An anti-disturbance control method for underwater robot manipulator based on deep reinforcement learning
By combining the expanded state observer and deep reinforcement learning algorithm, a limited time control strategy is designed to solve the problem of insufficient trajectory tracking accuracy and disturbance resistance in complex environments by underwater robotic robot arms, and efficient and robust control performance is achieved.
Patent Information
- Application Number
- CN202410569006.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-09
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-05-09
AI Technical Summary
The tracking accuracy of underwater robotic arms in complex underwater environments is not high and susceptible to water flow disturbances. The traditional control methods have limited effects and are difficult to achieve high-precision and efficient control.
Combining the expanded state observer and deep reinforcement learning algorithm, a limited time control strategy is designed to observe and compensate for uncertainty and external perturbations through ESO in real time, and to optimize control parameters online using deep reinforcement learning to achieve efficient control.
It significantly improves the control performance and anti-interference ability of the underwater robot robot arm system, ensures the task is completed quickly and accurately in complex environments, and improves operating efficiency and robustness.
Smart Images

Figure CN118466201B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of robot control technology, and in particular to an anti-disturbance control method for an underwater robot manipulator arm based on deep reinforcement learning. Background Art
[0002] Underwater manipulation has become an important means and development trend for the development of marine resources. When performing tasks such as deep-sea exploration, submarine pipeline maintenance, and underwater structure construction, underwater robot manipulators face extremely complex underwater environmental challenges. These environmental factors include but are not limited to water flow, water pressure, temperature changes, and underwater obstacles, so strict requirements are placed on the stability and precise control of underwater robot manipulators.
[0003] Traditional control methods, such as PID control and model predictive control, often have limited effectiveness in dealing with these uncertainties and nonlinear factors. In addition, the delay of underwater communication and the noise of sensors can also have an adverse effect on the control performance. Therefore, it is particularly important to develop a new control strategy that can adapt to these challenges and provide high-precision control.
[0004] In recent years, deep reinforcement learning has attracted attention due to its potential in dealing with high-dimensional continuous action spaces and uncertain environments. This paper aims to propose a new control method by combining finite-time control of an extended state observer and a deep reinforcement learning algorithm to improve the control performance and robustness of an underwater robot manipulator system in a complex underwater environment. Summary of the invention
[0005] The present invention provides an anti-disturbance control method for an underwater robot manipulator arm based on deep reinforcement learning, which solves the technical problems in the prior art that the trajectory tracking accuracy of the underwater robot manipulator arm system is low and it is susceptible to water flow disturbances. Compared with traditional control methods, the control method of the present invention can effectively improve the control performance, operating efficiency and anti-disturbance ability of the system.
[0006] The inventive concept of the present invention is: Since the underwater robot-manipulator system has the characteristics of strong coupling and high nonlinearity, the already complex underwater operations become more complicated and difficult to control. Therefore, the ESO is designed to observe, estimate and compensate for the uncertainty and external disturbances of the underwater robot manipulator system in real time, so as to reduce the influence of uncertainty and external disturbances on the control accuracy of the manipulator joints. In order to ensure that the underwater robot manipulator system can respond quickly and accurately reach the predetermined position and posture within a limited time, the present invention introduces a finite time control strategy, and combines the deep reinforcement learning algorithm to learn and optimize the control parameters online, so that the system can automatically adjust its action state according to environmental feedback and task requirements. By combining finite time control with a deep reinforcement learning algorithm and using the ESO module to provide state estimation, the present invention can achieve efficient control of the underwater robot manipulator system, improve the control performance of the system, and enable it to stably complete work in a complex underwater environment.
[0007] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: an anti-disturbance control method for an underwater robot manipulator arm based on deep reinforcement learning, comprising the following steps:
[0008] Step S1: Establishing a controlled object model of the underwater robot manipulator system;
[0009] Furthermore, in step S1, the underwater robot-mechanical arm includes an underwater robot body and mechanical arms located on the left and right sides of the underwater robot body, and the control model of the underwater robot mechanical arm is established as follows:
[0010]
[0011] Where q = [q1 q2 q3 q4 q5 q6] T , subscripts 1, 2 and 3 respectively represent joints 1, 2 and 3 of the left arm of the robot, and subscripts 4, 5 and 6 respectively represent the three joints of the right arm of the robot; q and Represent the joint angles and joint angular velocities of the left and right robotic arms respectively; is the controlled variable of the robot arm, where v1 = [uvw] T is the linear velocity of the underwater robot body, u, v, w represent the linear velocity of the underwater robot body along the three coordinate axes x, y, z in space, v2 = [lmp] T is the angular velocity of the underwater robot, l, m, and p represent the angular velocity of the hull around the three coordinate axes x, y, and z, respectively, and ζ and are the velocity and acceleration of the system respectively; is the inertia matrix including the additional mass term, is the Coriolis force and centripetal force matrix of the system, is the damping matrix, are the gravity and buoyancy terms, is the rotation matrix from the body coordinate system to the inertial coordinate system, is a generalized force, which is the control quantity of the robot manipulator, including the combined external force of the hull, the combined external torque of the hull and the joint torque of the manipulator. F ∑ =[X YZ] T is the total external force on the hull, M ∑ =[KMN] T is the total external moment of the hull, τ q =[τ q1 τ q2 τ q3 τ q4 τ q5 τ q6 ] T is the joint torque of the robot arm.
[0012] The model meets the following characteristics:
[0013] (1) The inertia matrix M of the system is symmetric and positive definite:
[0014] M=M T >0
[0015] In addition, the following inequality is satisfied:
[0016] λ min (M)≤||M||≤λ max (M)
[0017] where λ min (M), λ max (M) are the minimum and maximum eigenvalues of M respectively.
[0018] (2) is a skew-symmetric matrix:
[0019]
[0020] Right now
[0021]
[0022] and
[0023] ||C(a,b)c||≤C M ||b||||c||
[0024] C(a,α1b+α2c)=α1C(a,b)+α2C(a,c)
[0025] (3) The matrix D is a positive definite matrix:
[0026] D>0
[0027] and meet
[0028] ||D(q,a)-D(q,b)||≤D M ||ab||
[0029] Step S2: Design an Extended State Observer (ESO) to estimate and compensate for system uncertainties and external disturbances in real time;
[0030] Furthermore, in step S2, in order to improve the anti-interference capability of the underwater robot manipulator system, an extended state observer is designed to estimate the interference of the system and compensate the system.
[0031] S21: Convert the dynamic model of the underwater robot-manipulator system into a first-order system, as follows:
[0032]
[0033] S22: In view of the uncertainty parameters in the underwater robot manipulator system dynamics model and the difficulty in accurately measuring external disturbances in a complex underwater environment, the first-order system expression in equation (2) is simplified as follows:
[0034]
[0035] in, U=[U1 U2 U3 U4 U5 U6 U7 U8 U9 U 10 U 11 U 12 ] T ,f=[f1 f2f3 f4 f5 f6 f7 f8 f9 f 10 f 11 f 12 ] T , subscripts 1, 2 and 3 represent the position of the ship along the three coordinate axes x, y and z, respectively; subscripts 4, 5 and 6 represent the posture of the ship with the three coordinate axes x, y and z as the center, respectively; subscripts 7, 8 and 9 represent the three joints of the left manipulator, respectively; and subscripts 10, 11 and 12 represent the three joints of the right manipulator. U = M(q) -1 τ is the virtual control quantity, is the total disturbance effect of the system. At this time, the controlled variable of the robot arm is Can be controlled independently, where ζ iis the i-th system speed, i=1,2,3,4,5,6,7,8,9,10,11,12, U i is the ith virtual control quantity, i=1,2,3,4,5,6,7,8,9,10,11,12, f i is the total disturbance of the i-th system, i=1,2,3,4,5,6,7,8,9,10,11,12.
[0036] S23: Targeting ζ in S22 i ,i=1,2,3,4,5,6,7,8,9,10,11,12,design an extended state observer to observe the total disturbance f of the ith system i , i=1,2,3,4,5,6,7,8,9,10,11,12. The state equation is established as follows:
[0037]
[0038] S24: Define the observed value of the state quantity as z i1 、z i2 , z i1 For x i1 The observed value, z i2 f i The state observer is designed as follows:
[0039]
[0040] Among them, β i1 and β i2 is the observer gain, w i1 =z i1 -x i1 is the error value estimated by the state observer.
[0041] Step S3: design a finite time controller to achieve fast and accurate state tracking to ensure that the system reaches the desired state within a finite time;
[0042] Furthermore, in step S3, in order to ensure that the underwater robot arm can respond quickly and accurately reach a preset position and posture within a limited time, a finite time controller is designed to achieve fast and accurate state tracking.
[0043] S31: The i-th virtual control quantity is solved according to the following formula:
[0044]
[0045] Where Λ=k i e i , ei =x id -z i1 , x id is the controlled variable ζ i The setting value of
[0046] S32: Solve for the generalized force, the calculation formula is as follows:
[0047] τ=M(q)U (7)
[0048] Step S4: Using deep reinforcement learning, the control parameters are adaptively adjusted through the actor network and the critic network to respond to environmental changes.
[0049] Furthermore, in step S4, a deep reinforcement learning module is added to learn and optimize control parameters online, and the strategy is continuously improved through interaction with the environment to adapt to the complex underwater environment and task requirements. The details are as follows:
[0050] S41: Define the number n and range D of state s and action a, and set the learning parameters. According to the control target, the state is defined as the weighted sum of the tracking error of the velocity α i is an adjustable weighting parameter, and the control parameter β i1 , β i2 and k i , i = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, is the action to obtain the enhanced controller, that is, β i1 , β i2 and k i It can learn and update as the environment changes. The reward is combined with the weighted sum of tracking errors. The smaller the weighted sum of tracking errors, the greater the reward value. Learning parameters include learning rate, discount factor γ, variance of exploration noise, etc.
[0051] S42: Initialize the underwater robot-manipulator system, create an experience replay buffer to store experience during training, and design the actor network μ(s|θ μ ) and the critic network Q(s,a|θ Q ) architecture, including input layer, hidden layer and output layer, initialize network weight parameters, and create a target network whose parameters are a copy of the current network parameters θ Q′ ←θ Q ,θ μ′ ←θ μ . Where θ μ is the parameter of the actor network (strategy network), θ Q is the parameter of the critic network Q-network, μ(s|θ μ ) is the policy function, Q(s,a|θQ ) is the Q function.
[0052] S43: At each time step, the actor network outputs action instructions according to the current state to control the underwater robot manipulator to perform the corresponding action. In order to explore new actions, the sampling noise N is added to the actor strategy, that is, Observe the actual position of the underwater robot arm after executing the action, and then get the new state s t+1 and the reward value r r+1 .
[0053] S44: Change the current state s t , Action performed a t , the reward received t and the new state s t+1 Stored in the experience replay buffer, and a batch of experience samples P are randomly drawn from the experience replay buffer for training to break the temporal correlation between the data.
[0054] S45: Use the extracted experience samples to update the weights of the actor network and the critic network through the gradient descent method, by minimizing the loss To update the critic network, where yi = ri + γQ′ (si +1 ,μ′(si +1 |θ μ′ )|θ Q′ ), calculated using the target network with a discount factor γ, where P is the number of experience samples randomly drawn from the experience replay buffer. Using sampled policy gradient To update the actor network.
[0055] S46: Use a soft update strategy to regularly update the parameters of the current network to the target network to maintain the stability of the target network.
[0056] θ Q′ ←δθ Q +(1-δ)θ Q′ ,θ μ′ ←δθ μ +(1-δ)θ μ′
[0057] Where δ is the update coefficient, which is generally small.
[0058] S47: Repeat the steps of performing actions, observing results, storing experience, extracting samples and updating the network. When the predetermined number of training rounds is reached or the network performance reaches a satisfactory level, stop training. Otherwise, return to step S43 to continue the training process.
[0059] Compared with the prior art, the present invention has the following beneficial effects:
[0060] 1. The design of the finite-time control strategy enables the underwater robot-manipulator system to respond quickly and accurately reach the predetermined position and posture in a short time, significantly improving the efficiency of task execution.
[0061] 2. By using an extended state observer to estimate and compensate for system states and unknown disturbances in real time, the present invention significantly improves the system's resistance to external interference and parameter changes, ensuring the stability and reliability of control performance.
[0062] 3. The application of deep reinforcement learning algorithm enables the present invention to automatically adjust and optimize the control strategy according to the changes in the underwater environment and task requirements, effectively deal with the high-dimensional motion space and continuous motion problems in the control of the underwater robot manipulator system, and improve the system's adaptability and intelligence level.
[0063] 4. The control system of the present invention can complete underwater operation tasks more quickly, reduce energy consumption and operation time to a certain extent, and improve operation efficiency and economy. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.
[0065] Figure 1 Schematic diagram of the overall process of the anti-disturbance control method for the underwater robot manipulator arm based on deep reinforcement learning provided by the present invention.
[0066] Figure 2 Schematic diagram of the overall system structure of the underwater robot manipulator anti-disturbance control method based on deep reinforcement learning provided by the present invention.
[0067] Figure 3 A deep reinforcement learning algorithm flow chart of the underwater robot manipulator anti-disturbance control method based on deep reinforcement learning provided by the present invention.
[0068] Figure 4 Schematic diagram of the system speed curve of the underwater robot manipulator anti-disturbance control method based on deep reinforcement learning provided by the present invention.
[0069] Figure 5 Schematic diagram of the system acceleration curve of the anti-disturbance control method for the underwater robot manipulator arm based on deep reinforcement learning provided by the present invention.
[0070] Figure 6 Schematic diagram of the system torque curve of the anti-disturbance control method for the underwater robot manipulator arm based on deep reinforcement learning provided by the present invention. DETAILED DESCRIPTION
[0071] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.
[0072] Example 1
[0073] The technical solution provided in this embodiment is: an anti-disturbance control method for an underwater robot manipulator based on deep reinforcement learning, Figure 1 It is a flow chart of the control method, which includes the following four steps:
[0074] Step S1: Establishing a controlled object model of the underwater robot-manipulator system;
[0075] Step S2: Design an Extended State Observer (ESO) to perform real-time estimation and compensation of system uncertainties and external disturbances;
[0076] Step S3: design a finite time controller to achieve fast and accurate state tracking to ensure that the system reaches the desired state within a finite time;
[0077] Step S4: Using deep reinforcement learning, the control parameters are adaptively adjusted through the actor network and the critic network to respond to environmental changes.
[0078] See also Figures 1 to 3 , this embodiment provides an anti-disturbance control method for an underwater robot manipulator based on deep reinforcement learning, comprising the following steps:
[0079] Step S1: The underwater robot manipulator includes an underwater robot body and manipulators located on the left and right sides of the underwater robot body. The control model of the underwater robot manipulator is established as follows:
[0080]
[0081] Where q = [q1 q2 q3 q4 q5 q6] T , subscripts 1, 2 and 3 respectively represent joints 1, 2 and 3 of the left arm of the robot, and subscripts 4, 5 and 6 respectively represent the three joints of the right arm of the robot; q and Represent the joint angles and joint angular velocities of the left and right robotic arms respectively; is the controlled variable of the robot arm, where v1 = [uvw] T is the linear velocity of the underwater robot body, u, v, w represent the linear velocity of the underwater robot body along the three coordinate axes x, y, z in space, v2 = [lmp]T is the angular velocity of the underwater robot, l, m, and p represent the angular velocity of the hull around the three coordinate axes x, y, and z, respectively, and ζ and are the velocity and acceleration of the system respectively; is the inertia matrix including the additional mass term, is the Coriolis force and centripetal force matrix of the system, is the damping matrix, are the gravity and buoyancy terms, is the rotation matrix from the body coordinate system to the inertial coordinate system, is a generalized force, which is the control quantity of the robot manipulator, including the combined external force of the hull, the combined external torque of the hull and the joint torque of the manipulator. F ∑ =[X YZ] T is the total external force on the hull, M ∑ =[KMN] T is the total external moment of the hull, τ q =[τ q1 τ q2 τ q3 τ q4 τ q5 τ q6 ] T is the joint torque of the robot arm.
[0082] The model meets the following characteristics:
[0083] (1) The inertia matrix M of the system is symmetric and positive definite:
[0084] M=M T >0
[0085] In addition, the following inequality is satisfied:
[0086] λ min (M)≤||M||≤λ max (M)
[0087] where λ min (M), λ max (M) are the minimum and maximum eigenvalues of M respectively.
[0088] (2) is a skew-symmetric matrix:
[0089]
[0090] Right now
[0091]
[0092] and
[0093] ||C(a,b)c||≤C M ||b||||c||
[0094] C(a,α1b+α2c)=α1C(a,b)+α2C(a,c)
[0095] (3) The matrix D is a positive definite matrix:
[0096] D>0
[0097] and meet
[0098] ||D(q,a)-D(q,b)||≤D M ||ab||
[0099] Step S2: Design an Extended State Observer (ESO) to estimate and compensate for system uncertainties and external disturbances in real time;
[0100] Furthermore, in step S2, in order to improve the anti-interference capability of the underwater robot manipulator system, an extended state observer is designed to estimate the interference of the system and compensate the system.
[0101] S21: Convert the dynamic model of the underwater robot-manipulator system into a first-order system, as follows:
[0102]
[0103] S22: In view of the uncertainty parameters in the underwater robot-manipulator system dynamics model and the difficulty in accurately measuring external disturbances in a complex underwater environment, the first-order system expression in equation (2) is simplified as follows:
[0104]
[0105] in, U=[U1 U2 U3 U4 U5 U6 U7 U8 U9 U 10 U 11 U 12 ] T ,f=[f1 f2f3 f4 f5 f6 f7 f8 f9 f 10 f 11 f 12 ] T, subscripts 1, 2 and 3 represent the position of the ship along the three coordinate axes x, y and z, respectively; subscripts 4, 5 and 6 represent the posture of the ship with the three coordinate axes x, y and z as the center, respectively; subscripts 7, 8 and 9 represent the three joints of the left manipulator, respectively; and subscripts 10, 11 and 12 represent the three joints of the right manipulator. U = M(q) -1 τ is the virtual control quantity, is the total disturbance effect of the system. At this time, the controlled variable of the robot arm is Can be controlled independently, where ζ i is the i-th system speed, i=1,2,3,4,5,6,7,8,9,10,11,12, U i is the ith virtual control quantity, i=1,2,3,4,5,6,7,8,9,10,11,12, f i is the total disturbance of the i-th system, i=1,2,3,4,5,6,7,8,9,10,11,12.
[0106] S23: Targeting ζ in S22 i ,i=1,2,3,4,5,6,7,8,9,10,11,12,design an extended state observer to observe the total disturbance f of the ith system i , i=1,2,3,4,5,6,7,8,9,10,11,12. The state equation is established as follows:
[0107]
[0108] S24: Define the observed value of the state quantity as z i1 、z i2 , z i1 For x i1 The observed value, z i2 f i The state observer is designed as follows:
[0109]
[0110] Among them, β i1 and β i2 is the observer gain, w i1 =z i1 -x i1 is the error value estimated by the state observer.
[0111] Step S3: design a finite time controller to achieve fast and accurate state tracking to ensure that the system reaches the desired state within a finite time;
[0112] Furthermore, in step S3, in order to ensure that the underwater robot arm can respond quickly and accurately reach a preset position and posture within a limited time, a finite time controller is designed to achieve fast and accurate state tracking.
[0113] S31: The i-th virtual control quantity is solved according to the following formula:
[0114]
[0115] Where Λ=k i e i , e i =x id -z i1 , x id is the controlled variable ζ i The setting value of
[0116] S32: Solve for the generalized force, the calculation formula is as follows:
[0117] τ=M(q)U (7)
[0118] Step S4: Using deep reinforcement learning, the control parameters are adaptively adjusted through the actor network and the critic network to respond to environmental changes.
[0119] Furthermore, in step S4, a deep reinforcement learning module is added to learn and optimize control parameters online, and the strategy is continuously improved through interaction with the environment to adapt to complex underwater environments and mission requirements. The specific algorithm flow is as follows:
[0120] S41: Define the number n and range D of state s and action a, and set the learning parameters. According to the control target, the state is defined as the weighted sum of the tracking error of the velocity α i is an adjustable weighting parameter, and the control parameter β i1 , β i2 and k i (i=1,2,3,4,5,6,7,8,9,10,11,12) is the action to obtain the enhanced controller, that is, β i1 , β i2 and k i It can learn and update as the environment changes. The reward is combined with the weighted sum of tracking errors. The smaller the weighted sum of tracking errors, the greater the reward value. Learning parameters include learning rate, discount factor γ, variance of exploration noise, etc.
[0121] S42: Initialize the underwater robot arm system, create an experience replay buffer to store the experience during training, and design the actor network μ(s|θ μ) and the critic network Q(s,a|θ Q ) architecture, including input layer, hidden layer and output layer, initialize network weight parameters, and create a target network whose parameters are a copy of the current network parameters θ Q′ ←θ Q ,θ μ′ ←θ μ . Where θ μ is the parameter of the actor network (strategy network), θ Q is the parameter of the critic network (Q-network), μ(s|θ μ ) is the policy function, Q(s,a|θ Q ) is the Q function.
[0122] S43: At each time step, the actor network outputs action instructions according to the current state to control the underwater robot-manipulator to perform the corresponding action. In order to explore new actions, the sampling noise N is added to the actor strategy, that is, Observe the actual position of the underwater robot-manipulator after executing the action, and then get the new state s t+1 and the reward value r r+1 .
[0123] S44: Change the current state s t , Action performed a t , the reward received t and the new state s t+1 Stored in the experience replay buffer, and a batch of experience samples P are randomly drawn from the experience replay buffer for training to break the temporal correlation between the data.
[0124] S45: Use the extracted experience samples to update the weights of the actor network and the critic network through the gradient descent method. By minimizing the loss To update the critic network, where y i =r i +γQ′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ ), calculated using the target network with a discount factor γ, where P is the number of experience samples randomly drawn from the experience replay buffer. Using sampled policy gradient To update the actor network.
[0125] S46: Use a soft update strategy to regularly update the parameters of the current network to the target network to maintain the stability of the target network.
[0126] θ Q′ ←δθ Q+(1-δ)θ Q′ ,θ μ ←δθ μ +(1-δ)θ μ′
[0127] Where δ is the update coefficient, which is generally small.
[0128] S47: Repeat the steps of performing actions, observing results, storing experience, extracting samples and updating the network. When the predetermined number of training rounds is reached or the network performance reaches a satisfactory level, stop training. Otherwise, return to step S43 to continue the training process.
[0129] Example 2
[0130] join Figures 4 to 6 , Figure 4 The schematic diagram of the data curve of the hull linear velocity, hull angular velocity and joint velocity of the underwater robot manipulator system is shown in Figure 2. Figure 4 It can be seen that the hull linear velocity changes slightly in the early stage, the angular velocity oscillation amplitude gradually decreases, and the joint velocity changes continuously and smoothly over time in the whole process without jumps. The joint speed of the manipulator is accurately controlled, and the control algorithm makes the manipulator rotate to the predetermined position, while the hull quickly maintains unchanged. Figure 5 Figure 1 is a schematic diagram of the hull linear acceleration, hull angular acceleration and joint acceleration data curves of the underwater robot manipulator system. The hull linear acceleration and angular acceleration change slightly at 10s. Since the joint motor needs to overcome a large friction force when starting, the joint acceleration changes sharply at this time. After starting, the acceleration decreases. Figure 5 It can be seen that the acceleration can be quickly kept constant by adjusting the control strategy in real time to optimize the acceleration. Figure 6 Schematic diagram of the data curve of the hull linear force, hull moment and joint torque of the underwater robot manipulator system. Figure 6 It can be seen that the maximum torque of the joint appears in the motor starting stage, which is consistent with the change of the joint angular acceleration. Except for the change when the motor starts, there is no other obvious mutation point. The control program algorithm calculates and adjusts the force and torque in real time, so that the hull and the robotic arm can move quickly to the predetermined position.
[0131] The experimental results of Example 2 show that the anti-disturbance control method proposed in the present invention has higher tracking accuracy and faster response speed, enhances the robustness of the system and its anti-disturbance ability to complex underwater environments, and improves the overall operating efficiency of the system.
[0132] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for anti-disturbance control of underwater robot manipulator based on deep reinforcement learning, characterized in that: The following steps are involved: Step S1: Establish a control model of the underwater robot manipulator system; The step S1 is: The underwater robot manipulator includes the underwater robot body and the manipulators on the left and right sides of the underwater robot body. The control model of the underwater robot manipulator is established as follows: Where q = [q1 q2 q3 q4 q5 q6] T , subscripts 1, 2 and 3 respectively represent joints 1, 2 and 3 of the left mechanical arm of the underwater robot, and subscripts 4, 5 and 6 respectively represent the three joints of the right mechanical arm of the underwater robot; q and Represent the joint angles and joint angular velocities of the left and right robotic arms respectively; is the velocity vector of the underwater robot manipulator, where v1 = [uvw] T is the linear velocity of the underwater robot body, u, v, w represent the linear velocity of the underwater robot body along the three coordinate axes x, y, z in space, v2 = [lmp] T is the angular velocity of the underwater robot body, l, m, and p represent the rotation angular velocities of the underwater robot body around the three coordinate axes x, y, and z, respectively; is the inertia matrix including the additional mass term, is the Coriolis force and centripetal force matrix of the system, is the damping matrix, are the gravity and buoyancy terms, is the rotation matrix from the body coordinate system to the inertial coordinate system, is a generalized force, which is the control quantity of the underwater robot manipulator arm, including the underwater robot body's combined external force, the underwater robot body's combined external torque and the manipulator arm's joint torque. F ∑ =[XYZ] T is the combined external force of the underwater robot, M ∑ =[KMN] T is the total external torque of the underwater robot, τ q =[τ q1 τ q2 τ q3 τ q4 τ q5 τ q6 ] T is the joint torque of the robot arm; Step S2: Design an extended state observer, which estimates the uncertainty and external disturbance of the compensation system in real time; The step S2 comprises the following steps: S21: Convert the control model of the underwater robot manipulator system into a first-order system, as follows: S22: In view of the uncertainty parameters in the control model of the underwater robot manipulator system and the difficulty in accurately measuring external disturbances in a complex underwater environment, the first-order system expression in equation (2) is simplified as follows: in, U=[U1U2 U3 U4 U5 U6 U7 U8 U9 U 10 U 11 U 12 ] T ,f=[f1 f2 f3 f4f5 f6 f7 f8 f9 f 10 f 11 f 12 ] T , subscripts 1, 2 and 3 represent the positions of the underwater robot along the three coordinate axes x, y and z, respectively; subscripts 4, 5 and 6 represent the postures of the underwater robot with the three coordinate axes x, y and z as the center, respectively; subscripts 7, 8 and 9 represent the three joints of the left manipulator, respectively; subscripts 10, 11 and 12 represent the three joints of the right manipulator, respectively; U = M(q) -1 τ is the virtual control quantity, is the total disturbance of the system, and the controlled variable of the underwater robot arm is They are controlled independently, where ζ i is the i-th row element of ζ, i=1,2,3,4,5,6,7,8,9,10,11,12,U i is the ith virtual control quantity, i=1,2,3,4,5,6,7,8,9,10,11,12, f i is the total disturbance of the i-th system, i=1,2,3,4,5,6,7,8,9,10,11,12; S23: Targeting ζ in S22 i ,i=1,2,3,4,5,6,7,8,9,10,11,12,design an extended state observer to observe the total disturbance f of the ith system i , i=1,2,3,4,5,6,7,8,9,10,11,12, the state equation is established as follows: S24: Define the observed value of the state quantity as z i1 、z i2 , z i1 For x i1 The observed value, z i2 f i The observation value of , the state observer is designed as follows: Among them, β i1 and β i2 is the observer gain, w i1 =z i1 -x i1 is the error value estimated by the state observer; Step S3: design a finite time controller to achieve fast and accurate state tracking to ensure that the system reaches the desired state within a finite time; The step S3 comprises the following steps: S31: The i-th virtual control quantity is solved according to the following formula: Among them, Λ i =k i e i , e i =x id -z i1 , x id is the controlled variable ζ i The setting value of S32: Solve for the generalized force, the calculation formula is as follows: τ=M(q)U (7); Step S4: Using deep reinforcement learning, adaptively adjust control parameters through the actor network and the critic network to respond to environmental changes; The step S4 specifically comprises the following steps: S41: Define the number n and range D of state s and action a, and set the learning parameters. According to the control target, the state is defined as the weighted sum of the tracking error of the velocity. α i is an adjustable weighting parameter, and the control parameter β i1 , β i2 and k i , i = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, is the action to obtain the enhanced controller, that is, β i1 , β i2 and k i As the environment changes, learning and updating are achieved, and the reward value is weighted according to the tracking error. The value of is set. The smaller the weighted sum of tracking errors, the larger the reward value. The learning parameters include learning rate, discount factor γ, and variance of exploration noise. S42: Initialize the underwater robot arm system, create an experience replay buffer to store experience during training, design the actor network μ(s|θμ) and the critic network Q(s,a|θ Q ) architecture, including input layer, hidden layer and output layer, initialize network weight parameters, and create a target network whose parameters are a copy of the current network parameters θ Q′ ←θ Q ,θ μ′ ←θ μ ; where θ μ is the parameter of the actor network, θ Q is the parameter of the critic network, μ(s|θ μ ) is the policy function, Q(s,a|θ Q ) is the Q function; S43: At each time step, the actor network outputs action instructions according to the current state to control the underwater robot manipulator to perform the corresponding action. In order to explore new actions, the sampling noise N is added to the actor strategy, that is, Observe the actual position of the underwater robot arm after executing the action, and then get the new state s t+1 and the reward value r t+1 ; S44: Change the current state s t , Action performed a t , the reward received t and the new state s t+1 Stored in the experience replay buffer, and randomly extract a batch of experience samples P from the experience replay buffer for training to break the temporal correlation between the data; S45: Use the extracted experience samples to update the weights of the actor network and the critic network through the gradient descent method, by minimizing the loss To update the critic network, where y i =r i +γQ′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ ), calculated using the target network with a discount factor γ, P is the number of experience samples randomly drawn from the experience replay buffer, and the sampling policy gradient is used To update the actor network; S46: Use a soft update strategy to regularly update the parameters of the current network to the target network to maintain the stability of the target network; i Q′ ←sth Q +(1-δ)θ Q′ ,the μ′ ←d μ +(1-δ)θ μ′ Where δ is the update coefficient; S47: Repeat the steps of executing actions, observing results, storing experience, extracting samples and updating the network. When the predetermined number of training rounds is reached or the network performance reaches a satisfactory level, stop training, otherwise return to step S43 to continue the training process.
