Reinforced learning control method and device for input-saturated flexible double-connecting-rod mechanical arm

Abstract

The invention provides an input saturation flexible double-connecting-rod mechanical arm reinforcement learning control method and device, and relates to the technical field of robot intelligent control. The method comprises the steps that a hypothetical modal method is used for constructing a dynamic model of the flexible double-connecting-rod mechanical arm system; designing a reinforcement learning adaptive vibration controller with input saturation; and according to the reinforcement learning self-adaptive vibration controller with the input saturation, the flexible double-connecting-rod mechanical arm system is controlled, and an optimal control strategy is obtained. The invention provides a reinforcement learning algorithm with an auxiliary system for a flexible double-connecting-rod mechanical arm, and the auxiliary system is used for eliminating the influence of a saturated nonlinear function on the trajectory tracking of the mechanical arm. In addition, a reinforcement learning algorithm is designed to approach uncertain information of a flexible double-connecting-rod mechanical arm system, so that the flexible mechanical arm restrains elastic vibration while achieving trajectory tracking.

Description

Technical Field

[0001] The present invention relates to the technical field of robot intelligent control, and in particular to a reinforcement learning control method and device for a flexible double-link robot arm with input saturation. Background Art

[0002] The control research of flexible manipulators first needs to establish its dynamic model, which is usually described by PDE (Partial Differential Equation), which increases the complexity of control design. In order to simplify the complexity of the model, researchers often convert PDE into ODE (Ordinary Differential Equation) and reduce the dimension by methods such as assumed modal method.

[0003] Nonlinear saturation is a common problem in control systems, especially in actuators. The effect of input saturation is usually unavoidable. It requires sufficient input power, and the actuator input may be constrained by the amplitude and rate of the input torque. When the input of the actuator exceeds the maximum threshold, the actuator will maintain operation at the maximum value, which may lead to some unpredictable situations, such as increasing energy, which places high demands on the control design. Therefore, a compensation system needs to be added to compensate for the effect of saturation nonlinearity to ensure the stability of the flexible system.

[0004] In terms of control strategies, a variety of control strategies have been introduced in the field of flexible manipulator control in recent years, including PID control, adaptive control, fuzzy control, optimal control, and reinforcement learning control. In the face of model uncertainty, neural network control methods are widely used in controller design, which improves the learning ability and control effect of the system. The advantage of neural network control is that it reduces the dependence on partial differential equation models and reduces the demand for sensors. However, neural network control also faces the challenge of overfitting or underfitting, which affects its generalization ability and accuracy.

[0005] In the existing vibration control of flexible manipulators, it is difficult to achieve the ideal control effect and its flexible characteristics due to the elastic deformation of the flexible manipulator during movement, which makes its modeling more difficult. Secondly, the neural network may have overfitting or underfitting problems in the process of compensating for the uncertain information of the system, which will affect the accuracy of compensation. During the control process, an evaluation strategy is needed to evaluate the neural network and adjust the control torque in time to achieve better vibration suppression effect. In addition, in previous inventions, the influence of input saturation was not considered in the vibration suppression control of the flexible system. Since the nonlinear function of the actuator is unknown, ignoring the saturation nonlinearity will affect the control of the flexible manipulator system, making the system unreliable. Summary of the invention

[0006] In order to solve the problem that most flexible manipulator systems in the prior art use PDE for dynamic modeling, PDE modeling can describe flexible systems with high precision, but in practical applications, using PDE dynamic equations to build system models will make the controller design complicated. In traditional manipulator control, trajectory tracking is usually performed for rigid manipulators. However, for flexible manipulators, the influence of elastic vibration caused by its flexible structure on trajectory tracking must also be considered. In addition, neural network control is widely used in controller design due to its excellent learning ability, especially when dealing with model uncertainty and nonlinearity. However, neural networks may affect the compensation accuracy due to overfitting or underfitting, so new control strategies need to be designed to evaluate and adjust the control torque to optimize the vibration suppression effect. Due to the unknown nonlinear function of the actuator, ignoring the saturation nonlinearity will cause system vibration and reduce the technical problem of reliability. The embodiment of the present invention provides a reinforcement learning control method and device for a flexible double-link manipulator with input saturation. The technical solution is as follows:

[0007] On the one hand, a reinforcement learning control method for a flexible double-link manipulator with input saturation is provided. The method is implemented by a reinforcement learning control device for a flexible double-link manipulator with input saturation. The method comprises:

[0008] S1. Use the hypothetical modal method to construct the dynamic model of the flexible two-link robotic arm system.

[0009] S2. Design a reinforcement learning adaptive vibration controller with input saturation.

[0010] S3. According to the reinforcement learning adaptive vibration controller with input saturation, the flexible double-link robotic arm system is controlled to obtain the optimal control strategy.

[0011] Optionally, the dynamic model of the flexible double-link manipulator system in S1 is as shown in the following formula (1):

[0012]

[0013] Where J(Q) represents the inertia matrix, Q = [θ, l] T represents the state variables of the flexible two-link robotic arm system, represents the generalized coordinates that change with time, n1 represents the degrees of freedom of the first link, n2 represents the degrees of freedom of the second link, θ=[θ1,θ2] T represents the rotation angle of the flexible link, θ1 represents the rotation angle of the first link, θ2 represents the rotation angle of the second link, represents the second-order derivative of the state variable Q, represents the Coriolis matrix and the centripetal effect, represents the first-order derivative of the state variable Q, represents the stiffness matrix, represents the saturated nonlinear function of the torque τ, represents a set of real numbers, N represents the total degrees of freedom of the system, T represents the matrix transpose, and τ represents the torque.

[0014] Optionally, the design of a reinforcement learning adaptive vibration controller with input saturation in S2 includes:

[0015] S21. Design an evaluation neural network to evaluate the performance of the current control strategy.

[0016] S22. Design an execution neural network for generating control strategies. The execution neural network adopts a radial basis function neural network.

[0017] S23. Send the system information as input data to the evaluation neural network for processing to obtain the evaluation neural network output, and use the evaluation neural network output as the input of the execution neural network to guide the execution neural network to perform training.

[0018] S24. Construct a new Lyapunov function and prove that the closed-loop control system consisting of the dynamic model of the flexible two-link robotic arm system and the reinforcement learning adaptive vibration controller with input saturation has semi-globally consistent ultimate boundedness.

[0019] Optionally, the design evaluation neural network in S21 includes:

[0020] S211. Design a long-term cost function to quantitatively measure the performance of the control strategy within a preset time period.

[0021] Among them, the long-term cost function is shown in the following formula (2):

[0022]

[0023] In the formula, represents the optimal weight of evaluating the neural network, T represents matrix transposition, Z c represents the Gaussian basis function of the neural network, S c =f1=Q1-Q r , f1 represents the error calculated from the preset reference trajectory and the input state obtained by the flexible double-link robot system, Q1 represents the current state of the system, Q r represents the desired state of the system, ∈ c represents the approximation error of the evaluation neural network, t represents the lower bound of the integral, n represents the integral variable, ζ represents the discount factor of the future cost, and Υ(n) represents the instantaneous cost function.

[0024] S212. Design and evaluate the weight update rate of the neural network, as shown in the following formula (3):

[0025]

[0026] In the formula, Indicates the actual weight value of the neural network at the next moment, σ c >0 means evaluating the learning rate of the neural network. Indicates the actual weight value of the neural network at the current moment. Indicates S c The derivative of Indicates S c The gradient of , Υ(t) represents the instantaneous cost function.

[0027] Optionally, the design in S22 executes a neural network, including:

[0028] S221. Construct an auxiliary system to offset the effect of saturation input on vibration suppression.

[0029] S222. Based on the dynamic model of the flexible double-link robotic arm system, the initial control torque is derived in combination with the backstepping method.

[0030] S223. Use the execution neural network to approximate the unknown dynamic information of the dynamic model of the flexible double-link robotic arm system to obtain a new control torque.

[0031] Optionally, the auxiliary system in S221 is as shown in the following formula (4):

[0032]

[0033] In the formula, Indicates the value of the auxiliary variable at the next moment, K ξ >0, ξ represents the auxiliary variable, f2 represents the tracking error, T represents the matrix transpose, Δτ=S(τ)-τ, S(τ) represents the saturated nonlinear function of the torque τ,

[0034] Optionally, the new Lyapunov function in S24 is as shown in the following equation (5):

[0035]

[0036] Where V represents the Lyapunov function, f1 represents the tracking error, T represents the matrix transpose, f2 represents the tracking error, J represents the inertia matrix, ξ represents the auxiliary variable, represents the error of the action neural network weights, Represents the error in evaluating the neural network weights.

[0037] On the other hand, a reinforcement learning control device for a flexible double-link manipulator with input saturation is provided, and the device is applied to a reinforcement learning control method for a flexible double-link manipulator with input saturation. The device comprises:

[0038] A building block for constructing a dynamic model of a flexible two-link robotic arm system using the assumed modal method.

[0039] Design module for designing reinforcement learning adaptive vibration controllers with input saturation.

[0040] The output module is used to control the flexible two-link manipulator system according to the reinforcement learning adaptive vibration controller with input saturation to obtain the optimal control strategy.

[0041] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0042] In the present invention, a system modeling method of a flexible double-link manipulator under saturated nonlinear interference is considered, so that the system can more accurately describe the behavior of the flexible link system.

[0043] Compared with the flexible single-link robot arm, the present invention is verified on a flexible double-link robot arm with strong coupling and achieves good tracking and vibration suppression effects.

[0044] Aiming at the saturation nonlinear interference faced by FTLM, the present invention designs an auxiliary system, which can eliminate the influence of actuator saturation on the system and improve the robustness and tracking performance of the system.

[0045] The present invention takes into account the uncertainty of the FTLM system and designs a reinforcement learning control method with an auxiliary system to compensate for the uncertain information of the system. In addition, the method also has good self-learning ability, and can better obtain the most effective control strategy for the flexible system through iterative learning. Through a large number of experiments, it is proved that the proposed control method can effectively suppress the elastic vibration of the flexible link system and can accurately track the desired trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 It is a flow chart of a reinforcement learning control method for a flexible double-link manipulator with input saturation provided by an embodiment of the present invention;

[0048] Figure 2is a schematic structural diagram of a flexible double-link mechanical arm provided by an embodiment of the present invention;

[0049] Figure 3 is a block diagram of a control algorithm for a flexible double-link mechanical arm provided by an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of a FTLM experimental device provided in an embodiment of the present invention;

[0051] Figure 5 is a comparison chart of tracking trajectory performance of three control methods provided by an embodiment of the present invention;

[0052] Figure 6 is a comparison diagram of tracking errors of three control methods provided by an embodiment of the present invention;

[0053] Figure 7 is a comparison of elastic vibrations of three control methods provided by embodiments of the present invention;

[0054] Figure 8 is a control input comparison of three control methods provided by an embodiment of the present invention;

[0055] Fig. 9 It is a block diagram of a reinforcement learning control device for a flexible double-link robotic arm with input saturation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0057] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0058] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0059] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0060] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0061] The embodiment of the present invention provides a reinforcement learning control method for a flexible double-link mechanical arm with input saturation, which can be implemented by a reinforcement learning control device for a flexible double-link mechanical arm. Figure 1 The flow chart of the input saturated flexible double-link manipulator reinforcement learning control method shown in FIG. 1 is a flow chart of the input saturated flexible double-link manipulator reinforcement learning control method, and the processing flow of the method may include the following steps:

[0062] S1. Use the hypothetical modal method to construct the dynamic model of the flexible two-link robotic arm system.

[0063] In a feasible implementation, based on Hamilton's principle, the dynamic model of FTLM (Flexible Two Link Manipulaotr) is constructed by analyzing the moment of inertia, angular velocity, linear velocity, elastic vibration and other factors of the FTLM system. Figure 2 Figure 1 describes the notation used in mathematical modeling of the flexible manipulator. XOYZ is the inertial reference frame used to determine the position of the manipulator in space, and xoy is the rotational reference frame used to observe the elastic vibrations. i represents the torque of the i-th drive unit, ω i represents the elastic vibration of the ith link under the reference xoy, Y i (Q,t) represents the position of the i-th link, I oi represents the moment of inertia of the i-th connecting rod, I hi is the moment of inertia of the joint, B i represents the equivalent viscous damping coefficient, K si is the torsional stiffness constant.

[0064] Furthermore, the dynamic equation of the FTLM system with saturated nonlinearity is:

[0065]

[0066] Where, Q = [θ, l] T represents the state variables of the flexible two-link robotic arm system, represents the generalized coordinates that change with time, n1 represents the degrees of freedom of the first link, n2 represents the degrees of freedom of the second link, θ=[θ1,θ2] T represents the rotation angle of the flexible link, θ1 represents the rotation angle of the first link, θ2 represents the rotation angle of the second link, represents the second-order derivative of the state variable Q, the matrix J(Q) and represents the inertia matrix, Coriolis matrix and centripetal effect, and the size of each matrix is represents the first-order derivative of the state variable Q, represents the stiffness matrix, which is Design Matrix It is obliquely symmetrical. represents the saturated nonlinear function of the torque τ, represents a set of real numbers, N represents the total degrees of freedom of the system, T represents the matrix transpose, and τ represents the torque.

[0067] S2. Design a reinforcement learning adaptive vibration controller with input saturation.

[0068] In a feasible implementation, for the FTLM dynamic model constructed using the assumed modal method, the present invention proposes an adaptive controller based on an actor-critic network, which enables the FTLM to approximate the unknown system dynamics in real time through online learning, obtain the optimal control strategy, and reduce the design cost.

[0069] Furthermore, the present invention designs a control method with an auxiliary system based on an actor-critic strategy, and the algorithm has the reinforcement learning capability of the auxiliary system structure. The actor-critic algorithm combines strategy iteration and value iteration, wherein the actor network is responsible for generating the control strategy, and the critic network evaluates the performance of the current strategy, providing an iterative method for online learning: the actor network uses RBFNN (Radial Basis Function Neural Network) to gradually learn and improve from the interaction with the system environment, generating an appropriate control strategy, and the critic network can adjust the learning feedback based on the input of the reward or penalty signal, evaluate the performance of the current strategy through the state error of the current system, and provide feedback to improve the strategy, which can effectively handle dynamic uncertainty. In order to solve the influence of saturated nonlinear input on the control strategy, an auxiliary system is added to the reinforcement learning control strategy to offset the influence of saturated input on vibration suppression.

[0070] Figure 3 A visual representation of the control algorithm structure of the FTLM system based on reinforcement learning is provided. r The error f1 is calculated with the input state Q obtained by the FTLM system; the error is input into the execution network and the evaluation network respectively, so that the execution network and the evaluation network can be iteratively updated with each other, and then the auxiliary system is added to compensate for the influence of the saturated nonlinear function on the flexible system, so as to control the robot arm to move according to the preset trajectory.

[0071] Specifically, the above step S2 may include the following steps S21-S24:

[0072] S21. Design an evaluation neural network to evaluate the performance of the current control strategy.

[0073] The design evaluation neural network in S21 may include the following steps S211-S212:

[0074] S211. Design a long-term cost function to quantitatively measure the performance of the control strategy within a preset time period.

[0075] In a feasible implementation, RBFNN has the characteristics of estimating unknown nonlinear functions and self-learning ability. For a continuous function h(S), RBFNN can be expressed as h(S)=W T Z(S), where Z(S) = [z1(S),z2(S),…,z l (S)] is the Gaussian basis function of the neural network, W is the weight vector of the number of neural network nodes, η i represents the center of the Gaussian basis function. When the input S is close to η i When the radial basis function neural network is activated, the basis function will be activated, resulting in an increase in output. RBFNN can convert the compact set Any continuous function in can be estimated to any desired precision.

[0076] In the control strategy evaluation of neural networks, the reward or penalty signal brought by the input data plays a core role in fine-tuning the learning feedback. To this end, the present invention designs a long-term cost function to quantitatively measure the performance of the control strategy over a period of time. This cost function helps the neural network to adjust the control strategy in an orderly manner, thereby improving its performance. In this way, it can be ensured that the neural network can be continuously optimized when implementing the control strategy to achieve better results.

[0077] In order to achieve the purpose of optimizing the control strategy, the present invention designs a long-term cost function, the purpose is to find the optimal control that minimizes the cost. and Where S c =f1=Q1-Q r Based on these definitions, the long-term cost function is expressed as:

[0078]

[0079] In the formula, represents the optimal weight of evaluating the neural network, T represents matrix transposition, Z c represents the Gaussian basis function of the neural network, S c =f1=Q1-Q r , f1 represents the preset reference trajectory Qr The error calculated from the input state Q obtained by the FTLM system, Q1 represents the current state of the system, Q r represents the desired state of the system, ∈ c represents the approximation error of the evaluation neural network, t represents the lower bound of the integral, n represents the integral variable, ζ represents the discount factor of the future cost, and Υ(n) represents the instantaneous cost function.

[0080] Where ζ represents the discount factor of future costs. The instantaneous cost function Υ(t) can be expressed as:

[0081]

[0082] Where D and R are positive definite matrices, f1 represents the tracking error, and τ represents the torque. When the cost function is minimized, the vibration can be suppressed.

[0083] The critic network evaluates the cost function based on the control input, current trajectory and vibration information to determine whether the control torque is reasonable. Then, through the evaluation of the critic network, the actor network uses the radial basis function neural network to generate a new control torque to minimize the cost function, thereby ensuring the vibration suppression of the FTLM.

[0084] S212. Design and evaluate the weight update rate of the neural network, as shown in the following formula (4):

[0085]

[0086] In the formula, Indicates the actual weight value of the neural network at the next moment, σ c >0 means evaluating the learning rate of the neural network. Indicates the actual weight value of the neural network at the current moment. Indicates S c The derivative of Indicates S c The gradient of , Υ(t) represents the instantaneous cost function.

[0087] S22. Design an execution neural network for generating control strategies. The execution neural network adopts a radial basis function neural network.

[0088] Optionally, the design and execution of the neural network in S22 may include the following steps S221-S223:

[0089] S221. Construct an auxiliary system to offset the effect of saturation input on vibration suppression.

[0090] In a feasible implementation, an auxiliary system based on RL (Reinforcement Learning) controller is designed to solve the actuator constraint problem. The auxiliary system offsets the effect of actuator saturation on FTLM and improves the vibration suppression performance of the flexible system.

[0091] Specifically, in order to maintain the semi-global critical stability of the system, a specific virtual error term is added to the controller, and the execution neural network adjusts the control strategy in real time according to the system vibration error, tracking error, control torque at the previous moment, and feedback from the evaluation network, so as to optimize the vibration suppression and tracking performance of the system. However, in actual control systems, the system may be affected by various nonlinear inputs, among which saturation nonlinearity will inhibit the operation of the actuator and affect the control performance. The present invention offsets the influence of actuator saturation on the control system by constructing an auxiliary system. In this way, even in the presence of unknown saturation nonlinearity, the execution neural network can generate efficient control inputs, thereby enhancing the control performance of the system.

[0092] In order to eliminate the influence of the saturation function, the following auxiliary system ξ is designed as shown in the following formula (5):

[0093]

[0094] In the formula, Indicates the value of the auxiliary variable at the next moment, ξ represents the auxiliary variable, f2 represents the tracking error, T represents the matrix transpose, Δτ=S(τ)-τ, and S(τ) represents the saturated nonlinear function of the torque τ.

[0095] S222. According to the dynamic model of the flexible double-link manipulator system, the initial control torque is derived by combining the backstepping method:

[0096]

[0097] In the formula, τ represents the initial control torque, K2 represents a positive constant, λ1 represents the virtual control amount, Represents the first-order derivative of the virtual control variable λ1.

[0098] S223. Since the dynamic information of the robot is unknown, in order to solve this problem, the present invention uses an execution neural network to approximate the unknown dynamic information. Therefore, the new control torque is:

[0099]

[0100] in Represents the actual weight value of the neural network.

[0101] S23. In the control of the system, the system information is first used as input data and sent to the critic neural network for processing. The output of the critic neural network is then used as the input of the actor neural network to guide the actor network to train. This process is implemented based on the following formula:

[0102]

[0103] Then input the updated weights into the actor network for iterative update. a >0 is the learning rate.

[0104] S24. Construct a new Lyapunov function and prove that the closed-loop control system consisting of the dynamic model of the flexible two-link robotic arm system and the reinforcement learning adaptive vibration controller with input saturation has semi-globally consistent ultimate boundedness.

[0105] The new Lyapunov function is shown in the following formula (9):

[0106]

[0107] Where V represents the Lyapunov function, f1 represents the tracking error, T represents the matrix transpose, f2 represents the tracking error, J represents the inertia matrix, ξ represents the auxiliary variable, represents the error of the action neural network weights, Represents the error in evaluating the neural network weights.

[0108] By continuously deriving and transforming the function, the present invention can derive This conclusion proves that the designed closed-loop control system has semi-globally consistent ultimate boundedness.

[0109] S3. According to the reinforcement learning adaptive vibration controller with input saturation, the flexible double-link robotic arm system is controlled to obtain the optimal control strategy.

[0110] Experimental verification based on the proposed controller algorithm:

[0111] Experimental test platform introduction: In order to evaluate the performance and stability of the proposed algorithm, the present invention is experimentally verified on a flexible dual-link robotic arm system. A cost-effective simulation is achieved using a hardware-in-the-loop simulation test platform (HIL), which can integrate actual physical components for testing, thereby improving the reliability of the system and the quality of the design. Figure 4In the FTLM system shown, two DC motors are connected and driven at the connection of the link. The first stage uses a 3-inch wide flexible link, and the second stage uses a 1.5-inch wide flexible link. The digital angular position of the two precision brush DC motors Maxon 273759 and Maxon118752 is measured using a high-resolution orthogonal optical encoder with 1024 lines per revolution. In order to facilitate the application of the FTLM system, the present invention integrates QUARC and Simulink to connect the hardware and operation controller of the FTLM system. The data of the corresponding transformer and encoder are collected and processed using a data acquisition device (DAQ) and a power amplifier (AMPAQ), and then the algorithm is designed on the industrial computer to achieve precise control of the FTLM system. Due to the seamless integration of QUARC and Simulink, the Simulink model can run in real time on various targets (such as Windows and QNX). In addition, the experimental platform also includes other key components, including Hall effect sensors, joint limit switches, etc. The detailed basic parameters of the FTLM system are shown in Table 1.

[0112] Table 1

[0113]

[0114] Furthermore, experimental verification of the proposed control algorithm: In order to better observe the control effect of the proposed reinforcement learning control strategy with auxiliary system on the flexible two-link robotic arm system with input saturation, the present invention experiments with three control methods: PD control, NN control and RL control on the flexible two-link robotic arm system.

[0115] The present invention constructs PD control, neural network control and reinforcement learning control strategy with auxiliary system through Simulink toolbox of MATLAB. Among them, PD and neural network control are used to compare with the reinforcement learning control method with auxiliary system proposed in the present invention. The reference trajectory angle is set between 10° and 15°. Next, the parameter configuration of RL control strategy will be introduced in detail.

[0116] The control parameters of the reinforcement learning controller with auxiliary system are K1 = [28, 24] T , K2=[1.6,1.6] T ,α=50,S 1,max =0.45, S 1,min =-0.42, S 2,max =0.44, S 2,min =-0.4, ξ(0)=[0.01,0.01], K ξ =[1.8,1.8], η a =2,η c=0.5. In addition, the model parameters of the flexible double-link robot arm are designed as n1=n2=2, the actor neural network and the critic neural network have 256 nodes and 4 nodes respectively, and the learning rate of the neural network is selected as a and σ c They are 10 and 1 respectively.

[0117] The present invention verifies the PD control, NN control and RL control algorithms respectively. Figures 5 to 8 The experimental results of the FTLM for three different control methods are described. Subsequently, an in-depth evaluation based on these results will be presented.

[0118] Figure 5 and Figure 6 The experimental results of trajectory tracking and tracking error of the flexible dual-link manipulator are shown respectively. Figure 5 The experimental results of the two links show that the tracking performance under the RL strategy is significantly better than that of the NN and PD control methods. The maximum tracking error of the reinforcement learning control with an auxiliary system in the second half of the cycle is 8.2 degrees, and the maximum tracking errors of the PD control and NN control are both greater than the maximum tracking error of the RL control. For link 2, the maximum tracking error of the reinforcement learning control with an auxiliary system in the second half of the cycle is 10.1 degrees, and the maximum tracking errors of the PD control and NN control are both greater than the maximum tracking error of the RL control. Therefore, it can be concluded that the RL control with an auxiliary system designed for the flexible manipulator can have a smaller tracking error.

[0119] Figure 7 The experimental results of FTLM elastic vibration. For connecting rod 1, the elastic vibration of the reinforcement learning control with auxiliary system is maintained within 0.3 degrees in the second half of the cycle, and the elastic vibration of PD control and NN control is greater than that of RL control with auxiliary system. For connecting rod 2, the vibration of RL control is maintained within 0.75 degrees in the second half of the cycle, and the elastic vibration of PD control and NN control is greater than that of RL control with auxiliary system. In the whole vibration reduction process, RL control with auxiliary system has a good vibration suppression effect. Figure 8 The experimental results of the control input of the FTLM are shown. In the whole control process of the flexible two-link manipulator, the input of the RL control with auxiliary system is relatively small.

[0120] Therefore, compared with the reinforcement learning control strategy with auxiliary system designed for the flexible robotic arm, the control algorithm proposed in the present invention has good performance in tracking error, vibration suppression and steady-state error for the control of the flexible double-link robotic arm.

[0121] Existing flexible robotic arms face major challenges in modeling and vibration suppression. In order to reduce the complexity of the controller, the assumed modal method is used to construct the ordinary differential dynamics equation of the FTLM. The object of the present invention is to provide a control algorithm for a flexible two-link robotic arm that resists saturation input based on reinforcement learning. The saturated nonlinear function used in the present invention constrains the actuator. When the actuator input exceeds the maximum or minimum value of the saturation function, the actuator maintains operation at the maximum value, which may cause unpredictable situations. Therefore, an auxiliary system is designed to eliminate the influence of the saturated nonlinear function on the actuator. The present invention designs a reinforcement learning control method with online learning capability, which improves the vibration suppression capability of the flexible robotic arm system during trajectory tracking. Compared with the traditional neural network control framework, the controller proposed in the present invention has better self-learning and anti-saturation interference capabilities.

[0122] In the embodiment of the present invention, a system modeling method of a flexible double-link manipulator under saturated nonlinear interference is considered, so that the system can more accurately describe the behavior of the flexible link system.

[0123] Compared with the flexible single-link robot arm, the present invention is verified on a flexible double-link robot arm with strong coupling and achieves good tracking and vibration suppression effects.

[0124] Aiming at the saturation nonlinear interference faced by FTLM, the present invention designs an auxiliary system, which can eliminate the influence of actuator saturation on the system and improve the robustness and tracking performance of the system.

[0125] The present invention takes into account the uncertainty of the FTLM system and designs a reinforcement learning control method with an auxiliary system to compensate for the uncertain information of the system. In addition, the method also has good self-learning ability, and can better obtain the most effective control strategy for the flexible system through iterative learning. Through a large number of experiments, it is proved that the proposed control method can effectively suppress the elastic vibration of the flexible link system and can accurately track the desired trajectory.

[0126] Fig. 9 The block diagram of a device for reinforcing learning control of a flexible double-link manipulator with input saturation is shown according to an exemplary embodiment. The device is used for reinforcing learning control method of a flexible double-link manipulator with input saturation. Fig. 9 The device includes a construction module 310, a design module 320 and an output module 330.

[0127] in:

[0128] The construction module 310 is used to construct a dynamic model of the flexible double-link manipulator system by using an assumed modal method.

[0129] The design module 320 is used to design a reinforcement learning adaptive vibration controller with input saturation.

[0130] The output module 330 is used to control the flexible double-link manipulator system according to the reinforcement learning adaptive vibration controller with input saturation to obtain an optimal control strategy.

[0131] In the embodiment of the present invention, a system modeling method of a flexible double-link manipulator under saturated nonlinear interference is considered to better capture the behavior of the flexible link system. Compared with a flexible single-link manipulator, the present invention is verified on a flexible double-link manipulator with strong coupling and achieves good tracking and vibration suppression effects.

[0132] In order to deal with the saturated nonlinear interference suffered by FTLM, the present invention designs an auxiliary system, which can effectively eliminate the influence of actuator saturation on system performance and enhance the robustness and tracking accuracy of the system. At the same time, considering the uncertainty of the FTLM system, the present invention proposes a reinforcement learning method combined with an auxiliary system to compensate for the influence of system uncertainty. This method has the ability of self-learning and can obtain the most effective control strategy for the flexible system through a continuous iterative learning process. Finally, after comparative experimental tests, the present invention confirms that the proposed control method can effectively suppress the elastic vibration of the flexible linkage system and can accurately track the desired trajectory.

[0133] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0134] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0135] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0136] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0138] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0141] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0142] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A reinforcement learning control method for a flexible double-link manipulator with input saturation, characterized in that: The method comprises: S1. Use the hypothetical modal method to construct the dynamic model of the flexible double-link robotic arm system; S2. Design of a reinforcement learning adaptive vibration controller with input saturation; S3. Control the flexible double-link robotic arm system according to the reinforcement learning adaptive vibration controller with input saturation to obtain an optimal control strategy.

2. The input saturated flexible double-link robotic arm reinforcement learning control method according to claim 1, characterized in that: The dynamic model of the flexible double-link manipulator system in S1 is shown in the following formula (1): Where J(Q) represents the inertia matrix, Q = [θ, l] T represents the state variables of the flexible two-link robotic arm system, represents the generalized coordinates that change with time, n1 represents the degrees of freedom of the first link, n2 represents the degrees of freedom of the second link, θ=[θ1,θ2] T represents the rotation angle of the flexible link, θ1 represents the rotation angle of the first link, θ2 represents the rotation angle of the second link, represents the second-order derivative of the state variable Q, represents the Coriolis matrix and the centripetal effect, represents the first-order derivative of the state variable Q, represents the stiffness matrix, represents the saturated nonlinear function of the torque τ, represents a set of real numbers, N represents the total degrees of freedom of the system, T represents the matrix transpose, and τ represents the torque.

3. The input saturated flexible double-link robotic arm reinforcement learning control method according to claim 1, characterized in that: The design of the reinforcement learning adaptive vibration controller with input saturation in S2 includes: S21. Design an evaluation neural network to assess the performance of the current control strategy. S22, designing an execution neural network for generating a control strategy, wherein the execution neural network adopts a radial basis function neural network; S23, sending the system information as input data to the evaluation neural network for processing to obtain the evaluation neural network output, and using the evaluation neural network output as the input of the execution neural network to guide the execution neural network to perform training; S24. Construct a new Lyapunov function and verify that the closed-loop control system consisting of the dynamic model of the flexible two-link robotic arm system and the reinforcement learning adaptive vibration controller with input saturation has semi-globally consistent ultimate boundedness.

4. The input saturated flexible double-link mechanical arm reinforcement learning control method according to claim 3 is characterized in that: The design and evaluation neural network in S21 includes: S211. Design a long-term cost function to quantitatively measure the performance of the control strategy within a preset time period; Among them, the long-term cost function is shown in the following formula (2): In the formula, represents the optimal weight of evaluating the neural network, T represents matrix transposition, Z c represents the Gaussian basis function of the neural network, S c =f1=Q1-Q r , f1 represents the error calculated from the preset reference trajectory and the input state obtained by the flexible double-link robot system, Q1 represents the current state of the system, Q r represents the desired state of the system, ∈ c represents the approximation error of the evaluation neural network, t represents the lower bound of the integral, n represents the integral variable, represents the discount factor of future costs, γ(n) represents the instantaneous cost function; S212. Design and evaluate the weight update rate of the neural network, as shown in the following formula (3): In the formula, Indicates the actual weight value of the neural network at the next moment, σ c >0 indicates the learning rate of the evaluation neural network. Indicates the actual weight value of the neural network at the current moment. Indicates S c The derivative of Indicates S c The gradient of , γ(t) represents the instantaneous cost function.

5. The input saturated flexible double-link robotic arm reinforcement learning control method according to claim 3, characterized in that: The design of S22 implements a neural network, including: S221, construct an auxiliary system to offset the effect of saturation input on vibration suppression; S222, deriving an initial control torque according to the dynamic model of the flexible double-link manipulator system in combination with a backstepping method; S223. Use the execution neural network to approximate the unknown dynamic information of the dynamic model of the flexible double-link robotic arm system to obtain a new control torque.

6. The input saturated flexible double-link mechanical arm reinforcement learning control method according to claim 5, characterized in that: The auxiliary system in S221 is shown in the following formula (4): In the formula, Indicates the value of the auxiliary variable at the next moment, K ξ >0, ξ represents the auxiliary variable, f2 represents the tracking error, T represents the matrix transpose, Δτ=S(τ)-τ, S(τ) represents the saturated nonlinear function of the torque τ, 7. The input saturated flexible double-link mechanical arm reinforcement learning control method according to claim 3, characterized in that: The new Lyapunov function in S24 is shown in the following formula (5): Where V represents the Lyapunov function, f1 represents the tracking error, T represents the matrix transpose, f2 represents the tracking error, J represents the inertia matrix, ξ represents the auxiliary variable, represents the error of the action neural network weights, Represents the error in evaluating the neural network weights.

8. An input-saturated flexible double-link mechanical arm reinforcement learning control device, the input-saturated flexible double-link mechanical arm reinforcement learning control device is used to implement the input-saturated flexible double-link mechanical arm reinforcement learning control method according to any one of claims 1 to 7, characterized in that: The device comprises: A building module for building a dynamic model of a flexible two-link robotic arm system using an assumed modal method; Design module for designing reinforcement learning adaptive vibration controllers with input saturation; The output module is used to control the flexible double-link manipulator system according to the reinforcement learning adaptive vibration controller with input saturation to obtain an optimal control strategy.

9. The input saturated flexible double-link mechanical arm reinforcement learning control device according to claim 8, characterized in that: The dynamic model of the flexible double-link robotic arm system is shown in the following formula (1): Where J(Q) represents the inertia matrix, Q = [θ, l] T represents the state variables of the flexible two-link robotic arm system, represents the generalized coordinates that change with time, n1 represents the degrees of freedom of the first link, n2 represents the degrees of freedom of the second link, θ=[θ1,θ2] T represents the rotation angle of the flexible link, θ1 represents the rotation angle of the first link, θ2 represents the rotation angle of the second link, represents the second-order derivative of the state variable Q, represents the Coriolis matrix and the centripetal effect, represents the first-order derivative of the state variable Q, represents the stiffness matrix, represents the saturated nonlinear function of the torque τ, represents a set of real numbers, N represents the total degrees of freedom of the system, T represents the matrix transpose, and τ represents the torque.

10. The input saturated flexible double-link mechanical arm reinforcement learning control device according to claim 8, characterized in that: The design of the reinforcement learning adaptive vibration controller with input saturation includes: S21. Design an evaluation neural network to assess the performance of the current control strategy. S22, designing an execution neural network for generating a control strategy, wherein the execution neural network adopts a radial basis function neural network; S23, sending the system information as input data to the evaluation neural network for processing to obtain the evaluation neural network output, and using the evaluation neural network output as the input of the execution neural network to guide the execution neural network to perform training; S24. Construct a new Lyapunov function and verify that the closed-loop control system consisting of the dynamic model of the flexible two-link robotic arm system and the reinforcement learning adaptive vibration controller with input saturation has semi-globally consistent ultimate boundedness.

Images