Reinforcement learning control method and device for flexible double-link robotic arm with input saturation

Abstract

The present invention provides a reinforcement learning control method and device for a flexible two-link manipulator with input saturation, relating to the field of intelligent robot control technology. The method comprises: constructing a dynamic model of the flexible two-link manipulator system using a hypothetical modal method; designing a reinforcement learning adaptive vibration controller with input saturation; and controlling the flexible two-link manipulator system according to the reinforcement learning adaptive vibration controller with input saturation to obtain an optimal control strategy. The present invention proposes a reinforcement learning algorithm with an auxiliary system for the flexible two-link manipulator, utilizing the auxiliary system to eliminate the influence of saturated nonlinear functions on the manipulator's trajectory tracking. In addition, a reinforcement learning algorithm is designed to approximate the uncertainty information of the flexible two-link manipulator system, enabling the flexible manipulator to suppress elastic vibration while achieving trajectory tracking.

Description

Technical Field

[0001] The present invention relates to the field of robot intelligent control technology, and in particular to a reinforcement learning control method and device for an input-saturated flexible double-link robotic arm. Background Art

[0002] Control research for flexible robotic arms begins with establishing a dynamic model, typically using a partial differential equation (PDE), which complicates control design. To simplify the model, researchers often convert the PDE into an ordinary differential equation (ODE) and perform dimensionality reduction using methods such as the hypothetical modal method.

[0003] Nonlinear saturation is a common problem in control systems, particularly in actuators. While input saturation is often unavoidable, it requires sufficient input power, and the actuator input may be constrained by the magnitude and rate of the input torque. When the actuator input exceeds a maximum threshold, the actuator will operate at its maximum value, potentially leading to unpredictable behavior such as increasing energy, placing high demands on the control design. Therefore, a compensation system is required to compensate for the effects of saturation nonlinearity and ensure the stability of flexible systems.

[0004] In terms of control strategies, a variety of strategies have been introduced in the field of flexible robotic arm control in recent years, including PID control, adaptive control, fuzzy control, optimal control, and reinforcement learning control. To address model uncertainty, neural network control methods have been widely used in controller design, improving the system's learning capabilities and control effectiveness. The advantage of neural network control lies in its reduced reliance on partial differential equation models and reduced sensor requirements. However, neural network control also faces the challenge of overfitting and underfitting, which can affect its generalization ability and accuracy.

[0005] In the existing vibration control of flexible manipulators, it is difficult to achieve the ideal control effect and its flexible characteristics due to the elastic deformation of the flexible manipulator during movement, which makes its modeling more difficult. Secondly, the neural network may have overfitting or underfitting problems in the process of compensating for the uncertain information of the system, which will affect the accuracy of the compensation. During the control process, an evaluation strategy is needed to evaluate the neural network and adjust the control torque in time to achieve a better vibration suppression effect. In addition, in previous inventions, the influence of input saturation was not considered in the vibration suppression control of the flexible system. Since the nonlinear function of the actuator is unknown, ignoring the saturation nonlinearity will affect the control of the flexible manipulator system, making the system unreliable. Summary of the Invention

[0006] In order to solve the problem that most flexible manipulator systems in the prior art use PDE for dynamic modeling, PDE modeling can describe flexible systems with high precision. However, in practical applications, using PDE dynamic equations to build system models will make the controller design complicated. In traditional manipulator control, trajectory tracking is usually performed for rigid manipulators. However, for flexible manipulators, the influence of elastic vibration caused by its flexible structure on trajectory tracking must also be considered. In addition, neural network control is widely used in controller design due to its excellent learning ability, especially when dealing with model uncertainty and nonlinearity. However, neural networks may affect the compensation accuracy due to overfitting or underfitting, so new control strategies need to be designed to evaluate and adjust the control torque to optimize the vibration suppression effect. Since the nonlinear function of the actuator is unknown, ignoring the saturation nonlinearity will cause system vibration and reduce the technical problem of reliability. An embodiment of the present invention provides a reinforcement learning control method and device for a flexible two-link manipulator with input saturation. The technical solution is as follows:

[0007] In one aspect, a reinforcement learning control method for a flexible two-link manipulator with input saturation is provided. The method is implemented by a reinforcement learning control device for a flexible two-link manipulator with input saturation. The method comprises:

[0008] S1. Use the hypothetical modal method to construct a dynamic model of the flexible two-link robotic arm system.

[0009] S2. Design a reinforcement learning adaptive vibration controller with input saturation.

[0010] S3. Based on the reinforcement learning adaptive vibration controller with input saturation, the flexible double-link robotic arm system is controlled to obtain the optimal control strategy.

[0011] Optionally, the dynamic model of the flexible double-link manipulator system in S1 is as shown in the following formula (1):

[0012]

[0013] Where J(Q) represents the inertia matrix, Q = [θ, l] T represents the state variables of the flexible two-link robotic arm system, represents the generalized coordinates that change with time, n1 represents the degree of freedom of the first link, n2 represents the degree of freedom of the second link, and θ = [θ1, θ2] T represents the rotation angle of the flexible link, θ1 represents the rotation angle of the first link, θ2 represents the rotation angle of the second link, represents the second-order derivative of the state variable Q, represents the Coriolis matrix and centripetal effect, represents the first-order derivative of the state variable Q, represents the stiffness matrix, represents the saturated nonlinear function of the torque τ, represents a set of real numbers, N represents the total degrees of freedom of the system, T represents the matrix transpose, and τ represents the torque.

[0014] Optionally, the design of a reinforcement learning adaptive vibration controller with input saturation in S2 includes:

[0015] S21. Design an evaluation neural network to evaluate the performance of the current control strategy.

[0016] S22. Design an execution neural network for generating a control strategy. The execution neural network adopts a radial basis function neural network.

[0017] S23. Send the system information as input data to the evaluation neural network for processing to obtain the evaluation neural network output, and use the evaluation neural network output as the input of the execution neural network to guide the execution neural network to perform training.

[0018] S24. Construct a new Lyapunov function and prove that the closed-loop control system consisting of the dynamic model of a flexible two-link robotic arm system and a reinforcement learning adaptive vibration controller with input saturation has semi-globally consistent ultimate boundedness.

[0019] Optionally, the design evaluation neural network in S21 includes:

[0020] S211. Design a long-term cost function to quantitatively measure the performance of the control strategy within a preset time period.

[0021] Among them, the long-term cost function is shown in the following formula (2):

[0022]

[0023] Where, Represents the optimal weight of the evaluation neural network, T represents the matrix transpose, Z c represents the Gaussian basis function of the neural network, S c =f1=Q1-Q r , f1 represents the error calculated from the preset reference trajectory and the input state obtained by the flexible double-link manipulator system, Q1 represents the current state of the system, Q r represents the expected state of the system, ∈ c Represents the approximation error of the evaluation neural network, t represents the lower bound of the integral, n represents the integral variable, ζ represents the discount factor of future costs, and Υ(n) represents the instantaneous cost function.

[0024] S212. Design and evaluate the weight update rate of the neural network, as shown in the following formula (3):

[0025]

[0026] Where, Indicates the actual weight value of the neural network at the next moment, σ c >0 indicates the learning rate of the evaluation neural network, Indicates the actual weight value of the neural network at the current moment. Indicates S c The derivative of Indicates S c The gradient of , Υ(t) represents the instantaneous cost function.

[0027] Optionally, the design in S22 executes a neural network, including:

[0028] S221. Construct an auxiliary system to offset the effect of saturation input on vibration suppression.

[0029] S222. Based on the dynamic model of the flexible double-link manipulator system, the initial control torque is derived in combination with the backstepping method.

[0030] S223. Use the execution neural network to approximate the unknown dynamic information of the dynamic model of the flexible two-link manipulator system to obtain a new control torque.

[0031] Optionally, the auxiliary system in S221 is as shown in the following formula (4):

[0032]

[0033] Where, Indicates the value of the auxiliary variable at the next moment, K ξ >0, ξ represents the auxiliary variable, f2 represents the tracking error, T represents the matrix transpose, Δτ=S(τ)-τ, S(τ) represents the saturated nonlinear function of the torque τ,

[0034] Optionally, the new Lyapunov function in S24 is as shown in the following equation (5):

[0035]

[0036] Where V represents the Lyapunov function, f1 represents the tracking error, T represents the matrix transpose, f2 represents the tracking error, J represents the inertia matrix, ξ represents the auxiliary variable, represents the error of the action neural network weights, Represents the error in evaluating the neural network weights.

[0037] On the other hand, a reinforcement learning control device for a flexible double-link manipulator with input saturation is provided. The device is applied to a reinforcement learning control method for a flexible double-link manipulator with input saturation. The device comprises:

[0038] A building block for constructing a dynamic model of a flexible two-link robotic arm system using the assumed modal method.

[0039] Design module for designing reinforcement learning adaptive vibration controllers with input saturation.

[0040] The output module is used to control the flexible two-link robotic arm system according to the reinforcement learning adaptive vibration controller with input saturation to obtain the optimal control strategy.

[0041] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0042] In the present invention, a system modeling method of a flexible double-link manipulator under saturated nonlinear interference is considered, so that the system can more accurately describe the behavior of the flexible link system.

[0043] Compared with the flexible single-link manipulator, the present invention has been verified on a flexible double-link manipulator with strong coupling and achieved good tracking and vibration suppression effects.

[0044] In response to the saturation nonlinear interference faced by FTLM, the present invention designs an auxiliary system that can eliminate the influence of actuator saturation on the system and improve the robustness and tracking performance of the system.

[0045] This paper considers the uncertainty of the FTLM system and designs a reinforcement learning control method with an auxiliary system to compensate for the system's uncertainties. Furthermore, this method exhibits excellent self-learning capabilities, enabling iterative learning to better determine the most effective control strategy for the flexible system. Extensive experiments demonstrate that the proposed control method can effectively suppress the elastic vibration of the flexible linkage system and accurately track the desired trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 This is a flow chart of a reinforcement learning control method for a flexible double-link manipulator with input saturation provided by an embodiment of the present invention;

[0048] Figure 2is a schematic structural diagram of a flexible double-link robotic arm provided by an embodiment of the present invention;

[0049] Figure 3 This is a block diagram of a control algorithm for a flexible double-link robotic arm provided by an embodiment of the present invention;

[0050] Figure 4 Schematic diagram of the FTLM experimental equipment provided by an embodiment of the present invention;

[0051] Figure 5 1 is a comparison chart of the tracking trajectory performance of the three control methods provided by the embodiments of the present invention;

[0052] Figure 6 2 is a comparison diagram of tracking errors of three control methods provided by an embodiment of the present invention;

[0053] Figure 7 is a comparison of elastic vibrations of three control methods provided by embodiments of the present invention;

[0054] Figure 8 is a comparison of control inputs of the three control methods provided by the embodiments of the present invention;

[0055] Figure 9 This is a block diagram of an input-saturated flexible double-link robotic arm reinforcement learning control device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0057] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0058] In the embodiments of the present invention, the terms "image" and "picture" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, their intended meanings are the same. The terms "of," "corresponding," and "corresponding" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, their intended meanings are the same.

[0059] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0060] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0061] The embodiment of the present invention provides a reinforcement learning control method for a flexible double-link manipulator with input saturation, which can be implemented by a reinforcement learning control device for a flexible double-link manipulator. Figure 1 The flowchart of the reinforcement learning control method for a flexible two-link manipulator with input saturation is shown. The processing flow of the method may include the following steps:

[0062] S1. Use the hypothetical modal method to construct a dynamic model of the flexible two-link robotic arm system.

[0063] In a feasible implementation, based on Hamilton's principle, the dynamic model of FTLM (Flexible Two Link Manipulaotr) is constructed by analyzing the moment of inertia, angular velocity, linear velocity, elastic vibration and other factors of the FTLM system. Figure 2 As shown in Figure 1, it describes the symbols used in the mathematical modeling of the flexible manipulator. XOYZ is the inertial reference frame used to determine the position of the manipulator in space, and xoy is the rotational reference frame used to observe elastic vibrations. i represents the torque of the i-th drive unit, ω i represents the elastic vibration of the i-th link under the reference xoy, Y i (Q,t) represents the position of the i-th link, I oi represents the moment of inertia of the i-th connecting rod, I hi is the moment of inertia of the joint, B i represents the equivalent viscous damping coefficient, K si is the torsional stiffness constant.

[0064] Furthermore, the dynamic equation of the FTLM system with saturated nonlinearity is:

[0065]

[0066] Where Q = [θ, l] T represents the state variables of the flexible two-link robotic arm system, represents the generalized coordinates that change with time, n1 represents the degree of freedom of the first link, n2 represents the degree of freedom of the second link, and θ = [θ1, θ2] T represents the rotation angle of the flexible link, θ1 represents the rotation angle of the first link, θ2 represents the rotation angle of the second link, represents the second-order derivative of the state variable Q, the matrix J(Q) and represents the inertia matrix, Coriolis matrix and centripetal effect, and the size of each matrix is represents the first-order derivative of the state variable Q, Represents the stiffness matrix, which is represented by Design Matrix It is obliquely symmetrical. represents the saturated nonlinear function of the torque τ, represents a set of real numbers, N represents the total degrees of freedom of the system, T represents the matrix transpose, and τ represents the torque.

[0067] S2. Design a reinforcement learning adaptive vibration controller with input saturation.

[0068] In a feasible implementation, for the FTLM dynamic model constructed using the hypothetical modal method, the present invention proposes an adaptive controller based on the actor-critic network, which enables the FTLM to approximate the unknown system dynamics in real time through online learning, obtain the optimal control strategy, and reduce the design cost.

[0069] Furthermore, the present invention designs a control method with an auxiliary system based on an actor-critic strategy. This algorithm has the reinforcement learning capability of the auxiliary system structure. The actor-critic algorithm combines policy iteration and value iteration, wherein the actor network is responsible for generating the control strategy and the critic network evaluates the performance of the current strategy, providing an iterative method for online learning: the actor network uses RBFNN (Radial Basis Function Neural Network) to gradually learn and improve from the interaction with the system environment to generate an appropriate control strategy. The critic network can adjust the learning feedback based on the input of the reward or penalty signal, evaluate the performance of the current strategy through the state error of the current system, and provide feedback to improve the strategy. This method can effectively handle dynamic uncertainty. In order to address the impact of saturated nonlinear input on the control strategy, an auxiliary system is added to the reinforcement learning control strategy to offset the impact of saturated input on vibration suppression.

[0070] Figure 3 A visual representation of the control algorithm structure of the FTLM system based on reinforcement learning is provided. r The error f1 is calculated with the input state Q obtained by the FTLM system; the error is input into the execution network and the evaluation network respectively, so that the execution network and the evaluation network can be iteratively updated with each other, and then the auxiliary system is added to compensate for the influence of the saturated nonlinear function on the flexible system, thereby controlling the robot arm to move according to the preset trajectory.

[0071] Specifically, the above step S2 may include the following steps S21-S24:

[0072] S21. Design an evaluation neural network to evaluate the performance of the current control strategy.

[0073] The design evaluation neural network in S21 may include the following steps S211-S212:

[0074] S211. Design a long-term cost function to quantitatively measure the performance of the control strategy within a preset time period.

[0075] In a feasible implementation, RBFNN has the characteristics of estimating unknown nonlinear functions and self-learning ability. For a continuous function h(S), RBFNN can be expressed as h(S)=W T Z(S), where Z(S)=[z1(S),z2(S),…,z l (S)] is the Gaussian basis function of the neural network, W is the weight vector of the number of neural network nodes, η i Represents the center of the Gaussian basis function. When the input S is close to η i When the radial basis function neural network is activated, the basis function will be activated, resulting in an increase in output. RBFNN can convert the compact set Any continuous function within can be estimated to any desired precision.

[0076] In evaluating control strategies in neural networks, the reward or penalty signals generated by input data play a central role in fine-tuning learning feedback. To this end, the present invention designs a long-term cost function to quantitatively measure control strategy performance over time. This cost function helps the neural network systematically adjust the control strategy, thereby improving its performance. In this way, the neural network can ensure continuous optimization when implementing the control strategy, achieving better results.

[0077] In order to achieve the goal of optimizing the control strategy, the present invention designs a long-term cost function, the purpose of which is to find the optimal control that minimizes the cost. and Among them S c =f1=Q1-Q r Based on these definitions, the long-term cost function is expressed as:

[0078]

[0079] Where, Represents the optimal weight of the evaluation neural network, T represents the matrix transpose, Z c represents the Gaussian basis function of the neural network, S c =f1=Q1-Q r , f1 represents the preset reference trajectory Qr The error calculated from the input state Q obtained by the FTLM system, Q1 represents the current state of the system, Q r represents the expected state of the system, ∈ c Represents the approximation error of the evaluation neural network, t represents the lower bound of the integral, n represents the integral variable, ζ represents the discount factor of future costs, and Υ(n) represents the instantaneous cost function.

[0080] Where ζ represents the discount factor for future costs. The instantaneous cost function Υ(t) can be expressed as:

[0081]

[0082] Where D and R are positive definite matrices, f1 represents the tracking error, and τ represents the torque. When the cost function is minimized, the vibration can be suppressed.

[0083] The critic network evaluates the cost function based on the control input, current trajectory, and vibration information to determine whether the control torque is reasonable. Then, based on the critic network's evaluation, the actor network uses a radial basis function neural network to generate a new control torque to minimize the cost function, thereby ensuring vibration suppression of the FTLM.

[0084] S212. Design and evaluate the weight update rate of the neural network, as shown in the following formula (4):

[0085]

[0086] Where, Indicates the actual weight value of the neural network at the next moment, σ c >0 indicates the learning rate of the evaluation neural network, Indicates the actual weight value of the neural network at the current moment. Indicates S c The derivative of Indicates S c The gradient of , Υ(t) represents the instantaneous cost function.

[0087] S22. Design an execution neural network for generating a control strategy. The execution neural network adopts a radial basis function neural network.

[0088] Optionally, the design and execution of the neural network in S22 may include the following steps S221-S223:

[0089] S221. Construct an auxiliary system to offset the effect of saturation input on vibration suppression.

[0090] In one feasible implementation, an auxiliary system based on RL (Reinforcement Learning) controller is designed to address the actuator constraint problem. This auxiliary system offsets the impact of actuator saturation on the FTLM and improves the vibration suppression performance of the flexible system.

[0091] Specifically, to maintain the system's semi-global critical stability, a specific virtual error term is added to the controller. The execution neural network adjusts the control strategy in real time based on the system's vibration error, tracking error, the control torque at the previous moment, and feedback from the evaluation network to optimize the system's vibration suppression and tracking performance. However, in actual control systems, the system may be affected by various nonlinear inputs, among which saturation nonlinearity can inhibit the operation of the actuator and affect control performance. The present invention offsets the impact of actuator saturation on the control system by constructing an auxiliary system. In this way, even in the presence of unknown saturation nonlinearity, the execution neural network can generate efficient control inputs, thereby enhancing the system's control performance.

[0092] In order to eliminate the influence of the saturation function, the following auxiliary system ξ is designed, as shown in the following formula (5):

[0093]

[0094] Where, Indicates the value of the auxiliary variable at the next moment, ξ represents the auxiliary variable, f2 represents the tracking error, T represents the matrix transpose, Δτ = S(τ) - τ, and S(τ) represents the saturated nonlinear function of the torque τ.

[0095] S222. Based on the dynamic model of the flexible double-link manipulator system and the backstepping method, the initial control torque is derived as:

[0096]

[0097] In the formula, τ represents the initial control torque, K2 represents a positive constant, λ1 represents the virtual control amount, Represents the first-order derivative of the virtual control variable λ1.

[0098] S223. Since the dynamic information of the robot is unknown, in order to solve this problem, the present invention uses an execution neural network to approximate the unknown dynamic information. Therefore, the new control torque is:

[0099]

[0100] in Indicates the actual weight values ​​of the executed neural network.

[0101] S23. In the control of the system, the system information is first used as input data and fed into the critic neural network for processing. The output of the critic neural network is then used as the input of the actor neural network to guide the training of the actor network. This process is implemented based on the following formula:

[0102]

[0103] Then input the updated weights into the actor network for iterative update. a >0 is the learning rate.

[0104] S24. Construct a new Lyapunov function and prove that the closed-loop control system consisting of the dynamic model of a flexible two-link robotic arm system and a reinforcement learning adaptive vibration controller with input saturation has semi-globally consistent ultimate boundedness.

[0105] Among them, the new Lyapunov function is shown in the following formula (9):

[0106]

[0107] Where V represents the Lyapunov function, f1 represents the tracking error, T represents the matrix transpose, f2 represents the tracking error, J represents the inertia matrix, ξ represents the auxiliary variable, represents the error of the action neural network weights, Represents the error in evaluating the neural network weights.

[0108] By continuously derivatizing and transforming the function, the present invention can derive This conclusion proves that the designed closed-loop control system has semi-globally consistent ultimate boundedness.

[0109] S3. Based on the reinforcement learning adaptive vibration controller with input saturation, the flexible double-link robotic arm system is controlled to obtain the optimal control strategy.

[0110] Experimental verification based on the proposed controller algorithm:

[0111] Introduction to the experimental test platform: In order to evaluate the performance and stability of the proposed algorithm, the present invention was experimentally verified on a flexible dual-link robotic arm system. A hardware-in-the-loop simulation test platform (HIL) was used to achieve cost-effective simulation. This platform can integrate actual physical components for testing, thereby improving the reliability of the system and the quality of the design. Figure 4The illustrated FTLM system features two DC motors connected and driven at the link junction. The first stage utilizes a 3-inch-wide flexible link, while the second stage utilizes a 1.5-inch-wide flexible link. The digital angular position of the two precision brushed DC motors, Maxon 273759 and Maxon 118752, is measured using a high-resolution quadrature optical encoder with 1024 lines per revolution. To facilitate the application of the FTLM system, this paper integrates QUARC and Simulink to connect the FTLM system hardware and operational controller. Data from the corresponding transformers and encoders is collected and processed using a data acquisition device (DAQ) and a power amplifier (AMPAQ). Algorithms are then designed on an industrial computer to achieve precise control of the FTLM system. Due to the seamless integration of QUARC and Simulink, the Simulink model can run in real time on various targets, such as Windows and QNX. The experimental platform also includes other key components, including Hall-effect sensors and joint limit switches. Detailed basic parameters of the FTLM system are shown in Table 1.

[0112] Table 1

[0113]

[0114] Furthermore, experimental verification of the proposed control algorithm: In order to better observe the control effect of the proposed reinforcement learning control strategy with auxiliary system on the flexible two-link robotic arm system with input saturation, the present invention experiments on the three control methods of PD control, NN control and RL control on the flexible two-link robotic arm system.

[0115] This paper uses MATLAB's Simulink toolbox to construct PD control, neural network control, and reinforcement learning control strategies with auxiliary systems. PD and neural network control are compared with the reinforcement learning control method with auxiliary systems proposed in this paper. The reference trajectory angle is set between 10° and 15°. Next, the parameter configuration of the RL control strategy will be detailed.

[0116] The control parameters of the reinforcement learning controller with auxiliary system are K1 = [28, 24] T , K2=[1.6,1.6] T ,α=50,S 1,max =0.45, S 1,min =-0.42, S 2,max =0.44, S 2,min =-0.4, ξ(0)=[0.01,0.01], K ξ =[1.8,1.8], η a =2,η c=0.5. In addition, the flexible double-link manipulator model parameters are designed as n1=n2=2, the actor neural network and the critic neural network have 256 nodes and 4 nodes respectively, and the learning rate of the neural network is selected as a and σ c They are 10 and 1 respectively.

[0117] The present invention verifies the PD control, NN control and RL control algorithms respectively. Figures 5 to 8 The experimental results of the FTLM with three different control methods are described. Subsequently, an in-depth evaluation based on these results will be conducted.

[0118] Figure 5 and Figure 6 The experimental results of trajectory tracking and tracking error of the flexible dual-link manipulator are shown respectively. Figure 5 The experimental results for both links show that the tracking performance of the RL strategy significantly outperforms the NN and PD control methods. The maximum tracking error of the reinforcement learning control with the auxiliary system in the second half of the cycle was 8.2 degrees, and the maximum tracking errors of both PD and NN control were larger than those of the RL control. For link 2, the maximum tracking error of the reinforcement learning control with the auxiliary system in the second half of the cycle was 10.1 degrees, and the maximum tracking errors of both PD and NN control were larger than those of RL control. This indicates that RL control with auxiliary systems designed for flexible manipulators can achieve smaller tracking errors.

[0119] Figure 7 The experimental results of the FTLM's elastic vibration are shown below. For link 1, reinforcement learning control with an auxiliary system maintains elastic vibration within 0.3 degrees in the second half of the cycle. Both PD and NN control achieve greater elastic vibrations than RL control with an auxiliary system. For link 2, RL control maintains vibration within 0.75 degrees in the second half of the cycle. Both PD and NN control achieve greater elastic vibrations than RL control with an auxiliary system. Throughout the entire vibration reduction process, RL control with an auxiliary system demonstrates excellent vibration suppression. Figure 8 The experimental results of the control input of the FTLM are shown. During the entire control process of the flexible two-link manipulator, the input of the RL control with auxiliary system is relatively small.

[0120] Therefore, compared with the reinforcement learning control strategy with auxiliary system designed for the flexible robotic arm, the control algorithm proposed in this invention has good performance in tracking error, vibration suppression and steady-state error for the control of the flexible two-link robotic arm.

[0121] Existing flexible robotic arms face major challenges in modeling and vibration suppression. In order to reduce the complexity of the controller, the assumed modal method is used to construct the ordinary differential dynamics equations of the FTLM. The purpose of the present invention is to provide a control algorithm for a flexible two-link robotic arm that is resistant to saturation input based on reinforcement learning. The saturation nonlinear function used in the present invention constrains the actuator. When the actuator input exceeds the maximum or minimum value of the saturation function, the actuator maintains operation at the maximum value, which may cause unpredictable situations. Therefore, an auxiliary system is designed to eliminate the influence of the saturation nonlinear function on the actuator. The present invention designs a reinforcement learning control method with online learning capability, which improves the vibration suppression capability of the flexible robotic arm system during trajectory tracking. Compared with the traditional neural network control framework, the controller proposed in the present invention has better self-learning and anti-saturation interference capabilities.

[0122] In the embodiment of the present invention, a system modeling method of a flexible double-link manipulator under saturated nonlinear interference is considered, so that the system can more accurately describe the behavior of the flexible link system.

[0123] Compared with the flexible single-link manipulator, the present invention has been verified on a flexible double-link manipulator with strong coupling and achieved good tracking and vibration suppression effects.

[0124] In response to the saturation nonlinear interference faced by FTLM, the present invention designs an auxiliary system that can eliminate the influence of actuator saturation on the system and improve the robustness and tracking performance of the system.

[0125] This paper considers the uncertainty of the FTLM system and designs a reinforcement learning control method with an auxiliary system to compensate for the system's uncertainties. Furthermore, this method exhibits excellent self-learning capabilities, enabling iterative learning to better determine the most effective control strategy for the flexible system. Extensive experiments demonstrate that the proposed control method can effectively suppress the elastic vibration of the flexible linkage system and accurately track the desired trajectory.

[0126] Figure 9 This is a block diagram of a device for reinforcing learning control of a flexible double-link manipulator with input saturation according to an exemplary embodiment. The device is used for reinforcing learning control of a flexible double-link manipulator with input saturation. Figure 9 The device includes a construction module 310, a design module 320 and an output module 330.

[0127] in:

[0128] The construction module 310 is used to construct a dynamic model of the flexible double-link manipulator system using an assumed modal method.

[0129] The design module 320 is used to design a reinforcement learning adaptive vibration controller with input saturation.

[0130] The output module 330 is used to control the flexible double-link manipulator system according to the reinforcement learning adaptive vibration controller with input saturation to obtain an optimal control strategy.

[0131] In this embodiment, a system modeling method for a flexible dual-link manipulator under saturated nonlinear interference is considered to better capture the behavior of the flexible linkage system. This method has been validated on a flexible dual-link manipulator with strong coupling and achieved superior tracking and vibration suppression compared to a flexible single-link manipulator.

[0132] In order to deal with the saturation nonlinear interference suffered by FTLM, the present invention designs an auxiliary system that can effectively eliminate the impact of actuator saturation on system performance and enhance the robustness and tracking accuracy of the system. At the same time, considering the uncertainty of the FTLM system, the present invention proposes a reinforcement learning method combined with an auxiliary system to compensate for the impact of system uncertainty. This method has the ability of self-learning and can obtain the most effective control strategy for the flexible system through a continuous iterative learning process. Finally, after comparative experimental tests, the present invention confirms that the proposed control method can effectively suppress the elastic vibration of the flexible linkage system and accurately track the desired trajectory.

[0133] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0134] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0135] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0136] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0137] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0138] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0139] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0140] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0141] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0142] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A reinforcement learning control method for a flexible two-link manipulator with input saturation, characterized in that: The method comprises: S1. Using the hypothetical modal method, the dynamic model of the flexible double-link manipulator system under input saturation is constructed by analyzing the moment of inertia, angular velocity, linear velocity, and elastic vibration of the flexible double-link manipulator. S2. Design a reinforcement learning adaptive vibration controller with input saturation, including: S21. Design an evaluation neural network based on the error calculated between the preset reference trajectory and the input state obtained through the flexible two-link manipulator system to evaluate the performance of the manipulator control strategy under the current input saturation; S22. Construct an auxiliary system to offset the effect of saturated input on vibration suppression; design an execution neural network to approximate the unknown dynamic information in the new control torque. The execution neural network adjusts the control strategy in real time based on the system vibration error, tracking error, the previous control torque, and feedback from the evaluation network. S23, sending the system information as input data to the evaluation neural network for processing to obtain the evaluation neural network output, and using the evaluation neural network output as the input of the execution neural network to guide the execution neural network to perform training; S24. Construct a new Lyapunov function and verify that the closed-loop control system consisting of the dynamic model of the flexible two-link manipulator system and the reinforcement learning adaptive vibration controller with input saturation has semi-globally consistent ultimate boundedness; S3. Controlling the flexible double-link robotic arm system according to the reinforcement learning adaptive vibration controller with input saturation to obtain an optimal control strategy; The dynamic model of the flexible double-link manipulator system under input saturation in S1 is shown in the following formula (1): Where J(Q) represents the inertia matrix, Q = [θ, l] T represents the state variables of the flexible two-link robotic arm system, represents the generalized coordinates that change with time, n1 represents the degree of freedom of the first link, n2 represents the degree of freedom of the second link, and θ = [θ1, θ2] T represents the rotation angle of the flexible link, θ1 represents the rotation angle of the first link, θ2 represents the rotation angle of the second link, represents the second-order derivative of the state variable Q, represents the Coriolis matrix and centripetal effect, represents the first-order derivative of the state variable Q, represents the stiffness matrix, represents the saturated nonlinear function of the torque τ, represents a set of real numbers, N represents the total degrees of freedom of the system, T represents the matrix transpose, and τ represents the torque; The design and evaluation neural network in S21 includes: S211. Design a long-term cost function to quantitatively measure the performance of the robot's control strategy under input saturation within a preset time period; Among them, the long-term cost function is shown in the following formula (2): Where, Represents the optimal weight of the evaluation neural network, T represents the matrix transpose, Z c represents the Gaussian basis function of the neural network, S c =f1=Q1-Q r , f1 represents the error calculated from the preset reference trajectory and the input state obtained by the flexible double-link manipulator system, Q1 represents the current state of the system, Q r represents the expected state of the system, ∈ c represents the approximation error of the evaluation neural network, t represents the lower bound of the integral, n represents the integral variable, ζ represents the discount factor of future costs, and Υ(n) represents the instantaneous cost function; S212. Design and evaluate the weight update rate of the neural network, as shown in the following formula (3): Where, Indicates the actual weight value of the neural network at the next moment, σ c >0 indicates the learning rate of the evaluation neural network, Indicates the actual weight value of the neural network at the current moment. Indicates S c The derivative of Indicates S c The gradient of , Υ(t) represents the instantaneous cost function; The auxiliary system in S22 is shown in the following formula (4): Where, Indicates the value of the auxiliary variable at the next moment, K ξ >0, ξ represents the auxiliary variable, f2 represents the tracking error, T represents the matrix transpose, Δτ=S(τ)-τ, S(τ) represents the saturated nonlinear function of the torque τ, 2. The reinforcement learning control method for a flexible double-link manipulator with input saturation according to claim 1, characterized in that: The new Lyapunov function in S24 is shown in the following formula (5): Where V represents the Lyapunov function, f1 represents the tracking error, T represents the matrix transpose, f2 represents the tracking error, J represents the inertia matrix, ξ represents the auxiliary variable, represents the error of the action neural network weights, Represents the error in evaluating the neural network weights.

3. A reinforcement learning control device for a flexible double-link manipulator with input saturation, wherein the reinforcement learning control device for a flexible double-link manipulator with input saturation is used to implement the reinforcement learning control method for a flexible double-link manipulator with input saturation according to any one of claims 1 to 2, characterized in that: The device comprises: A construction module is used to construct a dynamic model of a flexible two-link manipulator system under input saturation by analyzing the moment of inertia, angular velocity, linear velocity, and elastic vibration of the flexible two-link manipulator using the assumed modal method; Design module for designing a reinforcement learning adaptive vibration controller with input saturation, including: S21. Design an evaluation neural network based on the error calculated between the preset reference trajectory and the input state obtained through the flexible two-link manipulator system to evaluate the performance of the manipulator control strategy under the current input saturation; S22. Construct an auxiliary system to offset the effect of saturated input on vibration suppression; design an execution neural network to approximate the unknown dynamic information in the new control torque. The execution neural network adjusts the control strategy in real time based on the system vibration error, tracking error, the previous control torque, and feedback from the evaluation network. S23, sending the system information as input data to the evaluation neural network for processing to obtain the evaluation neural network output, and using the evaluation neural network output as the input of the execution neural network to guide the execution neural network to perform training; S24. Construct a new Lyapunov function and verify that the closed-loop control system consisting of the dynamic model of the flexible two-link manipulator system and the reinforcement learning adaptive vibration controller with input saturation has semi-globally consistent ultimate boundedness; an output module, configured to control the flexible double-link robotic arm system according to the reinforcement learning adaptive vibration controller with input saturation to obtain an optimal control strategy; The dynamic model of the flexible double-link manipulator system under input saturation is shown in the following formula (1): Where J(Q) represents the inertia matrix, Q = [θ, l] T represents the state variables of the flexible two-link robotic arm system, represents the generalized coordinates that change with time, n1 represents the degree of freedom of the first link, n2 represents the degree of freedom of the second link, and θ = [θ1, θ2] T represents the rotation angle of the flexible link, θ1 represents the rotation angle of the first link, θ2 represents the rotation angle of the second link, represents the second-order derivative of the state variable Q, represents the Coriolis matrix and centripetal effect, represents the first-order derivative of the state variable Q, represents the stiffness matrix, represents the saturated nonlinear function of the torque τ, represents a set of real numbers, N represents the total degrees of freedom of the system, T represents the matrix transpose, and τ represents the torque.

Images