A reinforcement learning control method for a flexible dual-link robotic arm based on a perturbation observer
By combining a reinforcement learning control method based on a disturbance observer with a radial basis function neural network, a composite control system was designed. This system solved the problem of insufficient control accuracy of the flexible robotic arm under external disturbances, achieving fast and accurate trajectory tracking and elastic vibration suppression, and improving the robustness and self-learning ability of the system.
Patent Information
- Application Number
- CN202410186147.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-19
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-02-19
AI Technical Summary
Existing technologies lack sufficient control precision when facing model uncertainties and nonlinear elements in flexible robotic arm systems, especially when facing external disturbances, making it difficult to achieve fast and accurate trajectory tracking and elastic vibration suppression of flexible links.
A reinforcement learning control method based on disturbance observers is adopted, combined with radial basis function neural networks, to design a composite control system. The disturbance observer is used to perform feedforward compensation for the disturbance of the composite control system, and the effectiveness of the control system is verified by Lyapunov stability analysis.
It improves the robustness and tracking performance of the flexible robotic arm system, effectively suppresses elastic vibration, achieves fast and accurate trajectory tracking, and has good self-learning and anti-interference capabilities.
Smart Images

Figure CN117944050B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of robot intelligent control technology, specifically relating to a reinforcement learning control method for a flexible dual-link robotic arm based on a disturbance observer. Background Technology
[0002] The prerequisite for research on the control of flexible joint robotic arms is to establish a dynamic model of the controlled object. FTLM (Flexible Transformer Model) is a typical distributed parameter system. Researchers have proposed many methods for dynamic modeling of flexible robotic arm systems. This model is usually described using partial differential equations (PDEs), which increases the complexity and challenge of vibration control design for flexible robotic arms, especially when the model parameters cannot be accurately expressed. PDE-based methods provide an accurate description of the dynamic behavior of flexible systems; however, they involve a complex and resource-intensive modeling process. The main methods for establishing the dynamic equations of flexible robotic arms using PDEs include: Lagrange equations, Hamilton's principle, and Kane's method. While PDEs are commonly used to describe flexible robotic arm systems, they increase the complexity and challenge of vibration control. This can become more severe when the model parameters are not accurately expressed. Therefore, converting the PDE model to an ordinary differential equation (ODE) model helps determine appropriate vibration suppression strategies. ODE-based methods typically require model discretization to simplify the modeling process, thereby improving the convenience and efficiency of modeling. Currently, researchers have proposed a series of methods for dynamic modeling of flexible robotic arm systems through dimensionality reduction, such as the Assumed Modal Method (AMM), the Finite Element Method (FEM), and the Lumped Parameter Method (LPM).
[0003] Although flexible robotic arm systems have higher orders and more complex models than rigid robotic arm systems, the study of control methods for flexible robotic arms is essential due to their wide range of applications. In recent years, many control methods have been applied to the control of robotic arm systems, such as PID control, adaptive control, robust control, vibration control, fuzzy control, cooperative control, optimal control, and reinforcement learning control. When flexible robotic arms are in a state of model uncertainty and nonlinearity, commonly used research methods such as neural networks can be applied to controller design, expanding the research methods for vibration control of flexible robotic arms. Furthermore, by implementing neural network methods, flexible robotic arm systems can acquire a certain learning ability, enabling them to achieve the expected control effect more quickly. In addition, the control information required by the system during the control process is less than that required by partial differential equation models, which reduces the number of sensors required for flexible robotic arms. Neural network (NN) control addresses nonlinearity, uncertainty, and vibration problems in flexible robotic arms by learning and adjusting control parameters. Various neural networks, such as fuzzy neural networks and radial basis function neural networks (RBFNN), can be combined with other control methods, such as sliding mode control (SMC) or fuzzy control, to enhance performance and robustness. Neural network control can be applied to various modeling methods, such as AMM, FEM, or LPM, to capture dynamic behavior and vibration patterns. However, neural networks may suffer from overfitting or underfitting, affecting their generalization ability and accuracy.
[0004] To address this, those skilled in the art have proposed a reinforcement learning control method for a flexible dual-link robotic arm based on a disturbance observer. Summary of the Invention
[0005] The purpose of this invention is to provide a reinforcement learning control method for a flexible dual-link robotic arm based on a disturbance observer, so as to solve the problems mentioned in the background art.
[0006] The present invention achieves the above objectives through the following technical solutions:
[0007] A reinforcement learning control method for a flexible dual-link robotic arm based on a perturbation observer includes the following steps:
[0008] S1. Construct a dynamic model of a flexible double-link robotic arm;
[0009] S2. Design a composite control system based on a disturbance observer and a reinforcement learning controller. Propose a radial basis function neural network to determine the tracking error of the composite control system in order to approximate the uncertainty in the dynamic model.
[0010] S3. Feedforward compensation of disturbances in the composite control system is performed through the disturbance observer, and the controller equations of the composite control system are determined.
[0011] S4. Based on the controller equations and the pre-built flexible double-link robotic arm platform, a simulation experiment is conducted on the dynamic model, and Lyapunov stability analysis is used to verify the effectiveness of the composite control system.
[0012] As a further optimization of the present invention, in step S1, the Lagrange equation of the flexible double-link robotic arm is:
[0013]
[0014] The control force for each joint is denoted as τ.
[0015] Derive the dynamic model equations containing unknown perturbations, defining q = [θ, l] T Let θ be the state variable of the system, where θ = [θ1, θ2]. T For the angle of the flexible link, For virtual state, f dis For unknown spatiotemporal perturbations, F(q), Let J(q) be the inertia matrix, Coriolis matrix, and centripetal force matrix, respectively, and J(q) be the stiffness matrix.
[0016] As a further optimization of the present invention, step S2 includes: designing a control method for a composite disturbance observer based on the ctor-critic algorithm, and inputting a predetermined trajectory q into the host computer. r The tracking error e is obtained after calculating the output trajectory q. The tracking error is then input into the actor, critic neural network, and perturbation observer, respectively. The algorithm then uses the actor and critic to iteratively update each other to control the robotic arm trajectory to fit the predetermined trajectory q. r The perturbation observer output f dis The estimated value is incorporated into the design of the actor neural network; and an auxiliary variable e3 is introduced as part of the perturbation observer to approximate the uncertainties in the dynamic model.
[0017] As a further optimization of the present invention, step S2 also includes:
[0018] Define the evaluation function of the neural network as follows: and Where X c =e1=qq r The cost function is expressed as follows:
[0019]
[0020] Where v represents a constant, the instantaneous cost function is expressed as:
[0021]
[0022] Where C and Y are positive definite matrices used to balance trajectory error and control input, q r It is the expected trajectory described by a continuous function.
[0023] As a further optimization of the present invention, step S3 also includes:
[0024] Information from the control system is input into the critic neural network, and the output is input into the actor neural network for training, based on the following formula:
[0025]
[0026] The weights are then input into the Actor neural network for updating, where σ a For learning rate, This represents the cost estimation error.
[0027] As a further optimization of the present invention, the step of using Lyapunov stability analysis to verify the effectiveness of the composite control system includes:
[0028] The Lyapunov function is designed as follows:
[0029]
[0030] If we differentiate the above function, we get... This proves that the control system is effective.
[0031] The beneficial effects of this invention are as follows:
[0032] (1) The present invention considers a system modeling method for a flexible double-link robotic arm facing unknown disturbances, which enables the system to more accurately describe and predict the behavior of the flexible transmission system.
[0033] (2) Compared with single-link robotic arms, the present invention has been verified on more complex double-link robotic arms and has achieved good control results.
[0034] (3) To address the unknown external disturbances faced by FTLM, this invention designs a feedforward compensation system disturbance observer. It can reduce the impact of disturbances on the system and improve the system's robustness and tracking performance.
[0035] (4) This invention considers external disturbances in FTLM and designs a reinforcement learning control method based on a disturbance observer. Extensive simulations and experiments demonstrate that the proposed control method can effectively suppress elastic vibrations and deformations in the flexible transmission system and accurately track the desired trajectory. Furthermore, this method exhibits good self-learning and anti-interference capabilities, effectively handling unknown disturbances. Attached Figure Description
[0036] Figure 1 This is a flowchart illustrating the execution process of the method in this invention;
[0037] Figure 2 This is a schematic diagram of the flexible double-link robotic arm composite control system disclosed in this invention.
[0038] Figure 3 This is a flowchart of the algorithm execution in this invention;
[0039] Figure 4 This is a simplified experimental platform diagram of the flexible double-link system in this invention;
[0040] Figure 5-6 This is a schematic diagram of the angle tracking trajectory results of the flexible robotic arm in the embodiment;
[0041] Figure 7-8 This is a schematic diagram of the simulation results of the elastic deformation of the flexible robotic arm in the embodiment;
[0042] Figure 9-10 This is a schematic diagram of the angle tracking error results of the flexible robotic arm in the embodiment;
[0043] Figure 11-12 This is a schematic diagram of the control input results of the flexible robotic arm in the embodiment;
[0044] Figure 13 This is a flowchart illustrating the implementation of the present invention. Detailed Implementation
[0045] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.
[0046] Example 1
[0047] To address the insufficient control precision of existing technologies for flexible robotic arms facing external disturbances, this invention aims to provide a reinforcement learning control algorithm for a flexible dual-link robotic arm based on a disturbance observer. A control method is designed to achieve rapid and accurate planar positioning of the robot's end effector and minimize the elastic vibration of the flexible links. Two servo motors can apply driving forces at each rotational position of the flexible links. A key issue is the coupling between the two flexible links during motion, which makes system modeling and control design very complex. To simplify the system model, each flexible link can be modeled as an Euler-Bernoulli beam, and the dual-link flexible robotic arm is assumed to have a uniform cross-section and stable performance, neglecting air resistance and longitudinal deflection of the links.
[0048] like Figure 1 As shown, this embodiment proposes a reinforcement learning control method for a flexible dual-link robotic arm based on a disturbance observer, including the following steps:
[0049] S1. Construct a dynamic model of a flexible double-link robotic arm;
[0050] This includes pre-building a flexible double-link robotic arm platform, considering the influence of unknown spatiotemporal disturbances, and using these disturbances as one of the system inputs to obtain the relationship between the system's state variables and control inputs. The Lagrange equation for the flexible double-link robotic arm is:
[0051]
[0052] The control force for each joint is denoted as τ.
[0053] Derive the dynamic model equations containing unknown perturbations, defining q = [θ, l] T Let θ be the state variable of the system, where θ = [θ1, θ2]. T For the angle of the flexible link, For virtual state, f dis For unknown spatiotemporal perturbations, Let J(q) be the inertia matrix, Coriolis matrix, and centripetal force matrix, respectively, and J(q) be the stiffness matrix.
[0054] S2. Design of a reinforcement learning neural network controller based on a disturbance observer: Design a composite control system based on a disturbance observer and a reinforcement learning controller. Propose a radial basis function neural network to determine the tracking error of the composite control system in order to approximate the uncertainty in the dynamic model.
[0055] Combination Figure 2This is a schematic diagram of the flexible dual-link robotic arm composite control system disclosed in this invention. A control method based on an actor-critic algorithm and a composite disturbance observer is designed. This algorithm possesses reinforcement learning capabilities for the disturbance observer structure, and the overall disturbance term in the system includes unknown spatiotemporal disturbances. The actor-critic algorithm consists of policy evaluation and policy improvement, providing a flexible iterative method: the actor neural network gradually learns from system experience using a radial basis function neural network (RBFNN) and generates an appropriate control policy. On the other hand, the critic neural network estimates the evaluation function of the current policy to effectively handle dynamic uncertainties and disturbances. The critic neural network evaluates the performance of the current control policy by adjusting the learning feedback, which is based on the input of reward or penalty signals.
[0056] Combination Figure 3 This provides a visual representation of the control structure and algorithm flow in a DOB-based FTLM system. The predetermined trajectory q is input into the host computer. r The tracking error e is obtained after calculating the output trajectory q. The tracking error is then input into the actor, critic neural network, and perturbation observer, respectively. The algorithm then uses the actor and critic to iteratively update each other to control the robotic arm trajectory to fit the predetermined trajectory q. r The perturbation observer (DOB) output f dis The estimated values are incorporated into the design of the actor neural network; and an auxiliary variable e3 is introduced as part of the perturbation observer to approximate the uncertainties present in the dynamic model. Considering perturbations caused by unknown human-environment interactions, such as external disturbances or payload variations, the DOB is specifically designed to address these challenges, introducing the auxiliary variable e3 as part of our proposed DOB. This variable helps to effectively manage disturbance-related uncertainties, enabling the system to maintain its performance and stability even in the presence of external factors.
[0057] The above-mentioned control system also includes the following steps for implementation:
[0058] Evaluating neural network designs:
[0059] Critical neural networks play a crucial role in evaluating the effectiveness of mainstream control strategies by fine-tuning the learning feedback based on reward or penalty signals from the input data. To achieve this, a long-term cost function is formulated as a quantitative measure of the control strategy's performance over a period of time. By utilizing this cost function, critical neural networks can systematically adjust the control strategy to optimize its performance.
[0060] To achieve this, this embodiment designs a function to calculate long-term costs, aiming to find the optimal control that minimizes operating costs. The RBF neural network can estimate the nonlinear cost function through parallel computation and self-learning capabilities. For a continuous function f(X), the RBF neural network takes the form f(X) = W. T S(X), where It is the input vector; It is a weight vector of a neural network with l > 1 nodes, S(X) = [s1(X), s2(X), ..., s...]. l [(X)] are the basis functions of the neural network. Each s i (X) is represented using the Gaussian function.
[0061] μ i This is the center of the neural network distribution. When the input X is close to μ... i When this happens, the basis functions of the radial basis function neural network are activated, causing the output to increase. The width of the Gaussian function is determined by η. i This indicates that, with this configuration, radial basis function neural networks can accurately estimate compact sets. For any continuous function, achieve the required level of accuracy.
[0062] Preferably, step S2 further includes:
[0063] Define the evaluation function of the neural network as follows: and Where X c =e1=qq r The cost function is expressed as follows:
[0064]
[0065] Where v represents a constant, the instantaneous cost function is expressed as:
[0066]
[0067] Where C and Y are positive definite matrices used to balance trajectory error and control input, q r It is the expected trajectory described by a continuous function.
[0068] The evaluation neural network receives the current state and the actions generated by the actor neural network as input, and then outputs a value that represents the effectiveness of executing the actions generated by the actor neural network in the current state. The smaller this value, the better the policy. In this way, the evaluation neural network can evaluate and optimize the current policy, thereby updating the adaptive rate of the actor neural network to make better decisions.
[0069] The weight update rate of the neural network is evaluated as follows:
[0070]
[0071] Where σ c For learning rate,
[0072] Performing neural network design:
[0073] The execution neural network receives system state information, such as tracking and vibration errors, and the output torque from the previous moment. It includes a virtual error term to ensure semi-global critical stability. Based on the system state and feedback from the evaluation function, it dynamically adjusts the control input to optimize system performance. However, real-world control systems are often subject to various unknown disturbances. These disturbances can affect system performance, causing the control input generated by the execution neural network to fail to achieve the desired effect. By using a disturbance observer, disturbances in the system can be estimated in real time and compensated for. Thus, even in the presence of unknown disturbances, the execution neural network can generate effective control inputs, thereby improving system performance.
[0074] Based on the model equations of the flexible robotic arm, this embodiment derives the initial torque of the system using the backstepping method:
[0075]
[0076] Where e1 and e2 are the tracking errors defined by the system, taking into account the dynamic information of the system, such as and Not easily obtained, this embodiment uses an execution neural network to effectively approximate the uncertainties present in the system. For the disturbance estimate of the disturbance observer, the control torque after replacement with a neural network is:
[0077]
[0078] in, As an estimate of the weights.
[0079] S3. Feedforward compensation of disturbances in the composite control system is performed using a disturbance observer, and the controller equations of the composite control system are determined. The disturbance observer is designed as follows:
[0080] An auxiliary variable e3 is introduced as part of the solution proposed in this embodiment.
[0081] e3 = f dis +Θ(e2)
[0082] Where Θ(e2) is a function vector, we can obtain:
[0083]
[0084] It can be deduced that:
[0085]
[0086] Preferably, step S3 further includes:
[0087] Information from the control system is input into the critic neural network, and the output is input into the actor neural network for training, based on the following formula:
[0088]
[0089] The weights are then input into the Actor neural network for updating, where σ a For learning rate, This represents the cost estimation error.
[0090] S4. Based on the controller equations and a pre-built flexible double-link robotic arm platform, a simulation experiment was conducted on the dynamic model, and Lyapunov stability analysis was used to verify the effectiveness of the composite control system.
[0091] Preferably, Lyapunov stability analysis is used to verify the effectiveness of the composite control system, including:
[0092] The Lyapunov function is designed as follows:
[0093]
[0094] If we differentiate the above function, we get... This proves that the control system is effective.
[0095] According to embodiments of the present invention, a vibration control tracking algorithm for flexible robotic arm operation is proposed. An n-dimensional ODE system model of FTLM is introduced, which simplifies dynamic uncertainty analysis and controller design compared to a PDE system. Based on this, a composite controller integrating RL and a disturbance observer is proposed to achieve trajectory tracking and flexible vibration suppression. In this setup, the actor neural network formulates appropriate control strategies by continuously accumulating system experience, while the critic neural network provides feedback values by evaluating the effectiveness of the current control strategy.
[0096] The above control method will be further explained below with reference to specific implementation procedures.
[0097] To verify the effectiveness and stability of the algorithm proposed in this embodiment, experiments were conducted using a flexible dual-link robotic arm system platform. The hardware-in-the-loop (HIL) simulation test system provides cost-effective realism for the simulation and can integrate physical components for testing, thereby improving system reliability and design quality. The basic parameters of the robotic arm are shown in Table 1.
[0098] Table 1: Parameters of the Experimental Platform
[0099]
[0100] (1) Introduction to the experimental testing platform:
[0101] Figure 4 The diagram shows a simplified experimental platform for the FTLM system. This model connects the flexible module to a strain gauge amplifier board for real-time monitoring. The FTLM system, with two stepper motors mounted at the bottom of the connector, offers advantages in high precision and rapid response.
[0102] A semi-physical simulation experimental platform for FTLM was built using the Matlab / Simulink platform, providing software for seamless data interaction. Real-time data is transmitted to a data acquisition card, which drives the stepper motor. Feedback signals from the motor's photoelectric encoder and the arm strain gauge are processed and transmitted to an industrial computer. Upon receiving the processed feedback signal, the controller evaluates the current state of the system and adjusts the control signals accordingly. This realizes a closed-loop control system for FTLM, ensuring precise and stable operation. For a comprehensive understanding of the experimental setup, the parameters are summarized and given in Table 1.
[0103] To compare the control effects of traditional control methods, some advanced control methods, and the reinforcement learning control method with a disturbance observer mentioned in this invention on a flexible double-bar manipulator system, we conducted simulations and experiments on the flexible double-bar manipulator system using three control methods: PD control, neural network control, and reinforcement learning control.
[0104] In this section, we utilize the Simulink module in MATLAB to design PD, neural network control, and a reinforcement learning controller based on a perturbation observer. The PD and NN controls are used for comparison with the reinforcement learning control method based on a perturbation observer proposed in this invention. The reference trajectory angle θ id (i = 1, 2) Select an angle between 10° and 15°, and simulate an disturbance of f. dis = [0.3sin(t), 0.15sin(t)] T The design principles and parameter settings for three control methods are described below.
[0105] ① Given that introducing an integral term typically leads to problems such as integral saturation and instability in the system, flexible robotic arms in practical engineering have high requirements for stability and dynamic performance. Therefore, this invention only considers using the PD algorithm to control the flexible robotic arm. To improve joint positioning accuracy and achieve precise offset tracking of joints from a predetermined trajectory during the on-orbit operation of the flexible robotic arm, this paper designs a PD controller based on joint angle tracking and adjustment, the control form of which is as follows:
[0106]
[0107] PD control gain set to K p =[50, 30] T and K d =[1.2, 3.5] T .
[0108] ② To enhance the comprehensiveness and persuasiveness of this comparative experiment, a neural network controller was introduced as a comparison with the intelligent control algorithm. Similarly, the uncertainty term in the FTLM model can be approximated by a neural network, and the latest control input can be obtained through state feedback. The neural network controller is configured as follows:
[0109]
[0110] Neural network control gain selection K N1 =[35, 25] T and K N2 = [0.8, 0.6] T The neural network has 256 neurons, a center point of -1, a base width of 2.5, and an initial weight of 0 for each node.
[0111] ③ Reinforcement learning control based on perturbation observers:
[0112]
[0113] For reinforcement learning control gain selection, K1 = [28, 25]. T K2 = [0.8, 0.6] T K I =[50, 50] T The disturbance observer control parameters are D(e2) = [0.85, 0.6]. T The model parameters are chosen as n1 = n2 = 5. The evaluation and execution neural networks each have 256 nodes. The center parameter is -1 or 1, η. a =2, η c =0.5, the initial weights of each node are 0, and the learning rates of the two neural networks are chosen to be 100 and 0.1, respectively.
[0114] (2) Comparison of experimental results:
[0115] The parameter settings in the experiment are the same as in the simulation. The disturbance, consisting of sinusoidal functions simulating spatial and temporal variations, is denoted by f. dis = [0.3sin(t), 0.15sin(t)] T .
[0116] The input signal in the experiment was a square wave with a 5° phase difference between the first and second joints. The square wave was converted into an analog output signal, amplified, and then sent to the servo system to control the FTLM. The two drivers had separate command and control loops. Sensors recorded the rotation angle θ and vibration amplitude α, which were displayed in a real-time window.
[0117] The results show that PD and NN control cannot ensure that the servo system maintains a stable angular velocity output when faced with external disturbances. However, the servo system achieves stable output after adopting the RL control with DOB proposed in this paper. The disturbance observer can effectively reduce abrupt errors during angular velocity steering. Therefore, this experiment demonstrates the effectiveness of RL control with DOB.
[0118] To clearly compare the performance of each control algorithm, Figure 5-8 The link angle θ under each control method is shown separately. i The real-time superposition curve of the α vibration. Tracking error is shown in [the figure]. Figure 9-10 See the control input results. Figure 11-12 .
[0119] When a sinusoidal disturbance is present, both the PD and neural network control groups exhibit prolonged overshoot time, affecting the tracking performance of RL control with DOB. Figure 5 The θ1 shown accurately and quickly tracks the required trajectory, with a significantly shortened rise time and a resolution of significant overshoot.
[0120] Its damping performance, such as Figure 7 The α1 shown eventually reaches a stable state. Significant disturbances exist in the first stage between 5-8 seconds and 15-18 seconds. The oscillations originate from the inertia of the initial link when subsequent links stop moving. The disturbance range is stable at ±0.8° under PD control, ±0.4° under NN control, and only ±0.1° under RL control with DOB.
[0121] The error peaks occur at the 0th second of the first link's swing and the 10th second of the steering, while under RL control with DOB, these two error peaks, as well as the error in steady state, are minimized. Figure 4 as well as Figure 5As shown, although the PD and NN controllers based on the ODE model perform well in tracking the rotation angle of the flexible robotic arm, they still have many errors in tracking, and the time required to reach steady state is too long, resulting in a slow response speed that cannot meet the system's speed requirements. Their vibration control performance is also unsatisfactory, exhibiting a large steady-state amplitude. In contrast, the RL control with DOB, also based on the ODE model, can effectively suppress vibration, maintain continuous tracking of the rotation angle of the flexible link, and achieve a faster response speed and smaller tracking error. Therefore, the RL control with DOB designed for the flexible robotic arm is feasible and efficient.
[0122] To further verify the effectiveness of the proposed controller, we implemented three control algorithms—PD, NN, and RL control with DOB—on a physical test platform. The results of these experiments are as follows: Figures 5 to 12 As shown. Figures 5-8 The θ values of the three control methods were displayed in real time. i And the α curve. Tracking error as... Figure 9-10 As shown. Control inputs are as follows: Figure 11-12 As shown. Figure 8 The results show the minimum peak error, maximum tracking error, and steady-state error in trajectory tracking using RL control with DOB, indicating that it can track the desired trajectory more accurately and faster. The PD and NN control algorithms perform poorly in trajectory tracking, exhibiting large overshoot times and oscillations. The RL with DOB control algorithm has the smallest maximum vibration and the longest steady-state vibration range in vibration suppression, indicating that it can more effectively suppress the elastic vibration of flexible components in manipulators. The PD and NN control algorithms perform poorly in vibration suppression, exhibiting large residual vibrations. Figure 11-12 The results show that the RL control algorithm with DOB exhibits smaller fluctuations, with lower peak and steady-state values compared to the PD and NN control algorithms. This indicates that RL control with DOB can effectively utilize disturbance observers to compensate for disturbances in the system and reduce control torque consumption.
[0123] In conjunction with the above embodiments, this embodiment verifies the effectiveness of the proposed reinforcement learning composite controller through simulation experiments on a flexible dual-link robotic arm platform. Simultaneously, Lyapunov stability analysis proves the semi-global asymptotically uniform bounded property of the closed-loop system. Physical experimental results show that:
[0124] In trajectory tracking of flexible robotic arms, RL control using DOB has faster settling time and smaller tracking error than PD and neural network (NN) control.
[0125] (1) Compared with PD and NN control, the reinforcement learning control method based on disturbance observer significantly reduces the residual vibration generated after the robot arm motion ends and has a smaller tracking error convergence range.
[0126] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A reinforcement learning control method for a flexible dual-link robotic arm based on a perturbation observer, characterized in that, Includes the following steps: S1. Construct a dynamic model of a flexible double-link robotic arm; S2. Design a composite control system based on a disturbance observer and a reinforcement learning controller. Propose a radial basis function neural network to determine the tracking error of the composite control system in order to approximate the uncertainty in the dynamic model. S3. Feedforward compensation of disturbances in the composite control system is performed through the disturbance observer, and the controller equations of the composite control system are determined. S4. Based on the controller equations and the pre-built flexible double-link robotic arm platform, a simulation experiment is conducted on the dynamic model, and Lyapunov stability analysis is used to verify the effectiveness of the composite control system. Step S2 includes: designing a control method for a composite disturbance observer based on the actor-critic algorithm, and inputting a predetermined trajectory into the host computer. Output trajectory The tracking error was obtained after calculation. The tracking error is input into the actor, critic neural network, and perturbation observer, respectively. Then, the algorithm enables the actor and critic to iteratively update each other to control the robot arm trajectory to conform to the predetermined trajectory. The output of the perturbation observer The estimated values are used in the design of the actor neural network; and auxiliary variables are introduced. As part of the disturbance observer, it is used to approximate the uncertainties in the dynamic model; Step S2 also includes: Define the evaluation function of the neural network as follows: and ,in The cost function is expressed as follows: ; in, Representing a constant, the instantaneous cost function is expressed as: : ; in, and It is a positive definite matrix, used to balance trajectory error and control input. It is the expected trajectory described by a continuous function.
2. The reinforcement learning control method for a flexible dual-link robotic arm based on a perturbation observer according to claim 1, characterized in that: In step S1, the Lagrange equation for the flexible double-link robotic arm is: ; The control force of each joint is expressed as: ; Derive the dynamic model equations containing unknown perturbations, and define... For the system's state variables, For the angle of the flexible link, It is in a virtual state. For unknown spatiotemporal perturbations, These are the inertia matrix, the Coriolis matrix, and the centripetal force matrix, respectively. Here is the stiffness matrix.
3. The reinforcement learning control method for a flexible dual-link robotic arm based on a perturbation observer according to claim 1, characterized in that: Step S3 also includes: The information from the composite control system is input into the critic neural network, and the output is input into the actor neural network for training, based on the following formula: ; The weights are then input into the Actor neural network for updating, where For learning rate, , This represents the cost estimation error.
4. The reinforcement learning control method for a flexible dual-link robotic arm based on a perturbation observer according to claim 1, characterized in that: The verification of the effectiveness of the composite control system using Lyapunov stability analysis includes: The Lyapunov function is designed as follows: ; If we differentiate the above function, we get... This proves that the composite control system is effective.
Citation Information
Patent Citations
Flexible manipulator composite control method based on output redefinition
CN108388114A
Design method of self-adaptive neural network controller of electrically-driven flexible joint mechanical arm based on disturbance observer and command filter
CN114895564A
Self-adaptive neural network control method for hydraulic mechanical arm
CN117289612A
