Mechanical arm control method based on double evaluation networks and stability evaluation method thereof

Abstract

The invention provides a mechanical arm control method based on a double evaluation network and a stability evaluation method thereof, and relates to the technical field of friction coefficient calculation. The method comprises the steps that a kinematics equation of the flexible double-connecting-rod mechanical arm is established; designing a reference neural network to obtain an internal enhanced signal of the reference neural network; designing an evaluation neural network based on the internal reinforcement signal of the reference neural network to obtain a long-term cost control function, the long-term cost control function being used for finding optimal control for minimizing the operation cost; according to the method, the initial torque of the flexible double-connecting-rod mechanical arm is obtained based on the kinematics equation of the flexible double-connecting-rod mechanical arm, the replaced control force is obtained based on the action neural network, trajectory tracking control is conducted on an FTLM system, meanwhile, elastic vibration of the system is restrained, and the requirement that historical data or pre-existing knowledge is assigned for strengthening signals is eliminated.

Description

Technical Field

[0001] The invention relates to the technical field of friction coefficient, and in particular to a mechanical arm control method based on a dual evaluation network and a stability evaluation method thereof. Background Art

[0002] As a complex system with infinite degrees of freedom, the modeling and control of flexible manipulators have always been a challenge in the engineering field. Traditional modeling methods, such as Kane's method, Hamilton's principle, Newton-Euler equations and Lagrange equations, can accurately describe their dynamic characteristics, but they also increase the complexity of controller design. In order to simplify this process, researchers proposed a discretization modeling technique to transform partial differential equations into ordinary differential equations, thereby transforming the model into finite dimensions, improving the versatility of controller design and making vibration control more accurate.

[0003] Although research on control methods for flexible manipulators has made progress, most methods, including LQR control, boundary control, adaptive control, and sliding mode control, still rely on precise mathematical models, which are often difficult to implement in practical applications. To overcome this limitation, model-free control methods have emerged. They use the input and output data of the system for control and do not rely on precise mathematical models, such as PID control, neural network control, fuzzy control, specified performance control, and reinforcement learning control.

[0004] In the classic reinforcement learning algorithm, the evaluation network is a neural network used to approximate the value function. It outputs an estimated value of the value function based on the system state and control action, thereby guiding the learning and optimization of the system. However, the evaluation network requires a reinforcement signal to evaluate the behavior of the system, and this reinforcement signal is usually provided by the external environment or set manually. This leads to some problems, such as: the external reinforcement signal may be sparse, delayed, inaccurate or incomplete, making it difficult for the evaluation network to learn effective feedback information. Summary of the invention

[0005] In order to solve a technical problem existing in the prior art, the present invention proposes an adaptive dynamic programming architecture for a flexible two-link manipulator, aiming to perform trajectory tracking control on the FTLM system while suppressing the elastic vibration of the system. First, the flexible two-link manipulator is discretized and modeled using the assumed modal method to obtain the ordinary differential dynamics equation of the manipulator. Then, an ADP structure is designed to combine the reference network with the traditional critic network to form a dual critic network. The reference network updates the critic network by providing an internal reinforcement signal of the target, thereby achieving a more subtle approximation of the total cost signal. This internal target representation is dynamically adjusted according to the state of the system and the actions taken, eliminating the need to assign reinforcement signals to historical data or pre-existing knowledge. A manipulator control method based on a dual evaluation network, comprising:

[0006] Establish the kinematic equations of the flexible two-link robotic arm;

[0007] Design a reference neural network and obtain an internal reinforcement signal of the reference neural network;

[0008] Based on the internal reinforcement signal of the reference neural network, an evaluation neural network is designed to obtain a long-term cost control function, wherein the long-term cost control function is used to find an optimal control that minimizes the operating cost;

[0009] Based on the kinematic equation of the flexible double-link robotic arm, the initial torque of the flexible double-link robotic arm is obtained, and based on the action neural network, the replaced control torque is obtained.

[0010] Optionally, the kinematic equation for establishing the flexible double-link robotic arm is formula (1):

[0011]

[0012] Where τ(t) is the control torque of the joint at time t;

[0013] H(x) is the stiffness matrix;

[0014] A(x) is the inertia matrix;

[0015] are the Coriolis matrix and the centripetal force matrix;

[0016] The state quantity of the flexible manipulator system is x = [θ, p] T , θ=[θ 1 ,θ 2 ] T represents the joint angle, Represents the generalized coordinates of flexibility, with a total degree of freedom of N = n 1 +n 2 .

[0017] Optionally, the designing of a reference neural network to obtain an internal reinforcement signal of the reference neural network includes:

[0018] A reference neural network is designed, wherein the reference neural network includes an external reinforcement signal and an internal reinforcement signal, wherein the external reinforcement signal is determined according to the environment, and the external reinforcement signal is formula (2):

[0019]

[0020] Among them, c is the upper bound of the system expected error;

[0021] e 1 =xx r , e 1 is the tracking error of the flexible manipulator system, x r is the expected trajectory of the system;

[0022] Based on the back propagation rule, the internal reinforcement signal is obtained as formula (3):

[0023]

[0024] Among them, the range of R(t) is [-1,1];

[0025] is the expected weight of the reference network;

[0026] is the input of the reference neural network, τ is the control torque of the robot joint;

[0027] S r (U r ) is the basis function of the reference neural network;

[0028] ∈ r is the approximation error of the reference network.

[0029] Optionally, the error function of the reference neural network is defined as formula (4) and formula (5):

[0030]

[0031] e r (t) is the error function of the reference network and α is a discount factor less than 1. is the estimated value J of the evaluation network output d (t) is the expected value of the evaluation network output, E r (t) is the quadratic form of the error at time t. It is the error signal e r Half the square of (t), often used as a measure of system performance.

[0032] Optionally, the formula of the back propagation rule applied to the reference neural network is formula (6):

[0033]

[0034] Among them, σ r is the learning rate of the reference neural network; E r is the quadratic form of the error at time t. It is the error signal e r Half the square of (t), often used as a measure of system performance. are the estimated weights of the reference network.

[0035] Optionally, the designing and evaluating the neural network based on the internal reinforcement signal of the reference neural network to obtain the long-term cost control function comprises:

[0036] The evaluation neural network function is defined as formula (7) and formula (8):

[0037]

[0038] J is the output of the evaluation neural network, W c * To evaluate the expected weight of the network, ∈ c To evaluate the approximation error of the network, is the estimated weight S of the evaluation network c (U c ) is the basis function for evaluating the neural network, U C The input of the evaluation network To evaluate the estimated value of the network output;

[0039] Based on the definition of the evaluation neural network function, a long-term cost control function is obtained, which is formula (9):

[0040]

[0041] J(t) is the long-term cost function, ρ is a constant used to discount future costs, n∈t, κ(t) is the immediate cost function.

[0042] Optionally, the formula of the instantaneous cost function is formula (10):

[0043]

[0044] D is a positive definite constant matrix used to weigh trajectory errors;

[0045] R is a positive definite constant matrix used to weigh the control input;

[0046] The weight update rate of the evaluation neural network is as follows:

[0047]

[0048] σ c To evaluate the learning rate of the network, S c To evaluate the basis function of the network, U C The input of the evaluation network;

[0049] Optionally, obtaining the initial torque of the flexible double-link mechanical arm based on the kinematic equation of the flexible double-link mechanical arm, and obtaining the replaced control torque based on the action neural network includes:

[0050] Define the error amount of the flexible two-link robot: e 2 is the system tracking error variable, λ 1 for e 1 Virtual control quantity;

[0051] According to the backstepping method and K 1 is the control gain matrix of the system. Combined with the kinematic equation of the flexible double-link manipulator, e 1 , e 2 By taking the derivative with respect to time, we can obtain formula (12) and formula (13):

[0052]

[0053] Based on formula (12) and formula (13), the initial torque of the flexible double-link manipulator is obtained as formula (14):

[0054]

[0055] τ 0 is the initial torque of the flexible double-link manipulator, K 2 is the control gain matrix of the system;

[0056] Based on the uncertain terms existing in the motion neural network approximation system, the replaced control torque is obtained, and the replaced control torque is formula (15):

[0057]

[0058] Among them, τ 2 is the control torque after replacement, As an estimate of the action neural network weights,

[0059]

[0060] in, The formula is formula (16):

[0061]

[0062] σ a To enforce the learning rate of the network, β represents a positive constant.

[0063] A stability evaluation method for a robotic arm control method based on a dual evaluation network is used to evaluate the stability of the robotic arm control method based on the dual evaluation network, comprising the following steps:

[0064] Based on the Lyapunov function, the stability of the robot control method based on the dual evaluation network is evaluated to obtain the global Lyapunov function, and the global Lyapunov function V is formula (17):

[0065]

[0066] By taking the derivative of the global Lyapunov function, we obtain formula (18):

[0067]

[0068] Based on the basic inequality and formula (18), we get formula (19):

[0069]

[0070] Based on formula (19), the robot control method based on the dual evaluation network is obtained to be semi-globally consistent and ultimately bounded.

[0071] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0072] This paper proposes an innovative ADP structure to enhance the vibration reduction and trajectory accuracy of FTLM systems. This enhanced structure integrates a reference network with a typical evaluation network to form a dual evaluation network setup. The reference network enriches the evaluation network by providing an internal representation of the goal, which enables a more detailed approximation of the total cost signal. This internal goal representation is dynamically adjusted based on the state of the system and the actions taken, without the need for historical data or pre-existing knowledge to assign values ​​to the reinforcement signal;

[0073] This paper proposes an adaptive dynamic programming architecture for a flexible two-link manipulator, aiming to track the FTLM system and suppress the elastic vibration of the system. First, the flexible two-link manipulator is discretized and modeled using AMM to obtain the ordinary differential dynamic equations of the manipulator. Then, an ADP structure is designed to combine the reference network with the traditional critic network to form a dual critic network. The reference network updates the critic network by providing an internal reinforcement signal of the target, thereby achieving a more subtle approximation of the total cost signal. This internal target representation is dynamically adjusted according to the state of the system and the actions taken, eliminating the need to assign reinforcement signals to historical data or pre-existing knowledge. The effectiveness of the proposed ADP method based on the dual evaluation network is verified by experimental results. At the same time, the stability and robustness of the closed-loop system under ADP control are analyzed by the Lyapunov direct method, and the semi-global consistent ultimate boundedness of the tracking error and vibration error is proved. The experimental results show that:

[0074] (1) In the trajectory tracking of FTLM, the ADP method using a dual evaluation network significantly outperforms traditional RL control and NN control in reducing time cost and cumulative tracking error;

[0075] (2) In the vibration suppression of FTLM, the ADP method using a dual evaluation network significantly outperforms traditional RL control and NN control in suppressing the vibration of the flexible manipulator. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0077] Figure 1 (a) in FIG. 1 is a coordinate axis and symbol in the dynamic modeling of the flexible double-link provided in an embodiment of the present invention. Figure 1 (b) is the inertia moment of the intermediate load provided in the embodiment of the present invention;

[0078] Figure 2 ADP algorithm structure diagram provided by an embodiment of the present invention;

[0079] Figure 3 Comparison of tracking trajectory performance of three control methods provided in embodiments of the present invention;

[0080] Figure 4 Comparison of terminal vibrations of three control methods provided in embodiments of the present invention;

[0081] Figure 5Comparison of tracking errors of three control methods provided in embodiments of the present invention;

[0082] Figure 6 The control inputs of the three control methods provided in the embodiments of the present invention are compared. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "one" or "the" do not indicate a quantitative limitation, but indicate the existence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0085] It should be noted that the terms "up", "down", "left", "right", "front", "back", etc. used in the present invention are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0086] Among reinforcement learning control methods, especially adaptive dynamic programming (ADP) control methods, they have attracted increasing attention due to their adaptability and learning ability. ADP combines the concepts of optimal control and reinforcement learning, and provides a self-learning control method that can handle unknown dynamic systems, online learning and optimization of control strategies, and is particularly suitable for handling uncertainties and time-varying systems. Actor-Critic algorithm, an important branch of ADP, provides a classic framework for reinforcement learning. The action neural network determines the control input according to the strategy, while the evaluation neural network evaluates the value function. However, reinforcement learning control has poor adaptability in the absence of prior knowledge. To solve this problem, an innovative ADP method is proposed. The dual evaluation network design aims to reduce the dependence on prior information and improve the tracking control effect and optimization efficiency of the flexible manipulator, especially in complex systems with unknown system models.

[0087] The trajectory tracking and vibration control of flexible manipulators pose dual challenges to dynamic modeling and control design. The models of flexible systems are usually described by partial differential equations and ordinary differential equations. Although partial differential equations can describe the system model with high precision, the control design based on this model is highly complex and computationally intensive, which limits its practical application. Modeling based on ordinary differential equations can strike a balance between modeling accuracy and computational efficiency. Traditional manipulator control methods often ignore the vibration problem caused by the flexible structure, resulting in poor trajectory tracking performance. In addition, control methods based on reinforcement learning (RL) often rely heavily on artificially set prior models.

[0088] Although there are many successful applications of ADP / RL methods, many of them either use binary reinforcement signals or some specific discrete reinforcement signals, and some people hand-craft reinforcement signals using past experience or prior knowledge. For example, the square of the difference between the reference signal and the actual output of the system is set as the reinforcement signal, or different weights are defined for the reinforcement signals of each component, and some error components even have nonlinear coefficients. Therefore, it is hoped that there is a method that can assign reinforcement signals without any (or minimal) prior knowledge or human intervention, and adaptively adjust when the operating environment changes.

[0089] (1) The FTLM system is discretely modeled as ordinary differential equations using the assumed modal method. Based on this dynamic model, a dual-critic network ADP controller is developed for online updating.

[0090] (2) Compared with typical reinforcement learning and neural network control methods, the dual critic network framework proposed in this paper integrates the reference network with the critic network and innovatively introduces an adaptive internal reinforcement signal. This design reduces the dependence on external environmental information and prior knowledge, thereby enhancing the versatility of the control model. By accelerating learning speed and improving accuracy, the framework enables the FTLM system to achieve accurate trajectory tracking and effective vibration control through online learning.

[0091] (3) The vibration signal is introduced as an error term into the input of the reference network, which enables it to evaluate the system performance in real time and guide the update of the evaluation network, effectively suppressing the flexible vibration in the FTLM system without relying on a priori models or precise system models.

[0092] like Figures 1 to 6 As shown, based on this, the present invention provides a mechanical arm control method based on a dual evaluation network and a stability evaluation method thereof, the specific contents are as follows:

[0093] In a first aspect, the present invention provides a robot arm control method based on a dual evaluation network, comprising:

[0094] S1. Establish the kinematic equations of the flexible double-link robotic arm;

[0095] S2. Design a reference neural network and obtain an internal reinforcement signal of the reference neural network;

[0096] S3. Based on the internal reinforcement signal of the reference neural network, an evaluation neural network is designed to obtain a long-term cost control function, wherein the long-term cost control function is used to find the optimal control that minimizes the operating cost;

[0097] S4. Based on the kinematic equation of the flexible double-link robotic arm, the initial torque of the flexible double-link robotic arm is obtained, and based on the action neural network, the control torque after replacement is obtained.

[0098] In a specific implementation manner, S1, establishing the kinematic equation of the flexible double-link robotic arm specifically includes the following contents:

[0099] Figure 1 (a) shows the coordinate axes and symbols in the dynamic modeling of the flexible double link. The torque applied to the i-th joint is called τ i (t). The inertia matrices of the i-th link and its end effector are J li and J ti , and the inertia matrix of the joint itself is J hi During the motion, the position x along the i-th link i Its range is from base to maximum C i (usually not exceeding the connecting rod length L 1) Experienced by function O i (x i , t) captures the lateral dynamics. At the same time, the displacement Y i (x i , t) quantifies the position of the link at any time t in the XOYZ space, which ranges from 0 to C i This framework fully describes the dynamic behavior of each connecting rod under different working conditions. l1 and K l2 is the torsional stiffness constant of the first and second flexible links, J li (where i = 1, 2) is the moment of inertia of the intermediate load, such as Figure 1 (b) is shown. The kinematic equation for establishing the flexible double-link manipulator is formula (1):

[0100]

[0101] Among them, τ(t) is the control torque of the joint at time t;

[0102] H(x) is the stiffness matrix;

[0103] A(x) is the inertia matrix;

[0104] are the Coriolis matrix and the centripetal force matrix;

[0105] The control force of each joint is expressed as τ. On this basis, the system model equation containing unknown disturbances is further derived. Define x = [θ, p] T is the state quantity of the system, where θ=[θ 1 ,θ 2 ] T is the rotation angle of the flexible link, is the vibration modulus. Using A(x), denote the inertia matrix, Coriolis matrix and centripetal force matrix respectively, and the stiffness matrix is ​​represented by H(x).

[0106] In the subsequent steps, the ADP (Adaptive Dynamic Programming) control strategy of the dual evaluation network is mainly used, such as Figure 2As shown. Each component of the network is approximated using a radial basis function neural network (RBFNN). This dual evaluation architecture enhances the estimation of the value function through an internal goal representation mechanism. The reference network of this setting produces a continuous and dynamically adjustable internal enhancement signal that is derived from the state and control input of the system. The evaluation network evaluates the current system performance by comparing it with this reference signal, identifying the deviation between the actual results and the desired goals. This feedback information informs the efficiency of the system in achieving its goals. The action network is used to approximate the model uncertainties and control torques of the system. This approach minimizes the dependence on external enhancement signals or predefined knowledge, and more effectively reflects the internal goals of the system, promoting a more efficient learning and optimization process.

[0107] Figure 2 Provides a visual representation of the control structure and algorithm flow in the ADP-based FTLM system. The host computer inputs the predetermined trajectory x r , and the error e is obtained after calculating with the FTLM output trajectory x 1 ; Input the error into the action, evaluation and reference neural networks respectively, and then use the algorithm to iteratively update the action, evaluation and reference neural networks to control the trajectory of the robot to fit the predetermined trajectory x r .

[0108] In a specific implementation manner, S2, designing a reference neural network to obtain an internal reinforcement signal of the reference neural network includes the following contents:

[0109] In this step, the RBF neural network can estimate the nonlinear cost function through parallel computing and self-learning capabilities. For a continuous function f(U), the form of the RBF neural network is f(U) = W T S(U), where is the input vector, is the weight vector of the neural network with the number of nodes l>1, S(U)=[s 1 (U),s 2 (U), ..., s l (U)] is the basis function of the neural network. Each s i (U) are all represented by Gaussian functions.

[0110] μ i is the center of the neural network distribution. When the input U is close to μ i When , the basis function of the radial basis function neural network will be activated, resulting in an increase in output. The width of the Gaussian function is determined by η i With this configuration, the radial basis function neural network can accurately estimate the compact set to any continuous function in , to the desired level of accuracy.

[0111] Design a reference neural network, the reference neural network includes an external reinforcement signal and an internal reinforcement signal, wherein the external reinforcement signal is determined according to the environment, the external signal r(t) comes from the environment, and the tracking error e 1 =xx r At the same time, internal signals are generated by the reference network to guide specific behaviors within the system.

[0112] The external reinforcement signal is formula (2):

[0113]

[0114] Among them, e 1 =xx r , e 1 is the tracking error of the flexible manipulator system, x r is the expected trajectory of the system;

[0115] In formula (2) c is a fixed threshold for tracking error. A binary value of 0 indicates good tracking performance, while a value of -1 indicates poor performance.

[0116] In the feedforward phase, the internal reinforcement signal R(t) (in the range of [-1,1]) generated by the reference network is used as the input of the evaluation network. As the system state evolves, R(t) can be adaptively modified. In the feedback phase, the error function uses R(t) to adjust the parameters of the evaluation network. Therefore, R(t) is an important link between the reference network and the evaluation network, effectively integrating them into a unified module.

[0117] Based on the back propagation rule, the internal reinforcement signal is obtained as formula (3):

[0118]

[0119] Among them, the range of R(t) is [-1,1];

[0120] is the expected weight of the reference network;

[0121] is the input of the reference neural network, τ is the control torque of the robot joint;

[0122] S r (U r ) is the basis function of the reference neural network;

[0123] ∈ r is the approximation error of the reference network.

[0124] The error function of the reference neural network is defined as formula (4) and formula (5):

[0125]

[0126] Among them, e r (t) is the error function of the reference network, α is a discount factor less than 1, is the estimated value J of the evaluation network output d (t) is the expected value of the evaluation network output, E r (t) is the quadratic form of the error at time t. It is the error signal e r Half the square of (t), often used as a measure of system performance.

[0127] The formula of the back propagation rule applied to the reference neural network is formula (6):

[0128]

[0129] Among them, σ r is the learning rate of the reference neural network; E r is the quadratic form of the error at time t. It is the error signal e r Half the square of (t), often used as a measure of system performance, are the estimated weights of the reference network.

[0130] In a specific implementation, S3, based on the internal reinforcement signal of the reference neural network, an evaluation neural network is designed to obtain a long-term cost control function, wherein the long-term cost control function is used to find the optimal control that minimizes the operating cost, specifically including:

[0131] Critical neural networks play a vital role in evaluating the effectiveness of mainstream control strategies by fine-tuning learning feedback based on reward or penalty signals from input data. To achieve this goal, the present invention formulates a long-term cost function as a quantitative measure to evaluate the performance of a control strategy over a period of time. By utilizing this cost function, the critical neural network can systematically adjust the control strategy to optimize its performance.

[0132] To achieve this, we design a long-term cost function with the goal of finding the optimal control that minimizes the operating cost.

[0133] The evaluation neural network function is defined as formula (7) and formula (8):

[0134]

[0135] J is the output of the evaluation neural network, W c * To evaluate the expected weight of the network, ∈ cTo evaluate the approximation error of the network, is the estimated weight S of the evaluation network c (U c ) is the basis function for evaluating the neural network, U C The input of the evaluation network To evaluate the estimated value of the network output;

[0136] The neural network input Based on the definition of the evaluation neural network function, a long-term cost control function is obtained, which is formula (9):

[0137]

[0138] J(t) is the long-term cost function, ρ is a constant used to discount future costs, n∈t, κ(t) is the immediate cost function.

[0139] Optionally, the formula of the instantaneous cost function is formula (10):

[0140]

[0141] D is a positive definite constant matrix used to weigh trajectory errors;

[0142] R is a positive definite constant matrix used to weigh the control input;

[0143] The evaluation neural network receives the current state and the action generated by the action neural network as input, and then outputs a value that represents the effect of executing the action generated by the neural network in the current state. The smaller the value, the better the strategy. In this way, the evaluation neural network can evaluate and optimize the current strategy, thereby updating the adaptive rate of the execution neural network for better decision-making.

[0144] The weight update rate of the evaluation neural network is as follows:

[0145]

[0146] σ c To evaluate the learning rate of the network, S c To evaluate the basis function of the network, U C is the input of the evaluation network.

[0147] A specific implementation method, S4, based on the kinematic equation of the flexible double-link mechanical arm, obtain the initial torque of the flexible double-link mechanical arm, based on the action neural network, obtain the control torque after replacement, specifically including:

[0148] The execution neural network receives the system status information, such as the tracking error and vibration error of the system and the output torque at the previous moment. A virtual error term is set to ensure the stability of the system. According to the system status and the feedback of the evaluation function, the control input is dynamically adjusted to optimize the system performance.

[0149] Define the error amount of the flexible two-link robot: e 2 is the system tracking error variable, λ 1 for e 1 Virtual control quantity;

[0150] According to the backstepping method and K 1 is the control gain matrix of the system. Combined with the kinematic equation of the flexible double-link manipulator, e 1 , e 2 By taking the derivative with respect to time, we can obtain formula (12) and formula (13):

[0151]

[0152] Based on formula (12) and formula (13), the initial torque of the flexible double-link manipulator is obtained as formula (14):

[0153]

[0154] τ 0 is the initial torque of the flexible double-link manipulator, K 2 is the control gain matrix of the system;

[0155] Based on the uncertain terms existing in the motion neural network approximation system, the replaced control torque is obtained, and the replaced control torque is formula (15):

[0156]

[0157] Among them, τ 2 is the control torque after replacement, As an estimate of the action neural network weights,

[0158]

[0159] Among them, the system information is input into the evaluation neural network, and the output result is input into the action neural network for training. The formula is implemented. The formula is formula (16):

[0160]

[0161] σ a To enforce the learning rate of the network, β represents a positive constant.

[0162] On the other hand, the present invention provides a stability evaluation method for a robot arm control method based on a dual evaluation network, which is used to evaluate the stability of the robot arm control method based on the dual evaluation network, and comprises the following steps:

[0163] Based on the Lyapunov function, the stability of the robot control method based on the dual evaluation network is evaluated to obtain the global Lyapunov function, and the global Lyapunov function V is formula (17):

[0164]

[0165] By taking the derivative of the global Lyapunov function, we obtain formula (18):

[0166]

[0167] Based on the basic inequality and formula (18), we get formula (19):

[0168]

[0169] Based on formula (19), the robot control method based on the dual evaluation network is obtained to be semi-globally consistent and ultimately bounded.

[0170] Based on this method, the experimental verification is carried out as follows:

[0171] In order to verify the effectiveness and stability of the algorithm proposed in this example, a flexible double-link robotic arm system platform was used for experiments. The host computer uses Matlab software to design the control algorithm, and the output control signal enters the robotic arm motor through the analog-to-digital converter. The basic parameters of the robotic arm are shown in Table 1.

[0172] Table 1: Parameters of the flexible double-link robotic arm experimental platform

[0173]

[0174] In order to compare the control effects of other advanced control methods and the ADP control method in the present invention in a flexible double-link robotic arm system, three control methods, namely NN control, RL control and ADP control, were experimented on the flexible double-link robotic arm system.

[0175] In this section, the NN and RL controllers and the ADP controller proposed in this invention are designed using the Simulink module in MATLAB. The RL and NN controllers are used to compare with the ADP control method proposed in this invention. id(i=1, 2) is selected between 10° and 15°. The control gain of the ADP controller is set to K 1 =[20, 25] T ,K 2 =[0.5, 0.5] T ,α=[0.95,0.95] T ,β=[20,20] T ,c=2.5. The model parameters are set to n 1 =n 2 = 1. In addition, the actor NN has 256 nodes, the critic NN and the Reference NN have 64 nodes. The learning rates of the three NNs are a , σ a and σ c Set to 100, 50 and 1 respectively.

[0176] In order to evaluate the proposed control strategy, the present invention verifies the NN, RL and ADP algorithms respectively on the specified experimental settings. The experimental results are recorded in detail in Figures 3 to 6 middle.

[0177] Figure 3 and Figure 4 An overlay plot showing the comparison angle θ for each control strategy i and α i The vibration of Link 2 between 10 and 12 seconds is caused by the inertial effect of Link 1’s motion. It can be seen that Link 1 is less affected by Link 2’s motion than Link 1. The experimental results capture the real-time adaptation of the controller to the dynamic environment. Figure 5 The differences in tracking errors among the different control systems are described in detail. It is noteworthy that the ADP controller achieves remarkable accuracy in tracking the desired trajectory with the smallest peak and steady-state errors. Figure 6 The control input of the ADP method analyzed in Figure 2 shows a more stable input torque, indicating efficient energy usage and a sophisticated control approach that improves system stability and agility.

[0178] Analyzing the experimental results in detail, ADP control outperforms other methods in terms of tracking performance, especially in terms of maximum tracking error and steady-state error. Its maximum tracking error of the first link reaches the lowest level, which is 6.1 degrees, compared with 7.9 degrees for NN control and 6.9 degrees for RL control. ADP control is also in the lead in terms of vibration suppression, reducing the steady-state vibration of the first link to only 0.28 degrees, which is significantly better than 0.55 degrees under NN control and 0.52 degrees under RL control. Regarding control input, although the maximum input value of ADP control is similar to that of RL control, its steady-state input value is smaller, indicating higher efficiency and precision in control. Overall, ADP control demonstrates excellent performance in these key indicators, highlighting its potential and advantages in flexible manipulator control tasks.

[0179] In contrast, the NN and RL controllers showed larger errors and more obvious oscillation tendencies in trajectory tracking, resulting in poor performance. Similarly, these methods also showed low efficiency in suppressing vibrations within the system, with significant residual vibrations that could affect the life and reliability of the manipulator. In summary, ADP control not only reduced the error range and improved trajectory tracking performance, but also enhanced vibration suppression capabilities, demonstrating its effectiveness in complex dynamic environments.

[0180] The innovative ADP structure proposed in this paper is designed to enhance the vibration reduction and trajectory accuracy of the FTLM system. This enhanced structure integrates the reference network with the typical evaluation network to form a dual evaluation network setting. The reference network enriches the evaluation network by providing an internal representation of the goal, which can approximate the total cost signal in more detail. This internal goal representation is dynamically adjusted according to the state of the system and the actions taken, without the need for historical data or pre-existing knowledge to assign values ​​to the reinforcement signal;

[0181] This paper proposes an adaptive dynamic programming architecture for a flexible two-link manipulator, aiming to track the FTLM system and suppress the elastic vibration of the system. First, the flexible two-link manipulator is discretized and modeled using AMM to obtain the ordinary differential dynamic equations of the manipulator. Then, an ADP structure is designed to combine the reference network with the traditional critic network to form a dual critic network. The reference network updates the critic network by providing an internal reinforcement signal of the target, thereby achieving a more subtle approximation of the total cost signal. This internal target representation is dynamically adjusted according to the state of the system and the actions taken, eliminating the need to assign reinforcement signals to historical data or pre-existing knowledge. The effectiveness of the proposed ADP method based on the dual evaluation network is verified by experimental results. At the same time, the stability and robustness of the closed-loop system under ADP control are analyzed by the Lyapunov direct method, and the semi-global consistent ultimate boundedness of the tracking error and vibration error is proved. The experimental results show that:

[0182] (1) In the trajectory tracking of FTLM, the ADP method using a dual evaluation network significantly outperforms traditional RL control and NN control in reducing time cost and cumulative tracking error;

[0183] (2) In the vibration suppression of FTLM, the ADP method using a dual evaluation network significantly outperforms traditional RL control and NN control in suppressing the vibration of the flexible manipulator.

[0184] The following points need to be explained:

[0185] (1) The drawings of the embodiments of the present invention only relate to the structures related to the embodiments of the present invention, and other structures may refer to the general design.

[0186] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present invention, the thickness of the layers or regions is exaggerated or reduced, that is, these drawings are not drawn according to the actual scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element or there may be intermediate elements.

[0187] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to obtain new embodiments.

[0188] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A robotic arm control method based on a dual evaluation network, characterized in that: include: Establish the kinematic equations of the flexible two-link robotic arm; Design a reference neural network and obtain an internal reinforcement signal of the reference neural network; Based on the internal reinforcement signal of the reference neural network, an evaluation neural network is designed to obtain a long-term cost control function, wherein the long-term cost control function is used to find an optimal control that minimizes the operating cost; Based on the kinematic equation of the flexible double-link robotic arm, the initial torque of the flexible double-link robotic arm is obtained, and based on the action neural network, the replaced control torque is obtained.

2. The robot arm control method based on the dual evaluation network according to claim 1 is characterized in that: The kinematic equation for establishing the flexible double-link robotic arm is formula (1): Among them, τ(t) is the control torque of the joint at time t; H(x) is the stiffness matrix; A(x) is the inertia matrix; are the Coriolis matrix and the centripetal force matrix; The state quantity of the flexible manipulator system is x = [θ, p] T , θ=[θ1,θ2] T represents the joint angle, Represents the generalized coordinates of the flexible two-link robotic arm, and the total degree of freedom is N=n1+n2.

3. The robot arm control method based on the dual evaluation network according to claim 2 is characterized in that: The design of the reference neural network to obtain the internal reinforcement signal of the reference neural network includes: A reference neural network is designed, wherein the reference neural network includes an external reinforcement signal and an internal reinforcement signal, wherein the external reinforcement signal is determined according to the environment, and the external reinforcement signal is formula (2): Among them, c is the upper bound of the system expected error; e1=xx r , e1 is the tracking error of the flexible manipulator system, x r is the expected trajectory of the system; Based on the back propagation rule, the internal reinforcement signal is obtained as formula (3): Among them, the range of R(t) is [-1, 1]; is the expected weight of the reference network; is the input of the reference neural network, τ is the control torque of the robot joint; S r (U r ) is the basis function of the reference neural network; ∈r is the approximation error of the reference network.

4. The robot arm control method based on a dual evaluation network according to claim 3 is characterized in that: The error function of the reference neural network is defined as formula (4) and formula (5): e r (t) is the error function of the reference network and α is a discount factor less than 1. To evaluate the estimated value of the network output, J d (t) is the expected value of the evaluation network output, E r (t) is the quadratic form of the error at time t, E r (t) is the error signal e r Half of the square of (t), used to represent a measure of system performance.

5. The robot arm control method based on the dual evaluation network according to claim 4 is characterized in that: The formula of the back propagation rule applied to the reference neural network is formula (6): Among them, σ r is the learning rate of the reference neural network; E r is the quadratic form of the error at time t, which is the error signal e r Half the square of (t), often used as a measure of system performance, are the estimated weights of the reference network.

6. The robot arm control method based on a dual evaluation network according to claim 5, characterized in that: The designing of the evaluation neural network based on the internal reinforcement signal of the reference neural network to obtain the long-term cost control function comprises: The evaluation neural network function is defined as formula (7) and formula (8): J is the output of the evaluation neural network, W c * is the expected weight of the evaluation network, ∈ c To evaluate the approximation error of the network, is the estimated weight of the evaluation network, S c (U c ) is the basis function for evaluating the neural network, U C To evaluate the input of the network, To evaluate the estimated value of the network output; Based on the definition of the evaluation neural network function, a long-term cost control function is obtained, which is formula (9): J(t) is the long-term cost function, p is the constant for discounting future costs, n∈t, and κ(t) is the immediate cost function.

7. The robot arm control method based on a dual evaluation network according to claim 6, characterized in that: The formula of the instant cost function is formula (10): D is a positive definite constant matrix used to weigh trajectory errors; R is a positive definite constant matrix used to weigh the control input; The weight update rate of the evaluation neural network is as follows: σ c To evaluate the learning rate of the network, S c To evaluate the basis function of the network, U C is the input of the evaluation network.

8. The robot arm control method based on a dual evaluation network according to claim 7, characterized in that: The kinematic equation of the flexible double-link manipulator is used to obtain the initial torque of the flexible double-link manipulator, and the replaced control torque is obtained based on the action neural network, including: Define the error amount of the flexible two-link robot: e2 is the defined system tracking error variable, λ1 is the virtual control variable of e1; According to the backstepping method and K1 is the control gain matrix of the system. Combined with the kinematic equation of the flexible double-link manipulator, the derivatives of e1 and e2 with respect to time are obtained to obtain formulas (12) and (13): Based on formula (12) and formula (13), the initial torque of the flexible double-link manipulator is obtained as formula (14): τ0 is the initial torque of the flexible double-link manipulator, K2 is the control gain matrix of the system; Based on the uncertain terms existing in the motion neural network approximation system, the replaced control torque is obtained, and the replaced control torque is formula (15): Among them, τ2 is the control torque after replacement, As an estimate of the action neural network weights, in, The formula is formula (16): σ a To enforce the learning rate of the network, β represents a positive constant.

9. A stability evaluation method for a robotic arm control method based on a dual evaluation network, characterized in that: The method for evaluating the stability of the robot arm control method based on the dual evaluation network according to any one of claims 1 to 8 comprises the following steps: Based on the Lyapunov function, the stability of the robot control method based on the dual evaluation network is evaluated to obtain a global Lyapunov function, and the global Lyapunov function V is formula (17): By taking the derivative of the global Lyapunov function, we obtain formula (18): Based on the basic inequality and formula (18), we get formula (19): Based on formula (19), the robot control method based on the dual evaluation network is obtained to be semi-globally consistent and ultimately bounded.

Images