Mechanical arm trajectory tracking control method based on data-model fusion driving

Abstract

The invention provides a mechanical arm trajectory tracking control method and system combining data driving and model control. The method comprises the following steps: firstly, establishing a dynamic linearization model (PFDL) for a mechanical arm, and designing a model-free adaptive controller (MFAC); meanwhile, based on trajectory tracking errors, a sliding mode control (SMC) method is introduced to enhance the robustness and control performance of the system. According to the method, a double-delay deterministic strategy gradient algorithm (TD3) in reinforcement learning is creatively fused, and controller parameters are adaptively adjusted through online learning, so that the controller can dynamically find an optimal parameter combination under a fixed structure, and an optimal control effect is realized; according to the method, the dependence of traditional model driving on an accurate mathematical model is broken through, the method can adapt to variable environments and complex trajectories, the control precision and the error convergence speed are remarkably improved, and a new solution is provided for trajectory tracking control of the mechanical arm.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing and automated control technology, and is directed to technical innovations of a trajectory tracking control method and system for a robotic arm, and in particular to a robotic arm trajectory tracking control method based on data-model fusion drive. Background Art

[0002] As the core equipment of modern manufacturing industry, the widespread application of robotic arms in automated production lines has greatly promoted the transformation of production models; they can efficiently and accurately perform various tasks such as material handling, parts assembly, product packaging, etc., which not only significantly improves the efficiency and capacity of production lines, but also effectively reduces the dependence on human resources and labor costs; especially in the field of precision machining, the excellent precision control capability of robotic arms ensures the excellence and consistency of product quality, and meets the market's strict requirements for high-quality products; with the advent of the Industrial 4.0 era, the continuous development of robotic arm control technology has further promoted the digital transformation and intelligent upgrading of the manufacturing industry, and provided key technical support for building smart factories and realizing personalized customization; the national level has also successively introduced a series of support policies to encourage technological innovation, improve product quality and optimize the industrial ecology, providing solid policy guarantees and direction guidance for the vigorous development of the robotic arm industry.

[0003] The trajectory tracking performance of a robotic arm is the key to its ability to complete tasks in a variety of working environments. In high-precision fields such as automotive manufacturing and electronic manufacturing, the robotic arm needs to accurately track the predetermined trajectory to ensure the accuracy of operations such as installation, handling, and inspection. In extreme environment applications such as aerospace, the robotic arm needs to maintain a stable trajectory tracking capability under harsh conditions such as microgravity and strong radiation. Therefore, designing a trajectory tracking control method that is both accurate and robust is crucial to ensuring the reliable operation and efficient operation of the robotic arm in various fields.

[0004] However, the existing trajectory tracking control methods still have many shortcomings. Although the model-driven control method can achieve the control objectives by accurately modeling the physical mechanism and dynamic characteristics of the system, the modeling process is complex and time-consuming, and has high requirements on model accuracy, and is easily affected by modeling errors and environmental changes. The data-driven control method mainly relies on learning the relationship between the input and output of the system from actual data, but lacks the explanatory power of the system's inherent physical mechanism, has limited generalization ability, and is highly dependent on high-quality and large amounts of data.

[0005] Therefore, there is an urgent need for a trajectory tracking control method that can integrate the advantages of model-driven and data-driven to improve the control accuracy and error convergence speed of the robotic arm in a changing environment and complex trajectory; the present invention is proposed precisely to meet this technical demand. Summary of the invention

[0006] The technical problem to be solved by the present invention is to address the shortcomings of the existing technology and to achieve high precision, high robustness and high adaptability of robot arm trajectory tracking control by combining technical means such as dynamic linearization model, model-free adaptive control, sliding mode control and reinforcement learning, and further provide a robot arm trajectory tracking control method based on data-model fusion drive, which can significantly improve the robot arm trajectory tracking accuracy and error convergence speed.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: a robot arm trajectory tracking control method based on data-model fusion drive, comprising the following steps:

[0008] (1) Based on the dynamic linearization theory, a dynamic linearization model (PFDL) of a multi-degree-of-freedom manipulator system is constructed. Based on the dynamic linearization model, a control method of model-free adaptive sliding mode fusion control (IMFAC-SMC) is designed. This method uses a model-free adaptive controller (MFAC) as the base controller and a sliding mode controller (SMC) as the compensation controller. The main control torque is output by using the rolling optimization characteristics of the MFAC to gradually reduce the tracking error, and the compensating control torque is output by taking advantage of the robustness and rapid responsiveness of the SMC to quickly eliminate the error caused by uncertainty and noise.

[0009] (2) The reinforcement learning TD3 algorithm is used to self-adjust the parameters of the IMFAC-SMC controller to ensure that the local optimal parameter combination of the controller at the current moment is found to achieve the optimal control effect.

[0010] The fusion method of the present invention, on the one hand, utilizes the dynamic linearization method to improve the model-driven control, avoiding the tedious robot arm dynamics modeling process; on the other hand, it introduces the reinforcement learning TD3 algorithm for parameter self-adjustment to avoid the tedious manual parameter adjustment process, thereby significantly improving the control performance; in addition, since the method is based on the principle of automatic control feedback, it can break through the limitations of data quality compared to the data-driven control method that purely utilizes learning, thereby improving the mathematical interpretability and robustness of the controller.

[0011] The specific implementation process of step (1) includes:

[0012] 1) Define the control input and output dimensions of the robotic system;

[0013] 2) Given a dynamic linearization data model of the robotic arm;

[0014] 3) Design of model-free adaptive control method using dynamic linearization model;

[0015] 4) Design of discrete sliding mode compensation control method using dynamic linearization model;

[0016] 5) Demonstrate the model-free adaptive sliding mode fusion control process.

[0017] It can be seen from the above process that the present invention constructs a dynamic linearization model of the robot arm and uses the system input and output data to approximately describe the dynamic characteristics of the robot arm system; this method integrates the complexity of the nonlinear system, such as nonlinear behavior, time-varying parameters and structure, into the time-varying parameter matrix, thereby greatly simplifying the control design process; based on this dynamic linearization model, the present invention designs a model-free adaptive control method, which can achieve trajectory tracking without any prior mechanism model; at the same time, the introduction of sliding mode variable structure control provides a robust compensation for the system, further enhancing the control performance.

[0018] The data model of the manipulator dynamic linearization (PFDL) in step 2) is:

[0019] Δx(k+1)=Φ p，L (k)ΔU L (k)+D(k)

[0020] Among them, the robot arm model output is the joint angular velocity; Δx(k+1)=x(k+1)-x(k) is the output change between two adjacent moments, ΔU L (k) = U L (k)-U L (k-1) is the input change between two adjacent moments; the time-varying parameter matrix (PPJM) Φ p，L (k)=[Φ1(k),...,Φ L (k)]∈R n×nL ; D(k)∈R n×1 Represents modeling errors, parameter variations, measurement errors, and other uncertainties.

[0021] The specific design process of the model-free adaptive control method includes: based on the optimal control theory, designing the MFAC control input objective function, including the error term and the penalty term; minimizing the objective function to obtain the MFAC control law; when estimating the time-varying parameter matrix PPJM, designing the objective function, which also has the error term and the penalty term; minimizing the objective function and simplifying it by dimensionality reduction to obtain the parameter estimation algorithm.

[0022] The specific design process of the sliding mode variable structure control method includes: designing the discrete sliding mode surface and the discrete sliding mode reaching law; combining with the dynamic linearization model, the discrete sliding mode control law can be obtained.

[0023] The control law of the model-free adaptive sliding mode fusion controller (IMFAC-SMC) in step 5) is:

[0024] u(k)=u(k-1)+αΔu MFAC (k)+βΔu SMC (k)

[0025] Where α is the change in MFAC control law Δu MFAC The step size factor, β is the SMC control law Δu SMC The step size factor.

[0026] The specific implementation process of step (2) includes:

[0027] 1) Define the algorithm network composition;

[0028] 2) Define the physical meaning of state and action vector;

[0029] 3) Design the gradient calculation process for the current Actor network update;

[0030] 4) Design the gradient calculation process for the current critic network update;

[0031] 5) Summarize the updating method of network parameters in the entire network architecture;

[0032] 6) Design experience playback mechanism;

[0033] 7) Design a reward function that meets the task requirements.

[0034] The present invention converts the state feature vector S t =[e t ∫e t dt] as the state input of the TD3 agent, and the action vector a t =[Δλ t Δκ t Δε t Δα t Δβ t ] as the action output of the TD3 agent; where the tracking error e t =e(k)=x d (k)-x(k),x d (k) is the target tracking trajectory, x(k) is the actual trajectory measured by the sensor; [Δλ t Δκ t Δε t Δα t Δβ t ] are the changes of five key parameters in the IMFAC-SMC controller; the present invention utilizes the reinforcement learning algorithm TD3 and designs a reward function that matches the control task to achieve adaptive adjustment of the key parameters of the IMFAC-SMC controller, thereby improving the control performance of the controller and its ability to adapt to environmental changes.

[0035] The TD3 algorithm includes the following 6 networks: The current network includes the current Actor network π (S t ;θ) and two current critic networks Q1(S t , a t ; w1), Q2(S t , a t ; w2), the parameters are θ, w1, w2 respectively; the target network includes the target Actor network π′(S t+1 ;θ′) and two target critic networks Q′1(s t+1 , a′ t+1 ; w′1), Q′2(s t+1 , a′ t+1 ; w′2), the parameters are θ′, w′1, w′2 respectively; based on TD3 algorithm, a controller parameter self-tuning model is constructed.

[0036] The specific implementation process of step 5) includes: step 3) and step 4) explain the calculation process of the current network gradient; based on this, the current network uses the gradient ascent method to update the network parameters; the target network uses the soft update method to update the network parameters; in addition, a delayed policy update method is designed; specifically, the TD3 algorithm sets a fixed number of delays and only updates the Actor network in each delayed update step, and only updates the Critic network in other time steps; this delayed policy update method can suppress the over-estimation problem to a certain extent by reducing the frequency of policy updates, that is, the misleading caused by overestimation of the target Q value; this helps to improve the convergence and training effect of the algorithm, and makes the training process more stable.

[0037] In order to improve data efficiency and reduce sample correlation, the present invention introduces an experience replay mechanism, that is, the specific implementation process of step 6) includes: setting an experience replay buffer B, storing a state transition tuple (S t , a t , r t , S t+1 ), each time the network is updated, M state transition tuples (S t , a t , r t , S t+1 ), where r t Reward value for each moment.

[0038] The specific design process of the reward function that meets the task requirements includes four aspects of general control performance: error, output control torque, jitter amplitude, and whether it exceeds the torque limit; the intelligent agent automatically finds the optimal parameters to achieve the fastest convergence under the constraints and with a smaller steady-state error.

[0039] Accordingly, the present invention also provides a robot arm trajectory tracking control system based on data-model fusion drive, including a computer device; the computer device is programmed or configured to be used for the steps of the method described in the present invention.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows: Compared with the prior art, the beneficial effects of the present invention are mainly reflected in the following aspects:

[0041] Simplify the modeling process and improve applicability: The present invention adopts a dynamic linearization method to construct a linearized model of the robotic arm system, avoiding the cumbersome and complex robotic arm dynamics modeling process in traditional model-driven control; this method not only greatly simplifies the control design, but also improves the applicability and flexibility of the control method, so that the present invention can be more widely applied to different types of robotic arm systems.

[0042] Fusion control strategy to improve control performance: The present invention designs a data-model fusion robot trajectory tracking control framework TD3-IMFAC-SMC; this framework combines the advantages of model-free adaptive control (MFAC) and sliding mode control (SMC), MFAC is used as the base controller to provide basic control torque, while SMC is used as a compensation controller to quickly eliminate errors caused by uncertainty and noise through its robustness and fast responsiveness; this fusion control strategy significantly improves the trajectory tracking accuracy and error convergence speed.

[0043] Reinforcement learning adaptive parameter setting to optimize control effect: The present invention innovatively introduces the TD3 reinforcement learning algorithm to perform self-adjustment of parameters of the IMFAC-SMC controller; by designing a reward function that matches the control task, the TD3 algorithm can automatically find the optimal parameter combination, so that the controller can achieve the optimal control effect under the current environment; this adaptive parameter setting method not only avoids the tedious manual parameter adjustment process, but also significantly improves the performance and stability of the controller.

[0044] Ensure system stability and provide efficient solutions: Without prior knowledge, it is difficult for traditional reinforcement learning agents to directly control the torque of a multi-degree-of-freedom robotic arm. The present invention uses TD3 reinforcement learning to perform adaptive parameter setting to achieve precise control of compensation control, which not only optimizes control performance but also prioritizes system stability. This provides an efficient and reliable solution for complex robotic arm control.

[0045] Specially designed reward function significantly improves trajectory tracking effect: The present invention designs a reward function specifically for the robot arm trajectory tracking task, which comprehensively considers the general control performance in four aspects, namely, error, output control torque, jitter amplitude, and whether the torque limit is exceeded; the intelligent agent automatically searches for the optimal parameters, so that the trajectory tracking effect under the constraints is significantly improved, the steady-state error is smaller, and the convergence speed is faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the overall architecture of the data-model fusion drive control according to the present invention;

[0047] Figure 2 It is the framework of model-free adaptive sliding mode fusion controller (IMFAC-SMC);

[0048] Figure 3 It is the geometric interpretation of the dynamic linearization model (PFDL);

[0049] Figure 4 It is the overall training flow chart of the TD3 algorithm designed by the present invention;

[0050] Figure 5 It is the Sawyer robot experimental platform used in the embodiments of the present invention;

[0051] Figure 6 It is the MATLAB / Simulink-ROS communication framework designed by the embodiment of the present invention;

[0052] Figure 7 It is the built-in underlying control framework of the Sawyer robot;

[0053] Figure 8 This is a comparison chart of the experimental results of the method of the present invention and the built-in controller of the Sawyer robot;

[0054] Fig. 9 It is a comparison chart of experimental results of the method of the present invention and various control methods. DETAILED DESCRIPTION

[0055] The system architecture of the present invention is as follows Figure 1 As shown; a dynamic linearization model (PFDL) of a multi-degree-of-freedom manipulator system is constructed; a model-free adaptive sliding mode fusion controller (IMFAC-SMC) is designed based on the model; the manipulator state error and error integral are used as the state input of the TD3 intelligent agent, and the five key parameters of the IMFAC-SMC are used as the action output of the TD3 intelligent agent; the TD3 algorithm is used to dynamically adjust the key parameters of the IMFAC-SMC controller; the present invention finally obtains a new type of TD3-IMFAC-SMC controller.

[0056] The basic framework of the model-free adaptive sliding mode fusion controller (IMFAC-SMC) is as follows: Figure 2 As shown, the specific design steps are as follows:

[0057] (1) Define the control input and robot system output dimensions

[0058] Considering how each input change in a time window of a specific length affects the output change at the next moment, the control input U is defined as L (k) is a nL×1 The vector in the space is composed of all control input signals in the time window [k-L+1, k]. The specific expression is as follows:

[0059] U L (k)=[u T (k),u T (k-1),...,u T (k-L+1)] T #(1)

[0060] Where u(k)∈R n×1 Represents the joint control torque input, and the positive integer L is the control input length constant.

[0061] (2) Given the dynamic linearization data model of the robot arm

[0062] For the robot trajectory tracking control problem, the robot dynamic linearization (PFDL) data model is:

[0063] Δx(k+1)=Φ p，L (k)ΔU L (k)+D(k)#(2)

[0064] Among them, the robot arm model output is the joint angular velocity; Δx(k+1)=x(k+1)-x(k) is the output change between two adjacent moments, ΔU L (k) = U L (k)-U L (k-1) is the input change between two adjacent moments; the time-varying parameter matrix (PPJM) Φ p，L (k)=[Φ1(k),...,Φ L (k)]∈R n×nL ; D(k)∈R n×1 Represents modeling errors, parameter variations, measurement errors, and other uncertainties.

[0065] The geometric interpretation of this model is as follows Figure 3As shown in the figure, the dynamic linearization method approximates the dynamic characteristics of the nonlinear system by using the system input and output data; its core is to estimate the linearization coefficient Φ in real time. p，L (k) in order to capture the dynamic changes of the system; this method incorporates the complexity of the nonlinear system, such as nonlinear behavior, time-varying parameters and structure, into the time-varying parameter matrix Φ p，L (k) The control design process is simplified and the tedious robot arm dynamics modeling process is avoided.

[0066] 3) Design of model-free adaptive control method based on dynamic linearization model

[0067] Based on the optimal control theory, the following function is considered as the MFAC control input objective function:

[0068] J(u(k))=(x d (k+1)-x(k+1)) 2 +λ(u(k)-u(k-1)) 2 #(3)

[0069] Among them, x d (k+1) is the joint angular velocity command at time k+1, λ>0 is the penalty factor, and the penalty term is λ(u(k)-u(k-1)) 2 The function is to limit the change of the control input Δu(k) to ensure that the control input signal has a certain smoothness; it is inferred that the only state x(k+1) at the next moment is related to the control input u(k) at the current moment, that is,

[0070]

[0071] Based on the above inference, substitute equation (2) into equation (3) to minimize the objective function (3). In order to simplify the control law, the high-dimensional n×n inverse matrix is ​​reduced to a scalar:

[0072] u(k)=u(k-1)+αΔu MFAC (k)#(5)

[0073]

[0074] Among them, α∈(0,1] is the additional step size factor added to the control law to directly adjust the control input change, which is used to compensate for the information loss caused by the dimensionality reduction transformation. When estimating PPJM, such an objective function is established:

[0075]

[0076] in, is the estimated value of PPJMΦp,L(k); Λ>0 is the penalty factor of the objective function; minimizing the objective function (7) and simplifying it by dimensionality reduction, we can get the PPJM estimation algorithm:

[0077]

[0078] Among them, A∈(0, 2] is the additional step size factor added in the PPJM estimation algorithm.

[0079] 4) Design of discrete sliding mode compensation control method based on dynamic linearization model

[0080] The discrete sliding surface is designed as

[0081] s(k)=Ce(k)#(9)

[0082] The discrete sliding mode reaching law is designed as:

[0083]

[0084] Combine equations (2), (9), and (10), and use Δu to express the change in the control law of the SMC compensation control part. SMC (k) indicates that the sliding mode compensation control law can be obtained:

[0085]

[0086] 5) Model-free adaptive sliding mode fusion control process

[0087] Based on the dynamic linearization model, the present invention designs a control method of model-free adaptive sliding mode fusion control (IMFAC-SMC); the method uses a model-free adaptive controller (MFAC) as a base controller and a sliding mode controller (SMC) as a compensation controller; the MFAC rolling optimization characteristic is used to output the main control torque to gradually reduce the tracking error, and the robustness and rapid responsiveness of the SMC are used to output the compensation control torque to quickly eliminate the error caused by uncertainty and noise.

[0088] The model-free adaptive sliding mode fusion controller (IMFAC-SMC) requires two parts of controller design. Combining equations (5) and (11), the IMFAC-SMC control law is designed as:

[0089] u(k)=u(k-1)+αΔu MFAC (k)+βΔu SMC (k)#(12)

[0090] Where α is Δu MFAC The step size factor, β is Δu SMC The step size factor.

[0091] The overall process of TD3 network for adaptive adjustment of key parameters of IMFAC-SMC is as follows: Figure 1 As shown on the right, the robot state error and error integral are used as the state input of the TD3 agent, and the five key parameters of IMFAC-SMC are used as the action output of the TD3 agent. The TD3 algorithm is used to dynamically adjust the key parameters of the IMFAC-SMC controller.

[0092] 1) Define the TD3 algorithm network composition

[0093] The TD3 algorithm includes the following 6 networks: The current network includes the current Actor network π (S t ;θ) and two current Critic networks Q1(S t , a t ; w1), Q2(S t , a t ; w2), the parameters are θ, w1, w2 respectively; the target network includes the target Actor network π′(S t+1 ;θ′) and two target critic networks Q′1(s t+1 , a′ t+1 ; w′1), Q′2(s t+1 , a′ t+1 ; w′2), the parameters are θ′, w′1, w′2 respectively; based on TD3 algorithm, a controller parameter self-tuning model is constructed.

[0094] 2) Define the physical meaning of state and action vector

[0095] The present invention converts the state feature vector S t =[e t ∫e t dt] as the state input of the TD3 agent, and the action vector a t =[Δλ t Δκ t Δε t Δα t Δβ t ] as the action output of the TD3 agent; where the tracking error e t =e(k)=x d (k)-x(k),x d (k) is the target tracking trajectory, x(k) is the actual trajectory measured by the sensor; [Δλ t Δκ t Δε t Δα t Δβ t ] are the five key parameter changes in the IMFAC-SMC controller.

[0096] 3) Design the gradient calculation process for the current Actor network update

[0097] The current policy is defined as:

[0098] a t =π(S t ;θ)#(13)

[0099] Among them, action a t Indicates the current state S t The parameter change in the lower controller [Δλ t Δκ t Δε t Δα t Δβ t ].

[0100] The objective function of the TD3 algorithm can be defined as,

[0101]

[0102] Among them, γ is the discount factor, which adjusts the impact of long-term and short-term rewards; r t Indicates the current state S t The reward function value under .

[0103] Secondly, combined with the above parameters, the deterministic policy gradient algorithm is used to update the current Actor network π. The gradient calculation method is as follows:

[0104]

[0105] Among them, Q(S t , a t ; w) We make the following explanation: For the Actor network π(S t ; θ), its task is to do gradient ascent to find the maximum Q value. As the update proceeds, the two networks Q1 and Q2 will become more and more similar. Selecting Q1 or Q2 for the Actor network π(S t ; θ) update has little effect; the present invention selects Q=Q1 when the Actor network π is updated.

[0106] 4) Design the gradient calculation process for the current Critic network update

[0107] First, when calculating the target action value, add zero-mean Gaussian noise ∈~clip(N(0,σ 2 ), a min , a max ), the sampled action will be cut to [a min , a max ]; therefore,

[0108] a′ t+1 =π′(S t+1 ;θ′)+∈#(16)

[0109] Here, we choose not to add noise to the current Actor network π, so as to improve the utilization of the algorithm and make it more stable and convergent. In addition, we add noise to the target Actor network π′. The purpose of this design is to find a balance between exploration and utilization. Utilization is to make the best decision under the current information (i.e., no noise is added and deterministic actions are taken), while exploration is to try different behaviors and then collect more information (i.e., add noise and introduce randomness to explore the entire vector space). t+1 As the input of the target Critic network Q'. In order to alleviate the overestimation problem of the current Critic network Q, the target value Q required for its parameter update target It is calculated using the minimum value obtained from the two target Critic networks Q′, that is

[0110]

[0111] Secondly, combined with the above parameters, the parameters w1 and w2 of the two current Critic networks Q are updated by minimizing the loss function, and the gradient is calculated as follows:

[0112]

[0113] Among them, i = 1, 2, M is the state transition tuple (S t , a t , r t , S t+1 ) number.

[0114] 5) Summarize the update method of network parameters in the entire network architecture

[0115] Step 3) and step 4) calculate the update gradient of the current network; based on this, the current network uses the gradient ascent method to update the network parameters; the target network uses the soft update method to update the network parameters; the 6 network parameter update methods are as follows:

[0116]

[0117] θ′←τθ+(1-τ)θ′#(22)

[0118] w′ i ←τw i +(1-τ)w′ i #(twenty three)

[0119] Among them, d π , d Qare the learning rates of the current Actor network π and the current Critic network Q respectively, τ is the smoothing factor of soft update, 0<τ<<1.

[0120] In addition, a delayed policy update method is designed; specifically, the TD3 algorithm sets a fixed number of delays and only updates the Actor network in each delayed update step, and only updates the Critic network in other time steps; this delayed policy update method can suppress the over-estimation problem to a certain extent, that is, the misleading caused by overestimation of the target Q value, by reducing the frequency of policy updates; this helps to improve the convergence and training effect of the algorithm and make the training process more stable.

[0121] 6) Introducing experience replay mechanism

[0122] In order to improve data efficiency and reduce sample correlation, it is necessary to set up an experience replay buffer B, in which a state transition tuple (S t , a t , r t , S t+1 ), each time the network is updated, M state transition tuples (S t , a t , r t , S t+1 ).

[0123] 7) Design a reward function that meets the task requirements

[0124] The reward function design of the present invention mainly considers four aspects of general control performance: error, output control torque, jitter amplitude and whether it exceeds the torque limit; the intelligent agent automatically finds the optimal parameters so that convergence can be achieved as quickly as possible under the constraints and the steady-state error is smaller.

[0125] The reward function is designed as:

[0126] r t =r e +r u +r h -1000B#(24)

[0127] Among them, r e is the error penalty term, which is calculated as formula (25). When the error is large, logarithmic decay is used, that is, the error is forcibly suppressed. The purpose of this is to speed up the convergence of the error and quickly reach the optimal solution point (torque and error are balanced). When the error is small, a linear function is used to avoid the problem that when the error is small, the reward function changes too little and the convergence speed is slow. This idea is to significantly increase the value of the reward function when approaching the optimal solution to promote the reinforcement learning process to converge quickly to the optimal solution.

[0128]

[0129] r u is the control input penalty term, and its calculation method is as follows: (26); if the task requires high dynamic performance and steady-state performance, it is necessary to appropriately relax the controller input torque requirement.

[0130] r u =-||u t -1||2×10 -4 #(26)

[0131] r h is the oscillation amplitude penalty term, calculated as Equation (27); it is the steady-state oscillation amplitude of the discrete sliding mode motion when t→∞; in this equation, it can be seen that the jitter amplitude is the sum of the two parameters ε t and κ t If the two parameters are related, they can be optimized automatically; a large amplitude will introduce a penalty; however, the two parameters cannot be infinitely large or small, which will affect other control performance indicators, such as convergence speed, overshoot and jitter.

[0132]

[0133] B takes the value of 1 or 0, which is used to indicate whether the control input is out of range.

[0134] The overall training process of the TD3 algorithm is as follows Figure 4 As shown in the figure, at the beginning of training, initialization is first required, that is, initializing the parameters of the Actor network and the two Critic networks; initializing the target network parameters; initializing the experience replay buffer; then, sampling trajectories from the environment and storing the experience in the experience replay buffer; in this process, the Actor network generates actions, adds noise to promote exploration, and (S t , a t , r t , S t+1) The experience tuples are stored in the experience replay buffer; when the experience replay buffer reaches its storage capacity limit, the system randomly samples a small batch of experience tuples from it; these samples are then used to calculate the target Q value and the error of the Critic network, and the Critic network parameters are updated using the gradient descent method; in the parameter update process of the Actor network, a delayed update strategy is adopted, that is, the Actor network is updated only after a certain number of update steps; the Actor network uses the gradient ascent method to maximize the Q value output by the Critic network; then, the parameters of the target network are updated using the soft update method; finally, the above steps are repeated, and the parameters of the Actor and Critic networks are continuously optimized by repeatedly sampling and updating the network parameters from the experience replay buffer until the predetermined number of training rounds is reached or specific performance standards are met; the training process of the TD3 algorithm continuously collects experience, updates network parameters, and uses the data in the experience replay buffer for learning to gradually improve the performance of the Actor and Critic networks, thereby achieving better control of the robotic arm.

[0135] Comparative experiment

[0136] The present invention builds an IMFAC-SMC control system based on the MATLAB / simulink platform, and uses the TD3 algorithm designed specifically for the IMFAC-SMC controller to adaptively adjust key parameters in the controller; a seven-degree-of-freedom tandem robotic arm named Sawyer is selected as the actual controlled object; and the practical application value of the present invention is proved by comparison with other methods.

[0137] 1) Experimental platform construction

[0138] The present invention is based on Figure 5 The Sawyer robot platform shown conducted trajectory tracking experiments; the Sawyer robot is a seven-degree-of-freedom collaborative robotic arm developed based on the open source Robot Operating System (ROS).

[0139] Considering that MATLAB / simulink is more convenient to establish a control system, and in order to avoid the tediousness of porting MATLAB / simulink code to a real machine, the present invention considers using MATLAB / simulink and ROS to jointly communicate to achieve rapid verification of the control algorithm; the MATLAB / Simulink-ROS communication framework of the present invention is as follows Figure 6As shown in the figure, the communication framework consists of MATLAB / Simulink under the ROS system, the custom message format conversion node Communication, and the Sawyer robotic arm. During communication, all three parts are regarded as nodes, and the topic communication of the ROS system is used for data transmission.

[0140] The Sawyer robot provides an independent ROS Master, so that any workstation computer (located in this ROS host network) can connect to and control the Sawyer robot via ROS APIs; the basic communication process is: first, establish communication between the robotic arm and the melodic version of the ROS system in the desktop ubuntu18.04; then, MATLAB / Simulink publishes control commands as the publisher, and the information is received by the node Communication as the receiver, formatted and repackaged and sent to the Sawyer robotic arm; the role of the message format conversion node Communication is: Since Simulink cannot process character data, and the Sawyer node contains a lot of character data, it needs to be written to rewrite the message content.

[0141] 2) Experimental conditions setting

[0142] The Sawyer robot arm has three control modes: position control mode, speed control mode and torque control mode. Its underlying control framework is as follows: Figure 7 As shown; by connecting the host computer to the interface of these three control modes, control commands of different modes can be sent to realize the control of the robot arm; the trajectory tracking control methods of traditional industrial robot arms are almost all based on position or speed control modes; its essence is the PID control algorithm packaged in the underlying controller of each joint; the TD3-IMFAC-SMC control algorithm designed by the present invention adopts the torque control method, and its interface is the interface of the built-in torque control mode of Sawyer; in order to verify the superiority of this method over the position and speed control method of the robot arm, it is necessary to conduct a comparative experiment on the control effects of the three modes; in addition, in order to prove the superiority of TD3-IMFAC-SMC over other model-free control methods, it is also necessary to conduct comparative experiments on multiple model-free control methods; since the position and speed control mode of the Sawyer robot arm limits the joint angular velocity to 0.4rad / s, x is selected in all experiments in this section d =0.3cost is the target tracking trajectory of each joint; at the same time, in order to avoid damage to the robotic arm due to excessive torque, the torque output instructions of the present invention are limited to [-20, 20] N·m; in addition, the parameter settings of the TD3 algorithm are shown in Table 1.

[0143] Table 1 TD3 basic parameter settings

[0144]

[0145]

[0146] 4) Comparison between the proposed algorithm and the Sawyer robot position and speed control mode

[0147] The control effects of the three control modes are as follows: Figure 8 As shown; in terms of overshoot, the 3.8% under the TD3-IMFAC-SMC control algorithm is much smaller than the 16.7% under the position control mode and the 15.6% under the speed control mode; in terms of system response speed, the rise time is used to evaluate; the 0.12s under the TD3-IMFAC-SMC control algorithm is much smaller than the 1.135s under the speed control mode and the 0.145s under the position control mode; in terms of steady-state error fluctuation range, the [-0.02, 0.03] under the TD3-IMFAC-SMC control algorithm is slightly larger than [-0.013, 0.011], and much smaller than [-0.048, 0.073]; therefore, except that the TD3-IMFAC-SMC control algorithm is slightly inferior to the speed control mode in terms of steady-state error fluctuation range, the other control performance indicators are better than the position control mode and speed control mode set by the Sawyer underlying controller.

[0148] 5) Comparison of experimental results of various control methods

[0149] The effects of various control methods applied to real objects are as follows: Fig. 9 As shown; Fig. 9 (a) shows the joint angular velocity tracking effect. Fig. 9 (b) represents the magnitude of the input torque controlled by each model-free control algorithm; Fig. 9 As shown in (b), under the neural network sliding mode control (NNSMC) method, the robot arm vibrates violently, which is obviously not suitable for trajectory tracking control of the Sawyer robot arm alone; MFAC has no vibration phenomenon, while IMFAC-SMC and TD3-IMFAC-SMC will have a small amplitude vibration, and the vibration amplitude of TD3-IMFAC-SMC is smaller than that of IMFAC-SMC; Therefore, it can be seen that adding the reinforcement learning TD3 algorithm to automatically adjust the parameters of the IMFAC-SMC controller can reduce the vibration phenomenon of the robot arm; Fig. 9 As shown in (a), under the MFAC control method, the robot arm did not track the target trajectory within 10 seconds; the joint angular velocity jitter of the NNSMC control method was also serious; IMFAC-SMC and TD3-IMFAC-SMC could both achieve good control effects; since the PID control method has been compared in 4), the PID control method will not be discussed here.

[0150] In summary, in the actual machine experimental verification, the IMFAC-SMC control method is better than the traditional control method; after adding the TD3 reinforcement learning algorithm for parameter self-adjustment, the IMFAC-SMC controller forms a control algorithm called TD3-IMFAC-SMC; compared with the IMFAC-SMC controller with fixed parameters, the algorithm has enhanced environmental adaptability, better control effect and the control method is far superior to other traditional control methods.

Claims

1. A robot arm trajectory tracking control method based on data-model fusion drive, characterized in that: The following steps are involved: (1) Based on the dynamic linearization theory, a dynamic linearization model (PFDL) of a multi-degree-of-freedom manipulator system is constructed. Based on the dynamic linearization model, a control method of model-free adaptive sliding mode fusion control (IMFAC-SMC) is designed. This method uses a model-free adaptive controller (MFAC) as a base controller and a sliding mode controller (SMC) as a compensation controller. The main control torque is output by using the rolling optimization characteristics of the MFAC to gradually reduce the tracking error, and the compensating control torque is output by using the robustness and rapid responsiveness of the SMC to quickly eliminate the error caused by uncertainty and noise. (2) The reinforcement learning TD3 algorithm is used to self-adjust the parameters of the IMFAC-SMC controller to ensure that the local optimal parameter combination of the controller at the current moment is found to achieve the optimal control effect.

2. The robot arm trajectory tracking control method based on data-model fusion drive according to claim 1 is characterized in that: The specific implementation process of step (1) includes: 1) Define the control input and output dimensions of the robotic system; 2) Given a dynamic linearization data model of the robotic arm; 3) Design of model-free adaptive control method using dynamic linearization model; 4) Design of discrete sliding mode compensation control method using dynamic linearization model; 5) Demonstrate the model-free adaptive sliding mode fusion control process.

3. The robot arm trajectory tracking control method based on data-model fusion drive according to claim 2 is characterized in that: The data model of the manipulator dynamic linearization (PFDL) in step 2) is: Δx(k+1)=Φ p,L (k)ΔU L (k)+D(k) Among them, the robot arm model output is the joint angular velocity; Δx(k+1)=x(k+1)-x(k) is the output change between two adjacent moments, ΔU L (k) = U L (k)-U L (k-1) is the input change between two adjacent moments; the time-varying parameter matrix (PPJM) Φ p,L (k)=[Φ1(k),…,Φ L (k)]∈R n×nL ; D(k)∈R n×1 Represents modeling errors, parameter variations, measurement errors, and other uncertainties.

4. The robot arm trajectory tracking control method based on data-model fusion drive according to claim 2 is characterized in that: The specific design process of the model-free adaptive control method includes: based on the optimal control theory, designing the MFAC control input objective function, including the error term and the penalty term; minimizing the objective function to obtain the MFAC control law; when estimating the time-varying parameter matrix PPJM, designing the objective function, which also has the error term and the penalty term; minimizing the objective function and simplifying it by dimensionality reduction to obtain the parameter estimation algorithm.

5. The robot arm trajectory tracking control method based on data-model fusion drive according to claim 2 is characterized in that: The control law of the model-free adaptive sliding mode fusion controller (IMFAC-SMC) in step 5) is: u(k)=u(k-1)+αΔu MFAC (k)+βΔu SMC (k) Where α is the change in MFAC control law Δu MFAC The step size factor, β is the SMC control law Δu SMC The step size factor.

6. The robot arm trajectory tracking control method based on data-model fusion drive according to claim 1 is characterized in that: The specific implementation process of step (2) includes: 1) Define the algorithm network composition; 2) Define the physical meaning of state and action vector; 3) Design the gradient calculation process for the current Actor network update; 4) Design the gradient calculation process for the current critic network update; 5) Summarize the updating method of network parameters in the entire network architecture; 6) Design experience playback mechanism; 7) Design a reward function that meets the task requirements.

7. The robot arm trajectory tracking control method based on data-model fusion drive according to claim 6 is characterized in that: The state feature vector S t =[e t ∫e t dt] as the state input of the TD3 agent, and the action vector a t =[Δλ t Δκ t Δε t Δα t Δβ t ] as the action output of the TD3 agent; where the tracking error e t =e(k)=x d (k)-x(k),x d (k) is the target tracking trajectory, x(k) is the actual trajectory measured by the sensor; [Δλ t Δκ t Δε t Δα t Δβ t ] are the changes of five key parameters in the IMFAC-SMC controller; the present invention utilizes the reinforcement learning algorithm TD3 and designs a reward function that matches the control task to achieve adaptive adjustment of the key parameters of the IMFAC-SMC controller, thereby improving the control performance of the controller and its ability to adapt to environmental changes.

8. The robot arm trajectory tracking control method based on data-model fusion drive according to claim 7 is characterized in that: The TD3 algorithm includes the following 6 networks: The current network includes the current Actor network π (S t ;θ) and two current critic networks Q1(S t ,a t ;w1),Q2(S t ,a t ; w2), the parameters are θ, w1, w2 respectively; the target network includes the target Actor network π′(S t+1 ;θ′) and two target critic networks Q′1(S t+1 ,a′ t+1 ;w′1),Q′2(S t+1 ,a′ t+1 ; w′2), the parameters are θ′, w′1, w′2 respectively; based on the TD3 algorithm, a controller parameter self-tuning model is constructed.

9. The robot arm trajectory tracking control method based on data-model fusion drive according to claim 6, characterized in that: The process of implementing step 5) is as follows: first, the current network updates its parameters through the gradient ascent method, while the target network uses the soft update method to update its parameters; secondly, a delayed strategy update method is introduced, that is, by setting a fixed number of delays, the Actor network is updated only in a specific delayed update step, and only the Critic network is updated in other time steps.

10. The robot arm trajectory tracking control method based on data-model fusion drive according to claim 8, characterized in that: In order to improve data efficiency and reduce sample correlation, the experience replay mechanism is introduced. The specific implementation process of step 6) includes: setting an experience replay buffer B, storing a state transition tuple (S t ,a t ,r t ,S t+1 ), each time the network is updated, M state transition tuples (S t ,a t ,r t ,S t+1 ), where r t Reward value for each moment.

Images