Flexible joint mechanical arm tracking control method and device based on reinforcement learning

Abstract

The invention relates to the technical field of robot intelligent control, in particular to a flexible joint mechanical arm tracking control method and device based on reinforcement learning. The method comprises the following steps: estimating uncertainty and actuator faults in a system model by adopting an Actor-Critic network so as to ensure that the system performance is kept stable even if the system experiences unpredictable changes; the non-singular terminal sliding mode control method is adopted to avoid the occurrence of singular points, so that the continuity and robustness of the control law are ensured, and the accurate trajectory tracking of the robot system is also ensured. The method considers that model parameters of the flexible joint mechanical arm may deviate due to different parameters such as spring stiffness, connecting rod length and mass, so that the inherent uncertainty in the model is estimated by adopting reinforcement learning. According to the strategy based on reinforcement learning (RL) and non-singular terminal sliding mode control (NTSMC), the uncertainty and unknown interference of the system can be compensated while trajectory tracking is kept.

Description

Technical Field

[0001] The present invention relates to the technical field of robot intelligent control, and in particular to a flexible joint robot arm tracking control method and device based on reinforcement learning. Background Art

[0002] In the field of robotics research, trajectory tracking has always been a focus of attention in robotic arm research. It has been widely used in aerospace, automobile manufacturing, industrial production, and service industries. Although traditional rigid robotic arms perform well in tasks that require high repeatability and precision, they often have limited degrees of freedom and can only work along preset paths, which leads to poor adaptability to complex or variable environments. In addition, they may lack the necessary flexibility when interacting with humans or other dynamic systems, which may not only cause safety hazards during operation, but also reduce efficiency. With the advancement of automation technology and the rapid development of artificial intelligence, people have put forward higher requirements for the trajectory tracking capabilities of robotic arms. At present, the application scenarios of trajectory tracking are no longer limited to highly structured environments, but are getting closer and closer to the real complex world. In order to overcome these challenges, flexible joint robotic arms have emerged. Compared with traditional rigid robotic arms, flexible joint robotic arms have more degrees of freedom, can more accurately simulate the movements of human hands, and show higher flexibility and adaptability. The dynamic model of flexible joint robots is quite complex, and their nonlinear and strong coupling characteristics bring challenges to the design of controllers. Therefore, it is crucial to design effective control algorithms for flexible joint robots. In this context, reinforcement learning provides new possibilities for robot trajectory tracking. Domestic and foreign researchers have widely adopted control algorithms based on reinforcement learning in robot trajectory tracking research. By continuously interacting with the environment to learn the optimal strategy, the robot can achieve accurate trajectory tracking in various complex environments. This method greatly improves the trajectory tracking performance of the robot in unstructured environments.

[0003] Ideally, when the external environment or parameters do not change much, traditional control methods can still meet the control requirements. However, they have some significant disadvantages. First, these methods usually rely on an accurate understanding of the system model, which is difficult to achieve for highly nonlinear and complex systems because the simplified model may lose a lot of important information. Second, traditional control theory mainly targets linear steady-state systems, while systems in the real world often have nonlinear, time-varying and uncertain characteristics, which limits the application of traditional methods in these cases. In addition, traditional control methods also face challenges when dealing with multivariable systems and constrained control problems. They may find it difficult to meet the needs of real-time control, especially when the system dynamics change rapidly. In practical applications, when the controlled object has severe nonlinearity, uncertainty in the mathematical model, and drastic changes in the system operating point, the effectiveness of traditional control methods will be greatly limited. Therefore, the parameter adjustment process of traditional control systems is complex and time-consuming, and there are limitations in system scalability. Therefore, when facing modern complex and dynamic systems, these control methods may need to be combined with more advanced control strategies to improve their performance and adaptability. Summary of the invention

[0004] In order to solve the technical problem in the prior art that the potential risk of system uncertainty and actuator failure may have an adverse effect on the control accuracy, the embodiment of the present invention provides a flexible joint robot arm tracking control method and device based on reinforcement learning. The technical solution is as follows:

[0005] On the one hand, a flexible joint robot arm tracking control method based on reinforcement learning (RL) is provided, characterized in that the method comprises:

[0006] S1. Build a flexible joint robot platform, establish a coordinate system connecting the joints of the robot arm based on the DH (denavit-hartenberg, DH) method, and obtain the kinematic equation of the robot system;

[0007] S2. Based on the kinematic equations of the robot system, a reinforcement learning controller with sudden actuator failure is constructed to track and maintain the flexible joint robot platform while compensating for the uncertainty and unknown interference of the system; the design of the reinforcement learning controller with sudden actuator failure includes: designing an evaluation neural network and designing an execution neural network;

[0008] S3. Build a simulation platform, conduct simulation experiments based on a flexible joint robot platform and a reinforcement learning controller with sudden actuator failure, and complete the flexible joint robotic arm tracking control based on reinforcement learning.

[0009] Optionally, in S1, the kinematic equations of the robot system include:

[0010]

[0011] in, is the position of the connecting rod, is the speed of the connecting rod, is the acceleration of the connecting rod; is the inertia matrix; is the centrifugal force-Coriolis matrix; M(η)=M0(η)+M Δ (η) is gravity; C0(η), M0(η) is known; C Δ (η), M Δ (η) is unknown but bounded, is the control input, It is an actuator failure.

[0012] Optionally, in S2, designing an evaluation neural network includes:

[0013] Design Actor-Critic network, including Actor NN (neural network, NN) and CriticNN;

[0014] Among them, Actor NN is used to generate control strategies that directly interact with the environment and make decisions;

[0015] Critic NN is used to evaluate the quality and results of actions produced by the execution network and provide feedback signals to guide the execution network to adjust its strategy.

[0016] Optionally, based on the Actor-Critic network including:

[0017] Design the radial basis function neural network (RBFNN) as follows:

[0018] in, represents the weight, f(z) is a continuous function, and the input is S(z) represents the basis function:

[0019]

[0020] A long-term cost function is formulated to measure and optimize the overall performance of the robot after making decisions.

[0021] Optionally, a long-term cost function, comprising:

[0022]

[0023] Among them, J represents the discount constant of future costs; φ(t) represents the cost function; and t represents time.

[0024] Optionally, the weight update rate of the evaluation neural network is shown in the following formula:

[0025]

[0026] Among them, δ c >0 is the learning rate, Λ is S c (z c ), c represents a subscript, which is the abbreviation of critic; represents the weight update rate of the evaluation neural network; ψ represents a constant greater than 0; S c (z c ) represents the evaluation network basis function; Indicates S c (z c ) and the gradient of z c Product of derivatives;

[0027] After the evaluation neural network takes the current state as input, it will output a scalar value representing the estimate of the long-term reward that can be obtained by following the current policy starting from this state.

[0028] Optionally, design and implement a neural network, including:

[0029] The nonsingular terminal sliding mode control algorithm (NTSMC) is constructed according to the following formula:

[0030]

[0031] in, m=diag(m1,m2,…,m n ) is a positive definite matrix, q1 and q2 satisfy A positive odd number of

[0032] Calculate the initial input of the robot system according to the kinematic equations of the robot system;

[0033] The uncertainty in the system is approximated by the following formula:

[0034]

[0035] in, represents the predicted value of the weight, represents the ideal value, and S a (z a ) represents the basis function of executing the neural network, ε a represents the estimation error of the actor neural network;

[0036] The weight update rate of the neural network is:

[0037]

[0038] Among them, δ a >0 is the learning rate, is the cost estimation error.

[0039] On the other hand, a flexible joint robot arm tracking control device based on reinforcement learning is provided, and the device is applied to a flexible joint robot arm tracking control method based on reinforcement learning, and the device comprises:

[0040] The platform building module is used to build a flexible joint robot platform, establish a coordinate system connecting the joints of the robot arm based on the DH method, and obtain the kinematic equation of the robot system;

[0041] The tracking control design module is used to construct a reinforcement learning controller with sudden actuator failure based on the kinematic equations of the robot system, to track and maintain the flexible joint robot platform while compensating for the uncertainty and unknown interference of the system; the design of the reinforcement learning controller with sudden actuator failure includes: designing an evaluation neural network and designing an execution neural network;

[0042] The simulation experiment module is used to build a simulation platform, conduct simulation experiments based on a flexible joint robot platform and a reinforcement learning controller with sudden actuator failure, and complete the flexible joint robot arm tracking control based on reinforcement learning.

[0043] On the other hand, a flexible joint robot arm tracking control device based on reinforcement learning is provided, and the flexible joint robot arm tracking control device based on reinforcement learning includes: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned flexible joint robot arm tracking control methods based on reinforcement learning is implemented.

[0044] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned reinforcement learning-based flexible joint robot arm tracking control methods.

[0045] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0046] In the embodiment of the present invention, the uncertainty of the flexible joint robot system and the actuator failure are taken into consideration, and a reinforcement learning algorithm is proposed to compensate for the system uncertainty, thereby improving the fault tolerance of the system.

[0047] The present invention introduces non-singular terminal sliding mode control to enhance the robustness of the system, and proves through some necessary simulations and experiments that the proposed control method has a good tracking effect.

[0048] A complete robot system platform is built based on ROS. Through the communication mechanism, each node can work together efficiently to achieve real-time monitoring and control of the robot's joint status. Real-time monitoring of the status information of each joint of the robot. The platform realizes the efficient integration of software and hardware, providing the robot with powerful perception, decision-making and execution capabilities, enabling it to perform tasks in complex environments.

[0049] The present invention keeps the system output in a compact set by defining an appropriate Lyapunov function and ensures the control performance of the system by properly selecting design parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 It is a flow chart of an adaptive sparse signal recovery method for street fighting provided by an embodiment of the present invention;

[0052] Figure 2 A control system design diagram provided for an embodiment of the present invention;

[0053] Figure 3 A diagram showing the connection between the simulation platform and the computer IP provided by the embodiment of the present invention;

[0054] Figure 4 A diagram of the Gazebo simulation interface provided in an embodiment of the present invention;

[0055] Figure 5 A diagram of the enabling interface of a simulation robot provided by an embodiment of the present invention;

[0056] Figure 6 A trajectory tracking curve diagram of joint 1 provided in an embodiment of the present invention;

[0057] Figure 7 The error curve diagram of the joint 1 provided by the embodiment of the present invention;

[0058] Figure 8 A trajectory tracking curve diagram of joint 2 provided in an embodiment of the present invention;

[0059] Fig. 9 The error curve diagram of the joint 2 provided by the embodiment of the present invention;

[0060] Fig.10 An experimental platform diagram provided for an embodiment of the present invention;

[0061] Fig.11 A diagram showing the motion of a robotic arm provided by an embodiment of the present invention;

[0062] Fig.12 An IP diagram of a physical robot connected to a computer provided by an embodiment of the present invention;

[0063] Fig.13 is a diagram of a physical robot enabling interface provided by an embodiment of the present invention;

[0064] Fig.14 is a measured trajectory tracking curve diagram of the joint 1 provided in an embodiment of the present invention;

[0065] Fig.15 is a graph of the measured error of the joint 1 provided in an embodiment of the present invention;

[0066] Fig.16 is a measured trajectory tracking curve diagram of the joint 2 provided in an embodiment of the present invention;

[0067] Fig.17 is a measured error curve of the joint 2 provided in an embodiment of the present invention;

[0068] Fig.18 It is a block diagram of a flexible joint mechanical arm tracking control device based on reinforcement learning provided by an embodiment of the present invention;

[0069] Fig.19 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0071] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0072] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0073] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0074] The embodiment of the present invention provides a flexible joint robot arm tracking control method based on reinforcement learning, which can be implemented by a flexible joint robot arm tracking control device based on reinforcement learning, and the flexible joint robot arm tracking control device based on reinforcement learning can be a terminal or a server. Figure 1 The flowchart of the flexible joint manipulator tracking control method based on reinforcement learning is shown in the figure. The processing flow of the method may include the following steps:

[0075] S1. Build a flexible joint robot platform, establish a coordinate system connecting the joints of the robot arm based on the DH method, and obtain the kinematic equation of the robot system;

[0076] In a feasible implementation, in S1, the kinematic equation of the robot system includes:

[0077]

[0078] in, is the position of the connecting rod, is the speed of the connecting rod, is the acceleration of the connecting rod; is the inertia matrix; is the centrifugal force-Coriolis matrix; M(η)=M0(η)+M Δ (η) is gravity; C0(η), M0(η) is known; C Δ (η), M Δ (η) is unknown but bounded, is the control input, It is an actuator failure.

[0079] In a feasible implementation, the DH method is a method for describing and modeling robot kinematics. The method facilitates describing and calculating the posture and position of the robot movement by establishing a coordinate system connecting the joints of the robot arm, thereby simplifying the derivation of the robot kinematic equations.

[0080] S2. Based on the kinematic equations of the robot system, a reinforcement learning controller with sudden actuator failures is constructed to track and maintain the flexible joint robot platform while compensating for the uncertainty and unknown interference of the system; the design of the reinforcement learning controller with sudden actuator failures includes: designing an evaluation neural network and designing an execution neural network.

[0081] In a feasible implementation, the present invention designs a control algorithm based on Actor-Critic network and non-singular terminal sliding mode, which can compensate for the uncertainty and unknown disturbance of the system while maintaining trajectory tracking. Among them, Actor NN is responsible for generating control strategies that directly interact with the environment and make decisions, while Critic NN evaluates the quality and results of the actions generated by the execution network and provides feedback signals to guide the execution network to adjust its strategy to optimize long-term rewards. NTSMC designs a non-singular sliding surface to ensure that the control input remains bounded throughout the control process, enhancing the robustness of the system.

[0082] In a feasible implementation, in S2, designing an evaluation neural network includes:

[0083] Design Actor-Critic network, including Actor NN and Critic NN.

[0084] Among them, Actor NN is used to generate control strategies that directly interact with the environment and make decisions;

[0085] Critic NN is used to evaluate the quality and results of actions produced by the execution network and provide feedback signals to guide the execution network to adjust its strategy.

[0086] In one possible implementation, the evaluation neural network is often used in an action-evaluation algorithm to evaluate the value of the action selected by the execution network. The radial basis function neural network (RBFNN) is an efficient feedforward neural network, the core of which is a radial basis function (RBF) with local characteristics, and its output is only related to a certain local norm of the input vector. Due to the simplified training process and fast learning convergence, RBFNN can approximate any continuous nonlinear function with any desired accuracy. This ability makes RBFNN particularly effective in dealing with complex nonlinear problems and enhances its application in fault-tolerant control.

[0087] Design the following radial basis function neural network (RBFNN):

[0088] f(z)=w T S(z)

[0089] in, represents the weight, f(z) is a continuous function, and the input is S(z) represents the basis function;

[0090]

[0091] In order to achieve optimal control, the operating cost must be minimized. To achieve this goal, a long-term cost function is formulated to measure and optimize the overall performance of the robot after a series of decisions.

[0092] Define the state quantity y1 = η, The error vector is e1=y1-η d , Specifically define the evaluation neural network function as and where z c =e1,S(z c ) represents the basis function of the critic neural network.

[0093] In a feasible implementation, the long-term cost function includes:

[0094]

[0095] Among them, J represents the discount constant of future costs; φ(t) represents the cost function; and t represents time.

[0096] The instantaneous cost function φ(t) is designed as:

[0097] φ(t)=(η-η d ) T D(η-η d )+τ T Rτ

[0098] Among them, D and R are both positive definite matrices, η d represents the desired trajectory. In order to achieve optimal control, the cost function must be minimized.

[0099] The critique neural network takes the current state as input and outputs a scalar value that represents an estimate of the long-term reward that can be obtained by following the current policy starting from that state. The goal of the critique network is to estimate these values ​​as accurately as possible so that the execution network can use these estimates to improve its policy. The critique network can be trained to improve the accuracy of its value estimates by minimizing the difference between its predicted values ​​and actual rewards.

[0100] In a feasible implementation, the weight update rate of the evaluation neural network is shown in the following formula:

[0101]

[0102] Among them, δ c >0 is the learning rate, Λ is S c (z c ), c represents a subscript, which is the abbreviation of critic; represents the weight update rate of the evaluation neural network; ψ represents a constant greater than 0; S c (z c ) represents the evaluation network basis function; Indicates S c (z c ) and the gradient of z c Product of derivatives.

[0103] In one possible implementation, the main goal of executing the neural network is to maximize the cumulative reward by learning the optimal policy. It uses a neural network to approximate the function, taking the state as input and outputting the corresponding action.

[0104] In a feasible implementation, designing and executing a neural network includes:

[0105] The non-singular terminal sliding mode control algorithm (NTSMC) is constructed according to the following formula:

[0106]

[0107] in, m=diag(m1,m2,…,m n ) is a positive definite matrix, q1 and q2 satisfy A positive odd number of

[0108] According to the kinematic equation of the robot system, the initial input of the robot system is calculated as:

[0109]

[0110] in, is the uncertainty of the system;

[0111] The uncertainty in the system is effectively approximated using an executive neural network according to the following formula:

[0112]

[0113] in, represents the predicted value of the weight, represents the ideal value, and S a (z a ) represents the basis function of executing the neural network, ε a represents the estimation error of the actor neural network; therefore, the control input is:

[0114]

[0115] The weight update rate of the neural network is:

[0116]

[0117] Among them, δ a >0 is the learning rate, is the cost estimation error.

[0118] In a feasible implementation, Figure 2 The control system design diagram of the present invention is shown. Using the Lyapunov pair stability proof, the global Lyapunov function of the system is designed as

[0119]

[0120] After a series of derivative changes of the function, we can finally get The conclusion can prove that the designed closed-loop system is semi-globally consistent and ultimately bounded.

[0121] S3. Build a simulation platform, conduct simulation experiments based on a flexible joint robot platform and a reinforcement learning controller with sudden actuator failure, and complete the flexible joint robotic arm tracking control based on reinforcement learning.

[0122] In a feasible implementation, in order to prove the feasibility and effectiveness of the proposed control method, the method of the present invention uses the simulation model provided by the official to implement the control algorithm experiment on a computer with Ubuntu 16.04 operating system and ROS Noetic 20.04 environment installed to solve two forms of actuator failure: efficiency loss and bias failure. The Baxter robot is simulated through the Gazebo simulation environment, the corresponding parameters of the software are set, the robot model is imported, and the control node is started. The movement performance of the Baxter robot can be observed in the Gazebo environment, and the operation status and related information of the robot can be queried through the command line.

[0123] First, modify the device information in the baxter.sh script file so that the simulation platform is connected to the computer IP, such as Figure 3Then, execute the environment configuration script in the command line interface to start the simulation environment of the Baxter robot. Through the launch file, you can start Gazebo and load the simulation model of the robot, such as Figure 4 Finally, in a new terminal window, execute the Baxter simulation command and the robot enable command. After the robot is enabled, run the Python code you wrote to perform the simulation, as shown in the following figure. Figure 5 shown.

[0124] Data settings:

[0125] The basic parameters are shown in Table 1. The expected trajectory tracking is Actuator faults are:

[0126]

[0127] 1) PD control:

[0128] The design scheme of PD controller is:

[0129]

[0130] Among them, K p is the proportional gain, K d is the differential gain, and K p =0.88, K d =0.15.

[0131] Table 1 Basic parameters

[0132]

[0133] 2) Neural network control:

[0134] The design of the neural network controller is

[0135]

[0136] Where w represents the weight of the neural network and S(z) represents the radial basis function. C = [η1, η2] T , the number of nodes in the neural network is 256, the center parameter is ±1, the center variance is σ=2.05, and the initial weight is

[0137] 3) Reinforcement learning control:

[0138] The design of the Reinforcement Learning and Nonsingular Terminal Sliding Mode Control (RL-NTSMC) controller is:

[0139]

[0140] Where C0 = [η1, η2] T , m = 550, θ = 0.001, q = q1 / q2 = 5 / 3. The execution and evaluation neural networks each have 256 nodes, the center parameter is ±1, and the center variance is σ c = 0.5 and σ a =2.1, the initial weight is Robotic arm trajectory tracking simulation experiment:

[0141] Figure 6 and Figure 7 They represent the trajectory tracking curve and error curve of joint 1 respectively. Figure 8 and Fig. 9 It represents the trajectory tracking curve and error curve of joint 2. It can be seen from the figure that for joint 1, when the actuator has an efficiency loss failure (t=5s), the maximum tracking error of the RL-NTSMC control method is 0.0305rad; when the actuator has a deviation failure (t=9s), the maximum tracking error of the RL-NTSMC control method is 0.0057rad. For joint 2, the maximum tracking error of the RL-NTSMC control method is 0.0317rad; when the actuator has a bias failure, it is 0.0071rad for the RL-NTSMC control method. The maximum tracking errors of the NN and PD control methods are larger than those of the RL-NTSMC control method. It can be concluded that the algorithm of the present invention has good tracking performance and anti-interference ability.

[0142] Build the experimental platform:

[0143] In order to further demonstrate the effectiveness and superiority of the proposed technology, the present invention is experimentally verified on the Baxter robot. Each arm of Baxter has 7 degrees of freedom and is equipped with a series of joint actuators, which can operate safely around humans without setting up safety fences. Its joint assembly is composed of a series of elastic brakes, which can monitor and feedback external forces through springs between the engine or transmission device and the actuator. The Baxter robot provides a flexible, safe, and user-friendly operating environment through the Linux-based open source ROS platform. The joints of the Baxter robot are controlled through ROS nodes to achieve control of the robotic arm, making task programming and operation simple. The Baxter robot can adapt to changing working environments and safely cooperate with humans to complete various tasks. The experimental platform is as follows: Fig.10 As shown in Figure 2, the motion of the robot arm is as follows: Fig.11 shown.

[0144] First, connect the robot to the robot host computer via USB or Ethernet, and start the robot driver to establish a connection between the host computer and the robot body. After the host computer system successfully connects to the robot, modify the device information in the baxter.sh script file to connect the robot to the computer IP, such as Fig.12 Then, open the terminal, run the environment configuration script, and enable Baxter, as shown in the following example: Fig.13 After successful enabling, you can run the code. Fig.14 and Fig.15 They represent the trajectory tracking curve and error curve of joint 1 respectively. Fig.16 and Fig.17 Represents the trajectory tracking curve and error curve of joint 2.

[0145] As can be seen from the above figure, for joint 1, when the actuator has an efficiency loss failure, the maximum tracking error of the RL-NTSMC control strategy is 0.0343 rad, and the steady-state error is 0.0006 rad; when the actuator has a deviation failure, the maximum tracking error of the RL-NTSMC control method is 0.00621 rad, and the steady-state error is 0.0014 rad. For joint 2, when the actuator has an efficiency loss failure, the maximum tracking error of the RL-NTSMC control strategy is 0.0347 rad, and the steady-state error is 0.0002 rad; when the actuator has a deviation failure, the maximum tracking error of the RL-NTSMC control strategy is 0.0010 rad, and the steady-state error is 0.0003 rad. The maximum tracking errors of the NN and PD control methods are larger than those of the RL-NTSMC control method. It can be seen that the invented algorithm has good tracking performance and anti-interference ability.

[0146] In the embodiment of the present invention, the uncertainty of the flexible joint robot system and the actuator failure are taken into consideration, and a reinforcement learning algorithm is proposed to compensate for the system uncertainty, thereby improving the fault tolerance of the system.

[0147] The present invention introduces non-singular terminal sliding mode control to enhance the robustness of the system, and proves through some necessary simulations and experiments that the proposed control method has a good tracking effect.

[0148] A complete robot system platform is built based on ROS. Through the communication mechanism, each node can work together efficiently to achieve real-time monitoring and control of the robot's joint status. Real-time monitoring of the status information of each joint of the robot. The platform realizes the efficient integration of software and hardware, providing the robot with powerful perception, decision-making and execution capabilities, enabling it to perform tasks in complex environments.

[0149] The present invention keeps the system output in a compact set by defining an appropriate Lyapunov function and ensures the control performance of the system by properly selecting design parameters.

[0150] Fig.18 is a block diagram of a flexible joint robot arm tracking control device 300 based on reinforcement learning according to an exemplary embodiment. The device 300 is used in a flexible joint robot arm tracking control method based on reinforcement learning. Fig.18 The device includes a platform building module 310, a tracking control design module 320 and a simulation experiment module 330. Among them:

[0151] The platform building module 310 is used to build a flexible joint robot platform, establish a coordinate system connecting the joints of the robot arm based on the DH method, and obtain the kinematic equation of the robot system;

[0152] The tracking control design module 320 is used to construct a reinforcement learning controller with sudden actuator failure based on the kinematic equations of the robot system, and to track and maintain the flexible joint robot platform while compensating for the uncertainty and unknown interference of the system; wherein the design of the reinforcement learning controller with sudden actuator failure includes: designing an evaluation neural network and designing an execution neural network;

[0153] The simulation experiment module 330 is used to build a simulation platform, conduct simulation experiments based on a flexible joint robot platform and a reinforcement learning controller with sudden actuator failure, and complete the flexible joint robot arm tracking control based on reinforcement learning.

[0154] Optionally, the kinematic equations of the robot system include:

[0155]

[0156] in, is the position of the connecting rod, is the speed of the connecting rod, is the acceleration of the connecting rod; is the inertia matrix; is the centrifugal force-Coriolis matrix; M(η)=M0(η)+M Δ (η) is gravity; C0(η), M0(η) is known; C Δ (η), M Δ (η) is unknown but bounded, is the control input, It is an actuator failure.

[0157] Optionally, design and evaluate a neural network, including:

[0158] Design Actor-Critic network, including Actor NN and Critic NN.

[0159] Among them, Actor NN is used to generate control strategies that directly interact with the environment and make decisions;

[0160] Critic NN is used to evaluate the quality and results of actions produced by the execution network and provide feedback signals to guide the execution network to adjust its strategy.

[0161] Optionally, based on the Actor-Critic network including:

[0162] Design the following radial basis function neural network (RBFNN):

[0163] in, represents the weight, f(z) is a continuous function, and the input is S(z) represents the basis function:

[0164]

[0165] A long-term cost function is formulated to measure and optimize the overall performance of the robot after making decisions.

[0166] Optionally, a long-term cost function, comprising:

[0167]

[0168] Among them, J represents the discount constant of future costs; φ(t) represents the cost function; and t represents time.

[0169] Optionally, the weight update rate of the evaluation neural network is shown in the following formula:

[0170]

[0171] Among them, δ c >0 is the learning rate, Λ is S c (z c ), c represents a subscript, which is the abbreviation of critic; represents the weight update rate of the evaluation neural network; ψ represents a constant greater than 0; S c (z c ) represents the evaluation network basis function; Indicates S c (z c ) and the gradient of z c Product of derivatives;

[0172] After the evaluation neural network takes the current state as input, it will output a scalar value representing the estimate of the long-term reward that can be obtained by following the current policy starting from this state.

[0173] Optionally, design and implement a neural network, including:

[0174] The non-singular terminal sliding mode control algorithm NTSMC is constructed according to the following formula:

[0175]

[0176] in, m=diag(m1,m2,…,m n ) is a positive definite matrix, q1 and q2 satisfy A positive odd number of

[0177] Calculate the initial input of the robot system according to the kinematic equations of the robot system;

[0178] The uncertainty in the system is approximated by the following formula:

[0179]

[0180] in, represents the predicted value of the weight, represents the ideal value, and S a (z a ) represents the basis function of executing the neural network, ε a represents the estimation error of the actor neural network;

[0181] The weight update rate of the neural network is:

[0182]

[0183] Among them, δ a >0 is the learning rate, is the cost estimation error.

[0184] In the embodiment of the present invention, a trajectory tracking algorithm based on actor-critic reinforcement learning and non-singular synovial control is proposed for a flexible joint robot. The effectiveness of the proposed control method is verified through simulation on the Gazebo platform and experiments on the Baxter robot. At the same time, the semi-global asymptotic uniform boundedness of the closed-loop system is proved by Lyapunov stability analysis. The physical experimental results show that:

[0185] In simulation and experiments, the control algorithm proposed by the present invention can effectively achieve trajectory tracking and anti-interference capabilities;

[0186] Compared with the PD and NN control algorithms, the trajectory tracking effect of the algorithm proposed in the present invention is better.

[0187] Fig.19 is a structural diagram of a flexible joint robot arm tracking control device based on reinforcement learning provided by an embodiment of the present invention, such as Fig.19 As shown, the flexible joint robot arm tracking control device based on reinforcement learning may include the above Fig.18 The flexible joint robot arm tracking control device based on reinforcement learning is shown. Optionally, the flexible joint robot arm tracking control device 410 based on reinforcement learning may include a first processor 2001 .

[0188] Optionally, the reinforcement learning-based flexible joint robotic arm tracking control device 410 may further include a memory 2002 and a transceiver 2003 .

[0189] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0190] Combine the following Fig.19 The components of the flexible joint robot arm tracking control device 410 based on reinforcement learning are specifically introduced as follows:

[0191] The first processor 2001 is the control center of the flexible joint manipulator tracking control device 410 based on reinforcement learning, which can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0192] Optionally, the first processor 2001 can perform various functions of the flexible joint robot arm tracking control device 410 based on reinforcement learning by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0193] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Fig.19CPU0 and CPU1 are shown in FIG.

[0194] In a specific implementation, as an embodiment, the flexible joint robot arm tracking control device 410 based on reinforcement learning may also include multiple processors, such as Fig.19 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0195] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0196] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Fig.19 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0197] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0198] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Fig.19 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0199] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and track the interface circuit ( Fig.19 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0200] It should be noted that Fig.19 The structure of the reinforcement learning-based flexible joint robot arm tracking control device 410 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0201] In addition, the technical effects of the flexible joint robotic arm tracking control device 410 based on reinforcement learning can refer to the technical effects of the flexible joint robotic arm tracking control method based on reinforcement learning described in the above method embodiment, and will not be repeated here.

[0202] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0203] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0204] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable sensors. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0205] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0206] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0207] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0208] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0209] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0210] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention.

[0211] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A flexible joint robot arm tracking control method based on reinforcement learning, characterized in that: The method comprises: S1. Build a flexible joint robot platform, establish a coordinate system connecting the joints of the robot arm based on the DH method, and obtain the kinematic equation of the robot system; S2. Based on the kinematic equations of the robot system, a reinforcement learning controller with sudden actuator failure is constructed to track and maintain the flexible joint robot platform while compensating for the uncertainty and unknown interference of the system; wherein the design of the reinforcement learning controller with sudden actuator failure includes: designing an evaluation neural network and designing an execution neural network; S3. Build a simulation platform, conduct simulation experiments based on the flexible joint robot platform and the reinforcement learning controller with sudden actuator failure, and complete the flexible joint robot arm tracking control based on reinforcement learning.

2. The flexible joint robot arm tracking control method based on reinforcement learning according to claim 1 is characterized in that: In S1, the kinematic equations of the robot system include: in, is the position of the connecting rod, is the speed of the connecting rod, is the acceleration of the connecting rod; is the inertia matrix; is the centrifugal force-Coriolis matrix; M(η)=M0(η)+M Δ (η) is gravity; C0(η), , M0(η) is known; C Δ (η), 、M Δ (η) is unknown but bounded, is the control input, It is an actuator failure.

3. The flexible joint robot arm tracking control method based on reinforcement learning according to claim 2 is characterized in that: In S2, a neural network is designed and evaluated, including: Design Actor-Critic network, including Actor NN and Critic NN. Wherein, the Actor NN is used to generate control strategies that directly interact with the environment and make decisions; The Critic NN is used to evaluate the quality and results of actions produced by the execution network and provide feedback signals to guide the execution network to adjust its strategy.

4. The flexible joint robot arm tracking control method based on reinforcement learning according to claim 3 is characterized in that: The Actor-Critic network consists of: Design the radial basis function neural network RBFNN as follows: f(z)=w T S(z) in, represents the weight, f(z) is a continuous function, and the input is S(z) represents the basis function: A long-term cost function is formulated to measure and optimize the overall performance of the robot after making decisions.

5. The flexible joint robot arm tracking control method based on reinforcement learning according to claim 4 is characterized in that: The long-term cost function includes: Among them, J represents the discount constant of future costs; φ(t) represents the cost function; and t represents time.

6. The flexible joint robot arm tracking control method based on reinforcement learning according to claim 5 is characterized in that: The weight update rate of the evaluation neural network is shown in the following formula: Among them, δ c >0 is the learning rate, Λ is S c (z c ), c represents a subscript, which is the abbreviation of critic; represents the weight update rate of the evaluation neural network; ψ represents a constant greater than 0; S c (z c ) represents the evaluation network basis function; Indicates S c (z c ) and the gradient of z c Product of derivatives; After the evaluation neural network takes the current state as input, it will output a scalar value representing the estimate of the long-term reward that can be obtained by following the current policy starting from this state.

7. The flexible joint robot arm tracking control method based on reinforcement learning according to claim 6 is characterized in that: The design implements a neural network, including: The non-singular terminal sliding mode control algorithm (NTSMC) is constructed according to the following formula: in, m=diag(m1,m2,…,m n ) is a positive definite matrix, q1 and q2 satisfy A positive odd number of represents the derivative of the error e; Calculate the initial input of the robot system according to the kinematic equations of the robot system; The uncertainty in the system is approximated by the following formula: in, represents the predicted value of the weight, represents the ideal value, and S a (z a ) represents the basis function of executing the neural network, ε a represents the estimation error of the actor neural network; a represents the subscript, which is the abbreviation of actor; η represents the position of the connecting rod; The weight update rate of the neural network is: Among them, δ a >0 is the learning rate, is the cost estimation error.

8. A flexible joint robot arm tracking control device based on reinforcement learning, wherein the flexible joint robot arm tracking control device based on reinforcement learning is used to implement the flexible joint robot arm tracking control method based on reinforcement learning as claimed in any one of claims 1 to 7, characterized in that: The device comprises: The platform building module is used to build a flexible joint robot platform, establish a coordinate system connecting the joints of the robot arm based on the DH method, and obtain the kinematic equation of the robot system; A tracking control design module is used to construct a reinforcement learning controller with sudden actuator failure based on the kinematic equations of the robot system, to track and maintain the flexible joint robot platform while compensating for the uncertainty and unknown interference of the system; wherein the design of the reinforcement learning controller with sudden actuator failure includes: designing an evaluation neural network and designing an execution neural network; The simulation experiment module is used to build a simulation platform, conduct simulation experiments based on the flexible joint robot platform and the reinforcement learning controller with sudden actuator failure, and complete the flexible joint robot arm tracking control based on reinforcement learning.

9. A flexible joint robot arm tracking control device based on reinforcement learning, characterized in that: The flexible joint mechanical arm tracking control device based on reinforcement learning includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code can be called by a processor to execute the reinforcement learning-based flexible joint robot arm tracking control method as described in any one of claims 1 to 7.

Images