A non-boosting optimization iterative learning control method for robots

Abstract

The present invention discloses a non-lifting optimization iterative learning control method for a robot, which relates to the field of robot optimization control. The method comprises: converting a dynamic equation of a robot impedance control system into a discrete state space equation; describing a control process of the control system as a Markov decision process, and designing non-lifting optimization ILC methods for situations with and without system model parameter information, respectively, thereby reducing computational complexity compared to an optimization ILC method using a lifting technique; for situations with specific system model parameter information, solving a corresponding Bellman equation based on an approximate dynamic programming method, thereby obtaining a non-lifting optimization ILC method for situations with system model parameter information; and for situations without system model parameter information, optimizing a specified performance indicator using a least squares method based on a Q-learning method using measurable data, thereby obtaining a non-lifting optimization ILC method for situations without system model parameter information.

Description

Technical Field

[0001] The present invention relates to the field of robot optimization control, and in particular to a robot non-lifting optimization iterative learning control method. Background Art

[0002] Robotic control technology integrates mechanics, electronics, control, and computer science, and has a wide range of applications. Impedance control is a key solution to the high dynamics, nonlinearity, and time-varying nature of complex tasks like cutting and polishing, where the robot's end effector interacts with the external environment. Impedance control enables the robot to proactively adjust its impedance characteristics, allowing it to adapt flexibly to unknown environments.

[0003] For robotic systems performing repetitive motion tasks, iterative learning control, with its simple structure and high performance, can provide an excellent tracking control solution. It uses information from the system's operation to continuously correct the control inputs of the current batch, thereby achieving complete tracking of the desired trajectory within a limited time. Traditional iterative learning control suffers from the difficulty of selecting control gains, and optimized iterative learning control offers a solution to this problem. However, optimized iterative learning control relies heavily on accurate system model parameters. Furthermore, optimized iterative learning control using boosting techniques suffers from high computational complexity and requires excessive experimental batches to estimate system model information.

[0004] In response to the above-mentioned problems of optimized iterative learning control, non-boosting optimized iterative learning control methods with and without system model parameter information have emerged. Summary of the Invention

[0005] In response to the above-mentioned problems and technical needs, the inventors have proposed a robot non-boosting optimization iterative learning control method, which is applicable to trajectory tracking control in situations with and without system model parameter information. The technical solution of the present invention is as follows:

[0006] A robot non-boosting optimization iterative learning control method comprises the following steps:

[0007] The first step is to establish a dynamic model of the robot impedance control system, including:

[0008] The end of the robot impedance control system is covered with soft materials and is in contact with the external environment. The actual physical model of the robot impedance control system is constructed as a dynamic model, which is expressed as:

[0009]

[0010] Where M represents the mass of the robot, Δx rrepresents the motion displacement of the robot end, u f represents the input signal, c represents the impedance coefficient, f s (·) represents a smooth nonlinear function, f s (Δx r ) represents the motion displacement Δx r The force brought about

[0011] The second step is to construct the discrete state space equation of the robot impedance control system, including:

[0012] The motion displacement and velocity of the robot end are defined as state variables: Define the input variable as input signal u=u f , the output variable is the movement speed of the robot end Then the system dynamic model shown in formula (1) is described as:

[0013]

[0014] For the continuous nonlinear system model shown in Equation (2), the Jacobian linearization method is first used to find the equilibrium point Δx r =0[m] to obtain the linearized system model, and then discretize the linearized system model, and select the sampling period T that satisfies Shannon sampling theorem. s , the discrete state space equation of the robot impedance control system is obtained:

[0015]

[0016] Where t represents the sampling time, k represents the iteration batch, T represents the operation cycle of the batch process, and for each repeated operation cycle t∈[0, T], N sampling points are taken; and They represent the input, output and state vector of the robot impedance control system at the sampling time of the kth iteration batch t respectively; A, B and C are the parameter matrices of the discrete system and satisfy the CB full rank; it is assumed that the initial state of each iteration batch of the system remains consistent, that is, x k (0)=0;

[0017] Define the expected reference trajectory as y d (t), then the tracking error e k (t) is defined as:

[0018] e k (t) = y d (t)-y k (t) (4)

[0019] The third step is to transform the non-boosting optimization iterative learning control process of the discrete system into a Markov decision process, including:

[0020] For discrete system (3), different from traditional optimization iterative learning control, it is not converted into a time series lifting model. Define the five-tuple in:

[0021] Represents the state space, and defines the state as the tracking error e k (t+1); Represents the action space, defining the action as the input change Δu k+1 (t), that is:

[0022] Δu k+1 (t) = u k+1 (t)-u k (t) (5)

[0023] f represents the state transfer function, which is defined as:

[0024]

[0025] Where Δx k+1 (t) is the state change, that is:

[0026] Δx k+1 (t) = x k+1 (t)-x k (t) (7)

[0027] represents the profit function, which is defined as:

[0028]

[0029] Among them, the profit function It consists of two parts, namely the tracking error and the change of input signal between batches, and their priorities are represented by symmetric positive definite weight matrices Q and R, respectively, that is, Q = Q T >0 and R=R T >0; the tracking error and the induced norm of the input signal variation between batches are defined as:

[0030]

[0031] γ represents the discount factor and exists in the range γ∈(0,1], the size of γ determines the present value of future returns;

[0032] The state value function of the current batch is defined as:

[0033]

[0034] Step 4: Design a non-boosting optimization iterative learning control trajectory tracking algorithm when the system model parameter information is known, including:

[0035] The above-mentioned transformed Markov decision process is a linear quadratic adjustment problem along the iteration axis. The state value is quadratic, so Equation (10) can also be expressed as:

[0036]

[0037] Where P is a symmetric positive definite matrix;

[0038] At the same time, based on approximate dynamic programming, according to equations (8) and (9), equation (10) can also be expressed as:

[0039]

[0040] Substituting formula (11) into formula (12), we get:

[0041]

[0042] According to formula (13), Δu k+1 (t) Find the quadratic optimal solution and obtain the iterative learning update law:

[0043] u k+1 (t) = u k (t)+L e e k (t+1)+L x Δx k+1 (t) (14)

[0044] Among them, L e is the error term learning gain, L x are the learning gains of the state change terms between batches, which are:

[0045] L e =(γB T C T PCB+R) -1 γB T C T P (15)

[0046] L x =-(γB T C T PCB+R) -1 γB T C T PCA (16)

[0047] Substituting formula (11) into formula (13), we get:

[0048]

[0049] Formula (17) can be further expressed as:

[0050]

[0051] in,

[0052] D1=I-CBL e , D2=-CA-CBL x (19)

[0053] The Bellman equation for solving P in equations (15) and (16) is obtained from equation (18):

[0054] P=Q+L e T RL e +γ(I-CBL e ) T P(I-CBL e ) (20)

[0055] When the initial input u0(t) and initial state x0(t) of the robot impedance control system are given, the expected reference trajectory y d (t), symmetric positive definite weight matrices Q and R are selected, and a non-boosting optimization iterative learning control trajectory tracking algorithm is designed when the system model parameter information is known as follows:

[0056] Step 4.1: Perform u on the discrete system (3) k (t) to obtain x k (t+1) and y k (t+1), thus obtaining e k (t+1) and Δx k+1 (t);

[0057] Step 4.2: Strategy evaluation, get the updated P according to formula (20);

[0058] Step 4.3: Policy iteration, according to formula (15) and formula (16) to obtain the updated L e and L x ;

[0059] Step 4.4: Update the next batch of input u according to the iterative learning update rule (14) k+1 (t);

[0060] Step 4.5: Repeat step 4.1 until the number of sampling times reaches N;

[0061] Step 4.6: Repeat step 4.1 until the tracking error accuracy is less than the set value, then the algorithm ends;

[0062] Step 5: Design a non-boosting optimization iterative learning control trajectory tracking algorithm when model parameter information is unknown, including:

[0063] Based on the Q-learning algorithm, a non-boosting optimization iterative learning control trajectory tracking algorithm is designed when the model parameter information is unknown. The Q function is a state-action value function, defined as:

[0064]

[0065] The state value function is the policy Δu k+1 State e under (t) k (t+1) value function, so it has the same value as the Q function, that is:

[0066] V(e k (t+1))=Q(e k (t+1),Δu k+1 (t)) (22)

[0067] For the optimal iterative learning control problem, the Q function is written in the following quadratic form:

[0068]

[0069] in,

[0070]

[0071] The system parameter matrix is partitioned and defined as:

[0072]

[0073] and,

[0074] F ee =Q+γP,F eu =-γPCB,F ex =-γPCA (26)

[0075] F ue =-γB T C T P, F uu =R+γB T C T PCB, F ux =γB T C T PCA (27)

[0076] F xe =-γA T C T P, F xu =γAT C T PCB, F xx =γA T C T PCA (28)

[0077] According to formula (23), Δu k+1 (t) Find the quadratic optimal solution and obtain the iterative learning update law:

[0078] u k+1 (t) = u k (t)+L q τ k+1 (t+1) (29)

[0079] in,

[0080]

[0081]

[0082] To solve the continuous state space and action space problem, the experimental data is used to estimate the system parameter matrix Parameterize equation (23) as follows:

[0083]

[0084] in,

[0085]

[0086]

[0087]

[0088]

[0089] Regression vector Defined as Z k+1 The Kronecker product of itself, that is:

[0090]

[0091] Among them, z i for The i-th element of , and p=(n+m+l) 2 ;

[0092] According to formula (32), formula (21) can be expressed as:

[0093]

[0094] Since the unknown system parameter matrix is a symmetric matrix, the unknown system parameter vector It stores a total of p(p+1)2 independent elements, so L≥p(p+1)2 data samples are required to form a data matrix, and then use the least squares method to estimate the system parameter vector;

[0095] Define the data matrix as:

[0096]

[0097]

[0098] Then the unknown system parameter vector The least squares solution of is calculated as:

[0099]

[0100] When the initial input u0(t) and initial state x0(t) of the robot impedance control system are given, the expected reference trajectory y d (t), symmetric positive definite weight matrices Q and R are selected, and a non-boosting optimization iterative learning control trajectory tracking algorithm is designed when the system model parameter information is unknown as follows:

[0101] Step 5.1: Perform u on the discrete system (3) k (t) to collect τ k+1 (t+1), Z k+1 (t+1) and data;

[0102] Step 5.2: Update the next batch of input u according to the iterative learning update rule (29) k+1 (t);

[0103] Step 5.3: If L ≥ p(p+1) / 2, proceed to step 5.4; otherwise, proceed to step 5.6.

[0104] Step 5.4: Strategy evaluation, according to formula (38) and formula (41), we can get and will Restore to

[0105] Step 5.5: Policy iteration, according to formula (31), the updated and Thus forming L q ;

[0106] Step 5.6: Repeat step 5.1 until the number of sampling times reaches N;

[0107] Step 5.7: Repeat step 5.1 until the tracking error accuracy is less than the set value, and the algorithm ends.

[0108] The beneficial technical effects of the present invention are:

[0109] This application discloses a method for controlling a robot impedance control system as a controlled object for a system with repetitive motion characteristics and linearization, and describes the iterative learning control process of the robot impedance control system as a Markov decision process. For the case where there are system model parameters, the Bellman equation is solved based on the approximate dynamic programming method, thereby obtaining a non-boosting optimization iterative learning control tracking algorithm for the case where there is system model parameter information; for the case where there are no system model parameters, based on the Q-learning method, the optimization of the specified performance index is achieved by using the least squares method with measurable data, and a non-boosting optimization iterative learning control algorithm for the case where there is no system model parameter information is proposed. The advantage of the proposed method is that it does not require all the model information to be known, which reduces the computational complexity caused by the boosting technology and the number of experimental batches required for the model-free method, while having good convergence speed and error accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Figure 1 This is the actual physical model diagram of the robot impedance control system provided by this application.

[0111] Figure 2 This is a principle block diagram of the robot non-boosting optimization iterative learning control method provided in this application.

[0112] Figure 3 This is a 2-norm convergence diagram of the tracking error of the robot impedance control system provided by this application in the case of system model parameters.

[0113] Figure 4 This is a graph showing the expected output and actual output of the robot impedance control system provided by this application in the presence of system model parameters.

[0114] Figure 5 This is a 2-norm convergence diagram of the tracking error of the robot impedance control system provided by this application in the absence of system model parameters.

[0115] Figure 6 This is a graph showing the expected output and actual output of the robot impedance control system provided by this application in the absence of system model parameters.

[0116] Figure 7 This is a comparison diagram of the 2-norm convergence of the tracking error of the robot impedance control system provided by this application in the case without system model parameters and with system model parameters. DETAILED DESCRIPTION

[0117] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0118] Please refer to Figure 1 , which shows the actual physical model diagram of the robot impedance control system provided by this application. The end of the robot impedance control system is covered with soft material and needs to contact the external environment for operation. The input signal of the robot impedance control system with a given mass of M is u f , then the end of the robot produces a motion displacement Δx r , and the motion displacement Δx r Bring f s (Δx r ) of the force.

[0119] Figure 2 The principle block diagram of the robot non-lifting optimization iterative learning control method provided by the present application is shown. If the system model parameter information is known, the optimization iterative learning control algorithm with the system model parameters is selected, and the system model parameter information A, B and C are directly transferred to the non-lifting optimization iterative learning controller; if the system model parameter information is unknown, the iterative learning control algorithm without the system model parameters is selected, and the estimated system model parameter information is transferred to the non-lifting optimization iterative learning controller using the least squares estimation module based on the measured data Φ and Υ. Passed to the non-boosting optimized iterative learning controller.

[0120] The specific control process includes: at the kth batch and the tth moment, the input signal generated by the non-boosting optimization iterative learning controller is u k (t), which is applied to the robot impedance control system to obtain the output signal y k (t+1). k (t+1) and y in the expected trajectory memory d (t+1) and compare to get the tracking error e k (t+1). Next, e k (t+1) is compared with the tracking error accuracy setting value, such as e k If the error accuracy setting value is not reached at (t+1), the next iteration will be continued. k (t), tracking error e k (t+1), state information x k (t+1) and model information A, B, C are passed to the non-boosting optimization iterative learning controller to generate the next batch of input signals u k+1 (t), and the cycle continues until the tracking error accuracy is less than the tracking error accuracy setting value, then the algorithm stops. At this time, the input of the non-boosting optimization iterative learning controller is the optimal input signal.

[0121] In this embodiment, for Figure 1 The actual physical model of the robot impedance control system shown in formula (2) is set as follows:

[0122] M=0.1kg,c=0.3kg,f s (Δx r )=arctan(0.2Δx r ).

[0123] The operating cycle T of the control system is set to 1s, and the sampling time T s If it is set to 0.02s, the parameter matrices of the discrete state space model of the control system are:

[0124]

[0125] During the operation of the robot impedance control system, it is necessary to set a desired output. The desired reference trajectory set in this implementation is:

[0126] y d =sin(4πt / 5).

[0127] Select the symmetric positive definite weight matrix Q = I, R = 0.1I, and the initial input u0(t) = 0. For the case with system model parameters, select the discount factor γ = 0.99; for the case without system model parameters, select the discount factor γ = 0.8. The non-boosting optimization iterative learning controller designed in this application is implemented based on the STM32F103RCT6 chip. The input of the chip is the input signal u of the robot impedance control system, which is collected by the force sensor. The input signal u at the tth moment of the kth batch k (t) enters the STM32F103RCT6 chip through the conditioning circuit for storage and calculation, and constructs the non-boosting optimization iterative learning control law. The next batch of input signals obtained after CPU calculation is u k+1 (t),u k+1 (t) is then applied to the robot impedance control system via the RS232 communication module, and the system output tracking trajectory is continuously corrected until the key tracking time point is tracked. When the dynamic model of the robot impedance control system (2) is running, for the case of system model parameters, please refer to Figure 3 and Figure 4 , which respectively shows the tracking error 2-norm convergence diagram of the robot impedance control system in the case of system model parameters and the expected output and actual output curve of the robot impedance control system in the case of system model parameters. After running a certain number of iteration batches k, the tracking error gradually converges, and the system output can gradually track the expected target trajectory. For the case without system model parameters, please refer to Figure 5 and Figure 6, which respectively shows the tracking error 2-norm convergence diagram of the robot impedance control system in the case of no system model parameters and the expected output and actual output curve of the robot impedance control system in the case of no system model parameters. After running a certain number of iteration batches k, the tracking error gradually converges, and the system output can gradually track the expected target trajectory; Figure 7 A comparison chart of the 2-norm convergence of the tracking error of the robot impedance control system in the case without system model parameters and with system model parameters is shown. After a certain number of iteration batches, the control accuracy in the case without system model parameters is better than that in the case with system model parameters.

[0128] In summary, this application proposes a non-boosting optimized iterative learning control method for robots, describing the control process of the robot's impedance control system as a Markov decision process. Corresponding non-boosting optimized iterative learning control methods are proposed for both the case with and without system model parameters. This method does not require full knowledge of the model information, reducing the computational complexity associated with boosting techniques and the number of experimental batches required for model-free methods, while also achieving good convergence speed and error accuracy.

[0129] The above description is only a preferred embodiment of the present application, and the present invention is not limited to the above embodiment. It is understood that other improvements and variations directly derived or imagined by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included in the scope of protection of the present invention.

Claims

1. A robot non-boosting optimization iterative learning control method, characterized in that: The method comprises: The first step is to establish a dynamic model of the robot impedance control system, including: The end of the robot impedance control system is covered with soft material and is in contact with the external environment. The actual physical model of the robot impedance control system is constructed as a dynamic model, which is expressed as: Where M represents the mass of the robot, Δx r represents the motion displacement of the robot end, u f represents the input signal, c represents the impedance coefficient, f s (·) represents a smooth nonlinear function, f s (Δx r ) represents the motion displacement Δx r The force brought about The second step is to construct the discrete state space equation of the robot impedance control system, including: The motion displacement and motion speed of the robot end are defined as state variables: Define the input variable as input signal u=u f , the output variable is the movement speed of the robot end Then the system dynamic model shown in formula (1) is described as: For the continuous nonlinear system model shown in Equation (2), the Jacobian linearization method is first used to find the equilibrium point Δx r =0[m] to obtain the linearized system model, and then discretize the linearized system model, and select the sampling period T that satisfies Shannon's sampling theorem. s , the discrete state space equation of the robot impedance control system is obtained: Where t represents the sampling time, k represents the iteration batch, T represents the operation cycle of the batch process, and for each repeated operation cycle t∈[0, T], N sampling points are taken; and They represent the input, output and state vector of the robot impedance control system at the sampling time of the kth iteration batch t respectively; A, B and C are the parameter matrices of the discrete system and satisfy the CB full rank; it is assumed that the initial state of each iteration batch of the system remains consistent, that is, x k (0)=0; Define the expected reference trajectory as y d (t), then the tracking error e k (t) is defined as: e k (t)=y d (t)-y k (t) (4) The third step is to transform the non-boosting optimization iterative learning control process of the discrete system into a Markov decision process, including: Defining a quintuple in: Represents the state space, and defines the state as the tracking error e k (t+1); Represents the action space, defining the action as the input change Δu k+1 (t), that is: Δu k+1 (t)=u k+1 (t)-u k (t) (5) f represents the state transfer function, which is defined as: e k+1 (t+1)=e k (t+1)-CAΔx k+1 (t)-CBΔu k+1 (t) (6) Where Δx k+1 (t) is the state change, that is: Δx k+1 (t)=x k+1 (t)-x k (t) (7) represents the profit function, which is defined as: Among them, the profit function It consists of two parts, namely the tracking error and the change of input signal between batches, and their priorities are represented by symmetric positive definite weight matrices Q and R, respectively, that is, Q = Q T >0 and R=R T >0; the tracking error and the induced norm of the input signal change between batches are defined as: γ represents the discount factor and exists in the range γ∈(0,1], the size of γ determines the present value of future returns; The state value function of the current batch is defined as: Step 4: Design a non-boosting optimization iterative learning control trajectory tracking algorithm when the system model parameter information is known, including: The above-mentioned transformed Markov decision process is a linear quadratic adjustment problem along the iteration axis. The state value is quadratic, so Equation (10) can also be expressed as: Where P is a symmetric positive definite matrix; At the same time, based on approximate dynamic programming, according to equations (8) and (9), equation (10) can also be expressed as: Substituting formula (11) into formula (12), we get: According to formula (13), Δu k+1 (t) Find the quadratic optimal solution and obtain the iterative learning update law: u k+1 (t)=u k (t)+L e e k (t+1)+L x Δx k+1 (t) (14) Among them, L e is the error term learning gain, L x are the learning gains of the state change terms between batches, which are: L e =(γB T C T PCB+R) -1 γB T C T P (15) L x =-(γB T C T PCB+R) -1 γB T C T PCA (16) Substituting formula (11) into formula (13), we get: Formula (17) can be further expressed as: in, D1=I-CBL e ,D2=-CA-CBL x (19) The Bellman equation for solving P in equations (15) and (16) is obtained from equation (18): P=Q+L e T RL e +γ(I-CBL e ) T P(I-CBL e ) (20) When the initial input u0(t), initial state x0(t) of the robot impedance control system is given, the expected reference trajectory y d (t), the symmetric positive definite weight matrices Q and R are selected, and a non-boosting optimization iterative learning control trajectory tracking algorithm is designed when the system model parameter information is known as follows: Step 4.1: Perform u on the discrete system (3) k (t) to obtain x k (t+1) and y k (t+1), thus obtaining e k (t+1) and Δx k+1 (t); Step 4.2: Strategy evaluation, get the updated P according to formula (20); Step 4.3: Policy iteration, according to formula (15) and formula (16) to obtain the updated L e and L x ; Step 4.4: Update the next batch of input u according to the iterative learning update rule (14) k+1 (t); Step 4.5: Repeat step 4.1 until the number of sampling times reaches N; Step 4.6: Repeat step 4.1 until the tracking error accuracy is less than the set value, then the algorithm ends; Step 5: Design a non-boosting optimization iterative learning control trajectory tracking algorithm when model parameter information is unknown, including: Based on the Q-learning algorithm, a non-boosting optimization iterative learning control trajectory tracking algorithm is designed when the model parameter information is unknown. The Q function is a state-action value function, defined as: The state value function is the policy Δu k+1 State e under (t) k (t+1) value function, so it has the same value as the Q function, that is: V(e k (t+1))=Q(e k (t+1),Δu k+1 (t)) (22) For the optimal iterative learning control problem, the Q function is written in the following quadratic form: in, The system parameter matrix is partitioned and defined as: and, F ee =Q+γP,F eu =-γPCB,F ex =-γPCA (26) F ue =-γB T C T P,F uu =R+γB T C T PCB,F ux =γB T C T PCA (27) F xe =-γA T C T P,F xu =γA T C T PCB,F xx =γA T C T PCA (28) According to formula (23), Δu k+1 (t) Find the quadratic optimal solution and obtain the iterative learning update law: u k+1 (t)=u k (t)+L q τ k+1 (t+1) (29) in, To solve the continuous state space and action space problem, the experimental data is used to estimate the system parameter matrix Parameterize equation (23) as follows: in, Regression vector Defined as Z k+1 The Kronecker product of itself, that is: Among them, z i for The i-th element of , and p=(n+m+l) 2 ; According to formula (32), formula (21) can be expressed as: Since the unknown system parameter matrix is a symmetric matrix, the unknown system parameter vector It stores a total of p(p+1) / 2 independent elements, so L≥p(p+1) / 2 data samples are required to form a data matrix, and then use the least squares method to estimate the system parameter vector; Define the data matrix as: Then the unknown system parameter vector The least squares solution of is calculated as: When the initial input u0(t), initial state x0(t) of the robot impedance control system is given, the expected reference trajectory y d (t), symmetric positive definite weight matrices Q and R are selected, and a non-boosting optimization iterative learning control trajectory tracking algorithm is designed when the system model parameter information is unknown as follows: Step 5.1: Perform u on the discrete system (3) k (t) to collect τ k+1 (t+1), Z k+1 (t+1) and data; Step 5.2: Update the next batch of input u according to the iterative learning update rule (29) k+1 (t); Step 5.3: If L ≥ p(p+1) / 2, proceed to step 5.4; otherwise, proceed to step 5.

6. Step 5.4: Strategy evaluation, according to formula (38) and formula (41), we can get and will Restore to Step 5.5: Policy iteration, according to formula (31), the updated and Thus forming L q ; Step 5.6: Repeat step 5.1 until the number of sampling times reaches N; Step 5.7: Repeat step 5.1 until the tracking error accuracy is less than the set value, and the algorithm ends.

Images