A high-precision control method for flexible robotic arms integrating deep reinforcement learning

Abstract

The present invention proposes a high-precision control method for a flexible robotic arm integrating deep reinforcement learning, which belongs to the technical field of flexible robotic arm motion control, including establishing a mathematical model of the flexible robotic arm; designing a control law based on the model; establishing a Markov decision process of the system; designing a continuous reward function that considers a large range of tracking error characteristics; establishing an algorithm flow based on deep reinforcement learning; and designing a fusion law based on smoothing weighting. The high-precision control method for a flexible robotic arm integrating deep reinforcement learning proposed in the present invention not only has high trajectory tracking accuracy, but also realizes precise control without deploying sensors, and can effectively improve the trajectory tracking performance of the flexible robotic arm.

Description

Technical Field

[0001] The present invention belongs to the technical field of flexible robotic arm motion control, and in particular relates to a high-precision control method for a flexible robotic arm integrating deep reinforcement learning. Background Art

[0002] In the field of modern automation and robotics, flexible manipulator systems have been widely studied due to their lightweight structure and high motion flexibility. Compared with rigid manipulators, flexible manipulators can achieve a larger range of motion in a smaller space and consume less energy. However, due to their inherent elasticity, flexible manipulators produce complex vibrations when performing fast movements or precise trajectory tracking, which poses a huge challenge to control system design.

[0003] Traditional control methods such as adaptive control have limitations when dealing with flexible manipulator systems because they usually rely on accurate mathematical models and require the deployment of deformation sensors. Due to the nonlinearity and parameter uncertainty of flexible manipulator systems, accurate models are difficult to obtain. Control techniques such as linear quadratic regulators, pole placement, linear quadratic Gaussian control, state-dependent Riccati equations, and inverse dynamics-based control provide acceptable optimal performance between residual vibration suppression and manipulator position response accuracy. Although model-based controllers have achieved good results in some aspects of flexible manipulators, this model-based approach has an inherent disadvantage, that is, when the system order increases, the complexity of controller design will also increase. In order to take into account both high-speed operation and high-precision control, new control strategies need to be developed to fully utilize the flexible characteristics of the manipulator, while suppressing undesirable vibrations and ensuring the stability and operation accuracy of the system. Summary of the invention

[0004] The purpose of the present invention is to provide a high-precision control method for a flexible robotic arm integrating deep reinforcement learning, aiming to improve the trajectory tracking accuracy and suppress vibration of the flexible robotic arm.

[0005] Compared with the prior art, the present invention has the following significant advantages:

[0006] (1) This invention proposes for the first time to use a fusion control framework to control the motion control problem of a flexible robotic arm. The framework can dynamically assign different output weights according to the different states of the flexible robotic arm during the motion process, and can give full play to the respective advantages of the model-based method and the reinforcement learning method, which can not only ensure the trajectory tracking performance of the flexible robotic arm, but also improve the training speed of the flexible robotic arm.

[0007] (2) The reward function of the present invention is different from the reward function of traditional design. The reward function designed by the present invention can not only obtain continuous reward signals in the non-convergent area of ​​the trajectory, but also obtain reward signals with large differences under slight state changes in the convergent area of ​​the trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A system composition block diagram of a high-precision control method for a flexible robotic arm integrating deep reinforcement learning provided in an embodiment of the present invention.

[0009] Figure 2 for Figure 1 The deep reinforcement learning control flow chart in the flexible robotic arm fusion control framework shown.

[0010] Figure 3 for Figure 1 The tracking error of the end effector of the flexible manipulator fusion control framework is shown when tracking a quintic polynomial trajectory. DETAILED DESCRIPTION

[0011] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0012] Combination Figure 1-Figure 3 The present invention provides a high-precision control method for a flexible robotic arm integrating deep reinforcement learning, which specifically includes the following steps:

[0013] Step 1: Establish a mathematical model of the flexible robotic arm

[0014] The dynamic model of the flexible manipulator is established using the assumed modal method. The first two modes are taken and the expression is:

[0015]

[0016] Where: M is the mass of the end effector, L is the undeformed length of the flexible manipulator, x is the position on the flexible manipulator body, θ is the virtual rigid rotation angle of the flexible manipulator, is the virtual rigid angular velocity of the flexible manipulator, is the virtual rigid angular acceleration of the flexible manipulator, φ i and q i represents the modal function and modal coordinates of the i, i=1,2 order, φ iL represents the displacement of the i-th mode at the end of the flexible arm, t represents time, τ is the input torque, ρ is the linear density of the flexible manipulator, and EI is the bending section modulus.

[0017] Its matrix expression is:

[0018]

[0019] Where η=[θ,q1,q2] T is the generalized coordinate vector, M(η) is the inertia matrix, is the Coriolis force and centrifugal force matrix, K is the stiffness matrix, T τ =[τ,0,0] T is a non-conservative force vector.

[0020] Step 2: Design model-based control law

[0021] Take the state variable x1 = η, Take control input u1 = T τ , then the state space expression can be listed as:

[0022]

[0023] Wherein the state output y = F r is a vector, which indicates the two-dimensional coordinates of the end effector of the flexible manipulator:

[0024]

[0025] Where x 11 , x 12 , x 13 Represent the components of vector x1 respectively. The control input u1 is designed as:

[0026]

[0027] Where: represents the coordinates of the reference trajectory, ||·||2 represents the l2 norm of the vector, K s represents the control gain, T D represents the time differential constant, represents the rotation angle of the virtual rigid rod in the reference trajectory coordinate system, t represents the time, The meaning is as follows:

[0028]

[0029] In the formula, F ri , i=1,2 represents vector F r The ith element of .

[0030] Step 3: Establish the Markov decision process of the system

[0031] The flexible robotic arm system consists of two parts, namely the environment and the agent. The environment contains the flexible robotic arm and the drive motor. The agent is the proxy decision maker trained by the deep reinforcement learning algorithm. The agent will make decisions and take actions based on the current state given by the environment. The environment will feed back the state of the next moment and the reward of the current action to the agent based on the action taken by the agent. The reason why SAC is selected as the algorithm for training the agent is its good convergence and training stability. The SAC algorithm introduces the concept of entropy. Therefore, some definitions in the Markov decision process need to be rewritten. The Soft Bellman equation is rewritten as:

[0032]

[0033] In the formula, s t represents the state at time t, s t+1 represents the state at time t+1, a t represents the action at time t, Q(s t ,a t ) means in state s t Take action a t The action value function, r(s t ,a t ) means in state s t Take action a t reward, γ is the discount factor, V(s t+1 ) indicates state s t+1 The state value function under Represents the state value function V(s t+1 ) is the mathematical expectation of .

[0034] The state value function in formula (1) is written as:

[0035]

[0036] In formula (2), V(s t ) indicates state s t The state value function under α is a regularization coefficient used to control the importance of entropy, π(a t |s t ) means in state s t Take action a t The action strategy function, represents the action a selected from the strategy π t The mathematical expectation, H(π(·|s t )) indicates that the strategy π is in the current state s t The degree of randomness under, that is, its entropy, is defined as Where X represents a random variable, x is the observed value of the random variable X, p(x) is the probability density function of x, is the mathematical expectation of x that obeys the probability density function p(x).

[0037] Step 4: Design a continuous reward function that takes into account the characteristics of a large range of tracking errors

[0038] For the task of the flexible manipulator, the reward function is dense only within a small range of motion. In the early stages of the agent training, the sparse rewards it receives over a large range will lead to slow training. To this end, the reward function should allow the flexible manipulator system to receive continuous rewards within a large range of motion, and accelerate the convergence speed of the agent in the early stages of training to improve training efficiency. Its expression is:

[0039]

[0040] Where r is the reward value of the environment feedback, χ1, χ2 and χ3 are reward parameters, θ ref The reference angle of the virtual rigid rod is e θ =|θ-θ ref | represents the difference between the rotation angle and the reference angle of the virtual rigid rod, e θmax represents the set threshold value of the rotation error, represents the distance between the end effector and the reference trajectory, e dismax Indicates the set threshold of distance error, conditions 2 and 3 are e θ <e θmax A subset of , f(t) represents a limit value related to time t, the purpose of which is to weaken the influence of static deformation caused by gravity in the initial state.

[0041] Step 5: Establish a deep reinforcement learning algorithm process

[0042] In the SAC algorithm, we model two action value functions Q (with parameters ω1 and ω2) and a policy function π (with parameters θ). Based on the idea of ​​Double DQN, SAC uses two Q networks, but each time a Q network is used, a network with a small Q value is selected to alleviate the problem of overestimation of the Q value.

[0043] In order to improve the stability and sample efficiency of training, the experience replay and soft update target network methods are used here. The specific method of experience replay is to maintain a replay buffer and store the four-tuple data (s t ,a t ,r t ,s t+1 ) is stored in the replay buffer, and the Q network randomly samples some data from the replay buffer for training.

[0044] Different from the method in the DQN algorithm that directly copies the Q network parameters to the target Q network at regular intervals, the soft update method updates the target Q network slowly and gradually approaches the Q network.

[0045] The deep reinforcement learning algorithm flow of the flexible robotic arm is:

[0046] 1) Initialize the Critic network with random network parameters ω1, ω2 and θ respectively and Actor Network π θ (s);

[0047] 2) Copy the same parameters ω1 to ω2 to Initialize the target network separately and in and The target network and Parameters;

[0048] 3) Initialize the experience replay pool R;

[0049] 4) Obtain the initial state s1 of the environment;

[0050] 5) According to the current strategy π θ and status t Select action a t =π θ (s t );

[0051] 6) Execute action a t , get reward r t , the environment state becomes s t+1 ;

[0052] 7) will (s t ,a t ,r t ,s t+1 ) is deposited into the replay pool R;

[0053] 8) Return to step 5) until time step t;

[0054] 9) Sample N tuples from R {(s t ,a t ,r t ,s t+1 )} i＝1,...,N ;

[0055] 10) For each tuple, use the target network to calculate where y i is the discounted return of the i-th tuple, ri is the reward of the i-th tuple, j is the index of the j-th target network, Indicates that in state s i+1 and action a i+1 The following parameters are The action value function of the target network, π θ (a i+1 |s i+1 ) means in state s i+1 According to the policy function π with parameter θ θ Take action a i+1 The probability of a, the subscript i is the index of the number of sample tuples N, i = 1, 2, ..., N, a i+1 Represents the policy function π θ The next moment state s i+1 Actions taken;

[0056] 11) Update both Critic networks as follows: For j = 1, 2, minimize the loss function Where N is the number of sample tuples, and each symbol and subscript is associated with the meaning in step 10;

[0057] 12) Use the reparameterization technique to sample the action Then use the following loss function L π (θ) Update the current Actor network

[0058] The meanings of the symbols and subscripts are related to those in step 10. and They represent the policy function π with parameter θ respectively. θ In status i Downsampling Action The probability and parameter are ω j Value Network In status i and resampling actions the value of

[0059] 13) Update the coefficient α of the entropy regularization term;

[0060] 14) Update the target network and set the parameters of the target network The value of is updated to in

[0061] ω j ,j=1,2 are the parameters of the two Critic networks, τ is the update law factor;

[0062] 15) Return to step 9) until the number of training rounds is k;

[0063] 16) Return to step 4) until the sequence reaches e times.

[0064] Step 6: Design the fusion law

[0065] The fusion law can combine the advantages of the model-based control law and the deep reinforcement learning control law method, that is, the high tracking accuracy of the model-based control law is brought into play in the dynamic process of the flexible manipulator movement, and the high stability accuracy and vibration suppression effect of the deep reinforcement learning control law are brought into play in the steady-state process of the flexible manipulator movement. Its expression is:

[0066]

[0067] Where: u1 is the control input in equation (5), u2 = a t Represents the output of the deep reinforcement learning control law, that is, the output of the deep reinforcement learning algorithm according to the current strategy π in step 5 θ and status t Selected action a t =π θ (s t ). represents the angular velocity of the reference trajectory. Indicates the maximum traversal value of the reference trajectory angular velocity, that is, the maximum value that can be found by traversing from the current moment forward. K g is the time-delay gain. st It means that the input will be output after a delay of time t, and s is a positive constant. -1 It means that the input will be delayed by one time step before being output. Indicates a The mathematical expression of the relevant symbolic function is:

[0068]

[0069] In this example, the training parameters of the deep reinforcement learning agent are set to: action network learning rate 0.0003, value network learning rate 0.002, and entropy learning rate 0.0002. The target entropy is set to the negative action space dimension. The intuition is that the distribution over each action dimension should have a uniform probability distribution. Therefore, the target entropy is set to -1. The soft update parameter, the target smoothing factor, is set to 0.005. The buffer size is set to 10000. The DRL agent is trained with a sampling rate of 5ms and a fixed simulation time as the stopping criterion.

[0070] The parameters of the flexible manipulator and the model-based control law are set as follows: the length of the manipulator is 1.5m, the cross-sectional area is 2cm*1cm, the material density of the flexible manipulator is 2700kg / m^3, the Young's modulus is 70Gpa, and the Poisson's ratio is 0.33. The controller torque limit is ±60Nm, the end effector mass is 0.4kg, and the controller parameter K s 600, T D is 0.167, K g is 0.6, and the delay window length is 0.3s.

[0071] The embodiments of the present invention are described in detail above, but the contents described are only preferred embodiments of the present invention and are not intended to limit the present invention. Any equivalent replacements or improvements made according to the present invention should be included in the protection scope of the present invention.

Claims

1. A high-precision control method for a flexible robotic arm integrating deep reinforcement learning, characterized in that: A smooth weighted fusion law is used to improve the control effect of the flexible manipulator under the model-based control law and deep reinforcement learning control. The designed fusion law is: Where: u1 is the control input, u2 is the output of the deep reinforcement learning control law, represents the angular velocity of the reference trajectory, represents the maximum traversal value of the reference trajectory angular velocity, that is, the maximum value that can be found by traversing forward from the current moment, K g is the time-delay gain, e st It means that the input will be output after the delay time t, z -1 It means that the input will be delayed for one time step before output. Indicates a Related symbolic functions; The deep reinforcement learning algorithm process is: 1) Initialize the Critic network with random network parameters ω1, ω2 and θ respectively and Actor Network π θ (s); 2) Copy the same parameters ω1 to ω2 to Initialize the target network separately and in and The target network and Parameters; 3) Initialize the experience replay pool R; 4) Obtain the initial state s1 of the environment; 5) According to the current strategy π θ and status t Select action a t =π θ (s t ); 6) Execute action a t , get reward r t , the environment state becomes s t+1 ; 7) will (s t ,a t ,r t ,s t+1 ) is deposited into the replay pool R; 8) Return to step 5) until time step t; 9) Sample N tuples from R {(s t ,a t ,r t ,s t+1 )} i＝1,...,N ; 10) For each tuple, use the target network to calculate where y i is the discounted return of the i-th tuple, r i is the reward of the i-th tuple, j is the index of the j-th target network, γ is the discount factor, α is a regularization coefficient, Indicates that in state s i+1 and action a i+1 The following parameters are The action value function of the target network, π θ (a i+1 |s i+1 ) means in state s i+1 According to the policy function π with parameter θ θ Take action a i+1 The probability of a, the subscript i is the index of the number of sample tuples N, i = 1, 2, ..., N, a i+1 Represents the policy function π θ The next moment state s i+1 Actions taken; 11) Update both Critic networks as follows: For j = 1, 2, minimize the loss function Where N is the number of sample tuples, Indicates that in state s i and action a i The lower parameter is ω j , action value function of two critic networks with j=1,2; 12) Use the reparameterization technique to sample the action Then use the following loss function L π (θ) Update the current Actor network and They represent the policy function π with parameter θ respectively. θ In status i Downsampling Action The probability and parameter are ω j Value Network In status i and resampling actions the value of 13) Update the coefficient α of the entropy regularization term; 14) Update the target network and set the parameters of the target network The value of is updated to where ω j ,j=1,2 are the parameters of the two Critic networks, τ is the update law factor; 15) Return to step 9) until the number of training rounds is k; 16) Return to step 4) until the sequence is repeated e times; Where r is the reward value of the environment feedback, χ1, χ2 and χ3 are reward parameters, θ ref is the reference angle of the virtual rigid rod, e θ =|θ-θ ref | represents the difference between the rotation angle and the reference angle of the virtual rigid rod, e θmax represents the set threshold value of the rotation error, represents the distance between the end effector and the reference trajectory, e dismax represents the set threshold of the distance error, and f(t) represents a limit value related to time t; The control input u1 is: Where: represents the coordinates of the reference trajectory, ||·||2 represents the l2 norm of the vector, K s represents the control gain, T D represents the time differential constant, represents the rotation angle of the virtual rigid rod in the reference trajectory coordinate system, t represents the time, Indicates a with F ri The relevant symbolic function, F ri Represents the vector F r The i-th element of F r Represents the spatial two-dimensional coordinates of the end effector of the flexible robotic arm.

2. According to the high-precision control method of a flexible robotic arm integrating deep reinforcement learning as described in claim 1, it is characterized in that: The spatial two-dimensional coordinates of the end effector of the flexible robotic arm are: where x 11 , x 12 , x 13 They represent the components of the state variable x1 of the control rate, L is the undeformed length of the flexible manipulator, φ iL It represents the displacement of the i-th mode at the end of the flexible arm.

3. According to the high-precision control method of a flexible robotic arm integrating deep reinforcement learning as described in claim 2, it is characterized in that: The state variable x1 satisfies the following state space expression: State variables x1=η, M(x1)=M(η) is the inertia matrix, is the Coriolis force and centrifugal force matrix, and K is the stiffness matrix.

Images