Mechanical arm continuous trajectory tracking control system based on behavior cloning

Abstract

The invention relates to the technical field of mechanical arm control, in particular to a mechanical arm continuous trajectory tracking control system based on behavior cloning, which comprises a mechanical arm feature expression module, a proportional agent module, a behavior cloning module, a multi-agent module, a PI D control module, a reward determination module and an updating module, the mechanical arm is controlled through a multi-agent deep reinforcement learning method in combination with behavior cloning, the PI D agent and the DDR agent jointly output torque to control the same mechanical arm, and strategy networks of the two agents are optimized through two reward functions. The PI D intelligent body is used for outputting parameters of the PI D controller, then the PI D controller outputs torque to control the mechanical arm, so that the mechanical arm can track other tracks different from a training track, the DDR intelligent body directly outputs the torque to increase the anti-interference performance of the controller, the anti-interference capability is achieved in the environment with strong random interference and variability, and the mechanical arm can track other tracks. And the operation accuracy of the mechanical arm is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot arm control, and in particular to a robot arm continuous trajectory tracking control system based on behavior cloning. Background Art

[0002] In recent years, with the popularity of intelligent manufacturing, robot systems have developed rapidly and have been widely used in many industries such as photovoltaics and automobiles. Robot path planning and trajectory tracking are one of the key technologies for robot automation and intelligence. At present, many achievements have been made in the research on robot arm control. With the increase of control complexity, the uncertainty and system interference of robots should also be paid special attention in control design.

[0003] It is worth noting that the dynamic models of robots, especially in complex industrial applications, are often time-varying or uncertain. In order to adapt to this uncertainty, neural networks have been widely used in recent years to solve the control design problems of uncertain nonlinear dynamic robot systems.

[0004] Chinese Patent Publication No.: CN116512233B discloses a mechanical arm and a control system for the mechanical arm, which can rotate the device by setting a first motor, and can lift and lower the mounting column while rotating by setting a rotating plate, a telescopic rod and a first circular sleeve. A mechanical arm and a control system for the mechanical arm include a base and a first motor, the top of the base is fixedly connected to a support column, the top of the support column is provided with a mounting groove, the mounting column is slidably connected in the mounting groove, the output end of the first motor is fixedly connected to the bottom end of the mounting column, the top of the support column is provided with a rotating groove, the rotating groove is rotatably connected with a rotating plate, the top of the rotating plate is fixedly connected with a plurality of first circular sleeves, the plurality of first circular sleeves are slidably connected with telescopic rods, and the top of the telescopic rod is fixedly connected with a fixing ring; it can be seen that the above technical solution has the following problems: in the face of strong random interference and changeable environment, it does not have anti-interference ability, which affects the operation accuracy of the mechanical arm. Summary of the invention

[0005] To this end, the present invention provides a robot arm continuous trajectory tracking control system based on behavior cloning to overcome the problem in the prior art that the robot arm has no anti-interference ability in the face of strong random interference and changeable environments, which affects the operation accuracy of the robot arm.

[0006] To achieve the above object, the present invention provides a robot arm continuous trajectory tracking control system based on behavior cloning, comprising:

[0007] A robot arm feature expression module, which is used to model the robot arm through dynamic equations;

[0008] A proportional agent module that determines the output target action through a non-online actor-critic DRL algorithm;

[0009] The behavior cloning module is used to determine the target action output by the actor-critic DRL algorithm and the loss value shown by expert experience based on the error formula;

[0010] A multi-agent module, which includes a PID agent for outputting PID control parameters and a DDR agent for outputting control torque, and the two agents control the same robotic arm;

[0011] A PID control module, which is used to output torque based on the PID parameters output by the multi-agent module to control the robotic arm so that the robotic arm tracks a trajectory other than the training trajectory;

[0012] A reward determination module, which is used to determine the reward value of the corresponding agent through the corresponding reward function of each agent;

[0013] The update module is used to control the update of each agent.

[0014] Furthermore, the robot arm feature expression module is used to determine the dynamic equation, which is:

[0015]

[0016] Among them, q is the joint angle variable of the robot arm, is the joint angular velocity, τ is the torque of the robot, M(q) is an n×n matrix, and the n×n matrix is ​​determined as the inertia matrix. is the centrifugal force and Coriolis force matrix, G(q)∈Rn is the gravity matrix, n is the total number of joints of the robot arm, is a nonlinear interference, is the torque.

[0017] Further, the robot arm feature expression module is used to determine the torque;

[0018]

[0019] is the torque of the i-th joint of the robot, i = 1, 2, 3, ... n, n is the total number of joints of the robot;

[0020] The robot arm feature expression module is used to determine the torque constraint, which is:

[0021]

[0022] Among them, q min is the preset minimum torque, q maxThe maximum value of the preset torque.

[0023] Further, the robot arm feature expression module is used to determine nonlinear interference;

[0024]

[0025] in, Used to represent nonlinear interference, is the interference force on the i-th joint, i = 1, 2, 3, ... n, n is the total number of joints of the robot, and r is a random number;

[0026] The robot arm feature expression module is used to determine the constraint conditions of the interference force on the i-th joint;

[0027]

[0028] Wherein, Radn∈[0,1] is a random number, x is a preset trigger value, x∈[0,1]; when the random number Radn is greater than the preset trigger value, interference is triggered; G(r) is used to determine the unknown function of the interference torque in the case of interference triggering, G(r)∈(-K,K).

[0029] Further, the proportional agent module is used to determine the definition tuple;

[0030] Define the tuple as D = (S, A, π, Q, R, γ),

[0031] Among them, S represents the state space, A represents the action space, π represents the actor neural network to be trained, Q represents the critic neural network to be trained, R represents the reward space, and γ is the discount factor.

[0032] Further, the proportional agent module is used to determine the formula of the target action output by the target strategy of TD3;

[0033] a'(s')=clip(π φ′ (s′)+clip(∈,-c,c),a low ,a high )

[0034] Among them, a' is the target action, s' is the next state generated by the current action acting on the current state, π φ′ (s′) is the target strategy π φ′ The target action generated by the next action s' is ∈, which is the added random noise. The value range of ∈ is (-c, c), (a low , a high) is the size range of the target action output by the target strategy, and clip(a, b, c) means truncating a with the range (b, c);

[0035] TD3 is a twin delayed deep deterministic policy gradient algorithm. TD3 is a non-online actor-critic DRL algorithm designed specifically for continuous action space tasks.

[0036] The proportional agent module is used to determine Q, including:

[0037] The actor-critic DRL algorithm includes two different Q networks for estimating actions, and the proportional agent module determines the minimum Q value of the two Q values ​​as the update target to suppress continuous overestimation;

[0038] In the actor-critic DRL algorithm, the Q value is determined based on the Q value solution formula. The Q value solution formula is:

[0039]

[0040] Among them, r is the return value given by the environment, γ is the discount factor, It is a critic neural network composed of double Q networks, Q θ′i is a critic neural network with network parameters θ′ containing two Q networks;

[0041] The proportional agent module is used to determine π, including:

[0042] Used to determine the update formula of the actor network. The update formula of the actor network is:

[0043]

[0044] in, For the experience pool, is the policy gradient of the actor network during parameter update, and φ is the network parameter.

[0045] Furthermore, the error formula is:

[0046]

[0047] Among them (s i ,a i ) is the state and expert action of a round, N is the number of samples drawn from the dataset, f(s i ) is the predicted action of the network to be trained for the current state, and L(A) is the loss value.

[0048] Furthermore, the multi-agent module is used to determine the PID parameters through a PID controller expression, and the PID controller expression is

[0049]

[0050] in, is the torque output by the PID agent, P is the output of the proportional term, Kp is the proportional coefficient, e(t) represents the error at time t, i represents the output of the integral term, and K i is the integration coefficient, ∫0 t e(t)d t represents the integral of the error from the beginning to the current moment, D is the output of the differential term, K d is the differential coefficient, is the rate of change of error, and the three parameters Kp, K i and K d .

[0051] Furthermore, the PID reward function is,

[0052] r p (t)=-|v e ·e(t)|

[0053] Among them, r p (t) is the instantaneous reward of the PID agent at time t, e(t) represents the tracking error, e(t) = q(t) - q d (t), q(t) represents the position of the robot arm, qd(t) represents the desired position of the robot arm, and ve is the coefficient of the tracking error;

[0054] The reward determination module determines the expected discounted reward of the PID agent through the PID discounted reward function, and the PID discounted reward function is:

[0055]

[0056] Among them, R P (t) is the expected discounted reward of the PID agent after time step t;

[0057] The DDR reward function is,

[0058] r d (t)=-(|v e ·e(t)|)+v ed ·e d (t)

[0059] Among them, r d (t) is the instantaneous reward of the DDR agent at time t, e d (t) represents the change in error, ed (t)=|e(t-1)|-|e(t)|, e dv is the parameter of the error variation;

[0060] The reward determination module determines the expected discounted reward of the DDR agent through the DDR discounted reward function as follows:

[0061]

[0062] γ∈[0, 1] represents the discount factor.

[0063] Furthermore, the update module is used to control the PID agent update:

[0064] Used to initialize the review network with random network parameters θ1, θ2 and φ and action network π φ , initialize the target network parameters θ′1←θ1,θ′2←θ2,φ′←φ, initialize the replay experience pool And the pool of expert experience

[0065] Used in the expert experience pool Extract expert experience, use the expert experience to calculate the loss value according to the error formula, use the loss value as the Q value and use the actor network update formula to pre-train the actor network;

[0066] Output action a = π from the actor network φ (s) is input into the PID control module, which outputs torque to the environment to obtain the next state s′. The reward RM is obtained through the PID discounted reward function, and the current state transition tuple (s, a, r, s′) is stored as experience in the experience pool. In , if s′ is the end state of this round, reset the environment state;

[0067] When the number of experiences in the experience pool meets the number for updating, N batch samples are extracted for training update;

[0068] Use TD3's target strategy to output the target action formula and Q value solution formula to calculate the target action and Q value, where Q value is y(r, s′);

[0069] Update the critic network using the calculated Q value:

[0070]

[0071] When the number of updates of the critic network reaches a preset fixed number, the actor network is updated using the actor network update formula;

[0072] Soft update the target network of the two critic networks and the actor network:

[0073] θ′ i ←αθ i +(1-α)θ′ i

[0074] φ′←αφ+(1-α)φ′

[0075] Among them, α is a soft update weight not greater than 1;

[0076] The update module is used to control the DDR agent update:

[0077] Used to initialize the review network with random network parameters θ1, θ2 and φ and action network π φ , initialize the target network parameters θ′1←θ1,θ′2←θ2,φ′←φ, initialize the replay experience pool

[0078] Use the actor network to output action a = π φ (s) environment, get the next state s′, get the reward through the DDR discounted reward function, and store the current state transition tuple (s, a, r, s′) as experience in the experience pool In , if s′ is the end state of this round, reset the environment state;

[0079] When the number of experiences in the experience pool meets the number for updating, N batch samples are extracted for training update;

[0080] Use TD3's target strategy to output the target action formula and Q-value solution formula to calculate the target action and Q-value;

[0081] Update c using the calculated Q value r iti c network:

[0082]

[0083] When the number of updates of the critic network reaches a preset fixed number, the actor network is updated using the actor network update formula;

[0084] Put two c r iti c Network and acto r The target network of the network is soft updated:

[0085] θ′ i ←αθ i +(1-α)θi '

[0086] φ′←αφ+(1-α)φ′

[0087] Among them, α is a soft update weight not greater than 1.

[0088] Compared with the prior art, the beneficial effect of the present invention is that the robot arm characteristics can be accurately characterized by the robot arm feature expression module, and compared with other algorithms, the robot arm feature expression module does not need to consider the influence of constraint force, thus saving modeling time.

[0089] Furthermore, behavior cloning can be used for model pre-training and to reduce the difficulty of reward convergence of the model. The learning process of behavior cloning is to learn a mapping relationship from state to expert behavior, which is usually expressed by a neural network. The training purpose of behavior cloning is to make the agent perform the same actions when encountering the same situation as the expert. Therefore, the training process of behavior cloning can be understood as reducing the difference between the actions output by the neural network and the behaviors displayed by the expert experience. By introducing behavior cloning, the training speed of the neural network is improved.

[0090] Furthermore, TD3 can well solve the control problem of robot trajectory tracking and has good anti-interference ability. The TD3 with PID algorithm is designed by introducing the PID control module. The TD3 with PID algorithm uses a neural network to output PID controller parameters and then outputs torque through the PID control module to control the robot for trajectory tracking, which improves the tracking generality of the algorithm and overcomes the problem that the neural network has the characteristic of task fitting, resulting in poor performance when tracking other trajectories different from the training trajectory; for the trajectory tracking control task of a multi-joint robot in an environment with nonlinear interference and a changing environment, this solution introduces multi-agent deep reinforcement learning (MDRL) and combines the advantages of the two algorithms to solve the problem of weak anti-interference ability of the PID algorithm caused by the weak ability of the algorithm to respond to interference due to the need to output parameters to the PID control module first and then the PID control module outputs torque for control, thereby improving the tracking generality and anti-interference ability of the TD3with PID algorithm.

[0091] Furthermore, by introducing behavioral cloning, the problem of the MDRL algorithm being unable to converge or even having no positive convergence trend during training experiments was solved, thereby improving the training speed. In the face of complex tasks, functional decomposition was performed, and the PID agent was combined with the PID control module to improve the algorithm's generalization ability, and the DDR agent was used to improve the algorithm's anti-interference ability. Through the above solutions, the robotic arm can adapt to trajectory tracking tasks in different environments and have different functions to adapt to the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 It is a module block diagram of a robot arm continuous trajectory tracking control system based on behavior cloning according to an embodiment of the present invention;

[0093] Figure 2 A logic determination diagram for determining the interference force on the i-th joint by the characteristic expression module of the robot arm according to an embodiment of the present invention;

[0094] Figure 3 A block diagram of a multi-agent module and a multi-agent module joint output torque according to an embodiment of the present invention;

[0095] Figure 4 This is a flow chart of the steps of controlling the update of a DDR intelligent entity by an update module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0096] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0097] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0098] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0099] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0100] The present invention is referred to Figure 1 As shown, it is a module block diagram of a robot arm continuous trajectory tracking control system based on behavior cloning according to an embodiment of the present invention. A robot arm continuous trajectory tracking control system based on behavior cloning according to an embodiment of the present invention includes:

[0101] A robot arm feature expression module, which is used to model the robot arm through dynamic equations;

[0102] A proportional agent module that determines the output target action through a non-online actor-critic DRL algorithm;

[0103] The behavior cloning module is used to determine the target action output by the actor-critic DRL algorithm and the loss value shown by expert experience based on the error formula;

[0104] A multi-agent module, which includes a PID agent for outputting PID control parameters and a DDR agent for outputting control torque, and the two agents control the same robotic arm;

[0105] A PID control module, which is used to output torque based on the PID parameters output by the multi-agent module to control the robotic arm so that the robotic arm tracks a trajectory other than the training trajectory;

[0106] A reward determination module, which is used to determine the reward value of the corresponding agent through the corresponding reward function of each agent;

[0107] The update module is used to control the update of each agent.

[0108] Specifically, the robot arm feature expression module is used to determine the dynamic equation, which is:

[0109]

[0110] Among them, q is the joint angle variable of the robot arm, is the joint angular velocity, τ is the torque of the robot, M(q) is an n×n matrix, and the n×n matrix is ​​determined as the inertia matrix. is the centrifugal force and Coriolis force matrix, G(q)∈Rn is the gravity matrix, n is the total number of joints of the robot arm, is a nonlinear interference, is the torque.

[0111] Specifically, the robot arm feature expression module is used to determine the torque;

[0112]

[0113] is the torque of the i-th joint of the robot, i = 1, 2, 3, ... n, n is the total number of joints of the robot;

[0114] The robot arm feature expression module is used to determine the torque constraint, which is:

[0115]

[0116] Among them, q minis the preset minimum torque, q max The maximum value of the preset torque.

[0117] The robot arm feature expression module is used to determine nonlinear interference;

[0118]

[0119] in, Used to represent nonlinear interference, is the interference force on the i-th joint, i = 1, 2, 3, ... n, n is the total number of joints of the robot, and r is a random number;

[0120] See also Figure 2 As shown, it is a logical determination diagram of the interference force on the i-th joint determined by the characteristic expression module of the robot arm according to an embodiment of the present invention. The characteristic expression module of the robot arm according to the present invention is used to determine the constraint condition of the interference force on the i-th joint;

[0121]

[0122] Wherein, Radn∈[0,1] is a random number, x is a preset trigger value, x∈[0,1]; when the random number Radn is greater than the preset trigger value, interference is triggered; G(r) is used to determine the unknown function of the interference torque in the case of interference triggering, G(r)∈(-K,K).

[0123] Specifically, the size of G(r) corresponds to the random number r. The size of G(r) has nothing to do with the force of the robot arm and is an external interference. The size range of the interference can be determined by setting K, and the probability of interference can be determined by setting x.

[0124] Specifically, after the number of joints is determined, the above matrices can be obtained based on actual modeling.

[0125] Specifically, the robot arm feature expression module can accurately characterize the characteristics of the robot arm. Compared with other algorithms, the robot arm feature expression module does not need to consider the influence of constraint force, saving modeling time.

[0126] Specifically, by determining nonlinear interference, given an environment with random nonlinear interference and a target tracking trajectory, a multi-agent deep reinforcement learning controller is designed for the robot arm. Without knowing the dynamic model of the robot arm in advance, the robot arm can track the target tracking trajectory in a changing environment with interference. The content of this paper is the robot arm trajectory tracking task in an interference and changing environment. This algorithm innovatively combines two agents with different output torque modes and successfully trains through BC technology, which increases the control versatility and anti-interference ability during the tracking task. In actual control, the control versatility of the controller can increase its application prospects. As well as the ability to cope with special situations, the anti-interference ability can be enhanced The control stability of the controller is improved to adapt to more extreme application scenarios. Therefore, this study has important reference significance for the solutions of spraying, assembly and other tasks in harsh environments such as sea and desert and in bad weather.

[0127] Specifically, the proportional agent module is used to determine the definition tuple;

[0128] Define the tuple as D = (S, A, π, Q, R, γ),

[0129] Among them, S represents the state space, A represents the action space, π represents the actor neural network to be trained, Q represents the critic neural network to be trained, R represents the reward space, and γ is the discount factor.

[0130] The proportional agent module is used to determine the formula of the target action output by the target strategy of TD3;

[0131] a'(s')=clip(π φ′ (s′)+clip(∈,-c,c),a low, a high )

[0132] Among them, a' is the target action, s' is the next state generated by the current action acting on the current state, π φ′ (s′) is the target strategy π φ′ The target action generated by the next action s' is ∈, which is the added random noise. The value range of ∈ is (-c, c), (a low , a high ) is the size range of the target action output by the target strategy, and clip(a, b, c) means truncating a with the range (b, c);

[0133] TD3 is the twin delayed deep deterministic policy gradient algorithm. TD3 is a non-online actor-critic DRL algorithm designed specifically for continuous action space tasks.

[0134] Specifically, the proportional agent module is used to determine Q, including:

[0135] The actor-critic DRL algorithm includes two different Q networks for estimating actions, and the proportional agent module determines the minimum Q value of the two Q values ​​as the update target to suppress continuous overestimation;

[0136] In the actor-critic DRL algorithm, the Q value is determined based on the Q value solution formula. The Q value solution formula is:

[0137]

[0138] Among them, r is the return value given by the environment, γ is the discount factor, It is a critic neural network composed of a double Q network. Here, it refers to the evaluation network whose network parameter is θ′. Since Q is TD3, it is a double Q network, so there are two evaluation networks. Here, i refers to the i-th network, so Q θ′i It refers to a critic network system with two Q networks and a network parameter of θ′. The critic network, Q network and evaluation network all refer to a neural network used to evaluate the value of an action, but they are called differently in different places. Here, the two critic networks form a critic policy.

[0139] Specifically, DRL is deep reinforcement learning.

[0140] Specifically, the proportional agent module is used to determine π, including:

[0141] Used to determine the update formula of the actor network. The update formula of the actor network is:

[0142]

[0143] in, is the experience pool. The formula here is to calculate the policy gradient ▽φJ(φ) of the Actor network during parameter update, which is calculated for the Actor network parameter φ.

[0144] Specifically, the actor is used to continuously perform gradient updates to determine the maximum Q value.

[0145] Specifically, when the critic network is unstable, the output Q value is also unstable, which will also lead to unstable updates of the actor network. Therefore, the critic network is updated multiple times until the output Q value is accurate, and then the actor network is updated to make the training more stable.

[0146] Specifically, the error formula is:

[0147]

[0148] Among them (s i ,a i ) is the state and expert action of a round, N is the number of samples drawn from the dataset, f(s i ) is the predicted action of the network to be trained for the current state, and L(A) is the loss value.

[0149] Specifically, behavior cloning can be used for model pre-training and to reduce the difficulty of reward convergence of the model. The learning process of behavior cloning is to learn a mapping relationship from state to expert behavior, which is usually expressed by a neural network. The training goal of behavior cloning is to make the agent perform the same actions when encountering the same situation as the expert. Therefore, the training process of behavior cloning can be understood as reducing the difference between the actions output by the neural network and the behaviors demonstrated by the expert experience.

[0150] See also Figure 3 As shown, it is a module block diagram of a multi-agent module and a multi-agent module jointly output torque in an embodiment of the present invention. The multi-agent module of the present invention includes a PID agent for outputting PID control parameters and a DDR agent for outputting control torque. The two agents control the same robotic arm. The PID control module is used to determine the PID parameters and output torque based on the PID parameters to control the robotic arm so that the robotic arm tracks trajectories other than the training trajectories.

[0151] Specifically, the multi-agent module is used to determine the PID parameters through the PID controller expression, and the PID controller expression is:

[0152]

[0153] Among them, P is the output of the proportional term, Kp is the proportional coefficient, e(t) represents the error at time t, i represents the output of the integral term, and K i is the integration coefficient, ∫0 t e (t)d t represents the integral of the error from the beginning to the current moment, D is the output of the differential term, K d is the differential coefficient, is the rate of change of error, and the three parameters Kp, K i and Kd, It refers to the torque output by the PID controller. Because the torque output by the PID agent in the MDRL controller is τ1, the subscript 1 is also added here. And here it only explains the way the PID control module outputs the torque, not the actual output torque of the PID agent, so a horizontal line is added above to distinguish it.

[0154] Specifically, the full name of PID is Proportional Integral Derivative (PID), which is a feedback control algorithm.

[0155] Specifically, the PID agent only needs to find the PID parameters suitable for the current robotic arm to complete the tracking task, so its training speed is faster, but its anti-interference effect still needs to be compensated by the DDR agent.

[0156] Specifically, the PID reward function is,

[0157] r p (t)=-|v e ·e(t)|

[0158] Among them, r p (t) is the instantaneous reward of the PID agent at time t, e(t) represents the tracking error, e(t) = q(t) - q d (t), q(t) represents the position of the robot arm, qd(t) represents the desired position of the robot arm, and ve is the coefficient of the tracking error;

[0159] Specifically, the reward determination module determines the expected discounted reward of the PID agent through the PID discounted reward function, and the PID discounted reward function is,

[0160]

[0161] Among them, R P (t) is the discounted reward expected by the PID agent after time step t, and the reward sequence from the environment after time step t is r t+1 , r t+2 , r t+3 …, here it is just a formula for demonstration and no cut-off time is set, so there is no cut-off value behind it;

[0162] The DDR reward function is,

[0163] r d (t)=-(|v e ·e(t)|)+v ed ·e d (t)

[0164] Among them, r d (t) is the instantaneous reward of the DDR agent at time t, e d (t) represents the change in error, e d (t)=|e(t-1)|-|e(t)|, e dv is the parameter of the error variation;

[0165] The reward determination module determines the expected discounted reward of the DDR agent through the DDR discounted reward function as follows:

[0166]

[0167] γ∈[0,1] represents the discount factor.

[0168] Specifically, the PID agent pays more attention to the overall trajectory tracking performance, so the reward of the PID agent only uses the trajectory error as the reward value.

[0169] Specifically, using the error change as a reward can make the DDR agent more sensitive to tracking stability. The DDR agent directly outputs torque to the robot arm, so it can respond quickly to interference or system fluctuations, which can improve the system's anti-interference ability and tracking stability. Assume that after time step t, the error from the environment

[0170] Specifically, the reward sequence can be expressed as r t+1 , r t+2 , r t+3 …, then the sum of the weighted immediate rewards is the expected discounted reward.

[0171] Specifically, the two agents output torque in completely different ways. The PID agent outputs the PID control parameters and the PID control module outputs the control torque. The two agents also have different work priorities during the tracking task. The PID agent pays more attention to the overall tracking effect, while the DDR agent pays more attention to anti-interference ability and tracking stability. Due to the differences in the working logic and work priorities of the two agents, two different reward functions are used to reflect the work performance of each agent. While the two agents have the same common goal, they are more sensitive to different parameter changes. The two reward functions differentiate the overall task into internal rewards for the two agents, enhancing the collaboration of the two agents.

[0172] Specifically, TD3 can well solve the control problem of robot trajectory tracking and has good anti-interference ability. The TD3 with PID algorithm is designed by introducing the PID control module. The TD3 with PID algorithm uses a neural network to output PID controller parameters and then outputs torque through the PID control module to control the robot for trajectory tracking, which improves the tracking generality of the algorithm and overcomes the problem that the neural network has the characteristic of task fitting, resulting in poor performance when tracking other trajectories different from the training trajectory; for the trajectory tracking control task of a multi-joint robot in an environment with nonlinear interference and a changing environment, this solution introduces multi-agent deep reinforcement learning (MDRL) and combines the advantages of the two algorithms to solve the problem that the PID algorithm has weak anti-interference ability due to the fact that PID needs to output parameters to the PID control module first and then the PID control module outputs torque for control, which causes the algorithm to have weak response to interference, thereby improving the tracking generality and anti-interference ability of the TD3with PID algorithm.

[0173] Specifically, the introduction of behavioral cloning solved the problem of the MDRL algorithm being unable to converge or even having no positive convergence trend during training experiments, thereby improving the training speed. In the face of complex tasks, functional decomposition was performed, and the PID agent was combined with the PID control module to improve the algorithm's versatility, and the DDR agent was used to improve the algorithm's anti-interference ability. Through the above solutions, the robotic arm can adapt to trajectory tracking tasks in different environments and have different functional adaptation environments.

[0174] Specifically, the update module is used to control the PID agent update:

[0175] Used to initialize the review network with random network parameters θ1, θ2 and φ and action network π φ , initialize the target network parameters θ′1-θ1,θ′2←θ2,φ′←φ, initialize the replay experience pool And the pool of expert experience

[0176] Used in the expert experience pool Extract expert experience, use the expert experience to calculate the loss value according to the error formula, use the loss value as the Q value and use the actor network update formula to pre-train the actor network;

[0177] Output action a = π from the actor network φ (s) is input into the PID control module, which outputs torque to the environment to obtain the next state s′. The reward RM is obtained through the PID discounted reward function, and the current state transition tuple (s, a, r, s′) is stored as experience in the experience pool. In , if s′ is the end state of this round, reset the environment state;

[0178] When the number of experiences in the experience pool meets the number for updating, N batch samples are extracted for training update;

[0179] Use TD3's target strategy to output the target action formula and Q value solution formula to calculate the target action and Q value, where Q value is y(r, s′);

[0180] Update the critic network using the calculated Q value:

[0181]

[0182] When the number of updates of the critic network reaches a preset fixed number, the actor network is updated using the actor network update formula;

[0183] Soft update the target network of the two critic networks and the actor network:

[0184] θ i ′←αθ i +(1-α)θ i '

[0185] φ'←αφ+(1-α)φ′

[0186] Among them, α is a soft update weight not greater than 1.

[0187] See also Figure 4 As shown, it is a flow chart of the steps of controlling the updating of the DDR intelligent body by the updating module according to an embodiment of the present invention. The updating module according to the present invention is used to control the updating of the DDR intelligent body:

[0188] Used to initialize the review network with random network parameters θ1, θ2 and φ and action network π φ , initialize the target network parameters θ′1←θ1,θ′2←θ2,φ′←φ, and initialize the replay experience pool

[0189] Use the actor network to output action a = π φ (s) environment, get the next state s′, get the reward through the DDR discounted reward function, and store the current state transition tuple (s, a, r, s′) as experience in the experience pool In , if s′ is the end state of this round, reset the environment state;

[0190] When the number of experiences in the experience pool meets the number for updating, N batch samples are extracted for training update;

[0191] Use TD3's target strategy to output the target action formula and Q-value solution formula to calculate the target action and Q-value;

[0192] Update the critic network using the calculated Q value:

[0193]

[0194] When the number of updates of the critic network reaches a preset fixed number, the actor network is updated using the actor network update formula;

[0195] Put two c r ti c Network and acto r The target network of the network is soft updated:

[0196] θ′ i ←αθ i +(1-α)θ′ i

[0197] φ'←αφ+(1-α)φ′

[0198] Among them, α is a soft update weight not greater than 1.

[0199] Specifically, the two agents have different control characteristics. Different discounted reward functions and training processes are used to train the two agents in the same environment. During the training process, the behavior cloning module pre-trains the PID agent. Behavior cloning training requires the expert experience of the PID control module. During the training process of the TD3with PID algorithm, only the control parameters of the PID control module need to be output to achieve good control effects in the entire tracking task. Its exploration domain will be smaller than using TD3 to directly output torque to control the robotic arm. Therefore, the convergence speed of the TD3 with PID algorithm is faster than that of using only TD3. According to the easy training characteristics of the TD3 with PID algorithm, this paper first trains TD3 with PID, and then uses the trained expert model for tracking testing. During the tracking process, the state transition tuple formed by the current state, the expert action output by the model, the current reward, and the next state is collected as expert experience. Finally, the expert experience pool composed of expert experience is used to pre-train the PID agent with behavior cloning (BC). Although the use of BC technology makes the training process complicated, the use of BC reduces the overall training difficulty of MDRL and solves the problem that MDRL itself is very difficult to train. Through BC, the PID agent is pre-trained using the expert experience of the PID control module. Before formal training, a training set consisting of the expert experience of the PID controller is used. The gap between the decision-making of the PID agent and the expert decision-making under the same state is used as a loss to pre-update the PID agent, so that the agent has the ability to approach the expert decision-making level. The pre-strategy formed can output more appropriate PID parameters at the beginning of training to increase the experience of positive feedback, reduce the complexity of training, and solve the problem of difficult or even non-convergence of multi-agent systems.

[0200] Specifically, for the robot trajectory tracking problem with nonlinear interference and changeable environment, this solution combines behavior cloning (BC) with multi-agent deep reinforcement learning (MDRL) to control the robot. The multi-agent control algorithm includes a proportional-integral-derivative agent (PID agent) based on the Twin Delayed Deep Deterministic policy gradient algorithm (TD3), which is the PID agent, and a direct deep reinforcement learning agent (DDR agent) that directly outputs torque using a deep reinforcement learning strategy, which is the DDR agent. The two agents jointly output torque to control the same robot, and two reward functions are used to optimize the policy networks of the two agents. The PID agent is used to output the parameters of the PID controller, which then outputs the torque to control the robot, so that the robot can track other trajectories different from the training trajectory. The DDR agent directly outputs torque to increase the anti-interference ability of the controller. In order to overcome the difficulty of high complexity in the multi-agent training process, this paper introduces behavioral cloning technology in training, and uses the expert experience of the PID control module to pre-train the PID agent to form a pre-strategy. At the beginning of training, more suitable PID parameters can be output to increase effective experience to accelerate the reward convergence of the training process. This scheme has the best tracking effect in the face of random interference environments and tracking trajectories different from the training trajectories.

[0201] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0202] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A robot arm continuous trajectory tracking control system based on behavior cloning, characterized in that: include: A robot arm feature expression module, which is used to model the robot arm through dynamic equations; A proportional agent module that determines the output target action through a non-online actor-critic DRL algorithm; The behavior cloning module is used to determine the target action output by the actor-critic DRL algorithm and the loss value shown by expert experience based on the error formula; A multi-agent module, which includes a PID agent for outputting PID control parameters and a DDR agent for outputting control torque, and the two agents control the same robotic arm; A PID control module, which is used to output torque based on the PID parameters output by the multi-agent module to control the robotic arm so that the robotic arm tracks a trajectory other than the training trajectory; A reward determination module, which is used to determine the reward value of the corresponding agent through the corresponding reward function of each agent; The update module is used to control the update of each agent.

2. The robot arm continuous trajectory tracking control system based on behavior cloning according to claim 1 is characterized in that: The robot arm feature expression module is used to determine the dynamic equation, which is: Among them, q is the joint angle variable of the robot arm, is the joint angular velocity, τ is the torque of the robot, M(q) is an n×n matrix, and the n×n matrix is ​​determined as the inertia matrix. is the centrifugal force and Coriolis force matrix, G(q)∈Rn is the gravity matrix, n is the total number of joints of the robot arm, is a nonlinear interference, is the torque.

3. The robot arm continuous trajectory tracking control system based on behavior cloning according to claim 2 is characterized in that: The mechanical arm feature expression module is used to determine the torque; is the torque of the i-th joint of the robot, i = 1, 2, 3, ... n, n is the total number of joints of the robot; The robot arm feature expression module is used to determine the torque constraint, which is: Among them, q min is the preset minimum torque, q max The maximum value of the preset torque.

4. The robot arm continuous trajectory tracking control system based on behavior cloning according to claim 3 is characterized in that: The robot arm feature expression module is used to determine nonlinear interference; in, Used to represent nonlinear interference, is the interference force on the i-th joint, i = 1, 2, 3, ... n, n is the total number of joints of the robot, and r is a random number; The robot arm feature expression module is used to determine the constraint conditions of the interference force on the i-th joint; Wherein, Radn∈[0,1] is a random number, x is a preset trigger value, x∈[0,1]; when the random number Radn is greater than the preset trigger value, interference is triggered; G(r) is used to determine the unknown function of the interference torque in the case of interference triggering, G(r)∈(-K,K).

5. The robot arm continuous trajectory tracking control system based on behavior cloning according to claim 4 is characterized in that: The proportional agent module is used to determine the definition tuple; Define the tuple as D = (S, A, π, Q, R, γ), Among them, S represents the state space, A represents the action space, π represents the actor neural network to be trained, Q represents the critic neural network to be trained, R represents the reward space, and γ is the discount factor.

6. The robot arm continuous trajectory tracking control system based on behavior cloning according to claim 5 is characterized in that: The proportional agent module is used to determine the formula of the target action output by the target strategy of TD3; a′(s′)=clip(π φ′ (s′)+clip(∈,-c,c),a low ,a high ) Among them, a' is the target action, s' is the next state generated by the current action acting on the current state, π φ′ (s′) is the target strategy π φ′ The target action generated by the next action s' is ∈, which is the added random noise. The value range of ∈ is (-c, c), (a low , a high ) is the size range of the target action output by the target strategy, and clip(a, b, c) means truncating a with the range (b, c); TD3 is a twin delayed deep deterministic policy gradient algorithm. TD3 is a non-online actor-critic DRL algorithm designed specifically for continuous action space tasks. The proportional agent module is used to determine Q, including: The actor-criticDRL algorithm includes two different Q networks for estimating actions, and the proportional agent module determines the minimum Q value of the two Q values ​​as the update target to suppress continuous overestimation; In the actor-criticDRL algorithm, the Q value is determined based on the Q value solution formula. The Q value solution formula is: Among them, r is the return value given by the environment, γ is the discount factor, It is a critic neural network composed of double Q networks, Q θ′i is a critic neural network with network parameters θ′ containing two Q networks; The proportional agent module is used to determine π, including: Used to determine the update formula of the actor network. The update formula of the actor network is: in, For the experience pool, is the policy gradient of the actor network during parameter update, and φ is the network parameter.

7. The robot arm continuous trajectory tracking control system based on behavior cloning according to claim 6, characterized in that: The error formula is, Among them (s i ,a i ) is the state and expert action of a round, N is the number of samples drawn from the dataset, f(s i ) is the predicted action of the network to be trained for the current state, and L(A) is the loss value.

8. The robot arm continuous trajectory tracking control system based on behavior cloning according to claim 7 is characterized in that: The multi-agent module is used to determine the PID parameters through the PID controller expression. The PID controller expression is: in, is the torque output by the PID agent, P is the output of the proportional term, Kp is the proportional coefficient, e(t) represents the error at time t, i represents the output of the integral term, and K i is the integration coefficient, ∫0 t e(t)d t represents the integral of the error from the beginning to the current moment, D is the output of the differential term, K d is the differential coefficient, is the rate of change of error, and the three parameters Kp, K i and K d .

9. The robot arm continuous trajectory tracking control system based on behavior cloning according to claim 8, characterized in that: The PID reward function is, r p (t)=-|v e ·e(t)| Among them, r p (t) is the instantaneous reward of the PID agent at time t, e(t) represents the tracking error, e(t) = q(t) - q d (t), q(t) represents the position of the robot arm, q d (t) represents the desired position of the robot arm, v e is the coefficient of tracking error; The reward determination module determines the expected discounted reward of the PID agent through the PID discounted reward function, and the PID discounted reward function is: Among them, R P (t) is the expected discounted reward of the PID agent after time step t; The DDR reward function is, r d (t)=-(|v e ·e(t)|)+v ed ·e d (t) Among them, r d (t) is the instantaneous reward of the DDR agent at time t, e d (t) represents the change in error, e d (t)=|e(t-1)|-|e(t)|, e dv is the parameter of the error variation; The reward determination module determines the expected discounted reward of the DDR agent through the DDR discounted reward function as follows: γ∈[0, 1] represents the discount factor.

10. The robot arm continuous trajectory tracking control system based on behavior cloning according to claim 9, characterized in that: The update module is used to control the PID agent update: Used to initialize the review network with random network parameters θ1, θ2 and φ and action network π φ , initialize the target network parameters θ′1←θ1,θ′2←θ2,φ′←φ, initialize the replay experience pool And the pool of expert experience Used in the expert experience pool Extract expert experience, use the expert experience to calculate the loss value according to the error formula, use the loss value as the Q value and use the actor network update formula to pre-train the actor network; Output action a = π from the actor network φ (s) is input into the PID control module, which outputs torque to the environment to obtain the next state s′. The reward RM is obtained through the PID discounted reward function, and the current state transition tuple (s, a, r, s′) is stored as experience in the experience pool. In , if s′ is the end state of this round, reset the environment state; When the number of experiences in the experience pool meets the number for updating, N batch samples are extracted for training update; Use TD3's target strategy to output the target action formula and Q value solution formula to calculate the target action and Q value, where Q value is y(r, s′); Update the critic network using the calculated Q value: When the number of updates of the critic network reaches a preset fixed number, the actor network is updated using the actor network update formula; Soft update the target network of the two critic networks and the actor network: i i ′←αθ i +(1-α)θ i ′ φ′←αφ+(1-α)φ′ Among them, α is a soft update weight not greater than 1; The update module is used to control the DDR agent update: Used to initialize the review network with random network parameters θ1, θ2 and φ and action network π φ , initialize the target network parameters θ′1←θ1,θ′2←θ2,φ′←φ, initialize the replay experience pool Use the actor network to output action a = π φ (s) environment, get the next state s′, get the reward through the DDR discounted reward function, and store the current state transition tuple (s, a, r, s′) as experience in the experience pool In , if s′ is the end state of this round, reset the environment state; When the number of experiences in the experience pool meets the number for updating, N batch samples are extracted for training update; Use TD3's target strategy to output the target action formula and Q-value solution formula to calculate the target action and Q-value; Update the critic network using the calculated Q value: When the number of updates of the critic network reaches a preset fixed number, the actor network is updated using the actor network update formula; Soft update the target network of the two critic networks and the actor network: i i ′←αθ i +(1-α)θ i ′ φ′←αφ+(1-α)φ′ Among them, α is a soft update weight not greater than 1.

Images