A behavior cloning-based robot arm continuous trajectory tracking control system

Abstract

The present application relates to the technical field of mechanical arm control, and particularly relates to a mechanical arm continuous trajectory tracking control system based on behavior cloning, which comprises a mechanical arm feature expression module, a proportional intelligent agent module, a behavior cloning module, a multi-agent module, a PID control module, a reward determination module and an updating module, a method of multi-agent deep reinforcement learning is used to control the mechanical arm in combination with behavior cloning, two intelligent agents of a PID intelligent agent and a DDR intelligent agent jointly output torque to control the same mechanical arm, and two reward functions are used to optimize the strategy networks of the two intelligent agents, the PID intelligent agent is used to output the parameters of a PID controller, and then the PID controller outputs torque to control the mechanical arm, so that the mechanical arm can track other trajectories different from the training trajectory, the DDR intelligent agent directly outputs torque to increase the anti-interference performance of the controller, has anti-interference capability in the face of strong random interference and a variable environment, and improves the operation accuracy of the mechanical arm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical arm control, and in particular to a mechanical arm continuous trajectory tracking control system based on behavior cloning. BACKGROUND

[0002] In recent years, with the popularity of intelligent manufacturing, robot systems have developed rapidly and have been widely applied to many industries such as photovoltaic and automobile. Robot path planning and trajectory tracking are one of the key technologies of robot automation and intelligence. At present, the research on mechanical arm control has achieved many results. With the increase of control complexity, the uncertainty and system disturbance of the robot should also be paid special attention in the control design.

[0003] It is worth noting that the dynamic model of the robot, especially in complex industrial applications, is often time-varying or uncertain. In order to adapt to this uncertainty, in recent years, neural networks have been widely used to solve the control design problem of uncertain nonlinear dynamic robot systems.

[0004] Chinese patent publication No. CN116512233B discloses a mechanical arm and a control system for the mechanical arm. The device can rotate through the setting of the first motor. The mounting column can be lifted while rotating through the setting of the rotating plate, the telescopic rod and the first circular sleeve. The mechanical arm and the control system for the mechanical arm include a base, a first motor, a support column fixedly connected to the top end of the base, an installation slot formed in the top end of the support column, an installation column slidably connected in the installation slot, an output end of the first motor fixedly connected to the bottom end of the installation column, a rotating slot formed in the top end of the support column, a rotating plate rotatably connected in the rotating slot, a plurality of first circular sleeves fixedly connected to the top end of the rotating plate, a telescopic rod slidably connected in each of the first circular sleeves, and a fixing ring fixedly connected to the top end of the telescopic rod. It can be seen that the above technical solution has the following problems: in the face of strong random disturbance and variable environment, it does not have anti-interference ability, which affects the operation accuracy of the mechanical arm. SUMMARY

[0005] Therefore, the present application provides a mechanical arm continuous trajectory tracking control system based on behavior cloning to overcome the problem that in the prior art, in the face of strong random disturbance and variable environment, it does not have anti-interference ability, which affects the operation accuracy of the mechanical arm.

[0006] To achieve the above purpose, the present application provides a mechanical arm continuous trajectory tracking control system based on behavior cloning, which comprises:

[0007] A mechanical arm feature expression module is used to model the mechanical arm through a dynamic equation;

[0008] a proportion intelligent agent module configured to determine an output target action by an off-line actor-critic DRL algorithm;

[0009] a behavior cloning module configured to determine a loss value of the target action output by the actor-critic DRL algorithm and the expert experience demonstration based on an error formula;

[0010] a multi-agent module including a PID intelligent agent configured to output a PID control parameter and a DDR intelligent agent configured to output a control torque, both of which control the same robot arm;

[0011] a PID control module configured to output a torque based on the PID parameter output by the multi-agent module to control the robot arm to track a trajectory different from the training trajectory;

[0012] a reward determination module configured to determine a reward value of each agent by a corresponding reward function of the agent;

[0013] an update module configured to control each agent to update.

[0014] Further, the robot feature expression module is configured to determine a dynamic equation, the dynamic equation being:

[0015]

[0016] wherein q is a joint angle variable of the robot, is an angular velocity of the joint, τ is a torque of the robot, M(q) is an n*n matrix, the n*n matrix is determined as an inertia matrix, is a centrifugal force and Coriolis force matrix, G(q) ∈ Rn is a gravity matrix, n is a total number of joints of the robot, is a nonlinear disturbance, is a torque.

[0017] Further, the robot feature expression module is configured to determine the torque;

[0018]

[0019] is the torque of the i-th joint of the robot, i = 1, 2, 3, …, n, n is a total number of joints of the robot;

[0020] The robot feature expression module is configured to determine a constraint of the torque, the constraint of the torque being:

[0021]

[0022] wherein q min is a preset minimum value of the torque, q maxThis is the preset maximum torque value.

[0023] Furthermore, the robotic arm feature representation module is used to determine nonlinear disturbances;

[0024]

[0025] in, Used to represent nonlinear disturbances, Let r be the disturbance force on the i-th joint, i = 1, 2, 3, ..., n, where n is the total number of joints in the robotic arm, and r is a random number.

[0026] The robotic arm feature expression module is used to determine the constraint conditions of the disturbance force on the i-th joint;

[0027]

[0028] Where Radn∈[0,1] is a random number, x is a preset trigger value, x∈[0,1]; when the random number Radn is greater than the preset trigger value, interference is triggered; G(r) is used to determine the unknown function of the interference torque in the case of interference triggering, G(r)∈(-K,K).

[0029] Furthermore, the proportional agent module is used to determine the definition tuple;

[0030] Define the tuple as D = (S, A, π, Q, R, γ).

[0031] Where S represents the state space, A represents the action space, π represents the actor neural network to be trained, Q represents the critic neural network to be trained, R represents the reward space, and γ is the discount factor.

[0032] Furthermore, the proportional agent module is used to determine the formula for the target action output by the target policy of TD3;

[0033] a'(s′)=clip(π φ′ (s′)+clip(∈,-c,c),a low ,a high )

[0034] Where a' is the target action, s' is the next state generated by the current action acting on the current state, and π φ′ (s′) represents the target policy π. φ′ Based on the target action generated by the next action s', ∈ represents added random noise, and the range of ∈ is (-c, c), (a low a highThe target policy outputs the range of the target action, and clip(a, b, c) means to truncate and clip a using the range (b, c).

[0035] TD3 is a twin-delay deep deterministic policy gradient algorithm. TD3 is an offline actor-critic DRL algorithm specifically designed for continuous action space tasks.

[0036] The proportional agent module is used to determine Q, including:

[0037] The actor-critic DRL algorithm includes two different Q-networks for estimating actions. The proportional agent module determines the smaller of the two Q-values ​​as the update target to suppress continuous overestimation.

[0038] In the actor-critic DRL algorithm, the Q-value is determined based on the Q-value calculation formula, which is:

[0039]

[0040] Where r is the reward value given by the environment, and γ is the discount factor. It is a critic neural network composed of two Q networks, Q θ′i A critic neural network with two Q-networks and network parameters θ′;

[0041] The proportional agent module is used to determine π, including:

[0042] This is used to determine the update formula for the actor network. The update formula for the actor network is:

[0043]

[0044] in, For experience pool, φ represents the policy gradient of the actor network during parameter updates, and φ represents the network parameters.

[0045] Furthermore, the error formula is as follows:

[0046]

[0047] Where (s) i ,a i ) represents the state and expert action of a round, N is the number of samples drawn from the dataset, and f(s) i L(A) is the predicted action of the network to be trained for the current state, and L(A) is the loss value.

[0048] Furthermore, the multi-agent module is used to determine the PID parameters through the PID controller expression, which is:

[0049]

[0050] in, The torque output by the PID agent is P, where P is the proportional term output, Kp is the proportional coefficient, e(t) represents the error at time t, i represents the integral term output, and K... i It is the integral coefficient, ∫0 t e(t)d t K represents the integral of the error from the beginning to the current time, D is the output of the differential term, and K is the integral of the error from the beginning to the current time. d These are differential coefficients. It is the rate of change of error, which is output by TD3 as the three parameters Kp, K... i and K d .

[0051] Furthermore, the PID reward function is,

[0052] r p (t)=-|v e ·e(t)|

[0053] Where, r p (t) represents the instantaneous reward of the PID agent at time t, and e(t) represents the tracking error, e(t) = q(t) - q d (t), q(t) represents the position of the robotic arm, qd(t) represents the desired position of the robotic arm, and ve is the coefficient of the tracking error;

[0054] The reward determination module determines the expected discounted reward for the PID agent using a PID discounted reward function, which is:

[0055]

[0056] Among them, R P (t) represents the expected discounted reward for the PID agent after time step t;

[0057] The DDR reward function is,

[0058] r d (t)=-(|v e ·e(t)|)+v ed ·e d (t)

[0059] Where, r d (t) represents the instantaneous reward of the DDR agent at time t, e d (t) represents the change in error, ed (t)=|e(t-1)|-|e(t)|,e dv It is a parameter representing the amount of error change;

[0060] The reward determination module determines the expected discount reward for the DDR agent using the DDR discount reward function.

[0061]

[0062] γ∈[0,1] represents the discount factor.

[0063] Furthermore, the update module is used to control the update of the PID agent:

[0064] Used to initialize the comment network with random network parameters θ1, θ2, and φ. and Action Network π φ Initialize the target network parameters θ′1←θ1, θ′2←θ2, φ′←φ, and initialize the replay experience pool. and expert experience pool

[0065] Used in the expert experience pool The expert experience is extracted and used to calculate the loss value according to the error formula. The loss value is used as the Q value and the actor network is pre-trained using the update formula of the actor network.

[0066] The actor network outputs the action a = π. φ (s) Input the PID control module, the PID control module outputs torque to the environment, obtains the next state s′, obtains the reward RM through the PID discount reward function, and stores the current state transition tuple (s, a, r, s′) as experience in the experience pool. If s′ is the end state of this round, then reset the environment state;

[0067] If the amount of experience in the experience pool is sufficient for the number of updates, N batch samples are drawn for training and updating.

[0068] The target action and Q value are calculated using the target policy output formula and Q value solution formula of TD3. The Q value is y(r, s′).

[0069] Update the critic network using the calculated Q-value:

[0070]

[0071] When the number of updates to the critic network reaches a preset fixed number, the actor network is updated using the update formula of the actor network.

[0072] Perform a soft update on the target network of the two critic networks and the actor network:

[0073] θ′ i ←αθ i +(1-α)θ′ i

[0074] φ′←αφ+(1-α)φ′

[0075] Where α is a soft update weight that is no greater than 1;

[0076] The update module is used to control the update of the DDR agent:

[0077] Used to initialize the comment network with random network parameters θ1, θ2, and φ. and Action Network π φ Initialize the target network parameters θ′1←θ1, θ′2←θ2, φ′←φ, and initialize the replay experience pool.

[0078] Use the actor network to output the action a = π φ Given the environment (s), obtain the next state s′, receive the reward through the DDR discount reward function, and store the current state transition tuple (s, a, r, s′) as experience in the experience pool. If s′ is the end state of this round, then reset the environment state;

[0079] If the amount of experience in the experience pool is sufficient for the number of updates, N batch samples are drawn for training and updating.

[0080] The target action and Q value are calculated using the target policy output formula and Q value solution formula of TD3.

[0081] Update c using the calculated Q value. r iti c network:

[0082]

[0083] When the number of updates to the critic network reaches a preset fixed number, the actor network is updated using the update formula of the actor network.

[0084] Two c r iti c Network and Acto r The target network performs a soft update:

[0085] θ′ i ←αθ i +(1-α)θi ′

[0086] φ′←αφ+(1-α)φ′

[0087] Where α is a soft update weight that is no greater than 1.

[0088] Compared with the prior art, the beneficial effect of the present invention is that the robotic arm feature expression module can accurately characterize the characteristics of the robotic arm. Compared with other algorithms, the robotic arm feature expression module does not need to consider the influence of constraint forces, thus saving modeling time.

[0089] Furthermore, behavioral cloning can be used for model pre-training and to reduce the difficulty of reward convergence in models. The learning process of behavioral cloning involves learning a mapping relationship from states to expert behavior, which is usually expressed using neural networks. The training objective of behavioral cloning is to enable the agent to perform the same actions when encountering the same situation as the expert. Therefore, the training process of behavioral cloning can be understood as reducing the difference between the actions output by the neural network and the behaviors demonstrated by expert experience. Introducing behavioral cloning improves the training speed of neural networks.

[0090] Furthermore, TD3 effectively solves the control problem of robotic arm trajectory tracking and has good anti-interference capabilities. By introducing a PID control module, a TD3 with PID algorithm was designed. The TD3 with PID algorithm uses a neural network to output PID controller parameters, which are then output by the PID control module to control the robotic arm for trajectory tracking. This improves the tracking generality of the algorithm and overcomes the problem that neural networks, due to their task-fitting characteristics, perform poorly when tracking trajectories different from the training trajectory. For trajectory tracking control tasks of multi-joint robotic arms in environments with nonlinear disturbances and variable conditions, this solution introduces multi-agent deep reinforcement learning (MDRL) to combine the advantages of both algorithms. This solves the problem that the PID algorithm has weak anti-interference capabilities due to its weak response to disturbances caused by the need for the PID control module to output parameters first and then output torque for control. This improves the tracking generality and anti-interference capabilities of the TD3 with PID algorithm.

[0091] Furthermore, by introducing behavioral cloning, the problem of the MDRL algorithm failing to converge or even lacking a positive convergence trend during training experiments is solved, thus improving the training speed. For complex tasks, functional decomposition is performed. The PID agent combined with the PID control module enhances the algorithm's generality, while the DDR agent enhances the algorithm's anti-interference ability. Through the above solutions, the robotic arm can adapt to trajectory tracking tasks in different environments and has different functions to adapt to different environments. Attached Figure Description

[0092] Figure 1 This is a block diagram of the continuous trajectory tracking control system for a robotic arm based on behavior cloning, according to an embodiment of the present invention.

[0093] Figure 2 This is a logic diagram for determining the interference force on the i-th joint in the feature expression module of the robotic arm in this embodiment of the invention.

[0094] Figure 3 This is a block diagram of the multi-agent module and the module block diagram of the multi-agent module jointly outputting torque in an embodiment of the present invention;

[0095] Figure 4 This is a flowchart illustrating the steps of the update module controlling the DDR smart agent update in an embodiment of the present invention. Detailed Implementation

[0096] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0097] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0098] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0099] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0100] Please refer to this invention. Figure 1 The diagram shown is a block diagram of a continuous trajectory tracking control system for a robotic arm based on behavior cloning, according to an embodiment of the present invention. The system includes:

[0101] The robotic arm feature representation module is used to model the robotic arm using dynamic equations;

[0102] The proportional agent module is used to determine the output target action using an offline actor-critic DRL algorithm;

[0103] The behavior cloning module is used to determine the loss value between the target action output by the actor-critic DRL algorithm and the expert experience demonstration based on the error formula;

[0104] The multi-agent module includes a PID agent for outputting PID control parameters and a DDR agent for outputting control torque, with the two agents controlling the same robotic arm.

[0105] The PID control module is used to output torque based on the PID parameters output by the multi-agent module to control the robotic arm, so that the robotic arm can track other trajectories that are different from the training trajectory.

[0106] The reward determination module is used to determine the reward value of each agent through the corresponding reward function of each agent;

[0107] The update module is used to control the updates of each agent.

[0108] Specifically, the robotic arm feature representation module is used to determine the dynamic equation, which is:

[0109]

[0110] Where q is the joint angle variable of the robotic arm. Let ω be the joint angular velocity, τ be the torque of the robotic arm, and M(q) be an n×n matrix. The n×n matrix is ​​then defined as the inertia matrix. Let G(q)∈Rn be the centrifugal force and Coriolis force matrix, and let n be the total number of joints in the robotic arm. It is a nonlinear interference. It represents torque.

[0111] Specifically, the robotic arm feature expression module is used to determine the torque;

[0112]

[0113] Let be the torque of the i-th joint of the robotic arm, where i = 1, 2, 3, ..., n, and n is the total number of joints in the robotic arm;

[0114] The robotic arm feature representation module is used to determine the torque constraints, which are as follows:

[0115]

[0116] Where, q minq is the preset minimum torque. max This is the preset maximum torque value.

[0117] The robotic arm feature representation module is used to determine nonlinear interference;

[0118]

[0119] in, Used to represent nonlinear disturbances, Let r be the disturbance force on the i-th joint, i = 1, 2, 3, ..., n, where n is the total number of joints in the robotic arm, and r is a random number.

[0120] Please see Figure 2 As shown, it is a logic determination diagram for the robotic arm feature expression module to determine the interference force on the i-th joint in an embodiment of the present invention. The robotic arm feature expression module of the present invention is used to determine the constraint conditions of the interference force on the i-th joint.

[0121]

[0122] Where Radn∈[0,1] is a random number, x is a preset trigger value, x∈[0,1]; when the random number Radn is greater than the preset trigger value, interference is triggered; G(r) is used to determine the unknown function of the interference torque in the case of interference triggering, G(r)∈(-K,K).

[0123] Specifically, the magnitude of G(r) corresponds to the random number r, and the magnitude of G(r) is independent of the force exerted by the robotic arm, belonging to external interference. The magnitude range of the interference can be determined by setting K, and the probability of the interference occurring can be determined by setting x.

[0124] Specifically, after determining the number of joints, all of the above matrices can be derived from the actual model.

[0125] Specifically, the robotic arm feature expression module can accurately characterize the characteristics of the robotic arm. Compared with other algorithms, the robotic arm feature expression module does not need to consider the influence of constraints, thus saving modeling time.

[0126] Specifically, by identifying nonlinear disturbances, and given an environment with random nonlinear disturbances and a target tracking trajectory, a multi-agent deep reinforcement learning controller is designed for the robotic arm. This controller enables the robotic arm to track the target trajectory in a disturbed and variable environment without requiring prior knowledge of the robotic arm's dynamic model. This paper focuses on robotic arm trajectory tracking tasks in disturbed and variable environments. The algorithm innovatively combines two agents with different output torque methods and is successfully trained using BC technology, increasing the control generality and anti-interference capability during the tracking task. In practical control, the controller's control generality can enhance its application prospects.And the ability to cope with special cases, the anti-interference ability can increase The control stability of the controller is improved to adapt to more extreme application scenarios. Therefore, this study has important reference value for solutions to spraying, assembly and other tasks in harsh environments such as at sea and in deserts, as well as in severe weather.

[0127] Specifically, the proportional agent module is used to determine the definition tuple;

[0128] Define the tuple as D = (S, A, π, Q, R, γ).

[0129] Where S represents the state space, A represents the action space, π represents the actor neural network to be trained, Q represents the critic neural network to be trained, R represents the reward space, and γ is the discount factor.

[0130] The proportional agent module is used to determine the formula for the target action output by the target policy of TD3;

[0131] a'(s′)=clip(π φ′ (s′)+clip(∈,-c,c),a low, a high )

[0132] Where a' is the target action, s' is the next state generated by the current action acting on the current state, and π φ′ (s′) represents the target policy π. φ′ Based on the target action generated by the next action s', ∈ represents added random noise, and the range of ∈ is (-c, c), (a low a high The target policy outputs the range of the target action, and clip(a, b, c) means to truncate and clip a using the range (b, c).

[0133] TD3 is a twin-delay deep deterministic policy gradient algorithm. TD3 is an offline actor-critic DRL algorithm specifically designed for continuous action space tasks.

[0134] Specifically, the proportional agent module is used to determine Q, including:

[0135] The actor-critic DRL algorithm includes two different Q-networks for estimating actions. The proportional agent module determines the smaller of the two Q-values ​​as the update target to suppress continuous overestimation.

[0136] In the actor-critic DRL algorithm, the Q-value is determined based on the Q-value calculation formula, which is:

[0137]

[0138] Where r is the reward value given by the environment, and γ is the discount factor. It is a critic neural network composed of two Q-networks, where Q refers to the evaluation network with network parameters θ′. Since Q is a TD3 network (two Q-networks), there are two evaluation networks. Here, i refers to the i-th network, so Q θ′i This refers to a critic network system with two Q-networks and network parameters θ′. The critic network, Q-network, and evaluation network all refer to a neural network used to evaluate the value of an action; they are simply called differently in different contexts. Here, the two critic networks form a criticpolicy.

[0139] Specifically, DRL stands for Deep Reinforcement Learning.

[0140] Specifically, the proportional agent module is used to determine π, including:

[0141] This is used to determine the update formula for the actor network. The update formula for the actor network is:

[0142]

[0143] in, For the experience pool, the formula here is to calculate the policy gradient ▽φJ(φ) of the Actor network when updating parameters, which is calculated for the Actor network parameters φ.

[0144] Specifically, the actor is used to continuously update gradients in order to determine the maximum Q-value.

[0145] Specifically, when the critic network is unstable, its output Q-value also becomes unstable, which in turn leads to unstable updates in the actor network. Therefore, the critic network is updated multiple times until the output Q-value is accurate before updating the actor network, thus making the training more stable.

[0146] Specifically, the error formula is as follows:

[0147]

[0148] Where (s) i ,a i ) represents the state and expert action of a round, N is the number of samples drawn from the dataset, and f(s) i L(A) is the predicted action of the network to be trained for the current state, and L(A) is the loss value.

[0149] Specifically, behavior cloning can be used for model pre-training and to reduce the difficulty of reward convergence in models. The learning process of behavior cloning involves learning a mapping relationship from states to expert behavior, which is usually expressed using neural networks. The training objective of behavior cloning is to enable the agent to perform the same actions when encountering the same situation as the expert. Therefore, the training process of behavior cloning can be understood as reducing the difference between the actions output by the neural network and the behaviors demonstrated by the expert's experience.

[0150] Please see Figure 3 The diagram shown is a block diagram of the multi-agent module and the multi-agent module jointly outputting torque according to an embodiment of the present invention. The multi-agent module of the present invention includes a PID agent for outputting PID control parameters and a DDR agent for outputting control torque. The two agents control the same robotic arm. The PID control module is used to determine the PID parameters and output torque based on the PID parameters to control the robotic arm, so that the robotic arm tracks other trajectories different from the training trajectory.

[0151] Specifically, the multi-agent module is used to determine the PID parameters through a PID controller expression, the PID controller expression being:

[0152]

[0153] Where P is the output of the proportional term, Kp is the proportionality coefficient, e(t) represents the error at time t, i represents the output of the integral term, and K... i It is the integral coefficient, ∫0 t e (t)d t K represents the integral of the error from the beginning to the current time, D is the output of the differential term, and K is the integral of the error from the beginning to the current time. d These are differential coefficients. It is the rate of change of error, which is output by TD3 as the three parameters Kp, K... i And Kd, This refers to the torque output by the PID controller. Since the torque output by the PID agent in the MDRL controller is τ1, the subscript 1 is added here. Also, this is just an explanation of how the PID control module outputs torque, not the actual torque output by the PID agent, so a horizontal line is added above to distinguish it.

[0154] Specifically, PID stands for Proportional Integral Derivative (PID), which is a feedback control algorithm.

[0155] Specifically, the PID agent only needs to find the PID parameters suitable for the current robotic arm to complete the tracking task, so its training speed is relatively fast, but its anti-interference effect still needs to be compensated by the DDR agent.

[0156] Specifically, the PID reward function is:

[0157] r p (t)=-|v e ·e(t)|

[0158] Where, r p (t) represents the instantaneous reward of the PID agent at time t, and e(t) represents the tracking error, e(t) = q(t) - q d (t), q(t) represents the position of the robotic arm, qd(t) represents the desired position of the robotic arm, and ve is the coefficient of the tracking error;

[0159] Specifically, the reward determination module determines the expected discounted reward for the PID agent using a PID discounted reward function, which is:

[0160]

[0161] Among them, R P (t) represents the expected discounted reward for the PID agent after time step t, and the reward sequence from the environment after time step t is r. t+1 r t+2 r t+3 …This is just a demonstration using a formula and no deadline is set, so there is no deadline value following it;

[0162] The DDR reward function is,

[0163] r d (t)=-(|v e ·e(t)|)+v ed ·e d (t)

[0164] Where, r d (t) represents the instantaneous reward of the DDR agent at time t, e d (t) represents the change in error, e d (t)=|e(t-1)|-|e(t)|,e dv It is a parameter representing the amount of error change;

[0165] The reward determination module determines the expected discount reward for the DDR agent using the DDR discount reward function.

[0166]

[0167] γ∈[0,1] represents the discount factor.

[0168] Specifically, the PID agent focuses more on the overall trajectory tracking performance, so the reward of the PID agent only uses the trajectory error as the reward value.

[0169] Specifically, using error change as a reward makes the DDR agent more sensitive to tracking stability. The DDR agent directly outputs torque to the robotic arm, thus it can react quickly to disturbances or system fluctuations, improving the system's anti-interference capability and tracking stability. Assuming that after time step t, the environmental...

[0170] Specifically, the reward sequence can be represented as r t+1 r t+2 r t+3 …Then the sum of the weighted instant rewards is the expected discount reward.

[0171] Specifically, the two agents output torque in completely different ways. The PID agent outputs PID control parameters, which are then used by the PID control module to output control torque. Furthermore, the two agents have different priorities during the tracking task. The PID agent focuses more on the overall tracking effect, while the DDR agent prioritizes anti-interference capabilities and tracking stability. Due to the differences in their working logic and priorities, two different reward functions are used to reflect the performance of each agent. While both agents share the same common goal, they are more sensitive to changes in different parameters. These two reward functions differentiate the overall task, forming internal rewards for each agent and enhancing their collaborative capabilities.

[0172] Specifically, TD3 effectively solves the control problem of robotic arm trajectory tracking and has good anti-interference capabilities. By introducing a PID control module, a TD3 with PID algorithm was designed. The TD3 with PID algorithm uses a neural network to output PID controller parameters, which are then output by the PID control module to control the robotic arm for trajectory tracking. This improves the tracking generality of the algorithm and overcomes the problem that neural networks, due to their task-fitting characteristics, perform poorly when tracking trajectories different from the training trajectory. For trajectory tracking control tasks of multi-joint robotic arms in environments with nonlinear disturbances and changes, this solution introduces multi-agent deep reinforcement learning (MDRL) to combine the advantages of both algorithms. This solves the problem that the PID algorithm has weak anti-interference capabilities due to its weak response to disturbances caused by the need for the PID control module to output parameters first and then output torque for control. This improves the tracking generality and anti-interference capabilities of the TD3 with PID algorithm.

[0173] Specifically, by introducing behavioral cloning, the problem of the MDRL algorithm failing to converge or even showing a positive convergence trend during training experiments is solved, thus improving the training speed. For complex tasks, functional decomposition is performed, with the PID agent combined with the PID control module to improve the algorithm's generality and the DDR agent to improve the algorithm's anti-interference ability. Through the above solutions, the robotic arm can adapt to trajectory tracking tasks in different environments and has different functions to adapt to different environments.

[0174] Specifically, the update module is used to control the update of the PID agent:

[0175] Used to initialize the comment network with random network parameters θ1, θ2, and φ. and Action Network π φ Initialize the target network parameters θ′1-θ1, θ′2←θ2, φ′←φ, and initialize the replay experience pool. and expert experience pool

[0176] Used in the expert experience pool The expert experience is extracted and used to calculate the loss value according to the error formula. The loss value is used as the Q value and the actor network is pre-trained using the update formula of the actor network.

[0177] The actor network outputs the action a = π. φ (s) Input the PID control module, the PID control module outputs torque to the environment, obtains the next state s′, obtains the reward RM through the PID discount reward function, and stores the current state transition tuple (s, a, r, s′) as experience in the experience pool. If s′ is the end state of this round, then reset the environment state;

[0178] If the amount of experience in the experience pool is sufficient for the number of updates, N batch samples are drawn for training and updating.

[0179] The target action and Q value are calculated using the target policy output formula and Q value solution formula of TD3. The Q value is y(r, s′).

[0180] Update the critic network using the calculated Q-value:

[0181]

[0182] When the number of updates to the critic network reaches a preset fixed number, the actor network is updated using the update formula of the actor network.

[0183] Perform a soft update on the target network of the two critic networks and the actor network:

[0184] θ i ′←αθ i +(1-α)θ i ′

[0185] φ'←αφ+(1-α)φ′

[0186] Where α is a soft update weight that is no greater than 1.

[0187] Please see Figure 4 The diagram shown is a flowchart illustrating the steps of the update module controlling the DDR agent update in an embodiment of the present invention. The update module of the present invention is used to control the DDR agent update.

[0188] Used to initialize the comment network with random network parameters θ1, θ2, and φ. and Action Network π φ Initialize the target network parameters θ′1←θ1, θ′2←θ2, φ′←φ, and initialize the replay experience pool.

[0189] Use the actor network to output the action a = π φ Given the environment (s), obtain the next state s′, receive the reward through the DDR discount reward function, and store the current state transition tuple (s, a, r, s′) as experience in the experience pool. If s′ is the end state of this round, then reset the environment state;

[0190] If the amount of experience in the experience pool is sufficient for the number of updates, N batch samples are drawn for training and updating.

[0191] The target action and Q value are calculated using the target policy output formula and Q value solution formula of TD3.

[0192] Update the critic network using the calculated Q-value:

[0193]

[0194] When the number of updates to the critic network reaches a preset fixed number, the actor network is updated using the update formula of the actor network.

[0195] Two c r ti c Network and Acto r The target network performs a soft update:

[0196] θ′ i ←αθ i +(1-α)θ′ i

[0197] φ'←αφ+(1-α)φ′

[0198] Where α is a soft update weight that is no greater than 1.

[0199] Specifically, the two agents have different control characteristics. They are trained in the same environment using different discounted reward functions and training procedures. During the training process, the behavior cloning module pre-trains the PID agent. Behavior cloning training requires the expert experience of the PID control module. In the training process of the TD3 with PID algorithm, only the control parameters of the PID control module need to be output to achieve good control effect in the entire tracking task. Its exploration domain is smaller than that of the robotic arm directly outputting torque using TD3. Therefore, the convergence speed of the TD3 with PID algorithm is faster than that of using only TD3. Based on the ease of training of the TD3 with PID algorithm, this paper first trains TD3 with PID, and then uses the trained expert model for tracking test. During the tracking process, the state transition tuple formed by the current state, the expert action output by the model, the current reward, and the next state is collected as expert experience. Finally, the expert experience pool composed of expert experience is used to pre-train the PID agent using behavior cloning (BC). Although the use of BC technology makes the training process more complicated, it reduces the overall training difficulty of MDRL and solves the problem that MDRL itself is very difficult to train. By leveraging the expert experience of the PID control module through BC, the PID agent is pre-trained. Before formal training, a training set composed of expert experience from the PID controller is used. The difference between the PID agent's decisions under the same conditions and the expert's decisions is used as the loss to pre-update the PID agent, thereby enabling the agent to approach the level of expert decision-making. The resulting pre-policy can output more suitable PID parameters in the early stages of training to increase positive feedback experience, reduce training complexity, and solve the problem of convergence difficulties or even non-convergence in multi-agent systems.

[0200] Specifically, for the problem of robotic arm trajectory tracking in environments with nonlinear disturbances and variable conditions, this scheme combines Behavior Cloning (BC) with a Multi-agent Deep Reinforcement Learning (MDRL) approach to control the robotic arm. The MDRL algorithm includes a Proportional-Integral-Derivative (PID) agent (PID agent) based on the Twin Delayed Deep Deterministic policy gradient algorithm (TD3) and a Direct Deep Reinforcement Learning (DDR) agent that directly outputs torque using a deep reinforcement learning policy (DDR agent). Both agents jointly output torque to control the same robotic arm, and two reward functions are used to optimize the policy networks of the two agents. The PID agent outputs parameters for the PID controller, which then outputs torque to control the robotic arm, enabling it to track trajectories different from the training trajectory. The DDR agent directly outputs torque to increase the controller's anti-interference capability. To overcome the high complexity of multi-agent training, this paper introduces behavioral cloning technology during training. By utilizing the expert experience of the PID control module, the PID agent is pre-trained to form a pre-policy that can output appropriate PID parameters in the early stages of training, thereby increasing effective experience and accelerating reward convergence during the training process. This scheme achieves the best tracking performance in the face of random disturbance environments and tracking trajectories that differ from the training trajectory.

[0201] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0202] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A continuous trajectory tracking control system for a robotic arm based on behavior cloning, characterized in that, include: The robotic arm feature representation module is used to model the robotic arm using dynamic equations; The proportional agent module is used to determine the output target action using an offline actor-criticDRL algorithm; The behavior cloning module is used to determine the loss value between the target action output by the actor-criticDRL algorithm and the expert experience demonstration based on the error formula. The multi-agent module includes a PID agent for outputting PID control parameters and a DDR agent for outputting control torque, with the two agents controlling the same robotic arm. The PID control module is used to output torque based on the PID parameters output by the multi-agent module to control the robotic arm, so that the robotic arm can track other trajectories that are different from the training trajectory. The reward determination module is used to determine the reward value of each agent through the corresponding reward function of each agent; The update module is used to control the updates of each agent; The proportional agent module is used to determine the formula for the target action output by the target policy of TD3; a′(s′)=clip(π φ′ (s′)+clip(∈,-c,c),a low ,a high ) Where a' is the target action, s' is the next state generated by the current action acting on the current state, and π φ′ (s′) represents the target policy π. φ′ Based on the target action generated by the next action s', ∈ represents added random noise, and the range of ∈ is (-c, c), (a low ,a high The target policy outputs the size range of the target action, and clip(a,b,c) means to truncate and clip a using the range (b,c). TD3 is a twin-delay deep deterministic policy gradient algorithm. TD3 is an offline actor-critic DRL algorithm specifically designed for continuous action space tasks. The proportional agent module is used to determine Q, including: The actor-criticDRL algorithm includes two different Q-networks for estimating actions. The proportional agent module determines the smaller of the two Q-values ​​as the update target to suppress continuous overestimation. In the actor-criticDRL algorithm, the Q-value is determined based on the Q-value calculation formula, which is: Where r is the reward value given by the environment, and γ is the discount factor. It is a critic neural network composed of two Q networks, Q θ′i A critic neural network with two Q-networks and network parameters θ′; The proportional agent module is used to determine π, including: This is used to determine the update formula for the actor network. The update formula for the actor network is: in, For experience pool, φ represents the policy gradient of the actor network during parameter updates, and φ represents the network parameters.

2. The continuous trajectory tracking control system for a robotic arm based on behavior cloning according to claim 1, characterized in that, The robotic arm feature representation module is used to determine the dynamic equation, which is: Where q is the joint angle variable of the robotic arm. The joint angular velocity, Let M(q) be the torque of the robotic arm, and M(q) be an n×n matrix. Let the n×n matrix be the inertia matrix. Let G(q)∈Rn be the centrifugal force and Coriolis force matrix, and let n be the total number of joints in the robotic arm. It is a nonlinear interference. It represents torque.

3. The continuous trajectory tracking control system for a robotic arm based on behavior cloning according to claim 2, characterized in that, The robotic arm feature expression module is used to determine the torque; Let be the torque of the i-th joint of the robotic arm, where i = 1, 2, 3, ..., n, and n is the total number of joints in the robotic arm; The robotic arm feature representation module is used to determine the torque constraints, which are as follows: Where, q min q is the preset minimum torque. max This is the preset maximum torque value.

4. The continuous trajectory tracking control system for a robotic arm based on behavior cloning according to claim 3, characterized in that, The robotic arm feature representation module is used to determine nonlinear interference; in, Used to represent nonlinear disturbances, Let r be the disturbance force on the i-th joint, i = 1, 2, 3, ..., n, where n is the total number of joints in the robotic arm, and r is a random number. The robotic arm feature expression module is used to determine the constraint conditions of the disturbance force on the i-th joint; Where Radn∈[0,1] is a random number, x is a preset trigger value, x∈[0,1]; when the random number Radn is greater than the preset trigger value, interference is triggered; G(r) is used to determine the unknown function of the interference torque when interference is triggered, G(r)∈(-K,K), K is used to determine the magnitude range of the interference.

5. The continuous trajectory tracking control system for a robotic arm based on behavior cloning according to claim 4, characterized in that, The proportional agent module is used to define tuples; Define the tuple as D = (S, A, π, Q, R, γ). Where S represents the state space, A represents the action space, π represents the actor neural network to be trained, Q represents the critic neural network to be trained, R represents the reward space, and γ is the discount factor.

6. The continuous trajectory tracking control system for a robotic arm based on behavior cloning according to claim 5, characterized in that, The error formula is as follows: Where (s) i ,a i ) represents the state and expert action of a round, N is the number of samples drawn from the dataset, and f(s) i L(A) is the predicted action of the network to be trained for the current state, and L(A) is the loss value.

7. The continuous trajectory tracking control system for a robotic arm based on behavior cloning according to claim 6, characterized in that, The multi-agent module is used to determine the PID parameters through the PID controller expression, which is: in, The torque output by the PID agent is P, where P is the proportional term output, Kp is the proportional coefficient, e(t) represents the error at time t, i represents the integral term output, and K... i It is the integral coefficient. K represents the integral of the error from the beginning to the current time, D is the output of the differential term, and K is the integral of the error from the beginning to the current time. d These are differential coefficients. It is the rate of change of error, which is output by TD3 as the three parameters Kp, K... i and K d .

8. The continuous trajectory tracking control system for a robotic arm based on behavior cloning according to claim 7, characterized in that, The PID reward function is, r p (t)=-|v e ·e(t)| Where, r p (t) represents the instantaneous reward of the PID agent at time t, and e(t) represents the tracking error, e(t) = q(t) - q d q(t) represents the position of the robotic arm, and q d (t) represents the desired position of the robotic arm, v e The coefficient for tracking error; The reward determination module determines the expected discounted reward for the PID agent using a PID discounted reward function, which is: Among them, R P (t) represents the expected discounted reward for the PID agent after time step t; The DDR reward function is, r d (t)=-(|v e ·e(t)|)+v e d·e d (t) Where, r d (t) represents the instantaneous reward of the DDR agent at time t, e d (t) represents the change in error, e d (t)=|e(t-1)|-|e(t)|; The reward determination module determines the expected discount reward for the DDR agent using the DDR discount reward function. γ∈[0,1] represents the discount factor.

9. The continuous trajectory tracking control system for a robotic arm based on behavior cloning according to claim 8, characterized in that, The update module is used to control the update of the PID agent: Used to initialize the comment network with random network parameters θ1, θ2, and φ. and Action Network π φ Initialize the target network parameters θ′1←θ1, θ′2←θ2, φ′←φ, and initialize the replay experience pool. and expert experience pool Used in the expert experience pool The expert experience is extracted and used to calculate the loss value according to the error formula. The loss value is used as the Q value and the actor network is pre-trained using the update formula of the actor network. The actor network outputs the action a = π. φ (s) Input the PID control module, the PID control module outputs torque to the environment, obtains the next state s′, obtains the reward RM through the PID discount reward function, and stores the current state transition tuple (s, a, r, s′) as experience in the experience pool. If s′ is the end state of this round, then reset the environment state; If the amount of experience in the experience pool is sufficient for the number of updates, N batch samples are drawn for training and updating. The formula for outputting the target action and the Q-value calculation formula using the target policy of TD3 are used to calculate the target action and... Q value, Q value is y(r, s′); Update the critic network using the calculated Q-value: When the number of updates to the critic network reaches a preset fixed number, the actor network is updated using the update formula of the actor network. Perform a soft update on the target network of the two critic networks and the actor network: i i ′←αθ i +(1-α)θ i ′ φ'←αφ+(1-α)φ′ Where α is a soft update weight that is no greater than 1; The update module is used to control the update of the DDR agent: Used to initialize the comment network with random network parameters θ1, θ2, and φ. and Action Network π φ Initialize the target network parameters θ′1←θ1, θ′2←θ2, φ′←φ, and initialize the replay experience pool. Use the actor network to output the action a = π φ Given the environment (s), obtain the next state s′, receive the reward through the DDR discount reward function, and store the current state transition tuple (s,a,r,s′) as experience in the experience pool. If s′ is the end state of this round, then reset the environment state; If the amount of experience in the experience pool is sufficient for the number of updates, N batch samples are drawn for training and updating. The target action and Q value are calculated using the target policy output formula and Q value solution formula of TD3. Update the critic network using the calculated Q-value: When the number of updates to the critic network reaches a preset fixed number, the actor network is updated using the update formula of the actor network. Perform a soft update on the target network of the two critic networks and the actor network: i i ′←αθ i +(1-α)θ i ′ φ'←αφ+(1-α)φ′ Where α is a soft update weight that is no greater than 1.

Images