Adaptive variable impedance control methods and devices, electronic devices and storage media

Abstract

This invention provides an adaptive variable impedance control method and device, electronic device, and storage medium. Within the framework of reinforcement learning, this invention employs the Deep Deterministic Policy Gradient (DDPG) algorithm. The DDPG algorithm includes an Actor network and a Critical network. Through continuous learning in practical operations, the DDPG algorithm enables dual robotic arms to adjust their action strategies to maximize cumulative rewards. During the training phase, samples are randomly selected from the experience pool to train the Actor-Critical network to obtain the optimal network structure. Using the trained Actor-Critical network, the dual robotic arms can make optimal action choices based on the current state dataset to adapt to uncertain environments and achieve efficient grasping of target objects. This invention allows dual robotic arms to continuously optimize their behavior through reinforcement learning, thereby better adapting to complex working scenarios.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to an adaptive variable impedance control method and device, electronic device and storage medium. Background Technology

[0002] In industrial production, traditional industrial robots achieve specific tasks in industrial scenarios through position control. However, in applications requiring interaction with the environment, traditional position-based control methods are no longer adequate. In fields such as welding, polishing, and shaft-hole assembly, there is extensive and complex contact with the environment. If the industrial robot only moves along a designated path, any deviation from that path will generate significant environmental contact forces, potentially damaging the workpiece or even the robot itself. With increasing complexity and the need for flexible production processes, existing robots operating at independent workstations can no longer meet the ever-changing manufacturing demands. To adapt to the requirements of task complexity, intelligent operation, and system compliance in unstructured environments, two robots working together and cooperating demonstrate advantages in performing such tasks. In a dual-arm robot, the two arms maintain certain constraints during coordination to complete the coordination task. The basic idea of ​​pure position control is to first plan the trajectory of the target object and obtain the trajectory of the arm ends through the constraint relationship between the target object and the two arms. However, this control method does not consider the forces exerted by the arms on the target object, nor does it consider external disturbances to the target object. Therefore, how to achieve efficient grasping of target objects by dual robotic arms in uncertain and complex scenarios has become an urgent technical problem to be solved. Summary of the Invention

[0003] This invention provides an adaptive variable impedance control method and device, electronic device and storage medium, which enables a robot dual robotic arm to make optimal action selection based on the current state dataset, so as to adapt to uncertain environment and achieve efficient grasping of target objects, and allows the dual robotic arms to continuously optimize their behavior through reinforcement learning, thereby better adapting to complex working scenarios.

[0004] In a first aspect, embodiments of the present invention provide an adaptive variable impedance control method, comprising: An impedance model is constructed for the robot's dual robotic arms when grasping a target object. The internal and external forces acting on the target object are decoupled, and adaptive impedance control is applied to the internal and external forces respectively. The network parameters and experience pool of the impedance model are initialized. The experience pool is used to store the robot's experience tuples in the environment. The impedance model includes an Actor network and a Critic network. The Actor network is used to generate continuous actions, and the Critic network is used to evaluate the quality of the actions and output the corresponding action value function. The robot selects an action from its state space, executes the selected action, stores the experience tuples fed back from the environment into the experience pool, randomly samples a batch of data from the experience pool, calculates the loss of the Critic network and performs backpropagation, calculates the target Q value through the Critic network, updates the parameters of the Actor network to maximize the Q value, and trains iteratively until a preset number of iterations is reached to obtain a trained Actor-Critic network. Using the trained Actor-Critic network, the dual robotic arms perform actions to grasp the target object in a real-world environment.

[0005] In some embodiments, the method further includes: Establish a coordinate system for the dual-manipulator collaborative system. The position and orientation of the target object with respect to the reference coordinate system are solved using the following formula:

[0006] In the formula, This is the transformation matrix of the target object relative to the centroid coordinate system; Let be the 3x3 rotation matrix of the object relative to the coordinate system at its center of mass; This is the 3x1 position matrix of the target object relative to the coordinate system at its centroid; The transformation between the target object's center-of-mass coordinate system and the world coordinate system represents the constraint conditions between the target object and the robotic arm, expressed by the following formula:

[0007] In the formula, This represents the coordinate system at the center of mass of the target object obtained based on the kinematics of the right arm. Homogeneous coordinate transformation relative to the world coordinate system W; This represents the homogeneous coordinate transformation of the target object's center of mass coordinate system relative to the world coordinate system W, obtained based on the kinematic chain of the left arm. , Representing the right arm base coordinate system Left arm base coordinate system Homogeneous coordinate transformation relative to the world coordinate system W; Representing the coordinate system at the end of the right arm Left arm end coordinate system Homogeneous coordinate transformation relative to the corresponding robotic arm base coordinate system; , Representing the centroid coordinate system of the target object respectively Relative to the coordinate system of the right arm end Left arm end coordinate system Homogeneous coordinate transformation; The velocity constraint relationship is analyzed by the following formula, so that the position and velocity of the two arms remain consistent during the movement.

[0008] In the formula, ; Indicates the end effector of the robotic arm Relative to the world coordinate system speed; , Representing the centroid coordinate system of the target object respectively Relative to the world coordinate system Velocity and angular velocity; Represents the coordinate system of the robotic arm's end effector In the world coordinate system The position transformation matrix below; Represents the coordinate system of the robotic arm's end effector Relative to the target object's centroid coordinate system Position transformation matrix; Represents the target object's centroid coordinate system Relative to the world coordinate system The direction rotation matrix.

[0009] In some embodiments, decoupling the internal and external forces acting on the target object includes: Based on Newton's second law and Euler's equations, the state of the dual robotic arms grasping the target object is established, and the following dynamic equations of the target object are derived:

[0010] In the formula , , , , , The inertia matrix at the center of mass of the target object; This represents the vector force exerted on the target object by the two robotic arms. The mass-inertia matrix representing the target object; Represents the linear acceleration and angular acceleration of the target object during its motion; It is represented as the resultant force vector of the Coriolis force, gravity, and centrifugal force of the target object; This represents the vector force exerted on the target object by external disturbance force; transform the above equation into the following equation:

[0011] In the formula Represented as the left and right arms of a dual robotic arm. Represents the capture matrix; This represents the force exerted by the robotic arm on the target object; the grasping matrix is ​​decomposed to obtain the external force formula. and obtaining internal force formula :

[0012]

[0013] In the formula, yes The generalized inverse of a matrix.

[0014] In some embodiments, the equations for the impedance model are as follows:

[0015] In the formula, m is the inertia coefficient and b is the damping coefficient. For adaptive parameters, The error value representing the force, and These are the robot arm's motion speed and acceleration, respectively.

[0016] In some embodiments, the step of randomly sampling a batch of data from the experience pool, calculating the loss of the Critic network and performing backpropagation, calculating the target Q-value through the Critic network, and updating the parameters of the Actor network to maximize the Q-value includes: Based on a deterministic gradient strategy, the Actor network parameters are updated according to the action-value function. The deterministic behavior strategy is as follows:

[0017] in, For the policy function, Given the current state, the action of a deterministic policy is uniquely determined in state s, as shown in the following formula:

[0018] in, These are the policy parameters; during network training, multiple data points N are randomly sampled and used as the deterministic policy. Training data to measure deterministic policies This is expressed as follows:

[0019] in, Indicates from empirical distribution Sample a state , This indicates that the Critic network is for a given state. and actions The value of the entire expectation This means taking the expected value based on the sampled state and action; during training, the sample mean is used instead of the expected value. The Actor network learns the optimal policy by maximizing the output of the Critic network. The parameters of the Actor network are updated using gradient ascent as follows:

[0020] in, Indicates from the Actor strategy Sampling an action , This indicates that the Critic network is for a given state. and actions The value of the gradient term This represents the gradient of the Critic network with respect to the Actor's output action; Based on minimizing the loss function L( Update the parameters of the current value network Q. The loss function to be minimized is as follows:

[0021] Where N is the number of random samples.

[0022] In some embodiments, the step of randomly sampling a batch of data from the experience pool, calculating the loss of the Critic network and performing backpropagation, calculating the target Q-value through the Critic network, and updating the parameters of the Actor network to maximize the Q-value further includes: Based on a deterministic gradient policy, the behavior policy network is calculated according to the following formula. The gradient strategy is used to update the parameters of the Critic network;

[0023] in, and From empirical distribution The states and actions obtained from sampling From empirical distribution The next state obtained from the sampling, y is the output of the Critic network, representing the estimated value of action a in state s, and y is the target Q value. The goal of gradient estimation is to minimize the mean squared error between the Critic network output and the target Q value. This mean squared error is estimated using the sampled state, action, and next state, and then the expected value of the Critic network parameters is calculated. The gradient is used to update the parameters;

[0024] in, This is the current reward. It is a discount factor. and It is the target network; μ' and Q' are updated using a moving average method, as shown in the following equation:

[0025] in, , All are parameters. The learning rate is 0.001; in the network architecture, the policy network (Actor) is used to update... To output actions Critic value network adopts parameters Update the function to approximate the state-behavior value function. .

[0026] In some embodiments, the state space s of the cooperative motion process of the two robotic arms is defined as follows:

[0027] in, Representative force tracking error, Represents trajectory tracking error. This represents the actual force exerted during the control process of the dual robotic arms. This represents the actual trajectory during the control process of the dual robotic arms; The goal of the DDPG algorithm is to base the force tracking error on... and trajectory tracking error Output appropriate adaptive parameters As the output parameter of the DRL algorithm, the designed action space contains only one definite element, which represents the action at time t, and is the adaptive parameter ε. The action function is shown in the following equation:

[0028] in, For adaptive parameters; during the position and force tracking process of the robotic arm, it is necessary to evaluate each time step in real time, based on the actual force. Average value of force error The difference and the actual trajectory Average trajectory error The sum of the differences under different proportions is used as part of the reward / penalty function, called the basic reward / penalty part; let the force tracking error... Tracking error All values ​​remain at 0; force tracking error and trajectory tracking error As part of the reward / penalty function, it is called the additional incentive component. Different additional rewards and penalties are applied based on the range in which the force tracking error occurs. The reward and penalty function, which combines the basic reward and penalty components with the additional incentive components, is shown in the following formula:

[0029] in, , Here, H is the scaling factor, and H is the time period of the training process. for The average value of the time trajectory error. for The average value of the force error at any given time. Let be the actual force of the two robotic arms at time j. To determine the actual position of the dual robotic arms at time j, in the additional excitation section, according to... and Different additional rewards or penalties are given depending on the range in which it is located.

[0030] Secondly, embodiments of the present invention also provide an adaptive variable impedance control device, the device comprising: The construction module is used to build an impedance model of the robot's dual robotic arms when grasping a target object, decouple the internal and external forces acting on the target object, and perform adaptive impedance control on the internal and external forces respectively. An initialization module is used to initialize the network parameters and experience pool of the impedance model. The experience pool is used to store the robot's experience tuples in the environment. The impedance model includes an Actor network and a Critic network. The Actor network is used to generate continuous actions, and the Critic network is used to evaluate the quality of the actions and output the corresponding action value function. The training module is used to select actions from the state space of the robot, execute the selected actions, store the experience tuples fed back from the environment into the experience pool, randomly sample a batch of data from the experience pool, calculate the loss of the Critic network and perform backpropagation, calculate the target Q value through the Critic network, update the parameters of the Actor network to maximize the Q value, and cyclically train until the preset number of iterations is reached to obtain the trained Actor-Critic network. The execution module is used to perform the actions of the dual robotic arms to grasp the target object in a real environment using the trained Actor-Critic network.

[0031] Thirdly, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the adaptive variable impedance control method as described in the first aspect.

[0032] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for performing the adaptive variable impedance control method as described in the first aspect.

[0033] According to embodiments of the present invention, an adaptive variable impedance control method, device, electronic device, and storage medium are provided. The adaptive variable impedance control method includes: constructing an impedance model of a robot's dual robotic arms when grasping a target object; decoupling the internal and external forces acting on the target object; and performing adaptive impedance control on the internal and external forces respectively; initializing the network parameters and experience pool of the impedance model, wherein the experience pool stores the robot's experience tuples in the environment; the impedance model includes an Actor network and a Critic network, where the Actor network generates continuous actions, and the Critic network evaluates the quality of the actions and outputs the corresponding action value function; selecting an action from the robot's state space; after executing the selected action, storing the experience tuples fed back from the environment into the experience pool; randomly sampling a batch of data from the experience pool; calculating the loss of the Critic network and performing backpropagation; calculating the target Q value through the Critic network; updating the parameters of the Actor network to maximize the Q value; and iteratively training until a preset number of iterations is reached to obtain a trained Actor-Critic network; and using the trained Actor-Critic network to execute the robot's actions in a real environment to grasp the target object. Based on this, when constructing the impedance model of the dual robotic arms when grasping a target object, the response of the robotic arms when contacting the environment is considered. Therefore, the force on the target object is decomposed, the object's forces are decoupled, and adaptive impedance control is applied to the internal and external forces respectively, optimizing the overall control strategy and improving control accuracy. Within the framework of reinforcement learning, this embodiment of the invention employs the Deep Deterministic Policy Gradient (DDPG) algorithm. The DDPG algorithm includes an Actor network and a Critic network. The Actor network outputs continuous actions, and the Critic network evaluates the quality of the actions. Through continuous learning in actual operation, the DDPG algorithm enables the dual robotic arms to adjust their action strategies to maximize cumulative rewards. During the training phase, samples are randomly selected from the experience pool to train the Actor-Critic network to obtain the optimal network structure. By using the trained Actor-Critic network, the dual robotic arms can make optimal action choices based on the current state dataset to adapt to uncertain environments and achieve efficient grasping of target objects. This embodiment of the invention allows the dual robotic arms to continuously optimize their behavior through reinforcement learning, thereby better adapting to complex working scenarios. Attached Figure Description

[0034] Figure 1A This is a flowchart of an adaptive variable impedance control method provided in one embodiment of the present invention; Figure 1B This is a schematic diagram of the coordinate system of a dual-arm cooperative system provided in one embodiment of the present invention; Figure 2 This is a force analysis diagram of a target object provided in one embodiment of the present invention; Figure 3 This is a structural diagram of a DDPG network model provided in one embodiment of the present invention; Figure 4 This is a diagram illustrating the change in cumulative reward R during model training, provided in one embodiment of the present invention. Figure 5 This is a structural diagram of a deep reinforcement learning adaptive variable impedance control strategy provided in one embodiment of the present invention; Figure 6 This is a block diagram of a dual robotic arm control system provided in one embodiment of the present invention; Figure 7 This is a flowchart of a dual-arm cooperative adaptive impedance control based on deep reinforcement learning provided in one embodiment of the present invention; Figure 8 This is a hardware connection structure diagram of an experimental platform provided in one embodiment of the present invention; Figure 9A This is a constant force tracking diagram under deep reinforcement learning adaptive impedance control provided in one embodiment of the present invention; Figure 9B This is a constant trajectory tracking diagram under deep reinforcement learning adaptive impedance control provided in one embodiment of the present invention; Figure 10A This is a constant force tracking diagram under deep reinforcement learning adaptive impedance control provided in one embodiment of the present invention; Figure 10B This is a variable trajectory tracking diagram under deep reinforcement learning adaptive impedance control provided in one embodiment of the present invention; Figure 11A This is a variable force tracking graph under impedance control of deep reinforcement learning provided in one embodiment of the present invention; Figure 11B This is a variable trajectory tracking diagram under impedance control of deep reinforcement learning provided in one embodiment of the present invention; Figure 12 This is a schematic diagram of an adaptive variable impedance control device provided in one embodiment of the present invention; Figure 13 This is a schematic diagram of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0036] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the following drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0037] In this embodiment of the invention, the terms "furthermore," "exemplarily," or "optionally" are used as examples, illustrations, or descriptions and should not be construed as being more preferred or advantageous than other embodiments or designs. The use of the terms "furthermore," "exemplarily," or "optionally" is intended to present the relevant concepts in a specific manner.

[0038] First, let's analyze some of the terms used in this invention: Dual-arm collaborative control refers to two robotic arms or robots working collaboratively within a system environment to achieve a common goal. This involves synchronized motion, force and torque control, collision detection and avoidance, trajectory planning, etc. The goal of dual-arm collaborative control is to achieve efficient task execution, improve productivity, reduce human intervention, and enhance task accuracy and safety.

[0039] Adaptive variable impedance control using deep reinforcement learning is a machine learning approach designed to train an agent (typically a robot or control system) to learn optimal behavioral strategies in interaction with its environment. This learning is guided by reward and penalty signals, with the agent aiming to maximize long-term rewards. The parameters of the control system can be automatically adjusted to adapt to changes or uncertainties in the system's dynamic properties. In adaptive control, the control system can adjust its control strategy in real time to improve performance.

[0040] To facilitate a more convenient description of the working principle of the embodiments of the present invention, the following introduction of relevant technical scenarios is given first.

[0041] In industrial production, traditional industrial robots achieve specific tasks in industrial scenarios through position control. However, in applications requiring interaction with the environment, traditional position-based control methods are no longer adequate. In fields such as welding, polishing, and shaft-hole assembly, there is extensive and complex contact with the environment. If the industrial robot only moves along a designated path, any deviation from that path will generate significant environmental contact forces, potentially damaging the workpiece or even the robot itself. With increasing complexity and the need for flexible production processes, existing robots operating at independent workstations can no longer meet the ever-changing manufacturing demands. To adapt to the requirements of task complexity, intelligent operation, and system compliance in unstructured environments, two robots working together and cooperating demonstrate advantages in performing such tasks. In a dual-arm robot, the two arms maintain certain constraints during coordination to complete the coordination task. The basic idea of ​​pure position control is to first plan the trajectory of the target object and obtain the trajectory of the arm ends through the constraint relationship between the target object and the two arms. However, this control method does not consider the forces exerted by the arms on the target object, nor does it consider external disturbances to the target object. Therefore, how to achieve efficient grasping of target objects by dual robotic arms in uncertain and complex scenarios has become an urgent technical problem to be solved.

[0042] Based on this, the present invention provides an adaptive variable impedance control method and device, electronic device, and storage medium. The adaptive variable impedance control method includes: constructing an impedance model of a robot's dual robotic arms when grasping a target object; decoupling the internal and external forces acting on the target object; and performing adaptive impedance control on the internal and external forces respectively; initializing the network parameters and experience pool of the impedance model, where the experience pool stores the robot's experience tuples in the environment; the impedance model includes an Actor network and a Critic network, where the Actor network generates continuous actions and the Critic network evaluates the quality of the actions and outputs the corresponding action value function; selecting an action from the robot's state space; after executing the selected action, storing the experience tuples fed back from the environment into the experience pool; randomly sampling a batch of data from the experience pool; calculating the loss of the Critic network and performing backpropagation; calculating the target Q value through the Critic network; updating the parameters of the Actor network to maximize the Q value; and iteratively training until a preset number of iterations is reached to obtain a trained Actor-Critic network; and using the trained Actor-Critic network to execute the robot's actions in a real environment to grasp the target object. Based on this, when constructing the impedance model of the dual robotic arms when grasping a target object, the response of the robotic arms when contacting the environment is considered. Therefore, the force on the target object is decomposed, the object's forces are decoupled, and adaptive impedance control is applied to the internal and external forces respectively, optimizing the overall control strategy and improving control accuracy. Within the framework of reinforcement learning, this embodiment of the invention employs the Deep Deterministic Policy Gradient (DDPG) algorithm. The DDPG algorithm includes an Actor network and a Critic network. The Actor network outputs continuous actions, and the Critic network evaluates the quality of the actions. Through continuous learning in actual operation, the DDPG algorithm enables the dual robotic arms to adjust their action strategies to maximize cumulative rewards. During the training phase, samples are randomly selected from the experience pool to train the Actor-Critic network to obtain the optimal network structure. By using the trained Actor-Critic network, the dual robotic arms can make optimal action choices based on the current state dataset to adapt to uncertain environments and achieve efficient grasping of target objects. This embodiment of the invention allows the dual robotic arms to continuously optimize their behavior through reinforcement learning, thereby better adapting to complex working scenarios.

[0043] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0044] like Figure 1A As shown, Figure 1A This is a flowchart of an adaptive variable impedance control method provided in an embodiment of the present invention. The adaptive variable impedance control method may include, but is not limited to, steps S101 to S104.

[0045] Step S101: Construct an impedance model of the robot's dual robotic arms when grasping a target object, decouple the internal and external forces acting on the target object, and perform adaptive impedance control on the internal and external forces respectively. Step S102: Initialize the network parameters and experience pool of the impedance model. The experience pool is used to store the robot's experience tuples in the environment. The impedance model includes an Actor network and a Critic network. The Actor network is used to generate continuous actions, and the Critic network is used to evaluate the quality of the actions and output the corresponding action value function. Step S103: Select an action from the robot's state space. After executing the selected action, store the experience tuples of environmental feedback into the experience pool, randomly sample a batch of data from the experience pool, calculate the loss of the Critic network and perform backpropagation, calculate the target Q value through the Critic network, update the parameters of the Actor network to maximize the Q value, and train iteratively until the preset number of iterations is reached to obtain the trained Actor-Critic network. Step S104: Using the trained Actor-Critic network, perform the actions of the dual robotic arms to grasp the target object in a real-world environment.

[0046] In one embodiment, the motion trajectory planning of a single robotic arm is based on the estimation of the manipulated target and derived through kinematic transformation of the coordinate system. Therefore, the coordinate system of the dual-arm cooperative system is established as follows: Figure 1B As shown, in Figure 1B middle These are represented as the world coordinate system and the target object coordinate system, respectively. These are represented as the coordinate system of the right robotic arm base and the coordinate system of the left robotic arm base, respectively. Let the coordinates be represented as the right end-effector coordinate system and the left end-effector coordinate system, respectively. Based on the above coordinate systems, a transformation formula 1 for object-oriented dual-arm motion planning is proposed.

[0047] For the target object being manipulated, its position and orientation relative to the reference coordinate system are solved using Equation 1.

[0048] In the formula, This is the transformation matrix of the target object relative to the centroid coordinate system; Let be the 3x3 rotation matrix of the target object relative to the coordinate system at its centroid; This is the 3x1 position matrix of the target object relative to the coordinate system at its centroid.

[0049] The transformation between the target object's center-of-mass coordinate system and the world coordinate system into constraints between the target object and the robotic arm is expressed by the following equation 2:

[0050] In the formula, This represents the coordinate system at the center of mass of the target object obtained based on the kinematics of the right arm. Homogeneous coordinate transformation relative to the world coordinate system W; This represents the homogeneous coordinate transformation of the target object's center of mass coordinate system relative to the world coordinate system W, obtained based on the kinematic chain of the left arm. , Representing the right arm base coordinate system Left arm base coordinate system Homogeneous coordinate transformation relative to the world coordinate system W; Representing the coordinate system at the end of the right arm Left arm end coordinate system Homogeneous coordinate transformation relative to the corresponding robotic arm base coordinate system; , Representing the centroid coordinate system of the target object respectively Relative to the coordinate system of the right arm end Left arm end coordinate system Homogeneous coordinate transformation.

[0051] After constraining the positions of the two arms as described above, in order to achieve true real-time collaborative operation, in addition to position constraints, it is also necessary to ensure the consistency of the speed of the two arms during the movement. Therefore, it is necessary to analyze the speed constraint relationship through Equation 3 to ensure that the two arms maintain the consistency of position and speed during the movement and achieve collaborative operation.

[0052]

[0053] In the formula, ; Indicates the end effector of the robotic arm Relative to the world coordinate system speed; , Representing the centroid coordinate system of the target object respectively Relative to the world coordinate system Velocity and angular velocity; Represents the coordinate system of the robotic arm's end effector In the world coordinate system The position transformation matrix below; Represents the coordinate system of the robotic arm's end effector Relative to the target object's centroid coordinate system Position transformation matrix; Represents the target object's centroid coordinate system Relative to the world coordinate system The direction rotation matrix.

[0054] Force analysis of the target object as follows Figure 2 As shown, in Figure 2 middle, Let the forces and torques generated by the dual robotic arms and external forces acting on the surface of the target object be represented respectively. Based on Newton's second law and Euler's equations, the following dynamic equations for the target object are established to determine the state of the dual robotic arms grasping the object:

[0055] Simplified to equation (5):

[0056] In the formula , , , , , The inertia matrix at the center of mass of the target object; This represents the vector force exerted on an object by the two robotic arms. The mass-inertia matrix representing the target object; Represents the linear acceleration and angular acceleration of the target object during its motion; It is represented as the resultant force vector of the Coriolis force, gravity, and centrifugal force of the target object; The vector force representing the external disturbance force acting on the target object transforms equation (5) into equation (6).

[0057]

[0058] In the formula Represented as the left and right arms of a dual robotic arm. Represents the capture matrix; This represents the force exerted by the robotic arm on the target object, and the data can be directly collected through a six-dimensional force sensor. The grasping matrix can be decomposed into external force equation (7) and internal force equation (8):

[0059]

[0060] In the formula, yes The generalized inverse of a matrix.

[0061] In one embodiment, conventional impedance control implements a control strategy based on adjusting force according to position error, while admittance control implements a control strategy based on adjusting position according to force error. Therefore, position-based impedance control is also called admittance control. The impedance / admittance control strategy relies on a second-order "mass-impedance-spring" system. The dynamics of the target object are established using a second-order "mass-impedance-spring" system. The model of the impedance control strategy is Equation (18). In this invention, the task of dual-arm collaborative target object control is decomposed into an inner loop and an outer loop. The inner loop mainly controls the internal force. To prevent the robotic arm from damaging the target object, the outer ring controls the external force. Ensure the target object completes the collaborative task. From equations (7) and (8), we can see that the force exerted on the target object by the two robotic arms can be divided into internal force and external force. Therefore, equation (9) can be transformed into equation (13).

[0062]

[0063] In the formula , These represent the expected trajectory and the actual trajectory, respectively. , These represent the expected force and the actual force, respectively. ; Represents the inertia matrix; Represents the damping matrix; Represents the stiffness matrix; This represents the error value of the force.

[0064]

[0065] In the formula, for It is a single-dimensional quantity.

[0066] Substituting equation (10) into equation (9) yields equation (11):

[0067] From equation (11), we can obtain for:

[0068] Substituting equation (12) into equation (9), we obtain equation (13):

[0069] Applying the Laplace transform to equation (13), we get:

[0070] Therefore, the steady-state error of force tracking is as shown in equation (15):

[0071] As can be seen from equation (15), if we want the force to track the steady-state error... There are two methods. The first method is shown in equation (16), due to the stiffness under uncertain environment. and location Since everything is unknown, it is difficult to achieve. Equation (16) is not suitable. The second method is shown in equations (15) and (17), when hour, Similarly, due to the unknown environment, it is difficult to accurately obtain the desired trajectory. Therefore, the initial operating environment will be used. replace and order The improved equation is shown in equation (18).

[0072]

[0073] Based on the actual environment, appropriate inertia coefficient m and damping coefficient b should be selected to achieve stable force tracking requirements.

[0074] In uncertain environments, consider the operating environment trajectory. Environmental stiffness Since it is unknown, we combine all the unknown parameters in equation (18) to get (19).

[0075]

[0076] Set new adaptive parameters To replace the unknown parameter, as shown in equation (20).

[0077]

[0078] The adaptive impedance equation is obtained as shown in equation (21):

[0079] To optimize the impedance parameters and impedance parameter set, the force tracking control process of the robotic arm is designed as a Markov decision process (MDP)-based reinforcement learning framework suitable for different environments. Appropriate state and action values ​​are designed for the IIC (Intelligent Impedance Control) method, and a reward function is constructed. A safe learning mechanism is also added. The Markov decision process serves as the basic framework for RL (an interactive goal-oriented learning method), which includes (1) a state set S; (2) an action set A; and (3) state transition probabilities. (4) Reward probability (5) Attenuation factor (6) Online neural network parameters of actor / critic .

[0080] The basic process of RL is as follows: the agent interacts with the environment at time t and observes a certain feature of the environment's state. And output an action based on P. A. At the next moment t+1, based on the result of the action, the Agent obtains an immediate reward or penalty. and enter a new state. This process is then repeated continuously. This process is known as the strategy of the RL method. , can be represented by equation (22), which refers to the distribution of the action set in state s.

[0081]

[0082] Where 'a' is an action in action set A. A specific value, s, is an action in the action set S. Specific value.

[0083] Multiple learning sequences can be obtained through repeated interactions between the environment and the robotic arm. The average of the rewards and penalties for a given state (the weighted sum of all decaying rewards from a given moment onwards) is the expected reward for that state, or its value. Similarly, the average of the rewards for taking a uniform action in a given state is the expected reward for that state-action pair, or its value. Theoretically, through continuous learning, the values ​​of all states and state-action pairs can be obtained, thus leading to the state value function and the action value function.

[0084] According to the definition, cumulative rewards and penalties ( ) is denoted as equation (23).

[0085]

[0086] Where r is the reward, and the policy is... The element is random, therefore cumulative rewards and penalties apply. There are also multiple random possible values. To evaluate state s, a definite value needs to be defined to describe state s. However, cumulative rewards and penalties... Since the value is random, it cannot be used to describe state s; therefore, cumulative rewards and penalties are used. The expected value is used as the state value function The definition is as shown in equation (24).

[0087]

[0088] Accordingly, in strategy The value of performing action a in state s is called the state-behavior value function, which is defined as equation (25).

[0089]

[0090] The Bellman equation for the state value function is obtained from equation (24), which expresses the value of the current state s and the value of subsequent states. The relationship between their values ​​is shown in equation (26).

[0091]

[0092] The strategy is a mapping from the state space to the action space, as shown in equation (27).

[0093]

[0094] Therefore, by combining equations (25) and (27), we can obtain the state-behavior value function. Another way to express it is as shown in equation (28).

[0095]

[0096] Based on an adaptive impedance control system and Markov decision-making, this paper utilizes Deep Reinforcement Learning (DRL) to solve complex computations in high-dimensional dynamic environments. The impedance control strategy of the IIC method is solved using the Deep Deterministic Policy Gradient (DDPG) algorithm. DDPG is a type of DRL algorithm based on the Actor... The Critic architecture combines the features of value function updates and policy updates, and has certain advantages in solving problems in continuous motion space, making it suitable for force tracking control problems of robotic arms.

[0097] Based on the deterministic gradient strategy, the parameters of the Actor neural network are updated according to the predicted action value function value, and the deterministic behavior strategy is as shown in equation (29).

[0098]

[0099] in, For the policy function, Given the current state, the action of a deterministic policy is uniquely determined in state s, as shown in the following formula.

[0100]

[0101] in, These are the policy parameters. During network training, multiple data points N are randomly sampled and used as deterministic policies. Training data to measure deterministic policies This is shown in equation (31).

[0102]

[0103] in Indicates from empirical distribution Sample a state , This indicates that the Critic network is for a given state. and actions The value. The overall expectation. This represents the expected value for the sampled state and action. During training, this expected value is estimated by averaging a batch of samples, i.e., the sample mean is used instead of the expected value.

[0104] The Actor network learns the optimal policy by maximizing the output of the Critic network. The parameters of the Actor network are updated via gradient ascent using equation (32):

[0105] in, Indicates from the Actor strategy Sampling an action , This indicates that the Critic network is for a given state. and actions The value of the gradient term This represents the gradient of the Critic network's response to the Actor's output action. This gradient indicates how slightly changing the Actor network's parameters affects the Critic network's estimation of the value function.

[0106] Based on minimizing the loss function L( Update the parameters of the current value network Q. The loss function to be minimized is shown in equation (33):

[0107] Where N is the number of random samples.

[0108] Based on the deterministic gradient policy, the behavior policy network is calculated according to equation (34). The gradient strategy is used to update the parameters of the Critic neural network.

[0109]

[0110] in, and From empirical distribution The states and actions obtained from sampling From empirical distribution The next state obtained from the sampling, y is the output of the Critic network, representing the estimated value of action a in state s, and y is the target Q-value, typically calculated by the target network. The goal of this gradient estimation is to minimize the mean squared error between the Critic network's output and the target Q-value. This mean squared error is estimated using sampled states, actions, and the next state, and then the expected value is calculated for the Critic network parameters. The gradient is used to update the parameters.

[0111]

[0112] in, This is the current reward. It is a discount factor. and It is the target network.

[0113] The target network is a copy of the behavior network, and μ' and Q' are updated using a moving average method, as shown in Equation (36):

[0114] in , All are parameters, among which The learning rate is typically set to 0.001. In the network architecture, the policy network (Actor) is used to update... To output actions Critic value network adopts parameters Update the function to approximate the state-behavior value function. .

[0115] Based on the DDPG algorithm, its network structure model is as follows: Figure 3 As shown.

[0116] The state space s of the collaborative motion process of the two robotic arms is defined as shown in equation (37).

[0117]

[0118] in, Representative force tracking error, Represents trajectory tracking error. This represents the actual force exerted during the control process of the dual robotic arms. This represents the actual trajectory during the control process of the dual robotic arms.

[0119] The goal of the DDPG algorithm is to base the force tracking error on... and trajectory tracking error Output appropriate adaptive parameters Therefore, as the output parameter of the DRL algorithm, the designed action space contains only one definite element, which represents the action at time t, and is the adaptive parameter ε. The action function is shown in equation (38).

[0120] in, These are adaptive parameters.

[0121] During the position and force tracking process of the robotic arm, it is necessary to evaluate each time step in real time, based on the actual force. Average value of force error The difference and the actual trajectory Average trajectory error The sum of the differences under different proportions is used as part of the reward / penalty function, called the basic reward / penalty part. The goal of an adaptive impedance control system is to achieve good tracking performance, that is, to minimize the force tracking error. Tracking error All should be kept as close to zero as possible. Therefore, the force tracking error will be... and trajectory tracking error As part of the reward and punishment function, it is called the additional incentive component ( As shown in equation (40). In addition, in order to accelerate the training efficiency of the DRL model, different additional rewards and penalties will be given according to the interval in which the force tracking error is located. The reward and penalty function composed of the basic reward and penalty part and the additional incentive part is shown in equation (39).

[0122]

[0123] in, , Here, H is the scaling factor, and H is the time period of the training process. for The average value of the time trajectory error. for The average value of the force error at any given time. Let be the actual force of the two robotic arms at time j. To determine the actual position of the dual robotic arms at time j, in the additional excitation part, to accelerate the training efficiency of the DRL model, it will be based on... and Different intervals in which the network is located are given different additional rewards or penalties. This invention uses the Adam gradient descent algorithm as the optimizer for the Actor network. The Adam optimizer is a gradient descent algorithm widely used in deep learning. It has the characteristic of adaptive learning rate, which can effectively handle different gradient scales for different parameters, thereby improving training efficiency. The Adam optimizer is a robust and efficient optimizer that typically achieves good results in many deep learning tasks. Its adaptive learning rate characteristic makes the network more likely to converge.

[0124] The model is trained, and the training results are as follows: Figure 4 As shown, in the first 100 rounds, the algorithm is in the strategy exploration phase, and the cumulative reward / penalty increases rapidly. After 100 rounds, the cumulative reward / penalty curve begins to converge, indicating that a better strategy has been found.

[0125] Based on the Markov decision process, basic adaptive impedance control is combined with the DDPG algorithm to form an adaptive variable impedance control based on deep reinforcement learning of the reference model, such as... Figure 5 As shown.

[0126] In one embodiment, when the two arms collaboratively grip a common target object that interacts with a ring, the target object is subjected to both internal and external forces. Based on the above deep reinforcement learning adaptive impedance control strategy, this invention designs a dual-arm deep reinforcement learning adaptive impedance control strategy, the main control block diagram of which is shown below. Figure 6 As shown, Figure 6 The symmetrical control strategy of the dual robotic arms and the external impedance control and internal impedance control strategies of the dual robotic arm control system are shown.

[0127] exist Figure 6 middle, These are represented as actual external force and expected external force, respectively. These represent the actual internal force and the expected internal force, respectively. These are respectively expressed as the errors of the actual external force and the expected external force, and the errors of the actual internal force and the expected external force. These represent the position compensation generated by the external deep reinforcement learning adaptive impedance controller and the internal force deep reinforcement learning adaptive impedance controller, respectively. These represent the actual trajectory of the target object and the desired trajectory of the target object, respectively. The actual motion trajectories of the end effectors of the two robotic arms are represented respectively. They are decomposed into the actual motion trajectories of the end effectors of the left and right robotic arms by the closed-chain constraint condition of the two arms. These represent the actual motion trajectories of the end effectors of the left and right robotic arms, which are then converted into the actual motion angles of the joints of the left and right robotic arms through reverse motion.

[0128] In one embodiment, an impedance model is constructed for the dual robotic arms grasping a target object; When constructing the impedance model of the dual robotic arms when grasping a target object, this invention first considers the response of the robotic arms when in contact with the environment. Therefore, the force on the target object is decomposed, the internal and external forces are decoupled, and adaptive impedance control is performed on the internal and external forces respectively to optimize the overall control strategy and improve control accuracy.

[0129] During the process of the two robotic arms contacting the target object or target environment, the parameters of the robotic arms and control strategy will change. These parameters include the stiffness and damping of the robotic arms, which are used to describe their motion and force characteristics during the grasping process.

[0130] To train the robotic arm to behave in uncertain environments, this invention first requires collecting a dataset of the robotic arm's state during actual operation. This includes information such as the robotic arm's current position, velocity, and acceleration, as well as sensor data such as force and torque when the robotic arm comes into contact with the environment.

[0131] Within the framework of reinforcement learning, this invention selects the Deep Deterministic Policy Gradient (DDPG) algorithm. This algorithm consists of two key components: an Actor neural network for outputting continuous actions, and a Critic neural network for evaluating the quality of the actions. Through continuous learning in practical operation, the DDPG algorithm enables the robotic arm to adjust its action policy to maximize cumulative rewards.

[0132] During the training phase, this invention establishes an experience pool containing the current state of the robotic arm, the actions performed, the rewards obtained, and the next state. This invention randomly selects samples from the experience pool to train the Actor-Critic neural network to obtain the optimal network structure.

[0133] Ultimately, by using a trained Actor-Critic neural network, the robotic arm can make optimal action choices based on the current state dataset to adapt to uncertain environments and achieve efficient grasping of target objects. This invention allows the robotic arm to continuously optimize its behavior through reinforcement learning, thereby better adapting to complex work scenarios.

[0134] This invention utilizes the DDPG (Deep Deterministic Policy Gradient) algorithm to solve the deep reinforcement learning problem for a dual-arm robot gripping a target object in a continuous action space. The entire algorithm flow is as follows: Figure 7 As shown: First, during the initialization phase, the network parameters are initialized. This includes the Actor network and the Critic network, which are used to output continuous actions and evaluate the quality of actions, respectively. Simultaneously, an experience pool is set up to store the robot's experience in the environment, including tuples of states, actions, rewards, and the next state.

[0135] Secondly, the robot's state and motion space are clearly defined to ensure it can handle continuous operations in the real-world environment. An Actor network is designed to generate continuous actions, while a Critic network is used to evaluate the quality of the actions and output the corresponding action value function.

[0136] To improve training stability, a target network, a replica of the Actor and Critic networks, is introduced. The parameters of the target network are updated periodically at a certain ratio, thereby mitigating the instability problem during training.

[0137] Next, define the optimizer and loss function. Select appropriate optimizers for the Actor and Critic networks, typically the Adam optimizer, and define the mean squared error loss function for the Critic network to measure the difference between the estimated value and the target value.

[0138] Setting hyperparameters is also a crucial step, including the learning rate, discount factor, and soft update parameters. The choice of these hyperparameters directly affects the convergence and performance of the algorithm.

[0139] The training phase is the core of the algorithm. At each time step, the robot selects an action from the state space, generates the action through the Actor network, and adds some exploratory noise. After executing the selected action, it observes the environmental feedback, including rewards and the next state, and stores these experience tuples in the experience pool. Next, a batch of data is sampled from the experience pool, the loss of the Critic network is calculated, and backpropagation is performed to optimize the network parameters. The target Q-value is calculated through the target Critic network, the parameters of the Actor network are updated to maximize the Q-value, and finally, a soft update is performed to adjust the parameters of the target network. This training loop iterates continuously, gradually optimizing the Actor and Critic networks.

[0140] After training, the robot can perform actions using the trained Actor network and enter the real-world environment. To ensure the effectiveness of the algorithm, the robot's performance in the real-world environment can be evaluated periodically to adjust and optimize hyperparameters, ensuring the algorithm's robustness and generalization ability. Through this process, the DDPG algorithm enables the robot to adapt to complex, continuous action spaces, thereby achieving efficient reinforcement learning.

[0141] In one embodiment, to verify the practicality of the adaptive control algorithm, the present invention will, as shown in the example... Figure 8 The experiment was conducted on the experimental platform shown. The platform includes two UR5 robotic arms, a UR5 robotic arm control cabinet, a UR5 robotic arm teach pendant, a PC, a six-dimensional force sensor, and a switch. The two UR5 robotic arms communicate through their respective control cabinets. This invention uses the Ethernet communication protocol for communication between the UR5 robotic arms and the TCP communication protocol for communication between the six-dimensional force sensor and the PC. The two UR5 robotic arms and the six-dimensional force sensor are connected to the switch, which communicates with the PC via the TCP / IP communication protocol to transmit data. This allows the PC to directly control the movement of the robotic arms and read the data from the six-dimensional force sensor in real time.

[0142] In the simulation experiment, a standard force signal was set as the target force for the robot's end effector. A tracking force was set in the z-axis direction of the robot's end effector's operating space, while no target force values ​​were set in other axes. The target force value Fe = 8 in the z-axis direction was set, and the simulation time was set to 3 seconds. The force-displacement tracking effect was observed. Figure 9A and Figure 9B As shown, a deep reinforcement learning module was added to the traditional admittance control. In simulation experiments, three algorithms were compared and analyzed to highlight their respective advantages and disadvantages. Under constant force and trajectory conditions, deep reinforcement learning did not outperform the other two algorithms in tracking the desired force and trajectory, but the errors remained within 0.0005. Furthermore, deep reinforcement learning was the fastest algorithm to reach the desired force and trajectory. While the constant impedance controller had lower speed and error compared to deep reinforcement learning, it exhibited overshoot and oscillation in the early stages of simulation. Therefore, the overall performance of deep reinforcement learning was superior to the other two algorithms, verifying the feasibility of the adaptive variable impedance control algorithm based on deep reinforcement learning.

[0143] In the simulation experiment, a standard force signal was set as the target force for the robot's end effector. A tracking force was set in the z-axis direction of the robot's end effector's operating space, while no target force values ​​were set in other axes. The target force value Fe = 8 in the z-axis direction was set, and the simulation time was set to 3 seconds. The force-displacement tracking effect was observed. Figure 10A and Figure 10BAs shown, the simulation results of deep reinforcement learning adaptive impedance control for tracking the desired force and trajectory under constant force and constant trajectory are presented. Under the constant force and variable trajectory condition, deep reinforcement learning outperforms the other two algorithms in tracking the desired force and trajectory, and its convergence speed is relatively fast. While the constant impedance controller has the fastest convergence speed, it exhibits overshoot and oscillation in the early stages of simulation, and further oscillations in subsequent actions. The adaptive variable impedance controller has a larger error and a slower convergence speed. Therefore, deep reinforcement learning outperforms the other two algorithms overall, and its adaptive variable impedance control enables it to achieve better desired force and trajectory tracking.

[0144] In the simulation experiment, a standard force signal was set as the target force for the robot's end effector. The tracking force was set in the z-axis direction of the robot's end effector operating space, while no target force values ​​were set in other axes. The target force value Fe = 8 in the z-axis direction was set, and the trajectory value was set as Xe = 0.1 + 0.1sin(3π). The simulation time was set to 3 seconds, and the force-displacement tracking effect was observed. Figure 11A and Figure 11B The simulation results of the reference model adaptive impedance control for tracking the desired force and trajectory under variable force and trajectory conditions are shown. Under variable force and trajectory conditions, deep reinforcement learning outperforms the other two algorithms in tracking the desired force and trajectory, and its convergence speed is relatively fast. While the constant impedance controller has the fastest convergence speed, it exhibits overshoot and oscillation in the early stages of simulation. The adaptive variable impedance controller has a larger error, overshoot in the early stages of convergence, and a slower convergence speed. Therefore, deep reinforcement learning outperforms the other two algorithms overall. Furthermore, the reference model deep reinforcement learning adaptive variable impedance control can stabilize the force under dynamic trajectories, while also reducing the expected force error.

[0145] Compared to existing technologies, the deep reinforcement learning used in this invention possesses learning capabilities, allowing for optimization of control strategies through continuous trial and error and feedback, thereby improving system control performance. Compared to traditional control algorithms, deep reinforcement learning can better adapt to unknown environments and tasks. Furthermore, it can adjust control strategies based on real-time changes in the environment and task, achieving adaptive control. This adaptability enables the system to better adapt to complex and dynamic working environments. By using training data to improve system robustness, the system exhibits better adaptability to noise, interference, and uncertainty, achieving more precise force coordination control.

[0146] In addition, such as Figure 12 As shown, one embodiment of the present invention also discloses an adaptive variable impedance control device, the device comprising: Module 110 is used to build an impedance model of the robot's dual robotic arms when grasping a target object, decouple the internal and external forces acting on the target object, and perform adaptive impedance control on the internal and external forces respectively. The initialization module 120 is used to initialize the network parameters and experience pool of the impedance model. The experience pool is used to store the robot's experience tuples in the environment. The impedance model includes an Actor network and a Critic network. The Actor network is used to generate continuous actions, and the Critic network is used to evaluate the quality of the actions and output the corresponding action value function. The training module 130 is used to select actions from the robot's state space, and after the selected actions are executed, the experience tuples of environmental feedback are stored in the experience pool, and a batch of data is randomly sampled from the experience pool. The loss of the Critic network is calculated and backpropagation is performed. The target Q value is calculated through the Critic network, and the parameters of the Actor network are updated to maximize the Q value. The training is repeated until the preset number of iterations is reached to obtain the trained Actor-Critic network. The execution module 140 is used to perform the actions of the dual robotic arms to grasp target objects in a real environment using a trained Actor-Critic network.

[0147] The adaptive variable impedance control device of this invention is used to execute the adaptive variable impedance control method in the above embodiments. Its specific processing procedure is the same as that of the adaptive variable impedance control method in the above embodiments, and will not be described in detail here.

[0148] In addition, such as Figure 13 As shown, one embodiment of the present invention also discloses an electronic device, including: at least one processor 210; at least one memory 220 for storing at least one program; and when the at least one program is executed by the at least one processor 210, implementing the adaptive variable impedance control method as in any of the preceding embodiments.

[0149] In addition, one embodiment of the present invention discloses a computer-readable storage medium storing computer-executable instructions for performing the adaptive variable impedance control method as described in any of the preceding embodiments.

[0150] The system architecture and application scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0151] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0152] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0153] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process or execution thread, and components may be located on a single computer or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, or a network, such as the Internet interacting with other systems via signals).

Claims

1. An adaptive variable impedance control method, comprising: An impedance model is constructed for the robot's dual robotic arms when grasping a target object. The internal and external forces acting on the target object are decoupled, and adaptive impedance control is applied to the internal and external forces respectively. The network parameters and experience pool of the impedance model are initialized. The experience pool is used to store the robot's experience tuples in the environment. The impedance model includes an Actor network and a Critic network. The Actor network is used to generate continuous actions, and the Critic network is used to evaluate the quality of the actions and output the corresponding action value function. The robot selects an action from its state space, executes the selected action, stores the experience tuples fed back from the environment into the experience pool, randomly samples a batch of data from the experience pool, calculates the loss of the Critic network and performs backpropagation, calculates the target Q value through the Critic network, updates the parameters of the Actor network to maximize the Q value, and trains iteratively until a preset number of iterations is reached to obtain a trained Actor-Critic network. Using the trained Actor-Critic network, the action of the dual robotic arms is performed in a real environment to grasp the target object; The decoupling of the internal and external forces acting on the target object includes: Based on Newton's second law and Euler's equations, the state of the dual robotic arms grasping the target object is established, and the dynamic equation of the target object is established, which is then transformed into the following equation: In the formula Represented as the left and right arms of a dual robotic arm. Represents the capture matrix; This represents the force exerted by the robotic arm on the target object; the grasping matrix is ​​decomposed to obtain the external force formula. and obtaining internal force formula : In the formula, yes The generalized inverse of a matrix; The equations for the impedance model are as follows: In the formula, m is the inertia coefficient and b is the damping coefficient. For adaptive parameters, The error value representing the force, and These are the robot arm's motion speed and acceleration, respectively. , This represents the expected trajectory.

2. The method according to claim 1, characterized in that, The method further includes: Establish a coordinate system for the dual-manipulator collaborative system. The position and orientation of the target object with respect to the reference coordinate system are solved using the following formula: In the formula, This is the transformation matrix of the target object relative to the centroid coordinate system; Let be the 3x3 rotation matrix of the object relative to the coordinate system at its center of mass; This is the 3x1 position matrix of the target object relative to the coordinate system at its centroid; The transformation between the target object's center-of-mass coordinate system and the world coordinate system represents the constraint conditions between the target object and the robotic arm, expressed by the following formula: In the formula, This represents the coordinate system at the center of mass of the target object obtained based on the kinematics of the right arm. Homogeneous coordinate transformation relative to the world coordinate system W; This represents the homogeneous coordinate transformation of the target object's center of mass coordinate system relative to the world coordinate system W, obtained based on the kinematic chain of the left arm. , Representing the right arm base coordinate system Left arm base coordinate system Homogeneous coordinate transformation relative to the world coordinate system W; Representing the coordinate system at the end of the right arm Left arm end coordinate system Homogeneous coordinate transformation relative to the corresponding robotic arm base coordinate system; , Representing the centroid coordinate system of the target object respectively Relative to the coordinate system of the right arm end Left arm end coordinate system Homogeneous coordinate transformation; The velocity constraint relationship is analyzed by the following formula, so that the position and velocity of the two arms remain consistent during the movement. In the formula, ; Indicates the end effector of the robotic arm Relative to the world coordinate system speed; , Representing the centroid coordinate system of the target object respectively Relative to the world coordinate system Velocity and angular velocity; Represents the coordinate system of the robotic arm's end effector In the world coordinate system The position transformation matrix below; Represents the coordinate system of the robotic arm's end effector Relative to the target object's centroid coordinate system Position transformation matrix; Represents the target object's centroid coordinate system Relative to the world coordinate system The direction rotation matrix.

3. The method according to claim 1, characterized in that, The steps of randomly sampling a batch of data from the experience pool, calculating the loss of the Critic network and performing backpropagation, calculating the target Q-value through the Critic network, and updating the parameters of the Actor network to maximize the Q-value include: Based on a deterministic gradient strategy, the Actor network parameters are updated according to the action-value function. The deterministic behavior strategy is as follows: in, For the policy function, Given the current state, the action of a deterministic policy is uniquely determined in state s, as shown in the following formula: in, These are the policy parameters; during network training, multiple data points N are randomly sampled and used as the deterministic policy. Training data to measure deterministic policies This is expressed as follows: in, Indicates from empirical distribution Sample a state , This indicates that the Critic network is for a given state. and actions The value of the entire expectation This means taking the expected value based on the sampled state and action; during training, the sample mean is used instead of the expected value. The Actor network learns the optimal policy by maximizing the output of the Critic network. The parameters of the Actor network are updated using gradient ascent as follows: in, Indicates from the Actor strategy Sampling an action , This indicates that the Critic network is for a given state. and actions The value of the gradient term This represents the gradient of the Critic network with respect to the Actor's output action; Based on minimizing the loss function L( Update the parameters of the current value network Q. The loss function to be minimized is as follows: Where N is the number of random samples.

4. The method according to claim 3, characterized in that, The steps of randomly sampling a batch of data from the experience pool, calculating the loss of the Critic network and performing backpropagation, calculating the target Q-value through the Critic network, and updating the parameters of the Actor network to maximize the Q-value further include: Based on a deterministic gradient policy, the behavior policy network is calculated according to the following formula. The gradient strategy is used to update the parameters of the Critic network; in, and From the empirical distribution The states and actions obtained from sampling From empirical distribution The next state obtained from the sampling, y is the output of the Critic network, representing the estimated value of action a in state s, and y is the target Q value. The goal of gradient estimation is to minimize the mean squared error between the Critic network output and the target Q value. This mean squared error is estimated using the sampled state, action, and next state, and then the expected value of the Critic network parameters is calculated. The gradient is used to update the parameters; in, This is the current reward. It is a discount factor. and It is the target network; μ' and Q' are updated using a moving average method, as shown in the following equation: in, , All are parameters. The learning rate is 0.001; in the network architecture, the policy network (Actor) is used to update... To output actions Critic value network adopts parameters Update the function to approximate the state-behavior value function. .

5. The method according to claim 1, characterized in that, The state space s of the cooperative motion process of the two robotic arms is defined as follows: in, Representative force tracking error, Represents trajectory tracking error. This represents the actual force exerted during the control process of the dual robotic arms. This represents the actual trajectory during the control process of the dual robotic arms; The goal of the DDPG algorithm is to base the force tracking error on... and trajectory tracking error Output appropriate adaptive parameters As the output parameter of the DRL algorithm, the designed action space contains only one definite element, which represents the action at time t, and is the adaptive parameter ε. The action function is shown in the following equation: in, For adaptive parameters; during the position and force tracking process of the robotic arm, it is necessary to evaluate each time step in real time, based on the actual force. Average value of force error The difference and the actual trajectory Average trajectory error The sum of the differences under different proportions is used as part of the reward / penalty function, called the basic reward / penalty part; let the force tracking error... Tracking error All values ​​remain at 0; force tracking error and trajectory tracking error As part of the reward / penalty function, it is called the additional incentive component. Different additional rewards and penalties are applied based on the range in which the force tracking error occurs. The reward and penalty function, which combines the basic reward and penalty components with the additional incentive components, is shown in the following formula: in, , Here, H is the scaling factor, and H is the time period of the training process. for The average value of the time trajectory error. for The average value of the force error at any given time. Let be the actual force of the two robotic arms at time j. To determine the actual position of the dual robotic arms at time j, in the additional excitation section, according to... and Different additional rewards or penalties are given depending on the range in which it is located.

6. An adaptive variable impedance control device, characterized in that, The device includes: The construction module is used to build an impedance model of the robot's dual robotic arms when grasping a target object, decouple the internal and external forces acting on the target object, and perform adaptive impedance control on the internal and external forces respectively. An initialization module is used to initialize the network parameters and experience pool of the impedance model. The experience pool is used to store the robot's experience tuples in the environment. The impedance model includes an Actor network and a Critic network. The Actor network is used to generate continuous actions, and the Critic network is used to evaluate the quality of the actions and output the corresponding action value function. The training module is used to select actions from the state space of the robot, execute the selected actions, store the experience tuples fed back from the environment into the experience pool, randomly sample a batch of data from the experience pool, calculate the loss of the Critic network and perform backpropagation, calculate the target Q value through the Critic network, update the parameters of the Actor network to maximize the Q value, and cyclically train until the preset number of iterations is reached to obtain the trained Actor-Critic network. The execution module is used to perform the actions of the dual robotic arms to grasp the target object in a real environment using the trained Actor-Critic network. The decoupling of the internal and external forces acting on the target object includes: Based on Newton's second law and Euler's equations, the state of the dual robotic arms grasping the target object is established, and the dynamic equation of the target object is established, which is then transformed into the following equation: In the formula Represented as the left and right arms of a dual robotic arm. Represents the capture matrix; This represents the force exerted by the robotic arm on the target object; the grasping matrix is ​​decomposed to obtain the external force formula. and obtaining internal force formula : In the formula, yes The generalized inverse of a matrix; The equations for the impedance model are as follows: In the formula, m is the inertia coefficient and b is the damping coefficient. For adaptive parameters, The error value representing the force, and These are the robot arm's motion speed and acceleration, respectively. , This represents the expected trajectory.

7. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the adaptive variable impedance control method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing computer-executable instructions for performing the adaptive variable impedance control method as described in any one of claims 1 to 5.

Images