A robot shaft-hole assembly method based on deep reinforcement learning and admittance control

Abstract

The present invention provides a robot shaft hole assembly method based on deep reinforcement learning and admittance control, which belongs to the field of intelligent manufacturing; for the single-axis hole search stage and the multi-axis hole search stage, the present invention respectively designs a deep reinforcement learning network I and a deep reinforcement learning network II to train the output of the hole search action, and at the same time, adds hole search priori knowledge in this process to improve the exploration training efficiency of the deep reinforcement learning network; for the single-axis hole plugging stage and the multi-axis hole plugging stage, a variable admittance control method based on the deep reinforcement learning network III is designed to realize the smooth plugging of the robot, and at the same time, the plugging priori knowledge is added, and if the state is the priori state, the robot adjustment posture is directly output, and if not, it is input into the deep reinforcement learning network III for training, thereby improving the efficiency of the robot plugging; adopting the technical solution of the present invention, through the training of the deep reinforcement learning network, the adjustment strategy is output, and the robot can be controlled to complete the assembly of the single-axis hole and the multi-axis hole.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing, and more specifically, to a robot assembly method based on deep reinforcement learning and admittance control. Background Art

[0002] With the development of intelligent manufacturing technology, robot compliant assembly has always been the research focus and hotspot in the field of industrial robots. However, high-performance assembly still faces many difficulties. The study of robot compliant shaft-hole assembly methods is of great significance to the expansion of robots in the entire assembly field. In the field of robot compliant shaft-hole assembly, the main methods are three types: assembly methods based on passive compliant control, compliant assembly methods based on traditional control algorithms, and robot compliant assembly methods based on learning algorithms. Both the assembly methods based on passive compliant control and the compliant assembly methods based on traditional control algorithms require complex analysis of the contact model of the assembly or dynamic modeling, and the dynamic models of some parts are often very complex, which brings great difficulties to the involvement of the assembly algorithm. The robot compliant assembly method based on learning algorithms often requires a large amount of training data, low learning efficiency, and low assembly robustness. Secondly, for multi-axis hole assembly, the traditional control algorithms in the hole search and hole insertion stages of the assembly process have poor adaptability to different assembly environments and low efficiency. Therefore, in order to solve the above defects, a robot assembly method based on deep reinforcement learning and admittance control is proposed. Summary of the invention

[0003] In order to solve the above problems, the present invention aims to disclose a robot assembly method based on deep reinforcement learning and admittance control, especially an exploration and learning of hole searching and hole adjustment strategy actions in the shaft hole assembly process based on a deep reinforcement learning algorithm to achieve robot compliant assembly.

[0004] In order to achieve the above object, the technical solution adopted by the present invention mainly includes the following processes:

[0005] Step 1, the robot shaft hole assembly is divided into three stages: the free near hole stage outside the hole, the hole edge search stage, and the hole insertion stage; at the same time, the method of distinguishing the robot single-axis hole assembly and multi-axis hole assembly;

[0006] Step 2: Roughly locate the hole by visual positioning, control the robot movement, make the workpiece at the end of the robot reach the hole edge and contact the hole, and then the robot enters the hole edge search stage of assembly;

[0007] In the visual positioning method, the robot world coordinate system, tool coordinate system and workpiece coordinate system are first constructed, and the origin of the world coordinate system is the origin of the robot base coordinate system; secondly, markers are placed around the assembly holes, and the RGB image and depth information of the markers are obtained using a depth camera. The assembly holes are roughly positioned by camera calibration, depth image calibration, and hand-eye calibration and marker recognition in combination with the robot arm; the coordinates obtained by the rough positioning are the position and posture of the assembly holes in the robot world coordinate system;

[0008] Preferably, controlling the movement of the robot refers to controlling the movement of the robot through a program;

[0009] Step 3: In the hole edge search stage, firstly, design the hole search prior knowledge based on human hole search experience, preset the prior state and prior action, and form a prior knowledge memory bank. Secondly, collect the force / torque information and posture information of the robot as the input of the deep reinforcement learning network. The prior state classification model is used to determine whether the input is prior knowledge. If so, directly output the corresponding action. If not, input it into the deep reinforcement learning network for exploration and training. Output the robot position or posture adjustment strategy action so that the workpiece at the end of the robot can search holes smoothly. Construct the deep reinforcement learning network I for single-axis hole search and the deep reinforcement learning network II for multi-axis hole assembly respectively.

[0010] The prior actions of hole-searching prior knowledge based on human hole-searching experience include actions in four directions A=[+ΔX, -ΔX, +ΔX, -ΔX], where ΔX and ΔY are the step displacements of the workpiece at the end of the robot in the X-axis and Y-axis directions in the Cartesian world coordinate system, respectively. After being classified by the prior state classification model, one of the above four actions is output to adjust the position of the end of the robot, thereby ultimately ensuring that the robot can search holes smoothly.

[0011] The input S of the hole-searching prior knowledge prior state classification model based on human hole-searching experience is [P X , P Y , R X , R Y , R Z ], where P X , P Y is the position of the robot end workpiece in the X-axis and Y-axis directions in the Cartesian world coordinate system, R X , R Y , R zThe rotation angles of the robot end workpiece around the X-axis, Y-axis, and Z-axis in the world coordinate system. The significance of adding hole-searching prior knowledge based on human hole-searching experience is that when the action that the robot end workpiece needs to adjust is the movement amount on the X-axis and Y-axis of the hole plane, it does not need to explore and learn the action through the deep reinforcement learning network, but adjusts it according to the action preset by human experience, thereby accelerating the exploration efficiency of deep reinforcement learning in the hole-searching process;

[0012] Preferably, the prior state classification model adopts a support vector machine (SVM) model;

[0013] The force signal is collected by the force sensor at the end of the robot, including the contact force signal F of the workpiece at the end of the robot in the three coordinate axes X, Y, and Z directions. X , F Y , F Z , assembly torque signal T around the workpiece at the end of the robot X , T Y , T Z ;

[0014] The input S of the deep reinforcement learning network I for single-axis hole search is [P X , P Y , R X , R Y , R Z , F X , F Y , T X , T Y ], the input of the deep reinforcement learning network II for multi-axis hole assembly is S = [P X , P Y , R X , R Y , R z , F X , F Y , T X , T Y , T Z ], where P X , P Y , R X , R Y , R z is the position information of the end workpiece on the X and Y axes in the Cartesian world coordinate system, and the rotation amount around the X, Y, and Z axes. X , F Y , T X , T Y , T Z is the force and torque information of the robot end workpiece in the X-axis and Y-axis and the torque information of the robot end workpiece in the Z-axis; the output A of the deep reinforcement learning network I = [+ΔR X, -ΔR X , +ΔR Y , -ΔR Y ], the output of the deep reinforcement learning network II A=[+ΔR X , -ΔR X , +ΔR Y , -ΔR Y , +ΔR Z , -ΔR z ,] where ΔR X , ΔR Y , ΔR Z They are the step adjustment amounts of the robot end workpiece around the positive and negative directions of the X-axis, around the positive and negative directions of the Y-axis, and around the positive and negative directions of the Z-axis;

[0015] Preferably, the deep reinforcement learning network I and the deep reinforcement learning network II both adopt the DQN (Deep QNetwork) network structure. The input of the DQN model undergoes three convolutional layers and two fully connected layers of nonlinear transformation, and finally obtains the Q value of each action value at the output layer, and selects the optimal action according to the Q value; the update method of the DQN algorithm is: Where s′ and a′ represent the state and action of the next moment, r and γ represent the reward function and decay factor, respectively; the loss function of the DQN algorithm is

[0016] Step 4: Design the reward and punishment function, the number of training rounds, and the maximum number of steps for each round of training for the deep reinforcement learning network in the hole edge search stage; use the deep reinforcement learning network I and the deep reinforcement learning network II to conduct exploration training on single-axis hole search and multi-axis hole search respectively until the number of steps and reward value of each round converge;

[0017] The reward function where k n is the number of steps in the nth round of training, and the maximum number of steps in each round is set to k max The first type of reward function is positive reward, which is determined by the ratio between the number of steps trained in this round and the set maximum number of steps. The more steps taken to complete the assembly in each round, the lower the positive reward value obtained in this round; the second type is negative reward. When the end of the robot rotates in the same direction for m consecutive times, the behavior is set to deviate from the center of the hole, and a -1 reward is given;

[0018] The exploration training process is as follows: first, the robot end is controlled to move toward the center of the hole according to the roughly positioned hole posture; after the initial movement, the force / torque information and posture information of the workpiece at the end of the robot are collected; and the parameters therein are used to determine whether the state at this time is a priori state; if it is a priori state, the next action is performed according to the prior action; if it is not a priori state, the state information is input into the deep reinforcement learning network training to obtain the next motion state, and so on are repeated. Each round of training is completed until the hole search is successful or the number of training steps per round reaches the set maximum number of steps, then the training of this round is terminated, the reward and punishment value is calculated, and then the next round of training is carried out; when the collected Z-axis force suddenly becomes smaller, it means that the hole search is successful in this round; when the optimal action strategy is trained, the reward function and the number of training steps converge, indicating that the training is successful, and the workpiece at the end of the robot enters the hole insertion stage;

[0019] Step 5: During the hole-insertion stage, a constant desired force is set in the Z-axis direction of the robot end, and the forces and torques in other directions are all 0. At the same time, the robot end is moved to the set desired position, and the force / torque information of the robot end during the hole-insertion process is collected. The collected force / torque of the robot end is compared with the set desired force to obtain the real-time force deviation; the force deviation is input into the admittance controller, and the adjusted posture of the workpiece at the robot end is output after calculation by the admittance control algorithm; at the same time, the admittance control parameters are optimized in real time through the deep reinforcement learning algorithm to realize variable admittance control;

[0020] The expected force F d =[0,0,f d , 0, 0, 0], the sensor collects the contact force F = [f x , f y , f z , T x , T y , T z ], the force deviation is ΔF=F d -F;

[0021] The admittance control algorithm is: Where M, B, and K represent the robot mass matrix, stiffness matrix, and damping matrix, respectively. d is the expected position of the robot end workpiece in the Cartesian world coordinate system, and X is the real-time position of the robot end workpiece in the Cartesian world coordinate system;

[0022] Step 6. Design a priori knowledge memory bank of jacks based on human jack experience, including prior states and prior experience values. Construct a classifier to identify the input state of the deep reinforcement learning network. If the input state is a priori state, directly output the prior experience value, and then directly control the end motion of the robot workpiece. If the input state is not a priori state, send it to the deep reinforcement learning network for training, and then control the end motion of the robot through the admittance controller. Define the admittance control parameters that need to be trained, and simulate the admittance control algorithm. By selecting different admittance control parameters for simulation, determine the admittance control parameters that need to be trained, and predetermine the range of change of the admittance control parameters.

[0023] The input of the prior state of the socket based on human experience in socket insertion is the position and posture S=[Px, Py, Pz, Rx, Ry, Rz, Fx, Fy, Fz, Tx, Ty, Tz] of the robot end workpiece in the Cartesian world coordinate system, wherein Px, Py, and Pz are the positions of the robot end workpiece in the Cartesian world coordinate system on the X-axis, Y-axis, and Z-axis, respectively, Rx, Ry, and Rz are the rotation amounts of the robot end workpiece around the X-axis, Y-axis, and Z-axis in the Cartesian world coordinate system, respectively, and Fx, Fy, Fz and Tx, Ty, and Tz are the forces and torques of the robot end in the X-axis, Y-axis, and Z-axis directions, respectively; the prior experience value of the socket based on human experience in socket insertion is A=[+ΔRx, -ΔRx, +ΔRy, -ΔRy], wherein ΔRx and ΔRy represent the positive and negative rotation amounts of the robot end workpiece around the X-axis and the positive and negative rotation amounts around the Y-axis, respectively;

[0024] Preferably, the prior state classification model adopts a support vector machine (SVM) model;

[0025] The admittance control parameters include an inertia matrix, a damping matrix, and a damping matrix, wherein the inertia matrix is ​​a unit positive definite matrix; the training method thereof is to select one of the coefficient components in the damping matrix and one of the coefficient components in the damping matrix for training, and then obtain other coefficients in the damping matrix and the stiffness matrix according to the proportional relationship;

[0026] Step 7: Construct a deep reinforcement learning network III for variable admittance control that can output continuous actions, collect force / torque signals and posture signals during the plug-in process as state inputs for deep reinforcement learning, and the output of the deep reinforcement learning network is the parameter of admittance control; design the reward and punishment function of the deep reinforcement learning network III, the number of training rounds, and the maximum number of steps per training round;

[0027] The input and output of the deep reinforcement network III are as follows: for the single-axis hole insertion stage, the input of the deep reinforcement learning network III is set to S = [Px, Py, Pz, Rx, Ry, Fx, Fy, Fz, Tx, Ty]; for the multi-axis hole insertion stage, the input of the deep reinforcement learning network III is set to S = [Px, Py, Pz, Rx, Ry, Rz, Fx, Fy, Fz, Tx, Ty, Tz]; where Px, Py, and Pz are the robot end points, respectively. The positions of the end workpiece in the X-axis, Y-axis, and Z-axis in the Cartesian world coordinate system, Rx, Ry, and Rz are the rotation values ​​of the end workpiece of the robot around the X-axis, Y-axis, and Z-axis in the Cartesian world coordinate system, Fx, Fy, Fz and Tx, Ty, and Tz are the forces and moments of the end of the robot in the X-axis, Y-axis, and Z-axis directions respectively; after training through the reinforcement learning network III, the continuous stiffness coefficient and damping coefficient are output, where the stiffness coefficient and the damping coefficient are both within a certain range;

[0028] Preferably, the structure of the deep reinforcement learning network III selects the DDPG (Deep Deterministic Policy Gradient) network structure, which is divided into a critic module and an actor module, and contains four neural networks; the critic module updates the network parameter ω by using the temporal difference error method, and regularly copies ω to the target network. The actor module updates the network parameter θ by using the DPG algorithm, and the behavior strategy selects actions to act on the environment according to the output results of its policy network;

[0029] In the above, the reward function is where k n is the number of steps at the end of round n, k max is the maximum number of training steps set, Δh is the depth of the socket at the end of each round, and H is the depth of the set assembly completion; the entire reward function is divided into three parts. The first part is to give positive rewards based on the ratio of the number of assembly steps in each round to the set maximum number of steps. At the end of the assembly round, the more steps used, the less positive rewards are obtained; the second category is determined by the depth of the socket in each round. Within a reasonable threshold, the deeper the depth of the socket, the higher the positive reward; the last part is to give a -1 reward when the force exceeds the set threshold during the socket insertion process;

[0030] Step 8: During the hole insertion stage, the robot end is set to reach a certain insertion depth and the axis force / torque is within a certain threshold range, indicating that the insertion is successful. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present invention clearer, the specific implementation methods of the present invention are described in detail below:

[0032] Step 1: Divide the robot shaft hole assembly into three stages: the free near-hole stage outside the hole, the hole edge search stage, and the hole insertion stage; at the same time, distinguish the robot single-axis hole assembly and multi-axis hole assembly methods, and conduct experiments on both methods;

[0033] Step 2: Roughly locate the hole by visual positioning, control the robot movement, make the workpiece at the end of the robot reach the hole edge and contact the hole, and then the robot enters the hole edge search stage of assembly;

[0034] In the visual positioning method, the robot world coordinate system, tool coordinate system and workpiece coordinate system are first constructed, and the origin of the world coordinate system is the origin of the robot base coordinate system; secondly, markers are placed around the assembly holes, and the RGB image and depth information of the markers are obtained using a depth camera. The assembly holes are roughly positioned by camera calibration, depth image calibration, and hand-eye calibration and marker recognition in combination with the robot arm; the coordinates obtained by the rough positioning are the position and posture of the assembly holes in the robot world coordinate system;

[0035] Controlling the robot movement refers to controlling the robot movement with a program, using moveit in ROS to plan the trajectory, and realizing the movement of the workpiece at the end of the robot;

[0036] Step 3: In the hole edge search stage, firstly, design the hole search prior knowledge based on human hole search experience, preset the prior state and prior action, and form a prior knowledge memory bank. Secondly, collect the force / torque information and posture information of the robot as the input of the deep reinforcement learning network. The prior state classification model is used to determine whether the input is prior knowledge. If so, directly output the corresponding action. If not, input it into the deep reinforcement learning network for exploration and training. Output the robot position or posture adjustment strategy action so that the workpiece at the end of the robot can search holes smoothly. Construct the deep reinforcement learning network I for single-axis hole search and the deep reinforcement learning network II for multi-axis hole assembly respectively.

[0037] The prior actions of hole-searching prior knowledge based on human hole-searching experience include actions in four directions A=[+0.10, -0.10, +0.10, -0.10], which respectively represent the step displacements of the workpiece at the end of the robot in the X-axis and Y-axis directions in the Cartesian world coordinate system. After being classified by the prior state classification model, one of the above four actions is output to adjust the position of the end of the robot, and finally ensure that the robot can search holes smoothly;

[0038] The input S of the hole-searching prior knowledge prior state classification model based on human hole-searching experience is [P X , P Y , R X , R Y , R Z], where P X , P Y is the position of the robot end workpiece in the X-axis and Y-axis directions in the Cartesian world coordinate system, R X , R Y , R Z The rotation angles of the robot end workpiece around the X-axis, Y-axis, and Z-axis in the world coordinate system. The significance of adding hole-searching prior knowledge based on human hole-searching experience is that when the action that the robot end workpiece needs to adjust is the movement amount on the X-axis and Y-axis of the hole plane, it does not need to explore and learn the action through the deep reinforcement learning network, but adjusts it according to the action preset by human experience, thereby accelerating the exploration efficiency of deep reinforcement learning in the hole-searching process;

[0039] The prior state classification model adopts a support vector machine (SVM) model;

[0040] The force signal is collected by the force sensor at the end of the robot, including the contact force signal F of the workpiece at the end of the robot in the three coordinate axes X, Y, and Z directions. X , F Y , F Z , assembly torque signal T around the workpiece at the end of the robot X , T Y , T Z ;

[0041] The input S of the deep reinforcement learning network I for single-axis hole search is [P X , P Y , R X , R Y , R Z , F X , F Y , T X , T Y ], the input of the deep reinforcement learning network II for multi-axis hole assembly is S = [P X , P Y , R X , R Y , R Z , F X , F Y , T X , T Y , T Z ], where P X , P Y , R X , R Y , R Z is the position information of the end workpiece on the X and Y axes in the Cartesian world coordinate system, and the rotation amount around the X, Y, and Z axes. X , F Y , TX , T Y , T Z is the force and torque information of the robot end workpiece in the X-axis and Y-axis, and the torque information of the robot end workpiece in the Z-axis; the output of deep reinforcement learning network I A=[+0.05, -0.05, +0.05, -0.05], the output of deep reinforcement learning network II A=[+0.05, -0.05, +0.05, -0.05, +0.05, -0.05], which are the step adjustment amounts of the robot end workpiece around the positive and negative directions of the X-axis, around the positive and negative directions of the Y-axis, and around the positive and negative directions of the Z-axis respectively;

[0042] Both the deep reinforcement learning network I and the deep reinforcement learning network II adopt the DQN (Deep Q Network) network structure. The input of the DQN model undergoes three convolutional layers and two fully connected layers of nonlinear transformation, and finally obtains the Q value of each action value at the output layer. The optimal action is selected based on the Q value. The update method of the DQN algorithm is: Where s′ and a′ represent the state and action of the next moment, r and γ represent the reward function and decay factor, respectively; the loss function of the DQN algorithm is

[0043] Step 4: Design the reward and punishment function, the number of training rounds, and the maximum number of steps for each round of training for the deep reinforcement learning network in the hole edge search stage; use the deep reinforcement learning network I and the deep reinforcement learning network II to conduct exploration training on single-axis hole search and multi-axis hole search respectively until the number of steps and reward value of each round converge;

[0044] The reward function where k n is the number of steps in the nth round of training, and the maximum number of steps in each round is set to k max =40, number of episodes = 100. If the number of training steps and the reward function have not converged beyond this number of episodes, the training will be considered to have failed. The first type of reward function is positive reward, which is determined by the ratio between the number of steps in this round of training and the set maximum number of steps. The more steps are taken to complete the assembly in each round, the lower the positive reward value obtained in this round. The second type is negative reward. When the end of the robot rotates in the same direction for 6 consecutive times, the behavior is set to deviate from the center of the hole, and a -1 reward is given.

[0045] The exploration training process is as follows: first, the robot end is controlled to move toward the center of the hole according to the roughly positioned hole posture; after the initial movement, the force / torque information and posture information of the workpiece at the end of the robot are collected; and the parameters therein are used to determine whether the state at this time is a priori state; if it is a priori state, the next action is performed according to the prior action; if it is not a priori state, the state information is input into the deep reinforcement learning network training to obtain the next motion state, and so on are repeated. Each round of training is completed until the hole search is successful or the number of training steps per round reaches the set maximum number of steps, then the training of this round is terminated, the reward and punishment value is calculated, and then the next round of training is carried out; when the collected Z-axis force suddenly becomes smaller, it means that the hole search is successful in this round; when the optimal action strategy is trained, the reward function and the number of training steps converge, indicating that the training is successful, and the workpiece at the end of the robot enters the hole insertion stage;

[0046] Step 5: During the hole-insertion stage, a constant desired force is set in the Z-axis direction of the robot end, and the forces and torques in other directions are all 0. At the same time, the robot end is moved to the set desired position, and the force / torque information of the robot end during the hole-insertion process is collected. The collected force / torque of the robot end is compared with the set desired force to obtain the real-time force deviation; the force deviation is input into the admittance controller, and the adjusted posture of the workpiece at the robot end is output after calculation by the admittance control algorithm; at the same time, the admittance control parameters are optimized in real time through the deep reinforcement learning algorithm to realize variable admittance control;

[0047] The expected force F d =[0,0,f d , 0, 0, 0], the sensor collects the contact force F = [f x , f y , f z , T x , T y , T z ], the force deviation is ΔF=F d -F;

[0048] The admittance control algorithm is: Where M, B, and K represent the robot mass matrix, stiffness matrix, and damping matrix, respectively. d is the expected position of the robot end workpiece in the Cartesian world coordinate system, and X is the real-time position of the robot end workpiece in the Cartesian world coordinate system;

[0049] Step 6. Design a priori knowledge memory bank of jacks based on human jack experience, including prior states and prior experience values. Construct a classifier to identify the input state of the deep reinforcement learning network. If the input state is a priori state, directly output the prior experience value, and then directly control the end motion of the robot workpiece. If the input state is not a priori state, send it to the deep reinforcement learning network for training, and then control the end motion of the robot through the admittance controller. Define the admittance control parameters that need to be trained, and simulate the admittance control algorithm. By selecting different admittance control parameters for simulation, determine the admittance control parameters that need to be trained, and predetermine the range of change of the admittance control parameters.

[0050] The input of the prior state of the jack based on human jack experience is the position and posture S=[Px, Py, Pz, Rx, Ry, Rz, Fx, Fy, Fz, Tx, Ty, Tz] of the robot end workpiece in the Cartesian world coordinate system, wherein Px, Py, Pz are the positions of the robot end workpiece in the Cartesian world coordinate system on the X-axis, Y-axis, and Z-axis, respectively, Rx, Ry, Rz are the rotation amounts of the robot end workpiece around the X-axis, Y-axis, and Z-axis in the Cartesian world coordinate system, Fx, Fy, Fz and Tx, Ty, Tz are the forces and torques of the robot end in the X-axis, Y-axis, and Z-axis directions, respectively; the prior experience value of the jack based on human jack experience is A=[+0.05, -0.05, +0.05, -0.05], and the first two parameters and the last two parameters in A represent the positive and negative rotation amounts of the robot end workpiece around the X-axis and the positive and negative rotation amounts around the Y-axis in the Cartesian coordinate system, respectively;

[0051] The prior state classification model adopts a support vector machine (SVM) model;

[0052] The admittance control parameters include an inertia matrix, a damping matrix, and a damping matrix, wherein the inertia matrix is ​​a unit positive definite matrix; the training method thereof is to select one of the coefficient components in the damping matrix and one of the coefficient components in the damping matrix for training, and then obtain other coefficients in the damping matrix and the stiffness matrix according to the proportional relationship;

[0053] Step 7: Construct a deep reinforcement learning network III for variable admittance control that can output continuous actions, collect force / torque signals and posture signals during the plug-in process as state inputs for deep reinforcement learning, and the output of the deep reinforcement learning network is the parameter of admittance control; design the reward and punishment function of the deep reinforcement learning network III, the number of training rounds, and the maximum number of steps per training round;

[0054] The input and output of the deep reinforcement network III are: for the single-axis hole insertion stage, the input S of the deep reinforcement learning network III is set to [Px, Py, Pz, Rx, Ry, FX, F y, Fz, Tx, Ty], for the input S = [Px, Py, Pz, Rx, Ry, Rz, Fx, Fy, Fz, Tx, Ty, Tz] of the deep reinforcement learning network III in the multi-axis hole insertion stage; wherein Px, Py, Pz are the positions of the workpiece at the end of the robot in the Cartesian world coordinate system in the X-axis, Y-axis, and Z-axis, respectively, Rx, Ry, Rz are the rotation values ​​of the workpiece at the end of the robot around the X-axis, Y-axis, and Z-axis in the Cartesian world coordinate system, Fx, Fy, Fz and Tx, Ty, Tz are the forces and moments of the robot end in the X-axis, Y-axis, and Z-axis directions, respectively; after training through the reinforcement learning network III, continuous stiffness coefficient and damping coefficient are output, wherein the stiffness coefficient and damping coefficient are both within a certain range, the stiffness coefficient training action value range b∈(0.5, 5), and the damping coefficient training action value range is k∈(5, 75);

[0055] The structure of the deep reinforcement learning network III is the DDPG (Deep Deterministic Policy Gradient) network structure, which is divided into a critic module and an actor module, and contains four neural networks; the critic module updates the network parameter ω using the temporal difference error method and regularly copies ω to the target network. The actor module updates the network parameter θ using the DPG algorithm, and the behavior strategy selects actions to act on the environment based on the output of its policy network;

[0056] In the above, the reward function is where k n is the number of steps at the end of round n, k max =60 is the maximum number of training steps set, and the number of episodes is Episode = 120; Δh is the depth of the jack at the end of each round, and H is the depth of the set assembly completion; the entire reward function is divided into three parts. The first part is to give positive rewards based on the ratio of the number of assembly steps in each round to the set maximum number of steps. At the end of the assembly round, the more steps used, the less positive rewards are obtained; the second category is determined based on the depth of the jack in each round. Within a reasonable threshold, the deeper the depth of the jack, the higher the positive reward; the last part is to give a -1 reward, which means that the force exceeds the set threshold during the jacking process;

[0057] Step 8: During the hole insertion stage, the robot end is set to reach a certain insertion depth and the axis force / torque is within a certain threshold range, indicating that the insertion is successful.

Claims

1. A robot shaft hole assembly method based on deep reinforcement learning and admittance control, characterized in that: The following steps are involved: Step 1, the robot shaft hole assembly is divided into three stages: the free near hole stage outside the hole, the hole edge search stage, and the hole insertion stage; at the same time, the method of distinguishing the robot single-axis hole assembly and multi-axis hole assembly; Step 2: Roughly locate the assembly hole by visual positioning, control the robot movement, make the workpiece at the end of the robot reach the position of the hole edge, and the robot enters the hole edge search stage of assembly; Step 3: In the hole edge search stage, firstly, the hole search prior knowledge based on human hole search experience is designed, and the prior state and prior action are preset to form a prior knowledge memory bank. Secondly, the force / torque information and posture information of the robot are collected as the input of the deep reinforcement learning network. The prior state classification model is used to determine whether the input is prior knowledge. If so, the corresponding action is directly output. If not, it is input into the deep reinforcement learning network for exploration and training, and the robot position or posture adjustment strategy action is output, so that the workpiece at the end of the robot can search holes smoothly. The deep reinforcement learning network I for single-axis hole search and the deep reinforcement learning network II for multi-axis hole assembly are constructed respectively. Step 4: Design the reward and punishment function, the number of training rounds, and the maximum number of steps for each round of training for the deep reinforcement learning network in the hole edge search stage; use the deep reinforcement learning network I and the deep reinforcement learning network II to conduct exploration training on single-axis hole search and multi-axis hole search respectively until the number of steps and reward value of each round converge; Step 5: During the hole-insertion stage, a constant desired force is set in the Z-axis direction of the robot end, and the forces and torques in other directions are all 0. At the same time, the robot end is moved toward the set desired position, and the force / torque information of the robot end during the hole-insertion process is collected. The collected force / torque of the robot end is compared with the set desired force to obtain the real-time force deviation; the force deviation is input into the admittance controller, and the adjusted posture of the workpiece at the robot end is output after calculation by the admittance control algorithm; at the same time, the admittance control parameters are optimized in real time through the deep reinforcement learning algorithm to realize variable admittance control; Step 6. Design a priori knowledge memory bank of jacks based on human jack experience, including prior states and prior experience values. Construct a classifier to identify the input state of the deep reinforcement learning network. If the input state is a priori state, directly output the corresponding prior experience value, and then directly control the movement of the robot workpiece end. If the input state is not a priori state, send it to the deep reinforcement learning network for training to obtain the admittance control parameters. Define the admittance control parameters that need to be trained, and simulate the admittance control algorithm. By selecting different admittance control parameters for simulation, determine the admittance control parameters that need to be trained, and predetermine the range of variation of the admittance control parameters. Step 7: Construct a deep reinforcement learning network III for variable admittance control that can output continuous actions, collect force / torque signals and posture signals during the plug-in process as state inputs for deep reinforcement learning, and the output of the deep reinforcement learning network is the parameter of admittance control; design the reward and punishment function of the deep reinforcement learning network III, the number of training rounds, and the maximum number of steps per training round; Step 8: During the hole insertion stage, the robot end is set to reach a certain insertion depth and the axis force / torque is within a certain threshold range, indicating that the insertion is successful.

2. According to the robot shaft hole assembly method based on deep reinforcement learning and admittance control as described in claim 1, it is characterized in that: In step 3, the prior action based on human hole searching experience is the step displacement of the robot end workpiece in the X-axis and Y-axis directions in the Cartesian world coordinate system, and the input of the prior state classification model includes the position of the robot end workpiece in the X-axis and Y-axis directions in the Cartesian world coordinate system, and the rotation angles around the X-axis, Y-axis, and Z-axis directions.

3. According to the robot shaft hole assembly method based on deep reinforcement learning and admittance control as described in claim 1, it is characterized in that: In step 3, the input of the deep reinforcement learning network I is the position information of the robot's end workpiece in the X and Y axes in the Cartesian world coordinates, the rotation amounts around the X, Y and Z axes, and the force and torque information of the robot's end workpiece in the X and Y axes; the output includes four actions, which are the step adjustment amounts of the robot's end workpiece in the positive and negative directions around the X axis and the positive and negative directions around the Y axis.

4. According to the robot shaft hole assembly method based on deep reinforcement learning and admittance control as described in claim 1, it is characterized in that: In step 3, the input of the deep reinforcement learning network II is the position of the robot's end workpiece on the X-axis and Y-axis in the Cartesian world coordinates, the rotation angles around the X-axis, Y-axis, and Z-axis, the force of the robot's end workpiece on the X-axis and Y-axis, and the torque of the robot's end workpiece on the X-axis, Y-axis, and Z-axis; the output of the deep reinforcement learning network II is six actions, which are the positive and negative step adjustments of the robot's end workpiece rotating around the X-axis, Y-axis, and Z-axis.

5. According to the robot shaft hole assembly method based on deep reinforcement learning and admittance control as described in claim 1, it is characterized in that: In step 4, the reward and punishment function is divided into two types of rewards. The first type is positive reward, which is determined by the ratio between the number of steps trained in this round and the set maximum number of steps. The more steps are taken to complete the assembly in each round, the lower the positive reward value obtained in this round; the second type is negative reward. When the end of the robot rotates m times in a row in the same direction, the behavior is set to deviate from the center of the hole, and a -1 reward is given; when the collected Z-axis force suddenly becomes smaller, it means that the hole search in this round is successful, and the hole search in this round ends.

6. According to the robot shaft hole assembly method based on deep reinforcement learning and admittance control as described in claim 1, it is characterized in that: In step 6, the input of the prior state classifier is the position and posture of the robot end workpiece in the Cartesian world coordinate system, and the force / torque information acting on the robot end workpiece; the prior experience value is the positive and negative rotation amount of the robot end workpiece around the X-axis and the positive and negative rotation amount around the Y-axis.

7. According to the robot shaft hole assembly method based on deep reinforcement learning and admittance control as described in claim 1, it is characterized by: In step 6, one of the coefficient components in the damping matrix and one of the coefficient components in the damping matrix are selected for training, and then other coefficients in the damping matrix and the stiffness matrix are obtained according to the proportional relationship.

8. According to the robot shaft hole assembly method based on deep reinforcement learning and admittance control as described in claim 1, it is characterized in that: In the step 7, for the single-axis hole plugging stage, the input S of the deep reinforcement learning network III is set to [Px, Py, Pz, Rx, Ry, Fx, Fy, Fz, Tx, Ty], and for the multi-axis hole plugging stage, the input S of the deep reinforcement learning network III is set to [Px, Py, Pz, Rx, Ry, Rz, Fx, Fy, Fz, Tx, Ty, Tz]; wherein Px, Py, and Pz are the positions of the workpiece at the end of the robot in the Cartesian world coordinate system in the X-axis, Y-axis, and Z-axis, respectively, Rx, Ry, and Rz are the rotation values ​​of the workpiece at the end of the robot around the X-axis, Y-axis, and Z-axis in the Cartesian world coordinate system, respectively, and Fx, Fy, Fz and Tx, Ty, and Tz are the forces and torques of the robot end in the X-axis, Y-axis, and Z-axis directions, respectively; After training through the reinforcement learning network III, continuous stiffness coefficient and damping coefficient are output, where the stiffness coefficient and the damping coefficient are both within a certain range.

9. The robot shaft hole assembly method based on deep reinforcement learning and admittance control according to claim 1, characterized in that: In step 7, the reward function is where k n is the number of steps at the end of each round, k max is the set maximum number of training steps, Δh is the depth of the hole at the end of each round, and H is the set depth of assembly completion; the entire reward function is divided into three parts. The first part is to give positive rewards based on the ratio of the number of assembly steps in each round to the set maximum number of steps. At the end of the assembly round, the more steps used, the less positive rewards are obtained; the second type is determined according to the depth of the hole in each round. Within a reasonable threshold, the deeper the depth of the hole, the higher the positive reward; the last part is to give a -1 reward if the force exceeds the set threshold during the insertion process.

Images