Five-finger dexterous arm-cellphone robot grabbing method based on teaching learning

Abstract

The application discloses a five-finger dexterous arm-mobile phone robot grabbing method based on teaching learning, and belongs to the field of intelligent robots. The method comprises three parts of teaching of a mechanical arm end trajectory, teaching of a dexterous hand and demonstration of a grabbing task. The teaching of the mechanical arm is first based on dynamic motion primitive learning of a teaching path feature collected offline, and a new mechanical arm end trajectory is generated in combination with a position of a grabbed object perceived online by a depth camera; action teaching of the dexterous hand is based on an action reorientation neural network, action posture information of a teacher's hand is extracted, joint parameter suitable for the dexterous hand to grab different objects is generated; simulation demonstration combines the separately taught methods of the arm and the hand according to a grabbing task flow of the arm-mobile phone robot, generates motion parameters and transmits the motion parameters into a simulation scene, the arm-mobile phone robot performs operations step by step according to the generated parameters, successfully approaches and grabs a target object, and finally realizes the grabbing task of the arm-mobile phone robot based on teaching learning.

Description

Technical Field

[0001] This invention relates to the field of intelligent robots, and in particular to a grasping method for a five-finger dexterous arm-hand robot based on teaching learning. Background Technology

[0002] With the rise of the information and intelligent era, robots are not only playing a vital role in industrial production, but are also gradually permeating all aspects of people's daily lives. Through teaching and learning, robots can mimic the operations of human arms and hands, enabling them to complete relatively complex grasping tasks and providing people with more convenient services.

[0003] Currently, mainstream arm-hand robot teaching and learning methods can be broadly categorized based on the teaching equipment used: vision-based, teleoperation-based, and human-machine physical interaction-based teaching and learning. Vision-based and teleoperation-based teaching and learning use the motion data of a human operator's hand and arm as the learning object. However, due to differences in size, structure, and range of motion between the human operator and the arm-hand robot, directly using the human operator's data to control the robot's movement can lead to problems such as distorted movements and unstable object grasping. Teleoperation-based teaching and learning involves a human operator dragging the robot and recording the robot's motion data to reproduce the task. However, this teaching method is relatively complex, generates simple teaching trajectories, and has poor generalization ability.

[0004] To address the problems encountered during the teaching and learning process of the arm-hand robot, this invention introduces a five-finger dexterous arm-hand robot grasping method based on teaching and learning. This method not only improves the accuracy of robot teaching and learning but also enables the robot to have a certain generalization ability, allowing it to grasp objects of different sizes and positions. Summary of the Invention

[0005] This invention provides a grasping method for a five-finger dexterous arm-hand robot based on teaching learning, which aims to improve the teaching learning accuracy of the robot and enable the robot to have a certain generalization ability, so that it can achieve natural and stable grasping when facing unknown objects.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a grasping method for a five-finger dexterous arm-hand robot based on teaching learning, the specific steps of which are as follows:

[0007] Step S1: Build a robot grasping task scenario, and then divide the robot grasping task process into two parts that can be executed independently: a robotic arm and a dexterous hand.

[0008] Step S2: Based on the sub-tasks divided in Step S1, the running trajectory of the robotic arm is first taught. After obtaining the teaching data of the robotic arm end-effector trajectory through human-computer physical interaction, the path shape of the teaching trajectory is learned based on the dynamic motion primitive method. Combined with the grasping point position sensed by the RGB-D camera, the running trajectory of the robotic arm end-effector reaching the target position of the grasping point is generated. The robot efficiently approaches the grasping object at different positions according to the generated different trajectories.

[0009] Step S3: After the robotic arm reaches the target position through step S2, it first uses the MediaPipe Hands algorithm to obtain the key point three-dimensional coordinate data representing the current teacher's hand movement posture. Then, it builds a motion redirection neural network to process the collected hand teaching data and map the current teacher's hand movement onto the dexterous hand robot, thereby realizing the arm-hand robot's grasping of objects.

[0010] Step S4: Based on the task flow divided in Step S1, combine Step S2 and Step S3 to form a complete arm-hand robot grasping task flow. The robot performs operations step by step according to the flow, successfully approaches and grasps the target object, and finally successfully completes the entire grasping task.

[0011] Furthermore, the specific steps of step S1 are as follows:

[0012] Step S1-1: Build a simulation scene in the Gazebo simulation environment that includes an arm-hand robot, a depth camera, and a grasping object;

[0013] Step S1-2: Divide the object grasping process of the arm-hand robot in this task scenario into the motion processes of the robotic arm and the dexterous hand;

[0014] Furthermore, the specific steps of step S2 are as follows:

[0015] Step S2-1: Using a human-computer physical interaction teaching method, drag the robotic arm robot in the teaching mode to approach and grasp the object in a natural way, and record the three-dimensional coordinate data of the trajectory of the robotic arm end effector during operation.

[0016] Step S2-2: Based on the following DMP dynamic system equations, learn the characteristics of the teaching data;

[0017]

[0018]

[0019] Where τ is the total time, K is the elastic coefficient, D is the damping coefficient, and ω i These are the weight parameters to be learned. The mean is μ i Width is h i The Gaussian kernel function is given, where g is the desired endpoint position, y0 is the starting position, α is a coefficient variable that adjusts the system's convergence speed, and x acts as an internal clock to adjust the speed of the newly generated trajectory. The position, velocity, and acceleration y of the taught trajectory at the end effector of the robotic arm are known. demo (m) m = 1, 2, 3, ..., M; this can be solved by solving the equation: The weights ω of the nonlinear term f(x) in the system equation are obtained. i ;

[0020] Step S2-3: Use a depth camera to identify and obtain the three-dimensional coordinates of the grasped object relative to the robot arm's base coordinate system;

[0021] Step S2-4: Combining the initial position of the robot's end effector and the position of the target grasping point in the task environment, the learned DMP weight parameters ω are used... i This generates a new grasping motion trajectory for the robotic arm, and the end effector of the robotic arm moves according to the generated trajectory to grasp objects at different positions.

[0022] Furthermore, the specific steps of step S3 are as follows:

[0023] Step S3-1: Use an RGB camera to capture a video of the demonstrator's hand demonstration, and then use the MediaPipe Hands algorithm to identify and obtain the three-dimensional coordinate data of the key points of the hand in the image;

[0024] Step S3-2: Build a neural network model. The main part of the network is an encoder-decoder structure similar to a variational autoencoder. First, start from the input image I H Extracting attitude information D H Then, after encoding, decoding, reconstruction, and range correction, the joint parameters θ corresponding to the dexterous hand robot are obtained. R A loss function is introduced to optimize the parameters of the main body of the neural network; the joint parameters θ are optimized using an FK layer. R Convert to robot pose D R The FK layer is the formula for calculating the robot's forward kinematics based on the DH parameter method:

[0025]

[0026]

[0027] Where Q(0) represents the joint coordinates in the nth coordinate system, a n and θ n This represents the rotation angles around the X and Z axes between two adjacent reference coordinate systems, a.n and d n These represent the translations along the X and Z axes between two adjacent reference coordinate systems. The DH matrix can be used to transform the joint coordinates in each sub-coordinate system to the wrist-based coordinate system of the dexterous hand robot. The difference between the robot's pose and the instructor's hand pose is calculated using a loss function, as follows:

[0028] L=λ we L we +λ wj L wj +λ col L col

[0029] Among them, L we L wj L col The sub-tables represent the losses from fingertips, finger joints, and finger self-collision, λ we , λ wj , λ col The corresponding weight parameters.

[0030] Furthermore, the specific steps of step S3 are as follows:

[0031] Step S4-1: According to the task flow divided in Step 1, combine the teaching and learning processes of Step S2 and Step S3, and perform online teaching and learning of the robotic arm and dexterous hand separately in the corresponding steps to generate the motion parameters of the robot; Step S4-2: Apply the generated motion parameters to the arm-hand robot grasping task scenario built in S1 to realize the arm-hand robot grasping task in the Gazebo simulation environment.

[0032] Compared with the prior art, the present invention has at least the following advantages:

[0033] First, the teaching and learning framework of the arm-hand robot enables the robot to learn actions that are more reasonable and natural. Second, further mapping processing of the teaching data allows the robot's teaching and learning to go beyond simply replicating the taught task, and also to have a certain generalization ability, enabling it to grasp different objects in different positions, thus improving the intelligence of the arm-hand robot and making its grasping efficiency higher. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the grasping method of the five-finger dexterous arm robot based on teaching learning according to the present invention;

[0035] Figure 2 This is a flowchart of the grasping task of the five-finger dexterous arm robot of the present invention;

[0036] Figure 3This is a teaching structure diagram of the robotic arm based on depth camera perception and DMP method of the present invention;

[0037] Figure 4 This invention provides the robotic arm end-effector teaching and the trajectory path diagram generated based on DMP.

[0038] Figure 5 This is a diagram of the neural network structure for teaching the dexterous hand in this invention;

[0039] Figure 6 This is a schematic diagram illustrating how the five-fingered dexterous hand robot of the present invention can avoid self-collisions;

[0040] Figure 7 These are several basic posture diagrams of the five-finger dexterity hand generated after instruction according to the present invention.

[0041] Figure 8 These are illustrations showing the operation of the five-finger dexterous hand robot of the present invention grasping different objects.

[0042] Figure 9 This is a diagram illustrating the object grasping operation effect of the five-finger dexterous arm-hand robot of the present invention. Detailed Implementation

[0043] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0044] Figure 1 This is a schematic diagram of the grasping method for a five-finger dexterous arm robot based on teaching learning according to the present invention. It includes three parts: robotic arm teaching learning, dexterous hand teaching learning, and simulation demonstration. First, a simulation scenario for the arm-hand robot is built. For this scenario, the task flow of the arm-hand robot is divided into two parts: the robotic arm and the dexterous hand, which are performed separately. Then, the robotic arm and the dexterous hand robot are taught separately. The robotic arm part acquires teaching data through human-machine physical interaction. Then, based on the Dynamic Motion Primitives (DMP) method, the features of the teaching trajectory are learned. Combined with the initial position of the robotic arm's end effector and the position of the object to be grasped, a new motion trajectory similar to the shape of the teaching trajectory is generated. The robot approaches and grasps the object according to the generated trajectory. The dexterous hand part is based on a graph neural network. It maps the collected hand movement data of the teacher. Considering the differences in structural size and range of motion between the human hand and the dexterous hand, joint angle data suitable for the dexterous hand's motion control is generated. The dexterous hand robot performs the object grasping operation according to the generated joint data. Finally, the robot motion parameters generated from the two teaching parts are combined and passed to the robot for execution, thus completing the grasping task of the arm-hand robot. Figure 2 A flowchart illustrating how an arm-hand robot performs a grasping task with its robotic arm and dexterous hand distributed throughout. Figure 3This is a teaching structure diagram of the robotic arm combining a depth camera and a DMP according to the present invention. Figure 4 This invention provides the robotic arm end-effector teaching and the trajectory path diagram generated based on DMP. Figure 5 This is a diagram of the neural network structure for teaching the dexterous hand in this invention; Figure 6 This is a schematic diagram illustrating how the five-fingered dexterous hand robot of the present invention can avoid self-collisions; Figure 7 These are several basic posture diagrams generated by the dexterous hand of the present invention after being taught. Figure 8 These are operational illustrations of the five-fingered dexterous hand robot of the present invention grasping different objects; Figure 9 This is a diagram illustrating the object grasping operation of the five-finger dexterous arm robot of the present invention.

[0045] This invention proposes a grasping method for a five-finger dexterous arm-hand robot based on teaching-learning, the specific steps of which are as follows:

[0046] Step S1: First, build a robot grasping task scenario, and then divide the robot grasping task process into two parts that can be executed independently: the robotic arm and the dexterous hand.

[0047] Step S1-1: Build a simulation scene in Gazebo that includes an arm-hand robot, a depth camera, and a grasping object;

[0048] Step S1-2: For this task scenario, the process of the arm-hand robot grasping an object is divided into step-by-step task flows for the robotic arm and the dexterous hand. The resulting motion flow diagram is shown below. Figure 2 As shown.

[0049] Step S2: Based on the sub-tasks divided in Step S1, the robotic arm's trajectory is first taught. After acquiring the robotic arm's end-effector trajectory teaching data through human-machine physical interaction, the path shape of the taught trajectory is learned using the Dynamic Motion Primitives (DMP) method. Combined with the grasping point position sensed by the RGB-D camera, the trajectory of the robotic arm's end-effector reaching the target grasping point is generated, enabling the robot to approach and grasp the object more efficiently. The robotic arm's teaching process is as follows: Figure 3 As shown.

[0050] Step S2-1: Using a human-machine physical interaction teaching method, drag the robotic arm robot in teaching mode to approach and grasp the object in a manner similar to the movement habits of a human arm, and record the trajectory coordinate data and other data during the operation of the robotic arm end effector.

[0051] Step S1-2: Learn the features of the teaching data based on the DMP dynamic system equations:

[0052]

[0053]

[0054] Where τ is the total time, typically taken as 1, K is the elastic coefficient, D is the damping coefficient, and ω... i These are the weight parameters to be learned. The mean is μ i Width is h i The Gaussian kernel function is given, where g is the desired endpoint position, y0 is the starting position, α is a coefficient variable that adjusts the system's convergence speed, and x acts as an internal clock that can adjust the speed of the newly generated trajectory. The position, velocity, and acceleration y of the taught trajectory at the end effector of the robotic arm are known. demo (m) m = 1, 2, 3, ..., M. This can be solved by solving the equation: The weights ω of the nonlinear term f(x) in the system equation are obtained. i ;

[0055] Step S2-3: Use a depth camera to identify and capture the three-dimensional coordinates of the object. First, obtain the two-dimensional planar pixel coordinates of the target object, then obtain the three-dimensional coordinates in the camera coordinate system through camera focal length transformation, and finally obtain the three-dimensional coordinates relative to the robot arm base coordinate system through transformation matrix transformation.

[0056] Step S2-4: Combining the initial position of the robot's end effector in the task environment and the target grasping point position identified by the depth camera, the learned DMP weight parameters ω are used to... i Generate a new grasping motion trajectory for the robotic arm, including the robotic arm end-effector teaching trajectory and a new running trajectory generated based on DMP. Figure 4 As shown.

[0057] Step S3: After the robotic arm reaches the target position through step S2, it first uses the MediaPipe Hands algorithm to obtain the key point three-dimensional coordinate data representing the current teacher's hand movement posture. Then, it builds a motion redirection neural network to process the collected hand teaching data and map the current teacher's hand movement onto the dexterous hand robot, thereby realizing the arm-hand robot's grasping of the object.

[0058] Step S3-1: Use an RGB camera to capture a video of the demonstrator's hand demonstration, and then use the MediaPipe Hands algorithm to identify and obtain the three-dimensional coordinate data of the key points of the hand in the image;

[0059] Step S3-2: Build a... Figure 5 The neural network model shown has a main body similar to the encoding and decoding structure of a variational autoencoder (VAE). This network first processes the input image I... H Extracting attitude information D HThen, after encoding, decoding, reconstruction, and range correction, the joint parameters θ corresponding to the dexterous hand robot are obtained. R The range is in θ lower <θ R <θ upper Within, θ lower and θ upper These represent the upper and lower limits of finger joint movement in a dexterous hand robot.

[0060] To better reconstruct the instructor's hand posture, a loss function needs to be introduced to optimize the parameters of the main part of the neural network. First, the joint parameters θ are calculated using an FK layer. R Convert to robot pose D R The FK layer is a formula for calculating the robot's forward kinematics based on the DH parameter method, as follows:

[0061]

[0062]

[0063] Where Q(0) represents the joint coordinates in the nth coordinate system, a n and θ n This represents the rotation angles around the X and Z axes between two adjacent reference coordinate systems, a. n and d n It represents the translation along the X and Z axes between two adjacent reference coordinate systems. The DH matrix can be used to transform the joint coordinates in each sub-coordinate system to the robot base coordinate system.

[0064] The difference between the robot's pose and the instructor's hand pose is calculated using a reconstruction loss function, as follows:

[0065] L=λ we L we +λ wj L wj +λ col L col

[0066]

[0067]

[0068]

[0069] Among them, L we L wj L col The sub-tables represent the losses from fingertips, finger joints, and finger self-collision, λ we , λ wj , λ colThe corresponding weight parameters. The fingertip loss is for reconstructing the robot's fingertip. The fingertips between the instructor's fingers Relative to their respective wrist positions loss, l we The length from the fingertip to the wrist is used to ensure that the dexterous hand robot can contact and grasp objects as in the taught action; finger joint loss refers to the reconstruction of the robot's finger joint positions. The position of the finger joints between the demonstrator's joints Relative to their respective wrist positions loss, l wj The length from the finger joint to the wrist is used to maintain the consistency of the movements of each joint of the dexterous hand robot with the taught movements, ensuring the stability of object grasping; self-collision loss is the loss caused by self-collision between the fingers of the reconstructed robot due to the reconstruction deformation. When the joint position d between the fingers... i，j Less than the minimum safe distance d min This will result in an exponential loss. Considering self-collision loss can prevent dexterous hand robots from suffering such losses. Figure 6 Damage to the robot caused by self-collision. Through neural network mapping, dexterous hand robots can learn, for example... Figure 7 The shown postures and movements, and successfully grasped, such as Figure 8 The different objects shown.

[0070] Step S4: Based on the task flow divided in Step S1, combine Step S2 and Step S3 to form a complete arm-hand robot grasping task flow. The robot performs operations step by step according to the flow, successfully approaches and grasps the target object, and finally successfully completes the entire grasping task.

[0071] Step S4-1: According to the task flow divided in Step 1, combine the teaching and learning processes of Step S2 and Step S3, and perform online teaching and learning of the robotic arm and dexterous hand separately in the corresponding steps to generate the robot's motion parameters.

[0072] Step S4-2: Apply the generated motion parameters to the arm-hand robot grasping task scenario built in Step S1. The robot executes the operations step by step according to the process, successfully approaching and grasping the target object, ultimately realizing the arm-hand robot grasping task in the Gazebo simulation environment. The grasping effect is as follows: Figure 9 As shown.

Claims

1. A grasping method for a five-finger dexterous arm-hand robot based on teaching-learning, characterized in that, The specific steps are as follows: Step S1: Build a robot grasping task scenario, and then divide the robot grasping task process into two parts that can be executed independently: a robotic arm and a dexterous hand. Step S2: Based on the sub-tasks divided in Step S1, the running trajectory of the robotic arm is first taught. After obtaining the teaching data of the robotic arm end-effector trajectory through human-computer physical interaction, the path shape of the teaching trajectory is learned based on the dynamic motion primitive method. Combined with the grasping point position sensed by the RGB-D camera, the running trajectory of the robotic arm end-effector reaching the target position of the grasping point is generated. The robot efficiently approaches the grasping object at different positions according to the generated different trajectories. Step S3: After the robotic arm reaches the target position through step S2, it first uses the MediaPipe Hands algorithm to obtain the key point three-dimensional coordinate data representing the current teacher's hand movement posture. Then, it builds a motion redirection neural network to process the collected hand teaching data and map the current teacher's hand movement onto the dexterous hand robot, thereby realizing the arm-hand robot's grasping of objects. Step S4: Based on the task flow divided in step S1, combine steps S2 and S3 to form a complete arm-hand robot grasping task flow. The robot performs operations step by step according to the flow, successfully approaches and grasps the target object, and finally successfully completes the entire grasping task. The specific steps of step S3 are as follows: Step S3-1: Use an RGB camera to capture a video of the demonstrator's hand demonstration, and then use the MediaPipe Hands algorithm to identify and obtain the three-dimensional coordinate data of the key points of the hand in the image; Step S3-2: Build a neural network model. The main part of the network is an encoder-decoder structure similar to a variational autoencoder. First, start from the input image I H Extracting attitude information D H Then, after encoding, decoding, reconstruction, and range correction, the joint parameters θ corresponding to the dexterous hand robot are obtained. R A loss function is introduced to optimize the parameters of the main body of the neural network; the joint parameters θ are optimized using an FK layer. R Convert to robot pose D R The FK layer is the formula for calculating the robot's forward kinematics based on the DH parameter method: Where Q(0) represents the joint coordinates in the nth coordinate system, a n and θ n This represents the rotation angles around the X and Z axes between two adjacent reference coordinate systems, a. n and d n These represent the translations along the X and Z axes between two adjacent reference coordinate systems. The DH matrix can be used to transform the joint coordinates in each sub-coordinate system to the wrist-based coordinate system of the dexterous hand robot. The difference between the robot's pose and the instructor's hand pose is calculated using a loss function, as follows: L=λ we L we +λ wj L wj +λ col L col Among them, L we L wj L col The sub-tables represent the losses from fingertips, finger joints, and finger self-collision, λ we , λ wj , λ col The corresponding weight parameters.

2. The grasping method of a five-finger dexterous arm-hand robot based on teaching learning as described in claim 1, wherein step S1 is as follows: Step S1-1: Build a simulation scene in the Gazebo simulation environment that includes an arm-hand robot, a depth camera, and a grasping object; Step S1-2: Divide the object grasping process of the arm-hand robot in this task scenario into the motion flow of the robotic arm and the dexterous hand.

3. The grasping method of a five-finger dexterous arm-hand robot based on teaching learning as described in claim 1, wherein step S2 is as follows: Step S2-1: Using a human-computer physical interaction teaching method, drag the robotic arm robot in the teaching mode to approach and grasp the object in a natural way, and record the three-dimensional coordinate data of the trajectory of the robotic arm end effector during operation. Step S2-2: Based on the following DMP dynamic system equations, learn the characteristics of the teaching data; Where τ is the total time, K is the elastic coefficient, D is the damping coefficient, and ω i These are the weight parameters to be learned. The mean is μ i Width is h i The Gaussian kernel function is given, where g is the desired endpoint position, y0 is the starting position, α is a coefficient variable that adjusts the system's convergence speed, and x acts as an internal clock to adjust the speed of the newly generated trajectory. The position, velocity, and acceleration y of the taught trajectory at the end effector of the robotic arm are known. demo (m) m = 1, 2, 3, ..., M; this can be solved by solving the equation: The weights ω of the nonlinear term f(x) in the system equation are obtained. i ; Step S2-3: Use a depth camera to identify and obtain the three-dimensional coordinates of the grasped object relative to the robot arm's base coordinate system; Step S2-4: Combining the initial position of the robot's end effector and the position of the target grasping point in the task environment, the learned DMP weight parameters ω are used... i This generates a new grasping motion trajectory for the robotic arm, and the end effector of the robotic arm moves according to the generated trajectory to grasp objects at different positions.

4. The grasping method of a five-finger dexterous arm-hand robot based on teaching learning as described in claim 1, wherein step S3 is as follows: Step S4-1: According to the task flow divided in Step 1, combine the teaching and learning processes of Step S2 and Step S3, and execute the online teaching and learning of the robotic arm and dexterous hand separately in the corresponding steps to generate the robot's motion parameters. Step S4-2: Apply the generated motion parameters to the arm-hand robot grasping task scenario built in S1 to realize the arm-hand robot grasping task in the Gazebo simulation environment.

Images