A dexterous hand teleoperation method based on grasping motion prior
By generating high-quality grasping postures and fusing multimodal features, the problem of unstable grasping in visual gesture teleoperation is solved, achieving high success rate and smooth grasping motion, which is applicable to a variety of dexterous hand platforms.
Patent Information
- Application Number
- CN202610463013.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-09
- Publication Date
- 2026-06-26
AI Technical Summary
Existing visual gesture teleoperation methods suffer from unstable gesture estimation under complex lighting, occlusion, or rapid movement conditions, and lack prior knowledge of motion for grasping tasks, resulting in low grasping success rates and difficulty in cross-domain transfer.
By generating high-quality grasping postures, combining least squares optimization and multimodal feature fusion, grasping motion priors are constructed. Transformer temporal modeling is used to correct trajectory deviations caused by perceived noise, which are then deployed on the robotic arm for real-time correction.
It significantly improves the success rate of grasping and the smoothness of the trajectory, and significantly enhances the cross-domain generalization ability. The grasping success rate has increased from 10% to over 90%, and it is suitable for a variety of dexterous hand platforms.
Smart Images
Figure CN122274971A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of robot teleoperation and machine learning, and in particular to a dexterous hand teleoperation method based on grasping motion priors. Background Technology
[0002] Traditional teleoperation methods for robotic dexterous hands mainly fall into three categories: mechanical "twin hands" (such as TeleHand, whose structure corresponds one-to-one with the robotic hand), glove / exoskeleton methods (such as gloves equipped with sensors or force feedback), and camera + gesture reconstruction methods (unmarked, based on 3D gesture estimation). Among these, teleoperation methods based on visual gesture reconstruction have advantages such as not requiring the use of wearing devices, being intuitive and natural, and being easy to deploy, making them particularly suitable for long-term, multi-person collaborative scenarios.
[0003] However, existing visual gesture teleoperation methods still have significant drawbacks: First, the jitter and delay of the gesture estimation model under complex lighting, occlusion, or rapid movement lead to unstable joint trajectories and low grasping success rate; second, the lack of prior knowledge of motion for grasping tasks makes it difficult to automatically correct trajectory deviations introduced by perceptual noise, resulting in frequent grasping failures or collisions; third, the large gap between the simulation and real domains makes it difficult to directly transfer well-trained strategies to physical dexterity hands.
[0004] In recent years, data-driven grasping pose generation methods (such as GraspNet) have been able to generate reasonable grasping poses for object point clouds, but they only focus on the final state and ignore the motion process. Human motion capture priors have been shown to improve the smoothness and success rate of robot motion control, but existing work mostly focuses on the motion control of the robot itself, lacking modeling of grasping motion priors for high-degree-of-freedom dexterous hands. Therefore, how to construct grasping motion priors and integrate them into the teleoperation framework to improve trajectory correction capabilities under perceived noise and cross-domain generalization performance is a pressing technical problem in this field. Summary of the Invention
[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a dexterous hand telemanipulation method based on grasping motion priors.
[0006] The objective of this invention is achieved through the following technical solution: a dexterous hand telemanipulation method based on grasping motion prior, comprising: Randomly generate physically feasible target grasping postures. For each grasping task, initialize the hand posture. Based on the initial posture parameters and the target grasping posture, use least squares optimization to generate the true trajectory of the grasping posture with MANO parameters of the human hand and the noisy grasping reference trajectory. The MANO parameters of the hand generated from the true trajectory and reference trajectory are redirected to the five-finger manipulator to obtain the true trajectory and reference trajectory of the manipulator joints. Deploy the robot arm model to execute the ground truth trajectory, and collect trajectory datasets of images, robot arm proprioception, robot arm reference posture, and robot arm ground truth posture; Image features, ontology features, and reference features are extracted from the dataset and concatenated. The concatenated features are then input into the prediction model to predict the robot's pose at the current moment, and the model is trained using the ground truth pose. The pre-trained network is deployed to receive reference postures obtained from visual images, proprioception, and gesture estimation in real time, and outputs corrected joint commands to the robot controller.
[0007] Furthermore, the random generation of physically feasible target grasping postures includes: Input object point cloud and contact map, and generate high-quality human hand grasping pose parameters through GraspNet-1 Billion network. , where the attitude vector Represents the axial rotation of 16 joints, shape vector Represents the principal components of the hand shape, and the global rotation vector. This indicates the overall rotation of the wrist.
[0008] Furthermore, the ground truth trajectory of the grasping posture generated using least-squares optimization to generate the MANO parameters of the human hand includes: Each MANO dimension is parameterized using low-order polynomials to define the trajectory. ,in As basis functions, For coefficients, The optimization objective is in the form of weighted least squares: in For linear interpolation approximation, , and For hyperparameters, T is the capture time, k is the number of steps, and K is the total number of steps. Zero-boundary constraints for velocity and acceleration are added. , After solving the linear system and obtaining the coefficients, the true value trajectory is obtained by sampling. .
[0009] Furthermore, the reference trajectory is obtained by adding noise to the true trajectory.
[0010] Furthermore, the redirection of the MANO parameters of the generated true trajectory and reference trajectory to the five-fingered robotic hand includes: employing an inverse kinematics solver to minimize the objective function: in For positive kinematic functions, The posture of the robotic arm. For MANO parameters.
[0011] Furthermore, the prediction model is a multimodal feature Transformer fusion network with a 3-layer encoder structure. Each layer contains 8 attention heads. The multimodal features are concatenated and input, fused through a self-attention mechanism, and finally predicted as the ground truth pose at the current moment by a multilayer perceptron.
[0012] Furthermore, the extraction of image features, ontology features, and reference features from the dataset includes: visual observation. Extracting body and hand features using a pre-trained ResNet network; proprioception. Encoded by a multi-layer perceptron; reference pose sequence Temporal features are extracted using an LSTM network.
[0013] Furthermore, the loss function used in training the prediction model is: in Here, is the regularization coefficient, used to constrain the velocity smoothness of the predicted attitude at time t. To predict the true pose, The true pose serves as the monitoring signal.
[0014] According to another aspect of the specification, a dexterous hand teleoperation device based on grasping motion prior is also provided, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the dexterous hand teleoperation method based on grasping motion prior.
[0015] According to another aspect of the specification, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, implements the aforementioned dexterous hand teleoperation method based on grasping motion priors.
[0016] The beneficial effects of this invention are: 1. High-quality target grasping poses are generated using GraspNet, and a rich and natural grasping motion prior dataset is constructed by combining least squares trajectory optimization.
[0017] 2. By introducing multimodal feature fusion and Transformer temporal modeling, the motion prior network can perceive the current visual and ontological state, and effectively correct real-time trajectory deviations caused by perceptual noise by utilizing historical information from the reference trajectory.
[0018] 3. The prior network trained in simulation can be directly deployed in real-world scenarios, significantly improving cross-domain generalization ability and increasing the capture success rate from 10% without prior knowledge to over 90%, while also greatly improving trajectory smoothness.
[0019] 4. This invention is applicable to various dexterous hand platforms, requiring only adjustments to the redirection process, and has good versatility and scalability. Attached Figure Description
[0020] Figure 1 This is a flowchart illustrating the overall process of the dexterous hand telemanipulation device method based on grasping motion prior according to an embodiment of the present invention. Figure 2 This is a diagram illustrating the teleoperation effect of an embodiment of the present invention in a simulation scenario; Figure 3 This is a diagram illustrating the teleoperation effect of an embodiment of the present invention in a real-world scenario; Figure 4 This is a schematic diagram of a dexterous hand teleoperation device based on grasping motion priors according to an embodiment of the present invention. Detailed Implementation
[0021] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0022] This embodiment provides a dexterous hand telemanipulation method based on grasping motion priors, specifically applied to telemanipulation grasping tasks using a five-finger dexterous hand. It is equipped with a first-person view RGB camera and a dexterous hand joint angle sensor. The overall process is as follows: Figure 1 As shown, it includes target attitude generation, trajectory optimization, redirection, data acquisition, prior learning and deployment.
[0023] Step 1: Target Grasping Pose Generation In a simulation environment, point clouds and contact maps of various objects (such as spheres, cylinders, and boxes) are randomly generated. Using a GraspNet-1 Billion network, the point clouds and contact maps of the objects are taken as input, and high-quality MANO hand grasping posture parameters are output. .in It is a 48-dimensional joint axis rotation (16 joints). It is a 10-dimensional chiral principal component. For 3D wrist global rotation, multiple feasible grasping poses are generated for each object, and physically feasible poses (non-penetrating, force-closed) are selected.
[0024] Step 2: Generating the True Value and Reference Trajectory For each grasping task, the hand pose is randomly initialized. (Naturally open state). Set capture duration. Discretized into Step. Each MANO dimension is parameterized using a 5th-order polynomial to construct the trajectory. ,in As basis functions, For coefficients, , representing the complexity of the trajectory model, t being the current time step. The true trajectory is generated using a least-squares optimization method, which generates the trajectory from the initial pose by minimizing the squared error between the trajectory points and the boundary constraints. Grasp the target posture A smooth time series; the optimization objective is a weighted least squares form: in For linear interpolation approximation, , Add zero-boundary constraints for velocity and acceleration: , ; The hand posture trajectory at time 0 is constrained to be close to that at the initial time. Represents time t Second derivative, After solving the linear system and obtaining the coefficients, the true value trajectory is obtained by sampling. .
[0025] Solve the coefficients using least squares. This yields a smooth truth locus. Simultaneously, a noisy reference trajectory is generated: the noisy trajectory. Simulate perceived noise.
[0026] Step 3: Attitude Redirection Redirect the MANO parameter trajectory to the five-finger robotic hand. Forward kinematics of the five-finger robotic hand. Mapping joint angles to fingertip positions. MANO's forward kinematics. Map the 61-dimensional parameters to the same key points (such as the positions of fingertips and palms). For each time step, solve the inverse kinematics: Obtain the joint angle of the robotic arm .in For positive kinematic functions, The parameters are the MANO parameters. The true trajectory and reference trajectory are retargeted separately to obtain the true trajectory of the manipulator joints. and reference trajectory (Add noise).
[0027] Step 4: Multimodal Dataset Acquisition In the Isaac Gym simulation environment, a robotic arm model is deployed, and the object and hand poses are randomly initialized. The aforementioned ground truth trajectory is then executed. At each time step... Record first-person RGB images (84×84), Proprioception of the robotic hand (32 dimensions including joint angles and angular velocities), and the robot's reference posture. True value attitude of robotic arm This is a multimodal tracking trajectory dataset. Each successfully captured trajectory is considered a sample, and a total of 10,000 trajectories were collected to form the multimodal dataset.
[0028] Step 5: Feature Extraction For each time step Visual images Inputting a ResNet-50 (pre-trained on ImageNet, with the first few layers frozen) outputs 2048-dimensional features, which are then processed by global average pooling to obtain 512-dimensional visual features. Proprioception Inputting two layers of multilayer perceptron (MLP) encoding (256→128) yields 128-dimensional ontology features. Reference attitude sequence (10 × robot arm degrees of freedom) Input a single-layer LSTM (128 hidden layers), and take the output at the last moment as the temporal feature. The three are then combined to obtain multimodal data features.
[0029] Step Six: Prior Learning of Motion A Transformer encoder is constructed, taking the multimodal data feature sequence (one vector per time step) as input. A 3-layer encoder is used, with 8 attention heads per layer and a forward propagation dimension of 1024. After passing through LayerNorm, the output is passed through a 3-layer MLP (512→256→16) to predict the robot's pose at the current time step. During training, batches are randomly sampled from the dataset, with the input being... The visual, ontological, and reference sequences at each moment, with the supervision signal being the truth value. The loss function is: in To predict the first derivative of pose with respect to time, we predict the velocity term. This term penalizes overly rapid or jittery pose changes, making the output motion smoother. The velocity term is approximated by differencing consecutive frames. We use the Adam optimizer with a learning rate of 1e-4 and train for 50 epochs.
[0030] Step 7: Remote Operation Implementation A pre-trained prior network is deployed in a simulation environment: it receives visual images (with artificial noise added to test robustness) and proprioception in real time, as well as the initial reference posture of the gesture (noisy reference). The network outputs corrected joint commands and sends them to the robot controller. In a real-world scenario, a monocular RGB camera captures the operator's gestures. By estimating the key points of the hand and mapping them to MANO parameters, the robot's reference posture is obtained through redirection and then input into the prior network to obtain the final command. Experiments show that, with visual noise (±5 pixel jitter), the success rate of teleoperation without prior knowledge is only 10%, while the method of this invention reaches 92%. On real objects, even for unseen objects (such as irregular toys), the success rate remains above 85%. Figure 2 As shown, this application deploys simulated teleoperation on a shadow hand. As... Figure 3 As shown, this application uses xhand to deploy real-device remote operation on hands for various operational tasks.
[0031] Step 8: Expand the application The method of this invention can also be extended to other dexterous hands (such as Allegro Hand, Shadow Hand, Xhand), simply by replacing the redirection module. The trained prior network can be fine-tuned and quickly adapted to new platforms.
[0032] Corresponding to the aforementioned embodiment of a dexterous hand teleoperation method based on grasping motion prior, the present invention also provides an embodiment of a dexterous hand teleoperation device based on grasping motion prior.
[0033] See Figure 4 The present invention provides a dexterous hand teleoperation device based on grasping motion prior, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a dexterous hand teleoperation method based on grasping motion prior in the above embodiment.
[0034] The present invention provides an embodiment of a dexterous hand telemanipulation device based on grasping motion priors. This embodiment can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is an intelligent hand teleoperation device based on grasping motion priors provided by the present invention. (Except for...) Figure 4In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.
[0035] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0036] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0037] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a dexterous hand teleoperation method based on grasping motion priors as described in the above embodiments.
[0038] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.
[0039] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the aforementioned dexterous hand teleoperation method based on grasping motion prior.
[0040] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.
[0041] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A dexterous hand teleoperation method based on grasp motion priors, characterized in that, include: Randomly generate physically feasible target grasping postures. For each grasping task, initialize the hand posture. Based on the initial posture parameters and the target grasping posture, use least squares optimization to generate the true trajectory of the grasping posture with MANO parameters of the human hand and the noisy grasping reference trajectory. The MANO parameters of the hand generated from the true trajectory and reference trajectory are redirected to the five-finger manipulator to obtain the true trajectory and reference trajectory of the manipulator joints. Deploy the robot arm model to execute the ground truth trajectory, and collect trajectory datasets of images, robot arm proprioception, robot arm reference posture, and robot arm ground truth posture; Image features, ontology features, and reference features are extracted from the dataset and concatenated. The concatenated features are then input into the prediction model to predict the robot's pose at the current moment, and the model is trained using the ground truth pose. The pre-trained network is deployed to receive reference postures obtained from visual images, proprioception, and gesture estimation in real time, and outputs corrected joint commands to the robot controller.
2. The dexterous hand teleoperation method based on grasp motion priors of claim 1, wherein, The randomly generated physically feasible target grasping postures include: Input object point cloud and contact map, generate high-quality human hand grasp pose parameters through GraspNet-1Billion network where the pose vector represents the axis-angle rotation of 16 joints, the shape vector represents the principal components of hand shape, and the global rotation vector represents the global rotation of the wrist.
3. The method for dexterous hand telemanipulation based on grasping motion priors according to claim 1, characterized in that, The ground truth trajectory of the grasping posture generated using least squares optimization of the MANO parameters of the human hand includes: Each MANO dimension is parameterized using low-order polynomials to define the trajectory. ,in As basis functions, For coefficients, The optimization objective is in the form of weighted least squares: in For linear interpolation approximation, , and These are hyperparameters, where T is the capture time, k is the number of steps, and K is the total number of steps. The hand posture trajectory at time 0 is constrained to be close to that at the initial time. Represents time t Second derivative and add velocity and acceleration zero boundary constraints: , After solving the linear system and obtaining the coefficients, the true value trajectory is obtained by sampling. .
4. The method for dexterous hand telemanipulation based on grasping motion prior as described in claim 3, characterized in that, The reference trajectory is obtained by adding noise to the true trajectory.
5. The method for dexterous hand telemanipulation based on grasping motion priors according to claim 1, characterized in that, The process of redirecting the MANO parameters of the generated true trajectory and reference trajectory to the five-finger robotic hand includes: using an inverse kinematics solver to minimize the objective function: in For positive kinematic functions, The posture of the robotic arm. For MANO parameters, To use the MANO parameter positive kinematics function of the human hand.
6. The method for dexterous hand telemanipulation based on grasping motion prior as described in claim 1, characterized in that, The prediction model is a multimodal feature Transformer fusion network with a 3-layer encoder structure. Each layer contains 8 attention heads. The multimodal features are concatenated and input, fused through a self-attention mechanism, and finally predicted as the ground truth pose at the current moment by a multilayer perceptron.
7. The method for dexterous hand telemanipulation based on grasping motion priors according to claim 1, characterized in that, The extraction of image features, ontology features, and reference features from the dataset includes: visual observation. Extracting body and hand features using a pre-trained ResNet network; proprioception. Encoded by a multi-layer perceptron; reference pose sequence Temporal features are extracted using an LSTM network.
8. The method for dexterous hand telemanipulation based on grasping motion prior as described in claim 1, characterized in that, The loss function used in training the prediction model is: in Here, is the regularization coefficient, used to constrain the velocity smoothness of the predicted attitude at time t. To predict the true pose, The true pose serves as the monitoring signal. To predict the first derivative of attitude with respect to time.
9. A dexterous hand teleoperation device based on grasping motion priors, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a dexterous hand teleoperation method based on grasping motion priors as described in any one of claims 1-8.
10. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a dexterous hand teleoperation method based on grasping motion prior as described in any one of claims 1-8.