A method and system for generating dexterous hand grasping postures guided by a restricted graspable area

By using a restricted graspable area guidance method, and leveraging AI models and fine-tuning improvement modules, diverse and controllable grasping postures that meet task requirements are generated. This solves the problem of insufficient diversity and controllability of grasping postures in existing technologies, and can be applied to dexterous robotic arms, virtual reality/augmented reality, and AI-assisted animation editing.

CN116704160BActive Publication Date: 2026-11-10SHANGHAI JIAOTONG UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202310629004.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-05-30
Publication Date
2026-11-10
Estimated Expiration
2043-05-30

AI Technical Summary

Technical Problem

Existing technologies lack masked region guidance when generating robot grasping postures, resulting in insufficient diversity and controllability of the generated grasping postures, making it difficult to meet specific task requirements.

Method used

A restricted graspable region guidance method is adopted, and grasping postures are generated by AI model. Combined with fine-tuning and scoring modules, grasping postures that meet the task requirements are selected. Using the conditional variational encoder framework and Pointnet network structure, diverse grasping postures are generated by combining mask information, and the final posture is optimized by gradient method.

Benefits of technology

It enables the generation of diverse and controllable grasping postures that meet specific task requirements, improving the diversity and controllability of grasping postures. It is suitable for dexterous robotic hands, virtual operations in virtual reality/augmented reality, and AI-assisted animation editing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116704160B_ABST
    Figure CN116704160B_ABST
Patent Text Reader

Abstract

This invention provides a method and system for generating dexterous hand grasping postures guided by a limited graspable area, comprising: Step S1: obtaining the position, posture, and shape information of the target object, and uniformly converting it into an object point cloud, calculating the normal vector corresponding to each point cloud, and setting the graspable area of ​​the target object; Step S2: inputting the object point cloud and its feature information into an AI model grasping posture generation module to generate a grasping posture; Step S3: inputting the generated grasping posture into a fine-tuning and scoring module to filter out target grasping poses; Step S4: filtering grasping postures based on task and intent, and performing post-processing optimization. This invention mainly utilizes the given shape and posture of the target object to generate humanoid dexterous hand grasping postures that meet the requirements of subsequent tasks and conform to human operating habits. This invention can generate batches of candidate postures for AI-assisted animation editing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method and system for generating dexterous hand grasping postures guided by a limited graspable area. Background Technology

[0002] Humans interact with their surroundings extensively in daily activities, performing tasks with specific intentions through actions such as grasping, moving, and flipping target objects. Whether in the fields of robotics, mixed reality digital content technology, or intelligent animation generation, there is a demand for generating reasonable, human-like grasping postures based on target objects.

[0003] Patent document CN108381550B discloses a template-based method for rapid robot grasping posture planning. The method defines a grasping planning template; sets planning parameters; generates a grasping center sampling point from the grasping planning template according to the set planning parameters; checks if the grasping posture is usable; returns the currently attempted grasping posture as the result; detects whether the grasping posture has rotated less than 360 degrees around the center point; rotates the grasping posture around the sampling center point; and repeats this process until a usable grasping posture is found. However, this invention does not use a target grasping method guided by a mask region as a sub-generation method. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for generating dexterous hand grasping postures guided by a limited graspable area.

[0005] A method for generating dexterous hand grasping posture guided by a restricted graspable area according to the present invention includes:

[0006] Step S1: Obtain the position, pose, and shape information of the target object, and convert them into a point cloud. Calculate the normal vectors corresponding to each point cloud and set the grabbable area of ​​the target object.

[0007] Step S2: Input the object point cloud and its feature information into the AI ​​model grasping pose generation module to generate the grasping pose;

[0008] Step S3: Input the generated grasping posture into the fine-tuning and scoring module to filter out the target grasping posture;

[0009] Step S4: Further filter the grabbing posture based on the task and intent, and perform post-processing optimization.

[0010] Preferably, in step S1:

[0011] The graspable area is converted into a mask representation based on the preset potential graspable area of ​​the object corresponding to the task, or the pre-set rules, or the graspable area limited online in this scenario. This mask representation, together with the point cloud normal vector, serves as the object's point cloud feature information.

[0012] Preferably, in step S2:

[0013] The AI-based grasping pose generation module employs a conditional variational encoder framework to encode and decode the hand model's pose. The conditional quantity uses point cloud representations of the 3D shape information of the object to be grasped, including the normal vector information of each point and mask information indicating whether it is within a graspable region. The 3D shape encoding of the object uses an autoencoder network structure, with the encoder using a PointNet network structure and the decoder using a fully connected network. The output is the mask information of the points. A classification network determines whether the point is within a graspable region, using feature vectors encoded into a preset length as the conditional information for hand pose encoding. Grasping poses of different postures are obtained through batch sampling z.

[0014] The training process utilizes open-source 3D dexterous hand grasping data. During the training phase, different thresholds are set for the graspable region mask based on the distance from the object point cloud to the hand model. For the same grasping pose sampling data, multiple thresholds are set to construct the diversity of the mask region.

[0015] Preferably, in step S3:

[0016] The distance from the vertex point cloud of the hand model to the point cloud of the object is calculated under the initially obtained potential grasping pose state, and together with the pose vector and the position input fine-tuning improvement module, it is optimized in multiple loops.

[0017] The fine-tuning and improvement module takes the current hand pose and the distance between the fixed point of the hand model and the object point cloud under this pose as input, and outputs the correction amount of this pose. The new grasping pose is obtained by summing it with the initial pose.

[0018] The training of the fine-tuning and improvement module takes a small local perturbation of the marked pose as input and the deviation from the unperturbed true value as output. During the model training process, the perturbation amplitude is controlled so that the perturbed hand pose is within the preset human hand movement space.

[0019] The scoring module takes the generated grasping posture or the grasping posture processed by the fine-tuning and improvement module as input, calculates the distance from the vertex of the hand model to the object point cloud in the posture, and uses this distance to represent the grasping posture. The scoring module outputs a value in the range of 0-1 as the score for the grasping posture. By setting different thresholds, the grasping posture can be filtered. The training data of the scoring module is implemented by perturbing the ground truth grasping posture. The ideal grasping posture is marked as 1, and the perturbed grasping posture is marked as 0. Random noise is added to the movement and rotation of the hand model origin, and perturbation is added to some or all of the fingers.

[0020] Preferably, in step S4:

[0021] Set tasks and filtering rules according to specific task requirements or introduce an online decision-making process to further obtain a grasping posture that meets the task requirements; use gradient method to further optimize based on the distance from the vertex of the hand model to the point cloud, the distance from the object point cloud to the vertex of the hand model, and the distance from the vertex of the finger model to the object point cloud to obtain the final grasping posture.

[0022] A dexterous hand grasping posture generation system guided by a restricted graspable area according to the present invention includes:

[0023] Module M1: Obtains the position, pose, and shape information of the target object, converts it into a point cloud, calculates the normal vectors corresponding to each point cloud, and sets the graspable area of ​​the target object.

[0024] Module M2: Inputs the object point cloud and its feature information into the AI ​​model grasping pose generation module to generate the grasping pose;

[0025] Module M3: Inputs the generated grasping posture into the fine-tuning and scoring module to filter out the target grasping posture;

[0026] Module M4: Further filters the grabbing posture based on the task and intent, and performs post-processing optimization.

[0027] Preferably, in module M1:

[0028] The graspable area is converted into a mask representation based on the preset potential graspable area of ​​the object corresponding to the task, or the pre-set rules, or the graspable area limited online in this scenario. This mask representation, together with the point cloud normal vector, serves as the object's point cloud feature information.

[0029] Preferably, in module M2:

[0030] The AI-based grasping pose generation module employs a conditional variational encoder framework to encode and decode the hand model's pose. The conditional quantity uses point cloud representations of the 3D shape information of the object to be grasped, including the normal vector information of each point and mask information indicating whether it is within a graspable region. The 3D shape encoding of the object uses an autoencoder network structure, with the encoder using a PointNet network structure and the decoder using a fully connected network. The output is the mask information of the points. A classification network determines whether the point is within a graspable region, using feature vectors encoded into a preset length as the conditional information for hand pose encoding. Grasping poses of different postures are obtained through batch sampling z.

[0031] The training process utilizes open-source 3D dexterous hand grasping data. During the training phase, different thresholds are set for the graspable region mask based on the distance from the object point cloud to the hand model. For the same grasping pose sampling data, multiple thresholds are set to construct the diversity of the mask region.

[0032] Preferably, in module M3:

[0033] The distance from the vertex point cloud of the hand model to the point cloud of the object is calculated under the initially obtained potential grasping pose state, and together with the pose vector and the position input fine-tuning improvement module, it is optimized in multiple loops.

[0034] The fine-tuning and improvement module takes the current hand pose and the distance between the fixed point of the hand model and the object point cloud under this pose as input, and outputs the correction amount of this pose. The new grasping pose is obtained by summing it with the initial pose.

[0035] The training of the fine-tuning and improvement module takes a small local perturbation of the marked pose as input and the deviation from the unperturbed true value as output. During the model training process, the perturbation amplitude is controlled so that the perturbed hand pose is within the preset human hand movement space.

[0036] The scoring module takes the generated grasping posture or the grasping posture processed by the fine-tuning and improvement module as input, calculates the distance from the vertex of the hand model to the object point cloud in the posture, and uses this distance to represent the grasping posture. The scoring module outputs a value in the range of 0-1 as the score for the grasping posture. By setting different thresholds, the grasping posture can be filtered. The training data of the scoring module is implemented by perturbing the ground truth grasping posture. The ideal grasping posture is marked as 1, and the perturbed grasping posture is marked as 0. Random noise is added to the movement and rotation of the hand model origin, and perturbation is added to some or all of the fingers.

[0037] Preferably, in module M4:

[0038] Set tasks and filtering rules according to specific task requirements or introduce an online decision-making process to further obtain a grasping posture that meets the task requirements; use gradient method to further optimize based on the distance from the vertex of the hand model to the point cloud, the distance from the object point cloud to the vertex of the hand model, and the distance from the vertex of the finger model to the object point cloud to obtain the final grasping posture.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. This invention mainly proposes a target grasping sub-generation method guided by mask region, which takes intent information as a prerequisite to balance the diversity and controllability of grasping pose generation model based on AI generation model, and obtains grasping poses that meet specific task requirements in batches.

[0041] 2. This invention proposes a post-processing module that can be flexibly combined with multiple sub-modules, and provides training strategies for each model.

[0042] 3. This invention mainly utilizes the given shape and posture of the target object to generate a humanoid dexterous hand grasping posture that meets the requirements of subsequent tasks and conforms to human operating habits. The application scenarios of this invention include the generation of dexterous robotic hand grasping postures, the generation of virtual operation content of surrounding objects in virtual reality / augmented reality, and the generation of batch candidate postures for AI-assisted animation editing. Attached Figure Description

[0043] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0044] Figure 1 Flowchart of the method for generating pose for dexterous hand grasping;

[0045] Figure 2 This is a network structure diagram of the grasping pose generation module based on the CVAE model;

[0046] Figure 3 To fine-tune and improve the structure diagram of the module and the scoring network module;

[0047] Figure 4 This is a schematic diagram of the agile grasping posture generated in an embodiment of the present invention. Detailed Implementation

[0048] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0049] Example 1:

[0050] This invention provides a method for generating human-like dexterous hand grasping postures, which can balance the diversity and controllability of AI-generated grasping postures to a certain extent, and obtain grasping postures that meet specific task requirements in batches. The main process is as follows: Figure 1 As shown, firstly, the three-dimensional shape information of the target object is obtained and converted into a point cloud representation. Based on specific task requirements and other rules, potential graspable target areas are set. Then, the AI ​​model is used to generate diverse grasping postures in this area. After fine-tuning and scoring, the target grasping posture is selected. Then, potential grasping postures are further selected according to the next task. Finally, the gradient method is used to further optimize the spatial distance between the object shape and the hand model.

[0051] This invention proposes a grasping pose generation method guided by a potential graspable region. Combined with subsequent local improvement and evaluation modules, it generates batches of humanoid dexterous hand target grasping poses that meet task requirements. Applications of this invention include dexterous robotic hand grasping pose generation, virtual manipulation of surrounding objects in virtual reality / augmented reality, and the generation of batch candidate poses for AI-assisted animation editing.

[0052] The present invention provides a method for generating dexterous hand grasping posture guided by a limited graspable area, such as... Figures 1-4 As shown, it includes:

[0053] Step S1: Obtain the position, pose, and shape information of the target object, and convert them into a point cloud. Calculate the normal vectors corresponding to each point cloud and set the grabbable area of ​​the target object.

[0054] Specifically, in step S1:

[0055] The graspable area is converted into a mask representation based on the preset potential graspable area of ​​the object corresponding to the task, or the pre-set rules, or the graspable area limited online in this scenario. This mask representation, together with the point cloud normal vector, serves as the object's point cloud feature information.

[0056] Step S2: Input the object point cloud and its feature information into the AI ​​model grasping pose generation module to generate the grasping pose;

[0057] Specifically, in step S2:

[0058] The AI-based grasping pose generation module employs a conditional variational encoder framework to encode and decode the hand model's pose. The conditional quantity uses point cloud representations of the 3D shape information of the object to be grasped, including the normal vector information of each point and mask information indicating whether it is within a graspable region. The 3D shape encoding of the object uses an autoencoder network structure, with the encoder using a PointNet network structure and the decoder using a fully connected network. The output is the mask information of the points. A classification network determines whether the point is within a graspable region, using feature vectors encoded into a preset length as the conditional information for hand pose encoding. Grasping poses of different postures are obtained through batch sampling z.

[0059] The training process utilizes open-source 3D dexterous hand grasping data. During the training phase, different thresholds are set for the graspable region mask based on the distance from the object point cloud to the hand model. For the same grasping pose sampling data, multiple thresholds are set to construct the diversity of the mask region.

[0060] Step S3: Input the generated grasping posture into the fine-tuning and scoring module to filter out the target grasping posture;

[0061] Specifically, in step S3:

[0062] The distance from the vertex point cloud of the hand model to the point cloud of the object is calculated under the initially obtained potential grasping pose state, and together with the pose vector and the position input fine-tuning improvement module, it is optimized in multiple loops.

[0063] The fine-tuning and improvement module takes the current hand pose and the distance between the fixed point of the hand model and the object point cloud under this pose as input, and outputs the correction amount of this pose. The new grasping pose is obtained by summing it with the initial pose.

[0064] The training of the fine-tuning and improvement module takes a small local perturbation of the marked pose as input and the deviation from the unperturbed true value as output. During the model training process, the perturbation amplitude is controlled so that the perturbed hand pose is within the preset human hand movement space.

[0065] The scoring module takes the generated grasping posture or the grasping posture processed by the fine-tuning and improvement module as input, calculates the distance from the vertex of the hand model to the object point cloud in the posture, and uses this distance to represent the grasping posture. The scoring module outputs a value in the range of 0-1 as the score for the grasping posture. By setting different thresholds, the grasping posture can be filtered. The training data of the scoring module is implemented by perturbing the ground truth grasping posture. The ideal grasping posture is marked as 1, and the perturbed grasping posture is marked as 0. Random noise is added to the movement and rotation of the hand model origin, and perturbation is added to some or all of the fingers.

[0066] Step S4: Further filter the grabbing posture based on the task and intent, and perform post-processing optimization.

[0067] Specifically, in step S4:

[0068] Set tasks and filtering rules according to specific task requirements or introduce an online decision-making process to further obtain a grasping posture that meets the task requirements; use gradient method to further optimize based on the distance from the vertex of the hand model to the point cloud, the distance from the object point cloud to the vertex of the hand model, and the distance from the vertex of the finger model to the object point cloud to obtain the final grasping posture.

[0069] Example 2:

[0070] Example 2 is a preferred example of Example 1, and is used to illustrate the present invention in more detail.

[0071] The present invention also provides a dexterous hand grasping posture generation system guided by a restricted graspable area. The dexterous hand grasping posture generation system guided by a restricted graspable area can be implemented by executing the process steps of the dexterous hand grasping posture generation method guided by a restricted graspable area. That is, those skilled in the art can understand the dexterous hand grasping posture generation method guided by a restricted graspable area as a preferred embodiment of the dexterous hand grasping posture generation system guided by a restricted graspable area.

[0072] A dexterous hand grasping posture generation system guided by a restricted graspable area according to the present invention includes:

[0073] Module M1: Obtains the position, pose, and shape information of the target object, converts it into a point cloud, calculates the normal vectors corresponding to each point cloud, and sets the graspable area of ​​the target object.

[0074] Specifically, in module M1:

[0075] The graspable area is converted into a mask representation based on the preset potential graspable area of ​​the object corresponding to the task, or the pre-set rules, or the graspable area limited online in this scenario. This mask representation, together with the point cloud normal vector, serves as the object's point cloud feature information.

[0076] Module M2: Inputs the object point cloud and its feature information into the AI ​​model grasping pose generation module to generate the grasping pose;

[0077] Specifically, in module M2:

[0078] The AI-based grasping pose generation module employs a conditional variational encoder framework to encode and decode the hand model's pose. The conditional quantity uses point cloud representations of the 3D shape information of the object to be grasped, including the normal vector information of each point and mask information indicating whether it is within a graspable region. The 3D shape encoding of the object uses an autoencoder network structure, with the encoder using a PointNet network structure and the decoder using a fully connected network. The output is the mask information of the points. A classification network determines whether the point is within a graspable region, using feature vectors encoded into a preset length as the conditional information for hand pose encoding. Grasping poses of different postures are obtained through batch sampling z.

[0079] The training process utilizes open-source 3D dexterous hand grasping data. During the training phase, different thresholds are set for the graspable region mask based on the distance from the object point cloud to the hand model. For the same grasping pose sampling data, multiple thresholds are set to construct the diversity of the mask region.

[0080] Module M3: Inputs the generated grasping posture into the fine-tuning and scoring module to filter out the target grasping posture;

[0081] Specifically, in module M3:

[0082] The distance from the vertex point cloud of the hand model to the point cloud of the object is calculated under the initially obtained potential grasping pose state, and together with the pose vector and the position input fine-tuning improvement module, it is optimized in multiple loops.

[0083] The fine-tuning and improvement module takes the current hand pose and the distance between the fixed point of the hand model and the object point cloud under this pose as input, and outputs the correction amount of this pose. The new grasping pose is obtained by summing it with the initial pose.

[0084] The training of the fine-tuning and improvement module takes a small local perturbation of the marked pose as input and the deviation from the unperturbed true value as output. During the model training process, the perturbation amplitude is controlled so that the perturbed hand pose is within the preset human hand movement space.

[0085] The scoring module takes the generated grasping posture or the grasping posture processed by the fine-tuning and improvement module as input, calculates the distance from the vertex of the hand model to the object point cloud in the posture, and uses this distance to represent the grasping posture. The scoring module outputs a value in the range of 0-1 as the score for the grasping posture. By setting different thresholds, the grasping posture can be filtered. The training data of the scoring module is implemented by perturbing the ground truth grasping posture. The ideal grasping posture is marked as 1, and the perturbed grasping posture is marked as 0. Random noise is added to the movement and rotation of the hand model origin, and perturbation is added to some or all of the fingers.

[0086] Module M4: Further filters the grabbing posture based on the task and intent, and performs post-processing optimization.

[0087] Specifically, in module M4:

[0088] Set tasks and filtering rules according to specific task requirements or introduce an online decision-making process to further obtain a grasping posture that meets the task requirements; use gradient method to further optimize based on the distance from the vertex of the hand model to the point cloud, the distance from the object point cloud to the vertex of the hand model, and the distance from the vertex of the finger model to the object point cloud to obtain the final grasping posture.

[0089] Example 3:

[0090] Example 3 is a preferred example of Example 1, and is used to illustrate the present invention in more detail.

[0091] The process of a method for generating dexterous multi-finger grasping postures based on restricted graspable regions is as follows:

[0092] Step S1: Obtain the position, pose, and shape information of the target object, and convert it into a point cloud representation. Calculate the normal vector corresponding to each point cloud.

[0093] Step S2: Set the grabbable area of ​​the target object;

[0094] Step S3: Input the above object point cloud and its feature information into the AI ​​model restricted area capture generation module;

[0095] Step S4: Input the generated grasping posture into the fine-tuning and improvement module and the scoring module;

[0096] Step S5: Filtering based on task and intent-based grasping posture;

[0097] Step S6, post-processing optimization module.

[0098] In step 2, the graspable region can be converted into a mask representation based on the preset potential graspable region of the object corresponding to the task or the pre-set rules (such as the region where x>0) or the graspable region that is considered to be limited to the online context. This mask representation, together with the point cloud normal vector, serves as the object point cloud feature information.

[0099] In step 3, the AI ​​model-based grasping pose generation module takes object point cloud information and its feature information, as well as graspable region encoding, as input information. It obtains grasping poses of different postures through batch sampling z. It should be noted that this generation module can be trained using open-source 3D dexterous hand grasping data. During the training phase, the graspable region mask can be set with different thresholds based on the distance from the object point cloud to the hand model. In order to ensure the robustness of the model to different mask regions, multiple thresholds are set for the same grasping pose sampling data to build the diversity of the mask region. For example, multiple random samplings can be performed within the range of 30-200mm with a certain probability, or multiple segmentation points can be set at different intervals.

[0100] In step 4, the distance from the vertex point cloud of the hand model to the object point cloud in the initially obtained potential grasping pose state is calculated and combined with the pose vector and the position input fine-tuning and improvement module. It should be noted that this step can be optimized through multiple iterations. The training of this fine-tuning and improvement module takes a small local perturbation of the marked pose as input and the deviation from the unperturbed ground truth as output. Since this correction direction is only effective near the ground truth, the perturbation amplitude is controlled during model training, while ensuring that the perturbed hand pose remains within a reasonable range of human hand movement space.

[0101] In step 4, the grasping posture scoring module can take the grasping posture generated in step 3 or the fine-tuned grasping posture as input. Specifically, it first calculates the distance from the vertex of the hand model to the object point cloud in this posture, and uses this distance to represent the grasping posture. The scoring module outputs a value in the range of 0-1, which can be considered as the score of the grasping posture. By setting different thresholds, excellent grasping postures can be selected. The training data for this scoring module is also implemented by perturbing the ground truth grasping posture. The ideal grasping posture is marked as 1, and the perturbed grasping posture is marked as 0. It should be noted that compared with the fine-tuning improvement module, this model does not have the problem of local validity, and the perturbation amplitude can be set relatively large. At the same time, in order to cover the real feature space as much as possible, more random noise can be set for the movement and rotation of the hand model origin. To ensure that the hand posture is in a reasonable human hand space, a small amount of perturbation is randomly added to some or all of the fingers.

[0102] Since step 4 contains independent fine-tuning sub-modules and molecule-breaking modules, which have no dependency on each other, the order of the two sub-modules can be arbitrarily combined to jointly optimize and filter the generated grasping posture.

[0103] Step S5: After batch improving the filtering steps, further, tasks and filtering rules can be set according to specific task requirements, or a human online decision-making process can be introduced to further obtain a grasping posture that meets the task requirements.

[0104] Step S6 involves using the gradient method to further optimize the grasping posture based on information such as the distance from the hand model vertex to the point cloud, the distance from the object point cloud to the hand model vertex, and the distance from the finger model vertex to the object point cloud.

[0105] It should be noted that step S5 is optional and can be flexibly determined to exist or be set before or after step S6.

[0106] The main workflow modules include: object 3D information preprocessing module, target grasping area editing module, grasping posture generation module guided by restricted area, fine-tuning and evaluation module, rule filtering module, and post-processing module.

[0107] The AI-based grasping pose generation module primarily employs a Conditional Variational Autoencoder (CVAE) framework to encode and decode the hand model pose. The conditional quantity uses point cloud representations of the 3D shape information of the graspable object, including the normal vector information of each point and the mask information indicating whether it is within the graspable region. The 3D shape encoding of the object uses an autoencoder network structure, with the encoder using a PointNet network structure and the decoder using a fully connected network. The output is the mask information of the point, which is a classification network that determines whether the point is within the graspable region. The feature vector encoded into a fixed length is used as the conditional information for hand pose encoding. Figure 2 This demonstrates the complete network structure of this module, achieving end-to-end training during model training. In the model inference phase, the hand pose encoding module and point cloud mask decoding module are omitted, and instead... Figure 2 The gray area network takes the 3D shape information of the object to be grasped and the mask information of the graspable region as input, and outputs the potential grasping pose in this state. At the same time, the network has a random variable z obtained by Gaussian sampling. By sampling different z variables, it is possible to obtain diverse target grasping poses under the same input conditions.

[0108] The fine-tuning and scoring module mainly consists of two independent sub-modules: a fine-tuning and scoring module. The fine-tuning and scoring module takes the current hand pose and the distance between the hand model vertex and the object point cloud in that pose as input, and outputs a correction value for that pose. This correction is summed with the initial pose to obtain the new grasping pose. The scoring module takes the distance from the hand model vertex to the object point cloud as input and outputs a continuous variable in the range of 0-1. This variable serves as a score evaluating the reasonableness of the grasping pose; a higher score indicates a more reasonable grasping pose. Since the two sub-modules are independent, various combinations can be used in practice. For example, "ab" involves first fine-tuning and correcting the network, then scoring and evaluating it, and using thresholds such as TopK to sort and filter grasping poses. Similarly, "ba" can be used, where sorting and filtering are performed first, followed by grasping pose correction. Other similar combinations include "bab", "a(N)-b", and "ba(N)-b". Where “a(N)-b” means that after the initial posture has been improved through N iterations, the grasping posture is evaluated and screened.

[0109] Example 4:

[0110] Example 4 is a preferred example of Example 1, which is used to illustrate the present invention in more detail.

[0111] Step S1: Convert the object mesh into a point cloud and calculate the normal vector, sampling up to 4096 points, with the input vector size being (4096, 6).

[0112] Step S2: Set a boundary line of y>10mm to divide the object into grabbable and non-grabable regions. Add this mask to the input vector and update its size to (4096, 7).

[0113] Step S3: Train the CVAE grasping pose generation model using a 3D grasping dataset. Set the graspable area mask based on the distance from the object point cloud to the vertex of the hand model. Set the thresholds to 30mm, 40mm, 50mm, 60mm, 70mm, 80mm, 100mm, 150mm, and 200mm respectively. Assign multiple mask areas to the same grasping pose. The model is trained for 200 epochs.

[0114] Step S4: The network model is randomly generated from the restricted area of ​​the AI ​​model trained in Step S3, which was obtained in Step S2. 100 z variables are randomly sampled to obtain the 100 potential target grasping poses in the restricted area.

[0115] Step S5: Use the 3D grasping dataset to train the fine-tuning and improvement module model. Perturb the hand model position variables in a small range of 0-20mm, the global pose in 0-5 degrees, and the root joints of the fingers in 0-5 degrees. At the same time, set 10% of the unperturbed dataset as training data, and output the pose change as 0. The model is trained for a total of 50 epochs.

[0116] Step S6: Input the 100 potential grasping postures generated in step S4 into the fine-tuning and improvement module model trained in step S5, and perform 5 iterations.

[0117] Step S7: The scoring module model is trained using a 3D dataset. Negative samples are obtained by perturbing the ground truth. The hand model position variable is perturbed from 0 to 100 mm, the global pose variable from 0 to 10 degrees, and the root joint of the finger is perturbed from 0 to 10 degrees. The model is trained for a total of 50 epochs.

[0118] Step S8: Input the corrected grasping pose from step S6 into the evaluation network model trained in step S7, and select 10 grasping poses as the final output based on the model output score.

[0119] Step S9: Based on information such as the distance from the hand model vertices to the object point cloud, the distance from the object point cloud to the hand model vertices, and the distance from the finger model vertices to the object point cloud, and guided by the hand model and object model not being aligned, and the thumb / middle finger / index finger being in contact with the object surface, the gradient iterative loop is run 100 times to obtain the final grasping pose. The effect is as follows. Figure 4 As shown.

[0120] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0121] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for generating dexterous hand grasping posture guided by a restricted graspable area, characterized in that, include: Step S1: Obtain the position, pose, and shape information of the target object, and convert them into a point cloud. Calculate the normal vectors corresponding to each point cloud and set the grabbable area of ​​the target object. Step S2: Input the object point cloud and its feature information into the AI ​​model grasping pose generation module to generate the grasping pose; Step S3: Input the generated grasping posture into the fine-tuning and scoring module to filter out the target grasping posture; Step S4: Further filter the grabbing posture based on the task and intent, and perform post-processing optimization; In step S1: The graspable area is converted into a mask representation based on the preset potential graspable area of ​​the object corresponding to the task, or the graspable area in the corresponding scenario set in advance, or the graspable area limited online. This mask representation, together with the point cloud normal vector, serves as the object point cloud feature information. In step S2: The AI-based grasping pose generation module employs a conditional variational encoder framework to encode and decode the hand model's pose. The conditional quantity uses the 3D shape information of the object to be grasped, represented by a point cloud, which includes the normal vector information of each point and the mask information indicating whether it is within the graspable region. The 3D shape encoding of the object uses an autoencoder network structure, with the encoder using a PointNet network structure and the decoder using a fully connected network. The output is the mask information of the points. A classification network determines whether the point is within the graspable region and uses the feature vector encoded into a preset length as the conditional information for hand pose encoding. Grasping poses of different postures are obtained by batch sampling z. The training process utilizes open-source 3D dexterous hand grasping data. During the training phase, different thresholds are set for the graspable region mask based on the distance from the object point cloud to the hand model. For the same grasping pose sampling data, multiple thresholds are set to construct the diversity of the mask region.

2. The method for generating dexterous hand grasping posture guided by a limited graspable area according to claim 1, characterized in that, In step S3: The distance from the vertex point cloud of the hand model to the point cloud of the object is calculated under the initially obtained potential grasping pose state, and together with the corresponding pose vector and the position input fine-tuning improvement module, it is optimized in multiple loops. The fine-tuning and improvement module takes the current hand pose and the distance between the vertices of the hand model and the object point cloud in that pose as input, and outputs the correction amount for that pose. The new grasping pose is obtained by summing it with the initial pose. The training of the fine-tuning and improvement module takes a small local perturbation of the marked pose as input and the deviation from the unperturbed true value as output. During the model training process, the perturbation amplitude is controlled so that the perturbed hand pose is within the preset human hand movement space. The scoring module takes the generated grasping posture or the grasping posture processed by the fine-tuning and improvement module as input, calculates the distance from the vertex of the hand model to the object point cloud in the posture, and uses this distance to represent the grasping posture. The scoring module outputs a value in the range of 0-1 as the score for the grasping posture. By setting different thresholds, the grasping posture can be filtered. The training data of the scoring module is implemented by perturbing the ground truth grasping posture. The ideal grasping posture is marked as 1, and the perturbed grasping posture is marked as 0. Random noise is added to the movement and rotation of the hand model origin, and perturbation is added to some or all of the fingers.

3. The method for generating dexterous hand grasping posture guided by a limited graspable area according to claim 1, characterized in that, In step S4: Set tasks and filtering rules according to specific task requirements or introduce an online decision-making process to further obtain a grasping posture that meets the task requirements; use gradient method to further optimize based on the distance from the vertex of the hand model to the point cloud, the distance from the object point cloud to the vertex of the hand model, and the distance from the vertex of the finger model to the object point cloud to obtain the final grasping posture.

4. A dexterous hand grasping posture generation system guided by a restricted graspable area, characterized in that, include: Module M1: Obtains the position, pose, and shape information of the target object, converts it into a point cloud, calculates the normal vectors corresponding to each point cloud, and sets the graspable area of ​​the target object. Module M2: Inputs the object point cloud and its feature information into the AI ​​model grasping pose generation module to generate the grasping pose; Module M3: Inputs the generated grasping posture into the fine-tuning and scoring module to filter out the target grasping posture; Module M4: Further filters the grabbing posture based on the task and intent, and performs post-processing optimization; In module M1: The graspable area is converted into a mask representation based on the preset potential graspable area of ​​the object corresponding to the task, or the graspable area in the corresponding scenario set in advance, or the graspable area limited online. This mask representation, together with the point cloud normal vector, serves as the object point cloud feature information. In module M2: The AI-based grasping pose generation module employs a conditional variational encoder framework to encode and decode the hand model's pose. The conditional quantity uses the 3D shape information of the object to be grasped, represented by a point cloud, which includes the normal vector information of each point and the mask information indicating whether it is within the graspable region. The 3D shape encoding of the object uses an autoencoder network structure, with the encoder using a PointNet network structure and the decoder using a fully connected network. The output is the mask information of the points. A classification network determines whether the point is within the graspable region and uses the feature vector encoded into a preset length as the conditional information for hand pose encoding. Grasping poses of different postures are obtained by batch sampling z. The training process utilizes open-source 3D dexterous hand grasping data. During the training phase, different thresholds are set for the graspable region mask based on the distance from the object point cloud to the hand model. For the same grasping pose sampling data, multiple thresholds are set to construct the diversity of the mask region.

5. The dexterous hand grasping posture generation system guided by a limited graspable area according to claim 4, characterized in that, In module M3: The distance from the vertex point cloud of the hand model to the point cloud of the object is calculated under the initially obtained potential grasping pose state, and together with the corresponding pose vector and the position input fine-tuning improvement module, it is optimized in multiple loops. The fine-tuning and improvement module takes the current hand pose and the distance between the vertices of the hand model and the object point cloud in that pose as input, and outputs the correction amount for that pose. The new grasping pose is obtained by summing it with the initial pose. The training of the fine-tuning and improvement module takes a small local perturbation of the marked pose as input and the deviation from the unperturbed true value as output. During the model training process, the perturbation amplitude is controlled so that the perturbed hand pose is within the preset human hand movement space. The scoring module takes the generated grasping posture or the grasping posture processed by the fine-tuning and improvement module as input, calculates the distance from the vertex of the hand model to the object point cloud in the posture, and uses this distance to represent the grasping posture. The scoring module outputs a value in the range of 0-1 as the score for the grasping posture. By setting different thresholds, the grasping posture can be filtered. The training data of the scoring module is implemented by perturbing the ground truth grasping posture. The ideal grasping posture is marked as 1, and the perturbed grasping posture is marked as 0. Random noise is added to the movement and rotation of the hand model origin, and perturbation is added to some or all of the fingers.

6. The dexterous hand grasping posture generation system guided by a limited graspable area according to claim 4, characterized in that, In module M4: Set tasks and filtering rules according to specific task requirements or introduce an online decision-making process to further obtain a grasping posture that meets the task requirements; use gradient method to further optimize based on the distance from the vertex of the hand model to the point cloud, the distance from the object point cloud to the vertex of the hand model, and the distance from the vertex of the finger model to the object point cloud to obtain the final grasping posture.

Citation Information

Patent Citations

  • A template-based method for fast robot grasping posture planning

    CN108381550B

  • Robot grabbing detection method based on instance segmentation under single-view-angle point cloud

    CN110363815A

  • A method for controlling a grasping robot through a learning phase and a grasping phase

    EP4155036A1