Intra-class multi-specification tool oriented dexterous hand grabbing posture generation method

Through the multi-dimensional grasping knowledge coding and point cloud registration method based on intention, the grasping posture generation of the five-finger dexterity hand is optimized, and the problem of insufficient model adaptability in the existing technology is solved, and functional grasping of multi-special tools is realized , improves the grasping ability of the five-finger clever hands.

CN119089989APending Publication Date: 2024-12-06NANJING UNIV OF AERONAUTICS & ASTRONAUTICS

Patent Information

Application Number
CN202410963659.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing technology fails to effectively consider the tool's intention label and hand grab dataset encoding when generating the five-finger clever hand grab posture, resulting in insufficient model adaptability in the case of few samples, making it difficult to achieve functional grasp of multi-spec tools. Pick.

Method used

Using the multi-dimensional crawling knowledge coding method based on intention, the crawling pose parameters and intent labels of the basic tools are collected by collecting the crawling pose parameters and intent labels of the human hand to build a crawling pose generation network, and the crawling pose is optimized using point cloud registration and self-collision loss functions, combined with Dexterous hand kinematics optimize key points and realize functional capture of multi-special tools within the class.

Benefits of technology

The adaptability and functional grasping ability of the five-finger dexterity hand to multi-spec tools is improved, and the effect of grabbing posture generation is enhanced, especially in the case of few samples, which can effectively generate grabbing postures that are adapted to different ideas.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089989A_ABST
    Figure CN119089989A_ABST
Patent Text Reader

Abstract

The invention discloses an intra-class multi-specification tool-oriented dexterous hand grabbing posture generation method. A basic tool is selected for each class of tools, grabbing postures of the basic tool under different intentions are collected, the grabbing postures are coded, and a basic tool multi-dimensional grabbing knowledge data set is constructed; training according to the grabbing postures of the basic tools under different intentions to obtain a related grabbing posture generation network based on the intentions; on the basis of a grabbing attitude generation network trained by a basic tool, realizing the migration of grabbing attitudes under different intentions from the basic tool to an intra-class target tool through grabbing knowledge migration; and finally, a mapping rule between the human hand and the five-fingered dexterous hand is constructed through grabbing posture mapping, the mapping rule is optimized by utilizing an iterative posture optimization method, and the five-fingered dexterous hand grabs the intra-class target tools under different intentions. According to the method, the purpose that the dexterous hand generates the grabbing posture under the specified intention for any tool in the class under the condition of few samples is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method for generating grasping postures, and in particular to a method for generating grasping postures of dexterous hands for tools of multiple specifications within a category. Background Art

[0002] Many tools at the assembly site are designed according to people's usage habits. Therefore, compared with parallel grippers or special fixtures, the five-finger dexterous hand is more adaptable to this type of assembly environment and has strong generalization ability, which can realize the operation and use of complex parts and multiple tools at the assembly site. At present, the five-finger dexterous hand is getting closer and closer to the human hand in terms of size and mechanical function. However, from the perspective of its actual task execution, there is still a big gap compared with the human hand. The tools at the assembly site not only have many categories, but also have different size specifications within each category of tools. In order to realize human-machine collaboration and task allocation in the assembly scene, the robot needs to functionally grasp the required tools according to different intentions. In order to enhance the functional grasping ability of the five-finger dexterous hand for tools, it is necessary to start from the functional grasping of tools by human hands, and then realize the grasping of the five-finger dexterous hand.

[0003] Some of the existing methods for tool grasping posture generation process do not take into account the intention label, and ignore the functionality of hand-to-tool grasping; some data sets collected between hands and tools do not fully encode the grasping knowledge; some studies only focus on human hand grasping of tools, and do not transfer grasping to five-finger dexterous hands; the difficulty of collecting data on human hand grasping of tools under different intentions makes it difficult to collect enough data to make the grasping posture generation model adapt to all situations. When the data volume is large, the model becomes difficult to train and consumes a lot of computing resources. In the case of a small number of samples of human hand and tool grasping data, the grasping posture generation model has the problem of insufficient generalization.

[0004] In view of this, designing and implementing an intention-based dexterous hand grasping posture generation method for multi-specification tools within a class is of great significance for dexterous hand tool grasping, tool operation and human-machine collaboration. Summary of the invention

[0005] Purpose of the invention: The present invention provides a method for generating grasping postures of dexterous hands for tools of multiple specifications within a class, which not only solves the problem that the intention-based grasping posture generation network trained under a small number of samples is difficult to adapt to tools of multiple specifications within a class, but also solves the problem of generating functional grasping of tools by five-finger dexterous hands under specified intentions.

[0006] Technical solution: The method for generating a grasping posture of a dexterous hand for tools of multiple specifications within a category described in the present invention specifically comprises the following steps:

[0007] (1) For each type of tool, one is selected as the basic tool and the rest are selected as the target tools within the class. The grasping posture parameters corresponding to the hand parameterized MANO model of the basic tool under different intentions, the basic model transformation posture and the corresponding intention label are collected;

[0008] (2) Processing the collected data, the sampling point cloud of the basic tool model and its corresponding normal vector, multi-dimensional crawling knowledge and word vectors corresponding to the crawling intention together form a multi-dimensional crawling knowledge dataset of the basic tools in the category;

[0009] (3) Construct and train a grasping posture generation network based on intention-based multi-dimensional grasping knowledge encoding, including the construction of a hand-object multi-dimensional grasping knowledge generation network and the solution of grasping posture parameters;

[0010] (4) In the process of grasping knowledge transfer, a point cloud registration method is added to focus on the posture difference between the basic tool and the target tool. According to the grasping knowledge of the target tool obtained by transfer, the grasping posture transfer grasping posture solution method of inter-finger collision is optimized. In the grasping posture solution process, a hand self-collision loss function is proposed to avoid the occurrence of inter-finger collision, and the grasping posture parameters of the target tool under different intentions are obtained.

[0011] (5) The mapping relationship between the human hand and the five-finger dexterous hand is constructed by weighted key point position optimization based on the kinematics of the dexterous hand. According to the correspondence between the key points of the human hand and the key points of the dexterous hand, a weighted key point optimization objective function is proposed. The objective function is optimized using the iterative posture optimization method to realize the dexterous hand grasping posture generation of multi-specification target tools within the class under different intentions.

[0012] Furthermore, the implementation process of step (2) is as follows:

[0013] The basic tool model is transformed and moved to the origin as the center, and n point cloud coordinates and their corresponding normal vectors are sampled;

[0014] The n points sampled from the basic tool model and the grasping posture parameters corresponding to the hand parameterized model are used to restore the hand parameterized model posture, and the contact relationship between the hand parameterized model and the tool point cloud under the corresponding posture is encoded into a multi-dimensional grasping knowledge, which is encoded into a hand-object contact probability map, a hand-object contact part map, and a hand-object contact direction map; the above constitutes the multi-dimensional grasping knowledge;

[0015] Encode the grasping intention label in the form of word vectors to guide the generation direction of grasping posture and achieve functional grasping;

[0016] The captured data with few data samples are expanded accordingly to ensure the balance of captured data samples under different intentions.

[0017] Furthermore, the hand-object multi-dimensional grasping knowledge generation network construction implementation process in step (3) is as follows:

[0018] The hand-object multi-dimensional grasping knowledge generation network is an improved conditional variational autoencoder structure. For different hand-object grasping knowledge descriptions, a conditional variational autoencoder is constructed respectively, and each conditional variational autoencoder is connected in series to form a sequential conditional variational autoencoder. The tool point cloud features and the grasping intention features are concatenated as the conditional features of each conditional variational autoencoder; the conditional features and the multi-dimensional grasping knowledge are used as the input of each conditional variational autoencoder network, and at the same time as the conditional input of the network to guide the generation of hand-object multi-dimensional grasping knowledge of different basic tools under different intentions. The training goal is to ensure the consistency of the generated hand-object multi-dimensional grasping knowledge and the input multi-dimensional grasping knowledge encoding; in the test phase, the input conditional features and the vectors sampled in the latent variable space of the conditional variational autoencoder are jointly input into the decoder of the conditional variational autoencoder to generate diversified hand-object multi-dimensional grasping knowledge.

[0019] Furthermore, the process of solving the posture parameters in step (3) is as follows:

[0020] Based on the diverse hand-object multi-dimensional grasping knowledge, the grasping posture parameters are optimized and solved. First, the global posture of the grasping parameters is optimized, and then the hand posture parameters and shape parameters of the grasping are further optimized to obtain the grasping posture parameters corresponding to the hand-object multi-dimensional grasping knowledge under different grasping intentions.

[0021] Furthermore, the implementation process of step (4) is as follows:

[0022] Calculate the contact graph of hand-object interaction based on multiple grasping postures generated by the basic tool under specified intentions;

[0023] In view of the posture difference between the target tool and the basic tool in the class, the PCA point cloud registration method is used to calculate the posture transformation relationship between the basic tool and the target tool in the class, and the target tool is aligned with the posture of the basic tool through posture transformation;

[0024] The base tool and the aligned target tool are shaped encoded using DeepSDF, and linear interpolation is performed between the shape vectors encoded by the base tool and the target tool. The interpolated shape vectors are decoded to obtain the interpolation model corresponding to the shape vector. The contact map is migrated from the base tool to the target tool by taking advantage of the small shape difference between adjacent interpolation models.

[0025] When solving the corresponding grasping posture for the contact graph of the target tool, a self-collision loss function is designed to optimize the grasping posture.

[0026] Furthermore, the process of constructing the mapping relationship between the human hand and the five-finger dexterous hand based on the weighted key point position optimization of the dexterous hand kinematics described in step (5) is as follows:

[0027] According to the generated grasping posture of the human hand for the target tool, combined with the similarity between the human hand and the five-finger dexterous hand, different corresponding weights are set using the key points of the hand and the joints of the five-finger dexterous hand. The mapping relationship of the corresponding key point positions under different weights is optimized through the kinematics of the five-finger dexterous hand, and the grasping posture of the human hand for the tool is transferred to the five-finger dexterous hand, realizing the functional grasping of the tool by the five-finger dexterous hand under different intentions; the optimization goals of the mapping relationship are as follows:

[0028]

[0029] in, represents the position of the i-th hand key point, represents the position of the i-th key point corresponding to the kinematic optimization process of the five-finger dexterous hand, k i Represents the weight corresponding to the position difference of the key points between the i-th hand and the five-finger dexterous hand.

[0030] Furthermore, the implementation process of optimizing the objective function using the iterative posture optimization method in step (5) is as follows:

[0031] The mapping of the dexterous hand's grasping posture is divided into multiple iterations. The grasping posture parameters of the dexterous hand obtained by the last iteration are used to gradually approach the target posture. According to the structure of the dexterous hand and the characteristics of the grasping posture, a reasonable threshold is set to avoid the iterative posture optimization process falling into an infinite loop.

[0032] Furthermore, the hand-object contact probability map represents the contact probability between the hand and each point in the object point cloud, indicating which part of the object may be in contact with the hand; the hand-object contact direction map calculates the contact direction between the point on the object point cloud and the corresponding hand contact area.

[0033] Furthermore, the hand-object contact portion diagram represents dividing the hand into sixteen regions and calculating the contact conditions between the object and different regions of the hand.

[0034] Furthermore, the self-collision loss function is as follows:

[0035]

[0036] Among them, i and j represent the numbers of the 21 key points of the hand, and h i ,h j Represents the coordinates of 21 key points of the hand, dis(h i -h j) represents the distance between two different key points of the hand, δ ij Represents the threshold of the distance between two different key points of the hand, k ij Represents the weight of the distance loss between different keypoints.

[0037] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. The present invention improves the adaptability of the grasping posture generation network to multi-specification tools by using grasping knowledge transfer in the intention-based dexterous hand grasping posture generation method for multi-specification tools within a class;

[0039] 2. Compared with the existing methods, the present invention not only improves the adaptability to tools of various specifications, but also emphasizes the functionality of the five-finger dexterous hand in grasping tools. It combines the grasping intention with the grasping posture generation network encoded by multi-dimensional grasping knowledge, and proposes a grasping posture generation network based on intention-based multi-dimensional grasping knowledge encoding, which not only emphasizes the functionality of grasping, but also enhances the effect of grasping posture generation.

[0040] 3. The present invention performs network training with few samples, and combines the method of grasping knowledge transfer to realize functional grasping of other tools in the same category, and finally realizes the functional grasping of multiple specifications of tools in the same category by the five-finger dexterous hand. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the present invention;

[0042] Figure 2 is a schematic diagram of a grasping posture generation network proposed in the present invention;

[0043] Figure 3 It is a flow chart of the mapping algorithm between human hand and five-finger dexterous hand in the present invention. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below with reference to the accompanying drawings.

[0045] Aiming at the functional grasping of tools by five-fingered dexterous hands in assembly scenarios, in addition to the diversity of categories, tools in assembly scenarios also have multiple specifications and sizes within the tool category. This paper proposes a grasping posture generation method for dexterous hands with multiple specifications of tools within a category, such as Figure 1 As shown, the following steps are included:

[0046] Step 1: There are many types of tools in industrial scenes, and each type of tool has different specifications and models. For each type of tool, select one as the basic tool and the rest as the target tools in the class, and collect the grasping postures of the basic tools under different intentions. Process the collected data, and finally combine the sampling point cloud of the basic tool model and its corresponding normal vector, multi-dimensional grasping knowledge and word vector corresponding to the grasping intention to form a multi-dimensional grasping knowledge grasping dataset of the basic tools in the class. The details are as follows:

[0047] There are many types of tools in industrial scenes, recorded as (a, b, c...); a type of tool also has different specifications and models, recorded as (a o1 ,a o2 ,a o3 …); for a type of tool a, select one as the basic tool a o1 , and the rest are regarded as in-class target tools, denoted as (a o2 ,a o3 …); collect the basic tools a under different intentions o1 The grasping posture a h1 The grasping intention is used as a label to distinguish different functional areas of functional grasping, denoted as Int. The multi-dimensional grasping knowledge dataset includes the hand parameter model MANO and the corresponding grasping posture parameters, the basic tool model P and its corresponding posture [R p ,T p ] and the fetch intent Int.

[0048] The collected data is further processed to convert the basic tool model P into a point cloud, denoted as P o , calculate the center P of the point cloud o c , move the center of the point cloud to the origin, and transform the moved point cloud into the corresponding posture obtained by acquisition [R p ,T p ]Get P o ', from P o 'Sampling n points P o ' -n and its corresponding normal vector P o ' -v .

[0049] Encode the crawling intent Int in the form of One-Hot word vector. The length of the word vector is determined according to the number of intent tags. Each bit represents a different intent, recorded as Int vec .

[0050] The parameters corresponding to the hand parametric model are used to restore the posture of the hand parametric model, and the n points P obtained by sampling are used o '-n The hand parameterized model in the corresponding posture is compared with the tool point cloud P o ' -n The contact relationship between them is encoded into a multi-dimensional grasping knowledge, which is encoded as a hand-object contact probability graph k c , contact part diagram k p And the contact direction diagram k d , hand-object contact probability map k c represents the contact probability between the hand and each point in the object point cloud, indicating which part of the object may be in contact with the hand; hand-object contact part map k p It represents dividing the hand into 16 main areas and calculating the contact between the object and different areas of the hand; the hand-object contact direction map k d The main thing is to calculate the contact direction between the point on the object point cloud and the corresponding hand contact area.

[0051] The multi-dimensional grasping knowledge dataset of the basic tools in the class includes the point cloud P sampled by the above basic tools. o ' -n and its corresponding normal vector P o ' -v , multi-dimensional knowledge capture k c , k p With k d , grab the word vector Int of the intent encoding.

[0052] vec

[0053] In data collection, there may be a problem of uneven data distribution in multi-dimensional crawled knowledge data sets under different categories of tools or different intentions. To address this problem, the crawled data with fewer data samples are expanded accordingly to ensure that the crawled data samples under different categories of tools or different intentions are balanced, thereby improving the performance and generalization ability of the model.

[0054] The multi-dimensional crawling knowledge crawling dataset of the in-class basic tools composed of all the above features is divided into a training set and a test set in a ratio of 3:1.

[0055] Step 2: Construct a grasping posture generation network based on intention-based multi-dimensional grasping knowledge encoding, improve the conditional variational autoencoder, train a basic tool to generate grasping postures based on hand-object multi-dimensional grasping knowledge encoding under different intentions, and use the constructed grasping posture solver to optimize the grasping posture under hand-object multi-dimensional grasping knowledge. The main structure of the grasping posture generation network based on intention-based multi-dimensional grasping knowledge encoding is as follows: Figure 2 As shown, the details are as follows:

[0056] The main architecture of the hand-object multi-dimensional grasping knowledge generation network is an improved conditional variational autoencoder structure. c, k p With k d , respectively construct a conditional variational autoencoder CVAE c 、CVAE p 、CVAE d , concatenate each conditional autoencoder into a sequential conditional variational autoencoder, and sample the point cloud P obtained by the basic tool o ' -n and its corresponding normal vector P o ' -v Splicing, and extracting point cloud features through PointNet2 is recorded as P f The word vector Int encoded corresponding to the crawling intention feature Int vec concatenated as the conditional feature P of each conditional autoencoder f-int , conditional feature P f-int With three types of crawling knowledge k c , k p With k d As the input of each conditional autoencoder network, P f-int As the conditional input of the network, it guides the generation of hand-object multi-dimensional grasping knowledge under different intentions of basic tools in different categories. The training goal is to generate the hand-object multi-dimensional grasping knowledge k c ', k p ', k d 'With the input multi-dimensional crawling knowledge encoding k c , k p , k d consistency.

[0057] In the testing phase, the conditional feature P f-int The vector sampled from the latent variable space z in the conditional variational autoencoder is input into the decoder of the conditional variational autoencoder together to generate diverse hand-object multi-dimensional grasping knowledge.

[0058] The training process of the grasping posture generation network model based on multi-dimensional grasping knowledge encoding of intention is k c , k p With k d And the conditional feature P f-int , the output is k c ', k p ' and k d ', the amount of training data is 500 data in the hand-object multi-dimensional grasping data set corresponding to each grasping intention of each type of basic model, and 100 data are selected for verification, CVAE c 、CVAE p With CVAE dThe dimension z of the latent variable space is set to 16, the number of training rounds is 1000, the learning rate is 0.0008, the optimizer is Adam, and the training batch is set to 64.

[0059] The grasping posture generation network finally trained based on the multi-dimensional grasping knowledge encoding of the intention can be used to encode the multi-dimensional grasping knowledge k of the basic model under different intentions. c ', k p ' and k d 'Generate a better effect.

[0060] Diversified hand-object multi-dimensional grasping knowledge k generated by the hand-object multi-dimensional grasping knowledge generation network c ', k p ' and k d ', optimize and solve the grasping posture parameters. First, optimize the global posture of the grasping parameters, and then further optimize the grasping hand posture parameters, shape parameters, etc., and finally solve the grasping posture parameters corresponding to the hand-object multi-dimensional grasping knowledge under different grasping intentions.

[0061] Step 3: In view of the fact that the target tools in the same class mentioned in step 1 have multiple specifications, the generation effect of the grasping posture generation network based on the intention-based multi-dimensional grasping knowledge encoding of the basic tool trained in step 2 is reduced when generating the grasping posture. In order to solve this problem, the grasping knowledge transfer method for the target tools in the same class with different postures and the grasping posture optimization solution method for optimizing the collision relationship between fingers are adopted. The grasping knowledge on the basic tool is transferred to the target tool in the same class by using the similarity of the geometry and functional areas between the basic tool and the target tool in the same class. The corresponding grasping posture parameters are optimized and solved according to the grasping knowledge on the target tool obtained by the migration. The details are as follows:

[0062] Using the basic tool a obtained in step 2 o1 The grasping posture generation network based on intention-based multi-dimensional grasping knowledge encoding generates multiple grasping postures of basic tools under specified intentions and calculates the contact graph C of hand-object interaction. o1 .

[0063] The target tools in the same class have differences in size, shape, posture, etc. To address the differences in posture, the PCA point cloud registration method is used to calculate the posture transformation relationship [R, T] between the basic tool and the target tool in the class. o1-o2 , transform the target tool through the posture transformation [R,T] o1-o2 Align with the pose of the base tool.

[0064] The base tool and the aligned target tool are shaped encoded using DeepSDF, and linear interpolation is performed between the shape vectors encoded by the base tool and the target tool. The interpolated shape vector is decoded to obtain the interpolation model corresponding to the shape vector. The contact map C on the base tool is converted into the contact map C by using the small shape difference of adjacent interpolation models. o1 Gradually migrate from the basic tool to the target tool to obtain C o2 .

[0065] In the contact diagram C for the target tool o2 When solving the corresponding grasping posture, in order to avoid the self-collision between the dexterous hands, we designed a self-collision loss function L self-collision The grasping posture is optimized, and the corresponding grasping posture parameters of the dexterous hand are finally solved. The designed self-collision loss function is as follows:

[0066]

[0067] In the formula, i and j represent the numbers of the 21 key points of the hand, and h i ,h j Represents the coordinates of 21 key points of the hand, dis(h i -h j ) represents the distance between two different key points of the hand, δ ij Represents the threshold of the distance between two different key points of the hand, k ij Represents the weight of the distance loss between different key points. The loss function is designed to ensure that the distance dis(h i -h j ) is as large as possible from the set threshold δ ij , to avoid self-collision of the hands.

[0068] Step 4: Figure 3 As shown, according to the grasping posture of the human hand for the target tool within the class generated in step 3 above, the mapping relationship between the human hand and the five-finger dexterous hand based on the weighted key point position optimization of the dexterous hand kinematics is constructed, and the weighted key point optimization objective function is designed based on the corresponding relationship between the key points of the human hand and the key points of the dexterous hand. The grasping of the target tool within the class by the human hand under different intentions is mapped to the grasping of the target tool within the class by the five-finger dexterous hand under different intentions. So far, the dexterous hand grasping posture generation of multi-specification target tools within the class under different intentions is completed. The details are as follows:

[0069] The mapping relationship between the human hand and the five-finger dexterous hand is constructed based on the weighted key point position optimization of the dexterous hand kinematics, and the optimization objective function is designed as follows:

[0070]

[0071] In the formula, 21 represents the 21 key points of the hand. represents the position of the i-th hand key point, represents the position of the i-th key point corresponding to the kinematic optimization process of the five-finger dexterous hand, k i Represents the weight corresponding to the position difference of the corresponding key points between the i-th human hand and the five-finger dexterous hand, and its value is [0,1]. The optimization goal of the mapping relationship is to minimize the average difference between the corresponding key points so as to restore the grasping posture of the human hand as much as possible.

[0072] According to the parameters corresponding to the hand parametric model obtained by migration optimization, the 21 key point information of the hand in the hand parametric model is extracted through Manolayer. These 21 key point information correspond to the joints of the five-finger dexterous hand. The corresponding relationship is used to optimize the joint posture information of the five-finger dexterous hand. At the same time, it is noted that there is a certain range of motion (α1, α2) between the joints of the five-finger dexterous hand, and the solved joint postures must be limited within the range of motion.

[0073] According to the optimization objective function of the above mapping relationship, an iterative optimization process is designed to optimize the mapping relationship between the human hand and the five-finger dexterous hand. Starting from an initial posture of the dexterous hand, the iterative process is as close to the target posture as possible, which solves the problem of being easily trapped in the local optimum when the target posture and the initial posture are greatly different in the single-step posture optimization. The posture of the dexterous hand obtained by the previous step optimization is used as the initial value of the next step optimization. In the iterative optimization process, the number of iterative optimizations m is set and the corresponding threshold is set in combination with the grasping posture and the corresponding five-finger dexterous hand structure characteristics. The end of the iterative optimization process is divided into two cases. One is that the average error between the joint position and the key point of the hand during the optimization process is less than the set threshold, then the optimization process is ended; the other is that the optimization result of the grasping posture of the five-finger dexterous hand fails to converge to the set threshold within the set number of iterative optimizations m, and the maximum number of iterative optimizations is exceeded, then the mapping of the five-finger dexterous hand to the hand posture is forced to avoid wasting computing resources.

[0074] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for generating grasping postures of dexterous hands for tools of multiple specifications within a class, characterized in that: The following steps are involved: (1) For each type of tool, one is selected as the basic tool and the rest are selected as the target tools within the class. The grasping posture parameters corresponding to the hand parameterized MANO model of the basic tool under different intentions, the basic model transformation posture and the corresponding intention label are collected; (2) Processing the collected data, the sampling point cloud of the basic tool model and its corresponding normal vector, multi-dimensional crawling knowledge and word vectors corresponding to the crawling intention together form a multi-dimensional crawling knowledge dataset of the basic tools in the category; (3) Construct and train a grasping posture generation network based on intention-based multi-dimensional grasping knowledge encoding, including the construction of a hand-object multi-dimensional grasping knowledge generation network and the solution of grasping posture parameters; (4) In the process of grasping knowledge transfer, a point cloud registration method is added to focus on the posture difference between the basic tool and the target tool. According to the grasping knowledge of the target tool obtained by transfer, the grasping posture transfer grasping posture solution method of inter-finger collision is optimized. In the grasping posture solution process, a hand self-collision loss function is proposed to avoid the occurrence of inter-finger collision, and the grasping posture parameters of the target tool under different intentions are obtained. (5) The mapping relationship between the human hand and the five-finger dexterous hand is constructed by weighted key point position optimization based on the kinematics of the dexterous hand. According to the correspondence between the key points of the human hand and the key points of the dexterous hand, a weighted key point optimization objective function is proposed. The objective function is optimized using the iterative posture optimization method to realize the dexterous hand grasping posture generation of multi-specification target tools within the class under different intentions.

2. The method for generating grasping postures of dexterous hands for tools of multiple specifications within a category according to claim 1 is characterized in that: The implementation process of step (2) is as follows: The basic tool model is transformed and moved to the origin as the center, and n point cloud coordinates and their corresponding normal vectors are sampled; The n points sampled from the basic tool model and the grasping posture parameters corresponding to the hand parameterized model are used to restore the hand parameterized model posture, and the contact relationship between the hand parameterized model and the tool point cloud under the corresponding posture is encoded into a multi-dimensional grasping knowledge, which is encoded into a hand-object contact probability map, a hand-object contact part map, and a hand-object contact direction map; the above constitutes the multi-dimensional grasping knowledge; Encode the grasping intention label in the form of word vectors to guide the generation direction of grasping posture and achieve functional grasping; The captured data with few data samples are expanded accordingly to ensure that the captured data samples with different intentions are balanced.

3. The method for generating grasping postures of dexterous hands for tools of multiple specifications within a category according to claim 1 is characterized in that: The implementation process of constructing the hand-object multi-dimensional grasping knowledge generation network in step (3) is as follows: The hand-object multi-dimensional grasping knowledge generation network is an improved conditional variational autoencoder structure. For different hand-object grasping knowledge descriptions, a conditional variational autoencoder is constructed respectively, and each conditional variational autoencoder is connected in series to form a sequential conditional variational autoencoder. The tool point cloud feature and the grasping intention feature are concatenated as the conditional feature of each conditional variational autoencoder. The conditional features and multi-dimensional grasping knowledge are used as the input of each conditional variational autoencoder network. They are also used as the conditional input of the network to guide the generation of hand-object multi-dimensional grasping knowledge of different basic tools under different intentions. The training goal is to ensure the consistency of the generated hand-object multi-dimensional grasping knowledge and the input multi-dimensional grasping knowledge encoding. During the testing phase, the conditional features are input together with the vectors sampled in the latent variable space of the conditional variational autoencoder and input into the decoder of the conditional variational autoencoder to generate diverse hand-object multi-dimensional grasping knowledge.

4. The method for generating grasping postures of dexterous hands for tools of multiple specifications within a category according to claim 1 is characterized in that: The process of solving the attitude parameters in step (3) is as follows: Based on the diverse hand-object multi-dimensional grasping knowledge, the grasping posture parameters are optimized and solved. First, the global posture of the grasping parameters is optimized, and then the hand posture parameters and shape parameters of the grasping are further optimized to obtain the grasping posture parameters corresponding to the hand-object multi-dimensional grasping knowledge under different grasping intentions.

5. The method for generating grasping postures of dexterous hands for tools of multiple specifications within a category according to claim 1 is characterized in that: The implementation process of step (4) is as follows: Calculate the contact graph of hand-object interaction based on multiple grasping postures generated by the basic tool under specified intentions; In view of the posture difference between the target tool and the basic tool in the class, the PCA point cloud registration method is used to calculate the posture transformation relationship between the basic tool and the target tool in the class, and the target tool is aligned with the posture of the basic tool through posture transformation; The base tool and the aligned target tool are shaped encoded using DeepSDF, and linear interpolation is performed between the shape vectors encoded by the base tool and the target tool. The interpolated shape vectors are decoded to obtain the interpolation model corresponding to the shape vector. The contact map is migrated from the base tool to the target tool by taking advantage of the small shape difference between adjacent interpolation models. When solving the corresponding grasping posture for the contact graph of the target tool, a self-collision loss function is designed to optimize the grasping posture.

6. The method for generating grasping postures of dexterous hands for tools of multiple specifications within a category according to claim 1 is characterized in that: The process of constructing the mapping relationship between the human hand and the five-finger dexterous hand based on the weighted key point position optimization of the dexterous hand kinematics described in step (5) is as follows: According to the generated grasping posture of the human hand for the target tool, combined with the similarity between the human hand and the five-finger dexterous hand, different corresponding weights are set using the key points of the hand and the joints of the five-finger dexterous hand. The mapping relationship of the corresponding key point positions under different weights is optimized through the kinematics of the five-finger dexterous hand, and the grasping posture of the human hand for the tool is transferred to the five-finger dexterous hand, realizing the functional grasping of the tool by the five-finger dexterous hand under different intentions; the optimization goals of the mapping relationship are as follows: in, represents the position of the i-th hand key point, represents the position of the i-th key point corresponding to the kinematic optimization process of the five-finger dexterous hand, k i Represents the weight corresponding to the position difference of the key points between the i-th hand and the five-finger dexterous hand.

7. The method for generating grasping postures of dexterous hands for tools of multiple specifications within a category according to claim 1 is characterized in that: The implementation process of optimizing the objective function using the iterative posture optimization method in step (5) is as follows: The mapping of the dexterous hand's grasping posture is divided into multiple iterations. The grasping posture parameters of the dexterous hand obtained by the last iteration are used to gradually approach the target posture. According to the structure of the dexterous hand and the characteristics of the grasping posture, a reasonable threshold is set to avoid the iterative posture optimization process falling into an infinite loop.

8. The method for generating grasping postures of dexterous hands for tools of multiple specifications within a category according to claim 2 is characterized in that: The hand-object contact probability map represents the contact probability between the hand and each point in the object point cloud, indicating which part of the object may be in contact with the hand; the hand-object contact direction map calculates the contact direction between the point on the object point cloud and the corresponding hand contact area.

9. The method for generating grasping postures of dexterous hands for tools of multiple specifications within a category according to claim 2 is characterized in that: The hand-object contact portion diagram represents dividing the hand into sixteen regions and calculating the contact conditions between the object and different regions of the hand.

10. The method for generating grasping postures of dexterous hands for tools of multiple specifications within a category according to claim 5, characterized in that: The self-collision loss function is as follows: Among them, i and j represent the numbers of the 21 key points of the hand, and h i ,h j Represents the coordinates of 21 key points of the hand, dis(h i -h j ) represents the distance between two different key points of the hand, δ ij Represents the threshold of the distance between two different key points of the hand, k ij Represents the weight of the distance loss between different keypoints.

Citation Information

Patent Citations

  • Three-dimensional point cloud registration method based on evolutionary multi-task optimization

    CN116363179A

  • Workpiece grabbing method, device and equipment based on point cloud segmentation and storage medium

    CN117689887A

  • Five-finger manipulator grabbing and operating system based on single-track teaching

    CN117901147A

  • Robotic grasping prediction method based on triplet contrastive network

    WO2024087331A1

Cited By

  • Virtual gesture generation method and system based on contact modeling and contact area alignment

    CN119916946A

  • Grabbing type prior-based dexterous hand grabbing method and equipment and medium

    CN120095812A

  • Dexterous hand self-adaptive grabbing method, dexterous hand control system and storage medium

    CN121132718A