A dexterous hand grasping method and device based on grasping type prior and a medium

Abstract

The application discloses a dexterous hand grasping method and device based on grasping type prior, and a medium, wherein the method comprises the following steps: acquiring point cloud information of an object to be grasped, and judging a grasping type according to the point cloud information; wherein the grasping type comprises a palm type grasping type and a finger type grasping type; generating a contact heat map of the object to be grasped and a dexterous hand according to the point cloud information and the obtained grasping type; taking the contact heat map as a constraint, generating a preliminary grasping posture according to the point cloud information, that is, the displacement and rotation of the dexterous hand relative to the object to be grasped and the joint angle of the dexterous hand; and according to the grasping type, adopting a corresponding optimization strategy to optimize the preliminary grasping posture to generate a final grasping posture. The application guides the generation of the grasping posture through the grasping type prior, improves the applicability and stability of the dexterous hand grasping, and is suitable for the application of a robot dexterous hand in various common grasping scenes. The application can be widely applied to the field of robot grasping.

Description

Technical Field

[0001] This invention relates to the field of robotic grasping, and more particularly to a dexterous hand grasping method, device, and medium based on grasping type prior. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, dexterous hands are playing an increasingly important role in fields such as industrial automation, service robots, and medical rehabilitation. The core function of a dexterous hand is to flexibly adjust its grasping strategy according to the shape, material, and task requirements of an object to achieve stable and efficient grasping. The core issue of dexterous hand grasping technology is how to accurately predict the grasping posture of a given object based on its three-dimensional representation. The object grasping posture typically includes three parts: (1) the displacement of the dexterous hand relative to the center of the object to be grasped; (2) the rotation of the dexterous hand relative to the center of the object to be grasped; and (3) the angles of the various joints of the dexterous hand.

[0003] Currently, most dexterous hand grasping methods generate grasping postures based on the object's geometric information (such as point clouds). However, existing methods often ignore prior knowledge of the object's grasping type. For example, when facing an object with a convex surface (such as a vase), existing methods may generate inappropriate grasping postures, such as using the palm to grasp the body of the vase instead of using the fingers to grasp the top. Humans can quickly select appropriate grasping postures based on the object's shape, size, and characteristics when grasping objects. Therefore, if robots could learn from this prior information about the grasping type, the stability and accuracy of grasping would be greatly improved. Summary of the Invention

[0004] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a dexterous hand grasping method, device and medium based on grasping type prior.

[0005] The first technical solution adopted in this invention is:

[0006] A dexterous hand grasping method based on grasping type prior includes the following steps:

[0007] Obtain the point cloud information of the object to be grasped, and determine the grasping type based on the point cloud information; the grasping type includes palm-shaped grasping type and finger-shaped grasping type;

[0008] Based on the point cloud information and the obtained grasping type, a contact heat map between the object to be grasped and the dexterous hand is generated.

[0009] Using the contact heatmap as a constraint, a preliminary grasping posture is generated based on the point cloud information, namely the displacement and rotation of the dexterous hand relative to the object to be grasped, as well as the joint angles of the dexterous hand.

[0010] Based on the grasping type, an appropriate optimization strategy is adopted to optimize the initial grasping posture and generate the final grasping posture. Further, a pre-trained grasping type discrimination module is used to predict the grasping type. The input of the grasping type discrimination module is the point cloud information of the object, and the output is the corresponding grasping type.

[0011] The PointNet++ model is used to extract local features of the object point cloud, the Transformer model is used to capture global features of the object point cloud, and a Graph Convolutional Network (GNN) is used to extract topological features from the topology of the point cloud. The features are then fused through a self-attention mechanism, and the output grasp type is predicted.

[0012] Furthermore, the expression for the local feature is:

[0013] F local =f PointNet++ (P)

[0014] In the formula, F local P represents the local features of an object, where P is the object's point cloud.

[0015] The expression for the global feature is:

[0016] F global =f Transformer (P)

[0017] In the formula, F global P represents the local features of an object, where P is the object's point cloud.

[0018] The expression for the topological feature is:

[0019] F topo =f GNN (P)

[0020] In the formula, F topo P represents the local features of an object, where P is the object's point cloud.

[0021] The expression for the features fused using the self-attention mechanism is:

[0022] F final =Attention(F local ,F global ,F topo )

[0023] In the formula, F final Indicates the final characteristics after fusion;

[0024] The classification head (MLP) outputs the object's grasping type based on the fused features:

[0025]

[0026] In the formula, The predicted grasping type, either palm-type or finger-type.

[0027] Furthermore, a pre-trained contact heat map generation module is used to generate contact heat maps;

[0028] For palm-type and finger-type grasping types, two structurally identical conditional variational autoencoders (CVAEs) are trained to generate contact heatmaps of finger-type and palm-type grasping objects.

[0029] The conditional variational autoencoder generates a contact heatmap based on the object's point cloud information. The expression is:

[0030]

[0031] In the formula, (H contacy |P) represents the conditional probability distribution of the contact heatmap, and H contact For contact heatmaps, P is the point cloud of the object, μ is the mean, and σ is the standard deviation;

[0032] Contact thermal mapping is used to constrain key areas of the contact region on the object's surface to optimize the grasping posture of the dexterous hand. The objective function is:

[0033]

[0034] In the formula, z is a latent variable, p(H|z) is the output probability distribution of the generator; KL represents the Kullback-Leibler divergence; q(z|P) is , p(z) is ; the generated contact heat map H={h1,h2,…,h m}, where h j ∈[0,1] represents the contact strength of each contact point in the point cloud, which is used to constrain the grasping posture of the dexterous hand; For the desired operation.

[0035] Furthermore, a pre-trained grasping posture generation module is used to generate an initial grasping posture;

[0036] The grasping posture generation module works as follows:

[0037] The point cloud information of the object is serialized and embedded to obtain a high-dimensional feature representation, expressed as:

[0038] F embedding =f embedding (P)

[0039] In the formula, F embedding P represents the embedded features, where P is the object point cloud.

[0040] Spatial compression of the embedded features is achieved through a grid pooling layer, expressed as follows:

[0041] F pooled =f GridPool (F embedding )

[0042] In the formula, F pooled These are the features after pooling;

[0043] The spatial representation of features is optimized using conditional location encoding and attention mechanisms, expressed as follows:

[0044] F att =f Attention (F pooled )

[0045] In the formula, F att Features optimized through attention mechanisms;

[0046] The pose is captured by predicting the head output, and the expression is:

[0047]

[0048] In the formula, The predicted grasping pose includes the object's displacement, rotation, and the joint angles of the dexterous hand.

[0049] Furthermore, the loss function is calculated and backpropagation is performed:

[0050]

[0051] In the formula, λ1, λ2, and λ3 are weighting coefficients used to balance the effects of different loss terms; This represents displacement loss, where T is the actual displacement. To predict displacement, ∥∥ denotes the Euclidean norm; This represents the rotational displacement, where θ is the angle between the predicted rotation and the actual rotation; Indicates joint angle loss, J i For true joint angles, To predict joint angles, M represents the number of joints in a dexterous hand.

[0052] Furthermore, the step of optimizing the initial grasping posture using a corresponding optimization strategy based on the grasping type to generate the final grasping posture includes:

[0053] For palm-type grasping, firstly optimize displacement and rotation to bring the palm as close as possible to the grasping area to provide sufficient gripping force; then optimize the joint angles of the fingers to assist in grasping; finally, optimize to reduce unreasonable hand-object penetration.

[0054] For finger-type grasping, directly optimize the joint angle of the fingers to make the fingers fit as closely as possible to ensure sufficient gripping force; reduce unreasonable hand-object penetration through optimization.

[0055] Furthermore, the formula for optimizing the palm-shaped grasping type is:

[0056] E palm =Distance(palm,object)

[0057]

[0058] In the formula, E palm The distance between the hand and the object is represented by: palm (the palm portion of the dexterous hand) and object (the object to be grasped); R is the rotation of the dexterous hand relative to the center of the object; t is the rotation of the dexterous hand relative to the center of the object; Δ(R,t) represents the change in relative position between the hand and the object under the combined effects of rotation R and displacement t; E pen Indicates the penetration measurement of a hand object. Indicates posture The dexterous hand region under t,q, where p represents a sampling point on the surface of the palm or fingers in the dexterous hand model, and d p,object Let represent the distance from point p to the surface of the object's point cloud; q is the joint angle of the dexterous hand; Δ(R,t,q) represents the angle caused by rotation. The relative pose change between the hand and the object under the combined action of displacement t and joint angle q; η is the learning rate.

[0059] The formula for optimizing finger-type grasping is:

[0060] E finger{i} =Distance(finger{i},object)

[0061]

[0062] In the formula, E finger{i} This represents the distance between a finger and an object, where finger{i} represents the i-th finger, and there are 5 fingers in total; qpos represents the distance between a finger and an object. finger{i} This represents the joint angle of the i-th finger in the dexterous hand, Δ(qpos) finger{i} ) indicates the joint angle qpos finger{i} Changes in the relative posture between the fingers and the object under the influence of the action.

[0063] The second technical solution adopted in this invention is:

[0064] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a dexterous hand grasping method based on grasping type prior as described above.

[0065] The third technical solution adopted in this invention is:

[0066] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a dexterous hand grasping method based on grasping type prior as described above.

[0067] The fourth technical solution adopted in this invention is:

[0068] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions to cause the computer device to perform the method described above.

[0069] The beneficial effects of this invention are: by guiding the generation of grasping posture through prior knowledge of grasping type, this invention improves the applicability and stability of dexterous hand grasping, and is applicable to the application of robot dexterous hands in a variety of common grasping scenarios. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a flowchart of the steps of a dexterous hand grasping method based on grasping type prior in an embodiment of the present invention;

[0072] Figure 2 This is a schematic diagram of the structure of the capture type discrimination module in an embodiment of the present invention;

[0073] Figure 3 This is a schematic diagram of the grasping posture generation module in an embodiment of the present invention. Detailed Implementation

[0074] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0075] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0076] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0077] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0078] Currently, most existing dexterous hand grasping methods rely on the object's geometry and point cloud data to directly generate the grasping posture. However, these methods suffer from poor grasping adaptability and insufficient grasping stability. Specifically, when faced with objects with complex shapes or irregular surfaces, existing methods often fail to effectively determine the appropriate grasping type (such as finger grasping or palm grasping), resulting in inaccurate grasping postures or even failures. Furthermore, most methods neglect the proper planning of the contact area between the object and the dexterous hand, leading to uneven distribution of grasping force and further affecting grasping stability.

[0079] To address these issues, this invention proposes a dexterous hand grasping method based on grasping type prior. It fundamentally improves the adaptability and stability of grasping by introducing modules for grasping type discrimination, contact heat map generation, and grasping posture optimization. First, the grasping type discrimination module accurately determines the object's grasping type (finger-type or palm-type), ensuring the dexterous hand selects the most suitable grasping method. Next, the contact heat map generation module provides precise contact area information for the grasping posture, allowing for reasonable distribution of gripping force during grasping and preventing unnecessary hand-object penetration. Finally, combined with the grasping posture optimization module, this invention further optimizes displacement, rotation, and joint angles, ensuring the stability and efficiency of the grasping posture. Compared to traditional methods, the innovation of this invention lies in its modular design and multi-level optimization, which not only improves the grasping adaptability for different object types but also significantly enhances the stability and success rate during the grasping process, especially when dealing with objects with complex shapes and irregular surfaces.

[0080] Example 1

[0081] like Figure 1 As shown, this embodiment provides a dexterous hand grasping method based on grasping type prior information. By combining the object's point cloud information and grasping type prior information, it achieves efficient and stable grasping posture generation. The method specifically includes the following steps:

[0082] S1. Obtain the point cloud information of the object to be grasped, and determine the grasping type based on the point cloud information; the grasping type includes palm-type grasping type and finger-type grasping type.

[0083] In this embodiment, a grasping type discrimination module is constructed to determine the grasping type. The input of the grasping type discrimination module is the point cloud information of the object to be grasped, and the output is its grasping type (finger type or palm type).

[0084] Specifically, see Figure 2 The PointNet++ model is constructed to extract local features of the point cloud, the Transformer model captures global features, a Graph Convolutional Network (GNN) is constructed to learn features from the topology of the point cloud, and finally the various features are fused through a self-attention mechanism to form a grasping type discrimination module and output the grasping type.

[0085] As one implementation method, the crawling type discrimination module works as follows:

[0086] S21. Extract features based on point cloud information.

[0087] 1) Use the PointNet++ network to extract local features from the volumetric point cloud. PointNet++ extracts local features at different scales through hierarchical aggregation operations, as shown in the following formula:

[0088] F local =f PointNet++ (P)

[0089] Where F local P represents the local features of an object, where P is the object's point cloud.

[0090] 2) Use the Transformer model to capture global features and long-range dependencies in point cloud information, as shown in the following formula:

[0091] F global =f Transformer (P)

[0092] Where F global P represents the local features of an object, where P is the object's point cloud.

[0093] 3) Use a Graph Convolutional Network (GNN) to process the topology of the point cloud and extract topological features, as shown in the following formula:

[0094] F topo =f GNN (P)

[0095] Where F topo P represents the local features of an object, where P is the object's point cloud.

[0096] S22. Feature fusion: Features from PointNet++, Transformer, and GNN are fused using a self-attention mechanism, as shown in the following formula:

[0097] F final =Attention(F local ,F global ,F topo )

[0098] Where F final This represents the final characteristic after fusion.

[0099] S23. Predicted Output: Using the classification head (MLP), output the object's grasping type based on the fused features.

[0100]

[0101] in The predicted grasping type (palm-type or finger-type).

[0102] S24. Calculate the loss function and backpropagate, using cross-entropy loss to measure the difference between the predicted class and the true class:

[0103]

[0104] Where N is the number of training samples, y i It is the true class label of the i-th sample. That is the corresponding predicted probability.

[0105] In this embodiment, PointNet++ extracts local geometric features from the point cloud, preserving detailed information about the object's surface, making it suitable for handling complex object shapes. Transformer captures the global context of the object's point cloud, making it particularly suitable for modeling features with long-distance dependencies, such as the overall shape and structure of an object. GNN learns the object's morphological features from the topological structure of the point cloud, such as the connections between points, which is crucial for extracting structured features from complex objects. Finally, the self-attention mechanism is introduced for feature fusion to integrate local, global, and topological features, ensuring information complementarity between different features and improving the classification module's ability to discriminate the grasping type.

[0106] S2. Based on the point cloud information and the obtained grasping type, generate a contact heat map between the object to be grasped and the dexterous hand.

[0107] In this embodiment, a contact heatmap generation module is used to generate contact heatmaps. The contact heatmap generation module is trained by training two identical conditional variational autoencoders (CVAEs) for both palm-shaped and finger-shaped grasping objects. Using object point cloud information as input, the module predicts the contact heatmaps of the corresponding grasping type object and the dexterous hand, serving as constraints for subsequent grasping posture generation and post-optimization processes.

[0108] The contact thermal mapping generation module uses two conditional variational autoencoders (CVAEs) to generate contact thermal maps for finger-shaped and palm-shaped grasping objects, respectively. The CVAE model generates the contact thermal map based on the object's point cloud information. The formula is expressed as:

[0109]

[0110] Wherein, p(H contact |P) represents the conditional probability distribution of the contact heatmap, μ is the mean, and σ is the standard deviation. The training objective is to maximize the variational lower bound (ELBO), as shown in the formula:

[0111]

[0112] in, For the desired operation, D KL Kullback-Leibler divergence measures the difference between the true distribution and the generated distribution.

[0113] S3. Using the contact heatmap as a constraint, generate a preliminary grasping posture based on the point cloud information, namely the displacement and rotation of the dexterous hand relative to the object to be grasped, as well as the joint angles of the dexterous hand.

[0114] In this embodiment, a preliminary grasping posture is generated through a grasping pose generation module. The grasping pose generation module is trained by sequentially passing through a serialization layer, embedding layer, mesh pooling layer, sequential randomization layer, conditional position encoding, layer normalization, attention layer, layer normalization, multilayer perceptron, and prediction head, based on the point cloud information of the object, to predict and output the grasping posture, namely the displacement and rotation of the dexterous hand relative to the object, as well as the joint angles of the dexterous hand.

[0115] As an optional implementation, see [link to implementation details]. Figure 3 The working steps of the pose capture generation module include:

[0116] S41. After processing the object point cloud data through serialization, embedding, etc., a high-dimensional feature representation is obtained, as shown in the formula:

[0117] F embedding =f embedding (P)

[0118] Among them, F embedding These are the features after embedding.

[0119] S42. Spatial compression of point cloud data using a mesh pooling layer:

[0120] F pooled =f GridPool (F embedding )

[0121] S43. Optimize the spatial representation of features using conditional location encoding and attention mechanisms:

[0122] F att =f Attention (F pooled )

[0123] Among them, F att Features optimized through attention mechanisms.

[0124] S44. Capture pose by predicting head output:

[0125]

[0126] in, The predicted grasping pose includes the object's displacement, rotation, and the joint angles of the dexterous hand.

[0127] S45, Calculate the loss function and backpropagate:

[0128]

[0129] In the formula, λ1, λ2, and λ3 are weighting coefficients used to balance the effects of different loss terms; This represents displacement loss, where T is the actual displacement. To predict displacement, ∥∥ denotes the Euclidean norm; This represents the rotational displacement, where θ is the angle between the predicted rotation and the actual rotation; Indicates joint angle loss, J i For true joint angles, To predict joint angles, M represents the number of joints in a dexterous hand.

[0130] S4. Based on the grasping type, adopt the corresponding optimization strategy to optimize the initial grasping posture and generate the final grasping posture.

[0131] For example, post-optimization is performed on the generated initial grasping pose: different post-optimization strategies are applied to optimize the pose based on the grasping type.

[0132] 1) For palm-shaped hands, first optimize displacement and rotation to bring the palm as close as possible to the grasping area to provide sufficient gripping force; then optimize the joint angles of the fingers to assist in grasping; finally, optimize to reduce unreasonable hand-object penetration; the specific formula is as follows:

[0133] E palm =Distance(palm,object)

[0134]

[0135] In the formula, E palm The distance between the hand and the object is represented by: palm (the palm portion of the dexterous hand) and object (the object to be grasped); R is the rotation of the dexterous hand relative to the center of the object; t is the rotation of the dexterous hand relative to the center of the object; Δ(R,t) represents the change in relative position between the hand and the object under the combined effects of rotation R and displacement t; E pen Indicates the penetration measurement of a hand object. Indicates posture The dexterous hand region under t,q, where p represents a sampling point on the surface of the palm or fingers in the dexterous hand model, and d p,object Let represent the distance from point p to the surface of the object's point cloud; q is the joint angle of the dexterous hand; Δ(R,t,q) represents the angle caused by rotation. The relative pose change between the hand and the object under the combined action of displacement t and joint angle q; η is the learning rate.

[0136] 2) For finger-type grips, directly optimize the joint angles of the fingers to ensure they are as close to the object as possible, guaranteeing sufficient grip strength; subsequently, optimize for hand-object penetration issues as well, using the following formula:

[0137] E finger{i} =Distance(finger{i},object)

[0138]

[0139] In the formula, E finger{i} This represents the distance between a finger and an object, where finger{i} represents the i-th finger, and there are 5 fingers in total; qpos represents the distance between a finger and an object. finger{i} This represents the joint angle of the i-th finger in the dexterous hand, Δ(qpos) finger{i} ) indicates the joint angle qpos finger{i} Changes in the relative posture between the fingers and the object under the influence of the action.

[0140] In this embodiment, the advantage of this post-optimization strategy is that it is tailored to different gripping types, especially for the displacement and rotation optimization of palm-shaped gripping, which significantly improves the rationality of gripping force distribution and effectively reduces the problem of hand-object penetration in gripping failures.

[0141] As a preferred implementation, the method of this embodiment also includes the step of constructing a training set: selecting an existing dexterous grasping dataset, generating a contact heat map by analyzing the contact between the object point cloud and various parts of the dexterous hand, and labeling the grasping type according to the part that provides the main grasping force in the grasping data, including finger grasping and palm grasping, that is, the main grasping force is provided by the fingers or the palm respectively, to prepare data for subsequent training of each module.

[0142] Optionally, the aforementioned smart grasping dataset is a publicly available image dataset on the internet, such as DexGraspNet. Specifically, DexGraspNet is a simulation dataset that provides 1126 object classes and their corresponding grasping poses.

[0143] Specifically, the steps for constructing the training set include the following steps A1-A3:

[0144] A1. Point cloud data acquisition: By acquiring the point cloud data of the objects, the 3D shape information of each object is used for model training in subsequent steps.

[0145] A2. Contact Heatmap Generation: A contact heatmap is calculated using the point cloud of the object and the contact details between different parts of the dexterous hand. The contact heatmap represents the intensity of contact between different areas of the object's surface and the dexterous hand. Specifically, the following formula is used:

[0146]

[0147] Among them, H contact For contact thermal mapping, N is the number of points on the object's surface, P i Let S be the coordinates of the i-th point in the point cloud. j Let be the j-th part of the dexterous hand, and σ(·) be the function for calculating the contact strength.

[0148] A3. Grasping type labeling: Based on the main parts of the grasping force provided by the grasping force of the object in the data set, the grasping type of the object is labeled. It is mainly divided into palm grasping and finger grasping, that is, the force of grasping the object is mainly borne by the palm or the fingers respectively.

[0149] In a preferred embodiment, the method further includes a step of testing each trained module. A test set is selected from the smart grasping dataset, and the point cloud information of the object to be grasped in the test set is input into each of the trained modules according to a preset logic, outputting the corresponding preliminary grasping posture. During testing, the object's point cloud information is first input, and the grasping type discrimination module determines the object's grasping type. Based on the grasping type, the corresponding contact heatmap generation module is selected to generate the corresponding contact heatmap. Then, the contact heatmap is used as a constraint input to the grasping posture generation module to predict an appropriate grasping posture.

[0150] Specifically, the testing procedure includes the following steps B1-B5:

[0151] B1. Input point cloud data: Select the point cloud data P of the object to be grasped from the test set. test As input.

[0152] B2. Grasping Type Determination: Input the point cloud data into the trained grasping type determination module, and output the predicted grasping type:

[0153] y test =TypeClassifier(P test )

[0154] Among them, y test ∈{Finger,Palm}.

[0155] B3. Select the contact heatmap generation module: based on the predicted crawling type y test Select the corresponding contact heatmap generation module (CVAE model) and generate the point cloud data P. test Input a CVAE model and generate a contact heatmap:

[0156] H test =CVAE ytest (P test )

[0157] Among them, Htest It is the generated contact thermal map.

[0158] B4. Grasping posture prediction: Predicting the contact heatmap H test Point cloud data P test The input is sent to the grasping pose generation module to predict the initial grasping posture:

[0159]

[0160] in, This includes predicted displacement, rotation, and joint angles.

[0161] B5. Optimize the gripping posture using the generated contact heat map. First, calculate the contact heat map between the current gripping posture and the actual object:

[0162]

[0163] E contact =MSE(H test H gt )

[0164]

[0165] In the formula, H gt E represents the contact heat map between the actual dexterous hand and the object to be grasped. contact The contact error metric is MSE, which stands for Mean Square Error, and is used to measure the difference between the predicted contact heat map and the actual contact heat map. This represents the current dexterity hand posture parameters, including the rotation of the dexterity hand relative to the center of the object to be grasped. η is the learning rate, which is the displacement t relative to the center of the object to be grasped and the joint angle q.

[0166] In summary, the method of this invention addresses the problems of poor grasping adaptability and insufficient grasping stability in existing technologies by introducing modules such as grasping type discrimination, contact heat map generation, and grasping posture optimization. Through the synergistic effect of the grasping type discrimination module and the contact heat map generation module, the grasping posture generation process is optimized, ensuring the distribution of grasping force and the rational configuration of the dexterous hand. Furthermore, a more reasonable post-optimization strategy further improves the accuracy and efficiency of grasping. Compared with existing technologies, it has at least the following advantages and beneficial effects:

[0167] (1) The present invention improves the rationality and stability of the grasping posture and can generate an appropriate grasping posture according to the shape and characteristics of the object.

[0168] (2) The present invention optimizes the grasping posture generation process through the synergistic effect of the grasping type discrimination module and the contact heat map generation module, ensuring the distribution of grasping force and the reasonable configuration of dexterous hand.

[0169] (3) The method of the present invention can adapt to the grasping of different types of objects, and further improve the grasping accuracy and efficiency through post-optimization strategies.

[0170] Example 2

[0171] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to achieve the following: Figure 1 This illustrates a dexterous hand grasping method based on grasping type prior.

[0172] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.

[0173] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: a Central Processing Unit (CPU) and a modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0174] Since this electronic device is an electronic device corresponding to a dexterous hand grasping method based on grasping type prior according to an embodiment of the present invention, and the principle of solving the problem by this electronic device is similar to that of the method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0175] Example 3

[0176] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to achieve the following: Figure 1 This illustrates a dexterous hand grasping method based on grasping type prior.

[0177] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0178] Since the storage medium is the storage medium corresponding to a dexterous hand grasping method based on grasping type prior in an embodiment of the present invention, and the principle of the storage medium in solving the problem is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0179] Example 4

[0180] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of a dexterous hand grasping method based on grasping type prior according to various exemplary embodiments of the present application described above. The executable computer program code or "code" used to perform the various embodiments can be written in high-level programming languages ​​such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0181] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0182] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0183] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A dexterous hand grasping method based on grasping type prior, characterized in that, Includes the following steps: Obtain the point cloud information of the object to be grasped, and determine the grasping type based on the point cloud information; The grasping types include palm grasping type and finger grasping type; Based on the point cloud information and the obtained grasping type, a contact heat map between the object to be grasped and the dexterous hand is generated. Using the contact heatmap as a constraint, a preliminary grasping posture is generated based on the point cloud information, namely the displacement and rotation of the dexterous hand relative to the object to be grasped, as well as the joint angles of the dexterous hand. Based on the grasping type, the initial grasping posture is optimized using the corresponding optimization strategy to generate the final grasping posture; A pre-trained contact heat map generation module is used to generate contact heat maps; For palm-type and finger-type grasping types, two structurally identical conditional variational autoencoders are trained to generate contact heat maps of objects grasped by fingers and palms. The conditional variational autoencoder generates a contact heatmap based on the object's point cloud information. The expression is: In the formula, The conditional probability distribution of the contact heatmap. For contact thermal mapping, Point cloud for objects, The mean, Standard deviation; Contact thermal mapping is used to constrain key areas of the contact region on an object's surface to optimize the grasping posture of a dexterous hand. The objective function is: In the formula, As a latent variable, The output probability distribution of the generator; Indicates the Kullback-Leibler divergence; the generated contact heat map ,in This represents the contact strength of each contact point in the point cloud, used to constrain the grasping posture of the dexterous hand; For the desired operation.

2. The dexterous hand grasping method based on grasping type prior as described in claim 1, characterized in that, A pre-trained grasping type discrimination module is used to predict the grasping type. The input of the grasping type discrimination module is the point cloud information of the object, and the output is the corresponding grasping type. The PointNet++ model is used to extract local features of the object point cloud, the Transformer model is used to capture global features of the object point cloud, and a graph convolutional network is used to extract topological features from the topology of the point cloud. The features are then fused through a self-attention mechanism, and the output grasp type is predicted.

3. The dexterous hand grasping method based on grasping type prior as described in claim 2, characterized in that, The expression for the local feature is: In the formula, To represent local features of an object, Point cloud for objects; The expression for the global feature is: In the formula, Represents the global characteristics of an object. Point cloud for objects; The expression for the topological feature is: In the formula, Represents the topological features of an object. Point cloud for objects; The expression for the features fused using the self-attention mechanism is: In the formula, Indicates the final characteristics after fusion; The classification head outputs the object's grasping type based on the fused features: In the formula, This indicates the predicted crawling type.

4. The dexterous hand grasping method based on grasping type prior as described in claim 1, characterized in that, A pre-trained grasping posture generation module is used to generate an initial grasping posture; The working principle of the grasping posture generation module is as follows: The point cloud information of the object is serialized and embedded to obtain a high-dimensional feature representation, expressed as: In the formula, For the embedded features, Point cloud for objects; Spatial compression of the embedded features is achieved through a grid pooling layer, expressed as follows: In the formula, These are the features after pooling; The spatial representation of features is optimized using conditional location encoding and attention mechanisms, expressed as follows: In the formula, Features optimized through attention mechanisms; The pose is captured by predicting the head output, and the expression is: In the formula, The predicted grasping pose includes the displacement of the dexterous hand relative to the object to be grasped. Rotation and the joint angles of dexterous hands .

5. The dexterous hand grasping method based on grasping type prior according to claim 4, characterized in that, Calculate the loss function and backpropagate: In the formula, These are weighting coefficients used to balance the effects of different loss terms; Indicates displacement loss. For the actual displacement, To predict displacement, Denotes the Euclidean norm; Indicates rotational loss. To predict the angle between the rotation and the actual rotation; Indicates joint angle loss, For true joint angles, To predict joint angles, The number of joints in a dexterous hand.

6. The dexterous hand grasping method based on grasping type prior according to claim 1, characterized in that, The step of optimizing the initial grasping posture according to the grasping type and using corresponding optimization strategies to generate the final grasping posture includes: For palm-type grasping, firstly optimize displacement and rotation to bring the palm as close as possible to the grasping area to provide sufficient gripping force; then optimize the joint angles of the fingers to assist in grasping; finally, optimize to reduce unreasonable hand-object penetration. For finger-type grasping, directly optimize the joint angle of the fingers to make the fingers fit as closely as possible to ensure sufficient gripping force; reduce unreasonable hand-object penetration through optimization.

7. The dexterous hand grasping method based on grasping type prior as described in claim 6, characterized in that, The formula for optimizing the palm-shaped grasping type is: In the formula, A measure of the distance between the palm of the hand and an object. The palm portion representing a dexterous hand. Indicates the object to be grabbed; The rotation of the dexterous hand relative to the center of the object. This represents the displacement of the dexterous hand relative to the center of the object. Indicates rotation and displacement The relative positional change between the palm and the object under the combined effect; Indicates the penetration measurement of a hand object. Indicates posture The dexterity hand area below, This represents a sampling point on the surface of the palm or fingers in a dexterous hand model. Point Distance to the surface of the object's point cloud; For the joint angles of a dexterous hand. Indicates rotation Displacement and joint angle The relative pose changes between the palm and the object under the combined action; The learning rate; In the formula, This measures the distance between a finger and an object. This represents the i-th finger, of which there are 5 fingers in total. This indicates the joint angle of the i-th finger in a dexterous hand. Indicates the joint angle Changes in the relative posture between the fingers and the object under the influence of the action.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method as described in any one of claims 1 to 7.

Images