Optimization method for generating robust and physically feasible grasp poses for robotic hands

Abstract

The application discloses an optimization method for generating robust and physically feasible grasping poses for a robot dexterous hand, belonging to the field of robot control. The method comprises the following steps: step 1, predicting the 3D affordance of an object using a conditional variational autoencoder model, learning the geometric properties of the object in the data set, taking the point cloud of the object as the input, and outputting a 3D affordance region map related to hand operation; step 2, the dexterous hand follows the 3D affordance region map, based on the synergistic effect of the fingers and the palm, and generates a grasping pose by optimizing the total energy function containing a self-conflict optimization term between the fingers and a support optimization term, and the optimal grasping pose of the robot dexterous hand is obtained after optimization. The method uses a conditional variational autoencoder model to predict a high-precision operation region, and introduces a self-conflict optimization term between the fingers and a support optimization term between the palm and the object to optimize the generation of the grasping pose, realizes multi-finger coordination and grasping stability, and has robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, and in particular to an optimization method for generating robust and physically feasible grasping poses for robot dexterous hands to grasp objects. BACKGROUND

[0002] With the rapid development of humanoid robots, dexterous hand grasping has become an increasingly important research direction. In recent years, research has mainly focused on obtaining diverse and stable grasping strategies, involving various technical paths such as data-driven methods, optimization algorithms, and their combination.

[0003] In modern dexterous hand grasping tasks, the typical workflow usually includes the following two key steps: first, the 3D affordance of the object is predicted using deep learning algorithms, and by learning the geometric properties of the objects in the data set, the potential interaction area is identified; second, an optimization algorithm or reinforcement learning method is used to guide the hand pose to match the predicted 3D affordance, and finally the grasping operation is realized. However, although existing technologies have made significant progress in affordance prediction, the prediction results can only provide a rough approximation of the ideal contact area. Especially in complex object grasping tasks, the ambiguity of the affordance map can seriously affect the stability of the grasping motion, resulting in insufficient grasping accuracy and increased grasping failure rate.

[0004] On the other hand, due to the lack of a clear mapping relationship between the contact map and the grasping execution, even if a fine contact map is generated, it cannot be ensured that the grasping pose that meets human expectations is achieved. This limitation makes it challenging to use contact map prediction as a reliable supervisory signal. Existing methods only consider the contact points of the fingers and the object, ignoring the friction, support, and stability provided by other parts of the palm (such as the palm and the side of the hand) during grasping, thereby reducing the stability and diversity of tool use. In addition, due to the high degree of freedom of the dexterous hand, the fingers are prone to self-conflict during grasping, resulting in unfeasible grasping poses or execution failures, especially in complex tool grasping and use tasks, this problem is particularly prominent.

[0005] Therefore, the present application is proposed. SUMMARY

[0006] The purpose of the present application is to provide an optimization method for generating robust and physically feasible grasping poses for robot dexterous hands, which can output 3D affordance regions related to hand operations, generate more robust and coordinated dexterous hand grasping poses, and thus solve the above technical problems existing in the prior art.

[0007] The purpose of the present application is achieved by the following technical solutions:

[0008] An optimization method for generating robust and physically feasible grasping poses for a robot dexterous hand, for optimizing the grasping poses generated for a robot dexterous hand, comprising:

[0009] Step 1, using a conditional variational autoencoder model to predict the 3D affordance of the object to be grasped, learning the geometric properties of the objects in the data set, taking the object point cloud as input, and outputting a 3D affordance region graph related to hand operation;

[0010] Step 2, the robot dexterous hand optimizes the hand pose based on the synergistic action of the fingers and the palm according to the 3D affordance region graph related to the hand operation, by optimizing the total energy function containing the self-conflict optimization term between the fingers and the support optimization term between the palm and the object, to obtain the optimal grasping pose of the robot dexterous hand.

[0011] Compared with the prior art, the optimization method for generating robust and physically feasible grasping poses for a robot dexterous hand has the following beneficial effects:

[0012] By introducing the self-conflict optimization term between the fingers and the support optimization term between the palm and the object into the total energy function for optimizing the hand pose of the robot dexterous hand, the self-conflict optimization term between the fingers is used to effectively avoid the mutual penetration of the fingers and improve the physical feasibility of the grasping pose, and the support optimization term between the palm and the object is used to enhance the contact area between the palm and the object to improve the stability, continuity and safety of the grasping process, which can solve the limitations of existing methods in complex tool grasping tasks, especially the deficiencies in finger self-conflict and palm contact characteristics. Introducing the above two objective functions into the traditional contact point optimization and hand-object direction optimization can achieve higher grasping stability and success rate, especially in complex tool grasping tasks, which shows excellent generalization ability. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0014] Figure 1 The flowchart of the optimization method for generating robust and physically feasible grasping poses for a robot dexterous hand provided by the embodiments of the present application.

[0015] Figure 2 The system schematic diagram of the optimization method for generating robust and physically feasible grasping poses for a robot dexterous hand provided by the embodiments of the present application. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be described clearly and completely in connection with the specific contents of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application, which do not constitute a limitation of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the present application.

[0017] Firstly, the terms possibly used in the present application are explained as follows:

[0018] The term "and / or" means either of the two or both, for example, X and / or Y means three cases including "X" or "Y" or "X and Y".

[0019] The terms "include", "contain", "have", "possess" or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, the inclusion of a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.) should be interpreted as not only including the explicitly listed technical feature element, but also including other technical feature elements not explicitly listed in the art.

[0020] The term "consisting of" means excluding any technical feature element not explicitly listed. If this term is used in the claims, the term will make the claim closed, so that it does not contain technical feature elements other than the explicitly listed technical feature elements, except for conventional impurities related thereto. If the term only appears in a certain clause of the claim, it is only limited to the elements explicitly listed in that clause, and the elements described in other clauses are not excluded from the overall claim.

[0021] Unless otherwise specifically provided or limited, the terms "mount", "connect", "connect", "fix", and the like should be broadly understood, for example: can be fixedly connected, can be detachably connected, or integrally connected; can be mechanically connected, or electrically connected; can be directly connected, or indirectly connected through an intermediate medium, can be internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0022] The terms "central", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", and the like indicate the orientation or positional relationship shown in the drawings, which are only for the convenience of description and simplification of description, and do not mean that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting herein.

[0023] The schemes provided by the present application will be described in detail below. The contents not described in detail in the embodiments of the present application belong to the prior art known to those skilled in the art. If the specific conditions are not specified in the embodiments of the present application, the conventional conditions or the conditions recommended by the manufacturer are used. If the reagents or instruments used in the embodiments of the present application are not specified by the manufacturer, they are all conventional products that can be purchased on the market.

[0024] As shown in Figure 1 , Figure 2 The embodiments of the present application provide an optimization method for generating a robust and physically feasible grasping pose for a robot dexterous hand, which is an optimization method for generating a grasping pose for a robot to grasp an object, comprising the following steps:

[0025] Step 1, using a conditional variational autoencoder model to predict the 3D affordance of the object to be grasped, learning the geometric properties of the objects in the data set, taking the object point cloud as input, and outputting a 3D affordance region graph related to hand operation;

[0026] Step 2, the robot dexterous hand optimizes the hand pose based on the synergistic action of the fingers and the palm according to the 3D affordance region graph related to the hand operation by optimizing the total energy function containing the self-conflict optimization term between the fingers and the support optimization term between the palm and the object, and obtains the optimal grasping pose of the robot dexterous hand after optimization. Figure 2 Min(E) in the above formula represents the optimization process of minimizing the total energy function E.

[0027] Preferably, in step 1 of the above optimization method, the conditional variational autoencoder model used is composed of an encoder and a decoder; wherein,

[0028] The encoder can extract point-by-point features from the input object point cloud, generate class embedding features for the corresponding finger contact classes, splice the extracted point-by-point features and class embedding features to obtain spliced features, and obtain latent space distribution after processing the spliced features;

[0029] The decoder is in communication connection with the encoder, can receive the latent space distribution output by the encoder, sample the latent vector from the latent space distribution, and decode the concatenated feature obtained by splicing the copied latent vector and the point-by-point feature to generate a 3D affordance region graph according to the number of points in the object point cloud copy the latent vector second, after splicing the copied latent vector and the point-by-point feature to obtain the spliced feature, generate a 3D affordance region graph through decoding.

[0030] Preferably, in the above optimization method, the encoder is composed of a PointNet encoder, an embedding layer, a first splicing layer, a multi-layer perception and a maximum pooling layer; wherein,

[0031] The PointNet encoder can extract point-by-point features from the input object point cloud;

[0032] The embedding layer can generate category embedding features for the corresponding finger contact categories;

[0033] The first splicing layer is in communication connection with the PointNet encoder and the embedding layer respectively, and can splice the extracted point-by-point features and the category embedding features to obtain spliced features;

[0034] The multi-layer perception is connected with the first splicing layer and the maximum pooling layer respectively, and can process the spliced features output by the first splicing layer to obtain a latent space distribution.

[0035] Preferably, in the above optimization method, the decoder is composed of a sampling layer, a copying layer, a second splicing layer and a multi-layer perception decoder; wherein,

[0036] The sampling layer is in communication connection with the encoder, can receive the latent space distribution output by the encoder, and sample the latent vector from the latent space distribution;

[0037] The copying layer is in communication connection with the sampling layer, and can copy the latent vector according to the number of points in the object point cloud copy the latent vector second;

[0038] The second splicing layer is in communication connection with the copying layer and the PointNet encoder in the encoder, and can splice the copied latent vector and the point-by-point feature obtained from the PointNet encoder to obtain spliced features;

[0039] The multi-layer perception decoder is in communication connection with the second splicing layer, and can decode the spliced features output by the second splicing layer to generate a 3D affordance region graph.

[0040] Preferably, in the above optimization method, the complete loss function for training the conditional variational autoencoder model is:

[0041] ;

[0042] wherein, 、 and represent the weights of the corresponding loss 、 、 , whose values are 0.5, 0.9 and 0.01 respectively; is a multi-class cross-entropy loss function; is a multi-class Dice loss function; is a regularization loss function, which is a KL loss of maximizing the KL divergence between the latent space distribution and the standard Gaussian distribution, and are the mean vector and variance vector of the latent space output by the encoder respectively.

[0043] The conditional variational autoencoder model described above takes object point cloud as input, extracts features through the encoder (PointNet+MLP) and generates latent distribution, and outputs high-precision operation region prediction through the decoder combined with reconstruction loss (cross-entropy+Dice loss) and KL regularization loss.

[0044] Preferably, in step 2 of the above optimization method, the robot dexterous hand optimizes the grasping pose based on the synergistic action of the fingers and the palm according to the 3D affordance region map related to the hand operation by optimizing the total energy function containing the inter-finger self-conflict optimization term and the palm-object support optimization term in the following manner, and obtains the optimal grasping pose of the robot dexterous hand after optimization, including:

[0045] After the robot dexterous hand completes the understanding of the 3D affordance region map of the tool, the grasping pose is optimized by optimizing the following total energy function containing the inter-finger self-conflict optimization term and the palm-object support optimization term, and the total energy function is:

[0046] ;

[0047] wherein, represents the number of iterations of the current optimization process; represents the weight based on the hand-object distance energy term; represents the hand-object distance energy term based on the hand-object distance energy term; represents the total number of iterations; represents the weight of the total direction consistency energy term; represents the total direction consistency energy term; represents the weight of the network prediction loss; denotes a binary cross-entropy loss; denotes a weight of a hand-object penetration penalty energy term; denotes a hand-object penetration penalty energy term; denotes a weight of an inter-finger self-conflict optimization term; denotes an inter-finger self-conflict optimization term; denotes a weight of a palm-object inter-support optimization term that minimizes the nearest neighbor distance between the palm and the object; denotes a palm-object inter-support optimization term that minimizes the nearest neighbor distance between the palm and the object.

[0048] The present application utilizes a Conditional Variational Autoencoder (CVAE for short) to output a 3D affordance region related to hand operation; then, on the basis of traditional contact point optimization and hand-object direction optimization, an inter-finger self-conflict optimization term and a palm-object inter-support optimization term are further introduced , aiming to generate more robust and coordinated dexterous grasping poses.

[0049] Preferably, in the above optimization method, the hand-object distance energy term is:

[0050] ;

[0051] wherein, denotes the total number of all categories; i denotes a category corresponding to the artificially labeled contact point set ; is the artificially labeled contact point set; is the category corresponding to the contact point number; is the target point set corresponding to the 3D affordance region map generated by the Conditional Variational Autoencoder; denotes that the obtained minimum distance value is set to zero with a probability ; j denotes the index of the midpoint in the object target point set ; denotes the i-th contact point in the artificially labeled contact point set; denotes the j-th target point in the object target point set .

[0052] The total direction consistency energy term is:

[0053] ;​

[0054] wherein, , are the weighting coefficients of the directional consistent energy term acting on the fingertip and the weighting coefficients of the directional consistent energy term acting on the finger surface region, respectively; denotes the directional consistent energy term acting on the fingertip; denotes the directional consistent energy term acting on the finger surface region.

[0055] Preferably, in the above optimization method, the directional consistent energy term acting on the fingertip is:

[0056] ;

[0057] wherein, denotes the unit normal vector at the fingertip contact point ; denotes the point in the object target point set closest to the contact point ; denotes the unitized vector of , denotes the vector of the fingertip contact point pointing to the point on the object surface closest to it .

[0058] The directional consistent energy term acting on the finger surface region is:

[0059] ;

[0060] wherein, denotes the unit normal vector at the finger surface contact point , denotes the set of artificially labeled finger surface points corresponding to the category ; denotes the th finger surface contact point in the set of finger surface points ; denotes the point in the object target point set closest to the finger surface contact point ; denotes the unitized vector of , denotes the vector of the finger surface contact point pointing to the point on the object surface closest to it ;

[0061] and denote the set of finger surface points corresponding to the category and The nearest target point index is calculated as follows:

[0062] ;

[0063] .

[0064] Preferably, in the above optimization method, the binary cross-entropy loss for:

[0065] ;

[0066] in, This represents the probability of successful grasping predicted by a binary classification network based on the PointNet architecture, where the binary classification network uses the hand point cloud H and the object point cloud H as inputs. As input.

[0067] Preferably, in the above optimization method, the hand-object penetration penalty energy term for:

[0068] ;

[0069] in, This represents the number of points in the hand point cloud, with a value of 778. The number of points in the point cloud of an object; This indicates a modified linear unit activation function; This indicates the preset safe distance threshold; Let m represent the m-th palm point in the palm point cloud H, where m∈{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16}; This represents the nth object point in the object point cloud; Indicates palm point With object point The minimum distance between them.

[0070] The finger-inter-conflict optimization term Determined in the following ways, including:

[0071] Calculate the geometric center of the fingertip of each finger in the MANO hand model. The calculation formula is:

[0072] ;

[0073] in, Let f represent the set of fingertips of the f-th finger, where f∈{1,2,3,4,5}. Represents the set of fingertips of the f-th finger. The fingertip point set of the five fingers in the MANO hand model is , is the gth fingertip point of the fth finger.

[0074] The Euclidean distance between adjacent fingers is calculated respectively and the long distance between the thumb and the little finger , and the calculation formulas are respectively:

[0075] , ;

[0076] ;

[0077] According to the obtained Euclidean distance between adjacent fingers and the long distance between the thumb and the little finger , the defined finger-to-finger self-conflict optimization term is:

[0078] ;

[0079] wherein α is a minimum safety distance threshold set to avoid the risk of self-conflict caused by too close distance between fingers, and is set to ; and are weight parameters between adjacent fingers and distant fingers, respectively, and are set to and ; represents a rectified linear unit activation function, and this energy term will only be penalized when the distance between fingers is less than the set threshold, preventing the fingers from penetrating each other.

[0080] The palm-to-object support optimization term for minimizing the nearest neighbor distance between the palm and the object is:

[0081] ;

[0082] wherein represents a function for calculating the nearest neighbor distance between point sets, and the nearest neighbor is 1-neighbor, i.e. k=1 in the k-neighbor algorithm; represents the object point cloud of the object; P m represents the mth palm point in the palm point cloud in the hand point cloud H, and P represents the palm point cloud in the hand point cloud H. The points numbered 0-15 in the hand point cloud of the MANO hand model are selected as the representative points of the palm for optimization, and P is defined as: wherein is the number of palm point clouds, Total number of vertices of the MANO hand model.

[0083] The above-mentioned inter-finger self-conflict optimization term , for avoiding spatial conflicts between fingers, ensuring the physical executability of the grasp pose.

[0084] The above-mentioned palm-object support optimization term , for enhancing the contact area between the palm and the object, providing additional friction and support force, thereby improving the stability and robustness of the grasp.

[0085] The present application adopts a conditional variational autoencoder model, takes an object point cloud as input, extracts features through an encoder (PointNet+MLP) and generates a latent distribution, and an decoder combines a reconstruction loss (cross entropy+Dice loss) and a KL regularization loss to output a high-precision operation region prediction. By introducing the inter-finger self-conflict optimization term and the palm-object support optimization term into the total energy function for optimizing the generated grasp pose, the problems of pose conflict and insufficient stability in the existing method in complex tool grasping are solved. Among them, the inter-finger self-conflict optimization term is used to avoid spatial conflicts between fingers, ensuring the physical executability of the grasp pose; the palm-object support optimization term is used to enhance the contact area between the palm and the object, providing additional friction and support force, thereby improving the stability and robustness of the grasp. This method is significantly better than the existing method in terms of multi-finger coordination and grasp stability, and has a certain robustness.

[0086] In order to more clearly show the technical solutions provided by the present application and the technical effects produced, the schemes provided by the embodiments of the present application will be described in detail below with specific embodiments.

[0087] Embodiment 1

[0088] As shown in Figure 1 , Figure 2 , the present embodiment provides an optimization method for generating robust and physically feasible grasp poses for a robot dexterous hand, and the preferred specific implementation process manner includes the following steps:

[0089] Step 1, use a conditional variational autoencoder model (Conditional Variational Autoencoder, hereinafter referred to as CVAE) to predict the 3D affordance of the object, learn the geometric properties of the object in the data set, take the object point cloud as input, and output the 3D affordance region related to the hand operation;

[0090] Step 2, the dexterous hand optimizes the hand posture by optimizing the total energy function containing the self-conflict optimization term between fingers and the support optimization term between the palm and the object, based on the coordination between fingers and the palm, and finally obtains the optimal grasping posture of the dexterous hand. Figure 2 Min(E) in the formula represents the optimization process of minimizing the total energy function E.

[0091] The above step 1 is specifically:

[0092] The CVAE model is trained based on supervised learning, which is composed of an encoding stage of an encoder and a decoding stage of a decoder, takes an object point cloud as input, and outputs a 3D affordance region related to hand operation.

[0093] In the encoding stage, the encoder first extracts point-wise features from the input object point cloud using a PointNet encoder; at the same time, a category embedding feature is generated for the corresponding contact map using an embedding layer. Then, the two types of features are spliced, and the distribution of the latent space is obtained through a multi-layer perceptron (MLP) and a max-pooling layer .

[0094] In the decoding stage, the decoder samples a latent vector from the distribution of the latent space times, where represents the number of points in the point cloud. After splicing the copied latent vector with the point-wise feature, a 3D affordance map is generated through an MLP decoder. In the training of the CVAE model, two types of supervised loss functions are used: reconstruction loss, which is used to punish errors in the contact map reconstruction process; regularization loss, which is used to encourage the latent space to have good properties. The reconstruction loss is composed of multi-class cross-entropy loss and multi-class Dice loss , which enables the network to generate more accurate contact maps from input point clouds. The KL loss is used as the regularization loss. The definition is to maximize the KL divergence between the latent space distribution and the standard Gaussian distribution , which encourages the latent space distribution learned by the encoder to approach the prior standard Gaussian distribution. The complete loss function for training the CVAE can be represented as:

[0095] ;

[0096] where, , and respectively, whose values are set to 0.5, 0.9 and 0.01 respectively.

[0097] The step 2 is specifically:

[0098] After completing the 3D affordance region understanding of the tool, the dexterous hand needs to further generate a grasping pose and perform a grasping operation. By optimizing the total energy function The grasping pose is optimized, and the function is expressed as follows:

[0099] ;

[0100] Wherein, Indicates the number of iterations of the current optimization process, Indicates the total number of iterations. The two parameters are used to realize the adaptive weight adjustment strategy, to ensure that attention is paid to the preliminary fitting of the grasping pose in the early stage of optimization, and more attention is paid to the coordination between fingers and the physical rationality of hand-object contact in the later stage.

[0101] Since the dexterous grasping task has significant non-convexity, it is difficult for traditional optimization algorithms to obtain a global optimal solution, and the present application uses a binary classification network based on the PointNet architecture, uses a binary cross-entropy loss function, inputs the hand point cloud H and the object point cloud To predict whether the grasping is successful. In order to avoid matching errors in the optimization process, the binary cross-entropy loss is introduced as an energy term, defined as:

[0102] ;

[0103] Wherein, Indicates the grasping success probability predicted by the network.

[0104] In addition, in order to further improve the physical rationality of the grasping pose, the method of the present application also introduces a hand-object penetration penalty energy term , which is defined as:

[0105] ;

[0106] Wherein, Indicates the number of points of the hand point cloud, which is 778; Indicates the number of points of the object point cloud; Indicates a rectified linear unit activation function; Indicates a preset safety distance threshold; Indicates the mth palm point of the palm point cloud in the hand point cloud H, m belongs to {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16}; Indicates the nth object point in the object point cloud; represents the minimum distance between the palm point and the object point .

[0107] The total energy function is defined as follows The rest of the terms are explained.

[0108] The hand-object distance energy term aims to encourage the dexterous hand pose to align precisely with the target contact area by minimizing the Euclidean distance between the hand fingertips and the 3D affordance region. Its specific form is defined as follows:

[0109] ;

[0110] where and represent the point cloud subsets belonging to the categories , is the set of contact points labeled artificially, is the set of object target points corresponding to the 3D affordance region map generated by the CVAE model, is the number of contact points corresponding to the category , denotes the total number of all categories. To avoid falling into a local optimal solution during optimization, the operation is introduced, which sets part of the minimum distance values to zero with a probability during optimization, thereby introducing perturbations and enhancing the model's exploration ability in complex contact spaces, improving the robustness of pose optimization.

[0111] The total direction consistency energy term is composed of acting on the fingertips and acting on the surface area of the fingers, which calculates the squared Euclidean distance between the unit normal vectors of the hand point set and the unit direction vectors of the target point set (generated by 3D affordance).

[0112] Let denote the unit normal vector at point , denote the unitized vector of . Based on this, , and the total direction consistency energy term are defined as follows:

[0113] ;

[0114] ;

[0115] ;

[0116] in, and Respectively represent and and The nearest target point index is calculated as follows:

[0117] , ;

[0118] in, and These are the weighting coefficients for the consistency of fingertip and finger direction, respectively. The operation is consistent with the aforementioned definition, through probability. Setting some terms to zero enhances the randomness of the optimization and avoids getting trapped in local optima.

[0119] In particular, this invention introduces a self-conflict optimization term between fingers in the grasping posture optimization. The minimum distance between fingers is constrained to avoid spatial conflicts between them, ensuring that the generated posture conforms to the natural shape and physical constraints of the human hand, and guaranteeing the physical feasibility of the grasping posture. If the distance between fingers is too small, it may lead to physical collisions and damage to the mechanical structure, while if the distance is too large, it may generate an unreasonable grasping posture, reducing the stability and feasibility of grasping.

[0120] Let the fingertip set of the five fingers in the hand model be... ,in Let f represent the set of coordinates of the fingertip of the f-th finger in three-dimensional space. For each finger, calculate the geometric center of its fingertip. The definition is as follows:

[0121] ;

[0122] in, Represents the set of fingertips of the f-th finger. The number of points in the middle, It is the g-th fingertip of the f-th finger.

[0123] To fully capture the spatial constraints between fingers, the Euclidean distance between adjacent fingers was considered. The long distance between the thumb and little finger Its definition is as follows:

[0124] , , ;

[0125] Based on this, the finger self-conflict optimization term is defined. for:

[0126] ;

[0127] where is the set minimum safety distance threshold to avoid self-collision risk caused by too close distance between fingers, which is set to , and are the weight parameters between adjacent fingers and distant fingers, which are set to and , is the rectified linear unit activation function, which ensures that the energy term will only produce a penalty when the distance between fingers is less than the set threshold, preventing the fingers from penetrating each other. This energy term imposes constraints on the distance between fingers during optimization, ensuring that the fingers maintain a reasonable spatial layout. By introducing ReLU function, when the distance between fingers is less than, the energy term contributes to the loss function, otherwise it has no effect, effectively avoiding the penetration of fingers and improving the physical feasibility of the grasp pose.

[0128] In addition, for the grasping task of smooth and fragile objects, the invention introduces a palm support and friction enhancement optimization palm-object support optimization term to enhance the contact area between the palm and the object, providing additional support and friction for the palm contact, which helps to improve the stability, persistence and safety of the grasping process, enhancing the friction and support force during grasping, ensuring robust grasping of complex objects. Let P represent the point cloud of the palm part, and H represent the complete hand point cloud. The invention extracts the palm region from the hand point cloud, and in the method of the invention, the points numbered 0-15 in the MANO hand point cloud are selected as the representatives of the palm for optimization. The number of palm representative points , the total number of vertices of the MANO hand model , is defined as follows:

[0129] ;

[0130] To enhance the stability of grasping and maximize the contact area between the palm and the object, the invention minimizes the palm-object support optimization term defined as:

[0131] ;

[0132] where represents the function to calculate the nearest neighbor distance between point sets, represents the object point cloud. By introducing this optimization term, the contact area between the palm and the object can be effectively improved, enhancing the friction and support during the grasping process, combined with the tool operation possibility region, ultimately generating a more robust grasp pose.

[0133] In summary, the method of this invention introduces a self-conflicting term between fingers into the total energy function for optimizing the grasping posture. Optimization of support between the palm and the object This invention generates a more robust and coordinated dexterous hand grasping posture, significantly improving the rationality and feasibility of the grasping posture. The optimized grasping posture avoids conflicts in finger spacing and has a wider and more reasonable contact area between the palm and the object, thereby improving the stability and success rate of grasping. The method of this invention is effective and reasonable in complex grasping tasks. The above improvements not only enhance the rationality and stability of the dexterous hand grasping posture, but also provide a more adaptable solution for complex objects and diverse operational tasks. The optimized method of this invention effectively solves the problem of infeasible or failed grasping postures in complex tool grasping and use tasks.

[0134] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0135] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. An optimized method for generating robust and physically feasible grasping postures for a robot dexterous hand, characterized in that, Optimizing the grasping posture generated for a robot's dexterous hand includes: Step 1: Use the conditional variational autoencoder model to predict the 3DAffordance of the object to be grasped. By learning the geometric properties of the objects in the dataset, the point cloud of the object is used as input, and the output is a 3DAffordance region map related to hand operation. Step 2: Based on the 3DAffordance region map related to hand operation, the robot dexterous hand optimizes the hand posture by optimizing the total energy function, which includes self-conflict optimization terms between fingers and support optimization terms between the palm and the object, based on the cooperative action between the fingers and the palm. After optimization, the optimal grasping posture of the robot dexterous hand is obtained. In step 2, the robot dexterous hand generates a grasping posture based on the 3DAffordance region map related to hand operations, using the cooperative interaction between the fingers and palm. This is achieved by optimizing the total energy function, which includes self-conflict optimization terms between the fingers and support optimization terms between the palm and the object. The optimized posture is then obtained, including: After understanding the 3D affordance region map of the tool, the robotic dexterous hand optimizes the following total energy function, which includes optimization terms for inter-finger self-conflict and optimization terms for support between the hand and the object, to generate the grasping posture. for: ； in, This indicates the number of iterations in the current optimization process; This represents the weight based on the hand-object distance energy term; This represents the energy term based on the distance between the hand and the object; Indicates the total number of iterations; The weight of the overall directional consistency energy term; This represents the overall directional consistency energy term; The weights represent the network's prediction loss; This represents the binary cross-entropy loss; This indicates the weight of the energy term representing the penalty for object penetration. This indicates the energy level of the penalty for the object penalizing the target. This indicates the weight of the self-conflict optimization term between fingers; This indicates a self-conflict optimization term between fingers; The weights of the support optimization term between the hand and the object, representing the minimum nearest neighbor distance between the hand and the object; The term represents the support optimization term between the hand and the object that minimizes the nearest neighbor distance between the hand and the object.

2. The optimized method for generating robust and physically feasible grasping postures for a robot dexterous hand according to claim 1, characterized in that, In step 1, the conditional variational autoencoder model used consists of an encoder and a decoder; wherein, The encoder can extract point-by-point features from the input object point cloud and generate category embedding features for the corresponding finger contact category. The extracted point-by-point features and category embedding features are concatenated to obtain concatenated features. The latent spatial distribution is obtained after processing the concatenated features. The decoder, communicatively connected to the encoder, can receive the latent spatial distribution output by the encoder, sample latent vectors from the latent spatial distribution, and sort them according to the number of points in the object point cloud. Copy the latent vector Next, the copied latent vector is concatenated with the point-by-point features to obtain the concatenated features, and then the 3DAffordance region map is generated by decoding.

3. The optimized method for generating robust and physically feasible grasping postures for a robot dexterous hand according to claim 2, characterized in that, The encoder consists of a PointNet encoder, an embedding layer, a first stitching layer, a multilayer perceptron, and a max-pooling layer; wherein... The PointNet encoder can extract point-by-point features from the input object point cloud; The embedding layer can generate category embedding features for the corresponding finger contact category; The first stitching layer is communicatively connected to the PointNet encoder and the embedding layer, and can stitch together the extracted point-by-point features and category embedding features to obtain stitched features; The multilayer perceptron is connected to the first stitching layer and the max pooling layer respectively, and can process the stitching features output by the first stitching layer to obtain the potential spatial distribution.

4. The optimized method for generating robust and physically feasible grasping postures for a robot dexterous hand according to claim 2, characterized in that, The decoder consists of a sampling layer, a replication layer, a second splicing layer, and a multilayer perceptron decoder; wherein... The sampling layer is communicatively connected to the encoder and can receive the latent spatial distribution output by the encoder and sample latent vectors from the latent spatial distribution; The replication layer is communicatively connected to the sampling layer, and is configured according to the number of points in the object point cloud. Copy the latent vector Second-rate; The second stitching layer is communicatively connected to the copying layer and the PointNet encoder in the encoder, and can stitch the copied latent vector with the point-by-point features obtained from the PointNet encoder to obtain stitched features; The multilayer perceptron decoder is communicatively connected to the second stitching layer and can decode the stitching features output by the second stitching layer to generate a 3DAffordance region map.

5. The optimized method for generating robust and physically feasible grasping postures for a robot dexterous hand according to any one of claims 2-4, characterized in that, In the method described, the complete loss function for training the conditional variational autoencoder model is... for: ； in, , and Representing the corresponding losses , , The weights are 0.5, 0.9, and 0.01, respectively; For multi-class cross-entropy loss functions; For multiple types of Dice loss functions; The regularization loss function maximizes the distribution of the latent space. KL loss of the KL divergence between the standard Gaussian distribution and the standard Gaussian distribution. and These are the mean vector and variance vector of the latent space output by the encoder, respectively.

6. The optimized method for generating robust and physically feasible grasping postures for a robot dexterous hand according to claim 1, characterized in that, The energy term based on hand-object distance for: ； in, The total number of all categories is represented by 'i'; 'i' represents the category. Corresponding manually calibrated set of contact points Midpoint index; A set of manually calibrated fingertip contact points; For category The corresponding number of fingertip contact points; This is the set of object target points corresponding to the 3DAffordance region map generated by the conditional variational autoencoder model; Represented by probability Setting the obtained minimum distance value to zero can introduce perturbation during the optimization process and avoid getting trapped in local optima; j represents the set of object target points. Midpoint index; This represents the i-th fingertip contact point in the manually calibrated set of contact points; Represents the object target point set The j-th target point in the; The overall directional consistency energy term for: ； in, , These are the weighting coefficients for the energy term of fingertip orientation consistency and the weighting coefficients for the energy term of finger surface orientation consistency, respectively. This represents the energy term representing the directional consistency of the energy acting on the fingertips; This represents the directional consistency energy term acting on the surface area of ​​the finger.

7. The optimized method for generating robust and physically feasible grasping postures for a robot dexterous hand according to claim 6, characterized in that, The directional consistency energy term acting on the fingertips for: ； in, This represents the i-th fingertip contact point in the manually calibrated set of contact points. The unit normal vector at that location; Represents the object target point set The i-th fingertip contact point in the set of manually calibrated contact points The nearest point; express The normalized vector, This represents the i-th fingertip contact point in the manually calibrated set of contact points. Point to the closest point on the object's surface ; The directional consistency energy term acting on the finger surface area for: ； in, Indicates the contact point on the finger surface The unit normal vector at that location, Representation and Category The corresponding manually calibrated finger surface point set; Represents the point set on the finger surface The first in Each finger surface contact point; Represents the object target point set Contact point with the finger surface The nearest point; express The normalized vector, Indicates the contact point on the finger surface The point points to the nearest point on the object's surface. ; and Respectively represent and and The nearest target point index is calculated as follows: ； 。 8. The optimized method for generating robust and physically feasible grasping postures for a robot dexterous hand according to claim 1, characterized in that, The binary cross-entropy loss for: ； in, This represents the probability of successful grasping predicted by a binary classification network based on the PointNet architecture, where the binary classification network uses the hand point cloud H and the object point cloud H as inputs. As input.

9. The optimized method for generating robust and physically feasible grasping postures for a robot dexterous hand according to claim 1, characterized in that, The penetrating penalty energy item of the hand for: ； in, This represents the number of points in the hand point cloud, with a value of 778. The number of points in the point cloud of an object; This represents the modified linear unit activation function; This indicates the preset safe distance threshold; Let m represent the m-th palm point in the palm point cloud H, where m∈{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16}; This represents the nth object point in the object point cloud; Indicates palm point With object point The minimum distance between them; The finger-inter-conflict optimization term Determined in the following ways, including: Calculate the geometric center of the fingertip of each finger in the MANO hand model. The calculation formula is: ； in, Let f represent the set of fingertips of the f-th finger, where f∈{1,2,3,4,5}. Represents the set of fingertips of the f-th finger. The number of points in the MANO hand model, the fingertip point set of the five fingers is as follows: , It is the g-th fingertip of the f-th finger; Calculate the Euclidean distance between adjacent fingers respectively. The long distance between the thumb and little finger The calculation formulas are as follows: ， ； ； Based on the obtained Euclidean distance between adjacent fingers The long distance between the thumb and little finger Defined finger-to-finger self-conflict optimization term The is: ； in, To set a minimum safe distance threshold to avoid the risk of self-conflict caused by fingers being too close together, a setting is made. ; and These are the weight parameters between adjacent fingers and distant fingers, respectively, and are set as follows: and ; This indicates that the modified linear unit activation function will only incur a penalty when the distance between fingers is less than a set threshold, thus preventing fingers from penetrating each other. The palm-object support optimization term that minimizes the nearest neighbor distance between the palm and the object. for: ； in, This represents a function that calculates the nearest neighbor distance between point sets. The nearest neighbor refers to the 1-nearest neighbor, which is the case where k=1 in the k-nearest neighbor algorithm. Object point cloud representing an object; P m This represents the first point of the palm point cloud in the hand point cloud H. m One palm's width The point cloud H represents the palm point cloud. Points numbered 0 to 15 from the hand point cloud of the MANO hand model are selected as representative points of the palm for optimization. The definition of is: ,in The number of dots on the palm. This represents the total number of vertices in the MANO hand model.

Images