A grasp stability evaluation method based on hand-object interaction contact representation

Abstract

The present application relates to the technical fields of robot grasping, robot vision, point cloud geometry learning and end effector control, in particular to a grasping stability evaluation method based on hand-object interaction contact representation. The method comprises obtaining input data required for grasping stability evaluation; constructing hand surface and object surface according to the input data; constructing an effective contact point pair set according to the hand surface and the object surface; constructing hand-object interaction stability representation according to the effective contact point pair set; outputting a stability score of a candidate grasping pose using a grasping decision model based on the hand-object interaction stability representation; and performing grasping screening according to the stability score. The present application takes the geometric relationship of the hand-object contact interface as the basis for grasping stability discrimination, reduces the dependence on specific manipulator structure parameters, can adapt to multi-fingered dexterous hands, parallel grippers and heterogeneous end effectors, and improves the universality, robustness and reliability of candidate grasping screening.

Description

Technical Field

[0001] This invention belongs to the fields of robot grasping, robot vision, point cloud geometry learning and end effector control technology. Specifically, it relates to a grasping stability evaluation method, device, electronic device and storage medium based on hand-object interaction contact representation, which can be used for candidate grasping posture screening of multi-finger dexterous hands, three-finger grippers, parallel grippers, adsorption end effectors and heterogeneous robot hands. Background Technology

[0002] Robotic grasping systems typically begin by generating multiple candidate grasping postures using a grasping generation algorithm, followed by collision detection, reachability analysis, and stability assessment by subsequent modules. For regular objects and parallel grippers, the grasping width, grasping direction, and gripper center position can often provide some basis for judgment; however, in multi-fingered dexterous hands, complex objects, and irregular contact scenarios, it is difficult to determine whether candidate grasping postures can resist gravity, inertial forces, and external disturbances based on only a few geometric parameters.

[0003] Traditional grasping stability analysis often relies on mechanical models such as force closure, shape closure, or grasping matrices. These methods require accurate contact position, contact normal, friction coefficient, contact force distribution, and object mass inertia parameters. In real-world robot vision scenarios, these parameters are often difficult to obtain accurately, especially when the object's material is unknown, the point cloud is partially missing, or there is flexible contact between the finger and the object, limiting the applicability of analytical mechanical judgments.

[0004] In recent years, learning-based grasping evaluation methods have used neural networks to predict the probability of successful grasping from images, point clouds, or grasping parameters, which can compensate for modeling errors to some extent using data. However, some methods directly use the joint angles, gripper type, or fixed-dimensional grasping parameters of the robotic arm as input, making the model prone to being tied to a specific end effector. When the structure of the robotic arm, the number of fingers, or the surface geometry changes, the original model often needs to be retrained or have a large amount of calibration data.

[0005] For multi-fingered hands, grasping stability is essentially closely related to the contact relationship between the finger surfaces and the object surfaces. If the hand-object contact interface generated by different manipulators in candidate postures can be converted into a unified point cloud geometric representation, it is hoped that the model's dependence on specific hand shape parameters can be weakened, allowing the model to focus more on contact distribution, contact normals, contact coverage, and local geometric complementarity.

[0006] Therefore, a grasping stability assessment method is needed, which draws on information obtained from both vision and hand geometry. This method should be able to generate potential contact point pairs between the hand surface and the object surface based on candidate grasping postures, construct a unified hand-object interaction contact representation, and output a stability score by adapting to models with different contact numbers and distributions, thereby improving the universality and reliability of candidate grasping selection. Summary of the Invention

[0007] To address the problems of existing grasping stability assessment methods relying on precise mechanical parameters, difficulty in adapting to heterogeneous end effectors, and insufficient robustness to different contact scales, this invention provides a grasping stability assessment method based on hand-object interaction contact characterization. This method can be adapted to multi-finger dexterous hands, parallel grippers, and heterogeneous end effectors, improving the universality, robustness, and reliability of candidate grasping selection.

[0008] The embodiments of the present invention adopt the following technical solutions: In a first aspect, the present invention provides a method for evaluating grasping stability based on hand-object interaction contact characterization, specifically as follows: Obtain the input data required for crawl stability assessment, including the following sub-steps: The system acquires the object geometry information, robot hand specifications, and candidate grasping postures of the object to be evaluated. The object geometry information includes object point clouds, object mesh models, local surfaces reconstructed from depth maps, or combinations thereof. The robot hand specifications include at least one of hand kinematic parameters, link geometry models, finger surface meshes, joint constraints, and surface sampling rules. The candidate grasping postures include the hand base pose and hand joint configuration, or grasping parameters capable of recovering the hand base pose and hand joint configuration. Constructing the hand surface and object surface based on the input data includes the following sub-steps: Based on the robot hand specifications and candidate grasping postures, the position and posture of each link of the hand in a unified coordinate system are calculated using forward kinematics. By sampling the surface mesh of each link of the hand, a set of points on the hand surface is obtained; The hand surface normal vector is obtained by interpolating the surface normal vector of the hand surface mesh or by fitting the local neighborhood, and the hand surface is constructed accordingly. Sample or read a set of points on the object's surface from the object's geometric information; The surface normal vector of the object is obtained from the object mesh normal, principal component analysis of the point cloud neighborhood, or local plane fitting of the depth map, and the object surface is constructed accordingly. Constructing a set of effective contact point pairs based on the hand surface and the object surface includes the following sub-steps: Calculate the nearest neighbor distance from each point on the object's surface to the set of points on the hand's surface; Points on the surface of an object whose nearest neighbor is less than a preset distance threshold are identified as object contact points; The hand surface point closest to the contact point with the object is determined as the corresponding hand contact point; Effective contact point pairs are constructed based on the object contact point, the corresponding hand contact point, the object surface normal vector, and the hand surface normal vector; when the number of effective contact point pairs is greater than the preset number, the effective contact point pairs are sampled spatially uniformly, with confidence-priority sampling, or randomly. When the number of valid contact point pairs is less than or equal to the preset number, the valid contact point pairs are repeatedly sampled, filled with a mask, or marked with a low confidence flag. Based on the effective contact points, construct a characterization of the hand-object interaction stability of the set, including the following sub-steps: The coordinates of the object's contact point are used as the spatial coordinates of the point cloud. The object surface normal vector, the coordinates of the hand contact point, and the hand surface normal vector are used as point-by-point geometric features; Transform the coordinates of the object's contact point to the object's centered coordinate system or the local coordinate system of the candidate grasping posture; The coordinates of the object contact point, the coordinates of the hand contact point, the object surface normal vector, and the hand surface normal vector are scaled to form a unified input representation that adapts to objects of different sizes and different end effectors; Based on the stability characterization of hand-object interaction, the stability score of candidate grasping postures is output using a grasping decision model, including the following sub-steps: The hand-object interaction stability characterization is input into the local geometry encoder to obtain the local contact embedding for each effective contact point pair; The contact-aware multi-pooling module performs max pooling, average pooling, and standard deviation pooling on local contact embeddings, respectively. Dynamic fusion weights are generated based on the number of effective contact points, the distribution of contact areas, or the contact confidence level. The global contact descriptor is obtained by fusing the maximum pooling result, average pooling result, and standard deviation pooling result with the dynamic fusion weights. The global contact descriptor is input into the score prediction head and the Sigmoid function to obtain a stability score with a value range of 0 to 1; Based on the stability score, the data is filtered and processed, including the following sub-steps: Retain candidate grasping postures with stability scores greater than the stability score threshold; or select the top K candidate grasping postures in descending order of stability score. The selected candidate grasping postures are output to the motion planning module, collision detection module, or robot execution module.

[0009] Preferably, the nearest neighbor distance is the Euclidean distance from the object surface point to the set of hand surface points, the weighted distance with normal consistency constraints, or the directed distance considering the hand's closing direction.

[0010] Preferably, each element in the effective contact point pair set further includes at least one of the following: the distance between the object contact point and the hand contact point, the normal angle, and the contact confidence level.

[0011] Preferably, the local geometric encoder is a multilayer perceptron, a PointNet-style shared network, a graph neural network, or a local neighborhood attention network.

[0012] Preferably, when the number of effective contact point pairs is less than a preset threshold, the candidate grasping posture is determined as a low-confidence candidate, or the number of effective contact point pairs is used as an additional input to the grasping decision model, so that the grasping decision model can adaptively evaluate sparse contact situations.

[0013] Preferably, the grasping decision model is trained using labeled grasping test data, whereby the labels characterize whether the candidate grasping posture maintains successful grasping after closing, lifting, moving, or being subjected to external disturbances.

[0014] Preferably, the grasping decision model is trained using binary cross-entropy loss, ranking loss, class imbalance weighted loss, or a combination thereof.

[0015] Preferably, when training the grasping decision model, random perturbations are added to the coordinates of the object contact point, the coordinates of the hand contact point, the normal vector of the object surface, and the normal vector of the hand surface, and effective contact point pairs are randomly discarded, randomly resampled, or scaled to enhance the robustness of the model to perceived noise, occlusion, and different hand shape sampling densities.

[0016] Preferably, when the end effector is a parallel gripper, the candidate gripping posture further includes the gripper opening width or closing amount; when the end effector is a multi-fingered dexterous hand, the candidate gripping posture further includes the joint angles of each finger.

[0017] In a second aspect, the present invention provides a grasping stability assessment device based on hand-object interaction contact representation. The device includes: a data acquisition module, a hand surface generation module, a contact point pair construction module, a contact representation generation module, a grasping decision model, and a posture screening module. Each module is configured to execute the grasping stability assessment method based on hand-object interaction contact representation in the first aspect.

[0018] Thirdly, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, characterized in that the computer program, when executed by the processor, implements the grasping stability evaluation method based on hand-object interaction contact characterization in the first aspect.

[0019] Compared with existing technologies, the advantages of this invention are as follows: This invention uniformly transforms candidate grasping postures, hand specifications, and object geometry into a set of effective contact point pairs. Each effective contact point pair includes not only the object contact point position and the hand contact point position, but also the object surface normal vector and the hand surface normal vector, thereby simultaneously describing the spatial positional relationship and local surface orientation relationship of the contact. Compared with directly inputting the robot hand category or joint parameters, this representation is closer to the physical essence of grasping stability and is also easier to transfer across different hand shapes.

[0020] In the grasping decision model, this invention employs a local geometric encoder to extract the local contact embedding of each contact point pair; subsequently, a contact-aware multi-pooling module is used to calculate the maximum statistic, average statistic, and standard deviation statistic, and dynamically generate fusion weights based on the number of effective contact point pairs or contact confidence. Thus, the model can form a stable global contact descriptor regardless of whether the contact points are sparse, dense, or unevenly distributed.

[0021] This invention also provides training and inference selection strategies. During the training phase, labeled grasping test data is used to supervise the model's output stability score. The robustness of the model to perceived noise and hand shape changes is improved through coordinate perturbation, normal perturbation, random contact point discarding, and enhancement with different sampling densities. During the inference phase, the system can rank candidate grasping postures based on their stability scores and select the final execution posture by combining collision detection and motion planning.

[0022] In summary, the beneficial effects of this invention include: First, by adopting a unified geometric representation of hand-object contact, the dependence of the model on specific manipulator structural parameters is reduced; Second, by explicitly introducing the surface normals of the object and the hand, the model can identify opposing contact, tangential contact, and unfavorable contact; Third, by adapting contact perception multi-pooling to different numbers of contact points and different contact area sizes, the migration capability between heterogeneous end effectors is improved; Fourth, by outputting continuous stability scores, a quantifiable basis is provided for candidate grasping sorting, threshold screening, and subsequent motion planning. Attached Figure Description

[0023] Figure 1 A flowchart illustrating a grasping stability evaluation method based on hand-object interaction contact characterization provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the hand-object interaction stability assessment structure in a grasping stability assessment method based on hand-object interaction contact characterization provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of a grasping stability evaluation device based on hand-object interaction contact characterization provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0025] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0026] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0027] In the description of this invention, the expression “A and / or B” (where A and B are used to formally represent specific features) will be used. The corresponding expression includes the following three combinations: only A, only B, and a combination of A and B.

[0028] As used in this invention, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from a particular value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0029] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0030] Example 1: The following is combined with Figure 1 and Figure 2 This invention provides a detailed description of a grasping stability evaluation method based on hand-object interaction contact characterization, as described in the embodiments of the present invention. Figure 1 As shown, the method specifically includes the following steps: Step 101: Obtain the input data required for the crawl stability assessment.

[0031] The input data includes object geometry information, robot hand specifications, and candidate grasping postures. The object geometry information includes object point clouds, object mesh models, local surfaces reconstructed from depth maps, or combinations thereof. The robot hand specifications include at least one of hand kinematic parameters, link geometry models, finger surface meshes, joint constraints, and surface sampling rules. The candidate grasping postures include the hand base pose and hand joint configuration, or grasping parameters capable of recovering the hand base pose and hand joint configuration.

[0032] In one embodiment, when the end effector is a parallel gripper, the candidate gripping posture further includes the gripper opening width or closing amount; when the end effector is a multi-fingered dexterous hand, the candidate gripping posture further includes the joint angles of each finger.

[0033] In one embodiment, the robotic system first acquires the geometric information of the object to be evaluated. This geometric information can be a point cloud acquired by a depth camera, a CAD model, a triangular mesh model, or a local surface model obtained through multi-view depth fusion. For point cloud input, the system can first perform outlier removal, voxel downsampling, and normal estimation; for mesh input, the system can uniformly sample points on the object's surface.

[0034] The specifications of a robot hand include the hand kinematics chain, the geometry of each link, joint constraints, surface mesh, and the positive kinematic mapping from the joint angles to the hand surface. For multi-fingered dexterous hands, the hand specifications can be described by URDF, MJCF, or equivalent models; for parallel grippers, they can be described by the geometry of the gripper finger surfaces on both sides, the opening width, and the closing direction.

[0035] Candidate grasping poses can be generated by a grasping detection network, heuristic search, taught trajectories, or manually specified. The system unifies the candidate grasping poses, object geometry, and hand model into a single coordinate system. This coordinate system can be an object-centered coordinate system, a camera coordinate system, a robot base coordinate system, or a candidate grasping local coordinate system. To facilitate learning across objects and hand shapes, an object-centered coordinate system is preferred, and the coordinates are normalized according to the object scale.

[0036] Step 102: Construct the hand surface and the object surface based on the input data.

[0037] Specifically, based on the robot hand specifications and candidate grasping postures, the position and posture of each link of the hand in a unified coordinate system are calculated using forward kinematics; the surface mesh of each link of the hand is sampled to obtain a set of points on the hand surface; the hand surface normal vector is obtained by interpolation of the surface normal of the hand surface mesh or by local neighborhood fitting, and the hand surface is constructed accordingly; the set of points on the object surface is sampled or read from the object geometry information; the object surface normal vector is obtained by the object mesh normal, principal component analysis of the point cloud neighborhood or local plane fitting of the depth map, and the object surface is constructed accordingly.

[0038] In one embodiment, for a given candidate grasping posture, the system performs forward kinematics based on the hand base pose and joint configuration to obtain the position and orientation of each hand link in a unified coordinate system. Subsequently, the surface of each link is sampled to obtain a set of points on the hand surface. The sampling density can be adaptively set based on link area, contact importance, or computational budget.

[0039] In one embodiment, the hand surface normal vector can be obtained by interpolating the surface normals of the connecting rod surface mesh, or by performing planar fitting on the local neighborhood of the sampling point. For regular planar structures such as the gripper finger surfaces, the normal vector can be directly given by the hand model. The system can uniformly set the hand normal vector to point from the surface of the hand entity to the external space, so as to calculate the angle with the surface normal of the object.

[0040] In one embodiment, the set of points on the object surface can be provided directly from the object point cloud or sampled from the object mesh surface. For sensor point clouds, the system can estimate the object surface normal vector through local neighborhood principal component analysis and unify the normal orientation according to the camera's line of sight. If multi-view point cloud fusion results exist, the normal vectors of overlapping regions can be weighted and averaged to reduce single-view noise.

[0041] Step 103: Construct a set of effective contact point pairs based on the hand surface and the object surface.

[0042] Specifically, the nearest neighbor distance from each object surface point to the set of hand surface points is calculated. Object surface points with nearest neighbor distances less than a preset distance threshold are identified as object contact points. The hand surface point closest to the object contact point is identified as the corresponding hand contact point. Effective contact point pairs are constructed based on the object contact point, the corresponding hand contact point, the object surface normal vector, and the hand surface normal vector. When the number of effective contact point pairs is greater than a preset number, spatial uniform sampling, confidence-priority sampling, or random sampling are performed on the effective contact point pairs. When the number of effective contact point pairs is less than or equal to the preset number, repeated sampling, masking, or low-confidence marking are performed on the effective contact point pairs. The nearest neighbor distance is the Euclidean distance from the object surface point to the set of hand surface points, a weighted distance with normal consistency constraints, or a directed distance considering the hand closure direction. Each element in the set of effective contact point pairs also includes at least one of the following: the distance between the object contact point and the hand contact point, the normal angle, and the contact confidence.

[0043] In one embodiment, the system calculates the nearest neighbor distance from each object surface point to the set of hand surface points. If the nearest neighbor distance is less than a preset distance threshold, the object surface point is considered to be likely to make contact or close contact with the hand under the candidate grasping posture, and is identified as the object contact point, and the corresponding nearest hand surface point is identified as the hand contact point.

[0044] In one embodiment, the distance threshold can be set based on the object point cloud scale, hand model sampling resolution, depth sensor noise, and manipulator closure error. For example, for a fine multi-fingered hand, a smaller threshold can be used to emphasize real contact; for scenarios with sparse point clouds or sensing errors, the threshold can be appropriately increased and distance features and contact confidence can be introduced into subsequent models.

[0045] In addition to Euclidean distance, the system can also introduce normal consistency constraints. If the normal vectors of the object surface and the hand surface are approximately opposite in direction, the contact is more likely to provide stable support; if they are approximately in the same direction or the angle between them is too small, it may be a scuff or ineffective contact. The system can treat the normal angle as part of the contact confidence, rather than simply discarding the point, thus retaining more learnable information.

[0046] To control the input size, the system can resample the set of valid contact point pairs. When the number of contact points is too large, a fixed number of point pairs can be retained by spatial uniform sampling, normal diversity sampling, or confidence-priority sampling; when the number of contact points is too small, the model input format can be kept stable by repeated sampling, padding masks, or low-confidence markers.

[0047] Step 104: Construct a characterization of the stability of the interaction between the hand and the object based on the effective contact points.

[0048] Specifically, the coordinates of the object contact point are used as the spatial coordinates of the point cloud; the object surface normal vector, the hand contact point coordinates, and the hand surface normal vector are used as point-by-point geometric features; the object contact point coordinates are transformed into the object-centered coordinate system or the candidate grasping posture local coordinate system; the object contact point coordinates, hand contact point coordinates, object surface normal vectors, and hand surface normal vectors are scaled to form a unified input representation that adapts to objects of different sizes and different end effectors.

[0049] In one embodiment, for each valid pair of contact points, the system constructs basic features including the coordinates of the object contact points, the object surface normal vector, the coordinates of the hand contact points, and the hand surface normal vector. Further features may include the distance between the two contact points, the angle between their normals, the finger number to which the contact point belongs, the vector of the contact point relative to the object's centroid, and the contact confidence level.

[0050] To reduce the model's dependence on specific manipulator structures, this invention does not directly use the manipulator name or complete joint angles as the primary input for discrimination. Instead, it implicitly expresses the geometric role of the hand at the contact interface through the coordinates of the hand contact points and the hand normal vector. In this way, different hand shapes can be mapped to the same type of contact representation as long as surface points and normal vectors can be generated.

[0051] The object-centered coordinate system allows the model to focus on the relationship between the contact distribution and the object. For example, relatively balanced multi-sided contact around the object's center of mass is generally more stable than contact concentrated on one side; the hand's normal relative to the object's surface normal is more likely to provide effective normal support; and contact points covering the area above and to the sides of the object's center of mass generally provide stronger resistance to disturbances. All of the above information can be expressed by the contact characterization of this invention.

[0052] Step 105: Based on the stability representation of hand-object interaction, output the stability score of candidate grasping postures using the grasping decision model.

[0053] Specifically, the hand-object interaction stability characterization is input into a local geometric encoder to obtain the local contact embedding for each effective contact point pair. Max pooling, average pooling, and standard deviation pooling are performed on the local contact embeddings using a contact-aware multi-pooling module. Dynamic fusion weights are generated based on the number of effective contact point pairs, contact area distribution, or contact confidence. The max pooling, average pooling, and standard deviation pooling results are fused using these dynamic fusion weights to obtain a global contact descriptor. The global contact descriptor is then input into a score prediction head and a sigmoid function to obtain a stability score ranging from 0 to 1. The local geometric encoder can be a multilayer perceptron, a PointNet-style shared network, a graph neural network, or a local neighborhood attention network.

[0054] In one embodiment, the grasping decision model receives a point cloud input in the form of a set of valid contact point pairs. A local geometric encoder first encodes the basic features of each point pair to obtain a local contact embedding. The local geometric encoder can employ a multilayer perceptron, a PointNet-style shared network, a graph neural network, or a local neighborhood attention network.

[0055] Since the number of effective contact points generated by different candidate grasping postures can vary significantly, using only max pooling may ignore the overall contact distribution, using only average pooling may weaken key strong contact information, and using only standard deviation pooling is insufficient to express absolute contact intensity. Therefore, this invention employs a contact-aware multi-pooling module to simultaneously calculate the results of max pooling, average pooling, and standard deviation pooling.

[0056] The contact-aware multi-pooling module inputs the number of effective contact point pairs, contact confidence statistics, or contact area coverage into the weight generation network to obtain dynamic fusion weights for the three pooling statistics. When contact points are sparse, the model can increase the maximum pooling weight to focus on key contacts; when contact points are dense and evenly distributed, the model can increase the average pooling weight; when contact distribution varies greatly, the model can use standard deviation pooling to express the non-uniformity.

[0057] The fused global contact descriptor is used as the input score prediction head. The score prediction head can consist of several fully connected layers, normalization layers, and nonlinear activation functions, ultimately outputting a stability score between 0 and 1 via a sigmoid function. A higher score indicates a higher probability that the candidate grasping pose will maintain object stability during closure, lift-up, and perturbation.

[0058] Furthermore, the grasping stability evaluation method based on hand-object interaction contact representation also includes training a grasping decision model. This model can be trained using real robot grasping test data, physical simulation data, or a mixture of both. Each training sample includes object geometry, robot hand specifications, candidate grasping postures, and a grasping result label. The grasping result label can be determined based on criteria such as whether the object was successfully lifted, whether it fell during movement, or whether it remained stable after external disturbances.

[0059] The training loss can be a binary cross-entropy loss. When the ratio of successful to unsuccessful grabs in the training samples is unbalanced, class weights can be set for the loss. When it is necessary to learn the relative merits of candidates, a ranking loss can be added so that the score of successful samples is higher than the score of similar unsuccessful samples. This embodiment does not limit the specific loss function; ranking loss, class imbalance weighted loss, or a combination thereof can also be used when training the grabbing decision model.

[0060] In one embodiment, during the training of the grasping decision model, random perturbations are added to the object contact point coordinates, hand contact point coordinates, object surface normal vectors, and hand surface normal vectors. Effective contact point pairs are also randomly discarded, randomly resampled, or scale-perturbed to enhance the model's robustness to perceptual noise, occlusion, and different hand shape sampling densities. Specifically, various enhancements can be applied to the contact representation during the training phase, including adding Gaussian perturbations to the object contact point coordinates, randomly flipping a small number of normals, randomly discarding some effective contact points, changing the hand surface sampling density, perturbing the object scale, and simulating depth noise. Through these enhancements, the model can adapt to noise and occlusion in real-world perception systems.

[0061] When the goal is cross-hand transfer, the training set can include various manipulators and grippers. Since the input representation is uniformly defined as contact point coordinates and normal vectors, the model does not need to have a completely independent network structure for each hand type. For a new hand type, only its kinematics and surface geometry need to be provided to construct a hand-object interaction stability representation in the same format.

[0062] Step 106: Perform a capture and filtering process based on the stability score.

[0063] Specifically, candidate grasping postures with stability scores greater than the stability score threshold are retained; or the top K candidate grasping postures are selected from high to low according to their stability scores; and the filtered candidate grasping postures are output to the motion planning module, collision detection module, or robot execution module.

[0064] During the inference phase, the upstream grasping generation module typically provides multiple candidate grasping postures. The system constructs a hand-object interaction stability representation for each candidate posture and inputs it into the grasping decision model to obtain a stability score. Subsequently, candidates with scores greater than a threshold can be retained, or the top K candidates can be selected according to their scores for collision detection and motion planning.

[0065] The stability score can be used in conjunction with other evaluation metrics. For example, the system can weight and combine the stability score, robot reachability score, path collision cost, and execution time cost to obtain the final capture cost function. If the stability score is high but the path is unreachable, the next candidate is selected; if multiple candidates are stable, the candidate with the shorter motion path or the closest to the current end effector posture is given priority.

[0066] During robot execution, if the perception system updates the object point cloud or candidate grasping pose, the system can recalculate the hand-object interaction stability representation and update the stability score. For dynamic scenarios, this invention can also be combined with target tracking and grasping pose update methods to re-evaluate candidate poses updated with object movement.

[0067] In different grasping tasks, the stability score threshold can be set according to the task risk. For tasks involving fragile objects, valuable objects, or high-speed handling, the system can use a higher threshold, allowing only candidates with stability scores significantly higher than the experience safety line to enter the process; for low-risk sorting tasks, the system can appropriately lower the threshold to improve grasping coverage and operational efficiency.

[0068] In one embodiment, the stability score threshold can also be dynamically adjusted based on the object category, surface material, and end effector type. For example, for smooth metal objects, the system can increase the requirements for normal relative relationship and contact coverage; for soft objects, the system can allow for a larger proximity distance and use hand surface compliance as a threshold correction factor; for suction cup end effectors, the system can focus on contact area flatness and normal consistency.

[0069] When the upstream crawling and generation module outputs a large number of candidates, the system can first use lightweight rules to filter out obviously invalid candidates, and then run the crawling decision model on the remaining candidates. Lightweight rules include initial collision screening, lower limit of the number of contacts, end effector opening range, and workspace constraints. Through hierarchical filtering, the number of model inferences can be reduced and real-time performance can be improved.

[0070] During continuous operation, the system can automatically adjust the stability score threshold based on the recent capture success rate. If candidates with high stability scores still fail frequently, it indicates that there are unmodeled factors in the current environment, and the system increases the threshold and records the failure samples. If candidates with medium stability scores execute successfully for a long period of time, the system can appropriately relax the threshold to improve throughput.

[0071] For collaborative scenarios requiring manual review, the system can display stability scores and explanatory power on the user interface. Operators can view the critical contact areas, low-confidence contact positions, and system-suggested rankings corresponding to candidate grasping postures, thus enabling manual confirmation in complex or high-risk tasks. This embodiment improves the controllability of the method in actual production lines and experimental platforms.

[0072] Furthermore, to facilitate understanding of the stability assessment results by the robot system and operators, the grasping decision model can output several auxiliary explanatory quantities in addition to the stability score. These auxiliary explanatory quantities include high-contribution contact areas, low-confidence contact areas, contact point count statistics, normal relative relationship statistics, and the geometric relationship between the candidate pose and the object's centroid.

[0073] The system assigns importance weights to each valid contact point pair based on the contribution of local contact embedding to the final score. Contact points with higher importance typically correspond to key areas that support the object's weight, resist rotational disturbances, or form opposing grips. If high-contribution contact areas are concentrated on one side of the object, the system can prompt the upstream grasping and generation module to search for more balanced candidate poses.

[0074] In batch candidate selection scenarios, the system can output a list of candidate grasping postures. Each candidate includes a stability score, number of contacts, average contact distance, average normal angle, and a flag indicating whether it passes the low-contact gating. Based on this, the motion planning module can prioritize candidates with high stability and sufficient contact quality, reducing invalid planning attempts.

[0075] The above explanation and output do not change the calculation process of the stability score, but they can improve the diagnostics of the method in engineering systems. When a grasping failure occurs, developers can quickly determine whether the problem is a candidate pose generation issue, an object perception issue, a hand model calibration issue, or a model threshold setting issue based on the cause of the low score and the contact distribution.

[0076] The grasping stability evaluation method based on hand-object interaction contact representation also includes anomaly handling: when the number of effective contact point pairs is zero or lower than a preset threshold, it indicates that the candidate grasping posture has insufficient contact with the object under the current geometric relationship. The system can directly determine the candidate as low stability, or it can input it into the model and attach a low contact number label, causing the model to output a low confidence score.

[0077] When there are large areas of missing points in the object's point cloud, the system can construct partial contact representations based on the visible areas, while simultaneously reducing the stability score confidence. If the critical contact area of ​​a candidate pose is located in an invisible area, the system can request the robot to adjust its viewpoint or perform active perception before conducting a stability assessment.

[0078] When there are calibration errors between the hand model and the actual end effector, the fault tolerance can be improved by increasing the distance threshold, introducing contact distance features, and adding coordinate perturbations during the training phase. For flexible fingers or soft end effectors, the allowable contact distance range can be appropriately expanded, and surface compliance parameters can be added as additional features.

[0079] The grasping stability evaluation method based on hand-object interaction contact representation also includes cross-device migration and online calibration. In cross-device migration scenarios, the size, number of fingers, joint degrees of freedom, and surface sampling density of different robot hands may vary significantly. This invention expresses the actual contact interface through hand-object contact point pairs, enabling the model to bypass specific hand shape parameter differences at the input level. For new end effectors, only their geometric model and forward kinematics are required to generate a contact representation consistent with the training phase.

[0080] To further improve migration performance, the system can collect a small number of real-world capture samples in the early stages of a new device's deployment to fine-tune the fractional prediction head or normalization layer. Since the local geometry encoder has already learned the stability relationships between contact points, normals, and spatial distribution, a small number of samples are usually sufficient to calibrate the scale, compliance, and frictional characteristics of the new device.

[0081] The system can also maintain online statistics to correct for deviations between stability scores and actual capture results. For example, when a certain type of object frequently fails with high scores in actual execution, the system raises the screening threshold for that category or material; when an end effector performs stably within a specific number of contacts, the system can appropriately relax the candidate retention criteria.

[0082] The online calibration described above does not change the basic contact characterization of the present invention, but adapts the method to the deployment environment at the level of scoring threshold, confidence calibration and lightweight model fine-tuning, so that the method can maintain availability across different robot platforms, different sensor accuracies and different tasks.

[0083] In summary, this invention can serve as a candidate posture evaluation module in robot grasping systems, applicable to scenarios such as industrial sorting, warehouse picking, service robot pick-and-place, dexterous hand operation, and simulation-to-real-world migration. By unifying the contact relationships of different end effectors into a hand-object interaction contact representation, this invention improves the versatility of grasping stability assessment and the efficiency of engineering deployment.

[0084] Example 2: Based on the grasping stability evaluation method based on hand-object interaction contact characterization provided in the foregoing embodiments, the present invention also provides a grasping stability evaluation device based on hand-object interaction contact characterization that can be used to implement the above method, such as... Figure 3 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. The grasping stability evaluation device based on hand-object interaction contact characterization in this embodiment includes one or more processors 21 and a memory 22. Figure 3 Take a processor 21 as an example.

[0085] Processor 21 and memory 22 can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0086] The memory 22, as a non-volatile computer-readable storage medium for the grasping stability evaluation method based on hand-object interaction contact characterization, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the grasping stability evaluation method based on hand-object interaction contact characterization in the aforementioned embodiments. The processor 21 executes various functional applications and data processing of the grasping stability evaluation device based on hand-object interaction contact characterization by running the non-volatile software programs, instructions, and modules stored in the memory 22, thereby realizing the grasping stability evaluation method based on hand-object interaction contact characterization in the aforementioned embodiments.

[0087] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0088] The program instructions / modules are stored in memory 22. When executed by one or more processors 21, they perform the grasping stability evaluation method based on hand-object interaction contact characterization in the aforementioned embodiments.

[0089] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 3 One of the processors 21 enables the one or more processors to execute the grasping stability assessment method based on hand-object interaction contact characterization in the foregoing embodiments.

[0090] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0091] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for evaluating grasping stability based on hand-object interaction contact characterization, characterized in that, include: Obtain the input data required for crawl stability assessment, including the following sub-steps: The system acquires the object geometry information, robot hand specifications, and candidate grasping postures of the object to be evaluated. The object geometry information includes object point clouds, object mesh models, local surfaces reconstructed from depth maps, or combinations thereof. The robot hand specifications include at least one of hand kinematic parameters, link geometry models, finger surface meshes, joint constraints, and surface sampling rules. The candidate grasping postures include the hand base pose and hand joint configuration, or grasping parameters capable of recovering the hand base pose and hand joint configuration. Constructing the hand surface and object surface based on the input data includes the following sub-steps: Based on the robot hand specifications and candidate grasping postures, the position and posture of each link of the hand in a unified coordinate system are calculated using forward kinematics. By sampling the surface mesh of each link of the hand, a set of points on the hand surface is obtained; The hand surface normal vector is obtained by interpolating the surface normal vector of the hand surface mesh or by fitting the local neighborhood, and the hand surface is constructed accordingly. Sample or read a set of points on the object's surface from the object's geometric information; The surface normal vector of the object is obtained from the object mesh normal, principal component analysis of the point cloud neighborhood, or local plane fitting of the depth map, and the object surface is constructed accordingly. Constructing a set of effective contact point pairs based on the hand surface and the object surface includes the following sub-steps: Calculate the nearest neighbor distance from each point on the object's surface to the set of points on the hand's surface; Points on the surface of an object whose nearest neighbor is less than a preset distance threshold are identified as object contact points; The hand surface point closest to the contact point with the object is determined as the corresponding hand contact point; Effective contact point pairs are constructed based on the object contact point, the corresponding hand contact point, the object surface normal vector, and the hand surface normal vector; when the number of effective contact point pairs is greater than the preset number, the effective contact point pairs are sampled spatially uniformly, with confidence-priority sampling, or randomly. When the number of valid contact point pairs is less than or equal to the preset number, the valid contact point pairs are repeatedly sampled, filled with a mask, or marked with a low confidence flag. Based on the effective contact points, construct a characterization of the hand-object interaction stability of the set, including the following sub-steps: The coordinates of the object's contact point are used as the spatial coordinates of the point cloud. The object surface normal vector, the coordinates of the hand contact point, and the hand surface normal vector are used as point-by-point geometric features; Transform the coordinates of the object's contact point to the object's centered coordinate system or the local coordinate system of the candidate grasping posture; The coordinates of the object contact point, the coordinates of the hand contact point, the object surface normal vector, and the hand surface normal vector are scaled to form a unified input representation that adapts to objects of different sizes and different end effectors; Based on the stability characterization of hand-object interaction, the stability score of candidate grasping postures is output using a grasping decision model, including the following sub-steps: The hand-object interaction stability characterization is input into the local geometry encoder to obtain the local contact embedding for each effective contact point pair; The contact-aware multi-pooling module performs max pooling, average pooling, and standard deviation pooling on local contact embeddings, respectively. Dynamic fusion weights are generated based on the number of effective contact points, the distribution of contact areas, or the contact confidence level. The global contact descriptor is obtained by fusing the maximum pooling result, average pooling result, and standard deviation pooling result with the dynamic fusion weights. The global contact descriptor is input into the score prediction head and the Sigmoid function to obtain a stability score with a value range of 0 to 1; Based on the stability score, the data is filtered and processed, including the following sub-steps: Retain candidate grasping postures with stability scores greater than the stability score threshold; or select the top K candidate grasping postures in descending order of stability score. The selected candidate grasping postures are output to the motion planning module, collision detection module, or robot execution module.

2. The grasping stability evaluation method based on hand-object interaction contact characterization according to claim 1, characterized in that, The nearest neighbor distance is the Euclidean distance from the object surface point to the set of hand surface points, the weighted distance with normal consistency constraints, or the directed distance considering the hand's closing direction.

3. The grasping stability evaluation method based on hand-object interaction contact characterization according to claim 1, characterized in that, Each element in the set of effective contact points also includes at least one of the following: the distance between the object contact point and the hand contact point, the normal angle, and the contact confidence level.

4. The grasping stability evaluation method based on hand-object interaction contact characterization according to claim 1, characterized in that, The local geometric encoder is a multilayer perceptron, a PointNet-style shared network, a graph neural network, or a local neighborhood attention network.

5. The grasping stability evaluation method based on hand-object interaction contact characterization according to claim 1, characterized in that, When the number of effective contact point pairs is less than a preset threshold, the candidate grasping posture is judged as a low-confidence candidate, or the number of effective contact point pairs is used as an additional input to the grasping decision model so that the grasping decision model can adaptively evaluate sparse contact situations.

6. The grasping stability evaluation method based on hand-object interaction contact characterization according to claim 1, characterized in that, The grasping decision model is trained using labeled grasping test data, where the labels characterize whether the candidate grasping posture maintains successful grasping after closing, lifting, moving, or being subjected to external disturbances.

7. The grasping stability evaluation method based on hand-object interaction contact characterization according to claim 6, characterized in that, The grasping decision model is trained using binary cross-entropy loss, ranking loss, class imbalance weighted loss, or a combination thereof.

8. The grasping stability evaluation method based on hand-object interaction contact characterization according to claim 6, characterized in that, When training the grasping decision model, random perturbations are added to the coordinates of the object contact point, the coordinates of the hand contact point, the normal vector of the object surface, and the normal vector of the hand surface. Effective contact point pairs are randomly discarded, randomly resampled, or scaled to enhance the robustness of the model to perceived noise, occlusion, and different hand shape sampling densities.

9. The grasping stability evaluation method based on hand-object interaction contact characterization according to claim 1, characterized in that, When the end effector is a parallel gripper, the candidate gripping posture also includes the gripper opening width or closing amount; when the end effector is a multi-fingered dexterous hand, the candidate gripping posture also includes the joint angles of each finger.

10. A grasping stability evaluation device based on hand-object interaction contact characterization, characterized in that, The method includes a data acquisition module, a hand surface generation module, a contact point pair construction module, a contact characterization generation module, a grasping decision model, and a posture screening module, each of which is configured to perform the method described in any one of claims 1 to 9.

Images