3D hand-object grasping generation method and system based on fingertip perception and geometric guidance

Abstract

The application discloses a 3D hand-object grasping generation method and system based on fingertip perception and geometric guidance, and relates to the technical field of computer vision.The method comprises the following steps: performing feature extraction on a target object point cloud, generating a fingertip contact map, a global contact map, a hand region allocation map and a contact direction map by using a sequence conditional variational autoencoder, and constructing a multi-component interaction representation; correcting the multi-component interaction representation by using target object geometric information to obtain a corrected multi-component interaction representation; taking the corrected multi-component interaction representation as structured constraint information, combining a parameterized hand model and a piecewise signed distance function to establish a constraint, optimizing hand posture and shape parameters, and generating a three-dimensional hand grasping posture.The application solves the problems of grasping ambiguity and physically infeasible prediction of existing methods, significantly reduces the hand-object penetration degree, and improves the grasping stability and geometric consistency.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, robot manipulation, and 3D generation technology, and in particular to a 3D hand object grasping and generation method and system based on fingertip perception and geometric guidance. Background Technology

[0002] In the fields of computer vision, virtual reality, animation, and robotics, synthesizing natural, diverse, and physically plausible human hand grasping postures for arbitrary 3D objects is a fundamental challenge. Given a 3D object, the goal of grasping generation is not to predict a single deterministic hand posture, but to infer multiple plausible hand-object interaction configurations consistent with the object's geometry and human grasping patterns. Due to the extremely high degree of freedom of the human hand, recent research methods have typically introduced intermediate interaction representations, such as contact maps, hand region correspondences, and contact directions, to guide the generation or optimization of the final hand posture. These object-centered representations have shown great potential in modeling diverse grasping assumptions.

[0003] However, existing object-centric contact representations have significant limitations in capturing fine-grained interaction cues. First, the coarse-grained nature of contact representations leads to grasping ambiguity. A common strategy is to model object contact as a single global probability map, assigning a probability of being touched to each point on the object's surface. While this provides coarse-grained spatial guidance, it is severely inadequate in distinguishing different grasping configurations. Fingertip placement often has a significant impact on grasping patterns and stability; however, while different grasps may have highly similar global contact distributions, fingertip contact patterns can be fundamentally different. Relying solely on coarse-grained contact coverage will lead to confusion at the representational level between structurally disparate grasping patterns, severely limiting the diversity of generated grasps. Second, purely data-driven models often produce physically infeasible predictions. Even after generating interaction maps, data-driven methods often produce locally inconsistent predictions, especially for geometrically complex objects. The predicted contact map may be spatially discontinuous, the hand region allocation may be uncertain, and local contact directions may oscillate violently between neighboring points. More seriously, the pure data-driven model lacks explicit physical boundary constraints, and frequently generates physically infeasible interactive targets such as pointing into the interior of the object during the decoding stage. This leads to conflicts between contact constraints and penetration constraints during subsequent optimization, ultimately resulting in physically unreasonable phenomena such as hand hovering or hand-object penetration in the generated hand grasping posture. Summary of the Invention

[0004] In view of this, the present invention provides a 3D hand object grasping generation method and system based on fingertip perception and geometric guidance, which aims to solve the technical problems of insufficient representation of fine-grained hand object interaction relationships in existing methods, and poor consistency between the generated interaction representation and the geometric structure of the target object, resulting in confusion of grasping modes and physical unreasonable phenomena (such as penetration and suspension) in the generated posture.

[0005] In a first aspect, the present invention discloses a 3D hand object grasping generation method based on fingertip perception and geometric guidance, comprising the following steps: Obtain the point cloud of the target object, and perform feature extraction on the point cloud of the target object to obtain the geometric feature representation of the object; The geometric feature representation of the object is input into a pre-trained conditional variational autoencoder, which generates multi-component interactive representations step by step according to a preset hierarchical dependency order. The multi-component interactive representations include fingertip contact map, global contact map, hand region allocation map, and contact direction map. Based on the geometric information contained in the point cloud of the target object, the fingertip contact map, the global contact map, the hand region allocation map, and the contact direction map are corrected respectively to obtain a corrected multi-component interactive representation; The modified multi-component interaction representation is used as structured constraint information, and a joint optimization objective function is constructed by combining the parameterized hand model and the piecewise symbolic distance function. By minimizing the joint optimization objective function, the pose and shape parameters of the parameterized hand model are iteratively optimized to generate the target 3D human hand grasping pose.

[0006] As an optional implementation of the first aspect of this application, the step of generating multi-component interactive representations step by step according to a preset hierarchical dependency order by the conditional variational autoencoder includes: generating the fingertip contact map based on the object geometric feature representation and latent variables sampled from the latent space; the fingertip contact map is used to represent the probability that each sampling point in the target object point cloud is touched by a fingertip; generating the global contact map by combining the fingertip contact map as a condition with the object geometric feature representation; the global contact map is used to represent the probability that each sampling point in the target object point cloud is touched by the hand as a whole; generating the hand region allocation map by combining the fingertip contact map and the global contact map; the hand region allocation map is used to represent the probability that each sampling point in the target object point cloud corresponds to a predefined hand region; generating the contact direction map by combining the fingertip contact map, the global contact map, and the hand region allocation map; the contact direction map is used to represent the local hand-object contact direction vector corresponding to each sampling point in the target object point cloud.

[0007] As an optional implementation of the first aspect of this application, the fingertip contact map is corrected based on the geometric information contained in the target object point cloud, including: constructing a local density prior based on the neighborhood distribution of the target object point cloud, and weightedly fusing the local density prior with the fingertip contact map to obtain the corrected fingertip contact map.

[0008] As an optional implementation of the first aspect of this application, the step of correcting the global contact map based on the geometric information contained in the target object point cloud includes: constructing an axis-aligned bounding box based on the spatial distribution of the target object point cloud, calculating the object geometric center of the axis-aligned bounding box; constructing a distance prior based on the distance relationship between each sampling point in the target object point cloud and the object geometric center, and weightedly fusing the distance prior with the global contact map to obtain the corrected global contact map.

[0009] As an optional implementation of the first aspect of this application, the hand region allocation map is modified based on the geometric information contained in the target object point cloud, including: calculating the contact intensity of each sampling point in the target object point cloud based on the modified global contact map and the modified fingertip contact map; the contact intensity is calculated by performing a comprehensive operation on the modified global contact map and the modified fingertip contact map; the contact intensity is used to adjust the weight of the hand region allocation map to improve the confidence of the hand region allocation result corresponding to the contact intensity region, thereby obtaining an enhanced hand region allocation map.

[0010] As an optional implementation of the first aspect of this application, the contact orientation map is corrected based on the geometric information contained in the target object point cloud, including: for each sampling point in the target object point cloud, calculating the mean value of the direction vector corresponding to the neighboring sampling points in the contact orientation map to obtain the mean value of the neighboring directions; using the contact intensity to perform weighted fusion of the network predicted direction and the mean value of the neighboring directions in the contact orientation map; and normalizing the weighted fused direction vector to obtain the corrected contact orientation map.

[0011] As an optional implementation of the first aspect of this application, the joint optimization objective function includes a contact constraint term, a penetration constraint term, a direction constraint term, and a parameter regularization term; the contact constraint term is used to establish the contact relationship between the target object point cloud and the corresponding hand region; the penetration constraint term is used to apply a penetration penalty to the object sampling points located inside the parameterized hand model; the direction constraint term is used to keep the optimized hand posture consistent with the local direction information in the corrected multi-component interaction representation; the parameter regularization term is used to constrain the posture parameters and shape parameters of the parameterized hand model to suppress excessive joint movement and unreasonable hand deformation.

[0012] Secondly, this application discloses a 3D hand object grasping and generation system based on fingertip perception and geometric guidance, comprising: The feature extraction and generation module is used to acquire the point cloud of the target object, extract features from the point cloud to obtain the geometric feature representation of the object, and input the geometric feature representation of the object into a pre-trained conditional variational autoencoder. The conditional variational autoencoder generates multi-component interactive representations step by step according to a preset hierarchical dependency order. The multi-component interactive representations include a fingertip contact map, a global contact map, a hand region allocation map, and a contact direction map. The geometric guidance correction module is used to correct the fingertip contact map, the global contact map, the hand region allocation map, and the contact direction map respectively based on the geometric information contained in the target object point cloud, so as to obtain the corrected multi-component interactive representation; The posture optimization and recovery module is used to take the corrected multi-component interaction representation as structured constraint information, and combine it with the parameterized hand model and the piecewise symbolic distance function to construct a joint optimization objective function; by minimizing the joint optimization objective function, the posture parameters and shape parameters of the parameterized hand model are iteratively optimized to generate the target 3D human hand grasping posture.

[0013] Thirdly, this application discloses an electronic device, comprising: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the method as described in the first aspect.

[0014] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the method described in the first aspect.

[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention generates a multi-component interactive representation, including a fingertip contact map, a global contact map, a hand region allocation map, and a contact direction map, step by step according to a preset hierarchical dependency order through a conditional variational autoencoder. Fine-grained fingertip contact information is used as a priori conditions to guide the generation of subsequent global contact and region allocation. This feature helps to alleviate the ambiguity of grasping representation caused by the existing single global probability map, enabling the model to more fully characterize the fine-grained hand-object interaction relationship and improve the diversity and rationality of the generated grasping posture.

[0016] (2) This invention utilizes the geometric information contained in the point cloud of the target object to correct the multi-component interaction representation. By introducing distance prior, local density prior, and contact strength, the interaction graph predicted by the network is smoothed and enhanced. This feature helps to reduce local inconsistencies and physically infeasible predictions in the results generated by the pure data-driven model, improves the consistency between the interaction representation and the geometric structure of the target object, and provides structured geometric constraint information for subsequent attitude recovery.

[0017] (3) This invention combines the modified multi-component interactive representation with a parameterized hand model and a piecewise symbolic distance function to construct a joint optimization objective function, and explicitly establishes contact, orientation, and penetration constraints in three-dimensional space. This feature helps to coordinate the relationship between contact constraints, orientation constraints, and penetration constraints during the posture optimization process, reduce the degree of hand-object penetration and the possibility of hand suspension, thereby improving the geometric rationality and posture rationality of the generated three-dimensional human hand grasping posture. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 This is a flowchart of a 3D hand object grasping and generation method based on fingertip perception and geometric guidance according to an embodiment of the present invention; Figure 2 This is a framework diagram for generating 3D hand-held objects based on fingertip perception and geometric guidance, according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of a 3D hand object grasping and generation system based on fingertip perception and geometric guidance according to an embodiment of the present invention; Figure 4 This is a structural diagram of an electronic device disclosed in this invention. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0022] Example 1 Please see Figure 1 This invention provides a 3D hand-grasping generation method based on fingertip perception and geometric guidance. This method aims to synthesize physically plausible hand-grasping data for 3D objects, addressing issues such as insufficient interactive representation capabilities and poor physical plausibility of data-driven model-generated results in grasping generation tasks within the fields of computer vision and robotics.

[0023] like Figure 2 As shown, the method provided in this embodiment of the invention mainly includes three core stages: a multi-component interactive representation generation stage, a geometrically guided interactive representation correction stage, and a grasping posture generation stage. The specific implementation will be described in great detail below according to the steps of the technical solution of this invention.

[0024] S11: Obtain the point cloud of the target object, and perform feature extraction on the point cloud of the target object to obtain the geometric feature representation of the object.

[0025] In 3D human hand grasping generation tasks, the input is typically the 3D geometric data of the target object to be grasped. In this embodiment, point clouds are used as the 3D data representation of the target object. Let the input target object point cloud be represented as follows: ,in N This indicates the number of sampling points on the object's surface. Indicates the first i The three-dimensional spatial coordinates of each sampling point.

[0026] To obtain feature information that characterizes the three-dimensional geometric structure of the target object, feature extraction is required from the input point cloud. This embodiment employs a deep learning-based point cloud feature extraction network (such as PointNet++ or other point cloud feature extraction networks) to extract features from the target object's point cloud. The process involves processing. The feature extraction network extracts the local geometric structure and global shape information of the target object's point cloud through multi-level local region sampling and feature aggregation. After processing by the feature extraction network, a geometric feature representation of the object is obtained. This feature representation not only includes global shape context information, but also retains the information for each sampling point. The corresponding local high-dimensional feature vectors provide object geometric feature information for the subsequent generation of multi-component interactive representations.

[0027] S12: Input the geometric feature representation of the object into a pre-trained conditional variational autoencoder, and generate multi-component interactive representations step by step according to a preset hierarchical dependency order through the conditional variational autoencoder; the multi-component interactive representations include fingertip contact map, global contact map, hand region allocation map and contact direction map.

[0028] This embodiment employs a generative framework based on a Conditional Variational Autoencoder (CVAE) to learn the conditional probability distribution between the geometric features of the target object and the multi-component interaction representation, and to generate multiple interaction representations based on the same target object.

[0029] During the model training phase based on conditional variational autoencoders, the geometric features of the object are used as representations. and the corresponding training multi-component interaction representation As input, the mean and variance parameters of the latent variables are learned, and a latent space representation is constructed: in, and Let represent the mean and standard deviation of the latent variable distribution, respectively. This represents the latent variables learned by the conditional variational autoencoder. The distribution is represented by a Gaussian distribution. During training, reparameterization techniques are used to sample and obtain latent variables. : in, , For noise sampled from a standard normal distribution, This indicates element-wise multiplication.

[0030] During the test generation phase (i.e., the inference phase), the decoders (D1, D2, D3, D4) use the geometric features of the object and latent variables sampled from the pre-defined prior distribution of latent variables. Generate multi-component interaction representations. Unlike traditional methods that predict all interactive components simultaneously (which often leads to feature confusion and physical inconsistencies), this embodiment generates each interactive component level by level according to a preset hierarchical dependency order to establish conditional dependencies between different interactive components. The specific level-by-level generation process is as follows: (1) Generating the fingertip contact map: The model uses the fingertip contact map as the starting point for generation. The fingertip contact map is defined as follows: Fingertip contact information is used to characterize local fine-grained interaction features during the grasping process. The fingertip contact position is related to the grasping method and grasping stability. By prioritizing the generation of fingertip contact maps, fine-grained contact conditions are provided for the subsequent generation of global contact maps, hand region allocation maps, and contact direction maps.

[0031] (2) Generate a global contact map: The generated fingertip contact map As conditional features, combined with object geometric features and latent variables, a global contact map is generated. The global contact map is defined as follows: A global contact map describes the overall contact coverage of a human hand on the surface of a target object. The global contact map, together with the fingertip contact map, constitutes a dual-contact representation. By using the fingertip contact map as conditional information for the generation of the global contact map, the generation of the global contact map is guided by the fingertip contact position, thereby reducing representational confusion between different grasping modes.

[0032] (3) Generate a hand region allocation map: After obtaining the dual-contact representation, further combine... and Generate a hand region assignment map. The hand region assignment map is denoted as... , This indicates the number of predefined hand regions (e.g., dividing the hand into multiple independent regions such as the palm, thumb, and index finger). The hand region allocation map is used to establish the correspondence between sampling points on the target object surface and the predefined hand regions.

[0033] (4) Generate the contact direction diagram: Finally, combine all the generated interaction diagrams to generate the contact direction diagram. The contact direction diagram is denoted as... The contact direction vector describes the relative directional relationship between the sampling point and the corresponding hand contact area. This diagram illustrates the directional relationship of local hand-object contact, providing a basis for directional constraints in the subsequent grasping posture recovery process.

[0034] In the process of generating multi-component interactive representations, the first step is to use the point cloud of the target object. A fingertip contact map is generated, and then a global contact map is generated using the fingertip contact map as a condition. Subsequently, a hand region allocation map is generated by combining the fingertip contact map and the global contact map, and a contact direction map is generated by combining the fingertip contact map, the global contact map, and the hand region allocation map. The entire generation process can be represented as follows: Through the above step-by-step generation method, this example ultimately adopts a multi-component interactive representation. A unified description is provided for the interaction between the target object and the human hand. Among other things, Represents a probability distribution. This represents a fingertip contact diagram. Represents the global contact diagram. This diagram shows the distribution of the hand area. This represents the contact direction map. The fingertip contact map serves as conditional information in the generation of subsequent interactive components, maintaining hierarchical relationships between the interactive components, thereby obtaining a multi-component interactive representation that uniformly describes the interaction relationship between the target object and the human hand.

[0035] S13: Based on the geometric information contained in the target object point cloud, the fingertip contact map, the global contact map, the hand region allocation map, and the contact direction map are corrected respectively to obtain the corrected multi-component interactive representation.

[0036] Since the multi-component interaction representation generated by the conditional variational autoencoder may deviate from the local geometry of the target object, such as some high-contact-probability areas not matching the geometric features of the object surface, or the local contact direction changing discontinuously, this invention introduces a geometry-guided interaction representation correction mechanism, which uses the deterministic geometric information contained in the target object itself to correct the generated multi-component interaction representation.

[0037] (1) Correction of the global contact map: The global contact map describes the possible areas of overall contact between the hand and the object surface. In the process of enveloping grasping, the overall contact area is usually related to the main body area of ​​the target object. Therefore, in this embodiment, an axis-aligned bounding box is constructed based on the spatial distribution of the target object's point cloud, and the geometric center of the object in the axis-aligned bounding box is calculated; then, a distance prior is constructed using the distance between the sampling points on the target object surface and the geometric center of the object. Subsequently, the Euclidean distance between each sampling point on the object surface and the geometric center is calculated, and a distance prior is constructed through normalization and inversion operations. The closer a sampling point is to the geometric center, the larger its corresponding prior distance value. This prior distance is then compared with the global contact map predicted by the network. Perform weighted fusion: in, This represents the corrected global contact diagram. This indicates the adjustment of the fusion weights of the global contact map. The fusion weights can be determined based on the training set or validation set, for example, set to 0.7.

[0038] (2) Correction of fingertip contact map: Fingert contact is usually related to the local geometry of the target object's surface, such as local protrusions, edges, or other areas with significant geometric changes. In this embodiment, a local density prior is constructed based on the local neighborhood distribution of each sampling point in the target object's point cloud. For example, the number of adjacent sampling points within a preset radius neighborhood of each sampling point is counted, and the corresponding local density prior is determined accordingly. Compare it with the fingertip contact map predicted by the network. Perform weighted fusion: in, This represents the corrected fingertip contact diagram. This indicates the fusion weights for correcting the fingertip contact map.

[0039] After obtaining the corrected global contact map and fingertip contact map, the contact strength is further calculated to comprehensively represent the contact confidence level of each sampling point on the object surface. It is defined as follows: in, This represents the corrected global contact diagram. This represents the corrected fingertip contact map. The contact intensity integrates both global contact information and fingertip contact information, and is used as weighting information in the subsequent correction process of the hand area allocation map and contact direction map.

[0040] (3) Correction (enhancement) of hand region allocation map: The hand region allocation map often contains significant noise and uncertainty in the allocation results of low contact probability areas. Therefore, this embodiment first calculates the contact intensity. This is used to comprehensively represent the contact confidence level corresponding to each sampling point on the surface of an object. It utilizes contact strength. The hand region assignment map is adjusted (e.g., as an attention weight or confidence mask) to reduce the influence of assignment results corresponding to low contact confidence regions and increase the hand region assignment weights corresponding to sampling points with higher contact intensity, resulting in an enhanced hand region assignment map. This allows the hand region allocation results to focus more on the area actually involved in grasping and improves the reliability of the correspondence between the sampling points on the target object surface and the predefined hand region.

[0041] (4) Correction of contact pattern: the contact pattern predicted by the network In local areas, discontinuous orientation changes or local noise may exist, which can lead to conflicts between different local orientation constraints during subsequent grasping posture optimization. This embodiment smooths the orientation map based on the neighborhood relationship of the object's point cloud. For each sampling point, the average vector of the corresponding orientation vector of the neighboring sampling points in the contact orientation map is calculated and denoted as the neighborhood orientation mean. Then utilize contact strength The orientation pattern predicted by the network and neighborhood direction mean Perform weighted fusion: in This represents element-wise multiplication. After fusion, the direction vectors are normalized to maintain an effective three-dimensional direction representation, resulting in a corrected contact orientation pattern. .

[0042] After the above-mentioned global contact map correction, fingertip contact map correction, hand area allocation map enhancement, and contact direction map correction, the corrected multi-component interaction representation is obtained: This representation enables the modified multi-component interaction representation generated by the conditional variational autoencoder to further integrate the geometric information of the target object, thereby improving the consistency between the geometry of each interaction component and the target object. The modified multi-component interaction representation serves as constraint information for the subsequent grasping posture recovery module, guiding the grasping posture optimization based on the MANO parameterized hand model and the piecewise symbolic distance function.

[0043] S14: The modified multi-component interaction representation is used as structured constraint information, and a joint optimization objective function is constructed by combining the parameterized hand model and the piecewise symbolic distance function.

[0044] In this step, a Grasp Pose Recovery Module is introduced. This module is a nonlinear optimization solver based on a differentiable parameterized hand model and spatial geometry calculations, used to recover the 3D human hand grasping pose from the corrected multi-component interaction representation. Specifically, the Grasp Pose Recovery Module converts the corrected multi-component interaction representation, defined on the target object surface, obtained in the preceding steps, into the 3D hand pose corresponding to the parameterized hand model. Taking the corrected multi-component interaction representation and the target object point cloud as input, the module constructs and minimizes a joint optimization objective function, outputting the final 3D hand mesh and joint parameters.

[0045] This embodiment uses a MANO (Model with Articulated and Non-rigid Objects) parametric hand model to describe the geometry of the human hand. The MANO model uses low-dimensional pose parameters... (Controlling finger joint rotation) and shape parameters (Control the size and length of the hands).

[0046] To accurately establish the spatial relationship between the object surface and different hand regions, this invention introduces the Segmented Signed Distance Function (SDF). Traditional SDF can only calculate the shortest distance from a point to the entire hand mesh, unable to distinguish which finger it represents. This embodiment utilizes the SDF to establish the spatial relationship between the object surface and different hand regions. For the set of sampled points on the object surface, the grasping solver iteratively optimizes the MANO parameters, gradually ensuring the hand model satisfies the contact relationships, part correspondences, and directional relationships described by the interactive representation, thereby recovering the complete human hand grasping posture.

[0047] Based on this, a joint optimization objective function is constructed. : in, , , and These represent contact constraint terms, penetration constraint terms, direction constraint terms, and parameter regularization terms, respectively. , , and For the corresponding weights.

[0048] (1) Contact constraint terms Used to establish the contact relationship between the object's surface and the corresponding hand area. Based on the corrected global contact map. And the enhanced hand area allocation map The piecewise signed distance function is used to describe the distance relationship between the object's sampling points and different hand regions. Its expression is as follows: in, This represents the signed distance function corresponding to the hand region. This indicates the total number of predefined hand regions. b This represents the hand region index, and For object surface points with high contact values, this constraint drives the distance to the corresponding hand region to gradually approach zero, thereby bringing the hand surface closer to the predicted contact area.

[0049] (2) Directional constraint terms : Used to maintain consistency between the optimized hand pose and the local orientation information in the interaction representation, its expression is as follows: in, This represents the corrected contact pattern. This represents the direction vector obtained from the query corresponding to the hand region. This represents the orientation constraint weights constructed based on contact confidence, used to adjust the contribution of different object sampling points to the orientation constraint term. This constraint ensures that the optimized hand pose remains consistent with the predicted local interaction direction.

[0050] (3) Penetration constraint terms To avoid unreasonable hand-object penetration, a severe penalty is imposed on the sampling points of objects entering the hand model. This penalty is expressed as follows: This constraint penalizes the sampling points of objects that enter the hand model, thereby reducing unreasonable overlap between the hand and objects and improving the spatial rationality of the generated 3D human hand grasping posture.

[0051] (4) Parameter regularization term : Used to constrain MANO parameters, its expression form is: in, Indicates attitude parameters, This represents the shape parameter. This constraint can suppress excessive joint movement and unreasonable hand deformation, ensuring that the generated result conforms to the hand structure constraints expressed by the parametric hand model.

[0052] S15: By minimizing the joint optimization objective function, the posture parameters and shape parameters of the parameterized hand model are iteratively optimized to generate the target 3D human hand grasping posture.

[0053] In practice, the grab solver first updates the global translation and rotation of the hand model, bringing it into an optimizable position near the target object. Then, it updates the hand pose parameters. and shape parameters This process matches each hand region with the contact, location, and orientation information in the interaction representation. After iteration, the output shows the hand grasping result that meets the above optimization objectives.

[0054] Therefore, the human grabbing generation process based on model optimization can be represented as: in, This represents the multi-component interaction representation after geometric guidance correction. This indicates the process of recovering the grasping posture. This represents the final result of the manual grabbing process.

[0055] The revised multi-component interaction representation serves as structured constraint information in the subsequent optimization process, enabling model-based optimization to transform the object-centered interaction representation into a complete human hand grasping posture, thereby obtaining a grasping result that matches the geometry of the target object and meets spatial constraint requirements.

[0056] Overall training and testing process This invention includes a model training phase and a test generation phase. The model training phase is used to train a multi-component interactive representation generation network, and the test generation phase is used to generate the corresponding 3D human hand grasping posture based on the object to be grasped.

[0057] During the model training phase, surface sampling is first performed on the object models in the training set to obtain the point cloud of the target object. Based on the samples captured by the corresponding personnel, a multi-component interactive representation is constructed. The multi-component interaction representation includes a real fingertip contact map, a real global contact map, a real hand region allocation map, and a real contact direction map. The real fingertip contact map and the real global contact map together constitute a dual-contact representation, used to describe the local fingertip contact pattern and the overall hand-object contact range, respectively. The object point cloud and corresponding interaction information are input into a conditional variational autoencoder for training. A geometric guidance correction mechanism is embedded during the decoding and generation of the interaction map, enabling the generative network to learn the mapping relationship between the object's geometric structure and the hand-object interaction representation while improving the consistency between the interaction representation and the target object's geometric structure. During training, the network parameters are jointly optimized through reconstruction loss and latent spatial constraints. in, This represents the reconstruction loss between the generated interaction graph and the actual interaction graph. This represents the distribution constraint term of the latent variable. These are the weight coefficients. After training, an interactive generative network capable of generating diverse interactive representations is obtained.

[0058] During the test generation phase, only the point cloud of the target object to be captured is input. The point cloud of the target object is then used to extract features to obtain a geometric feature representation of the object. The trained generative network samples latent variables from a predefined prior distribution in the latent space and generates fingertip contact maps, global contact maps, hand region allocation maps, and contact direction maps sequentially according to a predefined hierarchical dependency order, thus obtaining multi-component interactive representations. Subsequently, The input geometric constraint correction module corrects the contact probability, hand area allocation relationship, and contact direction to obtain the corrected multi-component interaction representation. .

[0059] Finally, the corrected multi-component interactive representation is input into the grasping posture recovery module. The MANO parametric hand model is used to describe the human hand structure, and a piecewise signed distance function is combined to establish the spatial relationship between the object surface and different hand regions. By jointly optimizing contact constraints, orientation constraints, penetration constraints, and parametric constraints, the final 3D human hand grasping posture is recovered. in, This represents the final result of the manual grabbing process.

[0060] In summary, this invention proposes a method for generating hand grasping representations for 3D objects. It constructs a multi-component interactive representation consisting of a fingertip contact map, a global contact map, a hand region allocation map, and a contact direction map, describing the hand-object interaction relationship from multiple levels, including overall contact range, local contact patterns, hand region correspondence, and local contact direction. Based on this interactive representation, a four-level sequential conditional variational autoencoder is used to establish hierarchical dependencies between different interactive components, enabling the generation of diverse hand-object interactive representations. Furthermore, the generated interactive representation is corrected using the geometric information contained within the target object itself. Through global contact map correction, fingertip contact map correction, hand region allocation map correction, and contact direction map correction, the consistency between the interactive representation and the target object's geometry is improved. Subsequently, the corrected multi-component interactive representation is used as structured constraint information. Combined with the MANO parametric hand model and a piecewise symbolic distance function, the complete hand grasping posture is recovered by jointly optimizing contact constraints, direction constraints, penetration constraints, and parameter constraints. This invention unifies multi-component interactive modeling, geometrically guided interactive representation correction, and model-based posture recovery, achieving the generation of a complete 3D human hand grasping posture from the geometric structure of the target object. By fully utilizing fingertip contact information and the geometric features of the target object, constraint mechanisms are introduced at both the interactive representation and posture recovery levels to form a two-level constraint framework. While maintaining the diversity of grasping results, this improves the geometric consistency and spatial rationality between the generated results and the target object, and can be applied to fields such as robot grasping, virtual reality, human-computer interaction, and digital human generation.

[0061] Example 2 like Figure 3 As shown, the present invention also discloses a 3D hand object grasping and generation system based on fingertip perception and geometric guidance, which includes the following key modules: The feature extraction and generation module 11 is used to acquire the point cloud of the target object, extract features from the point cloud to obtain the geometric feature representation of the object, and input the geometric feature representation of the object into a pre-trained conditional variational autoencoder. The conditional variational autoencoder generates multi-component interactive representations step by step according to a preset hierarchical dependency order. The multi-component interactive representations include a fingertip contact map, a global contact map, a hand region allocation map, and a contact direction map. The geometric guidance correction module 12 is used to correct the fingertip contact map, the global contact map, the hand region allocation map and the contact direction map respectively based on the geometric information contained in the target object point cloud, so as to obtain the corrected multi-component interactive representation. The posture optimization and recovery module 13 is used to take the corrected multi-component interaction representation as structured constraint information, and construct a joint optimization objective function by combining the parameterized hand model and the piecewise symbolic distance function; by minimizing the joint optimization objective function, iteratively optimize the posture parameters and shape parameters of the parameterized hand model to generate the target three-dimensional human hand grasping posture.

[0062] Example 3 Furthermore, embodiments of this application also disclose an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0063] Figure 4 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the 3D hand object grasping generation method based on fingertip perception and geometric guidance disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be a computer.

[0064] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0065] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0066] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0067] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the data 223 in the memory 22. It can be Windows Server, Netware, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing a 3D hand-grabbing generation method based on fingertip perception and geometric guidance, as disclosed in any of the foregoing embodiments and executed by the electronic device 20, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0068] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed method for generating 3D hand objects based on fingertip perception and geometric guidance. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0069] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0070] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly in hardware, software modules executed by a processor, or a combination of both. The software module may be located in random access memory (RAM), memory, read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs (Compact Disc-Read Only Memory), or any other form of storage medium known in the art.

[0071] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0072] The solution provided by the present invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A 3D hand object grasping and generation method based on fingertip perception and geometric guidance, characterized in that, Includes the following steps: Obtain the point cloud of the target object, and perform feature extraction on the point cloud of the target object to obtain the geometric feature representation of the object; The geometric feature representation of the object is input into a pre-trained conditional variational autoencoder, which generates multi-component interactive representations step by step according to a preset hierarchical dependency order. The multi-component interactive representations include fingertip contact map, global contact map, hand region allocation map, and contact direction map. Based on the geometric information contained in the point cloud of the target object, the fingertip contact map, the global contact map, the hand region allocation map, and the contact direction map are corrected respectively to obtain a corrected multi-component interactive representation; The modified multi-component interaction representation is used as structured constraint information, and a joint optimization objective function is constructed by combining the parameterized hand model and the piecewise symbolic distance function. By minimizing the joint optimization objective function, the pose and shape parameters of the parameterized hand model are iteratively optimized to generate the target 3D human hand grasping pose.

2. The method according to claim 1, characterized in that, The step of generating multi-component interactive representations step by step according to a preset hierarchical dependency order using the conditional variational autoencoder includes: The fingertip contact map is generated based on the geometric feature representation of the object and latent variables sampled from the latent space; the fingertip contact map is used to represent the probability that each sampled point in the point cloud of the target object is touched by a fingertip; The global contact map is generated by combining the fingertip contact map with the geometric features of the object; the global contact map is used to represent the probability that each sampling point in the point cloud of the target object is touched by the hand as a whole. The hand region allocation map is generated by combining the fingertip contact map and the global contact map; the hand region allocation map is used to represent the probability of each sampling point in the target object point cloud corresponding to a predefined hand region; The contact direction map is generated by combining the fingertip contact map, the global contact map, and the hand region allocation map; the contact direction map is used to represent the local hand-object contact direction vector corresponding to each sampling point in the target object point cloud.

3. The method according to claim 1, characterized in that, Based on the geometric information contained in the point cloud of the target object, the fingertip contact map is corrected, including: A local density prior is constructed based on the neighborhood distribution of the target object's point cloud. The local density prior is then weighted and fused with the fingertip contact map to obtain the corrected fingertip contact map.

4. The method according to claim 3, characterized in that, Based on the geometric information contained in the point cloud of the target object, the global contact map is corrected, including: Construct an axis-aligned bounding box based on the spatial distribution of the target object's point cloud, and calculate the object's geometric center within the axis-aligned bounding box; A distance prior is constructed based on the distance relationship between each sampling point in the target object point cloud and the geometric center of the object. The distance prior is then weighted and fused with the global contact map to obtain a corrected global contact map.

5. The method according to claim 4, characterized in that, Based on the geometric information contained in the point cloud of the target object, the hand region allocation map is corrected, including: Based on the corrected global contact map and the corrected fingertip contact map, the contact intensity of each sampling point in the target object point cloud is calculated; the contact intensity is calculated by combining the corrected global contact map and the corrected fingertip contact map. The hand region allocation map is weighted using the contact intensity to increase the confidence of the hand region allocation results corresponding to the contact intensity region, thus obtaining an enhanced hand region allocation map.

6. The method according to claim 5, characterized in that, Based on the geometric information contained in the point cloud of the target object, the contact orientation pattern is corrected, including: For each sampling point in the point cloud of the target object, the mean value of the direction vector within its neighborhood is calculated to obtain the mean value of the neighborhood direction. The network-predicted direction and the mean of the neighborhood direction in the contact direction map are weighted and fused using the contact intensity; The weighted and fused direction vectors are normalized to obtain the corrected contact pattern.

7. The method according to claim 1, characterized in that, The joint optimization objective function includes contact constraint terms, penetration constraint terms, direction constraint terms, and parameter regularization terms; The contact constraint term is used to establish the contact relationship between the target object point cloud and the corresponding hand region; The penetration constraint term is used to apply a penetration penalty to object sampling points located inside the parameterized hand model; The orientation constraint term is used to keep the optimized hand posture consistent with the local orientation information in the modified multi-component interaction representation; The parameter regularization term is used to constrain the posture and shape parameters of the parameterized hand model to suppress excessive joint movement and unreasonable hand deformation.

8. A 3D hand-grasping and object generation system based on fingertip perception and geometric guidance, characterized in that, include: The feature extraction and generation module is used to acquire the point cloud of the target object and perform feature extraction on the point cloud of the target object to obtain the geometric feature representation of the object. The geometric feature representation of the object is input into a pre-trained conditional variational autoencoder, which generates multi-component interactive representations step by step according to a preset hierarchical dependency order. The multi-component interactive representations include fingertip contact map, global contact map, hand region allocation map, and contact direction map. The geometric guidance correction module is used to correct the fingertip contact map, the global contact map, the hand region allocation map, and the contact direction map respectively based on the geometric information contained in the target object point cloud, so as to obtain the corrected multi-component interactive representation; The posture optimization and recovery module is used to take the corrected multi-component interaction representation as structured constraint information, and combine it with the parameterized hand model and the piecewise symbolic distance function to construct a joint optimization objective function; by minimizing the joint optimization objective function, the posture parameters and shape parameters of the parameterized hand model are iteratively optimized to generate the target 3D human hand grasping posture.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of a 3D hand object grasping generation method based on fingertip perception and geometric guidance as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the steps of a 3D hand object grasping generation method based on fingertip perception and geometric guidance as described in any one of claims 1 to 7.

Images