Multi-size object-oriented dual-hand dexterous grasping data synthesis and grasping generation method and system and robot

Abstract

The application discloses a kind of dual-hand dexterous grasping data synthesis and grasping generation method, system and robot for multi-size object.Data synthesis method is: local grasping area is constructed on object surface, target area pair is determined and posture is initialized after geometric screening and mechanical stability evaluation;Decoupling optimization objective function including local force closure, contact distance and area consistency term is constructed, and left and right hand postures are decoupled and optimized;Feasible sample is screened and motion trajectory is generated by collision detection and inverse kinematics solution, and large-scale data set is constructed.Grasping generation method is: based on object point cloud cooperative prediction master-slave grasping perspective;Scale self-adaptive grasping anchor point is determined;Local features and global features are fused to construct conditional features;Generation model is used to generate dual-hand coordinated grasping posture in the form of pose increment relative to anchor point.The application significantly improves data generation efficiency and quality, and enhances the generalization ability of model to multi-size object and the executability of grasping.

Description

Technical Field

[0001] This invention belongs to the technical field of robot operation, specifically relating to a method, system, and robot for dexterous two-handed grasping of objects of various sizes, involving data synthesis and grasping generation. Background Technology

[0002] Dexterous grasping is a key foundational technology in the field of robotic manipulation, aiming to achieve stable and precise manipulation of real objects through multi-degree-of-freedom manipulators. While single-handed dexterous grasping has made some progress in handling small to medium-sized objects, it faces significant limitations in stability, force-bearing capacity, and operational flexibility when dealing with larger objects, objects with complex geometries, or objects requiring bilateral support. Therefore, dual-handed dexterous grasping has become an important development direction in the field of robotic manipulation.

[0003] Current research on bimanual dexterity grasping largely follows the technical paradigm of single-handed grasping, either directly extending single-handed grasping models to bimanual scenarios or relying on reinforcement learning and imitation learning to generate bimanual grasping strategies. However, bimanual systems significantly increase the degrees of freedom, and the grasping process must simultaneously ensure single-handed grip stability, bimanual spatial coordination, mechanical synergy, and kinematic accessibility constraints. This limits existing methods in terms of data generation efficiency, grasping stability, and adaptability to different object geometries and sizes.

[0004] Especially in data-driven methods, the lack of high-quality bimanual grasping data has become a core bottleneck for improving model performance. Existing bimanual grasping datasets are limited in size and have a narrow range of object sizes. Furthermore, data generation suffers from low initialization efficiency, high optimization difficulty, and difficulty in ensuring executability with real robots, making it impossible to support the training of highly generalized bimanual dexterity grasping models. Specific shortcomings include:

[0005] During the initialization phase of two-handed grasping, independently sampling the grasping points / poses of the left and right hands will lead to a quadratic increase in the number of candidate grasping pairs, resulting in a lot of invalid computation and extremely low data generation efficiency.

[0006] In the optimization stage of two-handed grasping, the left and right hands are placed in the same high-dimensional optimization space to perform joint force closure optimization. The two are highly coupled, which makes the optimization process complex, difficult to converge, and difficult to obtain high-quality grasping results.

[0007] In the grasping generation stage, the existing model directly predicts the absolute grasping posture of both hands, without explicitly modeling the coordination relationship between the two hands, and does not fully consider the influence of changes in the geometry and size of the object. The model has insufficient generalization ability for unseen objects.

[0008] The existing grasping data and generated results lack systematic constraints on the kinematic reachability of real robotic arms, and the generated grasping postures are difficult to execute directly in real robot systems.

[0009] In a high-dimensional two-handed grasping action space, weak constraints or low-quality data can lead to learning difficulties and poor convergence of generative models, and can easily cause physical infeasibility problems such as hand-object interpenetration and hand-hand self-collision. Summary of the Invention

[0010] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method, system, and robot for the synthesis and generation of data for dexterous two-handed grasping of objects of various sizes. This invention adopts a collaborative design approach at the data and model layers, structurally reduces the complexity of the search space, explicitly models the coordination relationship between the two hands, achieves efficient synthesis of high-quality two-handed grasping data, and provides stable grasping generation for objects of various sizes and complex geometries, while ensuring the executability of the grasping results in a real robot system.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] In a first aspect, the present invention provides a method for data synthesis of dexterous two-handed grasping of objects of multiple sizes, comprising the following steps:

[0013] Using the three-dimensional mesh model of the object as input, the farthest point sampling is used to select the reference point on the surface of the object and construct a local grasping area. After eliminating unstable contact areas, the candidate area pairs are geometrically screened. Force spinor space is introduced as a region stability evaluation index to select the candidate area pair with the highest score. Based on the area pair, the initial grasping posture of both hands is generated. The initial grasping posture of both hands includes the spatial translation and rotation parameters of the left and right dexterous wrists and the multi-degree-of-freedom joint configuration of the dexterous hands.

[0014] An optimization objective function is constructed with the grasping posture of both hands as the variable. The force closure objective of grasping with both hands is decoupled into the local force closure energy of each hand. At the same time, contact distance constraints and region consistency constraints are incorporated. During the optimization process, joint angle, self-collision, inter-hand collision and hand-object collision constraints are applied to obtain a physically stable and coordinated grasping posture of both hands.

[0015] Collision detection is performed on the optimized grasping posture, and solutions with collisions or penetrations are eliminated. Inverse kinematics are solved for the left and right robotic arms respectively to verify the reachability of wrist pose. Samples with collision-free inverse solutions are discarded. Pre-grasping and execution grasping postures are constructed for valid samples and continuous motion trajectories are generated by interpolation.

[0016] Based on the grasping data generated by the above steps, a two-handed dexterous grasping dataset covering objects of multiple sizes and geometric shapes is constructed. Each object is discretely scaled to multiple sizes, and large-scale two-handed dexterous grasping samples are synthesized in a GPU parallel simulation environment.

[0017] As a preferred technical solution, each two-handed grasping sample is defined as a combination of left and right hand grasping postures:

[0018]

[0019] in These represent the spatial translation and rotation parameters of the dexterous wrist, respectively. This indicates the multi-degree-of-freedom joint configuration of a dexterous hand.

[0020] As a preferred technical solution, the regional stability score of the force spinor space is calculated by projecting the force spinor boundary onto multiple orthogonal perturbation directions, taking the minimum projection value as the lower bound of the stability of the regional pair, and the calculation formula is as follows:

[0021]

[0022] in Denotes the first digit used to approximate the closure boundary. A force spinor, Indicates the first A predefined perturbation direction.

[0023] As a preferred technical solution, the objective function is:

[0024]

[0025] The energy of local force closure , It is used to measure the stable support ability of an object by a single hand, and is defined as:

[0026]

[0027] in This represents the force spinor perturbation along the six orthogonal force directions. This indicates a grasping contact model. This represents the contact force within the friction cone at the gripping point.

[0028] Secondly, this invention provides a method for generating two-handed dexterous grasping data for objects of various sizes. The method involves training a model and generating grasping data based on a two-handed dexterous grasping dataset constructed using this method, and includes the following steps:

[0029] Acquire point cloud data of the object to be grasped;

[0030] Cooperative prediction of gripping perspectives for both hands: Using the minimum bounding sphere of the object point cloud as a reference, a set of candidate gripping perspectives is constructed; based on the global geometric features of the object point cloud, the primary gripping perspective is predicted, and under the condition of the primary gripping perspective, a secondary gripping perspective that is coordinated with it is predicted, forming a coordinated gripping perspective combination for both hands.

[0031] Scale-adaptive gripping anchor point generation: Based on the geometric relationship between the gripping viewpoint and the object's minimum bounding sphere, the gripping anchor points for the left and right hands are determined, where the gripping anchor point is the intersection of the gripping viewpoint and the surface of the bounding sphere.

[0032] Grasping condition feature construction: Based on the global features of the point cloud, a set of point-level feature carriers are upsampled; taking the grasping anchor point as a spatial reference, local geometric features near the anchor point are extracted; the local geometric features, grasping viewpoint features and global geometric features of the object are fused to form a unified grasping condition feature.

[0033] Relative two-handed grasping generation: With the grasping condition features as constraints, a generation model is used to progressively sample and denoise the relative wrist poses and dexterous hand joint configurations of the left and right hands, and output the final two-handed dexterous grasping posture.

[0034] The relative wrist pose is described as the relative pose increment relative to the grasping anchor point, rather than the absolute pose in the object coordinate system.

[0035] As a preferred technical solution, the dual-gripping perspective collaborative prediction specifically includes:

[0036] Define the grabbing viewpoint in a scale-independent geometric space, where each grabbing viewpoint represents a direction of approach from the surface of the enclosing sphere toward the center of the object;

[0037] The system utilizes the statistical distribution of successfully crawled samples in the dataset for supervision. Through probabilistic modeling, it first predicts the primary crawling perspective most likely to generate stable crawls, and then predicts the secondary crawling perspective with the best synergy with the primary crawling perspective.

[0038] As a preferred technical solution, the method for extracting local geometric features in the construction of the grasping condition features is as follows:

[0039] Using the captured anchor point as a spatial reference, its nearest neighbor points are searched in the point cloud features, and local geometric features near the anchor point are extracted by K-nearest neighbor aggregation. The local geometric features include the curvature, normal change and local shape information of the potential contact area.

[0040] The crawling condition features Represented as:

[0041]

[0042] in For local geometric features, This is a view embedding representation derived from the view prediction process. This refers to the global geometric features of an object.

[0043] As a preferred technical solution, the generative model is any one of the following: a diffusion-based generative model, an autoregressive generative model, a conditional variational generative model, or an energy function-based generative model.

[0044] When using a generative model based on diffusion processes, the relative wrist poses and multi-degree-of-freedom joint configurations of the left and right hands are gradually denoised to obtain grasping results that satisfy hand coordination, geometric constraints, and physical rationality.

[0045] Thirdly, the present invention provides a two-handed dexterous grasping generation system for multi-size objects, which is applied to the two-handed dexterous grasping generation for multi-size objects, including a data acquisition module, a two-handed grasping perspective collaborative prediction module, a scale-adaptive grasping anchor point generation module, a grasping condition feature construction module, and a relative two-handed grasping generation module.

[0046] The data acquisition module is used to acquire point cloud data of the object to be captured;

[0047] The dual-handed grasping perspective collaborative prediction module is used to construct a set of candidate grasping perspectives with the minimum bounding sphere of the object point cloud as a reference; based on the global geometric features of the object point cloud, it predicts the main grasping perspective, and under the condition of the main grasping perspective, it predicts the second grasping perspective that is coordinated with it, forming a grasping perspective combination that is coordinated between the left and right hands.

[0048] The scale-adaptive grasping anchor point generation module is used to determine the grasping anchor points of the left and right hands based on the geometric relationship between the grasping viewpoint and the minimum bounding sphere of the object. The grasping anchor point is the intersection position of the grasping viewpoint and the surface of the bounding sphere.

[0049] The grasping condition feature construction module is used to upsample a set of point-level feature carriers based on the global features of the point cloud; extract local geometric features near the grasping anchor point as a spatial reference; and fuse the local geometric features, grasping viewpoint features, and global geometric features of the object to form a unified grasping condition feature.

[0050] The relative two-handed grasping generation module is used to progressively sample and denoise the relative wrist pose and dexterous hand joint configuration of the left and right hands using the grasping condition features as constraints, and output the final two-handed dexterous grasping posture.

[0051] The relative wrist pose is described as the relative pose increment relative to the grasping anchor point, rather than the absolute pose in the object coordinate system.

[0052] Fourthly, the present invention provides a robot, the robot comprising:

[0053] At least one processor; and,

[0054] A memory communicatively connected to the at least one processor; wherein,

[0055] The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the aforementioned method for generating a two-handed dexterous grasping mechanism for multi-sized objects.

[0056] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0057] 1. This invention overcomes the problems of low initialization efficiency and high proportion of invalid samples in traditional bi-handed grasping data synthesis;

[0058] Existing bi-handed grasping data generation methods are mostly direct extensions of single-handed grasping, typically initializing data by randomly sampling contact point pairs or grasping pose pairs on the object surface. In bi-handed scenarios, this leads to a quadratic increase in the search space, making large-scale initialization geometrically or mechanically infeasible, severely limiting data generation efficiency and quality.

[0059] This invention introduces a region-constraint-based bi-handed grasping initialization strategy during the data synthesis stage. By first selecting regions with stable grasping potential on the object surface, and then generating the initial bi-handed configuration within these regions, a large number of invalid grasping pairs are avoided from the outset. Experimental results show that the baseline method without region constraints achieves a success rate (SUC-L) of only 28.33% in desktop bi-handed grasping tasks, while the grasping success rate is significantly improved after introducing region constraints, laying a reliable foundation for subsequent optimization and execution.

[0060] 2. This invention avoids the instability and imbalance problems caused by the strong coupling of the objective function in the optimization of two-handed grasping;

[0061] In traditional two-handed grasping optimization, both hands are usually included in a unified force-closed objective function for joint optimization. This strong coupling can easily lead to instability in the optimization process, often resulting in one hand bearing the main support while the other hand contributes insufficiently, thus affecting the overall grasping quality.

[0062] To address this issue, this invention proposes a decoupled force-closure grasping optimization strategy. This strategy breaks down the two-handed grasping target into relatively independent optimization items for each hand, supplemented by region consistency and collision constraints. This significantly reduces the optimization difficulty while maintaining hand-hand coordination. Experimental results show that after gradually introducing the decoupled optimization and execution strategy, the desktop grasping success rate increased from 57.66% to 80.83%, indicating that this strategy effectively improves the quality of grasping optimization.

[0063] 3. This invention solves the problem that it is difficult to simultaneously satisfy physical stability and robotic arm reachability when grasping data with both hands;

[0064] Existing bi-handed grasping data often only focuses on the grasping stability of the floating hand, ignoring the kinematic constraints of the actual robotic arm execution, resulting in a large number of synthesized grasping configurations being unexecutable in real systems. This invention introduces an accessibility verification and filtering mechanism based on the inverse kinematics of the robotic arm into the data synthesis pipeline, eliminating unreachable grasping configurations during the data generation stage.

[0065] 4. This invention overcomes the problem of learning difficulties in high-dimensional bimanual grasping action space for generative models;

[0066] Dexterous grasping involves hand positions and high-degree-of-freedom joint configurations. Its motion space has high dimensions and complex distribution. If weak constraints or low-quality data are used directly for learning, the generated model often has difficulty converging or has insufficient generalization ability.

[0067] This invention constructs a large-scale, structurally consistent two-handed grasping dataset through the aforementioned high-quality data synthesis pipeline, enabling the generative model to effectively learn the inherent rules of two-handed collaborative grasping during training. Experimental results show that the generative model trained based on the data of this invention achieves a success rate (SUC-L) of 66.78% in desktop grasping tasks, significantly outperforming existing methods trained based on single-handed extended or weakly constrained data.

[0068] 5. This invention avoids the common problems of hand-object interleaving and hand-hand self-collision in the generation and grasping process;

[0069] Due to a lack of physical constraints or insufficient data quality, existing biphasic grasping generation models often suffer from problems such as finger interlacing, excessive hand-object embedding, or hand-object self-collision. This invention introduces strict collision detection and physical consistency constraints during the data synthesis stage, ensuring that the training data itself possesses good physical properties.

[0070] Correspondingly, the grasping mechanism generated by the generative model during the inference phase significantly outperforms the comparative methods in terms of object penetration depth (PD), hand self-penetration depth (SPD), and contact distance consistency (CDC), indicating that the technical solution can stably generate physically feasible and contact-balanced two-hand grasping configurations.

[0071] 6. This invention achieves an effective transfer from high-quality data synthesis to real two-handed robot execution;

[0072] The bi-handed grasping generation model, trained using data generated by the data synthesis pipeline of this invention, has been validated on various real objects and different bi-handed platforms. Experimental results show that the model can still generate stable and executable bi-handed grasping even when faced with unseen objects and multi-scale scenes, verifying the feasibility and robustness of the technical solution of this invention in real-world applications. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 This is a schematic diagram of a data synthesis method for dexterous two-handed grasping of objects of multiple sizes according to an embodiment of the present invention;

[0075] Figure 2 This is a diagram illustrating the architecture of a two-handed dexterous grasping model for objects of various sizes, as described in an embodiment of the present invention.

[0076] Figure 3 This is a block diagram of a bi-handed dexterous grasping generation system for multi-sized objects according to an embodiment of the present invention;

[0077] Figure 4 This is a structural diagram of the robot according to an embodiment of the present invention. Detailed Implementation

[0078] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0079] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0080] like Figure 1As shown, this embodiment provides a method for synthesizing data on two-handed dexterous grasping of objects of various sizes. Based on a region constraint and decoupling optimization pipeline, it aims to construct a high-quality dataset capable of supporting the training of a two-handed dexterous grasping generation model. The method includes the following steps:

[0081] (1) Using the three-dimensional mesh model of the object as input, through steps such as region constraint initialization, decoupled force closure optimization and kinematic feasibility verification, the dexterous gripping data of both hands that meet the physical stability and reachability of the robotic arm is systematically generated.

[0082] In terms of data representation, each two-handed grasping sample is defined as a combination of the grasping postures of the left and right hands:

[0083]

[0084] in These represent the spatial translation and rotation parameters of the dexterous wrist, respectively. This represents the multi-degree-of-freedom joint configuration of a dexterous hand. This unified representation provides a standardized foundation for subsequent grasping optimization, filtering, and learning modeling.

[0085] (2) Initialization of two-handed grasping based on region constraints;

[0086] To address the problems of rapidly expanding search space and extremely low random sampling efficiency during the initialization phase of bi-handed grasping, this invention proposes a bi-handed grasping initialization strategy based on region constraints. By pre-selecting regions with stable grasping potential on the object surface, the strategy guides the generation of subsequent grasping postures, thereby avoiding a large number of physically infeasible or geometrically unreasonable initial solutions.

[0087] Specifically, firstly, several reference points are selected on the object's surface using the farthest-point sampling method, and a local grasping area is constructed within the geodesic neighborhood of each reference point. To ensure the physical rationality of the grasping, areas with severe concavity that could lead to unstable contact are systematically eliminated. Subsequently, all candidate area pairs are geometrically screened to remove combinations where the area spacing is too close to accommodate coordinated two-handed operation.

[0088] Building upon this, the present invention further introduces the Grasp Wrench Space (GWS) as a regional stability evaluation index. For each candidate region pair, several contact points are sampled within the region, and its resistance to external disturbances is estimated using the Grasp Wrench boundary approximation method. Specifically, the Grasp Wrench boundary is projected onto multiple orthogonal disturbance directions, and the minimum projection value is taken as the lower bound of the stability of the region pair.

[0089]

[0090] in This represents the estimated boundary force spinor. This indicates a predefined perturbation direction. Finally, the regions are sorted according to their stability scores, and the region with the highest score is selected as the candidate region for the initialization of the two-handed grasping motion.

[0091] Based on the aforementioned high-quality region pairs, this invention generates corresponding initial grasping postures for both hands. The initial posture of each hand is determined by the palm position, palm orientation, and planar rotation angle around the palm normal. The finger joint angles are initialized based on the local curvature of the region, so that the hand can form a reasonable fit with the object surface in the initial state, thereby providing high-quality initial values ​​for subsequent optimization.

[0092] (3) Decoupled force-closed grasping optimization;

[0093] After obtaining the initial solution for two-handed grasping under regional constraints, this invention proposes a decoupled force-closed grasping optimization method to further optimize the initial grasping posture into a physically stable and coordinated high-quality grasping result.

[0094] Unlike traditional methods that treat both hands as a whole for joint optimization, this invention decouples the force closing target of the grasping by both hands, allowing each hand to focus on the stability of its local contact, thereby reducing the coupling complexity of the optimization problem and avoiding an unbalanced grasping mode where one hand dominates and the other hand contributes insufficiently.

[0095] Specifically, an optimization objective function is constructed with the grasping posture of both hands as the variable. Its energy term is composed of the local force closure energy of each hand, the contact distance constraint, and the region consistency constraint.

[0096]

[0097] The energy of local force closure , It is used to measure the stable support ability of an object by a single hand, and is defined as follows:

[0098]

[0099] in This represents the force spinor perturbation along the six orthogonal force directions. This indicates a grasping contact model. This represents the contact force within the friction cone at the gripping point. This energy is determined by a quadratic programming problem, where, for a given gripping point, the optimal contact force capable of resisting spinor perturbations in six orthogonal directions is solved. The contact distance term constrains the degree of contact between the hand contact point and the object surface, while the region consistency term ensures that the contact point remains near the gripping area selected in the initialization phase throughout the optimization process, thereby maintaining the overall structure of the coordinated two-handed gripping.

[0100] During the optimization process, joint angle constraints, self-collision constraints, inter-hand collision constraints, and hand-object collision constraints are applied simultaneously to ensure that the obtained grasping solution is executable at both the physical and geometric levels. Through the above decoupled optimization strategy, the grasping quality of both hands can be significantly improved while maintaining optimization stability.

[0101] (4) Feasibility screening and configuration generation of kinematically perceived two-handed grasping;

[0102] To ensure that the generated bimanual grasping data can be directly used in real robotic systems, this invention introduces kinematics-based feasibility verification and configuration generation steps after grasping optimization. Specifically, collision detection is first performed on each optimized bimanual grasping posture to eliminate solutions involving hand self-collisions, hand-to-hand collisions, or severe penetration by the object. Subsequently, inverse kinematics are solved for each of the left and right robotic arms to verify the reachability of the wrist pose within the robotic arm's workspace. Grasping samples for which collision-free inverse solutions cannot be obtained are discarded. After the above screening, this invention further constructs a pre-grasping posture and an execution grasping posture for each valid grasping sample, generating a continuous motion trajectory from pre-grasping to clamping through interpolation. This ensures that the generated data is valid not only in a static geometric sense but also satisfies the kinematic and dynamic constraints of a real robot during execution.

[0103] Based on the aforementioned bimanual dexterous grasping data generation pipeline, this invention systematically constructs a large-scale, high-quality bimanual dexterous grasping dataset. This dataset covers complex 3D object models from multiple sources, containing a total of 6351 different object instances, including 2397 objects from DexGraspNet and 3954 objects from Objaverse. To enhance the data's adaptability to changes in object size, each object is discretely scaled to 11 different sizes, ranging from 30 cm to 80 cm, thus significantly expanding the object scale distribution. Based on the aforementioned object set and scale settings, the data generation pipeline automatically synthesized 9.53 million bimanual dexterous grasping samples in a GPU parallel simulation environment. This dataset provides a solid data foundation for the subsequent training and evaluation of bimanual dexterous grasping generation models, and also possesses good generalization ability and practical value.

[0104] In another embodiment of this application, based on the aforementioned dexterous gripping dataset, this invention proposes a dexterous gripping generation method based on gripping perspective, used to directly generate a dexterous gripping posture with left and right hands working together from object point cloud data, such as... Figure 2 As shown, the specific steps include the following:

[0105] S1. Obtain the point cloud data of the object to be grasped;

[0106] Using a large number of successful two-handed grasping samples in the dataset as a priori, stable grasping generation of objects with different geometric shapes and sizes is achieved by explicitly modeling the relationship between the two-handed grasping perspective, object geometry, and relative grasping pose.

[0107] Specifically, the generative model uses object point clouds. As input, output a pair of dexterous grasping postures for the left and right hands. The grasping posture of each hand includes the wrist pose and the configuration of the multi-degree-of-freedom dexterous hand joints. To reduce the search space for grasping with both hands and enhance the stability of generation, the model does not directly regress the absolute grasping pose in 3D space, but introduces the grasping viewpoint and grasping anchor point as intermediate representations to impose structured constraints on the grasping generation process.

[0108] S2, Two-handed grasping perspective collaborative prediction, specifically including:

[0109] First, the model defines the grasping perspective in a scale-independent geometric space. Specifically, using the smallest bounding sphere of the object's point cloud as a reference, multiple discrete approach directions are uniformly sampled on the surface of the bounding sphere to form a set of candidate grasping perspectives. Each grasping perspective represents an approach direction from the surface of the bounding sphere towards the center of the object, used to characterize the overall spatial tendency of the dexterous hand approaching the object. Based on the global geometric features extracted from the object's point cloud, the model performs probabilistic modeling on the above candidate grasping perspectives. First, it predicts the primary grasping perspective most likely to produce a stable grasp, and then, under the condition of the primary grasping perspective, it predicts the secondary grasping perspective with optimal coordination with it, thus forming a pair of grasping perspectives that coordinate with both hands. This perspective prediction process is supervised by the statistical distribution of successful grasping samples in the dataset, enabling the model to learn the global coordination pattern of two-handed grasping during the training phase.

[0110] S3, Scale-adaptive anchor point generation;

[0111] After obtaining the two-handed grasping viewpoints, the model further introduces scale-adaptive grasping anchor points to constrain the generation of grasping pose. These anchor points are determined by the geometric relationship between the grasping viewpoints and the object's minimum bounding sphere; specifically, the position where each grasping viewpoint intersects the surface of the bounding sphere is defined as the corresponding hand's grasping anchor point. Based on these anchor points, the wrist pose is no longer predicted as an absolute pose in the object coordinate system, but rather as a relative pose relative to the grasping anchor point.

[0112]

[0113] in These represent the translation and rotation parameters of the anchor point, respectively. This represents the relative pose increment predicted by the model. Through this method, the grasping generation process automatically adapts to changes in object size, effectively reducing the grasping search space and improving the model's generalization ability to objects of different scales.

[0114] S4. Scale-adaptive anchor point generation;

[0115] In the grasping condition feature construction stage, this invention does not perform coordinate transformation or re-encode the original point cloud. Instead, it adopts a local feature aggregation strategy based on grasping anchor points. Specifically, firstly, based on the global features of the point cloud, a set of representative points are upsampled from the global point cloud using a point cloud decoder as point-level feature carriers, and corresponding decoder features are generated. Subsequently, using the grasping anchor point as a spatial reference, its nearest neighbor points are searched in the point cloud features, and local geometric features near the anchor point are extracted using the K-nearest neighbor aggregation method. These local features can accurately characterize the curvature, normal changes, and local shape information of the potential contact area, thereby avoiding the global features being dominated by object structures unrelated to grasping. Finally, the local geometric features, grasping viewpoint features, and global geometric features of the object are fused to form a unified grasping condition feature representation.

[0116]

[0117] The grasping viewpoint features are directly derived from the viewpoint embedding representation in the viewpoint prediction process, which makes the grasping condition features consistent with the grasping proximity direction at the geometric level.

[0118] S5. Generation of relative two-handed grasping based on diffusion process;

[0119] Under the constraints of the aforementioned grasping conditions, the model employs a relatively dexterous two-handed grasping generation method based on a diffusion process to jointly generate the two-handed grasping posture. Specifically, the model uses... As conditional feature inputs, the relative wrist poses and dexterity hand joint configurations of the left and right hands are progressively sampled and denoised to obtain grasping results that satisfy bimanual coordination, geometric constraints, and physical rationality. Since the generation process is always carried out under the joint constraints of the grasping perspective and the grasping anchor point, the generated bimanual grasping postures are more concentrated in spatial distribution and better match the statistical characteristics of real successful grasping in the dataset.

[0120] It is understood that, in the generative model part, this invention adopts a conditionally generated bi-handed grasping prediction method. To enhance the flexibility and generalization ability of the model structure, the model structure is not limited to a specific network form, and its alternatives include, but are not limited to: replacing the diffusion generation model with an autoregressive generation model, a conditional variational generation model, or an energy function-based generation model; replacing the relative representation of the grasping pose with an equivalent coordinate system expression or parameterized form; and replacing the joint generation process of bi-handed grasping with a staged generation method, such as first generating the master hand grasping and then generating the co-hand grasping.

[0121] Through the above technical solution, the present invention realizes a data-driven, perspective-constrained and geometrically adaptive two-handed dexterous grasping generation model, which can improve the stability and success rate of grasping and generating complex geometric objects and multi-scale objects while ensuring hand-hand coordination.

[0122] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0123] Based on the same idea as the two-handed dexterous grasping generation method for multi-sized objects in the above embodiments, the present invention also provides a two-handed dexterous grasping generation system for multi-sized objects. This system can be used to execute the above-described two-handed dexterous grasping generation method for multi-sized objects. For ease of explanation, the structural schematic diagram of the embodiment of the two-handed dexterous grasping generation system for multi-sized objects only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0124] Please see Figure 3 In another embodiment of this application, a dual-handed dexterous grasping generation system 100 for multi-size objects is provided. The system includes a data acquisition module 101, a dual-handed grasping perspective collaborative prediction module 102, a scale-adaptive grasping anchor point generation module 103, a grasping condition feature construction module 104, and a relative dual-handed grasping generation module 105.

[0125] The data acquisition module 101 is used to acquire point cloud data of the object to be captured;

[0126] The dual-hand grasping perspective collaborative prediction module 102 is used to construct a candidate grasping perspective set with the minimum bounding sphere of the object point cloud as a reference; predict the main grasping perspective based on the global geometric features of the object point cloud; and predict the second grasping perspective that is coordinated with the main grasping perspective under the condition of the main grasping perspective, so as to form a grasping perspective combination that is coordinated with the left and right hands.

[0127] The scale-adaptive gripping anchor point generation module 103 is used to determine the gripping anchor points of the left and right hands based on the geometric relationship between the gripping viewpoint and the minimum bounding sphere of the object. The gripping anchor point is the intersection position of the gripping viewpoint and the surface of the bounding sphere.

[0128] The grasping condition feature construction module 104 is used to upsample a set of point-level feature carriers based on the global features of the point cloud; extract local geometric features near the grasping anchor point as a spatial reference; and fuse the local geometric features, grasping view features and global geometric features of the object to form a unified grasping condition feature.

[0129] The relative two-handed grasping generation module 105 is used to use the grasping condition features as constraints, and employ a generation model to progressively sample and denoise the relative wrist pose and dexterous hand joint configuration of the left and right hands, and output the final two-handed dexterous grasping posture.

[0130] The relative wrist pose is described as the relative pose increment relative to the grasping anchor point, rather than the absolute pose in the object coordinate system.

[0131] It should be noted that the two-handed dexterous grasping generation system for multi-sized objects of the present invention corresponds one-to-one with the two-handed dexterous grasping generation method for multi-sized objects of the present invention. The technical features and beneficial effects described in the embodiments of the two-handed dexterous grasping generation method for multi-sized objects described above are applicable to the embodiments of two-handed dexterous grasping generation for multi-sized objects. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0132] Furthermore, in the above embodiments of the dual-handed dexterous grasping generation system for multi-sized objects, the logical division of each program module is merely illustrative. In practical applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the dual-handed dexterous grasping generation system for multi-sized objects can be divided into different program modules to complete all or part of the functions described above.

[0133] Please see Figure 4In one embodiment, a robot is provided that implements a method for generating a dexterous gripping mechanism for objects of multiple sizes. The robot 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a dexterous gripping mechanism for generating a mechanism for objects of multiple sizes 203.

[0134] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the robot 200, such as the robot 200's portable hard drive. In other embodiments, the first memory 202 can also be an external storage device of the robot 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the robot 200. Furthermore, the first memory 202 can include both internal storage units and external storage devices of the robot 200. The first memory 202 can be used not only to store application software and various types of data installed on the robot 200, such as the code for a dexterous two-handed grasping program 203 for multi-sized objects, but also to temporarily store data that has been output or will be output.

[0135] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the robot, connecting various components of the robot through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the robot 200 and process data.

[0136] Figure 4 Only a robot with parts is shown; those skilled in the art will understand that... Figure 4 The structure shown does not constitute a limitation on the robot 200, and may include fewer or more parts than shown, or combine certain parts, or have different part arrangements.

[0137] The first memory 202 in the robot 200 stores a dual-handed dexterous grasping generation program 203 for multi-sized objects, which is a combination of multiple instructions. When run in the first processor 201, it can achieve the following:

[0138] Acquire point cloud data of the object to be grasped;

[0139] Cooperative prediction of gripping perspectives for both hands: Using the minimum bounding sphere of the object point cloud as a reference, a set of candidate gripping perspectives is constructed; based on the global geometric features of the object point cloud, the primary gripping perspective is predicted, and under the condition of the primary gripping perspective, a secondary gripping perspective that is coordinated with it is predicted, forming a coordinated gripping perspective combination for both hands.

[0140] Scale-adaptive gripping anchor point generation: Based on the geometric relationship between the gripping viewpoint and the object's minimum bounding sphere, the gripping anchor points for the left and right hands are determined, where the gripping anchor point is the intersection of the gripping viewpoint and the surface of the bounding sphere.

[0141] Grasping condition feature construction: Based on the global features of the point cloud, a set of point-level feature carriers are upsampled; taking the grasping anchor point as a spatial reference, local geometric features near the anchor point are extracted; the local geometric features, grasping viewpoint features and global geometric features of the object are fused to form a unified grasping condition feature.

[0142] Relative two-handed grasping generation: With the grasping condition features as constraints, a generation model is used to progressively sample and denoise the relative wrist poses and dexterous hand joint configurations of the left and right hands, and output the final two-handed dexterous grasping posture.

[0143] The relative wrist pose is described as the relative pose increment relative to the grasping anchor point, rather than the absolute pose in the object coordinate system.

[0144] Furthermore, if the modules / units integrated into the robot 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0145] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0146] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for data synthesis in dexterous two-handed grasping of objects of multiple sizes, characterized in that, Includes the following steps: Using the three-dimensional mesh model of the object as input, the farthest point sampling is used to select the reference point on the surface of the object and construct a local grasping area. After eliminating unstable contact areas, the candidate area pairs are geometrically screened. Force spinor space is introduced as a region stability evaluation index to select the candidate area pair with the highest score. Based on the area pair, the initial grasping posture of both hands is generated. The initial grasping posture of both hands includes the spatial translation and rotation parameters of the left and right dexterous wrists and the multi-degree-of-freedom joint configuration of the dexterous hands. An optimization objective function is constructed with the grasping posture of both hands as the variable. The force closure objective of grasping with both hands is decoupled into the local force closure energy of each hand. At the same time, contact distance constraints and region consistency constraints are incorporated. During the optimization process, joint angle, self-collision, inter-hand collision and hand-object collision constraints are applied to obtain a physically stable and coordinated grasping posture of both hands. Collision detection is performed on the optimized grasping posture, and solutions with collisions or penetrations are eliminated. Inverse kinematics are solved for the left and right robotic arms respectively to verify the reachability of wrist pose. Samples with collision-free inverse solutions are discarded. Pre-grasping and execution grasping postures are constructed for valid samples and continuous motion trajectories are generated by interpolation. Based on the grasping data generated by the above steps, a two-handed dexterous grasping dataset covering objects of multiple sizes and geometric shapes is constructed. Each object is discretely scaled to multiple sizes, and large-scale two-handed dexterous grasping samples are synthesized in a GPU parallel simulation environment.

2. The method for data synthesis of dexterous two-handed grasping of objects of multiple sizes according to claim 1, characterized in that, Each two-handed grasping sample is defined as a combination of the grasping postures of the left and right hands: in These represent the spatial translation and rotation parameters of the dexterous wrist, respectively. This indicates the multi-degree-of-freedom joint configuration of a dexterous hand.

3. The method for data synthesis of dexterous two-handed grasping of objects of multiple sizes according to claim 1, characterized in that, The regional stability score of the force spinor space is calculated by projecting the force spinor boundary onto multiple orthogonal perturbation directions, taking the minimum projection value as the lower bound of the stability of the regional pair, and the formula is as follows: in Denotes the first digit used to approximate the closure boundary. A force spinor, Indicates the first A predefined perturbation direction.

4. The method for data synthesis of dexterous two-handed grasping of objects of multiple sizes according to claim 1, characterized in that, The objective function is: The energy of local force closure , It is used to measure the stable support ability of an object by a single hand, and is defined as: in This represents the force spinor perturbation along the six orthogonal force directions. This indicates a grasping contact model. This represents the contact force within the friction cone at the gripping point.

5. A method for generating dexterous two-handed grasping of objects of multiple sizes, characterized in that, The model training and grasping generation based on the bimanual dexterity grasping dataset constructed according to any one of claims 1-4 includes the following steps: Acquire point cloud data of the object to be grasped; Cooperative prediction of gripping perspectives for both hands: Using the minimum bounding sphere of the object point cloud as a reference, a set of candidate gripping perspectives is constructed; based on the global geometric features of the object point cloud, the primary gripping perspective is predicted, and under the condition of the primary gripping perspective, a secondary gripping perspective that is coordinated with it is predicted, forming a coordinated gripping perspective combination for both hands. Scale-adaptive gripping anchor point generation: Based on the geometric relationship between the gripping viewpoint and the object's minimum bounding sphere, the gripping anchor points for the left and right hands are determined, where the gripping anchor point is the intersection of the gripping viewpoint and the surface of the bounding sphere. Grasping condition feature construction: Based on the global features of the point cloud, a set of point-level feature carriers are upsampled; taking the grasping anchor point as a spatial reference, local geometric features near the anchor point are extracted; the local geometric features, grasping viewpoint features and global geometric features of the object are fused to form a unified grasping condition feature. Relative two-handed grasping generation: With the grasping condition features as constraints, a generation model is used to progressively sample and denoise the relative wrist poses and dexterous hand joint configurations of the left and right hands, and output the final two-handed dexterous grasping posture. The relative wrist pose is described as the relative pose increment relative to the grasping anchor point, rather than the absolute pose in the object coordinate system.

6. The method for generating dexterous two-handed grasping of objects of multiple sizes according to claim 5, characterized in that, The bi-handed grasping perspective collaborative prediction specifically includes: Define the grabbing viewpoint in a scale-independent geometric space, where each grabbing viewpoint represents a direction of approach from the surface of the enclosing sphere toward the center of the object; The system utilizes the statistical distribution of successfully crawled samples in the dataset for supervision. Through probabilistic modeling, it first predicts the primary crawling perspective most likely to generate stable crawls, and then predicts the secondary crawling perspective with the best synergy with the primary crawling perspective.

7. The method for generating dexterous two-handed grasping of objects of multiple sizes according to claim 5, characterized in that, In the construction of the crawling conditional features, the method for extracting local geometric features is as follows: Using the captured anchor point as a spatial reference, its nearest neighbor points are searched in the point cloud features, and local geometric features near the anchor point are extracted by K-nearest neighbor aggregation. The local geometric features include the curvature, normal change and local shape information of the potential contact area. The crawling condition features Represented as: in For local geometric features, This is a view embedding representation derived from the view prediction process. This refers to the global geometric features of an object.

8. The method for generating dexterous two-handed grasping of objects of multiple sizes according to claim 5, characterized in that, The generative model can be any one of the following: a diffusion-based generative model, an autoregressive generative model, a conditional variational generative model, or an energy function-based generative model. When using a generative model based on diffusion processes, the relative wrist poses and multi-degree-of-freedom joint configurations of the left and right hands are gradually denoised to obtain grasping results that satisfy hand coordination, geometric constraints, and physical rationality.

9. A two-handed dexterous grasping generation system for objects of multiple sizes, characterized in that, The dual-handed dexterous grasping generation for multi-sized objects, as described in any one of claims 5-8, includes a data acquisition module, a dual-handed grasping perspective collaborative prediction module, a scale-adaptive grasping anchor point generation module, a grasping condition feature construction module, and a relative dual-handed grasping generation module. The data acquisition module is used to acquire point cloud data of the object to be captured; The dual-handed grasping perspective collaborative prediction module is used to construct a set of candidate grasping perspectives with the minimum bounding sphere of the object point cloud as a reference; based on the global geometric features of the object point cloud, it predicts the main grasping perspective, and under the condition of the main grasping perspective, it predicts the second grasping perspective that is coordinated with it, forming a grasping perspective combination that is coordinated between the left and right hands. The scale-adaptive grasping anchor point generation module is used to determine the grasping anchor points of the left and right hands based on the geometric relationship between the grasping viewpoint and the minimum bounding sphere of the object. The grasping anchor point is the intersection position of the grasping viewpoint and the surface of the bounding sphere. The grasping condition feature construction module is used to upsample a set of point-level feature carriers based on the global features of the point cloud; extract local geometric features near the grasping anchor point as a spatial reference; and fuse the local geometric features, grasping viewpoint features, and global geometric features of the object to form a unified grasping condition feature. The relative two-handed grasping generation module is used to progressively sample and denoise the relative wrist pose and dexterous hand joint configuration of the left and right hands using the grasping condition features as constraints, and output the final two-handed dexterous grasping posture. The relative wrist pose is described as the relative pose increment relative to the grasping anchor point, rather than the absolute pose in the object coordinate system.

10. A robot, characterized in that, The robot includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a method for generating a two-handed dexterous grasping mechanism for multi-sized objects as described in any one of claims 5-8.

Images