Machine grabbing learning method fusing shape features, contact modeling and physical constraints

Abstract

The invention provides a machine grabbing learning method fusing shape features, contact modeling and physical constraints, and the machine grabbing learning method is a robot grabbing posture learning method capable of differentiating. According to the method, a three-dimensional point cloud serves as input, a deep neural network based on sparse voxel convolution is adopted to extract multi-scale shape features, and potential grabbing points and attitude parameters thereof are predicted. The differential contact modeling is realized by constructing a local area of direction perception and extracting geometric features reflecting the contact relationship between the mechanical claw and the object. Furthermore, the collision probability of the grabbing posture is estimated based on an implicit neural network, and a differentiable collision detection module is constructed. Meanwhile, physical constraints such as friction closing, surface alignment and geometric symmetry are introduced to serve as regular terms to jointly optimize grabbing scores, collision risks and stability; according to the method, end-to-end training is supported, efficient, robust and deployable grabbing posture estimation is achieved, and the method is suitable for various automatic grabbing tasks such as industrial robots and service robots.

Description

Technical Field

[0001] The present invention relates to the field of robot intelligent operation technology, and in particular to a machine grasping learning method integrating shape features, contact modeling and physical constraints. Background Art

[0002] In automated grasping tasks, grasping strategies based on 3D point clouds have become an important means to improve robot flexibility and generalization capabilities. However, existing methods generally suffer from the following key issues:

[0003] 1. Decoupling feature extraction and grasp modeling: Most methods rely on manually designed heuristic features or non-differentiable grasp candidate generation mechanisms, which cannot be jointly trained end-to-end with subsequent grasp scoring or optimization modules. As a result, grasping performance is limited by the quality of feature design and is difficult to adapt to complex environments.

[0004] 2. Contact modeling lacks direction awareness and differentiable structure: Some methods use fixed-shape (such as cylinders or cubes) regions to extract local geometric features in grasping posture prediction, but do not consider the geometric consistency of the grasping direction, lack sensitivity to modeling contact details, and have difficulty in gradient propagation and optimization during training.

[0005] 3. Inefficient collision detection and difficulty integrating with training processes: Traditional collision detection relies on sampling based on physical simulation or explicit geometric calculations, which are computationally expensive and difficult to integrate with neural networks. Some work has attempted to use voxel grid approximations for judgment, but their limited spatial resolution capabilities cannot meet the accuracy and continuity required for grasp planning.

[0006] In summary, current grasping methods have yet to achieve unified modeling and joint optimization of point cloud shape, contact details, and physical feasibility. In particular, there remains a significant gap in learning end-to-end differentiable grasping strategies. To achieve stable, efficient, and scalable robotic grasping behaviors, a novel approach is urgently needed that integrates efficient 3D feature representation, differentiable grasping modeling mechanisms, and physically plausible joint learning strategies. Summary of the Invention

[0007] The present invention proposes a machine grasping learning method that integrates shape features, contact modeling and physical constraints. Specifically, it relates to a differentiable grasping posture learning method that integrates shape feature extraction, contact modeling and physical constraints. The method is suitable for automated robot grasping tasks in complex scenarios, and is particularly suitable for high-precision and high-robustness grasping planning in industrial robots and service robots. It helps to solve the problems of structural separation, training difficulties and insufficient robustness in the existing technologies in grasping feature modeling, contact expression and physical feasibility assessment.

[0008] The present invention adopts the following technical solutions.

[0009] A machine grasping learning method integrating shape features, contact modeling and physical constraints, said method being used for robot grasping posture learning, comprises the following steps:

[0010] Step 1: Obtain three-dimensional point cloud data describing the target scene or object for input;

[0011] Step 2: Use a sparse voxel convolution-based neural network to encode the point cloud and extract multi-scale shape features;

[0012] Step 3: Predict point-level grasping scores based on the extracted features, and generate a set of grasping candidate points through non-maximum suppression;

[0013] Step 4: Construct a direction-aware local region for each candidate point and extract differentiable geometric features that reflect the contact relationship of the robotic gripper;

[0014] Step 5: Input the grasping posture parameters and point cloud features into the implicit neural network to predict the potential collision probability during the grasping process and realize differentiable collision detection;

[0015] Step 6: Introduce physical constraints, including friction closure, surface alignment, and geometric symmetry regularization mechanisms to improve the stability and feasibility of the grasping posture;

[0016] Step 7: Construct a joint loss function to train the grasp pose prediction network end-to-end, while optimizing grasp scoring, collision prediction, and physical regularization objectives.

[0017] Step 8: In the reasoning phase, the grasping score and collision risk are combined to screen and output the final executable optimal grasping posture.

[0018] In step 1, a depth camera is used to collect 3D point cloud data. The depth camera is fixedly installed at the end of the robot's mechanical arm used to perform the grasping task and close to the gripper, forming a tightly coupled visual grasping system with it; Figure 2 As shown in the left area of ​​​​the visual grasping system, the grasping posture is represented as a five-tuple They are: Grasp the center point position; Unit grab direction vector; The in-plane rotation angle around the v-axis; Grasp the initial opening width and closing depth;

[0019] like Figure 2 As shown in the right area of ​​, the following coordinate system is introduced:

[0020] World Coordinate System Align with the robot base;

[0021] Point cloud coordinate system: The coordinate system used by the depth camera sensor to collect point clouds. The default is Alignment;

[0022] Grab the local coordinate system The origin is the grip center x, the grip direction v defines the local z-axis, and the rotation angle Determine the local x-axis direction and the y-axis according to the right-hand rule for subsequent feature extraction and contact modeling; camera coordinate system Aligned with the optical center of the depth camera as the source of the original coordinates of the point cloud;

[0023] Assume that the position of the end of the gripper in the world coordinate system is the transformation matrix The fixed external parameters of the camera relative to the gripper are Then the global pose of the camera is: The depth map collected by the camera can be parsed into The point cloud coordinates shown below Then it can be converted into a three-dimensional point in the world coordinate system through the above formula Right now:

[0024] Convert all points to uniform Coordinate point cloud Serves as input data for subsequent grasp candidate generation and geometric constraint modeling modules;

[0025] This system uses the Intel RealSense D435i depth camera, which supports depth map output up to 1280×720 at 30 FPS and has a typical working distance of 0.2m to 3m. Within a range of 0.5m, its depth ranging accuracy is better than 2% (i.e., error is less than 10mm), which can meet the local geometric structure perception requirements of most tabletop grasping scenarios.

[0026] The depth camera is installed at the end of the robotic arm and moves synchronously with the end during the grasping process, causing geometric changes in the point cloud of consecutive frames;

[0027] In order to ensure the uniformity and geometric consistency of point cloud data in the world coordinate system, the following pose transformation processing method is adopted in the system:

[0028] First, the real-time pose of the camera in the world coordinate system consists of the following two parts:

[0029] in: The pose of the end effector in the grasping coordinate system in the world coordinate system is calculated through the forward kinematics of the manipulator. Specifically, the transformation matrix of the end effector in the base coordinate system is obtained by recursively multiplying each joint angle θ of the manipulator with the robot DH parameter model.

[0030] It represents the fixed rigid body transformation of the camera relative to the end effector, which is obtained offline through hand-eye calibration. The calibration operation must be completed before the robot system is deployed. The hand-eye calibration process is as follows: a checkerboard calibration plate is fixed in the workspace, and the calibration plate is observed in multiple end-effector postures to obtain a set of end-effector transformations under the base. and the calibration plate transformation B observed by the camera i By solving the following hand-eye calibration equations: A i X=XB i ,i=1,2,…,n

[0031] You can solve The Tsai-Lenz algorithm is used to achieve calibration accuracy at the millimeter level, meeting the requirements of high-precision grasping perception. Secondly, the depth image captured by the camera is converted into a three-dimensional point in the camera coordinate system by back-projection under the condition that the intrinsic parameter matrix K is known. With the help of the above-mentioned pose transformation, the corresponding world coordinates are obtained: Building a unified point cloud collection Although the camera constantly changes its posture during movement, the above high-precision posture estimation and coordinate transformation can ensure the consistency and reliability of the point cloud data.

[0032] In step 2, the neural network structure is ResUNet18 implemented based on MinkowskiEngine, which is used to efficiently process sparse point clouds and extract spatial hierarchical features.

[0033] In step 3, the candidate generation strategy is adopted, specifically:

[0034] The system input of the machine grasping learning method is a dense point cloud acquired from the camera. Each point p i Accompanied by the unit normal vector n(p i ), optionally including additional attributes such as color and depth.

[0035] To generate candidate grasping points, a sparse voxelized 3D convolutional network based on MinkowskiEngine (ResUNet18 structure) is used as the point cloud encoding backbone. The main process is as follows:

[0036] Step A1, sparse voxel encoding: input point cloud Perform voxelization and construct sparse tensors to extract spatial hierarchical features;

[0037] Step A2, Graspability Score Prediction: For each point p i Output a graspability score s grasp (p i )∈[0,1], indicating the feasibility of the point as the grasping center;

[0038] Step A3, Non-Maximum Suppression: Apply spatial suppression to the score map to filter out local maximum points;

[0039] Step A4, extraction of grasping candidates: based on the set threshold, filter the point set {x j}, as a candidate grasping center point; each candidate point x j It will be used in subsequent grasping posture generation, contact modeling and feasibility assessment modules.

[0040] In step 4, the contact modeling of the gripper contact relationship constructs a spherical local region based on the grasping direction and models the contact features in the local coordinate system to maintain directional consistency and differentiability;

[0041] The contact modeling is differentiable contact modeling, specifically:

[0042] like Figure 3 As shown above, to enhance the perception of local geometric structure in grasping posture, a direction-aware local feature extraction method is used to accurately model the contact area and guide the prediction of grasping parameters;

[0043] For each candidate grasp center x, under the constraint of the grasp direction v, the local point cloud neighborhood is defined as the direction conditional sphere region: Among them, r is the neighborhood radius;

[0044] To unify the direction representation, align the grasping direction v to the local z-axis and construct the rotation matrix R v , such that:

[0045] Get the point cloud {p i '};

[0046] Use lightweight PointNet to encode local point clouds and introduce a directional attention mechanism to enhance the response to the grasping direction:

[0047] in:

[0048] The final direction-aware local feature is the weighted sum: f local =∑ i a i ·h(p i ′)

[0049] Among them, h(p i ′) is the point feature output by the point cloud encoder;

[0050] The geometric consistency constraint of this step is as follows: In order to improve the consistency between the local feature expression and the geometric structure, a feature alignment loss term is introduced to constrain the direction perception feature and the reference geometric template feature f cylinder Alignment:

[0051] Through directional constraints and attention mechanism, differentiable modeling of the contact area is achieved, enhancing the geometric adaptability of the grasping posture.

[0052] In step 5, the implicit collision detection module is used for differentiable collision detection. The implicit collision detection module is implemented by a multi-layer perceptron combined with position encoding, which supports collision probability estimation of grasping postures in continuous space. Specifically:

[0053] like Figure 3 As shown in the lower area of ​​​​ Whether to use scene point cloud A collision occurs, and a differentiable neural collision function f is constructed θ (p i ,G), for each point p i Output the probability of being collided by the gripper in this posture: f θ (p i ,G)∈[0,1]

[0054] The overall collision is determined by whether the candidate pose collides with any point: Among them, ∈=0 is the judgment threshold, is the characteristic function;

[0055] To achieve end-to-end training, the neural network uses a multi-layer perceptron (MLP) to model the collision probability function, whose input is the combined features of the point and the grasping parameters:

[0056] where γ(·) represents the Fourier position encoding of the relative position between the point and the grasping center; f global Represents the global contextual features of the scene;

[0057] During training, the supervision label y i ∈{0,1} represents point p i Whether it is collided with the grasping posture G, the loss function uses binary cross entropy:

[0058] The collision function modeled by neural networks is differentiable and suitable for gradient-based grasping posture optimization, overcoming the limitations of traditional geometric collision detection in gradient propagation.

[0059] In step 6, the physical constraints are modeled using a differentiable physical constraint modeling method, specifically:

[0060] To further improve the physical feasibility and execution stability of the predicted grasping posture, three regularization terms based on physical and geometric principles are designed to constrain the contact stability, structural symmetry, and surface alignment.

[0061] a) Friction closure constraint, that is, the grasping point meets the friction closure condition to improve contact stability;

[0062] like Figure 4 As shown in the left area, considering the angle between the normal direction of the contact point and the fingertip connection vector, let the normal directions of the two contact points be n(u1) and n(u2). If the angle between them and the grasping direction is less than the friction cone half angle θ f =tan -1 (μ), then the friction closure condition is considered to be satisfied: ∠(n(u i ),±v)<θ f ,i=1,2

[0063] Where μ is the friction coefficient, which is set in the range of [0.5, 0.8] during training and is compatible with a variety of industrial materials;

[0064] The stability of the grasp depends on whether the angle between the contact point normal and the grasp axis falls within the friction cone. If the surface normals n(u1) and n(u2) of a pair of contact points u1 and u2 are facing each other and meet the friction angle constraint, then the grasp is stable and closed; the mathematical expression is as follows:

[0065]

[0066] Where p(·) represents the coordinates of the contact point, n(·) represents the unit normal vector, and c f is the friction threshold, which is defined by the friction coefficient μ: c f =cos(tan -1 μ)

[0067] To achieve differentiable optimization, the above conditions are constructed as the following loss function:

[0068]

[0069] b) Geometric symmetry constraints: the grasping contact points are symmetrically distributed around the object's center of mass to enhance grasping balance; specifically: Figure 4As shown in the right area, the left-right symmetry of the gripping posture helps to balance the force on the gripper. k For reference, we constrain the difference in Euclidean distances between the left and right contact points and the center, and define the symmetry loss:

[0070] in, and They represent the contact point positions of the left and right grippers in the i-th grasp, and N is the batch size;

[0071] c) Surface alignment constraint: The grasping point fits the object surface to reduce spatial offset errors. Specifically, to ensure that the predicted contact point falls close to the real object surface, a distance constraint based on the signed distance function (SDF) is introduced. Let SDF(u) represent the signed distance from point u to the object surface, and define the loss:

[0072]

[0073] Among them, d1 and d2 are the minimum and maximum acceptable contact distance thresholds, respectively, and δ1 and δ2 are balance coefficients; these regularization terms together define the complete physical constraint loss: Among them, α and β are adjustable weight hyperparameters.

[0074] In step 7, the joint loss function includes the joint training objective function, specifically:

[0075] To achieve end-to-end training of differentiable grasp pose estimation, grasp scoring, collision detection, and physical regularization terms are integrated into a unified training objective function; the details are as follows:

[0076]

[0077] in: Grasp scoring loss, which measures the accuracy of grasp confidence prediction for the current pose;

[0078] Collision detection loss, using binary cross entropy to supervise whether a collision occurs between point clouds and poses;

[0079] Physical constraint regularization term, including friction closure constraint Geometric symmetry constraints Surface alignment constraints Defined as:

[0080] In order to balance the impact of the three loss items on the total loss, a weighting coefficient is introduced, which can be adjusted according to the task requirements;

[0081] Through end-to-end joint optimization, the trained model not only improves the grasp scoring and discrimination capabilities, but also learns to avoid collisions and meet physical stability requirements, thereby generating more stable and executable grasping postures.

[0082] Step 8 includes the inference and crawl execution process, specifically:

[0083] In the deployment phase, the system of the machine grasping learning method receives the scene point cloud As input, the following steps are performed to generate and filter the grasping posture:

[0084] Step B1, grasp candidate generation: Use the point cloud backbone network to output the graspability score s of each point graspness , through non-maximum suppression (NMS) and threshold screening, a set of candidate grasping center points {x j};

[0085] Step B2: Grasp parameter prediction: For each candidate center point x j , using the direction-aware local feature extraction module to predict the corresponding grasping posture parameters

[0086] Step B3, collision judgment: for each candidate grasping posture G j , calculate the collision probability function C(P,G j ), filter out all candidates with collisions:

[0087] Step B4: Grab the score and sort: for the candidate set without collision Calculate the crawl score Q(P,G j ), and finally select the grasping posture with the highest score as the output:

[0088] Step B5, robot execution: set the optimal grasping posture parameter G * The data is transmitted to the robotic arm through the control interface to complete the grasping action.

[0089] The method is used for high-precision grasping planning tasks in industrial robots, service robots or other automation systems, and the working distance between the mechanical gripper and the grasping target ranges from 0.2m to 3m.

[0090] The advantages of this invention lie in that it constructs a unified microstructure through end-to-end encoding learning of 3D point clouds, and jointly models grasp point prediction, contact geometry modeling, collision detection, and physical stability constraints, effectively improving the feasibility and stability of grasping postures. Its advantages mainly include the following aspects:

[0091] 1. Shape-aware grasp candidate generation: A deep neural network based on sparse voxel convolution is used to encode the 3D point cloud, predict the grasp score of each point, and screen high-quality candidate grasp points through non-maximum suppression. This can achieve an efficient and spatially sparse-aware grasp candidate generation mechanism.

[0092] 2. Adopting a direction-aware contact modeling mechanism: For each grasping candidate point, a spherical local region is constructed with the grasping direction as the alignment reference. Direction-sensitive geometric features are extracted in the local coordinate system. Combined with the attention mechanism, the ability to perceive contact details is improved, while maintaining the differentiability of the structure and facilitating training and optimization.

[0093] 3. Adopting differentiable collision detection based on implicit neural fields: Designing a neural collision detection module that outputs collision probability distribution in continuous space by inputting capture parameters and point cloud features. This module can replace traditional voxel-based or geometry-based hard judgment methods, support end-to-end training, and enhance the model's detailed perception of collision risks.

[0094] 4. Adopt physically reasonable regularized modeling: Introducing physical constraints such as friction closure, surface alignment, and geometric symmetry as regularized loss functions during the learning process can effectively improve the stability, symmetry, and surface fit of the grasping posture, thereby enhancing the grasping success rate during actual execution.

[0095] 5. Adopt an end-to-end joint training strategy: Construct a multi-objective loss function that includes grasping scoring loss, collision detection loss, and physical regularization term. This can uniformly optimize various modules in a single neural network framework, making the model more generalizable and adaptable to real-world scenarios.

[0096] 6. Adopt a grasping posture executable evaluation mechanism: During the inference phase, the grasping score and collision prediction results are combined to screen physically feasible grasping postures with the lowest collision probability, which can improve the grasping success rate and robustness during actual deployment.

[0097] In summary, the present invention solves the problems of traditional methods in terms of inconsistent features, non-differentiable collisions, and unstable grasping by introducing direction-aware contact area modeling, differentiable neural collision detection, and multi-physics constraint optimization mechanisms into the grasping modeling process, and provides a high-precision and high-robustness grasping strategy learning method suitable for industrial and service robot scenarios.

[0098] The present invention discloses a differentiable robot grasping posture learning method that integrates shape features, contact modeling and physical constraints. The method takes a three-dimensional point cloud as input, and uses a deep neural network based on sparse voxel convolution to extract multi-scale shape features and predict potential grasping points and their posture parameters. By constructing a local area of ​​direction perception, the geometric features reflecting the contact relationship between the robotic claw and the object are extracted to achieve differentiable contact modeling. Furthermore, the collision probability of the grasping posture is estimated based on an implicit neural network, and a differentiable collision detection module is constructed. At the same time, physical constraints such as friction closure, surface alignment and geometric symmetry are introduced as regularization terms to jointly optimize the grasping score, collision risk and stability. The method supports end-to-end training, realizes efficient, robust and deployable grasping posture estimation, and is suitable for a variety of automated grasping tasks such as industrial robots and service robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0100] Attachment Figure 1 This is a schematic diagram of the overall framework structure of the proposed point cloud-based robot grasping posture optimization method (the figure shows the main functional modules of the system and their data flow relationship, including: input point cloud data P; grasping candidate generation module, used to predict multiple potential grasping postures from the point cloud; differentiable contact modeling is used to construct a direction-aligned geometric feature representation for the candidate grasping area; differentiable collision detection is used to estimate the collision risk between the grasping posture and the environment based on the point cloud neighborhood; differentiable physical constraints, including friction closure constraints, grasping symmetry constraints and surface alignment constraints, are used to improve grasping stability and executability; grasping score fusion module is used to comprehensively consider various scoring factors and select the final optimal grasping posture G * );

[0101] Attachment Figure 2 This is a schematic diagram of the geometric modeling and coordinate system definition of the robot's mechanical claw grasping system;

[0102] Attachment Figure 3 It is a schematic diagram of the algorithm flow of the grasping system;

[0103] Attachment Figure 4 It is a schematic diagram of the physical constraints in the present invention. DETAILED DESCRIPTION

[0104] This example proposes a differentiable grasping posture learning method that integrates shape feature extraction, contact modeling, and physical constraint optimization to support efficient estimation of stable and safe grasping postures from 3D point clouds. Figure 2-4 , which details the various key modules of the present invention and their physical and algorithmic implementations. Figure 2 Introduce the mechanical claw and environment configuration, Figure 3shows the algorithm flow chart, Figure 4 The physical constraint schematic is introduced.

[0105] As shown in the figure, a machine grasping learning method integrating shape features, contact modeling and physical constraints, which is used for robot grasping posture learning, includes the following steps:

[0106] Step 1: Obtain three-dimensional point cloud data describing the target scene or object for input;

[0107] Step 2: Use a sparse voxel convolution-based neural network to encode the point cloud and extract multi-scale shape features;

[0108] Step 3: Predict point-level grasping scores based on the extracted features, and generate a set of grasping candidate points through non-maximum suppression;

[0109] Step 4: Construct a direction-aware local region for each candidate point and extract differentiable geometric features that reflect the contact relationship of the robotic gripper;

[0110] Step 5: Input the grasping posture parameters and point cloud features into the implicit neural network to predict the potential collision probability during the grasping process and realize differentiable collision detection;

[0111] Step 6: Introduce physical constraints, including friction closure, surface alignment, and geometric symmetry regularization mechanisms to improve the stability and feasibility of the grasping posture;

[0112] Step 7: Construct a joint loss function to train the grasp pose prediction network end-to-end, while optimizing grasp scoring, collision prediction, and physical regularization objectives.

[0113] Step 8: In the reasoning phase, the grasping score and collision risk are combined to screen and output the final executable optimal grasping posture.

[0114] In step 1, a depth camera is used to collect 3D point cloud data. The depth camera is fixedly installed at the end of the robot's mechanical arm used to perform the grasping task and close to the gripper, forming a tightly coupled visual grasping system with it; Figure 2 As shown in the left area of ​​​​the visual grasping system, the grasping posture is represented as a five-tuple They are: Grasp the center point position; Unit grab direction vector; The in-plane rotation angle around the v-axis; Grasp the initial opening width and closing depth;

[0115] like Figure 2 As shown in the right area of ​​, the following coordinate system is introduced:

[0116] World Coordinate System Align with the robot base;

[0117] Point cloud coordinate system: The coordinate system used by the depth camera sensor to collect point clouds. The default is alignment;

[0118] Grab the local coordinate system The origin is the grip center x, the grip direction v defines the local z-axis, and the rotation angle Determine the local x-axis direction and the y-axis according to the right-hand rule for subsequent feature extraction and contact modeling; camera coordinate system Aligned with the optical center of the depth camera as the source of the original coordinates of the point cloud;

[0119] Assume that the position of the end of the gripper in the world coordinate system is the transformation matrix The fixed external parameters of the camera relative to the gripper are Then the global pose of the camera is: The depth map collected by the camera can be parsed into The point cloud coordinates shown below Then it can be converted into a three-dimensional point in the world coordinate system through the above formula Right now: Convert all points to uniform Coordinate point cloud Serves as input data for subsequent grasp candidate generation and geometric constraint modeling modules;

[0120] This system uses the Intel RealSense D435i depth camera, which supports depth map output up to 1280×720 at 30 FPS and has a typical working distance of 0.2m to 3m. Within a range of 0.5m, its depth ranging accuracy is better than 2% (i.e., error is less than 10mm), which can meet the local geometric structure perception requirements of most tabletop grasping scenarios.

[0121] The depth camera is installed at the end of the robotic arm and moves synchronously with the end during the grasping process, causing geometric changes in the point cloud of consecutive frames;

[0122] In order to ensure the uniformity and geometric consistency of point cloud data in the world coordinate system, the following pose transformation processing method is adopted in the system:

[0123] First, the real-time pose of the camera in the world coordinate system consists of the following two parts:

[0124] in: The pose of the end effector in the grasping coordinate system in the world coordinate system is calculated by the forward kinematics of the manipulator. Specifically, according to each joint angle θ of the manipulatori Recursively multiply the DH parameter model of the robot to obtain the transformation matrix of the end in the base coordinate system;

[0125] It represents the fixed rigid body transformation of the camera relative to the end effector, which is obtained offline through hand-eye calibration. The calibration operation must be completed before the robot system is deployed. The hand-eye calibration process is as follows: a checkerboard calibration plate is fixed in the workspace, and the calibration plate is observed in multiple end-effector postures to obtain a set of end-effector transformations under the base. and the calibration plate transformation B observed by the camera i By solving the following hand-eye calibration equations: A i X=XB i ,i=1,2,…,n

[0126] You can solve

[0127] It is implemented using the Tsai-Lenz algorithm, with calibration accuracy at the millimeter level, meeting the needs of high-precision grasping perception.

[0128] Secondly, the depth image captured by the camera is converted into a three-dimensional point in the camera coordinate system by back projection under the condition that the intrinsic parameter matrix K is known. With the help of the above-mentioned pose transformation, the corresponding world coordinates are obtained:

[0129] Building a unified point cloud collection

[0130] Although the camera constantly changes its posture during movement, the above high-precision posture estimation and coordinate transformation can ensure the consistency and reliability of the point cloud data.

[0131] In step 2, the neural network structure is ResUNet18 implemented based on MinkowskiEngine, which is used to efficiently process sparse point clouds and extract spatial hierarchical features.

[0132] In step 3, the candidate generation strategy is adopted, specifically:

[0133] The system input of the machine grasping learning method is a dense point cloud acquired from the camera. Each point p i Accompanied by the unit normal vector n(p i ), optionally including additional attributes such as color and depth.

[0134] To generate candidate grasping points, a sparse voxelized 3D convolutional network based on MinkowskiEngine (ResUNet18 structure) is used as the point cloud encoding backbone. The main process is as follows:

[0135] Step A1, sparse voxel encoding: input point cloud Perform voxelization and construct sparse tensors to extract spatial hierarchical features;

[0136] Step A2, Graspability Score Prediction: For each point p i Output a graspability score s grasp (p i )∈[0,1], indicating the feasibility of the point as the grasping center;

[0137] Step A3, Non-Maximum Suppression: Apply spatial suppression to the score map to filter out local maximum points;

[0138] Step A4, extraction of grasping candidates: based on the set threshold, filter the point set {x j}, as a candidate grasping center point; each candidate point x j It will be used in subsequent grasping posture generation, contact modeling and feasibility assessment modules.

[0139] In step 4, the contact modeling of the gripper contact relationship constructs a spherical local region based on the grasping direction and models the contact features in the local coordinate system to maintain directional consistency and differentiability;

[0140] The contact modeling is differentiable, specifically:

[0141] like Figure 3 As shown above, to enhance the perception of local geometric structure in grasping posture, a direction-aware local feature extraction method is used to accurately model the contact area and guide the prediction of grasping parameters;

[0142] For each candidate grasp center x, under the constraint of the grasp direction v, the local point cloud neighborhood is defined as the direction conditional sphere region: Among them, r is the neighborhood radius;

[0143] To unify the direction representation, align the grasping direction v to the local z-axis and construct the rotation matrix R v , such that:

[0144] Get the point cloud {p i '};

[0145] Use lightweight PointNet to encode local point clouds and introduce a directional attention mechanism to enhance the response to the grasping direction:

[0146] in:

[0147] The final direction-aware local feature is the weighted sum: f local =∑ i a i ·h(p i ′)

[0148] Among them, h(p i ′) is the point feature output by the point cloud encoder;

[0149] The geometric consistency constraint of this step is as follows: In order to improve the consistency between the local feature expression and the geometric structure, a feature alignment loss term is introduced to constrain the direction perception feature and the reference geometric template feature f cylinder Alignment:

[0150] Through directional constraints and attention mechanism, differentiable modeling of the contact area is achieved, enhancing the geometric adaptability of the grasping posture.

[0151] In step 5, the implicit collision detection module is used for differentiable collision detection. The implicit collision detection module is implemented by a multi-layer perceptron combined with position encoding, which supports collision probability estimation of grasping postures in continuous space. Specifically:

[0152] like Figure 3 As shown in the lower area of ​​​​ Whether to use scene point cloud A collision occurs, and a differentiable neural collision function f is constructed θ (p i ,G), for each point p i Output the probability of being collided by the gripper in this posture: f θ (p i ,G)∈[0,1]

[0153] The overall collision is determined by whether the candidate pose collides with any point: Among them, ∈=0 is the judgment threshold, is the indicator function;

[0154] To achieve end-to-end training, the neural network uses a multi-layer perceptron (MLP) to model the collision probability function, whose input is the combined features of the point and the grasping parameters:

[0155] where γ(·) represents the Fourier position encoding of the relative position between the point and the grasping center; fglobal Represents the global contextual features of the scene;

[0156] During training, the supervision label y i ∈{0,1} represents point p i Whether it is collided with the grasping posture G, the loss function uses binary cross entropy:

[0157] The collision function modeled by neural networks is differentiable and suitable for gradient-based grasping posture optimization, overcoming the limitations of traditional geometric collision detection in gradient propagation.

[0158] In step 6, the physical constraints are modeled using a differentiable physical constraint modeling method, specifically:

[0159] To further improve the physical feasibility and execution stability of the predicted grasping posture, three regularization terms based on physical and geometric principles are designed to constrain the contact stability, structural symmetry, and surface alignment.

[0160] a) Friction closure constraint, that is, the grasping point meets the friction closure condition to improve contact stability;

[0161] like Figure 4 As shown in the left area, considering the angle between the normal direction of the contact point and the fingertip connection vector, let the normal directions of the two contact points be n(u1) and n(u2). If the angle between them and the grasping direction is less than the friction cone half angle θ f =tan -1 (μ), then the friction closure condition is considered to be satisfied: ∠(n(u i ),±v)<θ f ,i=1,2

[0162] Where μ is the friction coefficient, which is set in the range of [0.5, 0.8] during training and is compatible with a variety of industrial materials;

[0163] The stability of the grasp depends on whether the angle between the contact point normal and the grasp axis falls within the friction cone. If the surface normals n(u1) and n(u2) of a pair of contact points u1 and u2 are facing each other and meet the friction angle constraint, then the grasp is stable and closed; the mathematical expression is as follows:

[0164]

[0165] Where p(·) represents the coordinates of the contact point, n(·) represents the unit normal vector, and c f is the friction threshold, which is defined by the friction coefficient μ: c f =cos(tan -1 μ)

[0166] To achieve differentiable optimization, the above conditions are constructed as the following loss function:

[0167]

[0168] b) Geometric symmetry constraints: the grasping contact points are symmetrically distributed around the object's center of mass to enhance grasping balance; specifically: Figure 4 As shown in the right area, the left-right symmetry of the gripping posture helps to balance the force on the gripper. k For reference, we constrain the difference in Euclidean distances between the left and right contact points and the center, and define the symmetry loss:

[0169] in, and They represent the contact point positions of the left and right grippers in the i-th grasp, and N is the batch size;

[0170] c) Surface alignment constraint: The grasping point fits the object surface to reduce spatial offset errors. Specifically, to ensure that the predicted contact point falls close to the real object surface, a distance constraint based on the signed distance function (SDF) is introduced. Let SDF(u) represent the signed distance from point u to the object surface, and define the loss:

[0171]

[0172] Among them, d1 and d2 are the minimum and maximum acceptable contact distance thresholds, respectively, and δ1 and δ2 are balance coefficients; these regularization terms together define the complete physical constraint loss: Among them, α and β are adjustable weight hyperparameters.

[0173] In step 7, the joint loss function includes the joint training objective function, specifically:

[0174] To achieve end-to-end training of differentiable grasp pose estimation, grasp scoring, collision detection, and physical regularization terms are integrated into a unified training objective function; the details are as follows:

[0175]

[0176] in: Grasp scoring loss, which measures the accuracy of grasp confidence prediction for the current pose;

[0177] Collision detection loss, using binary cross entropy to supervise whether a collision occurs between point clouds and poses; Physical constraint regularization term, including friction closure constraint Geometric symmetry constraints Surface alignment constraints Defined as:

[0178] In order to balance the impact of the three loss items on the total loss, a weighting coefficient is introduced, which can be adjusted according to the task requirements;

[0179] Through end-to-end joint optimization, the trained model not only improves the grasp scoring and discrimination capabilities, but also learns to avoid collisions and meet physical stability requirements, thereby generating more stable and executable grasping postures.

[0180] Step 8 includes the inference and crawl execution process, specifically:

[0181] In the deployment phase, the system of the machine grasping learning method receives the scene point cloud As input, the following steps are performed to generate and filter the grasping posture:

[0182] Step B1, grasp candidate generation: Use the point cloud backbone network to output the graspability score s of each point graspness , through non-maximum suppression (NMS) and threshold screening, a set of candidate grasping center points {x j};

[0183] Step B2: Grasp parameter prediction: For each candidate center point x j , using the direction-aware local feature extraction module to predict the corresponding grasping posture parameters

[0184] Step B3, collision judgment: for each candidate grasping posture G j , calculate the collision probability function C(P,G j ), filter out all candidates with collisions:

[0185] Step B4: Grab the score and sort: for the candidate set without collision Calculate the crawl score Q(P,G j ), and finally select the grasping posture with the highest score as the output:

[0186] Step B5, robot execution: set the optimal grasping posture parameter G * The data is transmitted to the robotic arm through the control interface to complete the grasping action.

[0187] The above process realizes a closed loop from raw point cloud perception to final grasping action, ensuring the physical feasibility and execution stability of the grasping posture.

[0188] The method is used for high-precision grasping planning tasks in industrial robots, service robots or other automation systems, and the working distance between the mechanical gripper and the grasping target ranges from 0.2m to 3m.

[0189] Example:

[0190] This example takes the grabbing of a mug in a desktop scenario as an example to explain in detail the entire process of grabbing posture generation and execution of the present invention:

[0191] The input point cloud is collected by an RGB-D camera, and the target object is a ceramic mug with a smooth body.

[0192] The point cloud backbone network analyzes the point cloud data and generates a set of candidate grasp center points x, one of which is located on the side of the cup body, and the grasp direction v points horizontally in the normal direction of the cup wall.

[0193] Gripper posture rotation angle Set it to 0 to ensure that the jaws open parallel to the tabletop.

[0194] In the candidate grasping area, local geometric features are extracted and combined with the directional attention mechanism to predict the gripper parameters w,d and contact area.

[0195] The angle between the contact point normal and the grasping direction is approximately 35°, satisfying the friction cone constraint with a friction coefficient of μ = 0.7, ensuring stable closure of the grasp.

[0196] The neural collision module outputs the collision probability f θ (p i ,G)<0.05, indicating that there is no obvious collision risk.

[0197] The grasp score Q(P,G) = 0.92, indicating that the grasp posture has a high execution confidence.

[0198] The final grasping posture G is set through the control interface * Send to the robot to realize the grasping action.

[0199] This example fully demonstrates the complete closed-loop process of the method of the present invention from point cloud perception, posture generation, physical constraint evaluation to grasping execution, and verifies the effectiveness and robustness of the algorithm in practical applications.

Claims

1. A machine grasping learning method that integrates shape features, contact modeling, and physical constraints, characterized by: The method is used for robot grasping posture learning and comprises the following steps: Step 1: Obtain three-dimensional point cloud data describing the target scene or object for input; Step 2: Use a sparse voxel convolution-based neural network to encode the point cloud and extract multi-scale shape features; Step 3: Predict point-level grasping scores based on the extracted features, and generate a set of grasping candidate points through non-maximum suppression; Step 4: Construct a direction-aware local region for each candidate point and extract differentiable geometric features that reflect the contact relationship of the robotic gripper; Step 5: Input the grasping posture parameters and point cloud features into the implicit neural network to predict the potential collision probability during the grasping process and realize differentiable collision detection; Step 6: Introduce physical constraints, including friction closure, surface alignment, and geometric symmetry regularization mechanisms to improve the stability and feasibility of the grasping posture; Step 7: Construct a joint loss function to train the grasp pose prediction network end-to-end, while optimizing grasp scoring, collision prediction, and physical regularization objectives. Step 8: In the reasoning phase, the grasping score and collision risk are combined to screen and output the final executable optimal grasping posture.

2. The machine grasping learning method integrating shape features, contact modeling, and physical constraints according to claim 1 is characterized by: In step 1, a depth camera is used to collect 3D point cloud data. The depth camera is fixedly installed at the end of the robot's mechanical arm used to perform grasping tasks and close to the gripper, forming a tightly coupled visual grasping system with it. In the visual grasping system, the grasping posture is represented as a five-tuple They are: Grasp the center point position; Unit grab direction vector; The in-plane rotation angle around the v-axis; Grasp the initial opening width and closing depth; Introduce the following coordinate system: World Coordinate System Align with the robot base; Point cloud coordinate system: The coordinate system used by the depth camera sensor to collect point clouds. The default is Alignment; Grab the local coordinate system The origin is the grip center x, the grip direction v defines the local z-axis, and the rotation angle Determine the local x-axis direction and the y-axis according to the right-hand rule for subsequent feature extraction and contact modeling; camera coordinate system Aligned with the optical center of the depth camera as the source of the original coordinates of the point cloud; Assume that the position of the end of the gripper in the world coordinate system is the transformation matrix The fixed external parameters of the camera relative to the gripper are Then the global pose of the camera is: The depth map collected by the camera is parsed into the coordinate system The point cloud coordinates shown below Then, the above formula is used to convert it into a three-dimensional point in the world coordinate system Right now: Convert all points to uniform Coordinate point cloud Serves as input data for subsequent grasp candidate generation and geometric constraint modeling modules; The depth camera is installed at the end of the robotic arm and moves synchronously with the end during the grasping process, causing geometric changes in the point cloud of consecutive frames; In order to ensure the uniformity and geometric consistency of point cloud data in the world coordinate system, the following pose transformation processing method is adopted in the system: First, the real-time pose of the camera in the world coordinate system consists of the following two parts: in: The pose of the end effector in the grasping coordinate system in the world coordinate system is obtained by the forward kinematics calculation of the manipulator; specifically, according to each joint angle θ of the manipulator i Recursively multiply the DH parameter model of the robot to obtain the transformation matrix of the end in the base coordinate system; Represents the fixed rigid body transformation of the camera relative to the end effector, obtained offline through hand-eye calibration. The calibration must be completed before the robot system is deployed. The hand-eye calibration process is as follows: fix the checkerboard calibration plate in the workspace, observe the calibration plate under multiple end-point postures, and obtain the transformation of the end-point under the base. and the calibration plate transformation B observed by the camera i By solving the following hand-eye calibration equations: A i X=XB i ,i=1,2,…,n You can solve The depth image captured by the camera is converted into a three-dimensional point in the camera coordinate system by back projection under the condition that the intrinsic parameter matrix K is known. With the help of the above-mentioned pose transformation, the corresponding world coordinates are obtained: Building a unified point cloud collection 3. The machine grasping learning method integrating shape features, contact modeling, and physical constraints according to claim 2 is characterized by: In step 2, the neural network structure is ResUNet18 implemented based on MinkowskiEngine, which is used to efficiently process sparse point clouds and extract spatial hierarchical features.

4. The machine grasping learning method integrating shape features, contact modeling, and physical constraints according to claim 2 is characterized by: In step 3, the candidate generation strategy is adopted, specifically: The system input of the machine grasping learning method is a dense point cloud acquired from the camera. Each point p i Accompanied by the unit normal vector n(p i ); To generate candidate grasping points, a sparse voxelized 3D convolutional network based on MinkowskiEngine is used as the point cloud encoding backbone. The process is as follows: Step A1, sparse voxel encoding: input point cloud Perform voxelization and construct sparse tensors to extract spatial hierarchical features; Step A2, Graspability Score Prediction: For each point p i Output crawlability score s grasp (p i )∈[0,1], indicating the feasibility of the point as the grasping center; Step A3, non-maximum suppression: apply spatial suppression to the score map to screen local maximum points; Step A4, extraction of grasping candidates: based on the set threshold, filter the point set {x j }, as a candidate grasping center point; each candidate point x j It will be used in subsequent grasping posture generation, contact modeling and feasibility assessment modules.

5. The machine grasping learning method integrating shape features, contact modeling, and physical constraints according to claim 2 is characterized by: In step 4, the contact modeling of the gripper contact relationship constructs a spherical local area based on the grasping direction and models the contact features in the local coordinate system to maintain directional consistency and differentiability. The contact modeling is differentiable contact modeling, specifically: To enhance the grasping posture's awareness of local geometric structures, a direction-aware local feature extraction method is used to accurately model the contact area and guide the grasping parameter prediction. For each candidate grasp center x, under the constraint of the grasp direction v, the local point cloud neighborhood is defined as the direction conditional sphere region: Among them, r is the neighborhood radius; To unify the direction representation, align the grasping direction v to the local z-axis and construct the rotation matrix R v , such that: Get the point cloud {p i '}; Use lightweight PointNet to encode local point clouds and introduce a directional attention mechanism to enhance the response to the grasping direction: in: r i =-||p i -x|| The final direction-aware local feature is the weighted sum: d local =∑ i a i ·h(p i ′) Among them, h(p i ′) is the point feature output by the point cloud encoder; The geometric consistency constraint of this step is as follows: In order to improve the consistency between the local feature expression and the geometric structure, a feature alignment loss term is introduced to constrain the direction perception feature and the reference geometric template feature f cylinder Alignment: Through directional constraints and attention mechanism, differentiable modeling of the contact area is achieved, enhancing the geometric adaptability of the grasping posture.

6. The machine grasping learning method integrating shape features, contact modeling, and physical constraints according to claim 2 is characterized by: In step 5, the implicit collision detection module is used for differentiable collision detection. The implicit collision detection module is implemented by a multi-layer perceptron combined with position encoding, which supports collision probability estimation of grasping postures in continuous space. Specifically: In order to determine the candidate grasping posture Whether to use scene point cloud A collision occurs, and a differentiable neural collision function f is constructed θ (p i ,G), for each point p i Output the probability of being collided by the gripper in this posture: f θ (p i ,G)∈[0,1] The overall collision is determined by whether the candidate pose collides with any point: Among them, ∈=0 is the judgment threshold, is the indicator function; To achieve end-to-end training, the neural network uses a multi-layer perceptron (MLP) to model the collision probability function, whose input is the combined features of the point and the grasping parameters: where γ(·) represents the Fourier position encoding of the relative position between the point and the grasping center; f global Represents the global contextual features of the scene; During training, the supervision label y i ∈{0,1} represents point p i Whether it is collided with the grasping posture G, the loss function uses binary cross entropy: The collision function modeled by neural networks is differentiable and used for gradient-based grasping posture optimization.

7. The machine grasping learning method integrating shape features, contact modeling, and physical constraints according to claim 2 is characterized by: In step 6, the physical constraints are modeled using a differentiable physical constraint modeling method, specifically: a) Friction closure constraint, that is, the grasping point meets the friction closure condition to improve contact stability; Considering the angle between the normal direction of the contact point and the fingertip connection vector, let the normal directions of the two contact points be n(u1) and n(u2). If the angle between them and the grasping direction is less than the friction cone half angle θ f =tan -1 (μ), then the friction closure condition is considered to be satisfied: ∠(n(u i ),±v)<θ f ,i=1,2 Where μ is the friction coefficient; If the surface normals n(u1) and n(u2) of a pair of contact points u1 and u2 are facing each other and the friction angle constraint is satisfied, then the grip is considered stable and closed; the mathematical expression is as follows: Where p(·) represents the coordinates of the contact point, n(·) represents the unit normal vector, and c f is the friction threshold, which is defined by the friction coefficient μ: c f =cos(tan -1 μ) To achieve differentiable optimization, the above conditions are constructed as the following loss function: b) Geometric symmetry constraints: the grasping contact points are symmetrically distributed around the object's center of mass, enhancing grasping balance; Specifically: take the object center C k For reference, we constrain the difference in Euclidean distances between the left and right contact points and the center, and define the symmetry loss: in, and They represent the contact point positions of the left and right grippers in the i-th grasp, and N is the batch size; c) Surface alignment constraint: The grasping point fits the object surface to reduce the spatial offset error. Specifically, to ensure that the predicted contact point falls close to the real object surface, a distance constraint based on the signed distance function (SDF) is introduced. Let SDF(u) represent the signed distance from point u to the object surface, and define the loss: Where d1 and d2 are the minimum and maximum acceptable contact distance thresholds, respectively, and δ1 and δ2 are balance coefficients. The regularization terms of the above constraints together define the complete physical constraint loss: Among them, α and β are adjustable weight hyperparameters.

8. The machine grasping learning method integrating shape features, contact modeling, and physical constraints according to claim 2 is characterized by: In step 7, the joint loss function includes a joint training objective function. Specifically, to achieve end-to-end training of differentiable grasp pose estimation, the grasp score, collision detection, and physical regularization terms are combined into a unified training objective function. The details are as follows: in: Grasp scoring loss, which measures the accuracy of grasp confidence prediction for the current pose; Collision detection loss, which uses binary cross entropy to supervise whether a collision occurs between the point cloud and the pose; Physical constraint regularization term, including friction closure constraint Geometric symmetry constraints Surface alignment constraints Defined as: In order to balance the impact of the three loss items on the total loss, a weighting coefficient is introduced, which can be adjusted according to the task requirements; Through optimization, the trained model can improve its grasp scoring and discrimination capabilities while learning to avoid collisions and meet physical stability requirements.

9. The machine grasping learning method integrating shape features, contact modeling, and physical constraints according to claim 2, characterized in that: Step 8 includes the inference and crawl execution process, specifically: In the deployment phase, the system of the machine grasping learning method receives the scene point cloud As input, the following steps are performed to generate and filter the grasping posture: Step B1, grasp candidate generation: Use the point cloud backbone network to output the graspability score s of each point graspness , through non-maximum suppression (NMS) and threshold screening, a set of candidate grasping center points {x j }; Step B2: Grasp parameter prediction: For each candidate center point x j , using the direction-aware local feature extraction module to predict the corresponding grasping posture parameters Step B3, collision judgment: for each candidate grasping posture G j , calculate the collision probability function C(P,G j ), filter out all candidates with collisions: Step B4: Grab the score and sort: for the candidate set without collision Calculate the crawl score Q(P,G j ), and finally select the grasping posture with the highest score as the output: Step B5, robot execution: set the optimal grasping posture parameter G * The data is transmitted to the robotic arm through the control interface to complete the grasping action.

10. The machine grasping learning method integrating shape features, contact modeling and physical constraints according to claim 2 is characterized by: The method is used for high-precision grasping planning tasks in industrial robots, service robots or other automation systems, and the working distance between the mechanical gripper and the grasping target ranges from 0.2m to 3m.

Images