Fine-grained contact representation dexterous hand grabbing method based on diffusion model

Abstract

The invention discloses a dexterous hand grabbing method based on fine-grained contact representation of a diffusion model, and particularly relates to the technical field of robots, which comprises the following steps: constructing a three-stage framework of contact graph generation, contact matrix prediction and grabbing posture generation, designing a finger separation contact graph generation model based on the diffusion model, predicting each fingertip contact area and direction from the object point cloud, and generating a fine-grained contact matrix graph; a direction consistency optimization algorithm is used, physical constraints and a contact diagram are fused, and it is ensured that the finger force application direction is matched with the normal of the surface of the object; the finger separation contact accuracy problem is solved through modes such as multi-task learning, physical robot experiments are carried out on platforms such as Shadow Hand to verify the actual grabbing effect, an independent contact probability matrix graph is generated for each finger, and the generated contact probability matrix graph fuses the object grabbing direction; a layered optimization strategy is adopted, learning aided optimization and lightweight model construction are reinforced, and the dynamic contact optimization efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robots, and in particular relates to a dexterous hand grasping method based on fine-grained contact representation of a diffusion model. Background Art

[0002] Currently, research on dexterous robotic grasping techniques, both domestically and internationally, is primarily divided into traditional approaches and those based on modern technologies. Traditional grasping methods rely on coarse-grained contact information, such as the overall contact area or key points, to plan grasping strategies. This approach has significant shortcomings when it comes to high-precision operations and is unable to meet the growing demand for refined manipulation. For example, when grasping tiny, delicate parts, it's impossible to precisely control the force and position of each finger, resulting in unstable grasping or damage to the object.

[0003] With the development of deep learning and sensor technology, learning-based methods and grasping technologies that utilize multimodal information have become research hotspots. Many learning-based methods use deep neural networks to learn grasping features and strategies from large amounts of data, and can directly predict grasping parameters or key information, such as contact points and contact maps. However, these methods still face many challenges. On the one hand, existing research lacks fine-grained predictions of the contact maps of individual fingers, making it difficult to fully utilize the advantages of multi-fingered dexterous hands. Each finger has a unique function during the grasping process, and the lack of fine-grained predictions can lead to poor coordination between fingers in complex tasks, affecting the grasping effect. On the other hand, existing grasp generation methods often generate grasps without detailed finger adjustments, making it difficult for robots to quickly adapt and find the optimal grasping posture when faced with objects of different shapes and sizes, resulting in poor performance on complex tasks. Summary of the Invention

[0004] The purpose of the present invention is to provide a dexterous hand grasping method based on fine-grained contact representation of the diffusion model. Through the fine-grained contact prediction framework of the diffusion model, the advantages of large model technology and diffusion model generation are deeply integrated to construct an object specification-aware grasping decision framework, which is committed to overcoming the difficulty of generating refined grasping gestures for objects of multiple specifications and realizing more intelligent, efficient and adaptable object grasping.

[0005] To achieve the above objectives, a dexterous hand grasping method based on fine-grained contact representation of a diffusion model is provided, which includes the following steps:

[0006] Step S1: Acquire the three-dimensional point cloud data of the target object.

[0007] Step S2: Use Blender software to preprocess the manipulator model and divide the manipulator area, extract the geometric features of each area and generate the hand point cloud feature vector.

[0008] Step S3: obtaining distance data between the object and each area of the hand based on the three-dimensional point cloud data of the target object and the hand point cloud feature vector, and preprocessing the three-dimensional point cloud data of the target object.

[0009] Step S4: Construct a lightweight diffusion model, input the preprocessed object 3D point cloud data and distance data, and combine time embedding and attention mechanism to generate the contact probability map of each finger.

[0010] Step S5: Fuse the contact probability map of each finger and the object semantic segmentation information to construct a fine-grained contact matrix.

[0011] Step S6: Based on the fine-grained contact matrix, the palm posture and finger joint angles are adjusted through hierarchical optimization, and the grasping action is dynamically optimized in combination with reinforcement learning.

[0012] Furthermore, the method for obtaining the three-dimensional point cloud data of the target object in step S1 specifically includes:

[0013] Use the depth camera to collect 3D point cloud data of the target object for Kinect Azure.

[0014] Based on the ContactPose dataset, different grasping postures of the robot are provided, which accurately represents the point cloud data volume and retains key features.

[0015] Furthermore, the method of using Blender software to pre-process the manipulator model and divide the manipulator area in step S2, and extracting geometric features of the hand and the object to generate point cloud feature vectors specifically includes:

[0016] The hand model was preprocessed using Blender software, and the robotic hand was divided into five independent regions: thumb, index finger, middle finger, ring finger, and little finger. The geometric features of each region were extracted to generate a hand feature point cloud.

[0017] Furthermore, the method of performing calculation based on the target object three-dimensional point cloud data and the hand point cloud feature vector in step S3 to obtain distance data between the object and each area of the hand, and preprocessing the target object three-dimensional point cloud data specifically includes:

[0018] The contact distance between each area of the hand and the object point cloud is calculated using the Euclidean distance formula to obtain the distance data between the object and each area of the hand, and the obtained distance data is trained using a data-driven method.

[0019] The preprocessing of the three-dimensional point cloud data of the target object is to use a data-driven method to perform ±30° random rotation and 0.8-1.2 times scaling on the three-dimensional point cloud data, and to perform horizontal flipping and brightness adjustment on the depth image.

[0020] Furthermore, the method of constructing a lightweight diffusion model in step S4, inputting preprocessed object 3D point cloud data and distance data, and combining time embedding and attention mechanism to generate each finger contact probability map specifically includes:

[0021] A lightweight diffusion model is constructed. The input includes preprocessed object point cloud data with dimensions N×6, including x / y / z coordinates and normal vectors nx / ny / nz, as well as calculated contact distance information. The object specification parameters are encoded into a 128-dimensional vector using Word2Vec. Through the temporal embedding module and the attention mechanism, a contact probability map of each finger with a resolution of 1024×1024 is generated. The ObMan dataset is used as the training benchmark, and the cross-entropy loss function is used to optimize the model. Its expression is:

[0022]

[0023] Among them, i is the finger index, j is the index of the point in the point cloud, and y i，j For real contact tags, Predict probabilities for the model.

[0024] At the same time, a joint angle association regularization term is introduced to constrain the spatial consistency of adjacent finger contact areas, and the joint angle association regularization term is determined according to the actual constraint conditions of the finger joint movement.

[0025] Furthermore, the method of fusing the finger contact probability maps and the object semantic segmentation information to construct a fine-grained contact matrix in step S5 specifically includes the following steps:

[0026] (1) Contact position encoding: The contact probability map is threshold segmented with a threshold of 0.7. The areas with probability values ≥ 0.7 are extracted as potential contact areas. The three-dimensional coordinates of the top 50 high-probability contact points are selected from them. The local feature vector of each point is extracted through the PointNet++ network to represent the geometric context information of the contact point.

[0027] (2) Normal matching calculation: define the fingertip direction vector, obtain the object surface normal vector through the target object point cloud normal vector estimation, calculate the cosine similarity between the fingertip direction vector and the object surface normal, set the threshold to 0.8, and encode it as 1 when the similarity is ≥ 0.8, otherwise it is 0, forming a 0-1 binary normal matching variable.

[0028] (3) Finger coordination constraint: According to the mechanical structure of the dexterous hand, a joint angle correlation model is established, and the thumb and index finger grip angle θ1∈[30°,120°] and the other finger bending angles θ2∈

[0029] [0°, 90°] and the angle difference between adjacent finger joints is ≤ 30°, generating the joint angle feasible domain matrix.

[0030] (4) Constructing a fine-grained contact matrix: Construct a 6×(3+1+1)-dimensional matrix, fill the corresponding rows with the average three-dimensional coordinates, normal matching degree, and joint constraint status of the top-50 high-probability contact points of each finger, and fill the non-contact category rows with default values;

[0031] The object semantic segmentation information is the semantic level classification and labeling of different functional components or structural areas of the target object through computer vision technology, wherein the information provides task-oriented constraints for the grasping strategy.

[0032] Furthermore, the method described in step S6 for adjusting the palm posture and finger joint angles through hierarchical optimization based on the fine-grained contact matrix and dynamically optimizing the grasping action in combination with reinforcement learning specifically includes:

[0033] (1) Coarse adjustment of palm posture: Control the wrist joint motor to adjust the palm toward the center of the object, with the error controlled within ±5mm.

[0034] (2) Finger joint fine-tuning: Based on the fine-grained contact matrix feedback, the joint angles of each key finger point are fine-tuned to ensure that the angle between the fingertip force direction and the object normal is less than 30°, thus generating the optimal grasping posture.

[0035] (3) Optimization by deep reinforcement learning: In the deep reinforcement learning framework, the state space S contains the geometric features of the object, the finger joint angles, and the contact pressure; the adjustment range of each joint angle in the action space A is ±10°, with an accuracy of 0.5°; the reward function is designed as:

[0036] R=0.6R success +0.3R contact +0.1R time

[0037] Among them, R success To capture the binary reward, R contact is the normalized value of the contact area, R time Penalty for time-consuming crawling;

[0038] The PPO algorithm is used to train the policy network to achieve the optimal grasping joint value generation in dynamic environments.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. Enhanced fine-grained contact representation: A diffusion model is used to generate an independent contact probability map for each finger, with a resolution of 1024×1024. The contact area positioning accuracy is improved to ±2mm, and the misjudgment rate of single-finger contact areas is reduced by 40% compared to traditional global contact maps. By integrating object material, size, and other specification information with semantic labels of functional components, contact map generation becomes task-oriented. For example, contact points are preferentially generated in the handle area, making it suitable for grasping tool-type objects.

[0041] 2. Improved dynamic optimization efficiency: The hierarchical optimization strategy reduces the degree-of-freedom search space from 60 dimensions (for the entire hand) to 20 dimensions (3 dimensions for the palm and 17 dimensions for the fingers), reducing computational time by 65% and enabling real-time grasp planning (response time <200ms). The reinforcement learning framework, through 100,000 simulation training runs, has increased the grasping success rate under dynamic interference from 72% to 89%, demonstrating a significant advantage in grasping curved objects (such as spheres and cones).

[0042] 3. Generalization and Robustness: Multimodal data fusion (point cloud, image, and tactile) and domain randomization enhancement (simulating different lighting and noise environments) enable the model to achieve a grasping success rate of over 85% for both industrial parts (such as gears and bolts) and everyday objects (such as cups and mobile phones). Finger coordination constraints and normal matching verification effectively prevent object slippage and joint overruns during grasping, improving operational stability in complex scenarios.

[0043] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 is a flow chart of the crawling method of the present invention;

[0046] Figure 2 This is a diagram of the hierarchical optimization and reinforcement learning framework of the present invention;

[0047] Figure 3 A diagram of the geometric structure of the 3D model of the present invention;

[0048] Figure 4 A visual identification diagram of the fingertip contact points of the hand model of the present invention;

[0049] Figure 5 It is the whole colored picture of the finger of the present invention;

[0050] Figure 6 This is the interaction posture diagram of the hand and cup model of the present invention. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention. Unless otherwise specified, the technical means used in the embodiments are conventional means well known to those skilled in the art.

[0052] The embodiment of the present invention provides a dexterous hand grasping method based on fine-grained contact representation of a diffusion model. Figure 1 , Figure 1 This is a flowchart of a dexterous hand grasping method based on fine-grained contact representation of a diffusion model provided by an embodiment of the present invention. The specific implementation steps are as follows:

[0053] Step S1: Acquire the three-dimensional point cloud data of the target object.

[0054] Step S2: Use Blender software to preprocess the manipulator model and divide the manipulator area, extract the geometric features of each area and generate the hand point cloud feature vector.

[0055] Step S3: Calculate based on the three-dimensional point cloud data of the target object and the hand point cloud feature vector to obtain the distance data between the object and each area of the hand, and pre-process the three-dimensional point cloud data of the target object.

[0056] Step S4: Construct a lightweight diffusion model, input the preprocessed object 3D point cloud data and distance data, and combine time embedding and attention mechanism to generate the contact probability map of each finger.

[0057] Step S5: Fuse the contact probability map of each finger and the object semantic segmentation information to construct a fine-grained contact matrix.

[0058] Step S6: Based on the fine-grained contact matrix, the palm posture and finger joint angles are adjusted through hierarchical optimization, and the grasping action is dynamically optimized in combination with reinforcement learning.

[0059] Optionally, the method for obtaining the three-dimensional point cloud data of the target object in step S1 specifically includes:

[0060] Use the depth camera to collect 3D point cloud data of the target object for Kinect Azure (including x / y / z coordinates and normal vectors nx / ny / nz, with a dimension of N×6).

[0061] Based on the ContactPose dataset, different grasping postures of the robot are provided, which accurately represents the point cloud data volume and retains key features.

[0062] Optionally, the method of using Blender software to pre-process the manipulator model and divide the manipulator area, and extracting geometric features of the hand and the object to generate point cloud feature vectors in step S2 specifically includes:

[0063] The hand model was preprocessed using Blender software and the robotic hand was divided into five independent regions: thumb, index finger, middle finger, ring finger, and little finger. As shown in the annotation logic of the ContactMapDataset class in the experimental verification and parameter configuration, the geometric features (such as curvature and convexity) of each region were extracted to generate a hand feature point cloud.

[0064] Optionally, the method of performing calculation based on the target object three-dimensional point cloud data and the hand point cloud feature vector in step S3 to obtain distance data between the object and each area of the hand, and preprocessing the target object three-dimensional point cloud data specifically includes:

[0065] The contact distance between each area of the hand and the object point cloud is calculated using the Euclidean distance formula to obtain the distance data between the object and each area of the hand, and the obtained distance data is trained using a data-driven method.

[0066] The preprocessing of the target object's three-dimensional point cloud data is to use a data-driven method to randomly rotate the three-dimensional point cloud data by ±30°, scale it by 0.8-1.2 times, horizontally flip the depth image, and adjust the brightness. The specific parameters are as shown in the generate_synthetic_data function in the experimental verification and parameter configuration, generating diverse training samples to improve the model's generalization ability.

[0067] Optionally, the method of constructing a lightweight diffusion model in step S4, inputting preprocessed object 3D point cloud data and distance data, and combining time embedding and attention mechanism to generate a contact probability map for each finger specifically includes:

[0068] A lightweight diffusion model is constructed. The input includes preprocessed object point cloud data with dimensions N×6, including x / y / z coordinates and normal vectors nx / ny / nz, as well as calculated contact distance information. The object specification parameters are encoded into a 128-dimensional vector using Word2Vec. Through the temporal embedding module and the attention mechanism, a contact probability map of each finger with a resolution of 1024×1024 is generated. The ObMan dataset is used as the training benchmark, and the cross-entropy loss function is used to optimize the model. Its expression is:

[0069]

[0070] Among them, i is the finger index, j is the index of the point in the point cloud, and y i，j For real contact tags, Predict probabilities for the model.

[0071] At the same time, a joint angle association regularization term is introduced to constrain the spatial consistency of adjacent finger contact areas, and the joint angle association regularization term is determined according to the actual constraint conditions of the finger joint movement.

[0072] Specifically, a lightweight diffusion model is constructed: the main body of the model is ContactConditionedUNet, which extends UNet2DConditionModel and introduces a sensor condition encoder. It inputs preprocessed object point cloud data (N×6) and a 128-dimensional object specification vector generated by Word2Vec (including shape, size, and material semantic encoding). Combined with time embedding (Time Embedding) and attention mechanism (such as the CrossAttnDownBlock2D module in experimental verification and parameter configuration), a 100-step denoising process is used to generate a contact probability map of each finger (resolution 1024×1024, such as the ContactDiffusionGenerator inference process in experimental verification and parameter configuration).

[0073] Training and regularization optimization: Based on the ObMan dataset, the actual contact area of each finger is manually marked as a supervisory signal (such as the ContactMapDataset class simulates the sensor data logic in the experimental verification and parameter configuration), and the cross entropy loss function is used (the formula is as shown in the experimental verification and parameter configuration). loss = mse_loss + 0.3 * contact_loss) to optimize model parameters; add joint angle association regularization terms in the output layer (such as the encoding of joint constraints by sensor_encoder in experimental verification and parameter configuration) to ensure the spatial consistency of adjacent finger contact areas and avoid penetration conflicts (such as the minimum distance constraint between the thumb and index finger contact areas).

[0074] Optionally, the method of fusing the finger contact probability maps and the object semantic segmentation information to construct a fine-grained contact matrix in step S5 specifically includes the following steps:

[0075] (1) Contact position encoding: The contact probability map is threshold segmented with a threshold value of 0.7. The area with a probability value ≥ 0.7 is extracted as the potential contact area. The three-dimensional coordinates of the top 50 high-probability contact points are selected from them. The local feature vector of each point is extracted through the PointNet++ network to represent the geometric context information of the contact point (such as curvature, neighborhood structure, etc.).

[0076] (2) Normal matching calculation: define the fingertip direction vector (e.g., the thumb tip points to the palm, and the other fingertips are perpendicular to the fingertip plane), obtain the object surface normal vector (nx, ny, nz) through the target object point cloud normal vector estimation (e.g., the calculation method based on K nearest neighbor), calculate the cosine similarity between the fingertip direction vector and the object surface normal, set the threshold to 0.8, and encode it as 1 (valid contact) when the similarity is ≥ 0.8, otherwise it is 0, forming a 0-1 binary normal matching variable.

[0077] (3) Finger coordination constraint: Based on the mechanical structure of the dexterous hand, a joint angle association model is established. For example, the thumb and index finger grip angle θ1∈[30°,120°], the other finger bending angles θ2∈[0°,90°] are set, and the angle difference between adjacent finger joints is ≤30°. The joint angle feasible domain matrix is generated (the elements are 0 or 1, indicating whether the joint angles meet the physical constraints).

[0078] (4) Construct a fine-grained contact matrix: Construct a 6×(3+1+1)-dimensional matrix, in which 6 rows correspond to the thumb, index finger, middle finger, ring finger, little finger and non-contact categories, and 5 columns correspond to 3D coordinates (X, Y, Z), normal matching (0 / 1), and joint constraint status (0 / 1) respectively; the average 3D coordinates, normal matching (the result of the majority of valid points) and joint constraint status (based on whether the current joint angle is within the feasible domain) of the top-50 high-probability contact points of each finger are filled into the corresponding rows, and the non-contact category rows are filled with default values (such as the coordinates are the center of mass of the object, the normal matching and joint constraint are 0).

[0079] Definition and fusion logic of object semantic segmentation information:

[0080] (1) Object semantic segmentation information: refers to the semantic classification and labeling of different functional components or structural areas of the target object through computer vision technology, such as distinguishing key parts such as handles, planes, protrusions, and depressions; this information provides task-oriented constraints for grasping strategies, such as preferentially generating contact points in the "handle" area to adapt to the grasping needs of tool-type objects, or avoiding "vulnerable parts" to prevent damage to the object.

[0081] (2) Fusion method:

[0082] Semantic weight modulation: Based on the semantic segmentation results of the object (such as the handle area weight ω = 1.2, the plane area weight ω = 0.8), the original probability value of the contact probability map is weighted and adjusted. The formula is: adjusted probability = original probability × semantic category weight, which enhances the contact point priority of the target functional area.

[0083] Semantic filtering mechanism: Filters contact points in non-target areas based on semantic labels (e.g., prohibiting contact points from being generated on "fragile surfaces"), re-screens the top 50 high-probability contact points, and updates the coordinate and normal matching in the contact matrix to ensure that the contact distribution meets the functional characteristics of the object and the requirements of the grasping task.

[0084] Through the above steps, the fine-grained contact matrix integrates the geometric characteristics of the contact position, the physical constraints of the normal direction, the movement restrictions of the finger joints, and the semantic guidance of the functional structure of the object, forming multi-dimensional constraint information, providing precise input for subsequent grasping posture optimization, and significantly improving the grasping stability and task adaptability of the dexterous hand for complex objects.

[0085] Alternatively, as Figure 2 As shown, the method described in step S6 for adjusting the palm posture and finger joint angles through hierarchical optimization based on the fine-grained contact matrix and dynamically optimizing the grasping action in combination with reinforcement learning specifically includes:

[0086] (1) Rough adjustment of palm posture: Figure 2 The "palm coarse adjustment" module on the left uses the wrist joint motor to adjust the palm toward the center of the object, and uses position sensor feedback control to ensure that the error between the palm center and the object center is within ±5mm (as specified in the manipulator control accuracy parameter in experimental verification and parameter configuration), reducing the degree of freedom search space from 60 dimensions to 20 dimensions.

[0087] (2) Finger joint fine-tuning: Figure 2 The "finger fine-tuning" module on the right adjusts the angles of the key joints of each finger (fingertips, proximal phalangeal joints) based on the high-probability contact points of the contact matrix C through the inverse kinematics algorithm to ensure that the angle between the force direction of the fingertip and the object normal is less than 30° (such as the specific application of the "normal matching degree" constraint in experimental verification and parameter configuration).

[0088] (3) Optimization by deep reinforcement learning: In the deep reinforcement learning framework, the state space S contains the geometric features of the object (point cloud curvature, semantic labels), 15-dimensional finger joint angles (3 joints per finger), 5-dimensional contact pressure (such as Figure 2The range of the joint angle adjustment in the action space A is ±10°, with an accuracy of 0.5°, as shown in the parameter settings of the generate method in Experimental Verification and Parameter Configuration. The reward function is designed as follows:

[0089] R=0.6R success +0.3R contact +0.1R time

[0090] Among them, R success To capture the binary reward, R contact is the normalized value of the contact area, R time Penalty for time-consuming crawling.

[0091] The PPO algorithm is used to train the strategy network to achieve the optimal grasping joint value generation in a dynamic environment. In this way, the PPO algorithm is used to train the strategy network (such as the train_model function process in the experimental verification and parameter configuration), and the contact point distribution is optimized through 100,000 simulation trainings to achieve the optimal joint value generation in a dynamic environment (such as automatically adjusting the finger bending curvature for spherical objects, corresponding to Figure 2 (see the arrow logic in “Generating Optimal Grasping Joint Values in Dynamic Environments”).

[0092] Figure 2 (Hierarchical optimization framework): Shows the two-layer structure of "palm coarse adjustment - finger fine adjustment". The reinforcement learning module optimizes the distribution of contact points through the state-action-reward loop, which is completely consistent with the mathematical model in step 6.

[0093] Figure 3 Diagram of the geometric structure for 3D modeling: This shows the process of constructing the three-dimensional geometric model of the target object and the dexterous hand; using Blender software, the robotic hand (such as Shadow Hand) is divided into five independent regions (thumb, index finger, etc.), and the geometric features of each region (such as curvature and convexity) are extracted to generate the hand point cloud feature vector; at the same time, point cloud modeling is performed on the target object (such as a cup and a gear) to provide a geometric basis for subsequent contact distance calculation and contact map generation; the division boundaries of each hand region and the key parts of the object (such as the handle and curved surface) are marked in the figure, reflecting the specific implementation of geometric feature extraction and preprocessing in steps S2 and S3.

[0094] Figure 4This is a visual identification map of the fingertip contact points of the hand model: the contact probability map visualization result of a single finger (such as the index finger). In the 1024×1024 resolution contact probability map generated by the lightweight diffusion model, the highlighted area indicates the area with high fingertip contact probability (threshold ≥ 0.7), corresponding to the output result of "Generate contact probability map for each finger" in step S4; the contact probability values are marked with different colors or brightness in the map to intuitively show the potential contact position between the fingertip and the object surface, providing input data for the contact position encoding and normal matching calculation in step S5.

[0095] Figure 5 Coloring the fingers as a whole: This diagram shows the overall coloring effect of each finger of the dexterous hand, with different colors corresponding to different fingers (such as red for the thumb and blue for the index finger) to distinguish the independent contact areas of each finger. Combined with the logic of "fusing the contact probability maps of each finger" in step S5, this diagram embodies the basis for constructing a fine-grained contact matrix - each finger generates an independent contact probability map, and adjusts the contact point priority through semantic segmentation information (such as prioritizing the grip handle area). The diagram also marks the joint angle constraint range of each finger (such as the thumb and index finger grip angle θ1∈[30°,120°]), corresponding to the "finger collaborative constraint" mechanism in step S5.

[0096] Figure 6 This is the interaction posture diagram of the hand and cup model: it shows the actual posture of the dexterous hand grasping the cup based on the fine-grained contact matrix, reflecting the result of "layer-by-layer optimization adjustment of palm posture and finger joint angles" in step S6; in the figure, the palm is facing the center of the object (coarse adjustment error ≤±5mm), and each finger adjusts the joint angle (such as the index finger bending angle θ2∈[0°,90°]) according to the feedback of the contact matrix to ensure that the angle between the fingertip force direction and the normal of the cup surface is less than 30°; in addition, the distribution of contact points after reinforcement learning optimization is marked in the figure (such as the high-probability contact points between the thumb and index finger in the handle area of the cup), verifying the effect of dynamic optimization on improving grasping stability.

[0097] Experimental verification and parameter configuration;

[0098] Specifically, the diffusion model architecture is as follows:

[0099] The steps to customize the contact map dataset loader (classContactMapDataset(Dataset)) are as follows:

[0100]

[0101] The steps to simulate physical sensor data (replace with real data in actual applications) are as follows:

[0102]

[0103]

[0104] #The steps to obtain the corresponding sensor data are as follows:

[0105]

[0106] The steps to generate synthetic training data (replace with real data collection for actual application) are as follows:

[0107] os.makedirs(output_dir,exist_ok=True)

[0108] for iinrange(num_samples):

[0109] #The steps to create a blank image are as follows:

[0110] img=Image.new('L',(512,512),0)

[0111] draw = ImageDraw.Draw(img)

[0112] #The steps to randomly generate contact points are as follows:

[0113]

[0114] #The steps to draw Gaussian distribution contact points are as follows:

[0115]

[0116]

[0117] #In the data preparation module, the ContactMapDataset class is a custom dataset loader.

[0118] #In the __init__ method, collect image paths, define image transformations, and simulate sensor data.

[0119] #__len__ and __getitem__ methods are used to return the number of samples and obtain specific sample data respectively;

[0120] The #generate_synthetic_data function is used to generate synthetic training data, create an output directory, and loop to generate images with random contact points for saving.

[0121] (ContactMapDataset class): Displays the custom dataset loader structure, through _getitem_The method fuses the contact map image with the sensor data (simulating the 6-dimensional vector output of the physical sensor, such as Figure 3 sensor_data input).

[0122] The steps to extend UNet to support sensor condition input (classContactConditionedUNet(UNet2DConditionModel)) are as follows:

[0123]

[0124]

[0125] #The steps of sensor condition encoder are as follows:

[0126]

[0127] #The steps to encode sensor data are as follows:

[0128] sensor_emb=self.sensor_encoder(sensor_data)

[0129] sensor_emb=sensor_emb.view(-1,1024,1,1)

[0130] #The steps of the original UNet processing are as follows:

[0131] x = self.conv_in(x)

[0132] #The steps to inject sensor conditions in the middle layer are as follows:

[0133]

[0134] #...(The rest of the UNet processing flow) steps are as follows:

[0135] returnsuper().forward(x,timesteps)

[0136] #Model architecture part, ContactConditionedUNet class extends

[0137] UNet2DConditionModel,

[0138] The #__init__ method initializes the UNet parameters and defines the sensor-conditioned encoder.

[0139] The #forward method first encodes the sensor data, convolves the input image, injects the sensor conditions into the UNet middle layer, and completes the remaining processing flow.

[0140] (ContactConditionedUNet network): Inject sensor conditions into the middle layer of UNet (for example, add sensor_emb input to the CrossAttnDownBlock2D module in the figure) to achieve the fusion of contact map generation and joint constraints.

[0141] The steps of deftrain_model() are as follows:

[0142] #The initialization steps are as follows:

[0143] device=torch.device('cuda'if torch.cuda.is_available()else'cpu')

[0144] #The steps to generate synthetic data (actual application uses real data) are as follows:

[0145] generate_synthetic_data(num_samples=1000)

[0146] #The steps of the data set are as follows:

[0147] dataset=ContactMapDataset('. / contact_data')

[0148] dataloader=DataLoader(dataset,batch_size=4,shuffle=True)

[0149] #The steps of the model are as follows:

[0150] model=ContactConditionedUNet().to(device)

[0151] noise_scheduler=DDPMScheduler(num_train_timesteps=1000)

[0152] #The optimizer steps are as follows:

[0153] optimizer=torch.optim.AdamW(model.parameters(),lr=1e-4)

[0154] lr_scheduler=get_cosine_schedule_with_warmup(

[0155] optimizer,

[0156] num_warmup_steps=500,

[0157] num_training_steps=len(dataloader)*100 )

[0159] #The steps of the training loop are as follows:

[0160] for epoch inrange(100):

[0161] progress_bar=tqdm(dataloader,desc=f'Epoch{epoch}')

[0162] for batch in progress_bar:

[0163] clean_images=batch['image'].to(device)

[0164] sensor_data=batch['sensor'].to(device)

[0165] #The steps to add noise are as follows:

[0166] noise=torch.randn_like(clean_images)

[0167] timesteps = torch.randint(

[0168] 0,noise_scheduler.num_train_timesteps,

[0169] (clean_images.shape[0],),device=device )

[0171] noise_images=noise_scheduler.add_noise(clean_images,noise,timesteps)

[0172] #The steps to predict noise are as follows:

[0173] noise_pred=model(noisy_images,timesteps,sensor_data).sample

[0174] #The steps to calculate the loss (adding physical constraints) are as follows:

[0175] mse_loss=F.mse_loss(noise_pred,noise)

[0176] #The steps for contact force distribution constraint are as follows:

[0177] contact_loss=F.mse_loss(

[0178] torch.sigmoid(noise_pred).mean(dim=[2,3]),

[0179] sensor_data[:,:1].expand(-1,noise_pred.shape[1]) )

[0181] loss=mse_loss+0.3*contact_loss

[0182] #The steps of back propagation are as follows:

[0183] optimizer.zero_grad()

[0184] loss.backward()

[0185] optimizer.step()

[0186] lr_scheduler.step()

[0187] progress_bar.set_postfix(loss=loss.item())

[0188] #The steps to save the model are as follows:

[0189] torch.save(model.state_dict(),'contact_diffusion.pth'=

[0190] #The training process is implemented through the train_model function, which first initializes the device, generates synthetic data, creates a dataset and loader,

[0191] #Then initialize the model, noise scheduler, optimizer, etc.; in the training loop, load the data and add noise. After the model predicts the noise, calculate the loss including the mean square error and contact force distribution constraint.

[0192] #Update the parameters and learning rate through back propagation, and finally save the trained model.

[0193] The steps of the """production environment inference pipeline""" (class ContactDiffusionGenerator) are as follows:

[0194] def__init__(self,model_path='contact_diffusion.pth'):

[0195] self.device=torch.device('cuda'if torch.cuda.is_available()else'cpu')

[0196] #The steps to load the model are as follows:

[0197] self.model=ContactConditionedUNet().to(self.device)

[0198] self.model.load_state_dict(torch.load(model_path))

[0199] self.model.eval()

[0200] self.noise_scheduler=DDPMScheduler(num_train_timesteps=1000)

[0201] #The steps for optimizing settings are as follows:

[0202] if torch.cuda.is_available():

[0203] self.model=torch.compile(self.model)

[0204] def generate(self,sensor_input,num_inference_steps=50):

[0205] """Generate contact map"""

[0206] The steps with torch.no_grad() are as follows:

[0207] #Prepare input steps as follows:

[0208] sensor_tensor=torch.FloatTensor(sensor_input).unsqueeze(0).to(self.device)

[0209] x=torch.randn((1,1,512,512),device=self.device)

[0210] #The steps of gradual denoising are as follows:

[0211]

[0212] #The post-processing steps are as follows:

[0213] image=(x.clamp(-1,1)+1) / 2#[-1,1]->[0,1]

[0214] image=image.squeeze().cpu().numpy()

[0215] return Image.fromarray((image*255).astype(np.uint8))

[0216] #The reasoning application is done by the ContactDiffusionGenerator class,

[0217] The #__init__ method initializes the device, loads the pre-trained model and sets the evaluation mode, initializes the noise scheduler, and compiles the optimized model in a GPU environment.

[0218] The #generate method receives sensor input and the number of inference steps, prepares the data, and then gradually removes noise. After post-processing, it converts the generated image into a PIL image and returns it.

[0219] When verifying the Shadow Hand platform, the hardware parameters mentioned above

[0220] The ContactDiffusionGenerator class (which supports GPU acceleration and model compilation optimization) has achieved indicators such as "contact area positioning accuracy of ±2mm" and "computational time reduction of 65%" through the above model architecture and algorithm flow.

[0221] The invention provides a fine-grained matrix contact map for each finger, annotating each finger with a different color. This intuitively introduces a new concept, D(F, O), for the use of the manipulator. For the contact points between the fingers and the object, Euclidean distance is used to represent the distance of contact, and points with high contact probability are intuitively displayed in the contact map. Finger coordination constraints are introduced, and joint angle correlation regularization terms are added to the model output layer to ensure spatial consistency of adjacent finger contact areas and avoid penetration conflicts.

[0222] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A dexterous hand grasping method based on fine-grained contact representation of diffusion model, characterized by: The crawling method comprises the following steps: Step S1: Acquire three-dimensional point cloud data of the target object; Step S2: Use Blender software to preprocess the manipulator model and divide the manipulator area, extract the geometric features of each area to generate the hand point cloud feature vector; Step S3: obtaining distance data between the object and each area of the hand based on the three-dimensional point cloud data of the target object and the hand point cloud feature vector, and preprocessing the three-dimensional point cloud data of the target object; Step S4: Construct a lightweight diffusion model, input the pre-processed object 3D point cloud data and distance data, and combine time embedding and attention mechanism to generate the contact probability map of each finger; Step S5: Fusing the contact probability map of each finger and the object semantic segmentation information to construct a fine-grained contact matrix; Step S6: Based on the fine-grained contact matrix, the palm posture and finger joint angles are adjusted through hierarchical optimization, and the grasping action is dynamically optimized in combination with reinforcement learning.

2. The dexterous hand grasping method based on fine-grained contact representation of the diffusion model according to claim 1 is characterized in that: The method for obtaining the three-dimensional point cloud data of the target object in step S1 specifically includes: Use the depth camera to collect 3D point cloud data of the target object for Kinect Azure; Based on the ContactPose dataset, different grasping postures of the robot are provided, which accurately represents the point cloud data volume and retains key features.

3. The dexterous hand grasping method based on fine-grained contact representation of the diffusion model according to claim 1 is characterized in that: The method described in step S2 of preprocessing the manipulator model using Blender software and dividing the manipulator area, extracting the geometric features of the hand and the object to generate the point cloud feature vector specifically includes: The hand model was preprocessed using Blender software, and the robotic hand was divided into five independent regions: thumb, index finger, middle finger, ring finger, and little finger. The geometric features of each region were extracted to generate a hand feature point cloud.

4. The dexterous hand grasping method based on fine-grained contact representation of a diffusion model according to claim 1 is characterized in that: The method of obtaining the distance data between the object and each area of the hand based on the three-dimensional point cloud data of the target object and the hand point cloud feature vector in step S3, and preprocessing the three-dimensional point cloud data of the target object specifically includes: The contact distance between each area of the hand and the object point cloud is calculated using the Euclidean distance formula to obtain the distance data between the object and each area of the hand. The obtained distance data is then trained using a data-driven method. The preprocessing of the three-dimensional point cloud data of the target object is to use a data-driven method to perform ±30° random rotation and 0.8-1.2 times scaling on the three-dimensional point cloud data, and to perform horizontal flipping and brightness adjustment on the depth image.

5. The dexterous hand grasping method based on fine-grained contact representation of a diffusion model according to claim 1 is characterized in that: The method of constructing a lightweight diffusion model in step S4, inputting preprocessed object 3D point cloud data and distance data, and combining time embedding and attention mechanism to generate a contact probability map for each finger specifically includes: A lightweight diffusion model is constructed. The input includes preprocessed object point cloud data with dimensions N×6, including x / y / z coordinates and normal vectors nx / ny / nz, as well as calculated contact distance information. The object specification parameters are encoded into a 128-dimensional vector using Word2Vec. A time embedding module and an attention mechanism are used to generate a contact probability map for each finger with a resolution of 1024×1024. The ObMan dataset is used as the training benchmark, and the cross-entropy loss function is used to optimize the model. Its expression is: Among them, i is the finger index, j is the index of the point in the point cloud, and y i，j For real contact tags, Predict probabilities for the model; At the same time, a joint angle association regularization term is introduced to constrain the spatial consistency of adjacent finger contact areas, and the joint angle association regularization term is determined according to the actual constraint conditions of the finger joint movement.

6. The dexterous hand grasping method based on fine-grained contact representation of a diffusion model according to claim 1 is characterized in that: The method of fusing the finger contact probability map and the object semantic segmentation information to construct a fine-grained contact matrix in step S5 specifically includes the following steps: (1) Contact position encoding: The contact probability map is segmented by threshold, with the threshold set to 0.

7. The regions with probability values ≥ 0.7 are extracted as potential contact regions. The 3D coordinates of the top 50 high-probability contact points are selected from them. The local feature vectors of each point are extracted through the PointNet++ network to represent the geometric context information of the contact points. (2) Normal matching calculation: define the fingertip direction vector, obtain the object surface normal vector through the target object point cloud normal vector estimation, calculate the cosine similarity between the fingertip direction vector and the object surface normal, set the threshold to 0.8, and encode it as 1 when the similarity is ≥ 0.8, otherwise it is 0, forming a 0-1 binary normal matching variable; (3) Finger coordination constraint: Based on the mechanical structure of the dexterous hand, a joint angle correlation model is established. The thumb and index finger grip angle θ1∈[30°,120°], the other finger bending angles θ2∈[0°,90°] are set, and the angle difference between adjacent finger joints is ≤30°. The joint angle feasible domain matrix is generated. (4) Constructing a fine-grained contact matrix: Construct a 6×(3+1+1)-dimensional matrix, fill the corresponding rows with the average three-dimensional coordinates, normal matching degree, and joint constraint status of the top-50 high-probability contact points of each finger, and fill the non-contact category rows with default values; The object semantic segmentation information is the semantic level classification and labeling of different functional components or structural areas of the target object through computer vision technology, wherein the information provides task-oriented constraints for the grasping strategy.

7. The dexterous hand grasping method based on fine-grained contact representation of a diffusion model according to claim 1 is characterized in that: The method described in step S6 of adjusting the palm posture and finger joint angles through hierarchical optimization based on the fine-grained contact matrix and dynamically optimizing the grasping action in combination with reinforcement learning specifically includes: (1) Coarse adjustment of palm posture: Control the wrist joint motor to adjust the palm toward the center of the object, with the error controlled within ±5mm; (2) Finger joint fine-tuning: Based on fine-grained contact matrix feedback, the joint angles of each key finger point are fine-tuned to ensure that the angle between the fingertip force direction and the object normal is less than 30°, generating the optimal grasping posture; (3) Optimization by deep reinforcement learning: In the deep reinforcement learning framework, the state space S contains the geometric features of the object, the finger joint angles, and the contact pressure; the adjustment range of each joint angle in the action space A is ±10°, with an accuracy of 0.5°; the reward function is designed as: <h2 style=";text-align:left;direction:ltr">R=0.6R<h2 style=";text-align:left;direction:ltr"> success <h2 style=";text-align:left;direction:ltr"> +0.3R<h2 style=";text-align:left;direction:ltr"> contact <h2 style=";text-align:left;direction:ltr"> +0.1R<h2 style=";text-align:left;direction:ltr"> time Among them, R success To capture the binary reward, R contact is the normalized value of the contact area, R time Penalty for time-consuming crawling; The PPO algorithm is used to train the policy network to achieve the optimal grasping joint value generation in dynamic environments.

Images