A dexterous hand grasping method based on fine-grained contact representation of diffusion model

Abstract

The application discloses a dexterous hand grasping method based on a fine-grained contact representation of a diffusion model, and particularly relates to the technical field of robots, and comprises the following steps: constructing a three-stage framework of "contact map generation-contact matrix prediction-grasping pose generation", designing a diffusion model-based interdigital contact map generation model, predicting the contact area and direction of each fingertip from an object point cloud, and generating a fine-grained contact matrix map; using a direction consistency optimization algorithm, fusing physical constraints and a contact map, and ensuring that the force direction of the fingers matches the surface normal of the object; solving the interdigital contact accuracy problem through multi-task learning and other methods, performing physical robot experiments on platforms such as Shadow Hand to verify the actual grasping effect, generating an independent contact probability matrix map for each finger, and the generated contact probability matrix map fuses the direction of the grasped object; and adopting a hierarchical optimization strategy, reinforcement learning assisted optimization and lightweight model construction, and improving the dynamic contact optimization efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robots, and particularly relates to a dexterous hand grasping method based on fine-grained contact representation of a diffusion model. BACKGROUND

[0002] At present, the research on dexterous hand grasping technology of robots at home and abroad mainly includes traditional methods and methods based on modern technology. The traditional grasping method mainly depends on coarse-grained contact information, such as the overall contact area or key points, to plan the grasping strategy. This method has obvious shortcomings when facing high-precision operation requirements, and it is difficult to meet the increasing demand for fine operation. For example, in the grasping of small and precise parts, it is difficult to accurately control the force and position of each finger, resulting in unstable grasping or damage to the object.

[0003] With the development of deep learning and sensor technology, learning-based methods and grasping technology using multi-modal information have become a research hotspot. Many learning-based methods learn grasping features and strategies from a large amount of data through deep neural networks, and can directly predict grasping parameters or key information such as contact points and contact maps. However, these methods still face many challenges. On the one hand, existing research lacks fine-grained prediction of individual finger contact maps, making it difficult to fully utilize the advantages of multi-fingered dexterous hands. Each finger has a unique function in the grasping process, and the lack of fine-grained prediction will result in poor coordination between fingers in complex tasks, affecting the grasping effect. On the other hand, existing grasp generation methods often generate grasps without detailed finger adjustments, making it difficult for robots to quickly adapt and find the best grasping pose when facing objects of different shapes and sizes, and their performance on complex tasks is not satisfactory. SUMMARY

[0004] The purpose of the present application is to provide a dexterous hand grasping method based on fine-grained contact representation of a diffusion model, which deeply integrates large model technology and diffusion model generation advantages through a fine-grained contact prediction framework of the diffusion model, constructs an object specification perception grasping decision framework, and strives to solve the problem of generating fine-grained grasping gestures for multi-specification objects, and realizes more intelligent, efficient and adaptable object grasping.

[0005] In order to achieve the above purpose, a dexterous hand grasping method based on fine-grained contact representation of a diffusion model is provided, which includes the following steps:

[0006] Step S1: Obtain the three-dimensional point cloud data of the target object.

[0007] Step S2: preprocess the robot hand model using Blender software and divide the robot hand area, and extract the geometric features of each area to generate a hand point cloud feature vector.

[0008] Step S3: Based on the target object three-dimensional point cloud data and hand point cloud feature vector, the distance data of each region of the object and the hand is obtained, and the target object three-dimensional point cloud data is preprocessed.

[0009] Step S4: A lightweight diffusion model is constructed, the preprocessed object three-dimensional point cloud data and distance data are input, and each finger contact probability map is generated by combining time embedding and attention mechanism.

[0010] Step S5: A fine-grained contact matrix is constructed by fusing each finger contact probability map and object semantic segmentation information.

[0011] Step S6: Based on the fine-grained contact matrix, the palm posture and finger joint angle are adjusted through hierarchical optimization, and the grasping action is dynamically optimized combined with reinforcement learning.

[0012] Further, the method for obtaining the target object three-dimensional point cloud data in step S1 specifically includes:

[0013] The target object three-dimensional point cloud data is collected by using a depth camera, i.e., Kinect Azure.

[0014] Based on the ContactPose dataset, different grasping postures of the robot hand are provided, which accurately represent the amount of point cloud data and retain key features.

[0015] Further, the method for preprocessing the robot hand model and dividing the robot hand regions by using Blender software in step S2 to extract the geometric features of the hand and the object to generate the point cloud feature vector specifically includes:

[0016] The hand model is preprocessed by using Blender software, the robot hand is divided into five independent regions, i.e., thumb, index finger, middle finger, ring finger, and little finger, the geometric features of each region are extracted to generate the hand feature point cloud.

[0017] Further, the method for calculating based on the target object three-dimensional point cloud data and the hand point cloud feature vector in step S3 to obtain the distance data of each region of the object and the hand, and preprocessing the target object three-dimensional point cloud data specifically includes:

[0018] The contact distance between each region of the hand and the object point cloud is calculated by using the Euclidean distance formula, the distance data of each region of the object and the hand is obtained, and the obtained distance data is trained by using a data-driven method.

[0019] The preprocessing of the target object three-dimensional point cloud data is using a data-driven method to perform ±30° random rotation, 0.8-1.2 times scaling on the three-dimensional point cloud data, and performing horizontal flipping and brightness adjustment on the depth image.

[0020] Further, the method for constructing the lightweight diffusion model in step S4 includes the following steps:

[0021] The lightweight diffusion model is constructed by inputting the pre-processed object point cloud data and distance data, combining time embedding and attention mechanism to generate each finger contact probability map, and using the ObMan dataset as the training benchmark to optimize the model using the cross-entropy loss function, and the expression is:

[0022]

[0023] where i is the finger index, j is the index of the point cloud, y i，j is the real contact label, is the model prediction probability.

[0024] Meanwhile, a joint angle correlation regular term is introduced to constrain the spatial consistency of adjacent finger contact areas, and the joint angle correlation regular term is determined according to the actual constraint condition of finger joint movement.

[0025] Further, the method for fusing each finger contact probability map and object semantic segmentation information to construct a fine-grained contact matrix in step S5 includes the following steps:

[0026] (1) Contact position coding: threshold segmentation is performed on the contact probability map, the threshold is set to 0.7, and the region with a probability value greater than or equal to 0.7 is extracted as a potential contact area, and the three-dimensional coordinates of the top-50 high-probability contact points are selected, and the local feature vector of each point is extracted through the PointNet++ network to represent the geometric context information of the contact point.

[0027] (2) Normal matching degree calculation: define the finger tip direction vector, obtain the object surface normal vector through the target object point cloud normal vector estimation, calculate the cosine similarity between the finger tip direction vector and the object surface normal, set the threshold to 0.8, and encode 1 when the similarity is greater than or equal to 0.8, otherwise 0, to form a 0-1 binary normal matching degree variable.

[0028] (3) Finger coordination constraint: according to the mechanical structure of the dexterous hand, a joint angle correlation model is established, the thumb and index finger are set to hold an angle θ1∈[30°,120°], the other finger bending angle θ2∈

[0029] [0°,90°] and the difference between adjacent finger joint angles is less than or equal to 30°, and a joint angle feasible region matrix is generated.

[0030] (4) Constructing fine-grained contact matrix: a 6x(3+1+1) matrix is constructed, and the average three-dimensional coordinates of the top-50 high-probability contact points of each finger, the normal matching degree, and the joint constraint state are filled into the corresponding row, and the non-contact category row is filled with default values;

[0031] The object semantic segmentation information is a semantic level classification and labeling of different functional components or structural regions of the target object through computer vision technology, wherein the information provides task-oriented constraints for the grasping strategy.

[0032] Further, the method for adjusting the palm posture and the finger joint angle by hierarchical optimization based on the fine-grained contact matrix in step S6 specifically comprises:

[0033] (1) Palm posture coarse adjustment: control the wrist joint motor to adjust the palm towards the center of the object, and the error is controlled within ±5mm.

[0034] (2) Finger joint fine adjustment: based on the feedback of the fine-grained contact matrix, the joint angle of each finger key point is fine adjusted to ensure that the angle between the finger force direction and the object normal is less than 30°, and the optimal grasping posture is generated.

[0035] (3) Deep reinforcement learning optimization: in the deep reinforcement learning framework, the state space S includes the geometric features of the object, the joint angles of the fingers and the contact pressure; the adjustment range of each joint angle in the action space A is ±10°, and the accuracy is 0.5°; the reward function is designed as:

[0036] R=0.6R success +0.3R contact +0.1R time

[0037] Wherein, R success is a binary reward for successful grasping, R contact is a normalized value of contact area, and R time is a time penalty for grasping;

[0038] The PPO algorithm is used to train the strategy network to generate the optimal grasping joint value in the dynamic environment.

[0039] Compared with the prior art, the beneficial effects of the present application are:

[0040] 1. Fine-grained contact representation enhancement: A diffusion model is used to generate an independent contact probability map for each finger, with a resolution of 1024x1024, and the contact area positioning accuracy is improved to ±2mm. Compared with the traditional global contact map, the single-finger contact area misjudgment rate is reduced by 40%. The contact map generation is task-oriented by fusing object material, size and other specification information and functional component semantic labels, such as generating contact points preferentially in the handle area, which is suitable for tool object grasping.

[0041] 2. Dynamic optimization efficiency improvement: The hierarchical optimization strategy reduces the degree of freedom search space from 60 dimensions (full hand freedom) to 20 dimensions (hand palm 3 dimensions + finger 17 dimensions), reducing the calculation time by 65%, meeting the real-time grasping planning (response time <200ms). Reinforcement learning framework improves the grasping success rate from 72% to 89% under dynamic interference through 100,000 times of simulation training, especially in curved object (such as sphere, cone) grasping.

[0042] 3. Generalization ability and robustness: Multi-modal data fusion (point cloud, image, touch) and domain randomization enhancement (simulate different light, noise environment) make the model's grasping success rate for industrial parts (such as gears, bolts) and daily objects (such as cups, mobile phones) both exceed 85%; finger coordination constraints and normal matching degree verification effectively avoid object slipping and joint over-limiting in the grasping process, improving the operation stability in complex scenarios.

[0043] On the basis of the implementation manners provided by the above aspects of the present application, further combinations can be made to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 The flowchart of the grasping method of the present application;

[0046] Figure 2 The hierarchical optimization and reinforcement learning framework of the present application;

[0047] Figure 3 The geometric structure building diagram of the 3D modeling of the present application;

[0048] Figure 4 The visual identification diagram of the fingertip contact point of the hand model of the present application;

[0049] Figure 5 The overall coloring diagram of the finger of the present application;

[0050] Figure 6 Interaction posture graph of the hand and the cup model of the present application. DETAILED DESCRIPTION

[0051] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiment of the present application will be described clearly and completely in the following combined with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, not all. Based on the embodiment in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the protection scope of the present application. If not specifically pointed out, the technical means used in the embodiment is the conventional means familiar to the person skilled in the art.

[0052] The embodiment of the present application provides a dexterous hand grasping method based on a fine-grained contact representation of a diffusion model. Figure 1 , Figure 1 The embodiment of the present application provides a flow chart of a dexterous hand grasping method based on a fine-grained contact representation of a diffusion model. The specific implementation steps are as follows:

[0053] Step S1: Obtain the three-dimensional point cloud data of the target object.

[0054] Step S2: preprocess the robot hand model and divide the robot hand area by using the Blender software, extract the geometric features of each area, and generate the hand point cloud feature vector.

[0055] Step S3: Based on the three-dimensional point cloud data of the target object and the hand point cloud feature vector, the distance data of the object and the hand area is obtained, and the three-dimensional point cloud data of the target object is preprocessed.

[0056] Step S4: Construct a lightweight diffusion model, input the preprocessed three-dimensional point cloud data of the object and the distance data, and generate the contact probability map of each finger by combining the time embedding and attention mechanism.

[0057] Step S5: Fuse the contact probability map of each finger and the object semantic segmentation information to construct a fine-grained contact matrix.

[0058] Step S6: Based on the fine-grained contact matrix, adjust the palm posture and finger joint angle through hierarchical optimization, and dynamically optimize the grasping action combined with reinforcement learning.

[0059] Optionally, the method for obtaining the three-dimensional point cloud data of the target object in step S1 specifically comprises:

[0060] The target object three-dimensional point cloud data (including x / y / z coordinates and normal vectors nx / ny / nz, dimension N*6) is collected by a Kinect Azure depth camera.

[0061] Different grasping poses of the robot hand are provided based on the ContactPose dataset, accurately representing the amount of point cloud data and retaining key features.

[0062] Optionally, the method for preprocessing the robot hand model and dividing the robot hand area in step S2 includes:

[0063] The hand model is preprocessed using Blender software, and the robot hand is divided into five independent areas: thumb, index finger, middle finger, ring finger, and little finger. The geometric features (such as curvature and convexity) of each area are extracted to generate hand feature point clouds, as shown in the ContactMapDataset class labeling logic in the experimental verification and parameter configuration.

[0064] Optionally, the method for calculating the distance data between the object and each area of the hand based on the target object three-dimensional point cloud data and hand point cloud feature vectors in step S3 includes:

[0065] The contact distance between each area of the hand and the object point cloud is calculated using the Euclidean distance formula to obtain the distance data between the object and each area of the hand, and the obtained distance data is trained using a data-driven method.

[0066] The preprocessing of the target object three-dimensional point cloud data uses a data-driven method to perform ±30° random rotation, 0.8-1.2 times scaling, horizontal flipping, and brightness adjustment on the depth image to generate diversified training samples and improve the model generalization ability.

[0067] Optionally, the method for constructing a lightweight diffusion model in step S4 includes:

[0068] A lightweight diffusion model is constructed, inputting pre-processed object point cloud data, with a dimension of N*6, including x / y / z coordinates and normal vectors nx / ny / nz, and calculated contact distance information, wherein the object specification parameters are encoded into a 128-dimensional vector by Word2Vec; through a time embedding module and an attention mechanism, a contact probability map with a resolution of 1024*1024 is generated for each finger, and the ObMan dataset is used as a training benchmark, and a cross-entropy loss function is used to optimize the model, and its expression is:

[0069]

[0070] wherein i is the finger index, j is the index of the point cloud, y i，j is the real contact label, is the model prediction probability.

[0071] At the same time, a joint angle correlation regular term is introduced to constrain the spatial consistency of adjacent finger contact areas, and the joint angle correlation regular term is determined according to the actual constraint condition of finger joint movement.

[0072] Specifically, the lightweight diffusion model is constructed: the main body of the model is ContactConditionedUNet, which extends UNet2DConditionModel and introduces a sensor condition encoder, inputs pre-processed object point cloud data (N*6) and 128-dimensional object specification vectors (including shape, size, and material semantic encoding) generated by Word2Vec, combines time embedding (Time Embedding) and attention mechanism (such as CrossAttnDownBlock2D module in experimental verification and parameter configuration), and generates a contact probability map for each finger (resolution 1024*1024, such as ContactDiffusionGenerator inference process in experimental verification and parameter configuration) through a 100-step denoising process.

[0073] Training and regularization optimization: based on the ObMan dataset, manually label the actual contact area of each finger as a supervision signal (such as the ContactMapDataset class in experimental verification and parameter configuration simulates sensor data logic), and use a cross-entropy loss function (formula in experimental verification and parameter configuration loss =mse_loss+0.3*contact_loss) to optimize the model parameters; add a joint angle correlation regular term (such as the encoding of the sensor_encoder in experimental verification and parameter configuration) to the output layer to ensure the spatial consistency of adjacent finger contact areas and avoid penetration conflicts (such as the minimum distance constraint between the thumb and the index finger contact area).

[0074] Optionally, the method of fusing each finger contact probability map and object semantic segmentation information to construct a fine-grained contact matrix in step S5 specifically includes the following steps:

[0075] (1) Contact position encoding: threshold segmentation is performed on the contact probability map, the threshold is set to 0.7, and the region with a probability value greater than or equal to 0.7 is extracted as a potential contact region. The three-dimensional coordinates of the top-50 high-probability contact points are selected, the local feature vector of each point is extracted through the PointNet++ network to represent the geometric context information (such as curvature, neighborhood structure, etc.) of the contact point.

[0076] (2) Normal matching degree calculation: define the finger tip direction vector (such as the thumb tip pointing to the palm center, and the other finger tips perpendicular to the finger palm plane), obtain the object surface normal vector (nx, ny, nz) through the target object point cloud normal vector estimation (such as a K-neighbor-based calculation method), calculate the cosine similarity between the finger tip direction vector and the object surface normal, set the threshold to 0.8, and encode 1 (effective contact) when the similarity is greater than or equal to 0.8, otherwise 0, forming a 0-1 binary normal matching degree variable.

[0077] (3) Finger coordination constraint: according to the mechanical structure of the dexterous hand, a joint angle correlation model is established, for example, set the thumb and index finger to hold angle θ1∈[30°, 120°], other finger bending angle θ2∈[0°, 90°] and adjacent finger joint angle difference ≤30°, generate a joint angle feasible region matrix (elements are 0 or 1, indicating whether the joint angle satisfies the physical constraint).

[0078] (4) Constructing a fine-grained contact matrix: a 6x(3+1+1) matrix is constructed, where the 6 rows correspond to the thumb, index finger, middle finger, ring finger, little finger, and non-contact category, and the 5 columns correspond to the 3-dimensional coordinates (X, Y, Z), the normal matching degree (0 / 1), and the joint constraint state (0 / 1). The average three-dimensional coordinates of the Top-50 high-probability contact points of each finger, the normal matching degree (taking the majority of valid points), and the joint constraint state (according to whether the current joint angle is within the feasible region) are filled into the corresponding row. The non-contact category row is filled with default values (such as the object centroid, normal matching degree and joint constraint are 0).

[0079] Definition and fusion logic of object semantic segmentation information:

[0080] (1) Object semantic segmentation information: refers to the semantic level classification and labeling of different functional components or structural regions of the target object through computer vision technology, such as distinguishing handles, planes, protrusions, and recesses. This information provides task-oriented constraints for grasping strategies, such as generating contact points in the "handle" area to adapt to tool object grasping requirements, or avoiding "fragile parts" to avoid damage to the object.

[0081] (2) Fusion mode:

[0082] Semantic weight modulation: According to the object semantic segmentation result (such as handle area weight ω = 1.2, plane area ω = 0.8), the original probability value of the contact probability map is weighted and adjusted, and the formula is: adjusted probability = original probability × semantic class weight, which enhances the priority of the contact points in the target functional area.

[0083] Semantic filtering mechanism: Based on the semantic label, the non-target area contact points are filtered (such as prohibiting the generation of contact points on the "fragile surface"), the top-50 high-probability contact points are re-screened, and the coordinate and normal matching degree in the contact matrix are updated to ensure that the contact distribution meets the object functional characteristics and the grasping task requirements.

[0084] Through the above steps, the fine-grained contact matrix integrates the geometric features of the contact position, the physical constraints of the normal direction, the motion restrictions of the finger joints, and the semantic guidance of the object functional structure, forming multi-dimensional constraint information, which provides accurate input for subsequent grasping pose optimization, and significantly improves the grasping stability and task adaptability of the dexterous hand to complex objects.

[0085] Optionally, as shown in Figure 2 the step S6 based on the fine-grained contact matrix, the palm posture and finger joint angle are adjusted through hierarchical optimization, and the method of dynamically optimizing the grasping action by combining reinforcement learning specifically includes:

[0086] (1) Palm posture coarse adjustment: as shown in Figure 2 the left "palm coarse adjustment" module, the palm orientation is adjusted to the center of the object through the wrist joint motor, and the position sensor feedback control is used to ensure that the error between the palm center and the object center is within ±5mm (such as the experimental verification and parameter configuration of the manipulator control precision parameter), reducing the degree of freedom search space from 60 to 20.

[0087] (2) Finger joint fine adjustment: as shown in Figure 2 the right "finger fine adjustment" module, based on the high-probability contact points of the contact matrix C, the angles of the key joints (finger tip, proximal phalanx joint) of each finger are adjusted through inverse kinematics algorithm to ensure that the angle between the finger force direction and the object normal is <30° (such as the specific application of "normal matching degree" constraint in experimental verification and parameter configuration).

[0088] (3) Deep reinforcement learning optimization: In the deep reinforcement learning framework, the state space S includes object geometric features (point cloud curvature, semantic label), 15-dimensional finger joint angles (3 joints per finger), and 5-dimensional contact pressure (such as Figure 2The state-action-reward module is labeled); the joint angle adjustment range of the action space A is ±10°, and the accuracy is 0.5°, as verified in the experiment and the parameter setting of the generate method in the parameter configuration; and the reward function is designed as:

[0089] R = 0.6R success + 0.3R contact + 0.1R time

[0090] wherein R success is the binary reward for successful grasping, R contact is the normalized contact area value, and R time is the grasping time penalty.

[0091] The PPO algorithm is used to train the policy network to generate optimal grasping joint values in a dynamic environment. Thus, the PPO algorithm is used to train the policy network (such as the train_model function flow in the experiment and parameter configuration), and the contact point distribution is optimized through 100,000 times of simulation training to generate optimal joint values in a dynamic environment (such as automatically adjusting the finger bending arc for a spherical object, corresponding Figure 2 to the arrow logic of "optimal grasping joint value generation in a dynamic environment" in the figure).

[0092] Figure 2 (Hierarchical optimization framework): The two-layer structure of "palm coarse adjustment-finger fine adjustment" is displayed, and the reinforcement learning module optimizes the contact point distribution through the state-action-reward cycle, which completely corresponds to the mathematical model in step 6.

[0093] Figure 3 3D modeling geometry setup diagram: shows the three-dimensional geometric model construction process of the target object and the dexterous hand; the mechanical hand (such as Shadow Hand) is divided into regions (thumb, index finger, etc. 5 independent regions) through Blender software, and the geometric features (such as curvature, convexity) of each region are extracted to generate hand point cloud feature vectors; at the same time, the target object (such as a cup, a gear) is modeled by point cloud, providing geometric basis for subsequent contact distance calculation and contact map generation; the division boundaries of each region of the hand and the key parts of the object (such as handle, curved surface) are labeled in the diagram, which embodies the specific implementation of geometric feature extraction and preprocessing in steps S2 and S3.

[0094] Figure 4A visual identification map for the fingertip contact point of the hand model: the contact probability map visualization result of a single finger (such as the index finger), in a 1024x1024 resolution contact probability map generated by a lightweight diffusion model, the highlighted area represents the area with high fingertip contact probability (threshold value ≥ 0.7), which corresponds to the output result of "generating each finger contact probability map" in step S4; the contact probability value is marked with different colors or brightness in the figure, which intuitively shows the potential contact position of the fingertip and the object surface, and provides input data for the contact position encoding and normal matching degree calculation in step S5.

[0095] Figure 5 A finger overall coloring map: shows the overall coloring effect of each finger of the dexterous hand, different colors correspond to different fingers (such as red for the thumb and blue for the index finger), which is used to distinguish the independent contact area of each finger; combined with the logic of "fusing each finger contact probability map" in step S5, this map embodies the basis for building a fine-grained contact matrix - each finger generates an independent contact probability map, and the contact point priority is adjusted through semantic segmentation information (such as the priority of grabbing the handle area); the joint angle constraint range of each finger (such as the angle θ1 between the thumb and the index finger θ1 ∈ [30°, 120°]) is marked in the figure, which corresponds to the "finger coordination constraint" mechanism in step S5.

[0096] Figure 6 An interaction pose map of the hand and the cup model: shows the actual pose of the dexterous hand grabbing the cup based on the fine-grained contact matrix, which embodies the result of "adjusting the palm pose and finger joint angle in layers" in step S6; in the figure, the palm is facing the center of the object (coarse error ≤ ± 5mm), and each finger adjusts the joint angle according to the contact matrix feedback (such as the bending angle θ2 of the index finger θ2 ∈ [0°, 90°]), which ensures that the angle between the fingertip force direction and the normal of the cup surface is < 30°; In addition, the contact point distribution after reinforcement learning optimization (such as the high-probability contact point of the thumb and the index finger in the handle area of the cup) is marked in the figure, which verifies the improvement effect of dynamic optimization on the stability of grabbing.

[0097] Experimental verification and parameter configuration

[0098] Specifically, the diffusion model architecture is as follows:

[0099] The steps of customizing the contact map dataset loader (classContactMapDataset(Dataset)) are as follows:

[0100]

[0101] The steps of simulating physical sensor data (actual application replaced with real data) are as follows:

[0102]

[0103]

[0104] The steps to obtain the corresponding sensor data are as follows:

[0105]

[0106] The steps to generate synthetic training data (actual application replaces real data collection) are as follows:

[0107] os.makedirs(output_dir,exist_ok=True)

[0108] for i in range(num_samples):

[0109] The steps to create a blank image are as follows:

[0110] img = Image.new('L', (512, 512), 0)

[0111] draw = ImageDraw.Draw(img)

[0112] The steps to randomly generate contact points are as follows:

[0113]

[0114] The steps to draw Gaussian distribution contact points are as follows:

[0115]

[0116]

[0117] In the data preparation module, the ContactMapDataset class is a custom dataset loader,

[0118] In the __init__ method, collect image paths, define image transformations, and simulate sensor data,

[0119] The __len__ and __getitem__ methods are used to return the number of samples and obtain specific sample data, respectively;

[0120] The generate_synthetic_data function is used to generate synthetic training data, create an output directory, and loop to generate images with random contact points for saving.

[0121] (ContactMapDataset class): Show the structure of a custom dataset loader, which _getitem_The method fuses the contact map image with sensor data (6-dimensional vector simulating physical sensor output, like Figure 3 sensor_data input).

[0122] The steps to extend UNet to support sensor condition input (class ContactConditionedUNet (UNet2DConditionModel)) are as follows:

[0123]

[0124]

[0125] The steps of the sensor condition encoder are as follows:

[0126]

[0127] The steps of the encoded sensor data are as follows:

[0128] sensor_emb = self.sensor_encoder(sensor_data)

[0129] sensor_emb = sensor_emb.view(-1, 1024, 1, 1)

[0130] The steps of the original UNet processing are as follows:

[0131] x = self.conv_in(x)

[0132] The steps of injecting sensor conditions at intermediate layers are as follows:

[0133]

[0134] The steps of the rest of the UNet processing flow are as follows:

[0135] return super().forward(x, timesteps)

[0136] The model architecture part, the ContactConditionedUNet class extends

[0137] UNet2DConditionModel,

[0138] The __init__ method initializes the UNet parameters and defines the sensor condition encoder.

[0139] The forward method first encodes the sensor data, then injects the sensor condition into the intermediate layer of UNet after convolution processing of the input image, and completes the remaining processing flow.

[0140] (ContactConditionedUNet network): Inject sensor conditions into the intermediate layer of UNet (such as the CrossAttnDownBlock2D module adding sensor_emb input in the figure), realize the fusion of contact map generation and joint constraint.

[0141] The steps of deftrain_model() are as follows:

[0142] The steps of initialization are as follows:

[0143] device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

[0144] The steps of generating synthetic data (actual application uses real data) are as follows:

[0145] generate_synthetic_data(num_samples=1000)

[0146] The steps of the dataset are as follows:

[0147] dataset = ContactMapDataset('. / contact_data')

[0148] dataloader = DataLoader(dataset, batch_size=4, shuffle=True)

[0149] The steps of the model are as follows:

[0150] model = ContactConditionedUNet().to(device)

[0151] noise_scheduler = DDPMScheduler(num_train_timesteps=1000)

[0152] The steps of the optimizer are as follows:

[0153] optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)

[0154] lr_scheduler = get_cosine_schedule_with_warmup(

[0155] optimizer,

[0156] num_warmup_steps = 500,

[0157] num_training_steps = len(dataloader) * 100 )

[0159] # Steps for the training loop are as follows:

[0160] for epoch in range(100):

[0161] progress_bar = tqdm(dataloader, desc=f'Epoch {epoch}')

[0162] for batch in progress_bar:

[0163] clean_images = batch['image'].to(device)

[0164] sensor_data = batch['sensor'].to(device)

[0165] # Steps for adding noise are as follows:

[0166] noise = torch.randn_like(clean_images)

[0167] timesteps = torch.randint(

[0168] 0, noise_scheduler.num_train_timesteps,

[0169] (clean_images.shape[0],), device=device )

[0171] noisy_images = noise_scheduler.add_noise(clean_images, noise, timesteps)

[0172] # Steps for predicting noise are as follows:

[0173] noise_pred = model(noisy_images, timesteps, sensor_data).sample

[0174] # The steps to calculate the loss (with physical constraints) are as follows:

[0175] mse_loss = F.mse_loss(noise_pred, noise)

[0176] # The steps to impose the contact force distribution constraint are as follows:

[0177] contact_loss = F.mse_loss(

[0178] torch.sigmoid(noise_pred).mean(dim=[2, 3]),

[0179] sensor_data[:, :1].expand(-1, noise_pred.shape[1]) )

[0181] loss = mse_loss + 0.3 * contact_loss

[0182] # The steps for backpropagation are as follows:

[0183] optimizer.zero_grad()

[0184] loss.backward()

[0185] optimizer.step()

[0186] lr_scheduler.step()

[0187] progress_bar.set_postfix(loss=loss.item())

[0188] # The steps to save the model are as follows:

[0189] torch.save(model.state_dict(), 'contact_diffusion.pth' =

[0190] # The training process is implemented through the train_model function, which first initializes the device, generates synthetic data, creates a dataset and loader,

[0191] # Then initialize the model, noise scheduler, optimizer, etc.; in the training loop, load data and add noise, the model computes the loss including mean squared error and contact force distribution constraints after predicting the noise,

[0192] # Update parameters and learning rate via backpropagation, and finally save the trained model.

[0193] The steps of the """production environment inference pipeline"" "(class ContactDiffusionGenerator) are as follows:

[0194] def__init__(self,model_path='contact_diffusion.pth'):

[0195] self.device=torch.device('cuda'if torch.cuda.is_available()else'cpu')

[0196] # Load the model as follows:

[0197] self.model=ContactConditionedUNet().to(self.device)

[0198] self.model.load_state_dict(torch.load(model_path))

[0199] self.model.eval()

[0200] self.noise_scheduler=DDPMScheduler(num_train_timesteps=1000)

[0201] # Optimization settings are as follows:

[0202] if torch.cuda.is_available():

[0203] self.model=torch.compile(self.model)

[0204] def generate(self,sensor_input,num_inference_steps=50):

[0205] """Generate contact map"""

[0206] The steps with torch.no_grad() are as follows:

[0207] The steps to prepare the input are as follows:

[0208] sensor_tensor = torch.FloatTensor(sensor_input).unsqueeze(0).to(self.device)

[0209] x = torch.randn((1, 1, 512, 512), device=self.device)

[0210] The steps for step-by-step denoising are as follows:

[0211]

[0212] The steps for post-processing are as follows:

[0213] image = (x.clamp(-1, 1) + 1) / 2 # [-1, 1] -> [0, 1]

[0214] image = image.squeeze().cpu().numpy()

[0215] return Image.fromarray((image * 255).astype(np.uint8))

[0216] The inference application is completed by the ContactDiffusionGenerator class,

[0217] The __init__ method initializes the device, loads the pre-trained model, and sets the evaluation mode, initializes the noise scheduler, and compiles the optimized model in the GPU environment.

[0218] The generate method receives sensor input and inference steps, prepares data for step-by-step denoising, and converts the generated image to a PIL image after post-processing.

[0219] In the Shadow Hand platform verification, the hardware parameters are as follows:

[0220] The ContactDiffusionGenerator class mentioned above supports GPU acceleration and model compilation optimization. Its "contact area positioning accuracy ±2mm" and "computing time reduction of 65%" indicators are achieved through the above model architecture and algorithm process.

[0221] The application provides a fine-grained matrix contact map for each finger, each finger is marked with different colors, and a new concept D(F, O) is introduced for the use of the robot hand; for the contact points of the fingers and the object, a Euclidean distance is used to represent the distance of the contact, and points with high contact possibility can be intuitively displayed in the contact map. The finger coordination constraint is introduced, and a joint angle correlation regularization term is added to the model output layer, so that the spatial consistency of adjacent finger contact areas is ensured, and penetration conflicts are avoided.

[0222] The above describes one embodiment of the application in detail, but the content is only the preferred embodiment of the application, and cannot be considered as limiting the scope of the application. Any equivalent changes and improvements made within the scope of the application should still belong to the patent scope of the application.

Claims

1. A dexterous hand grasping method based on a diffusion model-based fine-grained contact representation, characterized by, The grabbing method comprises the following steps: Step S1: obtaining three-dimensional point cloud data of a target object; Step S2: preprocessing a robot hand model by using Blender software, dividing the robot hand area, extracting geometric features of each area, and generating a hand point cloud feature vector; Step S3: obtaining distance data of the object and each area of the hand based on the three-dimensional point cloud data of the target object and the hand point cloud feature vector, and preprocessing the three-dimensional point cloud data of the target object; Step S4: constructing a lightweight diffusion model, inputting the preprocessed three-dimensional point cloud data of the object and the distance data, combining time embedding and attention mechanism to generate a contact probability map of each finger; Step S5: fusing the contact probability map of each finger and the object semantic segmentation information to construct a fine-grained contact matrix; Step S6: based on the fine-grained contact matrix, adjusting the palm posture and finger joint angle through hierarchical optimization, and dynamically optimizing the grabbing action combined with reinforcement learning.

2. The dexterous hand grasping method based on a diffusion model-based fine-grained contact representation according to claim 1, wherein, The method for obtaining three-dimensional point cloud data of a target object in step S1 comprises: Collecting three-dimensional point cloud data of a target object by using a depth camera, namely Kinect Azure; Based on the ContactPose dataset, different grabbing postures of the robot hand are provided, which accurately represent the amount of point cloud data and retain key features.

3. The dexterous hand grasping method based on a diffusion model-based fine-grained contact representation according to claim 1, wherein, The method for preprocessing a robot hand model by using Blender software, dividing the robot hand area, extracting geometric features of the hand and the object, and generating a point cloud feature vector in step S2 comprises: Preprocessing the hand model by using Blender software, dividing the robot hand into five independent areas, namely the thumb, index finger, middle finger, ring finger, and little finger, extracting geometric features of each area, and generating a hand feature point cloud.

4. The dexterous hand grasping method based on a diffusion model-based fine-grained contact representation according to claim 1, wherein, The method for obtaining distance data of the object and each area of the hand based on the three-dimensional point cloud data of the target object and the hand point cloud feature vector, and preprocessing the three-dimensional point cloud data of the target object in step S3 comprises: Calculating the contact distance between each area of the hand and the object point cloud by using the Euclidean distance formula, obtaining the distance data of the object and each area of the hand, and training the obtained distance data by using a data-driven method; The preprocessing of the three-dimensional point cloud data of the target object is a data-driven method for ±30° random rotation, 0.8-1.2 times scaling, horizontal flipping, and brightness adjustment of the depth image.

5. The dexterous hand grasping method based on diffusion model based fine-grained contact representation according to claim 1, wherein, The method for constructing a lightweight diffusion model, inputting the preprocessed three-dimensional point cloud data of the object and the distance data, combining time embedding and attention mechanism to generate a contact probability map of each finger in step S4 comprises: Constructing a lightweight diffusion model, inputting the preprocessed object point cloud data with a dimension of N×6, including x / y / z coordinates and normal vectors nx / ny / nz, and the calculated contact distance information, wherein the object specification parameters are encoded into a 128-dimensional vector by using Word2Vec; generating a contact probability map of each finger with a resolution of 1024×1024 by using a time embedding module and an attention mechanism; and taking the ObMan dataset as a training benchmark, and optimizing the model by using a cross-entropy loss function, and the expression is: where i is the index of the finger, j is the index of the point in the point cloud, y i，j is the real contact label, is the model predicted probability; At the same time, a joint angle correlation regularization term is introduced to constrain the spatial consistency of the contact area of ​​adjacent fingers. The joint angle correlation regularization term is determined according to the actual constraint conditions of finger joint movement.

6. The dexterous hand grasping method based on a diffusion model-based fine-grained contact representation according to claim 1, wherein, The method for constructing a fine-grained contact matrix by fusing the contact probability maps of each finger and the semantic segmentation information of the object in step S5 specifically includes the following steps: (1) Contact position encoding: Threshold segmentation is performed on the contact probability map. The threshold is set to 0.

7. Regions with probability values ​​≥ 0.7 are extracted as potential contact regions. The three-dimensional coordinates of the Top-50 high-probability contact points are selected from them. The local feature vectors of each point are extracted through the PointNet++ network to represent the geometric context information of the contact points. (2) Normal matching degree calculation: Define the fingertip direction vector, obtain the surface normal vector of the object by estimating the normal vector of the target object point cloud, calculate the cosine similarity between the fingertip direction vector and the surface normal, set the threshold to 0.8, when the similarity is ≥0.8, it is encoded as 1, otherwise it is 0, forming a 0-1 binary normal matching degree variable; (3) Finger coordination constraint: Based on the mechanical structure of the dexterous hand, a joint angle correlation model is established, setting the thumb and index finger grip angle θ1∈[30°,120°], the other finger bending angle θ2∈[0°,90°], and the difference between adjacent finger joint angles ≤30°, and generating the joint angle feasible domain matrix; (4) Construct a fine-grained contact matrix: Construct a 6×(3+1+1) dimensional matrix, fill the corresponding rows with the average three-dimensional coordinates, normal matching degree and joint constraint state of the Top-50 high-probability contact points of each finger, and fill the non-contact category rows with default values; The object semantic segmentation information is a semantic classification and annotation of different functional components or structural regions of the target object using computer vision technology. This information provides task-oriented constraints for the grasping strategy.

7. The dexterous hand grasping method based on diffusion model based fine-grained contact representation of claim 1, wherein, The method described in step S6, which involves adjusting the hand posture and finger joint angles through hierarchical optimization based on a fine-grained contact matrix and dynamically optimizing the grasping action using reinforcement learning, specifically includes: (1) Coarse adjustment of hand posture: Control the wrist joint motor to adjust the palm toward the center of the object, with the error controlled within ±5mm; (2) Finger joint fine-tuning: Based on fine-grained contact matrix feedback, the joint angle of each finger key point is fine-tuned to ensure that the angle between the force direction of the fingertip and the normal of the object is less than 30°, thus generating the optimal grip posture; (3) Optimization of deep reinforcement learning: In the deep reinforcement learning framework, the state space S includes the geometric features of the object, the angles of the finger joints, and the contact pressure; the adjustment range of each joint angle in the action space A is ±10°, with an accuracy of 0.5°; the reward function is designed as follows: R = 0.6R success +0.3R contact +0.1R time wherein R success is a success binary reward for grasping, R contact is a contact area normalized value, R time is a grasping time penalty; The PPO algorithm is used to train the policy network to generate optimal gripping joint values ​​in dynamic environments.

Images