Dexterous grabbing planning method based on conditional diffusion generation and accessibility perception

Abstract

The invention belongs to the technical field of robots, and particularly relates to a dexterous grabbing planning method based on conditional diffusion generation and accessibility perception, which comprises the following steps of: initializing a grabbing posture, and acquiring a multi-fingered dexterous hand grabbing simulation data set, the simulation data set comprises scene point cloud data, stably grabbed palm posture data and finger joint angle data; the multi-fingered dexterous hand grabbing simulation data set is input into a conditional diffusion generation model, a complete grabbing posture is obtained, the conditional diffusion generation model is used for extracting local features in a scene point cloud, and the complete grabbing posture is obtained according to the local features; according to the reachability graph of the mechanical arm, training a stacking auto-encoder, and obtaining a grabbing reachability evaluator; the complete grabbing posture is input into the grabbing accessibility evaluator, and the accessibility probability is obtained; and on the basis of the reachable probability, in combination with the grabbing stability probability, a target grabbing posture is obtained, and grabbing is conducted according to the target grabbing posture.

Description

Technical Field

[0001] This invention belongs to the field of robotics technology, and in particular relates to a clever grasping planning method based on conditional diffusion generation and reachability perception. Background Technology

[0002] Multi-fingered dexterous hands, with their human-like multi-joint structure and highly free maneuverability, have become an important research direction in the field of robotic grasping, especially in complex applications such as precision assembly and human-robot collaboration. However, grasping planning based on multi-fingered dexterous hands faces many challenges. First, many existing grasping datasets are based on two-finger grasping, which are not directly applicable to multi-fingered grasping scenarios and cannot effectively solve physical constraints in multi-fingered grasping, such as contact stability, force closure, collision detection, and joint angle limitations.

[0003] Secondly, traditional grasping planning methods often rely too heavily on the global geometric features of objects. However, in real-world scenarios, due to occlusion, environmental complexity, or viewpoint limitations, it is impossible to obtain complete object geometric information. This causes modeling methods based on global features to often fail when information is missing, making it impossible to generate grasping postures that conform to actual contact conditions.

[0004] Furthermore, regarding the kinematic reachability of robotic arms, most studies place the object within the known reachable area of ​​the robotic arm, assuming that all generated grasping postures can be executed. However, in real-world environments, the object's position may exceed the robotic arm's workspace, rendering the grasping posture kinematically unreachable. Existing planning systems typically lack mechanisms for quantifying unreachability, failing to effectively filter out kinematically unreachable grasping postures. The theoretically "optimal" grasping posture generated may be unexecutable due to robotic arm movement limitations.

[0005] Currently, relevant technical personnel have designed and researched some methods for agile grasping planning, but some shortcomings still exist: methods based on point cloud filtering and random sampling to generate grasping postures do not introduce differentiable force closure optimization, which easily leads to hand-object penetration or unstable grasping; methods based on voxel networks to generate training data are limited by voxel resolution and lack sensitivity to capturing key local geometric features such as object protrusions and depressions; schemes that rely on RGB image semantic features to predict grasping postures through gesture detection networks are prone to posture deviations when three-dimensional geometric information is missing, and do not integrate physical constraints such as joint angle limitations; methods that optimize trajectories by combining dynamic interactive representations with behavior cloning and reinforcement learning do not include force closure conditions in the loss function, and the generated posture may not meet the force closure requirement, leading to grasping failure; currently, many schemes only study the grasping posture but do not evaluate the reachability of the robotic arm for the grasping posture, which may prevent the generated posture from being executed due to failure of inverse kinematics solution.

[0006] In summary, existing technologies still have significant shortcomings in terms of dataset physical rationality, generative model generalization, and accessibility awareness. Therefore, there is an urgent need for a unified framework that integrates large-scale, high-quality datasets, grasping posture generation based on local feature awareness, and kinematic accessibility assessment to achieve stable and efficient grasping planning for multi-fingered dexterous hands in complex scenarios. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a dexterous grasping planning method based on conditional diffusion generation and reachability awareness. This method enables stable and efficient grasping planning for multi-finger dexterous hands in complex scenarios.

[0008] To achieve the above objectives, this invention provides a clever grasping planning method based on conditional diffusion generation and reachability awareness, comprising: Initialize the grasping posture and obtain a multi-finger dexterity hand grasping simulation dataset, wherein the simulation dataset includes: scene point cloud data, stable grasping palm posture data and finger joint angle data; The multi-fingered dexterous hand grasping simulation dataset is input into the conditional diffusion generation model to obtain the complete grasping posture. The conditional diffusion generation model is used to extract local features in the scene point cloud and obtain the complete grasping posture based on the local features. A stacked autoencoder is trained based on the robotic arm reachability map to obtain a grasping reachability estimator. Input the complete grasping posture into the grasping reachability estimator to obtain the reachability probability; Based on the reachability probability and the grasping stability probability, the target grasping posture is obtained, and grasping is performed according to the target grasping posture.

[0009] Optionally, initialize the grasping posture and obtain the multi-finger dexterity hand grasping simulation dataset, including: Generate the 6D pose of a two-finger gripper by calling the GraspNet API; The two-finger gripper 6D pose is converted into a multi-finger dexterous hand palm pose, and a pre-grasping joint angle is assigned to obtain the initial multi-finger grasping posture. Based on the differential force closure energy, contact attraction energy, penetration repulsion energy, self-penetration energy, and joint over-limit energy, multiple energy functions are obtained; The initial multi-finger grasping posture is optimized by gradient descent using the multi-energy function to obtain the optimized grasping posture; The optimized grasping posture was verified in a physical simulator, and the filtered results formed the multi-finger dexterous hand grasping simulation dataset.

[0010] Optionally, the multiple energy function is: ; in, It is a multi-energy function. Energy that can be closed by a minute force To attract energy through contact, In order to penetrate repulsive energy, As self-penetrating energy, For joint excess energy, , , and These represent the weights of the corresponding energy terms.

[0011] Optionally, the conditional diffusion generation model includes: a local feature extraction module and a grasping pose generation module; The local feature extraction module is used to extract local features from the scene point cloud; The grasping posture generation module is used to obtain the complete grasping posture based on the local features.

[0012] Optionally, extracting local features from the scene point cloud includes: Calculate each point on the surface of the object crawling score The crawling score , used to represent points The grabbability of the surrounding area; Points with the target grasp score and located on the object surface are selected as feature seed points, and a preset number of feature seed points are sampled from the farthest point to obtain the local features.

[0013] Optionally, obtaining the complete grasping posture based on the local features includes: Based on the local features, obtain the feature vector of the feature seed point; Using the aforementioned feature vector as a condition, the initial wrist pose is encoded into a target dimension vector; A forward diffusion process is applied to the target dimension vector to generate noisy samples; The noisy samples were used to train a denoising network to predict the diffusion rate, and the wrist posture vector was obtained by inverse denoising through ordinary differential equations. The denoised wrist pose vector is projected back into the preset space through matrix factorization (SVD) to obtain the target wrist pose. The target wrist posture and the feature vector of the feature seed point are concatenated and input into the MLP to regress the joint angle and obtain the complete grasping posture.

[0014] Optionally, training a stacked autoencoder based on the robotic arm reachability map to obtain a grasping reachability estimator includes: Discretize the workspace of the robotic arm into an equally spaced voxel grid; Based on the equidistant voxel grid, the end pose is randomly sampled using forward kinematics, and reachable voxels are marked. When constructing the accessibility map using the forward kinematics, if the ratio of newly discovered voxels is lower than the threshold, switch to inverse kinematics uniform sampling to complete the unexplored areas and obtain a complete accessibility map; Training samples are generated by uniformly sampling hand pose and hand pose reachability labels from the complete reachability map; The stacked autoencoder is trained using the training samples to obtain the crawling reachability estimator.

[0015] Optionally, training a stacked autoencoder using the training samples to obtain the grasping reachability estimator includes a pre-training phase and a fine-tuning phase; The pre-training stage greedily trains the autoencoder layer by layer to minimize the mean square error between the input and the reconstruction. The fine-tuning phase uses reachability tags as supervision and updates all weights through backpropagation.

[0016] Compared with the prior art, the present invention has the following advantages and technical effects: 1. This invention utilizes a differentiable force closure optimization method to generate a high-quality dataset to support model training, employs a diffusion generation model guided by local features to predict grasping postures and joint angles, and constructs an accessibility evaluator to assess the accessibility of grasping to avoid failure in the inverse kinematics solution of the robotic arm. This framework forms a data-model-evaluation closed loop, taking into account grasping diversity, physical stability, and practical feasibility, providing a complete grasping planning solution for multi-fingered dexterous hands.

[0017] 2. This invention proposes a conditional diffusion generation model based on local features to achieve high-quality and diverse grasping posture generation. Combining a local feature extraction module and a grasping posture generation module, the grasping posture distribution is modeled using a conditional diffusion generation model, and joint angles are predicted using MLP, while multiple loss functions are jointly trained. This framework learns complex grasping posture distributions from local geometric features, generating diverse grasping postures that conform to physical constraints, exhibiting strong generalization ability.

[0018] 3. This invention designs a grasping reachability estimator to enhance the feasibility of grasping in real-world scenarios. Based on a stacked autoencoder, the reachability estimator learns the mapping relationship between hand posture and the kinematic reachability of the robotic arm, effectively evaluating the reachability of the predicted grasping posture. This avoids kinematic inverse kinematics failure due to the unreachability of the target object, improving the practicality and reliability of grasping planning in actual robotic systems and ensuring the feasibility of the entire process from grasping posture prediction to actual robotic arm operation. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a smart grasping planning method based on conditional diffusion generation and reachability awareness according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the dexterous hand collision contact model in the dexterous grasping planning method based on conditional diffusion generation and reachability perception in an embodiment of the present invention. Figure 3 This is a schematic diagram of dexterous hand posture initialization in the dexterous grasping planning method based on conditional diffusion generation and reachability awareness in an embodiment of the present invention. Figure 4 This is the reachability map of the robotic arm in the smart grasping planning method based on conditional diffusion generation and reachability awareness in an embodiment of the present invention. Figure 5 This is a network architecture diagram of the crawling reachability evaluation in the smart crawling planning method based on conditional diffusion generation and reachability awareness according to an embodiment of the present invention. Figure 6 This is a diagram illustrating the overall algorithm framework of the smart grasping planning method based on conditional diffusion generation and reachability awareness, as described in an embodiment of the present invention. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0022] This embodiment proposes a clever grasping planning method based on conditional diffusion generation and reachability awareness, such as... Figure 1 As shown, the specific steps include: Initialize the grasping posture and obtain the multi-finger dexterity hand grasping simulation dataset, which includes: scene point cloud data, stable grasping palm posture data and finger joint angle data; The multi-finger dexterous hand grasping simulation dataset is input into the conditional diffusion generation model to obtain the complete grasping posture. The conditional diffusion generation model is used to extract local features in the scene point cloud and obtain the complete grasping posture based on the local features. A stacked autoencoder is trained based on the reachability map of the robotic arm to obtain a grasping reachability estimator. Input the complete grasping posture into the grasping reachability estimator to obtain the reachability probability; Based on the reachability probability and the grasping stability probability, the target grasping posture is obtained, and grasping is performed according to the target grasping posture.

[0023] Specifically, this embodiment includes the following steps: First, a large-scale multi-fingered dexterous hand grasping simulation dataset is constructed for model training. The grasping posture is initialized through the GraspNet API, and the grasping posture is optimized by combining the weights corresponding to multiple energy functions such as differential force closure, contact, penetration, and joint angles. Second, a conditional diffusion generation model based on local features is constructed. Point-by-point local features are extracted from the grasping region, and the grasping posture distribution is modeled based on the local features. The grasping posture distribution is predicted through the conditional diffusion generation model, and the joint angles are predicted through MLP. Simultaneously, the target segmentation loss, grasping degree loss, mean squared error loss, and joint angle loss are linearly combined as the total loss to achieve joint training of the modules in the multi-fingered dexterous hand grasping generation model. Finally, a grasping reachability estimator is designed to learn the potential mapping relationship between the hand posture and the kinematic reachability of the robotic arm, and to evaluate the kinematic reachability of the predicted hand posture.

[0024] More specifically, step 1: Initialize the grasping posture using the GraspNet API, optimize the initial grasping posture by combining the weights corresponding to multiple energy functions such as differential force closure, contact, penetration and joint angle, and filter out stable and collision-free grasping postures through a simulation physics simulator to build a large-scale multi-finger dexterity hand grasping simulation dataset.

[0025] Step 2: Identify the grasping region and extract point-by-point local features. Model the grasping posture distribution based on these local features, predict the grasping posture distribution using a conditional diffusion generation model, and predict joint angles using an MLP. Design a loss function and jointly train each module in the multi-fingered dexterous hand grasping generation model. In the conditional diffusion generation model, the "condition" refers to the local features in the scene point cloud, and the corresponding grasping posture is generated based on these "conditions." To train the conditional diffusion generation model guided by local features, a dataset is first constructed in Step 1, collecting scene point cloud data and grasping posture data corresponding to successfully grasped objects in the simulation environment (in this embodiment, the grasping posture refers to the palm posture + finger joint angles).

[0026] Step 3: Design a grasping reachability evaluator, build a dataset based on the robotic arm reachability map to train the network, learn the potential mapping relationship between hand pose and robotic arm kinematic reachability, and evaluate the predicted hand pose kinematic reachability.

[0027] Furthermore, the grasping posture is initialized, and the simulation dataset for multi-finger dexterity hand grasping is obtained, including: Generate the 6D pose of a two-finger gripper by calling the GraspNet API; The two-finger gripper 6D pose is converted into the multi-finger dexterous hand palm pose, and a pre-grasping joint angle is assigned to obtain the initial multi-finger grasping posture. Based on the differential force closure energy, contact attraction energy, penetration repulsion energy, self-penetration energy, and joint over-limit energy, multiple energy functions are obtained; The initial multi-finger grasping posture is optimized by gradient descent using a multi-energy function to obtain the optimized grasping posture; The optimized grasping posture was verified in a physical simulator, and the filtered results formed a multi-finger dexterous hand grasping simulation dataset.

[0028] Specifically, to construct the dataset, this embodiment selects 32 and 43 everyday objects with high-quality 3D mesh models from the YCB dataset and GraspNet-1 Billion dataset, respectively, as the base objects in the dataset. The dataset is constructed using a differentiable force closure optimization method to synthesize grasping posture data. This method uses a differentiable force closure estimator as an energy term and combines it with weights corresponding to multiple energy functions such as contact, penetration, and joint angles to optimize and synthesize the grasping posture. The calculation of each energy term depends on the geometric interaction between the dexterous hand and the object. To accurately represent these geometric relationships, this embodiment constructs a collision contact model for a 6-DOF time-dependent dexterous hand. Specifically, firstly, a dexterous hand collider is established based on the dexterous hand URDF file, and contact candidate points are uniformly selected on the surface of the dexterous hand finger collider. Then, the geometric center of each finger joint in the collider is used as the penetration point, and an appropriate penetration radius is selected based on the thickness of each finger joint.

[0029] In the differentiable force closure optimization method, the differentiable force closure energy term causes a set of contact points to form a force closure, expressed as: ; in: ; ; In the formula, G is a block matrix describing the relationship between contact force and force spinor. It is a 3×3 identity matrix. Let i be the three-dimensional coordinates of the i-th contact point. for The z-component, for The y-component.

[0030] To ensure contact between the multi-fingered dexterous hand and the object and to prevent penetration, a pair of attractive and repulsive energy functions are designed: ; ; in, It is a hand mesh Surface point clouds, Indicates the inside of the hand grid. It is the distance from the point to the grid. Point Is it inside the hand grid? Let be the three-dimensional coordinates of the i-th contact point.

[0031] However, the above The mesh fails for thin objects because the penetration energy term is always zero. Therefore, this embodiment designs a self-penetration energy function for thin objects, using penetration points on different finger joints. When the distance between two contact points is less than a distance threshold, self-penetration is considered to have occurred. Its expression is: ; in, Represents hand grid The set of penetration points and These are two differences in the set. This represents the distance between the two points. This is the distance threshold.

[0032] To prevent the finger joint angles corresponding to the optimized grasping posture from exceeding the limit, an energy function for the excessive joint angle is designed: ; in, Indicates the first The actual angle of each joint and These are the maximum and minimum allowable angles for the joint, respectively.

[0033] Furthermore, the multiple energy function is: ; in, It is a multi-energy function. Energy that can be closed by a minute force To attract energy through contact, In order to penetrate repulsive energy, As self-penetrating energy, For joint excess energy, , , and These represent the weights of the corresponding energy terms.

[0034] In each optimization iteration, six contact points are randomly sampled from the candidate contact points and the total energy term is calculated. With continuous iteration, the grasping posture gradually approaches the optimal posture from the initial posture. Finally, the synthesized grasping posture is verified in the IsaacGym physics simulator, and unstable grasps are filtered out.

[0035] Differential force closure optimization methods are highly sensitive to the initial hand pose. Optimizing from a random initial pose to a suitable target pose is not only time-consuming but also yields inconsistent optimization results. To address this, this invention designs a hand pose initialization method: It calls the GraspNet API to synthesize a large number of diverse two-finger gripper 6D poses for objects in the scene. For each two-finger 6D pose, the relocalization from the two-finger 6D pose to the multi-finger dexterity hand's palm pose is achieved by aligning the dexterity hand's palm orientation with the two-finger gripper orientation and translating a certain distance in the opposite direction of the dexterity hand's wrist. Simultaneously, a set of pre-grasping joint angles is defined to initialize the dexterity hand pose.

[0036] Because the initial poses retain diverse characteristics, the final synthesized multi-finger dexterous hand dataset is stable, collision-free, and diverse.

[0037] Furthermore, the conditional diffusion generation model includes: a local feature extraction module and a grasping pose generation module; The local feature extraction module is used to identify the grabbable region and extract point-by-point local features; The grasping pose generation module is used to obtain the complete grasping pose based on point-by-point local features.

[0038] Specifically, to predict stable grasping postures in multi-object scenes, this invention designs a conditional diffusion generative model guided by local features. This model consists of two parts: a local feature extraction module and a grasping posture generation module. The local feature extraction module identifies the graspable region and extracts point-by-point local features; the grasping posture generation module models the grasping posture distribution based on these local features. The combination of the generative model and local conditions enables the network to learn from a large number of local geometric changes in the dataset, thereby significantly improving generalization efficiency.

[0039] Furthermore, identifying the grabbable region and extracting point-by-point local features includes: Calculate each point on the surface of the object crawling score Among them, the crawling score , used to represent points The grabbability of the surrounding area; Points with target capture scores and located on the object surface are selected as feature seed points, and a preset number of feature seed points are sampled from the farthest point to obtain point-by-point local features.

[0040] Specifically, the local feature extraction module extracts features from single-view depth point clouds. Extract point-level features and generate object segmentation scores for each point. and crawling score This identifies the crawlable region. Based on these scores, a subset of high-scoring points is selected as feature seed points, and their local features are input into the subsequent crawling generation module. Compared to conditional methods based on global features, this approach achieves more efficient generalization.

[0041] In each training scenario, for each point on the object's surface Define crawl score This score represents the graspability of the area around the tag. Essentially, it's calculated by having each grasp tag heuristically "vote" for its nearest neighbors within its palm's reach. For each grasp tag in the scene... If point Located within the reach of its palm (distance threshold) ),but right The voting value is: ; in, To crawl The palm of the hand To crawl mass fraction, This is the attenuation coefficient.

[0042] The capture score for point p is the logarithm of the sum of all votes: ; in, For the set of all valid crawls in the scene, experience shows This reflects the number of effective captures in its vicinity.

[0043] To extract local features and predict point-level grasping accuracy, given a scene point cloud ResUNet14, built on the Minkowski Engine, is used for each point. Extracting feature vectors This feature is then fed into the MLP to predict... crawling score and object segmentation fraction Finally, the top 1% of object points in terms of capture accuracy were selected and downsampled using farthest point sampling (FPS) to... =1024 points. These points are called feature seed points, denoted as... Its eigenvectors and crawling score This will be used for subsequent grasping posture generation.

[0044] Furthermore, obtaining the complete grasping posture based on point-by-point local features includes: Based on the local features point by point, obtain the feature vector of the feature seed point; The initial wrist pose is encoded into a target dimension vector based on the feature vector. A forward diffusion process is applied to the target dimension vector to generate noisy samples; A denoising network was trained using noisy samples to predict the diffusion rate, and the wrist posture vector was obtained by inverse denoising through ordinary differential equations. The denoised wrist pose vector is projected back into the preset space through matrix factorization (SVD) to obtain the target wrist pose. The target wrist pose and the feature vector of the feature seed point are concatenated and then input into the MLP to regress the joint angle and obtain the complete grasping pose.

[0045] Specifically, the pose generation module receives the feature vector of the determined feature seed point. A conditional diffusion generative model is used to generate multiple skillful grasping poses near the feature seed point. These grasping poses are then determined based on their estimated log-likelihood and the grasping degree score (GS) of the feature seed point. s Sort the data. The skillful grasping posture relative to the feature seed points is denoted as: ; in and This represents the wrist pose relative to the feature seed point. Represents the joint angles of the hand. Given a point cloud of a scene. Feature seed points The conditional probability distribution of all valid captures in its vicinity is as follows: ; in It is a point The predicted visual features. This embodiment assumes that in conditions, The distribution of is multimodal and quite complex, while and conditions, The distribution is single-modal. Therefore, a conditional generative model is used to predict... The conditional distribution, and based on a deterministic model. and predict .

[0046] A denoised diffusion probability model is used to approximate the probability distribution. To be Embedded in Euclidean space, the rotation matrix Flatten and translate By concatenating the vectors, we obtain a 12D vector representation: ; Subsequently, the conditional denoising model was trained. The random 12D Gaussian noise vector is transformed through an iterative process. Denoising to obtain an effective wrist pose vector At the diffusion time step The formula for generating noisy samples is: ; in , The diffusion coefficient is a predefined value. (Denoising model) Predict the diffusion rate and achieve noise reduction by solving ordinary differential equations: ; in , The time step interval is used. Finally, the denoised image is processed using SVD. Projection Back Space, obtain The sample probability is estimated by solving partial differential equations. The grasping postures are then sorted using the following formula: ; in These are the weighting coefficients. The wrist pose is sampled from the diffusion model. Then, the features of the feature seed points are... and Input MLP to predict joint angles To form a complete grasping posture .

[0047] More specifically, the local feature extraction module and the grasping pose generation module employ joint training for model training. In each gradient step, random samples are taken from the dataset. A scene, the depth point cloud of the scene is denoted as The corresponding object point mask is The truth point level capture score is Subsequently, random sampling was performed in the scene. One crawled tag Each crawl tag Represented relative to its "corresponding point". attitude in the reference frame .

[0048] Training loss consists of object segmentation loss , capture rate loss diffusion loss and joint angle loss The four components together form a linear combination of the total loss: ; in, , , and These are the object segmentation loss, gripping degree loss, diffusion loss, and joint angle loss, and their expressions are as follows: ; ; ; ; in, This represents the actual diffusion rate.

[0049] Furthermore, a stacked autoencoder is trained based on the robotic arm's reachability map to obtain a grasping reachability estimator, including: Discretize the workspace of the robotic arm into an equally spaced voxel grid; Based on an equally spaced voxel grid, the end pose is randomly sampled using forward kinematics, and reachable voxels are labeled. When constructing an accessibility map using forward kinematics, if the ratio of newly discovered voxels is below a threshold, switch to inverse kinematics uniform sampling to complete unexplored areas and obtain a complete accessibility map. Training samples are generated by uniformly sampling hand pose and hand pose reachability labels from the complete reachability map; A stacked autoencoder is trained using training samples to obtain a crawling reachability estimator.

[0050] Furthermore, a stacked autoencoder is trained using training samples to obtain a grasping reachability estimator, including a pre-training phase and a fine-tuning phase. During the pre-training phase, the autoencoder is trained layer by layer greedily to minimize the mean square error between the input and the reconstruction. The fine-tuning phase uses reachability tags as supervision and updates all weights through backpropagation.

[0051] Specifically, the conditional diffusion generation model based on local feature guidance proposed in step 2 only outputs a stable grasping posture during the grasping prediction stage. However, in real-world multi-object scenarios, the target object may be located in an inaccessible position for the robotic arm. Directly driving the robotic arm to approach the grasping posture with the highest grasping stability will lead to kinematic inverse solution failure due to the inaccessibility of the target object. Therefore, the robot needs to have the ability to evaluate the kinematic reachability of the predicted grasping posture. To this end, this embodiment proposes a grasping reachability estimator based on a stacked autoencoder to evaluate the kinematic reachability of each candidate grasping posture.

[0052] The grasping reachability estimator uses a stacked autoencoder (SAE) based on softmax regression (SR) as the backbone of its network structure. It uses SAE for kinematic feature extraction and SR for reachability label classification. The network architecture of the grasping reachability estimator consists of four layers: an input layer, two hidden layers of sizes 5 and 3, and a final softmax regression layer. The network operates in a 12-dimensional hand gesture. As input, the output is the reachability probability of the evaluated hand pose. and unreachability probability .

[0053] To enable the grasping reachability assessment network to fully learn and extract the motion features of the robotic arm, a reachability dataset containing a complete representation of the robotic arm's reachable workspace is needed for training. The reachability graph is the set of all possible poses that the end effector can reach. This invention constructs a reachability dataset based on the robotic arm reachability graph. The specific generation process of the reachability graph is as follows: First, the workspace of the robotic arm is discretized into a voxel grid of equal size, and the workspace is randomly explored using the Forward Kinematics (FK) method. The FK method sets the reachability label of most voxels in the workspace to 1 (i.e., the calculated voxel center pose is reachable). While constructing the reachability graph using the FK method, the ratio of newly discovered voxels in the generated reachability graph is calculated in real time. Once the ratio reaches a predefined threshold, it indicates that the exploration process of the reachable space is trapped in calculated voxels due to the randomness of the FK method. At this point, the Inverse Kinematics method is triggered, continuing to explore the workspace not reached by the FK method through uniform sampling, thereby completing the reachability graph constructed by the FK method. By uniformly sampling the hand pose and its reachability label in the constructed reachability graph, a reachability dataset containing 20,000 reachability labels is built.

[0054] The entire network employs a phased training approach. During the pre-training phase, SAE is trained using a cascading greedy algorithm, first encoding to extract features, then decoding to reconstruct the input. The encoding and decoding processes are formulated as follows: ； ； in, and These represent the weight matrices for the encoding and decoding processes, respectively. and This represents the corresponding bias vector. It is the activation function of the autoencoder. The goal of training a layer of an autoencoder is to adjust the weights. , , , To minimize the reconstruction error function between the input hand pose and the reconstructed hand pose: ; After obtaining the SAE pre-trained weights, use This serves as a training set to improve SR's ability to capture reachability label classification. This is the activation level of the last layer of SAE. The input is the reachability label of the hand gesture. Finally, to improve the predictive performance and robustness of the multi-finger dexterity hand grasping planning model, the weights of the entire network are fine-tuned using the backpropagation algorithm.

[0055] Finally, the attainable probability of the hand position. Multiply by the stability probability of the hand posture Obtain the probability of crawling The grasping posture corresponding to the maximum grasping feasibility probability is the optimal grasping posture.

[0056] Among them, the crawling stability probability is given a crawling configuration. When this configuration satisfies the conditional probability of force closure, the formula is: .in It is a binary stability measure (1 indicates stable, 0 indicates unstable). The model output is generated through conditional diffusion after training. The reachable grasp probability is given a grasp configuration. and robotic arm When this configuration satisfies the conditional probability of kinematic reachability, the formula is: .in It is a binary reachability measure (1 indicates reachability, 0 indicates unreachability). The grasping reachability estimator learns the mapping relationship between hand pose and reachability, and directly outputs the reachability grasp probability value.

[0057] The following is a detailed description of this embodiment with reference to the accompanying drawings: like Figure 1 As shown, this embodiment specifically includes: Step 1: Initialize the grasping posture using the GraspNet API, optimize the initial grasping posture by combining the weights of multiple energy functions such as differential force closure, contact, penetration and joint angle, and filter out stable and collision-free grasping postures through a simulation physics simulator to build a large-scale multi-finger dexterity hand grasping simulation dataset.

[0058] To construct the dataset, this invention selects 32 and 43 everyday objects with high-quality 3D mesh models from the YCB dataset and GraspNet-1 Billion dataset, respectively, as the base objects for the dataset. The grasping gestures generated during dataset construction are primarily based on a differentiable force closure optimization method. This method uses a differentiable force closure estimator as an energy term and combines it with weights corresponding to multiple energy functions such as contact, penetration, and joint angles to optimize and synthesize the grasping posture. The calculation of each energy term depends on the geometric interaction between the pre-dexterous hand and the object. To accurately represent these geometric relationships, this invention constructs a collision contact model for a 6-DOF time-dependent dexterous hand, such as... Figure 2 As shown. Specifically, firstly, a dexterous hand collider is created based on the dexterous hand URDF file, and contact candidate points are uniformly selected on the surface of the dexterous hand finger collider. Then, the geometric center of each finger joint in the collider is used as the penetration point, and an appropriate penetration radius is selected according to the thickness of each finger joint.

[0059] In the differentiable force closure optimization method, the differentiable force closure energy term causes a set of contact points to form a force closure, expressed as: ; in: ; ; To ensure contact between the multi-fingered dexterous hand and the object and to prevent penetration, a pair of attractive and repulsive energy functions are designed: ; ; in It is a hand mesh Surface point clouds, Indicates the inside of the hand grid. It is the distance from the point to the grid. Point Is it inside the hand grid?

[0060] However, the above The mesh fails for thin objects because the penetration energy term is always zero. Therefore, this invention designs a self-penetration energy function for thin objects, using penetration points on different finger joints. When the distance between two contact points is less than a distance threshold, self-penetration is considered to have occurred. Its expression is: ; in Represents hand grid The set of penetration points and These are two differences in the set. This represents the distance between the two points. This is the distance threshold.

[0061] To prevent the finger joint angles corresponding to the optimized grasping posture from exceeding the limit, an energy function for the excessive joint angle is designed: ; in, Indicates the first The actual angle of each joint and These are the maximum and minimum allowable angles for that joint, respectively.

[0062] The final total energy function is: ; in, , , and These represent the weights of the corresponding energy terms.

[0063] In each optimization iteration, six contact points are randomly sampled from the candidate contact points and the total energy term is calculated. With continuous iteration, the grasping posture will gradually approach the optimal posture from the initial posture. Finally, the generated grasping posture is verified in the IsaacGym physics simulator, and unstable grasps are filtered out.

[0064] Differentiable force closure optimization methods are highly sensitive to the initial hand pose. Optimizing from a random initial pose to a suitable target pose is not only time-consuming but also yields inconsistent optimization results. Therefore, a hand pose initialization method is designed: the GraspNet API is called to synthesize a large number of diverse 6D poses of a two-finger gripper for objects in the scene; for example... Figure 3 As shown, for each two-finger 6D pose, the repositioning from the two-finger 6D pose to the multi-finger dexterous hand palm pose is achieved by aligning the palm orientation of the dexterous hand with the orientation of the two-finger gripper and translating a certain distance in the opposite direction of the wrist; at the same time, a set of pre-grasp joint angles are defined to initialize the dexterous hand pose.

[0065] Because the initial poses retain diverse characteristics, the final synthesized multi-finger dexterous hand dataset is stable, collision-free, and diverse.

[0066] Step 2: To predict stable grasping postures in multi-object scenes, this invention designs a conditional diffusion generative model guided by local features. This model consists of two parts: a local feature extraction module and a grasping posture generation module. The local feature extraction module identifies the graspable region and extracts point-by-point local features; the grasping posture generation module models the grasping posture distribution based on local features. The combination of the generative model and local conditions allows the network to learn from a large number of local geometric changes in the dataset, thereby greatly improving generalization efficiency. The following are more detailed steps for constructing the local feature extraction module and the grasping posture generation module: Step 2.1: Modeling the local feature extraction module.

[0067] The local feature extraction module extracts features from single-view depth point clouds. Extract point-level features and generate object segmentation scores for each point. and crawling score This identifies the crawlable region. Based on these scores, a subset of high-scoring points is selected as feature seed points, and their local features are input into the subsequent crawling generation module. Compared to conditional methods based on global features, this approach achieves more efficient generalization.

[0068] In each training scenario, for each point on the object's surface Define crawl score This score represents the graspability of the area around the tag. Essentially, it's calculated by having each grasp tag heuristically "vote" for its nearest neighbors within its palm's reach. For each grasp tag in the scene... If point Located within the reach of its palm (distance threshold) ),but right The voting value is: ; in, To crawl The palm of the hand To crawl mass fraction, This is the attenuation coefficient.

[0069] The capture score for point p is the logarithm of the sum of all votes: ; in For the set of all valid crawls in the scene, experience shows This reflects the number of effective captures in its vicinity.

[0070] To extract local features and predict point-level grasping accuracy, given a scene point cloud ResUNet14, built on the Minkowski Engine, is used for each point. Extracting feature vectors This feature is then fed into the MLP to predict... crawling score and object segmentation fraction Finally, the top 1% of object points in terms of capture accuracy were selected and downsampled using farthest point sampling (FPS) to... =1024 points. These points are called feature seed points, denoted as... Its eigenvectors and crawling score This will be used for subsequent grasping posture generation.

[0071] Step 2.2: Model the grasping posture generation module.

[0072] The pose generation module receives the feature vector of the feature seed point determined in step 2.1. A generative model is used to generate multiple skillful grasping poses near the feature seed point. These grasping poses are then determined based on their estimated log-likelihood and the grasping degree score (GS) of the feature seed point. s Sort the data. The skillful grasping posture relative to the feature seed points is denoted as: ; in and This represents the wrist pose relative to the feature seed point. Represents the joint angles of the hand. Given a point cloud of a scene. Feature seed points The conditional probability distribution of all valid captures in its vicinity is as follows: ; in It is a point Predicted visual features. This paper assumes that in conditions, The distribution of is multimodal and quite complex, while and conditions, The distribution is single-modal. Therefore, a conditional generative model is used to predict... The conditional distribution, and based on a deterministic model. and predict .

[0073] A denoised diffusion probability model is used to approximate the probability distribution. To be Embedded in Euclidean space, the rotation matrix Flatten and translate By concatenating the vectors, we obtain a 12D vector representation: ; Subsequently, the conditional denoising model was trained. The random 12D Gaussian noise vector is transformed through an iterative process. Denoising to obtain an effective wrist pose vector At the diffusion time step The formula for generating noisy samples is: ; in , The diffusion coefficient is a predefined value. (Denoising model) Predict the diffusion rate and achieve noise reduction by solving ordinary differential equations: ; in , The time step interval is used. Finally, the denoised image is processed using SVD. Projection Back Space, obtain The sample probability is estimated by solving partial differential equations. The grasping postures are then sorted using the following formula: ; in These are the weighting coefficients. The wrist pose is sampled from the diffusion model. Then, the features of the feature seed points are... and Input MLP to predict joint angles To form a complete grasping posture .

[0074] Step 2.3: Joint training and loss function design.

[0075] In the local feature extraction module and the grasping pose generation module mentioned in steps 2.1 and 2.2, a joint training approach is adopted. In each gradient step, random samples are taken from the dataset. A scene, the depth point cloud of the scene is denoted as The corresponding object point mask is The truth point level capture score is Subsequently, random sampling was performed in the scene. One crawled tag Each crawl tag Represented relative to its "corresponding point". attitude in the reference frame .

[0076] Training loss consists of object segmentation loss , capture rate loss diffusion loss and joint angle loss The four components together form a linear combination of the total loss: ; in, , , and These are the object segmentation loss, gripping degree loss, diffusion loss, and joint angle loss, and their expressions are as follows: ; ; ; ; in, This represents the actual diffusion rate.

[0077] Step 3: Construct a crawling reachability estimator based on a stacked autoencoder to evaluate the kinematic reachability of each candidate crawl.

[0078] The conditional diffusion generation model based on local feature guidance proposed in step 2 only outputs a stable grasping posture during the grasping prediction stage. However, in real-world multi-object scenarios, the target object may be located in an inaccessible position for the robotic arm. Directly driving the robotic arm to approach the grasping posture with the highest grasping stability will lead to inverse kinematics failure due to the inaccessibility of the target object. Therefore, the robot needs to have the ability to evaluate the kinematic reachability of the predicted grasping posture. To this end, this invention proposes a grasping reachability estimator based on a stacked autoencoder to evaluate the kinematic reachability of each candidate grasping posture.

[0079] The reachability estimator employs a stacked autoencoder (SAE) based on softmax regression (SR) as the backbone of its network structure. It uses SAE for kinematic feature extraction and SR for reachability label classification. Figure 4 As shown, the network architecture of the reachability estimator consists of four layers: an input layer, two hidden layers of sizes 5 and 3, and a final softmax regression layer. The network is positioned in a 12-dimensional hand gesture. As input, the output is the reachability probability of the evaluated hand pose. and unreachability probability .

[0080] To enable the grasping reachability assessment network to fully learn and extract the motion features of the robotic arm, a reachability dataset containing a complete representation of the robotic arm's reachable workspace is needed for training. The reachability graph is the set of all possible poses that the end effector can reach. This invention constructs a reachability dataset based on the robotic arm reachability graph. The specific generation process of the reachability graph is as follows: First, the workspace of the robotic arm is discretized into a voxel mesh of equal size, and the workspace is randomly explored using the Forward Kinematics (FK) method. The FK method sets the reachability label of most voxels in the workspace to 1 (i.e., the calculated voxel center pose is reachable). While constructing the reachability graph using the FK method, the ratio of newly discovered voxels in the generated reachability graph is calculated in real time. Once the ratio reaches a predefined threshold, it indicates that the exploration process of the reachable space is trapped in the calculated voxels due to the randomness of the FK method. At this point, the Inverse Kinematics method is triggered, continuing to explore the workspace not reached by the FK method through uniform sampling, thereby completing the reachability graph constructed by the FK method. The final robotic arm reachability graph is as follows: Figure 5 As shown, by uniformly sampling the hand poses and their reachability labels in the reachability map, an reachability dataset containing 20,000 reachability labels was constructed.

[0081] The entire network employs a phased training approach. During the pre-training phase, SAE is trained using a cascading greedy algorithm, first encoding to extract features, then decoding to reconstruct the input. The encoding and decoding processes are formulated as follows: ； ； in, and These represent the weight matrices for the encoding and decoding processes, respectively. and This represents the corresponding bias vector. It is the activation function of the autoencoder. The goal of training a layer of an autoencoder is to adjust the weights. , , , To minimize the reconstruction error function between the input hand pose and the reconstructed hand pose: ; After obtaining the SAE pre-trained weights, use This serves as a training set to improve SR's ability to capture reachability label classification. This is the activation level of the last layer of SAE. The input is the reachability label of the hand gesture. Finally, to improve the predictive performance and robustness of the multi-finger dexterity hand grasping planning model, the weights of the entire network are fine-tuned using the backpropagation algorithm.

[0082] Finally, the attainable probability of the hand position. Multiply by the stability probability of the hand posture Obtain the probability of crawling The optimal grasping posture corresponds to the maximum feasible grasping probability. The overall algorithm framework is as follows: Figure 6 As shown.

[0083] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A clever grasping planning method based on conditional diffusion generation and reachability awareness, characterized in that, include: Initialize the grasping posture and obtain a multi-finger dexterity hand grasping simulation dataset, wherein the simulation dataset includes: scene point cloud data, stable grasping palm posture data and finger joint angle data; The multi-fingered dexterous hand grasping simulation dataset is input into the conditional diffusion generation model to obtain the complete grasping posture. The conditional diffusion generation model is used to extract local features in the scene point cloud and obtain the complete grasping posture based on the local features. A stacked autoencoder is trained based on the robotic arm reachability map to obtain a grasping reachability estimator. Input the complete grasping posture into the grasping reachability estimator to obtain the reachability probability; Based on the reachability probability and the grasping stability probability, the target grasping posture is obtained, and grasping is performed according to the target grasping posture.

2. The agile grasping planning method based on conditional diffusion generation and reachability awareness according to claim 1, characterized in that, Initialize the grasping posture and obtain the multi-finger dexterity hand grasping simulation dataset, including: Generate the 6D pose of a two-finger gripper by calling the GraspNet API; The two-finger gripper 6D pose is converted into a multi-finger dexterous hand palm pose, and a pre-grasping joint angle is assigned to obtain the initial multi-finger grasping posture. Based on the differential force closure energy, contact attraction energy, penetration repulsion energy, self-penetration energy, and joint over-limit energy, multiple energy functions are obtained; The initial multi-finger grasping posture is optimized by gradient descent using the multi-energy function to obtain the optimized grasping posture; The optimized grasping posture was verified in a physical simulator, and the filtered results formed the multi-finger dexterous hand grasping simulation dataset.

3. The agile grasping planning method based on conditional diffusion generation and reachability awareness according to claim 2, characterized in that, The multiple energy function is: ； in, It is a multi-energy function. Energy that can be closed by a minute force To attract energy through contact, In order to penetrate repulsive energy, As self-penetrating energy, For joint excess energy, , , and These represent the weights of the corresponding energy terms.

4. The agile grasping planning method based on conditional diffusion generation and reachability awareness according to claim 1, characterized in that, The conditional diffusion generation model includes: a local feature extraction module and a grasping posture generation module; The local feature extraction module is used to extract local features from the scene point cloud; The grasping posture generation module is used to obtain the complete grasping posture based on the local features.

5. The agile grasping planning method based on conditional diffusion generation and reachability awareness according to claim 4, characterized in that, Extracting local features from scene point clouds includes: Calculate each point on the surface of the object crawling score The crawling score , used to represent points The grabbability of the surrounding area; Points with the target grasp score and located on the object surface are selected as feature seed points, and a preset number of feature seed points are sampled from the farthest point to obtain the local features.

6. The agile grasping planning method based on conditional diffusion generation and reachability awareness according to claim 4, characterized in that, Obtaining the complete grasping posture based on the local features includes: Based on the local features, obtain the feature vector of the feature seed point; Using the aforementioned feature vector as a condition, the initial wrist pose is encoded into a target dimension vector; A forward diffusion process is applied to the target dimension vector to generate noisy samples; The noisy samples were used to train a denoising network to predict the diffusion rate, and the wrist posture vector was obtained by inverse denoising through ordinary differential equations. The denoised wrist pose vector is projected back into the preset space through matrix factorization (SVD) to obtain the target wrist pose. The target wrist posture and the feature vector of the feature seed point are concatenated and input into the MLP to regress the joint angle and obtain the complete grasping posture.

7. The agile grasping planning method based on conditional diffusion generation and reachability awareness according to claim 1, characterized in that, Training a stacked autoencoder based on the robotic arm reachability map to obtain a grasping reachability estimator includes: Discretize the workspace of the robotic arm into an equally spaced voxel grid; Based on the equidistant voxel grid, the end pose is randomly sampled using forward kinematics, and reachable voxels are marked. When constructing the accessibility map using the forward kinematics, if the ratio of newly discovered voxels is lower than the threshold, switch to inverse kinematics uniform sampling to complete the unexplored areas and obtain a complete accessibility map; Training samples are generated by uniformly sampling hand pose and hand pose reachability labels from the complete reachability map; The stacked autoencoder is trained using the training samples to obtain the crawling reachability estimator.

8. The agile grasping planning method based on conditional diffusion generation and reachability awareness according to claim 7, characterized in that, The stacked autoencoder is trained using the training samples to obtain the grasping reachability estimator, which includes a pre-training phase and a fine-tuning phase. The pre-training stage greedily trains the autoencoder layer by layer to minimize the mean square error between the input and the reconstruction. The fine-tuning phase uses reachability tags as supervision and updates all weights through backpropagation.

Images