Method and device for predicting grabbing pose of dexterous hand

Abstract

The invention belongs to the technical field of manipulators, and relates to a dexterous hand grabbing pose prediction method and device, and the method comprises the steps: obtaining an RGBD image of a target object; segmenting the RGBD image according to the semantic information of the RGBD image to obtain an image mask corresponding to the grabbable part of the target object; acquiring a depth point cloud of the target object; performing calculation processing on the depth point cloud according to the image mask to obtain a grabbable point cloud corresponding to a part of or all the image masks; and based on the grabbable point cloud, using a pose prediction model to predict and obtain the grabbing pose of the dexterous hand. According to the technical scheme, the semantic information of the RGBD image can be fully utilized in the process of predicting the grabbing pose of the dexterous hand, and it is ensured that the pose prediction model only uses part or all of the point clouds of the grabbable part of the target object as input to predict the grabbing pose of the dexterous hand; the technical problem that according to an existing dexterous hand grabbing pose prediction method, due to weak semantics of the point cloud, the predicted grabbing pose is wrong is solved.

Description

Technical Field

[0001] The present application relates to the technical field of manipulators, and in particular to a method and device for predicting the grasping posture of a dexterous hand. Background Art

[0002] A dexterous hand is a humanoid robotic arm with a palm and fingers that can precisely grasp various objects. Current research on grasping strategies for dexterous hands focuses on end-to-end grasping execution strategies and segmented strategies that first identify the grasping posture and then execute the grasping strategy. End-to-end grasping execution strategies have poor interpretability, require massive amounts of data for training, and exhibit poor generalization to new scenarios. However, segmented strategies that first identify the grasping posture and then execute the grasping strategy offer better interpretability. For segmented strategies, identifying the grasping posture is a critical prerequisite for their implementation.

[0003] Currently, recognition of the grasping pose of dexterous hands is primarily based on predictions based on deep learning models. These predictions include the overall pose SE3 of the dexterous hand and the finger joint angles Qpos. Because object point clouds contain rich geometric prior information, prediction models often use them as input, such as mainstream methods like GraspTTA, Unidexgrasp, and GenDexGrasp. Furthermore, due to the high degrees of freedom of dexterous hands, the search space required for computation is very large. To address this large search space, probabilistic models such as IPDF and Flow are often used to predict the overall pose SE3 and finger joint angles Qpos. These models are then sampled to obtain the grasping pose.

[0004] However, because probabilistic sampling can only generate one prediction parameter at a time, it is an object-level prediction and cannot correspond to a specific object part. Furthermore, to avoid predicting a grasping pose that collides with the environment, both the object point cloud and the background point cloud are typically required as input to the prediction model. However, due to the weak semantics of point clouds, the prediction model may mistakenly predict background point clouds as foreground points, resulting in an incorrect predicted grasping pose. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a method and device for predicting the grasping posture of a dexterous hand, so as to solve the technical problem that the existing dexterous hand grasping posture prediction method may cause the predicted grasping posture to be incorrect due to the weak semantics of the point cloud.

[0006] In a first aspect, an embodiment of the present application provides a method for predicting a grasping posture of a dexterous hand, the method comprising:

[0007] Get the RGBD image of the target object;

[0008] Segmenting the RGBD image according to semantic information of the RGBD image to obtain an image mask corresponding to a graspable part of the target object;

[0009] Acquire a depth point cloud of the target object;

[0010] performing computational processing on the depth point cloud according to the image mask to obtain a graspable point cloud corresponding to part or all of the image mask;

[0011] The dexterous hand grasping posture is predicted based on the graspable point cloud using a posture prediction model.

[0012] Optionally, the calculating and processing the depth point cloud according to the image mask to obtain a graspable point cloud corresponding to part or all of the image mask specifically includes:

[0013] The depth point cloud is filtered according to the image mask to obtain a graspable point cloud corresponding to all the image masks.

[0014] Optionally, the calculating and processing the depth point cloud according to the image mask to obtain a graspable point cloud corresponding to part or all of the image mask comprises the following steps:

[0015] filtering the depth point cloud according to the image mask to obtain foreground point clouds corresponding to all of the image masks;

[0016] A point cloud prediction model is used based on the foreground point cloud to predict a graspable point cloud corresponding to a portion of the image mask.

[0017] Optionally, the calculating and processing the depth point cloud according to the image mask to obtain a graspable point cloud corresponding to part or all of the image mask comprises the following steps:

[0018] Based on the depth point cloud, an initial point cloud is predicted using a point cloud prediction model;

[0019] The initial point cloud is filtered according to the image mask to obtain a graspable point cloud corresponding to a portion of the image mask.

[0020] Optionally, it is characterized in that the calculation step of obtaining the grasping posture of the dexterous hand based on the graspable point cloud predicted by the posture prediction model includes:

[0021] Based on the graspable point cloud, a simplified prediction model is used to predict a grasping pose of a simplified actuator;

[0022] Recalling a pre-stored posture conversion relationship between the dexterous hand and the simplified actuator;

[0023] The grasping posture of the dexterous hand is calculated according to the grasping posture of the simplified actuator and the posture conversion relationship.

[0024] Optionally, the calculating and obtaining the grasping posture of the dexterous hand according to the grasping posture of the simplified actuator and the posture conversion relationship comprises the following steps:

[0025] An initial posture is calculated based on the grasping posture of the simplified actuator and the posture conversion relationship;

[0026] generating a dexterous hand point cloud based on the initial posture, wherein the dexterous hand point cloud is a depth point cloud of the dexterous hand in the initial posture;

[0027] Inputting the dexterous hand point cloud and the depth point cloud of the target object into ContactNet to predict the ContactMap of the target object;

[0028] Calculate the minimum distance value between each valid contact point of the ContactMap and each point of the dexterous hand point cloud, use the minimum distance value as the current loss value of the loss function, update the initial posture by using an iterative algorithm to optimize the loss function until the preset convergence condition is met, and use the initial posture when the preset convergence condition is met as the dexterous hand grasping posture.

[0029] Optionally, the simplified actuator is a two-finger gripper.

[0030] Optionally, the posture prediction model uses a diffusion model.

[0031] Optionally, segmenting the RGBD image according to semantic information of the RGBD image to obtain an image mask corresponding to a graspable part of the target object specifically includes:

[0032] An image mask corresponding to a graspable part of the target object is obtained by performing any one of instance segmentation processing, open vocabulary detection processing, and affordance detection processing on the RGBD image.

[0033] In a second aspect, an embodiment of the present application provides a dexterous hand grasping posture prediction device, the device comprising:

[0034] Image acquisition module, used to acquire RGBD images of target objects;

[0035] a mask generation module, configured to segment the RGBD image according to semantic information of the RGBD image to obtain an image mask corresponding to a graspable part of the target object;

[0036] A point cloud acquisition module, used to acquire the depth point cloud of the target object;

[0037] a point cloud computing module, configured to perform computational processing on the depth point cloud according to the image mask to obtain a graspable point cloud corresponding to part or all of the image mask;

[0038] The posture prediction module is used to predict the grasping posture of the dexterous hand based on the graspable point cloud using a posture prediction model.

[0039] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0040] The dexterous hand grasping posture prediction method of the embodiment of the present application obtains an RGBD image of the target object, segments the RGBD image according to the semantic information of the RGBD image, and obtains an image mask of the graspable part of the target object; and obtains a depth point cloud of the target object, calculates and processes the depth point cloud according to the image mask to obtain a graspable point cloud corresponding to part or all of the image mask, and then uses a posture prediction model to predict the dexterous hand grasping posture based on the graspable point cloud, thereby fully utilizing the semantic information of the RGBD image in the process of predicting the dexterous hand grasping posture, ensuring that the posture prediction model only uses part or all of the point cloud of the graspable part of the target object as input to predict the dexterous hand grasping posture, thereby solving the technical problem that the existing dexterous hand grasping posture prediction method may cause the predicted grasping posture to be incorrect due to the weak semantics of the point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 A schematic diagram of a flow chart of a method for predicting a dexterous hand grasping posture according to an embodiment of the present application;

[0043] Figure 2 for Figure 1 A schematic diagram of a specific flow chart of step S400 in one embodiment;

[0044] Figure 3 for Figure 1 A schematic diagram of a specific flow chart of step S400 in another embodiment;

[0045] Figure 4 for Figure 1 Schematic diagram of calculation steps of the pose prediction model in step S500;

[0046] Figure 5 for Figure 4 Detailed flow chart of step A300 in FIG.

[0047] Figure 6 A schematic diagram of an example obtained by executing the dexterous hand grasping posture prediction method provided in one embodiment of the present application;

[0048] Figure 7 For Figure 6 Schematic diagram of the corresponding dexterous hand and simplified actuator posture;

[0049] Figure 8 A schematic diagram of the structure of a dexterous hand grasping posture prediction system provided in one embodiment of the present application;

[0050] Figure 9 A schematic diagram of the structure of a dexterous hand grasping posture prediction device provided in one embodiment of the present application;

[0051] Figure 10 A schematic diagram of the structure of a computer device provided in one embodiment of the present application.

[0052] Reference numerals:

[0053] 100. Dexterous hand grasping posture prediction system; 110. Controller; 120. Image generation device; 130. Point cloud generation device; 200. Dexterous hand grasping posture prediction device; 210. Image acquisition module; 220. Mask generation module; 230. Point cloud acquisition module; 240. Point cloud computing module; 250. Posture prediction module; 300. Computer device; 310. Memory; 320. Processor; 330. Network interface. DETAILED DESCRIPTION

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0055] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0056] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0057] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0058] See also Figure 1-7 The first part of the embodiment of the present application provides a method for predicting a grasping posture of a dexterous hand, comprising the following steps:

[0059] S100, obtaining an RGBD image of a target object;

[0060] S200, segmenting the RGBD image according to semantic information of the RGBD image to obtain an image mask corresponding to a graspable part of the target object;

[0061] S300, obtaining a depth point cloud of the target object;

[0062] S400, performing calculation processing on the depth point cloud according to the image mask to obtain a graspable point cloud corresponding to part or all of the image mask;

[0063] S500: Predicting a grasping posture of the dexterous hand using a posture prediction model based on the graspable point cloud.

[0064] It should be noted that the "grasping posture" of the dexterous hand in this application refers to the position and posture of the dexterous hand relative to the target object when performing a grasping operation. The term "grasping" in this application refers to the dexterous hand bending its palm and / or fingers to contact the object, thereby performing various human-like grasping, gripping, pinching, twisting, clamping, and hooking actions on the object, and does not refer solely to the act of grasping or gripping. In this application, the target object is the object of the dexterous hand's grasping action, which can be a single or multiple real-world objects. An RGBD image of a target object refers to an RGBD image containing the target object, which generally also includes background objects. A graspable portion of a target object refers to the portion of the target object that the dexterous hand can successfully grasp. For example, if the target objects are a laundry detergent bottle, a water cup, and a Bluetooth headset case, the graspable portions of the target object can be the handle of the laundry detergent bottle, the body of the water cup, and the entire surface of the Bluetooth headset case, respectively. For another example, if the target objects are water cups 1 and 2 placed side by side, the graspable portion of the target object can be defined as the body of water cup 1. In this application, the semantic information of an RGBD image refers to the abstract knowledge extracted from the color information and depth information of the RGBD image that can describe the category, attribute, function or spatial relationship of an object.

[0065] In step S100, illustratively, the RGBD image of the target object can be directly acquired by an RGBD camera such as structured light (Structured Light), time-of-flight (ToF) or active stereo vision (Active Stereo).

[0066] In step S200, various existing RGBD image semantic segmentation algorithms can be used to segment the RGBD image according to the semantic information of the RGBD image to obtain an image mask. This application is not limited to this algorithm, as long as it can ensure that an image mask corresponding to the graspable part of the target object is generated. It can be understood that because RGBD images have strong semantic information, segmenting the RGBD image according to the semantic information of the RGBD image can more accurately obtain an image mask corresponding to the graspable part of the target object. Therefore, in the subsequent step S500, the grasping pose of the dexterous hand can be predicted based only on the graspable point cloud corresponding to part or all of the image mask, avoiding the prediction of the grasping pose of the dexterous hand based on the point cloud of the non-graspable part of the target object, thereby improving the accuracy of the predicted grasping pose of the dexterous hand.

[0067] In step S300, the depth point cloud of the target object refers to a set of discrete points on the surface of the target object in three-dimensional space, where each point contains three-dimensional coordinates and depth information. For example, the depth point cloud of the target object can be directly obtained using common depth sensors such as RGBD cameras, LiDAR, or binocular / multi-camera stereo vision sensors, or can be indirectly generated using algorithms such as monocular depth estimation and structure from motion (SfM).

[0068] In step S400, the specific steps of calculating and processing the image mask based on the depth point cloud are not limited. It may include but is not limited to the following steps such as "filtering" and "predicting using a point cloud prediction model", as long as it can ensure that the graspable point cloud corresponds to part or all of the image mask. The word "corresponding" here means that each point of the graspable point cloud can be uniquely aligned with a pixel point in the RGBD image located in the image mask area. It can be understood that because the graspable point cloud corresponds to part or all of the image mask, and the image mask corresponds to the graspable part of the target object, the graspable point cloud is a point cloud belonging to the graspable part of the target object.

[0069] In step S500, the posture prediction model refers to a computational model that can predict the dexterous hand grasping posture based on point cloud data (e.g., graspable point cloud). It can be obtained by directly selecting various existing computational models that can predict the dexterous hand posture based on point cloud data, or by combining existing computational models or adding new computational steps. It can be understood that in the embodiment of the present application, since the dexterous hand grasping posture is predicted based on point cloud data (i.e., graspable point cloud) by directly using the posture prediction model, multiple dexterous hand grasping postures can be predicted for the target object by only executing steps S100-S500 once, thereby realizing point-level prediction of the dexterous hand grasping posture, providing more optional solutions for the dexterous hand grasping strategy, and increasing the diversity of the dexterous hand grasping strategy.

[0070] It should be noted that, since the depth point cloud needs to be calculated and processed to obtain the graspable point cloud in step S400, the depth point cloud must be calculated in the form of a feature vector, and the graspable point cloud obtained thereby must also be in the form of a feature vector. That is, the depth point cloud and graspable point cloud involved in step S400 specifically refer to the multidimensional feature vector of the depth point cloud and the multidimensional feature vector of the graspable point cloud, rather than the point cloud data itself. The multidimensional feature vector in this application refers to the feature vector containing the geometric information of each point in the point cloud. Similarly, since the graspable point cloud needs to be calculated in step S500, the foreground point cloud, initial point cloud and graspable point cloud need to be calculated in the specific steps of the following components of S400. Therefore, the graspable point cloud involved in step S500 and the foreground point cloud, initial point cloud and graspable point cloud involved in the specific steps of the following components of S400 all refer to the multidimensional feature vectors of each point cloud, rather than the individual point clouds themselves. Similarly, the dexterous hand point cloud in the following step A330 also refers to its multi-dimensional feature vector.

[0071] Exemplarily, the multidimensional feature vector of the depth point cloud can be obtained by performing the following steps after step S300: inputting the depth point cloud into a point cloud feature extraction model for feature extraction to obtain a multidimensional feature vector for each point of the depth point cloud.

[0072] Among them, the multidimensional feature vector of each point of the depth point cloud refers to a feature vector containing the geometric information of each point of the depth point cloud, that is, it contains the coordinates (x, y, z) of each point of the depth point cloud in three-dimensional space, and can also contain additional features such as normal vector and curvature that represent geometric information or semantic information on this basis. For example, when the depth point cloud consists of N points, the multidimensional feature vector of its X-th point can be simplified to NX(3+C), where "3" represents the three-dimensional space coordinates and "C" represents the C additional features in addition to the three-dimensional space coordinates. Specifically, the point cloud feature extraction model can be, for example, a neural network model such as PointNet, PointNet++, or PointCNN.

[0073] The dexterous hand grasping posture prediction method of the embodiment of the present application obtains an RGBD image of the target object, segments the RGBD image according to the semantic information of the RGBD image, and obtains an image mask of the graspable part of the target object; and obtains a depth point cloud of the target object and uses a posture prediction model to predict the dexterous hand grasping posture, thereby fully utilizing the semantic information of the RGBD image in the process of predicting the dexterous hand grasping posture, ensuring that the posture prediction model only uses part or all of the point cloud of the graspable part of the target object as input to predict the dexterous hand grasping posture, thereby solving the technical problem that the existing dexterous hand grasping posture prediction method may cause the predicted grasping posture to be incorrect due to the weak semantics of the point cloud.

[0074] In one embodiment, step S200 specifically includes:

[0075] An image mask corresponding to a graspable part of the target object is obtained by performing any one of instance segmentation processing, open vocabulary detection processing, and affordance detection processing on the RGBD image.

[0076] Instance segmentation involves assigning a class label to each pixel in an RGBD image, thereby generating an image mask representing the graspable parts of the target object based on the class label. For example, convolutional neural network (CNN) methods such as SegNet, U-Net, and FuseNet can be used to assign a class label to each pixel in an RGBD image, thereby segmenting the RGBD image.

[0077] Open vocabulary detection involves locating the graspable parts of a target object in an RGBD image using a user-defined vocabulary, and then generating an image mask of the graspable parts of the target object. For example, open vocabulary detection can be implemented using the following process: extracting global features from the RGBD image using a CLIP encoder; encoding user-defined vocabulary (e.g., "the body of a water cup") into text features; and calculating the similarity between the global features and the text features to generate an image mask of the graspable parts of the target object.

[0078] Affordance detection processing refers to extracting image feature information from RGBD images, and then predicting the functional label of the target object (such as the AffCorner dataset) through a neural network model (such as using the AffordanceNet dataset) based on the image feature information to generate an image mask of the graspable part of the target object.

[0079] It can be understood that in this embodiment, by using any of the processing methods of instance segmentation, open vocabulary detection, and affordance detection to segment the RGBD image, an image mask can be accurately obtained, thereby enabling step S400 to be successfully executed, thereby fully utilizing the semantic information of the RGBD image and ensuring that the pose prediction model only uses the point cloud of the graspable part of the target object as input to predict the grasping pose of the dexterous hand. In addition, when the open vocabulary detection processing method is used to segment the RGBD image to obtain the image mask, it also achieves the guidance of the dexterous hand grasping pose generation based on the fusion of image information and text information (or voice information, etc.), which is conducive to the multimodal information interaction between the dexterous hand and the external environment.

[0080] In some other embodiments, step S200 may also execute other existing image semantic segmentation algorithms to segment the RGBD image according to the semantic information of the RGBD image to obtain the image mask, which is not limited in this application.

[0081] In one embodiment, step S400 specifically refers to: filtering the depth point cloud according to the image mask to obtain a graspable point cloud corresponding to all the image masks.

[0082] It should be noted that the above-mentioned “filtering” refers to using the image mask as a filtering condition to remove points in the depth point cloud that cannot correspond to the image mask, thereby obtaining a graspable point cloud corresponding to the entire image mask.

[0083] It can be understood that in this embodiment, the depth point cloud is filtered according to the image mask to obtain a graspable point cloud corresponding to the entire image mask, so that the dexterous hand grasping posture can be directly predicted based on the graspable point cloud by executing step S500, making the overall calculation process relatively simple.

[0084] See also Figure 2 In one embodiment, step S400 includes the following steps:

[0085] S410, filtering the depth point cloud according to the image mask to obtain foreground point clouds corresponding to all the image masks;

[0086] S420 : Predicting a graspable point cloud corresponding to a portion of the image mask using a point cloud prediction model based on the foreground point cloud.

[0087] It should be noted that the "filtering" in step S410 refers to using the image mask as a filtering condition to remove points in the depth point cloud that cannot correspond to the image mask, thereby obtaining a foreground point cloud corresponding to the entire image mask. The point cloud prediction model in step S420 refers to an existing computing model that can perform calculations based on the input original point cloud data (such as the foreground point cloud) to remove some unnecessary point cloud data and output more ideal useful point cloud data (such as a graspable point cloud). Therefore, by executing step S420, a graspable point cloud corresponding only to part of the image mask can be obtained. Exemplarily, the point cloud prediction model of this embodiment can use a multilayer perceptron (MLP) to facilitate the smooth and efficient execution of step S410 and ensure accurate prediction of the graspable point cloud. In some other embodiments, the point cloud prediction model can also use a convolutional neural network (CNN).

[0088] It can be understood that although the graspable part of the target object is the part that enables the dexterous hand to successfully perform the grasping action, it does not mean that every point of the corresponding foreground point cloud is suitable for the dexterous hand to successfully perform the grasping action. For example, when the target object is a laundry detergent bottle, the graspable part of the target object is the handle of the laundry detergent bottle, but not every point of the foreground point cloud corresponding to the handle is suitable for the dexterous hand to successfully perform the grasping action. In this example, the graspable point cloud is obtained by using the point cloud prediction model based on the foreground point cloud prediction, so that the graspable point cloud only corresponds to a part of the image mask, and then the dexterous hand grasping pose is predicted based on the graspable point cloud using the pose prediction model, rather than directly predicting the dexterous hand grasping pose based on the foreground point cloud using the pose prediction model. This can make the final predicted dexterous hand grasping pose more accurate.

[0089] See also Figure 3 In another embodiment, step S400 includes the following steps:

[0090] S410, using a point cloud prediction model to predict an initial point cloud based on the depth point cloud;

[0091] S420: Filter the initial point cloud according to the image mask to obtain a graspable point cloud corresponding to a portion of the image mask.

[0092] It should be noted that the point cloud prediction model in step S410 refers to an existing computational model that can perform computational processing based on the input raw point cloud data (e.g., a depth point cloud) to remove some unnecessary point cloud data and thereby output more ideal useful point cloud data (e.g., an initial point cloud). Therefore, the initial point cloud obtained by executing step S410 may only correspond to a portion of the image mask. The "filtering" in step S420 refers to using the image mask as a filtering condition to remove points in the initial point cloud that do not correspond to the image mask, thereby obtaining a foreground point cloud that corresponds to the portion of the image mask.

[0093] Understandably, the depth point cloud corresponding to all points on the surface of the target object is not always suitable for the dexterous hand to successfully perform a grasping action. Therefore, in step S410, an initial point cloud can be first predicted based on the depth point cloud using a point cloud prediction model, making the initial point cloud more suitable for the dexterous hand to successfully perform a grasping action than the depth point cloud. In this example, by first using the point cloud prediction model to predict the initial point cloud based on the depth point cloud, then filtering the initial point cloud according to the image mask to obtain a graspable point cloud, and finally using the pose prediction model to predict the grasping pose of the dexterous hand based on the graspable point cloud, rather than directly using the pose prediction model to predict the grasping pose of the dexterous hand based on the depth point cloud or the initial point cloud, the final predicted grasping pose of the dexterous hand can be made more accurate.

[0094] Exemplarily, the point cloud prediction model can use a multilayer perceptron (MLP) or a convolutional neural network (CNN). When the point cloud prediction model uses a multilayer perceptron, step S410 can be executed smoothly and efficiently and accurate prediction of the initial point cloud can be ensured.

[0095] In one embodiment, the posture prediction model in step S500 uses a diffusion model, so that the specific execution steps of step S500 are: using the diffusion model to predict the grasping posture of the dexterous hand based on the graspable point cloud.

[0096] Among them, the diffusion model refers to a type of generative model that decomposes the data generation process into two complementary stages: the forward diffusion process (Forward Process) and the reverse generation process (Reverse Process) by simulating diffusion phenomena in nature (such as heat diffusion, ink diffusion in water, etc.), such as DDPM (Denoising Diffusion Probabilistic Models), DDIM (Denoisson Diffusion Implicit Models) and Latent Diffusion Models. The aforementioned forward diffusion process refers to gradually adding Gaussian noise to the original data (such as the crawlable point cloud in this application) until the data is completely converted into a random noise distribution, and the aforementioned reverse generation process refers to training the neural network model to learn how to gradually remove noise in order to generate new data from the noise.

[0097] Specifically, in one embodiment, the diffusion model in step S500 is a conditional diffusion model. By using the graspable point cloud obtained in step S400 as a condition of the conditional diffusion model, the conditional diffusion model can be used to predict the grasping posture of the dexterous hand.

[0098] See also Figure 4 In one embodiment, the step of calculating the dexterous hand grasping posture by the posture prediction model based on the graspable point cloud in step S500 includes:

[0099] A100, using a simplified prediction model to predict a grasping pose of a simplified actuator based on the graspable point cloud;

[0100] A200, calling a pre-stored posture conversion relationship between the dexterous hand and the simplified actuator;

[0101] A300. Calculate the grasping posture of the dexterous hand according to the grasping posture of the simplified actuator and the posture conversion relationship.

[0102] Among them, the simplified prediction model can use a multilayer perceptron (MLP) to ensure that the simplified prediction model can accurately predict the grasping posture of the simplified actuator based on the graspable point cloud. It should be noted that since the input and output of the multilayer perceptron selected for the simplified prediction model in step A100 here are different from the input and output of the multilayer perceptron selected for the point cloud prediction model in the above step S400, the model parameters of the two are also different in this application. A simplified actuator refers to other end effectors that can realize grasping actions with less degrees of freedom than a dexterous hand, such as a common mechanical gripper. The grasping posture of a simplified actuator is the overall posture and related parameters of its motion joints when the simplified actuator is used to realize the grasping action on the target object.

[0103] The posture conversion relationship in step A200 can be calculated in advance based on the structural size parameters of the simplified actuator and the dexterous hand itself and the specific grasping actions that the dexterous hand needs to perform for the target object (such as grasping, holding, pinching, etc.) and pre-stored. For example, it can be stored in the memory of the controller of the dexterous hand grasping posture prediction system described below, so that it can be successfully called when executing step A200, thereby enabling step A300 to calculate the dexterous hand grasping posture based on the posture conversion relationship and the grasping posture of the simplified actuator.

[0104] It can be understood that in this embodiment, by first using a simplified prediction model to predict the grasping pose of the simplified actuator, and then calling the pose conversion relationship and calculating the grasping pose of the dexterous hand based on the grasping pose of the simplified actuator and the pose conversion relationship, since the simplified actuator has fewer degrees of freedom than the dexterous hand, the solution search space of the simplified prediction model can be effectively reduced, thereby effectively reducing the solution search space of the pose prediction model, and correspondingly solving the problem of the large solution search space required in the existing dexterous hand grasping pose prediction method. In addition, in the dexterous hand grasping pose calculation process, only the simplified prediction model is used to predict the grasping pose of the simplified actuator based on the graspable point cloud. Since the simplified actuator has fewer degrees of freedom than the dexterous hand, the simplified prediction model will consider more local geometric information of the graspable point cloud during calculation, so that the dexterous hand grasping pose prediction method of this application has better generalization and transferability. In addition, using the dexterous hand grasping posture prediction method of this embodiment, the corresponding dexterous hand grasping posture can be predicted according to the various grasping actions that the dexterous hand needs to perform on the target object, so that the prediction method of this embodiment can be widely used in various dexterous hand grasping posture prediction scenarios.

[0105] See also Figure 6-7 , Figure 6A schematic diagram of an example obtained by executing the dexterous hand grasping pose prediction method provided in an embodiment of the present application, which represents the example results obtained after executing each step of the dexterous hand grasping pose prediction method in the form of intuitively visible pictures, wherein sequence number ① refers to the above-mentioned segmentation of the RGBD image according to the semantic information of the RGBD image to obtain an image mask corresponding to the graspable part of the target object; sequence number ② refers to the above-mentioned input of the depth point cloud into the point cloud feature extraction model for feature extraction to obtain a multi-dimensional feature vector for each point of the depth point cloud; Figure 6 NX(3+C) in the figure is the multidimensional feature vector of the Xth point of the simplified depth point cloud composed of N points; serial number ③ refers to the above-mentioned filtering of the depth point cloud according to the image mask to obtain the foreground point cloud corresponding to the entire image mask; serial number ④ refers to the above-mentioned prediction of the graspable point cloud corresponding to part of the image mask using the point cloud prediction model based on the foreground point cloud; serial number ⑤ refers to the above-mentioned prediction of the graspable point cloud using the simplified prediction model to obtain the grasping pose of the simplified actuator; serial number ⑥ refers to the following calculation of the initial pose based on the grasping pose of the simplified actuator and the pose conversion relationship; serial number ⑦ refers to the following steps A330 and A340. Figure 7 and Figure 6 correspond, Figure 7 A schematic diagram of the postures of the dexterous hand and the simplified actuator when calculating the posture conversion relationship between the dexterous hand and the simplified actuator.

[0106] See also Figure 6-7 In one embodiment, the simplified actuator uses a two-finger gripper. For example, the bending angle of the fingers of the dexterous hand can be adjusted to Figure 7 The posture shown in the figure makes the width between the thumb and index finger of the dexterous hand equal to the width between the two fingers of the two-finger gripper, and makes the remaining fingers of the dexterous hand and the index finger perpendicular to each other. Under this condition, the posture transformation relationship between the dexterous hand and the simplified actuator is calculated. In this embodiment, the grasping posture of the simplified actuator includes the overall orientation, rotation angle, width between the two fingers, and gripping depth of the two-finger gripper.

[0107] See also Figure 5 In one embodiment, step A300 includes the following steps:

[0108] A310, calculating an initial posture according to the grasping posture of the simplified actuator and the posture conversion relationship;

[0109] A320: Generate a dexterous hand point cloud based on the initial posture, where the dexterous hand point cloud is a depth point cloud of the dexterous hand in the initial posture;

[0110] A330, inputting the dexterous hand point cloud and the depth point cloud of the target object into ContactNet to predict a ContactMap of the target object;

[0111] A340. Calculate the minimum distance between each valid contact point of the ContactMap and each point of the dexterous hand point cloud, use the minimum distance value as the current loss value of the loss function, update the initial posture by using an iterative algorithm to optimize the loss function until a preset convergence condition is met, and use the initial posture when the preset convergence condition is met as the grasping posture of the dexterous hand.

[0112] It should be noted that the initial pose in step A310 is calculated directly based on the grasping pose of the simplified actuator and the pose transformation relationship. This means the pose required for the dexterous hand to successfully grasp the target object using the simplified actuator posture corresponding to the pose transformation relationship. The specific steps for generating the dexterous hand point cloud based on the initial pose in step A320 may, for example, include obtaining a 3D model of the dexterous hand and calculating the dexterous hand point cloud based on the initial pose and the 3D model of the dexterous hand. The ContactNet in step A330 is a deep learning-based neural network model that can be used to predict contact areas on the surface of an object suitable for grasping by the dexterous hand, for example, to predict a ContactMap. The ContactMap (contact probability distribution map) in steps A330 and A340 is used to quantify the contact suitability between the dexterous hand and various areas on the surface of the target object, representing the probability that each point on the target object surface can achieve effective contact with the dexterous hand. The point where the predetermined probability value is reached is the effective contact point in step A340. The "loss function" in step A340 refers to a function that calculates the minimum distance value from the initial pose corresponding to the current loss value (i.e., the minimum distance value). For example, the iterative algorithm used in step A340 may be a gradient descent method, and "satisfying a preset convergence condition" in step A340 may be that the number of iterative calculations reaches a preset number, or that the error between the current initial pose and the previous initial pose is less than a preset error value, or that the current loss value is less than a preset loss value.

[0113] In this embodiment, since the dexterous hand point cloud and the target object's depth point cloud respectively describe the geometric shape, surface details, and other geometric information of the dexterous hand and the target object, the ContactMap of the target object can be predicted by inputting the two into ContactNet in step A330. Furthermore, since each valid contact point in the ContactMap represents a point suitable for contact when the dexterous hand grasps the target object, the calculated minimum distance value can be used as the current loss value of the loss function in step A340 to represent the error between the current initial pose and the dexterous hand grasping pose to be predicted. Furthermore, in step A340, the initial pose can be updated using an iterative algorithm to optimize the loss function until a preset convergence condition is satisfied. The initial pose that satisfies the preset convergence condition is used as the dexterous hand grasping pose, thereby achieving the purpose of accurately predicting the dexterous hand grasping pose.

[0114] It can be understood that the initial posture obtained by executing step A310 is only predicted by treating the dexterous hand as the corresponding simplified actuator posture, and does not fully consider all the degrees of freedom of all the fingers of the dexterous hand. Therefore, it cannot guarantee that the dexterous hand will successfully perform the grasping action. For example, if the dexterous hand directly uses the initial posture to grasp the target object, some fingers (such as fingers on the dexterous hand that do not correspond to the simplified actuator) may interfere with the target object or background objects. By completely executing steps A310-A340 of this embodiment and using an iterative algorithm to update the initial posture, the initial posture can be made more suitable for the dexterous hand to grasp the target object, further ensuring the accuracy of the predicted dexterous hand grasping posture, thereby ensuring the high degree of freedom and flexibility of the dexterous hand's grasping.

[0115] See also Figure 8 The second part of the embodiment of the present application provides a dexterous hand grasping pose prediction system 100, so that the dexterous hand grasping pose prediction method of the embodiment of the present application can be implemented based on the prediction system 100. The prediction system 100 includes a controller 110, an image generation device 120 and a point cloud generation device 130. The controller 110 is used to execute the above-mentioned dexterous hand grasping pose prediction method. The image generation device 120 is used to generate an RGBD image of the target object, and the image generation device 120 is communicatively connected to the controller 110 so that the controller 110 can obtain the RGBD image of the target object; the point cloud generation device 130 is used to generate a depth point cloud of the target object, and the point cloud generation device 130 is communicatively connected to the controller 110 so that the controller 110 can obtain the depth point cloud of the target object.

[0116] It is understood that the controller 110 in the embodiment of the present application is not necessarily equivalent to the control device of the dexterous hand itself in the dexterous hand grasping posture prediction method described in this application. The controller 110 in the embodiment of the present application is only used to execute the dexterous hand grasping posture prediction method to generate the dexterous hand grasping posture, while the control device of the dexterous hand itself is used to control the dexterous hand to perform specific actions based on the dexterous hand grasping posture. Therefore, the controller 110 in the embodiment of the present application can be set on the dexterous hand or set independently of the dexterous hand.

[0117] Exemplarily, the controller 110 of the embodiment of the present application can be, but is not limited to: a computer terminal (Personal Computer, PC); an industrial control computer terminal (Industrial Personal Computer, IPC); a mobile terminal; a server; a system including a terminal and a server, and implemented through the interaction between the terminal and the server; a programmable logic controller (Programmable Logic Controller, PLC); a field programmable gate array (Field-Programmable Gate Array, FPGA); a digital signal processor (Digital Signal Processor, DSP) or a microcontroller unit (Microcontroller unit, controller), etc.

[0118] For example, the image generation device 120 of the embodiment of the present application may be, but is not limited to, a depth camera such as a structured light, time-of-flight (ToF), or active stereo vision camera. The point cloud generation device 130 of the embodiment of the present application may be, but is not limited to, a depth camera such as a structured light, time-of-flight (ToF), or active stereo vision camera, a LiDAR, a 3D scanner, or a multi-view camera.

[0119] It should be noted that when the image generating device 120 and the point cloud generating device 130 both use the same depth camera, they can actually use the same device.

[0120] As an implementation of the above method, an embodiment of the present application also provides a dexterous hand grasping posture prediction device. The embodiment of the device corresponds to the above-mentioned prediction method embodiment. The device can be specifically applied to the controller 110 of the dexterous hand grasping posture prediction system 100 described in the embodiment of the present application.

[0121] See also Figure 9 In one embodiment, the dexterous hand grasping posture prediction device 200 includes:

[0122] An image acquisition module 210 is used to acquire an RGBD image of a target object;

[0123] a mask generation module 220 for segmenting the RGBD image according to semantic information of the RGBD image to obtain an image mask corresponding to a graspable part of the target object;

[0124] The point cloud acquisition module 230 is used to acquire the depth point cloud of the target object;

[0125] a point cloud computing module 240 for computing and processing the depth point cloud according to the image mask to obtain a graspable point cloud corresponding to part or all of the image mask;

[0126] The pose prediction module 250 is configured to predict the grasping pose of the dexterous hand based on the graspable point cloud using a pose prediction model.

[0127] Specifically, in one embodiment, the point cloud computing module 240 includes:

[0128] a foreground point cloud computing submodule, configured to filter the depth point cloud according to the image mask to obtain foreground point clouds corresponding to all the image masks;

[0129] The graspable point cloud prediction submodule is configured to predict a graspable point cloud corresponding to a portion of the image mask using a point cloud prediction model based on the foreground point cloud.

[0130] Specifically, in another embodiment, the point cloud computing module 240 includes:

[0131] An initial point cloud submodule, configured to predict an initial point cloud based on the depth point cloud using a point cloud prediction model;

[0132] The submodule for filtering the graspable point cloud is configured to filter the initial point cloud according to the image mask to obtain a graspable point cloud corresponding to a portion of the image mask.

[0133] Specifically, in one embodiment, the posture prediction module 250 includes:

[0134] A simplified prediction submodule, configured to predict a grasping pose of a simplified actuator based on the graspable point cloud using a simplified prediction model;

[0135] A conversion relationship calling submodule, used to call a pre-stored posture conversion relationship between the dexterous hand and the simplified actuator;

[0136] The posture calculation submodule is used to calculate the grasping posture of the dexterous hand according to the grasping posture of the simplified actuator and the posture conversion relationship.

[0137] More specifically, in one embodiment, the posture calculation submodule includes:

[0138] An initial posture calculation submodule, configured to calculate an initial posture according to the grasping posture of the simplified actuator and the posture conversion relationship;

[0139] a dexterous hand point cloud generation submodule, configured to generate a dexterous hand point cloud based on the initial posture, wherein the dexterous hand point cloud is a depth point cloud of the dexterous hand in the initial posture;

[0140] A probability distribution map generation submodule is used to input the dexterous hand point cloud and the depth point cloud of the target object into ContactNet to predict the ContactMap of the target object;

[0141] The iterative optimization submodule is used to calculate the minimum distance value between each effective contact point of the ContactMap and each point of the dexterous hand point cloud, use the minimum distance value as the current loss value of the loss function, update the initial posture by using an iterative algorithm to optimize the loss function until the preset convergence condition is met, and use the initial posture when the preset convergence condition is met as the grasping posture of the dexterous hand.

[0142] See also Figure 10 , an embodiment of the present application also provides a computer device 300. The computer device 300 can be a terminal or a server. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network), as well as basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this application.

[0143] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device 300 here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0144] The memory 310 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory 310 may be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 may also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory 310 may also include both the internal storage unit of the computer device 300 and its external storage device. In this embodiment, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as the program code of the dexterous hand grasping pose prediction method. In addition, the memory 310 can also be used to temporarily store various data that has been output or will be output.

[0145] In some embodiments, the processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 320 is generally used to control the overall operation of the computer device 300. In this embodiment, the processor 320 is used to execute program code stored in the memory 310 or process data, such as executing program code for a dexterous hand grasping pose prediction method.

[0146] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0147] An embodiment of the present application also provides a computer-readable storage medium, which stores a control program of the dexterous hand grasping posture prediction method. The control program of the dexterous hand grasping posture prediction method can be executed by at least one processor to enable the at least one processor to perform the steps of the dexterous hand grasping posture prediction method as described above.

[0148] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A method for predicting the grasping posture of a dexterous hand, characterized in that: The following steps are involved: Get the RGBD image of the target object; Segmenting the RGBD image according to semantic information of the RGBD image to obtain an image mask corresponding to a graspable part of the target object; Acquire a depth point cloud of the target object; performing computational processing on the depth point cloud according to the image mask to obtain a graspable point cloud corresponding to part or all of the image mask; The dexterous hand grasping posture is predicted based on the graspable point cloud using a posture prediction model.

2. The dexterous hand grasping posture prediction method according to claim 1, characterized in that: The calculating and processing the depth point cloud according to the image mask to obtain a graspable point cloud corresponding to part or all of the image mask specifically includes: The depth point cloud is filtered according to the image mask to obtain a graspable point cloud corresponding to all the image masks.

3. The dexterous hand grasping posture prediction method according to claim 1, characterized in that: The calculating and processing the depth point cloud according to the image mask to obtain a graspable point cloud corresponding to part or all of the image mask comprises the following steps: filtering the depth point cloud according to the image mask to obtain foreground point clouds corresponding to all of the image masks; A point cloud prediction model is used based on the foreground point cloud to predict a graspable point cloud corresponding to a portion of the image mask.

4. The dexterous hand grasping posture prediction method according to claim 1, characterized in that: The calculating and processing the depth point cloud according to the image mask to obtain a graspable point cloud corresponding to part or all of the image mask comprises the following steps: Based on the depth point cloud, an initial point cloud is predicted using a point cloud prediction model; The initial point cloud is filtered according to the image mask to obtain a graspable point cloud corresponding to a portion of the image mask.

5. The dexterous hand grasping posture prediction method according to any one of claims 1 to 4, characterized in that: The calculation steps of the posture prediction model to obtain the grasping posture of the dexterous hand based on the graspable point cloud prediction include: Based on the graspable point cloud, a simplified prediction model is used to predict a grasping pose of a simplified actuator; Recalling a pre-stored posture conversion relationship between the dexterous hand and the simplified actuator; The grasping posture of the dexterous hand is calculated according to the grasping posture of the simplified actuator and the posture conversion relationship.

6. The dexterous hand grasping posture prediction method according to claim 5, characterized in that: The method of calculating the grasping posture of the dexterous hand according to the grasping posture of the simplified actuator and the posture conversion relationship comprises the following steps: An initial posture is calculated based on the grasping posture of the simplified actuator and the posture conversion relationship; generating a dexterous hand point cloud based on the initial posture, wherein the dexterous hand point cloud is a depth point cloud of the dexterous hand in the initial posture; Inputting the dexterous hand point cloud and the depth point cloud of the target object into ContactNet to predict the ContactMap of the target object; Calculate the minimum distance value between each valid contact point of the ContactMap and each point of the dexterous hand point cloud, use the minimum distance value as the current loss value of the loss function, update the initial posture by using an iterative algorithm to optimize the loss function until the preset convergence condition is met, and use the initial posture when the preset convergence condition is met as the dexterous hand grasping posture.

7. The dexterous hand grasping posture prediction method according to claim 5 or 6, characterized in that: The simplified actuator is a two-finger gripper.

8. The method for predicting the grasping posture of a dexterous hand according to any one of claims 1 to 4, characterized in that: The posture prediction model uses a diffusion model.

9. The method for predicting the grasping posture of a dexterous hand according to claim 1, characterized in that: Segmenting the RGBD image according to the semantic information of the RGBD image to obtain an image mask corresponding to the graspable part of the target object specifically includes: An image mask corresponding to a graspable part of the target object is obtained by performing any one of instance segmentation processing, open vocabulary detection processing, and affordance detection processing on the RGBD image.

10. A dexterous hand grasping posture prediction device, characterized in that: The device comprises: Image acquisition module, used to acquire RGBD images of target objects; a mask generation module, configured to segment the RGBD image according to semantic information of the RGBD image to obtain an image mask corresponding to a graspable part of the target object; A point cloud acquisition module, used to acquire the depth point cloud of the target object; a point cloud computing module, configured to perform computational processing on the depth point cloud according to the image mask to obtain a graspable point cloud corresponding to part or all of the image mask; The posture prediction module is used to predict the grasping posture of the dexterous hand based on the graspable point cloud using a posture prediction model.

Images