Training method, grasping method, device and electronic device for gesture generation model

Abstract

The present application provides a training method, a grasping method, a device and an electronic device for a gesture generation model. The training method for the gesture generation model includes: obtaining a first training data set; constructing a gesture generation model and a meta-learner, wherein the gesture generation model is used to predict gesture information when a dexterous hand grasps an object, and the meta-learner is used to adjust the model parameters of the gesture generation model during the training process; using the first training data set and the meta-learner to train the gesture generation model to obtain a trained gesture generation model. The present application can obtain a gesture generation model with high prediction accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of humanoid robot dexterous hands, and in particular to a training method, grasping method, device and electronic equipment for a gesture generation model. Background Art

[0002] The dexterous hand is the end effector of the embodied robot interacting with the environment, and is a machine that deeply imitates the human hand. The adaptive grasping of objects by the dexterous hand in complex and unknown environments is limited by the ability of the model, but the existing model has the problem of low accuracy in predicting the grasping gesture, which in turn leads to the problem of low grasping efficiency when the dexterous hand uses the current model for complex, unknown or irregularly shaped objects. Therefore, a model with high accuracy in predicting the grasping gesture of the dexterous hand is urgently needed. Summary of the invention

[0003] Multiple aspects of the present application provide a training method, a grasping method, a device and an electronic device for a gesture generation model to solve the problem of low prediction accuracy of existing models used in dexterous hands.

[0004] The first aspect of the present application provides a method for training a gesture generation model, comprising:

[0005] Obtain a first training data set;

[0006] Constructing a gesture generation model and a meta-learner. The gesture generation model is used to predict the gesture information when the dexterous hand grasps an object. The meta-learner is used to adjust the model parameters of the gesture generation model during the training process.

[0007] The first training data set and the meta-learner are used to train the gesture generation model to obtain a trained gesture generation model.

[0008] A second aspect of the present application provides a grasping method, comprising: determining target point cloud data of a target object;

[0009] Inputting the target point cloud data into a gesture generation model for processing to obtain gesture information, wherein the gesture generation model is trained by the training method of the gesture generation model of the first aspect;

[0010] Control the dexterous hand to grasp the target object according to the gesture information.

[0011] A third aspect of the present application provides a training device for a gesture generation model, comprising:

[0012] An acquisition module, used to acquire a first training data set;

[0013] A building module is used to build a gesture generation model and a meta-learner. The gesture generation model is used to predict the gesture information when the dexterous hand grasps an object. The meta-learner is used to adjust the model parameters of the gesture generation model during the training process.

[0014] The training module is used to train the gesture generation model using the first training data set and the meta-learner to obtain a trained gesture generation model, and the meta-learner is used to adjust the model parameters of the gesture generation model during the training process.

[0015] A fourth aspect of the present application provides a gripping device, comprising:

[0016] A determination module, used for determining target point cloud data of a target object;

[0017] A processing module, used for inputting the target point cloud data into a gesture generation model for processing to obtain gesture information, wherein the gesture generation model is trained by the training method of the gesture generation model of the first aspect;

[0018] The control module is used to control the dexterous hand to grasp the target object according to the gesture information.

[0019] A fifth aspect of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the first aspect and / or the second aspect when executing the computer program.

[0020] In a sixth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method of the first aspect and / or the second aspect.

[0021] In a seventh aspect, the present application provides a computer program product, which includes: a computer program, which is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the method of the first aspect and / or the second aspect.

[0022] The present application is applied in the grasping scenario of dexterous hands. The present application obtains a first training data set; constructs a gesture generation model and a meta-learner, wherein the gesture generation model is used to predict gesture information when the dexterous hands grasp an object, and the meta-learner is used to adjust the model parameters of the gesture generation model during the training process; uses the first training data set and the meta-learner to train the gesture generation model to obtain a trained gesture generation model, which can obtain a gesture generation model with high prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1 An application scenario diagram of a training method for a gesture generation model provided by an exemplary embodiment of the present application;

[0025] Figure 2 A flowchart of a method for training a gesture generation model provided by an exemplary embodiment of the present application;

[0026] Figure 3 A flowchart of another method for training a gesture generation model provided for an exemplary embodiment of the present application;

[0027] Figure 4 A flowchart of a method for grasping provided by an exemplary embodiment of the present application;

[0028] Figure 5 A structural block diagram of a training device for a gesture generation model provided by an exemplary embodiment of the present application;

[0029] Figure 6 A structural block diagram of a gripping device provided for an exemplary embodiment of the present application;

[0030] Figure 7 A schematic structural diagram of an electronic device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0032] The grasping method of dexterous hands is limited by the model and the algorithm used by the model. The models trained by related technologies are difficult to cope with the diversity and uncertainty of objects, which makes the grasping efficiency of dexterous hands low when facing complex, unknown or irregularly shaped objects, and lacks flexibility and generalization ability. To solve this problem, related technologies use large-scale simulation data sets to train models, but it is still difficult to ensure the stability and generalization ability of the models in the real world.

[0033] Based on the above problems, the present application integrates a meta-learner to train a gesture generation model, wherein the meta-learner can evaluate the training effect of the gesture generation model, and then adjust the training strategy of the gesture generation model, thereby improving the accuracy and generalization ability of the gesture generation model in predicting gesture information.

[0034] In addition, an application scenario of the embodiment of the present application is as follows Figure 1 ,exist Figure 1 In the example, a dexterous hand and an object to be grasped are included. A trained posture generation model needs to be used in advance to generate posture information, and the dexterous hand grasps the object to be grasped according to the posture information.

[0035] in, Figure 1 This is just an exemplary application scenario, and the embodiment of the present application can be applied to any scenario where a dexterous hand grasps many objects. The embodiment of the present application does not limit the specific application scenario.

[0036] Figure 2 A flowchart of a method for training a gesture generation model provided by an exemplary embodiment of the present application specifically includes the following steps:

[0037] S201, obtaining a first training data set.

[0038] In an embodiment of the present application, the first training data set includes multiple sub-data sets, each sub-data set corresponds to a category of sample objects. The sub-data sets include: multiple first training samples and multiple second training samples, the first training samples include: first sample three-dimensional point cloud data and first sample gesture information, and the second training samples include: second sample three-dimensional point cloud data and second sample gesture information.

[0039] In an embodiment of the present application, model training data is first prepared, and the model training data is simulation data. For example, a large-scale robot dexterous hand grasping data set (UniDexGrasp) based on simulation is used as the model training data.

[0040] The model training data includes: a first training data set and a second training set, wherein the first training data set includes multiple sub-data sets, each sub-data set includes multiple training samples, and multiple training samples of the same sub-data set are sample objects from the same category. For example, if the category is water cup, then multiple training samples of water cups constitute a sub-data set.

[0041] For example, the first training data set includes: 100 sub-datasets (distributed as sub-dataset a1 to sub-dataset a100), which correspond to 100 categories, among which the category of the sample object corresponding to the sub-dataset a1 is a water cup, the category of the sample object corresponding to the sub-dataset a2 is a paper towel, and the sample object corresponding to the sub-dataset a3 is a mobile phone. The sub-datasets also have corresponding sample object categories, which are not listed one by one here.

[0042] Furthermore, each sub-dataset includes multiple training samples, and the multiple training samples can be divided into two groups, namely, multiple first training samples of the first group and multiple second training samples of the second group. For example, sub-dataset a1 includes: 1000 training samples, 400 training samples are first training samples of the first group, and 600 training samples are second training samples of the second group.

[0043] Furthermore, in the present application, each training sample (such as the first training sample or the second training sample) includes: sample three-dimensional point cloud data and sample gesture information, wherein the sample three-dimensional point cloud data is the three-dimensional point cloud data of the sample object, and the sample gesture information is the gesture information used by the dexterous hand to accurately grasp the sample object.

[0044] S202, construct a gesture generation model and a meta-learner.

[0045] The gesture generation model is used to predict the gesture information when the dexterous hand grasps an object. The meta-learner is used to adjust the model parameters of the gesture generation model during the training process.

[0046] The gesture generation model and the meta-learner may have the same model structure, and the gesture generation model may be a transformer (a deep learning model architecture) model or other models, without limitation.

[0047] Furthermore, the gesture generation model can predict gesture information based on the input data.

[0048] In the embodiment of the present application, the initial model parameters of the gesture generation model constructed can be Represented by. The initial network parameters of the constructed meta-learner can be used express.

[0049] S203: Use the first training data set and the meta-learner to train the gesture generation model to obtain a trained gesture generation model.

[0050] In an embodiment of the present application, the first training data set is used to train a gesture generation model, and the meta-learner is used to adjust the learning strategy during the training process of the gesture generation model.

[0051] Specifically, refer to Figure 3, using the first training data set and the meta-learner to train the gesture generation model, to obtain a trained gesture generation model, including: for the current sub-data set, performing the following steps

[0052] S301, training a gesture generation model according to the current sub-dataset.

[0053] Among them, training the gesture generation model according to the current sub-dataset includes: for each first training sample of the current sub-dataset, looping through the following steps: inputting the first sample three-dimensional point cloud data into the gesture generation model for processing to obtain first predicted gesture information; and updating the model parameters of the gesture generation model according to a first loss value of the first predicted gesture information relative to the first sample gesture information.

[0054] For example, for the sub-dataset ai, it includes 400 first training samples, namely the first training samples To the first training sample The first training sample (j ranges from 1 to 400) includes: the first sample 3D point cloud data and the first sample gesture information .

[0055] Then during the training process, for each first training sample of the sub-dataset ai , the first sample 3D point cloud data Input the gesture generation model for processing to obtain the first predicted gesture information , according to the first predicted gesture information Relative to the first sample gesture information The first loss value , update the model parameters of the gesture generation model.

[0056] For example, if the current sub-dataset is sub-dataset a1, then for the current sub-dataset a1, the model parameters of the gesture generation model are , first take the first training sample The first sample 3D point cloud data Input gesture generation model (model parameters are ) to obtain the first predicted gesture information , according to the first predicted gesture information Relative to the first sample gesture information The first loss value , update the model parameters of the gesture generation model to Then the first training sample The first sample 3D point cloud data Input gesture generation model (model parameters are ) to obtain the first predicted gesture information , according to the first predicted gesture information Relative to the first sample gesture information The first loss value , update the model parameters of the gesture generation model to , and the gesture generation model is trained in this cycle. After the training of each first training sample of the sub-dataset a1 is completed, the model parameters of the gesture generation model are obtained as follows: .

[0057] In an embodiment of the present application, the above training method is a case where each first training sample is trained only once. In addition, each first training sample can be trained multiple times, wherein the model parameters of the gesture generation model are iteratively updated during the training process.

[0058] Furthermore, the calculation of the first loss value may adopt a preset loss function, such as a negative log-likelihood loss function.

[0059] Among them, the log-likelihood loss function is as follows:

[0060] ;

[0061] Where: L represents the first loss value, y represents the sample gesture information, and p represents the predicted gesture information.

[0062] Furthermore, the gesture information includes at least one of posture information, position information and joint angle information of the dexterous hand, wherein the posture information is represented by rotation angle information and the position information is represented by translation distance information.

[0063] In an embodiment of the present application, the gesture generation model adopts a loss reduction training method so that under the condition of inputting the first sample three-dimensional point cloud data, the output predicted gesture information is closer to the sample gesture information.

[0064] Furthermore, after updating the model parameters of the gesture generation model, it also includes: for each second training sample, inputting the second sample three-dimensional point cloud data into the gesture generation model for processing to obtain second predicted gesture information; for each second training sample, determining a second loss value of the second predicted gesture information relative to the second sample gesture information; determining the gradient value of the current model parameters and the target learning rate of the meta-learner; and updating the network parameters of the meta-learner according to the second loss value, the gradient value and the target learning rate.

[0065] In an embodiment of the present application, the training process also includes training a meta-learner, thereby improving the accuracy of the target network parameters output by the meta-learner.

[0066] The multiple first training samples in the sub-dataset are used to train the gesture generation model, and the multiple second training samples are used to evaluate the performance of the gesture generation model and train the meta-learner.

[0067] Specifically, the calculation of the second loss value may refer to the calculation of the first loss value described above, which will not be described in detail here.

[0068] Furthermore, determining the target learning rate of the meta-learner includes: determining the prediction accuracy of the gesture generation model based on the second predicted gesture information of multiple second training samples and the second sample gesture information; obtaining the current learning rate of the meta-learner; and determining the target learning rate of the meta-learner based on the prediction accuracy and the current learning rate.

[0069] Specifically, there are n second training samples in total, corresponding to n second sample gesture information and n second predicted gesture information, wherein the prediction accuracy is determined by the following function:

[0070] ;

[0071] Among them, MSE represents the prediction accuracy, represents the i-th second sample gesture information, Represents the i-th second predicted gesture information.

[0072] In addition, the current learning rate of the meta-learner is , then the target learning rate of the meta-learner is .

[0073] In the embodiment of the present application, the learning rate of the meta-learner is updated once for each sub-data set, wherein the learning rate of the meta-learner is also updated iteratively.

[0074] For example, the second training sample is 600.

[0075] S302, determining training process data of a gesture generation model.

[0076] The training process data includes: the gradient value of the current model parameter, at least one of the performance index and the training process parameter.

[0077] It can be understood that after the gesture generation model is trained using a sub-dataset, the training process data can be determined.

[0078] Specifically, the current model parameters after the gesture generation model is trained using sub-dataset a1 are: .

[0079] In the embodiment of the present application, the training process data may also include other data in the training process, such as current model parameters, category data or sub-datasets, wherein the current model parameters include: weight parameters and / or bias parameters of the gesture generation model, and the performance indicators include loss values ​​and / or accuracy rates in the training process. The training process parameters include: number of training iterations and / or learning rate. The category data includes: description data of the category of the sub-dataset, such as a glass cup. The sub-dataset input to the meta-learner is a plurality of first training samples.

[0080] In the embodiment of the present application, the training process data may also include data in other training processes, which is not limited.

[0081] S303: Input the training process data into the meta-learner for processing to obtain the target model parameters of the gesture generation model

[0082] Among them, the meta-learner can process the training process data to obtain the target model parameters.

[0083] S304: Update the model parameters of the gesture generation model to target model parameters.

[0084] In the embodiment of the present application, the target model parameters are used to update the model parameters of the gesture generation model. For example, the model parameters of the gesture generation model after training in S301 are , the target model parameters are expressed as , then the target model parameters As model parameters of the gesture generation model.

[0085] S305: Determine whether the current training process meets the preset training requirements.

[0086] If the current training process meets the preset training requirements, the gesture generation model is determined to be trained. If the current training process does not meet the preset training requirements, the next sub-dataset is determined to be the current sub-dataset to continue the training steps.

[0087] It can be understood that the current training process includes: the number of training iterations or the prediction accuracy of the current gesture generation model, etc., and the preset training requirements are such as the number of iterations required, the prediction accuracy required for the current gesture generation model, etc. In the embodiment of the present application, the preset training requirements can be set according to the needs, and this is not limited.

[0088] Furthermore, if it is determined that the current training process meets the preset training requirements, the training is terminated and the current gesture generation model is determined to be a trained gesture generation model. Otherwise, the next sub-dataset (sub-dataset a2) is used as the current sub-dataset to execute S301 to S304 to iteratively update the model parameters of the gesture generation model.

[0089] The gesture generation model obtained by the current training is repeatedly executed from S301 to S304 using the next sub-data set to continue training the gesture generation model, and S301 to S304 are looped until a trained gesture generation model is obtained.

[0090] In an embodiment of the present application, after using a first training data set and a meta-learner to train a gesture generation model to obtain a trained gesture generation model, it also includes: obtaining a second training data set, the second training data set including: multiple third training samples of different categories; using multiple third training samples and a meta-learner to train the gesture generation model to obtain a trained gesture generation model.

[0091] In the embodiment of the present application, after the gesture generation model and the meta-learner are trained using the first training data set, the learning rate of the gesture generation model is , the learning rate of the meta-learner is , the learning rates of the gesture generation model and the meta-learner can be adjusted. For example, the learning rate of the gesture generation model is adjusted to , the learning rate of the meta-learner is Then, the second training data set is used to train the gesture generation model. The specific training process may be to use the second training data set as a sub-data set and perform training with reference to the above-mentioned methods of S301 to S304, which will not be described in detail here.

[0092] In an embodiment of the present application, the second training data set includes third training samples of each category. Using the second training data set to train the gesture generation model can further improve the generalization ability of the gesture generation model and the prediction accuracy of gesture information on objects of different categories.

[0093] In addition, after obtaining the trained gesture generation model, the embodiment of the present application can test and evaluate the gesture generation model. Among them, an independent test data set can be used to evaluate the performance of the gesture generation model. The test data set contains gesture types, scenes and conditions different from the training data set to verify the generalization ability of the gesture generation model.

[0094] In summary, the present application initializes the gesture generation model and the meta-learner, cyclically uses the meta-learner to adjust the model parameters of the gesture generation model, and trains the gesture generation model with the first training sample; the meta-learner dynamically adjusts the model parameters according to the generation effect and learning progress of the current gesture information. In addition, the generalization ability of the gesture generation model can be improved by training the gesture generation model according to sub-data sets of target objects of different categories in sequence. In addition, the generation effect of the model on the second sample data can also be evaluated. According to the generation effect, the meta-learner will further adjust the learning strategy to improve the accuracy and generalization ability of the gesture information generated by the gesture generation model.

[0095] Figure 4 A flowchart of a grasping method provided for an exemplary embodiment of the present application specifically includes the following steps:

[0096] S401, determining target point cloud data of a target object.

[0097] Among them, determining the target point cloud data of the target object includes: acquiring a color image and a depth image, both the color image and the depth image include: the target object to be captured; processing the color image through a target detection network to obtain the target pixel coordinates of the target object; and determining the target point cloud data of the target object according to the target pixel coordinates and the depth image.

[0098] It can be understood that the present application uses a visual perception module to collect color images and depth images of the capture scene where the target object is located, wherein the color image and the depth image are aligned in time and space. Furthermore, the visual perception module used in the present application uses a depth camera, and the color image and the depth image are aligned in time and space by setting the reading frequency of the depth camera and the camera intrinsic parameters of the depth camera.

[0099] Then, the color image is processed by the target detection network to obtain the target pixel coordinates and grasping force threshold of the target object. Among them, the target detection network is pre-trained and can be YOLOV8 (a target detection network) or other target detection networks, which is not limited. The target detection network can be trained using sample images, pixel coordinates of sample objects in sample images, and sample grasping force thresholds. The specific training method is not limited in this application.

[0100] Furthermore, according to the target pixel coordinates and the depth image, target point cloud data of the target object can be segmented from the depth image.

[0101] In addition, the present application may also determine the target point cloud data of the target object by other methods, which are not limited here.

[0102] S402, inputting the target point cloud data into a gesture generation model for processing to obtain gesture information.

[0103] The gesture generation model is obtained by training the above-mentioned gesture generation model training method. In the embodiment of the present application, the gesture information includes: at least one of the posture information, position information and joint angle information of the dexterous hand.

[0104] S403, controlling the dexterous hand to grasp the target object according to the gesture information.

[0105] In one embodiment, controlling the dexterous hand to grasp the target object according to gesture information includes: determining the position information of the target object according to the target point cloud data; determining the grasping path according to the gesture information and the position information of the target object; and controlling the dexterous hand to grasp the target object according to the grasping path.

[0106] Specifically, the position information of the target object can be determined based on the target point cloud data based on the camera intrinsic parameters and other parameters. After determining the position information of the target object, the distance information between the target object and the dexterous hand can be further determined based on the position information of the target object and the position information of the dexterous hand, and then the kinematic inverse solution method is used to determine the joint rotation angle of the manipulator according to the distance information and gesture information, wherein the dexterous hand is the end effector of the manipulator. It can be understood that the grasping path is represented by the joint rotation angle of the manipulator, and the manipulator rotates according to the joint rotation angle, which can drive the dexterous hand to reach the specified grasping position, and the target object can be grasped at this grasping position.

[0107] In one embodiment, after processing the color image (RGB image) using a target detection network, a grasping force threshold is obtained. After controlling the dexterous hand to grasp the target object according to the gesture information, it also includes: obtaining the real-time contact force of the dexterous hand; when the real-time contact force is greater than the maximum contact force, determining that the target object has been grasped.

[0108] In an embodiment of the present application, the dexterous hand realizes the tactile function of the dexterous hand by setting multiple tactile sensors on the fingers and palm.

[0109] Specifically, for each tactile sensor, a pressure value sequence can be collected every preset time (such as t milliseconds, t is a positive number), and then the abnormal values ​​(such as the maximum pressure value and the minimum pressure value) in the pressure value sequence can be removed by wavelet transform to obtain a new pressure value sequence. Then the new pressure value sequence is averaged to obtain the real-time contact force. When the real-time contact force is greater than the grasping force threshold, it is determined that the grasping is completed, thereby achieving the grasping of the target object. In the embodiment of the present application, the real-time contact force refers to the contact force value between the current dexterous hand and the target object.

[0110] It is understandable that there will be no tactile feedback before the dexterous hand contacts the target object, that is, the tactile sensor does not detect the pressure value. When the maximum contact force is less than the real-time grasping force, it is determined that the target object has been stably grasped and the grasping action has been completed.

[0111] Furthermore, after the dexterous hand completes grasping the target object, the dexterous hand is controlled to stop.

[0112] In the embodiments of the present application, the present application significantly improves the adaptive grasping ability of the dexterous hand by integrating the rapid adaptability, visual perception ability and tactile perception ability of the meta-learner. Specifically, firstly, the present application uses a large-scale first training data set and a second training data set to train the posture generation model, and dynamically adjusts the model parameters of the posture generation model through the meta-learner, thereby enhancing the generalization ability and learning efficiency of the posture generation model. Secondly, the present application uses a visual perception module to obtain color images and depth images, and combines the target detection network to determine the target point cloud data and grasping force threshold of the target object to be grasped, thereby improving the grasping accuracy, and reducing the grasping operation error by introducing real-time tactile feedback. Finally, the posture generation model is optimized by combining the meta-learner, visual perception ability and tactile perception ability, realizing efficient and accurate grasping of unknown objects in complex and changing environments, and improving the intelligence level and operational performance of the dexterous hand.

[0113] Reference Figure 5 , is a structural block diagram of a training device 50 for a gesture generation model provided in the present application, and the training device 50 for a gesture generation model includes:

[0114] An acquisition module 501 is used to acquire a first training data set;

[0115] A construction module 502 is used to construct a gesture generation model and a meta-learner, wherein the gesture generation model is used to predict gesture information when the dexterous hand grasps an object, and the meta-learner is used to adjust model parameters of the gesture generation model during training;

[0116] The training module 503 is used to train the gesture generation model using the first training data set and the meta-learner to obtain a trained gesture generation model. The meta-learner is used to adjust the model parameters of the gesture generation model during the training process.

[0117] In an optional embodiment, the first training data set includes a plurality of sub-data sets, each sub-data set corresponds to a category of sample objects, and the training module 503 is specifically used to:

[0118] For the current sub-dataset, perform the following training steps:

[0119] Train a gesture generation model based on the current sub-dataset;

[0120] Determine training process data of the gesture generation model, the training process data including: a gradient value of a current model parameter, a performance indicator, and at least one of a training process parameter;

[0121] The training process data is input into the meta-learner for processing to obtain the target model parameters of the gesture generation model;

[0122] Adjusting the model parameters of the gesture generation model to the target model parameters;

[0123] If the current training process meets the preset training requirements, it is determined that a trained gesture generation model is obtained; if the current training process does not meet the preset training requirements, the next sub-dataset is determined to be the current sub-dataset to continue the training steps.

[0124] In an optional embodiment, the current sub-dataset includes: a plurality of first training samples, the first training sample includes: first sample three-dimensional point cloud data and first sample gesture information, and the training module 503 is specifically used to:

[0125] For each first training sample, perform the following steps:

[0126] Inputting the first sample three-dimensional point cloud data into the gesture generation model for processing to obtain first predicted gesture information;

[0127] The model parameters of the gesture generation model are updated according to a first loss value of the first predicted gesture information relative to the first sample gesture information.

[0128] In an optional embodiment, the current sub-dataset further includes: a plurality of second training samples, the second training samples including: second sample three-dimensional point cloud data and second sample gesture information, and the training module 503 is further used to update the model parameters of the gesture generation model after:

[0129] For each second training sample, inputting the second sample three-dimensional point cloud data into the gesture generation model for processing to obtain second predicted gesture information;

[0130] For each second training sample, determining a second loss value of the second predicted gesture information relative to the second sample gesture information;

[0131] Determine the gradient value of the current model parameters and the target learning rate of the meta-learner;

[0132] Update the network parameters of the meta-learner according to the second loss value, the gradient value and the target learning rate.

[0133] In an optional embodiment, when determining the target learning rate of the meta-learner, the training module 503 is specifically configured to:

[0134] Determining a prediction accuracy rate of a gesture generation model according to second predicted gesture information of a plurality of second training samples and second sample gesture information;

[0135] Get the current learning rate of the meta-learner;

[0136] Determine the target learning rate of the meta-learner based on the prediction accuracy and the current learning rate.

[0137] In an optional embodiment, the acquisition module 501 is further configured to, after using the first training data set and the meta-learner to train the gesture generation model and obtaining the trained gesture generation model,

[0138] Acquire a second training data set, where the second training data set includes: a plurality of third training samples of different categories;

[0139] The training module 503 is further used to train the gesture generation model using multiple third training samples and a meta-learner to obtain a trained gesture generation model.

[0140] The training device for the gesture generation model provided in the present application can implement the training method for the gesture generation model mentioned above. Please refer to the above for details and will not repeat them here.

[0141] Reference Figure 6 , is a structural block diagram of a grasping device 60 provided in the present application, and the grasping device 60 includes:

[0142] A determination module 601 is used to determine target point cloud data of a target object;

[0143] Processing module 602, used for inputting the target point cloud data into a gesture generation model for processing to obtain gesture information, wherein the gesture generation model is trained by any of the training methods for the gesture generation model described above;

[0144] The control module 603 is used to control the dexterous hand to grasp the target object according to the gesture information.

[0145] In an optional embodiment, the determination module 601 is specifically used to obtain a color image and a depth image, and the color image and the depth image both include: a target object to be captured;

[0146] The color image is processed through the target detection network to obtain the target pixel coordinates of the target object;

[0147] The target point cloud data of the target object is determined according to the target pixel coordinates and the depth image.

[0148] In an optional embodiment, the control module 603, for controlling the dexterous hand to grasp the target object according to the gesture information, includes:

[0149] Determine the location information of the target object according to the target point cloud data;

[0150] Determine the grasping path based on the gesture information and the location information of the target object;

[0151] The dexterous hand is controlled to grasp the target object according to the grasping path.

[0152] In an optional embodiment, after the color image is processed by the target detection network, a grasping force threshold is also obtained, and an acquisition module (not shown) is further included, which is used to acquire the real-time contact force of the dexterous hand after the dexterous hand grasps the target object according to the gesture information;

[0153] The determination module 602 is further configured to determine that the target object is grasped successfully when the real-time contact force is greater than the maximum contact force.

[0154] The grasping device provided in the present application can implement the above-mentioned grasping method. Please refer to the above for details and will not be repeated here.

[0155] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and are only used to distinguish between different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "second", "first", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "second" and "first" to be different types.

[0156] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. Figure 7 As shown, the electronic device 70 includes: a processor 71, and a memory 72 communicatively connected to the processor 71, and the memory 72 stores computer-executable instructions.

[0157] Among them, the processor executes the computer execution instructions stored in the memory to implement the training method and / or grasping method of the gesture generation model provided by any of the above method embodiments, and the specific functions and technical effects that can be achieved are not repeated here.

[0158] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the training method and / or grasping method of any of the above-mentioned gesture generation models.

[0159] An embodiment of the present application also provides a computer program product, which includes: a computer program, which is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes any of the above-mentioned training methods and / or grasping methods of the gesture generation model.

[0160] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0161] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0163] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0164] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0165] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0166] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A training method for a gesture generation model, characterized in that: include: Acquire a first training data set, wherein the first training data set includes a plurality of sub-data sets, each sub-data set corresponds to a category of sample objects, and the category is a category of sample objects grasped by a dexterous hand; Constructing a gesture generation model and a meta-learner, wherein the gesture generation model is used to predict gesture information when a dexterous hand grasps an object, and the meta-learner is used to adjust model parameters of the gesture generation model during training; Using the first training data set and the meta-learner to train the gesture generation model to obtain a trained gesture generation model; The step of using the first training data set and the meta-learner to train the gesture generation model to obtain a trained gesture generation model includes: For the current sub-dataset, perform the following training steps: Training the gesture generation model according to the current sub-dataset, the current sub-dataset comprising: a plurality of first training samples, the first training samples comprising: first sample three-dimensional point cloud data and first sample gesture information, the first sample gesture information being gesture information used by a dexterous hand to accurately grasp a sample object; Determine training process data of the gesture generation model, wherein the training process data includes: a gradient value of a current model parameter, a performance indicator, and at least one of a training process parameter; Inputting the training process data into the meta-learner for processing to obtain target model parameters of the gesture generation model; Adjusting the model parameters of the gesture generation model to be the target model parameters; If the current training process meets the preset training requirements, it is determined that a trained gesture generation model is obtained; if the current training process does not meet the preset training requirements, the next sub-data set is determined to be the current sub-data set to continue executing the training steps.

2. The training method of the gesture generation model according to claim 1, characterized in that: The step of training the gesture generation model according to the current sub-dataset includes: For each first training sample, perform the following steps: Inputting the first sample three-dimensional point cloud data into the gesture generation model for processing to obtain first predicted gesture information; The model parameters of the gesture generation model are updated according to a first loss value of the first predicted gesture information relative to the first sample gesture information.

3. The training method for the gesture generation model according to claim 2, characterized in that: The current sub-dataset further includes: a plurality of second training samples, wherein the second training samples include: second sample three-dimensional point cloud data and second sample gesture information, and after the updating of the model parameters of the gesture generation model, further includes: For each second training sample, inputting the second sample three-dimensional point cloud data into the gesture generation model for processing to obtain second predicted gesture information; For each second training sample, determining a second loss value of the second predicted gesture information relative to the second sample gesture information; Determining the gradient value of the current model parameter and the target learning rate of the meta-learner; Update the network parameters of the meta-learner according to the second loss value, the gradient value and the target learning rate; Determining the target learning rate of the meta-learner includes: Determining a prediction accuracy rate of the gesture generation model according to second predicted gesture information of a plurality of second training samples and the second sample gesture information; Obtaining a current learning rate of the meta-learner; A target learning rate of the meta-learner is determined according to the prediction accuracy and the current learning rate.

4. The training method for a gesture generation model according to any one of claims 1 to 3, characterized in that: After the gesture generation model is trained using the first training data set and the meta-learner to obtain the trained gesture generation model, the method further includes: Acquire a second training data set, where the second training data set includes: a plurality of third training samples of different categories; The gesture generation model is trained using the multiple third training samples and the meta-learner to obtain a trained gesture generation model.

5. A grasping method, characterized in that: include: Determine target point cloud data of the target object; Inputting the target point cloud data into a gesture generation model for processing to obtain gesture information, wherein the gesture generation model is trained by the gesture generation model training method according to any one of claims 1 to 4; The dexterous hand is controlled to grasp the target object according to the gesture information.

6. The grasping method according to claim 5, characterized in that: The step of determining target point cloud data of the target object comprises: Acquire a color image and a depth image, wherein the color image and the depth image both include: a target object to be captured; Processing the color image through a target detection network to obtain target pixel coordinates of a target object; Determine target point cloud data of a target object according to the target pixel coordinates and the depth image; Controlling the dexterous hand to grasp the target object according to the gesture information comprises: Determine the location information of the target object according to the target point cloud data; Determining a grasping path according to the gesture information and the position information of the target object; Controlling the dexterous hand to grasp the target object according to the grasping path; After the color image is processed by the target detection network, a grasping force threshold is obtained, and after the dexterous hand is controlled to grasp the target object according to the gesture information, the method further includes: acquiring the real-time contact force of the dexterous hand; When the real-time contact force is greater than the maximum contact force, it is determined that the target object is grasped completely.

7. A training device for a gesture generation model, characterized in that: include: An acquisition module, configured to acquire a first training data set, wherein the first training data set includes a plurality of sub-data sets, each sub-data set corresponds to a category of sample objects, and the category is a category of sample objects grasped by a dexterous hand; A construction module, used to construct a gesture generation model and a meta-learner, wherein the gesture generation model is used to predict gesture information when a dexterous hand grasps an object, and the meta-learner is used to adjust model parameters of the gesture generation model during training; a training module, configured to train the gesture generation model using the first training data set and the meta-learner to obtain a trained gesture generation model, wherein the meta-learner is configured to adjust model parameters of the gesture generation model during the training process; The first training data set includes a plurality of sub-data sets, each sub-data set corresponds to a category of sample objects, and the training module is specifically used for: For the current sub-dataset, perform the following training steps: Training the gesture generation model according to the current sub-dataset, the current sub-dataset comprising: a plurality of first training samples, the first training samples comprising: first sample three-dimensional point cloud data and first sample gesture information, the sample gesture information being gesture information used by a dexterous hand to accurately grasp a sample object; Determine training process data of the gesture generation model, wherein the training process data includes: a gradient value of a current model parameter, a performance indicator, and at least one of a training process parameter; Inputting the training process data into the meta-learner for processing to obtain target model parameters of the gesture generation model; Adjusting the model parameters of the gesture generation model to be the target model parameters; If the current training process meets the preset training requirements, it is determined that a trained gesture generation model is obtained; if the current training process does not meet the preset training requirements, the next sub-data set is determined to be the current sub-data set to continue executing the training steps.

8. A gripping device, characterized in that: include: A determination module, used for determining target point cloud data of a target object; a processing module, used for inputting the target point cloud data into a gesture generation model for processing to obtain gesture information, wherein the gesture generation model is trained by the training method for the gesture generation model according to any one of claims 1 to 4; A control module is used to control the dexterous hand to grasp the target object according to the gesture information.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

Images