Adaptive grasping method for dexterous hand, dexterous hand control system and storage medium

Abstract

The application relates to a dexterous hand self-adaptive grasping method, a dexterous hand control system and a storage medium, the dexterous hand self-adaptive grasping method comprising the following steps: acquiring multi-modal perception data; after processing point cloud data to generate a point cloud global feature vector corresponding to a grasping target, mapping tactile patch information to a dexterous hand workspace to generate a simulated tactile signal; processing the point cloud global feature vector and the simulated tactile signal by using a Glow-CVAE conditional generation model to generate a candidate grasping gesture set comprising joint posture and pose parameters; determining the stability corresponding to each candidate grasping gesture, selecting a target grasping gesture from the candidate grasping gesture set according to the stability, and grasping the grasping target by using the target grasping gesture. Through the application, the problem that the adaptability to special-shaped parts and flexible objects is limited and the grasping stability is not high in the related art is solved, and the beneficial effects of high adaptability and stability of grasping diversified articles in a complex scene are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of embodied intelligence and intelligent robot control, in particular to a dexterous hand self-adaptive grasping method, a dexterous hand control system and a storage medium. BACKGROUND

[0002] Current logistics industry sorting operation still mainly relies on manual operation and traditional mechanical devices, and the automation level has significant room for improvement. Although leading enterprises at home and abroad have carried out automation exploration, they still face technical challenges brought by the diversity of goods. Industry practice shows that the sorting efficiency of a robot system equipped with self-adaptive grippers is more than 40% higher than that of traditional schemes. "Quality improvement and efficiency increase" and "flexible adaptation" have become the core technical demands, driving the engineering exploration of multi-degree-of-freedom dexterous hand technology. In view of the above-mentioned needs, significant progress has been made in the research on grasping strategies of dexterous hands. Dexterous hand is defined as a multi-finger multi-degree-of-freedom manipulator (usually 3-5 finger structure, total degree of freedom ≥ 20), which can adapt to different shaped objects through various modalities such as wrapping and pinching. Its high degree of freedom provides a hardware basis for grasping of irregular objects, but also brings computational complexity of control and planning.

[0003] In recent years, there are many mainstream technical methods around grasping strategy generation: grasping pose estimation method based on RGB vision guidance uses RGB image semantic analysis to estimate the graspable area and attitude. The core technologies include: deep learning framework directly regresses grasping pose parameters and quality score through CNN; unsupervised domain adaptive learning (k-means clustering divides the foreground object); multi-view information fusion improves the robustness of the occlusion scene; the advantages are low perception cost and fast reasoning speed, and the defects are lack of depth information leading to three-dimensional positioning error. Grasping strategy synthesis method based on three-dimensional point cloud data determines the stable contact position and 6-degree-of-freedom pose through point cloud geometric analysis; it has three-dimensional shape perception ability and performs better than planar vision in multi-angle grasping and obstacle avoidance scene, but has high computational complexity, is sensitive to sensor accuracy, and needs to combine force control to deal with flexible objects. The development of 6D grasping algorithm and benchmark dataset continues to promote the progress in this field. The imitation learning method based on human demonstration adopts the "demonstration-learning" paradigm, which obtains grasping strategy through human demonstration, and the implementation path includes: intuitive demonstration and action trajectory recording; the core advantage is high sample efficiency, which trains the strategy model through supervised learning methods such as behavior cloning to reduce trial-and-error cost; but it faces the problem of "covariate shift", which leads to unstable performance in real environment. The strategy learning method based on human operation trajectory extracts general strategies from large-scale human operation trajectory data, and the data sources include human hand operation trajectory library and teleoperation robot trajectory data; not only the data collection and labeling cost is high, but also there is a problem of insufficient multi-modal information fusion capability, which makes it difficult to cope with complex scenes.

[0004] In summary, for the increasingly diverse complex application scenario requirements, the existing dexterous hand grasping strategy technology has limited adaptability to special-shaped parts and flexible objects, and the grasping stability is not high, so there is an urgent need for an adaptive flexible grasping technology that can achieve robust grasping of diversified objects. SUMMARY

[0005] Embodiments of the present application provide a dexterous hand adaptive grasping method, a dexterous hand control system and a storage medium to at least solve the problem of limited adaptability to special-shaped parts and flexible objects and low grasping stability in related technologies.

[0006] In a first aspect, embodiments of the present application provide a dexterous hand adaptive grasping method, comprising: acquiring multi-modal perception data, wherein the multi-modal perception data includes point cloud data corresponding to a grasping target acquired from an RGB-D image corresponding to a grasping action, and tactile patch information of the dexterous hand to the grasping target; after processing the point cloud data to generate a point cloud global feature vector corresponding to the grasping target, mapping the tactile patch information to the dexterous hand workspace to generate a simulated tactile signal; using a Glow-CVAE conditional generation model to process the point cloud global feature vector and the simulated tactile signal to generate a candidate grasping gesture set including joint posture and pose parameters; wherein each candidate grasping gesture of the candidate grasping gesture set corresponds to one joint posture and one pose parameter, and the Glow-CVAE conditional generation model is generated according to the point cloud data of a preset training template image and the real grasping gesture information corresponding to the preset training template image; determining the stability of each candidate grasping gesture, and selecting a target grasping gesture from all candidate grasping gestures of the candidate grasping gesture set according to the stability, and grasping the grasping target with the target grasping gesture, wherein the stability is determined according to the contact matching degree, surface envelope degree, grasping penetration and self-collision degree of the dexterous hand and the grasping target.

[0007] In a second aspect, embodiments of the present application provide a dexterous hand control system, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the dexterous hand adaptive grasping method as described in the first aspect when executing the computer program.

[0008] In a third aspect, embodiments of the present application provide a storage medium having a computer program stored thereon, which is executed by a processor to implement the dexterous hand adaptive grasping method as described in the first aspect.

[0009] Compared with the related art, the dexterous hand adaptive grasping method, the dexterous hand control system and the storage medium provided by the embodiments of the present application obtain multi-modal perception data, wherein the multi-modal perception data includes point cloud data corresponding to a grasping target obtained from an RGB-D image corresponding to a grasping action, and tactile patch information of the dexterous hand to the grasping target; after processing the point cloud data to generate a point cloud global feature vector corresponding to the grasping target, the tactile patch information is mapped to a dexterous hand workspace to generate a simulated tactile signal; a Glow-CVAE conditional generation model is used to process the point cloud global feature vector and the simulated tactile signal to generate a candidate grasping gesture set including joint posture and pose parameters; wherein each candidate grasping gesture of the candidate grasping gesture set corresponds to one joint posture and one pose parameter, and the Glow-CVAE conditional generation model is generated according to the point cloud data of a preset training template image and the real grasping gesture information corresponding to the preset training template image; the stability of each candidate grasping gesture is determined, and a target grasping gesture is selected from all candidate grasping gestures in the candidate grasping gesture set according to the stability, and the target grasping gesture is used to grasp the grasping target, wherein the stability is determined according to the contact matching degree, surface envelope degree, grasping penetration and self-collision degree of the dexterous hand and the grasping target; the problem of limited adaptability to special-shaped parts and flexible objects and low grasping stability in the related art is solved, and the beneficial effects of high adaptability and stability of grasping diversified objects in complex scenes are achieved.

[0010] The details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0011] The drawings described herein are intended to provide further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0012] Figure 1 is a hardware structure block diagram of a terminal according to the dexterous hand adaptive grasping method of the embodiments of the present application;

[0013] Figure 2 is a flowchart of the dexterous hand adaptive grasping method according to the embodiments of the present application;

[0014] Figure 3 is a construction flowchart of the Glow-CVAE conditional generation model according to the embodiments of the present application;

[0015] Figure 4 is a flowchart of the dexterous hand adaptive grasping method according to the preferred embodiments of the present application;

[0016] Figure 5 is a structural block diagram of a dexterous hand control system according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be described and illustrated below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the scope of the present application. In addition, it can be understood that, although the efforts made in this development process can be complex and lengthy, some changes in design, manufacture or production and the like made by those of ordinary skill in the art related to the content disclosed in the present application on the basis of the technical content disclosed in the present application are only routine technical means and should not be understood as insufficient disclosure of the content disclosed in the present application.

[0018] In the present application, the phrase "embodiments" means that the specific features, structures or characteristics described in conjunction with the embodiments can be included in at least one embodiment of the present application. The appearance of this phrase at various places in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0019] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terms "a", "an", "one", "this", and similar referents in the context of describing the application are to be construed to be inclusive, not exclusive. The terms "comprise", "comprising", "comprises", "include", "including", "includes", "contain", "containing", "contains", and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product, or apparatus that comprises a list of steps or units (elements) is not necessarily limited to the listed steps or units, but can include additional steps or units not expressly listed or inherent to such process, method, product, or apparatus. The terms "coupled", "connected", "coupling", and similar referents in the context of this application are to be construed as not necessarily implying a direct or indirect connection between two or more elements, but can include an electrical connection, whether direct or indirect. The term "plurality" means two or more. The term "and / or" describes association between name objects, and means that there can be three types of association: single association, joint association, and separate association. The terms "first", "second", "third", and the like, merely identify names of similar objects, and do not necessarily indicate a specific order or sequence.

[0020] The method embodiments provided by the present embodiment can be executed in a terminal, a computer, or a similar computing device. Taking the case of running on a terminal, Figure 1 is a hardware structure diagram of a terminal of the dexterous hand self-adaptive grasping method of the present embodiment. As shown in Figure 1 , the terminal 10 can include one or more (only one is shown in Figure 1 ) processor 102 (the processor 102 can include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the above terminal can also include a transmission device 106 for communication function and an input and output device 108. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, and it does not limit the structure of the above terminal. For example, the terminal 10 can include more or fewer components than those shown in Figure 1 , or have a different configuration from that shown in Figure 1 .

[0021] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as a computer program corresponding to the adaptive grasping method of the dexterous hand in the embodiments of the present application. The processor 102 can execute various functional applications and data processing, i.e., implement the above method, by running the computer program stored in the memory 104. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, which can be connected to the terminal 10 through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0022] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network can include a wireless network provided by a communication provider of the terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC) which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module which is used to communicate with the Internet in a wireless manner.

[0023] The embodiments provide an adaptive grasping method of a dexterous hand running on the above terminal, Figure 2 is a flowchart of the adaptive grasping method of the dexterous hand according to the embodiments of the present application, as Figure 2 shown, the flow includes the following steps:

[0024] In step S201, multi-modal perception data is acquired, wherein the multi-modal perception data includes point cloud data corresponding to a grasping target acquired from an RGB-D image corresponding to a grasping action, and tactile patch information of the dexterous hand to the grasping target.

[0025] In the embodiments, by acquiring multi-modal perception data, including point cloud data collected in the RGB-D image, tactile patch information (including force distribution of the tactile patch) of the dexterous hand to the grasping target, etc., fusion of tactile and visual is achieved, and the accuracy and stability of grasping are greatly improved. Single modal perception data is difficult to cope with complex application scenarios, and multi-technology fusion is needed to realize complementary advantages, and multi-modal fusion realizes dual capabilities of visual geometric cognition and tactile physical cognition, which not only solves the problem of poor adaptability of the dexterous hand to special-shaped parts and soft packaging, but also guarantees the stability of grasping.

[0026] In step S202, after processing the point cloud data to generate the point cloud global feature vector corresponding to the grasping target, the tactile patch information is mapped to the dexterous hand workspace to generate a simulated tactile signal.

[0027] In this embodiment, by processing the point cloud data to generate the point cloud global feature vector corresponding to the grasping target, and mapping the tactile patch information to the dexterous hand workspace to generate a simulated tactile signal, the differences in size and initial pose of different objects are eliminated, the computational load and learning difficulty are reduced, and efficient grasping of the dexterous hand is achieved.

[0028] In step S203, the point cloud global feature vector and the simulated tactile signal are processed by using a Glow-CVAE conditional generation model to generate a candidate grasping gesture set including joint poses and pose parameters; each candidate grasping gesture in the candidate grasping gesture set corresponds to a joint pose and a pose parameter, and the Glow-CVAE conditional generation model is generated according to the point cloud data of the preset training template image and the real grasping gesture information corresponding to the preset training template image.

[0029] In this embodiment, by using the Glow-CVAE conditional generation model to process the point cloud global feature vector and the simulated tactile signal to generate a candidate grasping gesture set including joint poses and pose parameters, the adaptability of the dexterous hand grasping can be higher; since the Glow-CVAE conditional generation model fuses the object features of the grasping target and the pose features of the dexterous hand, the fusion of tactile and visual is achieved, greatly improving the adaptability of grasping. The conditional autoencoder CVAE is used to introduce a condition variable, which not only realizes more accurate data generation, but also enhances the robustness and stability of the model by optimizing the matching of the latent distribution and the prior distribution.

[0030] In step S204, the stability of each candidate grasping gesture is determined, and a target grasping gesture is selected from all candidate grasping gestures in the candidate grasping gesture set according to the stability, and the target grasping gesture is used to grasp the grasping target, wherein the stability is determined according to the contact matching degree, surface envelope degree, grasping penetration and self-collision degree of the dexterous hand and the grasping target.

[0031] In this embodiment, by determining the stability of each candidate grasping gesture, and selecting a target grasping gesture from all candidate grasping gestures in the candidate grasping gesture set according to the stability, and grasping the grasping target with the target grasping gesture, the stability of the candidate grasping gesture can be evaluated comprehensively according to the contact matching degree, surface envelope degree, grasping penetration and self-collision degree, so that the dexterous hand grasping target object is not easy to slip and fall when disturbed, the damage rate of grasping is reduced, and the stability of grasping is ensured.

[0032] Through the steps S201 to S204, the multi-modal perception data is obtained, wherein the multi-modal perception data includes: point cloud data corresponding to the grasping target obtained from the RGB-D image corresponding to the grasping action, and the tactile patch information of the dexterous hand to the grasping target; after processing the point cloud data to generate the point cloud global feature vector corresponding to the grasping target, the tactile patch information is mapped to the dexterous hand workspace to generate the simulated tactile signal; the Glow-CVAE conditional generation model is used to process the point cloud global feature vector and the simulated tactile signal to generate a candidate grasping gesture set including joint posture and pose parameters; wherein each candidate grasping gesture in the candidate grasping gesture set corresponds to a joint posture and a pose parameter, and the Glow-CVAE conditional generation model is generated according to the point cloud data of the preset training template image and the real grasping gesture information corresponding to the preset training template image; the stability of each candidate grasping gesture is determined, and the target grasping gesture is selected from all candidate grasping gestures in the candidate grasping gesture set according to the stability, and the target grasping gesture is used to grasp the grasping target, wherein the stability is determined according to the contact matching degree, surface envelope degree, grasping penetration and self-collision degree of the dexterous hand and the grasping target; the problem of limited adaptability to special-shaped parts and flexible objects and low grasping stability in the related art is solved, and the beneficial effects of high adaptability and stability of diversified objects grasping in complex scenes are achieved.

[0033] It should be noted that in the present embodiment, the stability is determined according to the contact matching degree, surface envelope degree, grasping penetration and self-collision degree of the dexterous hand and the grasping target; the optimization target can also add other stability constraint terms, such as: adding a force closure index and an attraction term to encourage the fingertips of the dexterous hand to approach the surface of the grasping target, adding a joint limit regularization term to avoid the finger joints exceeding a reasonable range, etc. Not only does it eliminate more detailed interference and improve the authenticity and practicality of grasping, but also the ContactNet algorithm has high computational efficiency and high adaptability to complex scenes, achieving fast and robust grasping.

[0034] In some embodiments, the Glow-CVAE conditional generation model is used to process the point cloud global feature vector and the simulated tactile signal to generate a candidate grasping gesture set including joint posture and pose parameters, which is achieved by the following steps:

[0035] Step 21, using the CVAE generation network to perform vector splicing on the point cloud global feature vector and the simulated tactile signal to generate a first joint feature.

[0036] Step 22, using the variational autoencoder of the Glow-CVAE conditional generation model to infer the first joint feature to generate a posterior distribution parameter, wherein the posterior distribution parameter includes a distribution parameter mean vector and a distribution parameter variance vector.

[0037] Step 23, reparameterization sampling is performed on the distribution parameter mean vector and the distribution parameter variance vector to generate the latent variable.

[0038] Step 24, after splicing the latent variable and the point cloud global feature vector to generate the joint latent feature vector, the joint latent feature vector is decoded by using the decoder of the Glow-CVAE conditional generation model to generate the candidate grasp gesture set.

[0039] By using the CVAE generation network in the above steps, the point cloud global feature vector and the simulated haptic signal are vector-spliced to generate a first joint feature; the variational autoencoder of the Glow-CVAE conditional generation model is used to infer the posterior distribution parameter, wherein the posterior distribution parameter includes a distribution parameter mean vector and a distribution parameter variance vector; reparameterization sampling is performed on the distribution parameter mean vector and the distribution parameter variance vector to generate the latent variable; after splicing the latent variable and the point cloud global feature vector to generate the joint latent feature vector, the joint latent feature vector is decoded by using the decoder of the Glow-CVAE conditional generation model to generate the candidate grasp gesture set, which can make the dexterous hand grasping more adaptive, and the conditional variational autoencoder CVAE introduces a conditional variable, which not only realizes more accurate data generation, but also enhances the robustness of the model and improves the training stability by optimizing the matching of the latent distribution and the prior distribution.

[0040] In some embodiments, the CVAE generation network is used to vector-splice the point cloud global feature vector and the simulated haptic signal to generate a first joint feature, which is achieved by the following steps:

[0041] Step 31, the point cloud global feature vector is processed into an object feature vector corresponding to the grasping target by using the PointNet network, and the current pose feature vector corresponding to the dexterous hand is extracted from the simulated haptic signal by using the PointNet network.

[0042] Step 32, the object feature vector and the current pose feature vector are concatenated to generate a multi-dimensional vector, and the joint distribution information is extracted from the multi-dimensional vector by using the target fully connected layer of the CVAE generation network to obtain the first joint feature.

[0043] The point cloud global feature vector is processed into the object feature vector corresponding to the grasping target by using the PointNet network in the above steps, and the PointNet network is used to extract the current posture feature vector corresponding to the dexterous hand from the simulated tactile signal; the object feature vector and the current posture feature vector are concatenated to generate a multi-dimensional vector, and the joint distribution information is extracted from the multi-dimensional vector through the target full connection layer of the CVAE generation network to obtain the first joint feature; the object feature of the grasping target and the posture feature of the dexterous hand are fused, the fusion of tactile and visual is realized, and the adaptability of grasping is improved, and the grasping efficiency is also improved.

[0044] In some embodiments, the point cloud data is processed to generate a point cloud global feature vector corresponding to the grasping target, including the following steps:

[0045] Step 41, after the coordinate normalization processing of the point cloud data to generate normalized point cloud data, the normalized point cloud data is normalized to generate normalized point cloud coordinates located in a preset unit sphere, wherein one normalized point cloud coordinate represents a three-dimensional coordinate corresponding to one point cloud data, and the normalization processing includes the following processing: translation alignment, coordinate scale standardization.

[0046] Step 42, using a multi-layer perceptron corresponding to the PointNet network, all normalized point cloud coordinates are feature extracted to obtain a corresponding point cloud global feature vector.

[0047] After the coordinate normalization processing of the point cloud data to generate normalized point cloud data, the normalized point cloud data is normalized to generate normalized point cloud coordinates located in a preset unit sphere in the above steps, wherein one normalized point cloud coordinate represents a three-dimensional coordinate corresponding to one point cloud data, and the normalization processing includes the following processing: translation alignment, coordinate scale standardization; using a multi-layer perceptron corresponding to the PointNet network, all normalized point cloud coordinates are feature extracted to obtain a corresponding point cloud global feature vector, the normalization processing eliminates the differences of different object sizes and initial postures, reduces the learning difficulty, and the PointNet network is simple and efficient, which improves the grasping efficiency.

[0048] In some embodiments, the tactile patch information is mapped to the dexterous hand workspace to generate a simulated tactile signal, which is achieved by the following steps: based on a Gaussian kernel function, the tactile patch information is mapped to the dexterous hand workspace to obtain a simulated tactile signal.

[0049] In this embodiment, the Gaussian kernel function has strong non-linear fitting ability and can process complex data distribution, so that the grasping adaptability is higher.

[0050] In some embodiments, a stability corresponding to each candidate grasping gesture is determined, and a target grasping gesture is selected from all candidate grasping gestures in the candidate grasping gesture set according to the stability, including the following steps:

[0051] Step 51, based on the improved ContactNet network constructed according to the preset target function, each candidate grasping gesture is evaluated to obtain a plurality of grasping loss parameters corresponding to each candidate grasping gesture, wherein the grasping loss parameters at least include one of the following: contact matching error parameter, surface envelope degree penalty parameter, grasping penetration loss parameter, and self-collision loss, and the target function is determined based on the sub-functions corresponding to the plurality of grasping loss parameters.

[0052] Step 52, all grasping loss parameters corresponding to each candidate grasping gesture are sequentially normalized and weighted to obtain a total grasping loss value corresponding to each candidate grasping gesture, wherein the stability includes the total grasping loss value.

[0053] Step 53, the candidate grasping gesture with the minimum total grasping loss value is selected from all candidate grasping gestures in the candidate grasping gesture set to obtain the target grasping gesture.

[0054] Through the above steps, each candidate grasping gesture is evaluated based on the improved ContactNet network constructed according to the preset target function to obtain a plurality of grasping loss parameters corresponding to each candidate grasping gesture, wherein the grasping loss parameters at least include one of the following: contact matching error parameter, surface envelope degree penalty parameter, grasping penetration loss parameter, and self-collision loss, and the target function is determined based on the sub-functions corresponding to the plurality of grasping loss parameters; all grasping loss parameters corresponding to each candidate grasping gesture are sequentially normalized and weighted to obtain a total grasping loss value corresponding to each candidate grasping gesture, wherein the stability includes the total grasping loss value; the candidate grasping gesture with the minimum total grasping loss value is selected from all candidate grasping gestures in the candidate grasping gesture set to obtain the target grasping gesture; wherein the contact matching error parameter is used to represent the energy loss of the contact matching degree between the dexterous hand and the grasping target, the surface envelope degree penalty parameter is used to represent the energy loss of the surface envelope degree of the dexterous hand to the grasping target, the grasping penetration loss parameter is used to represent the degree of grasping penetration of the dexterous hand relative to the grasping target, and the self-collision loss is used to represent the degree of self-collision of the dexterous hand; the stability evaluation of the candidate grasping gesture is comprehensively evaluated by the contact matching degree, the surface envelope degree, the grasping penetration and the self-collision degree, so that the dexterous hand grasping the target object is not easy to slip and fall when disturbed, the grasping damage rate is reduced, the grasping stability is improved, and high adaptability of grasping diversified objects such as special-shaped parts and flexible objects in complex scenes is realized.

[0055] In some embodiments, each candidate grasping gesture is evaluated using the improved ContactNet network, by the following steps:

[0056] Step 61, based on the point cloud data corresponding to the grasping target, the improved ContactNet network is used to predict the expected contact distribution probability corresponding to each point in the point cloud data.

[0057] Step 62, according to the point cloud data and the candidate grasping gesture, the current contact distribution probability corresponding to each point in the point cloud data is calculated.

[0058] Step 63, based on the expected contact distribution probability and the current contact distribution probability corresponding to all points in the point cloud data, the mean square error is calculated to obtain the contact matching error corresponding to each candidate grasping gesture, wherein the contact matching error parameter includes the contact matching error.

[0059] In this embodiment, the PointNet processing point cloud calculation is simple and efficient, and the grasping gesture generated by the ContactNet network has higher physical consistency and naturalness in the real scene, and the grasping action has high stability.

[0060] In some embodiments, each candidate grasping gesture is evaluated using the improved ContactNet network, including the following steps:

[0061] Step 71, using the improved ContactNet network, the optimized grasping gesture corresponding to each candidate grasping gesture is predicted.

[0062] Step 72, the signed distance of each point of the point cloud data and the corresponding optimized grasping gesture is calculated, and based on the signed distance corresponding to all points of the point cloud data, the signed distance field of the optimized grasping gesture and the grasping target is determined, to obtain the grasping penetration loss parameter corresponding to the corresponding candidate grasping gesture.

[0063] By the above steps of using the improved ContactNet network to predict the optimized grasping gesture corresponding to each candidate grasping gesture, calculating the signed distance of each point of the point cloud data and the corresponding optimized grasping gesture, and based on the signed distance corresponding to all points of the point cloud data, determining the signed distance field of the optimized grasping gesture and the grasping target, and obtaining the grasping penetration loss parameter corresponding to the corresponding candidate grasping gesture, the grasping gesture is generated simply and efficiently, and the grasping gesture is closer to the real natural, and the grasping action has high stability.

[0064] In some embodiments, each candidate grasping gesture is evaluated using the improved ContactNet network, and further includes the following steps:

[0065] Step 81, using the improved ContactNet network, predicting a predicted grasping gesture corresponding to each candidate grasping gesture.

[0066] Step 82, selecting a plurality of target joint link surfaces from the predicted grasping gesture, and calculating the pitch field of two target joint link surfaces in the plurality of target joint link surfaces, wherein the target joint link surface at least includes one of the following: a finger link surface, a palm component link surface.

[0067] Step 83, selecting the pitch field with the smallest pitch field distance from all pitch fields to obtain the self-collision loss corresponding to the candidate grasping gesture.

[0068] The construction and training method of the neural network model (Glow-CVAE conditional generation model) will be introduced below, Figure 3 The construction flowchart of the Glow-CVAE conditional generation model according to the embodiments of the present application is shown in Figure 3 , which includes the following steps:

[0069] Step S301, obtaining the point cloud data of the preset training template image. Generally, the surface point set is obtained from the RGB-D depth camera or sampled in the simulation environment, and the collected RGB-D image is subjected to light normalization processing to eliminate the influence of uneven light, facilitating subsequent model training. Specifically, a fixed number N of points are randomly sampled to represent the surface shape of the grasped target object, and each point contains three-dimensional coordinates (x, y, z). The processed data set is divided into a training set, a test set and a validation set.

[0070] Step S302, constructing a Glow-CVAE conditional generation model.

[0071] Step S303, training the constructed Glow-CVAE conditional generation model. In the training process, the improved YOLOv11 network structure is mainly used to train the processed data set. At the same time, in order to ensure the physical reasonableness of the hand posture in the training, an additional regularization term can also be introduced to punish parameters exceeding the joint range or unreasonable hand shape changes; but in actual operation, the main reliance is on the real grasping gesture information of the data set itself to ensure reasonableness.

[0072] Step S304, obtaining the trained Glow-CVAE conditional generation model.

[0073] The embodiments of the present application will be described and explained below through preferred embodiments.

[0074] Figure 4 The flowchart of the dexterous hand self-adaptive grasping method according to the preferred embodiments of the present application is shown in Figure 4 , which includes the following steps:

[0075] Step S401, obtain the point cloud data of the preset training template image. Generally, the surface point set is obtained from the RGB-D depth camera or sampled in the simulation environment. Specifically, a fixed number N of points are randomly sampled to represent the surface shape of the grasped target object, and each point contains three-dimensional coordinates (x, y, z).

[0076] Step S402, construct a Glow-CVAE conditional generation model.

[0077] Step S403, train the constructed Glow-CVAE conditional generation model, and obtain the trained Glow-CVAE conditional generation model.

[0078] Step S404, obtain multi-modal perception data. Including: point cloud data corresponding to the grasped target obtained from the RGB-D image corresponding to the grasping action, and tactile patch information of the dexterous hand to the grasped target.

[0079] Step S405, first, the point cloud data is subjected to coordinate normalization processing to generate normalized point cloud data; then the normalized point cloud data is subjected to normalization processing to generate normalized point cloud coordinates located in a preset unit sphere, wherein a normalized point cloud coordinate represents a three-dimensional coordinate corresponding to a point cloud data, and the normalization processing includes the following processing: translation alignment, coordinate scale standardization. In order to eliminate the differences of different object sizes and initial poses, the point cloud data is subjected to normalization processing:

[0080] First, the point cloud coordinates corresponding to the grasped target are subjected to translation alignment: taking the geometric center of the grasped target as the reference, the point cloud data is translated to make the center located at the origin. Then scale standardization is performed: the maximum distance D max of all point clouds of the grasped target relative to the origin is calculated i ; and the point cloud data is scaled according to the scale, so that the grasped target as a whole is normalized to the unit sphere range, which can be expressed as: for the object surface point p i , the center point is calculated ; and the normalized coordinate is represented as: , so that the grasped target as a whole is placed in the unit sphere, wherein N represents the number of points collected on the surface of the grasped target, and p i represents the coordinates of the i-th point, i∈[1, N].

[0081] Coordinate system alignment: define a unified world coordinate system in the desktop scene, set the desktop plane as the XY plane, the gravity direction as the Z axis, and the initial pose of the grasped target conforms to the global coordinate system.

[0082] Step S406, using a multi-layer perceptron corresponding to the PointNet network, the normalized point cloud coordinates are feature extracted to obtain the corresponding point cloud global feature vector. PointNet shares a set of multi-layer perceptron MLP for each point, maps 3D coordinates to a high-dimensional feature space, and forms a 1024-dimensional vector through several fully connected layers and nonlinear activation for each point: Then, a max pooling function is used to take the convergence in the point dimension to obtain a global feature vector independent of the point arrangement. The above PointNet can be used to extract the global feature of the first point cloud, which can be expressed as the following function:

[0083]

[0084] Where, ψ is a point feature extraction subnetwork composed of MLP, max represents taking the maximum value in each dimension; and the obtained is the global feature vector of the object point cloud. This global feature condenses the geometric information of the object shape.

[0085] Step S407, map the tactile patch information to the dexterous hand workspace to generate a simulated tactile signal. The tactile patch information is mapped to the dexterous hand workspace based on a Gaussian kernel function, and the Gaussian function used is as follows:

[0086]

[0087] Where, S i is the output signal value of the i-th tactile sensor; j is the contact point index, j∈[1,m]; m is the number of contact points; is the normal force of the n-th contact point, P j is the coordinate of the j-th contact point, C i is the coordinate of the j-th contact point, C i is the coordinate of the i-th tactile center; is the variance.

[0088] Step S408, using the CVAE generation network, the point cloud global feature vector and the simulated tactile signal are concatenated to generate the first joint feature. First, respectively using the PointNet network, the point cloud global feature vector corresponding to the grasping target and the simulated tactile signal are processed, and the object feature vector and the current posture feature vector of the dexterous hand corresponding to the grasping target are obtained; the object feature vector of the grasping target and the current posture feature vector of the dexterous hand are concatenated into a 2048-dimensional vector [F h ;F0], and the first joint feature is obtained by extracting the information of the joint distribution. This simple concatenation retains all the information of the two modalities.

[0089] Step S409, the first joint feature is inferred by using the variational autoencoder of the Glow-CVAE conditional generation model to generate the posterior distribution parameter, wherein the posterior distribution parameter includes a distribution parameter mean vector and a distribution parameter variance vector.

[0090] Step S410, the distribution parameter mean vector and the distribution parameter variance vector are reparameterized to generate the latent variable; then the latent variable and the point cloud global feature vector are spliced to generate the joint latent feature vector, and then the joint latent feature vector is decoded by using the decoder of the Glow-CVA conditional generation model to generate the candidate grasp gesture set. In the CVAE decoder, the latent variable sampled is spliced with the object feature F0 of the grasping target as [z; F0] as the input of the decoder. The latent variable of the CVAE is usually represented by a low-dimensional Gaussian vector, and the latent variable space with a dimension of |z|=64 is selected in the embodiment, and the latent variable carries the posture information of the diversified grasp gesture. The decoding network structure is as follows: the decoder of the CVAE is a conditional generation network, which maps the latent variable z to the model parameters of the MANO model (a three-dimensional hand parameterized model) . The decoding input includes the latent variable and the conditional input object feature. Specifically, the latent variable z and the object feature F0 are concatenated into a 1088-dimensional vector [F0; z]. Then, the parameter value of the hand is gradually regressed through a multi-layer fully connected network: the decoder MLP structure is . Finally, the 61-dimensional MANO model parameters are output, which include the joint angle of the hand, the global translation and the global rotation (and possibly the hand shape parameter, a total of 61 degrees of freedom), and the input is input into the pre-defined MANO layer, and the corresponding hand three-dimensional mesh is generated.

[0091] Step S411: Based on the improved ContactNet network constructed according to the preset objective function, each candidate grasping gesture is evaluated to obtain multiple grasping loss parameters corresponding to each candidate grasping gesture. These grasping loss parameters include at least one of the following: contact matching error parameter, surface envelope penalty parameter, grasping penetration loss parameter, and self-collision loss. The objective function is determined based on the sub-functions corresponding to these multiple grasping loss parameters. ContactNet is used to evaluate and optimize the rationality of hand-object contact during the grasping generation stage. The improved ContactNet objective function integrates contact matching degree, surface envelope degree, grasping penetration, and self-collision degree to encourage fingers to correctly touch objects and avoid penetrating objects or themselves. Let the current grasping gesture be H, the object point cloud be P0, and the expected contact distribution predicted by ContactNet be c*(p), which is the target contact score of the grasping target surface point cloud. Simultaneously, the actual contact distribution c is calculated from the current hand pose geometry. H (p). The objective function includes the following terms:

[0092] Contact matching error parameter L contact Encourage finger touch on designated object areas. Compare the expected contact probability at each point on the target with the current actual contact probability, using mean squared error to measure the difference. The calculation formula is: Where c*(p) is typically obtained by applying a nonlinear mapping to the distance from the object point to the dexterous hand. The distance d is normalized to [0,1] as the contact score; L contact Punish hand gestures that deviate from the desired contact pattern.

[0093] Capture the penetration loss parameter L penetration The constraint prevents the hand from penetrating the interior of the object. The penetration depth at a point on the object is calculated using the hand's signature distance field (SDF): For each point p on the object surface, the signed distance SDF from it to the hand surface is calculated. H (p), positive outside the hand surface and negative inside. The negative distance values ​​of all points falling inside the hand are summed and squared to obtain the gripping penetration loss, calculated using the following formula:

[0094]

[0095] This factor increases when the hand and the object intersect, prompting the finger to move out of the object during optimization.

[0096] Self-collision loss L self To prevent fingers or palms from intersecting, different joint connection surfaces in the hand model can be selected, and the minimum distance between them can be calculated. Similarly, using distance detection or geometric detection methods, penalties are applied if certain finger-palm components intrude into each other. This can be defined as follows: , traverse the pairs of hand segments i, j that should not collide, accumulate their negative squared distance. In implementation, the hand is approximated by simple geometric primitives, and the penetration depth between segments is computed to approximate the energy, self Ensure that the fingers do not self-penetrate when adjusting the grasp pose.

[0097] Other stability constraints L others : In the tabletop grasping scenario, also include a tabletop plane penetration penalty L plane , to ensure that the hand does not penetrate the tabletop during grasping. Its form is similar to object penetration: if a keypoint of the hand such as the fingertips or the palm bottom is below the tabletop height, penalize the squared distance of the exceeding part. In addition, to ensure the grasp stability, the optimization objective can also add a force closure indicator and an attraction term to encourage the fingertips to be close to the object surface; add joint limit regularization L joint to avoid exceeding the reasonable range.

[0098] Step S412, sequentially normalize and weight all grasp loss parameters corresponding to each candidate grasp gesture to obtain a total grasp loss value E grasp corresponding to each candidate grasp gesture. The calculation formula is as follows:

[0099]

[0100] Among them, is the contact matching error weight, L wrap is the surface envelope penalty parameter, is the surface envelope penalty weight, is the grasp penetration loss weight, is the self-collision loss weight, is the weight of other stability constraints, and the above weights are used to balance the influence of different metrics on optimization; if the other stability constraints are not considered, L is not added in calculation.

[0101] Step S413, select a candidate grasp gesture with the smallest total grasp loss value from all candidate grasp gestures in the candidate grasp gesture set to obtain a target grasp gesture.

[0102] It should be noted that the steps shown in the above process or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here. For example, step S401 and step S402, step S403 and step S404, step S406 and step S407.

[0103] The embodiments also provide a dexterous hand adaptive grasping device for implementing the above embodiments and preferred embodiments, which have been described and will not be repeated. As used below, the terms "module", "unit", "sub-unit" and the like can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware or a combination of software and hardware can also be implemented and conceived.

[0104] Figure 5 is a structural block diagram of a dexterous hand adaptive grasping device according to the embodiments of the present application, as shown in the figure, the device comprises: an acquisition module 51, a data processing module 52, a generation module 53, a stability evaluation module 54. Figure 5

[0105] The acquisition module 51 is configured to acquire multi-modal perception data, wherein the multi-modal perception data comprises: point cloud data corresponding to a grasping target acquired from an RGB-D image corresponding to a grasping action, and tactile patch information of the grasping target by the dexterous hand.

[0106] The data processing module 52 is coupled to the acquisition module 51 and configured to, after processing the point cloud data to generate a point cloud global feature vector corresponding to the grasping target, map the tactile patch information to a dexterous hand workspace to generate a simulated tactile signal.

[0107] The generation module 53 is coupled to the data processing module 52 and configured to process the point cloud global feature vector and the simulated tactile signal using a Glow-CVAE conditional generation model to generate a candidate grasping gesture set comprising joint poses and pose parameters; wherein each candidate grasping gesture of the candidate grasping gesture set corresponds to a joint pose and a pose parameter, and the Glow-CVAE conditional generation model is generated according to point cloud data of a preset training template image and real grasping gesture information corresponding to the preset training template image.

[0108] The stability evaluation module 54 is coupled to the generation module 53 and configured to determine a stability corresponding to each candidate grasping gesture, and select a target grasping gesture from all candidate grasping gestures of the candidate grasping gesture set according to the stability, and grasp the grasping target with the target grasping gesture, wherein the stability is determined according to a contact matching degree, a surface envelope degree, a grasping penetration and self-collision degree of the dexterous hand and the grasping target.

[0109] ​The adaptive grasping device of the dexterous hand provided by the embodiment of the present application adopts multi-modal perception data, wherein the multi-modal perception data comprises: point cloud data corresponding to a grasping target obtained from an RGB-D image corresponding to a grasping action, and tactile patch information of the grasping target to the dexterous hand; after the point cloud data is processed to generate a point cloud global feature vector corresponding to the grasping target, the tactile patch information is mapped to a dexterous hand workspace to generate a simulated tactile signal; a Glow-CVAE conditional generation model is used to process the point cloud global feature vector and the simulated tactile signal to generate a candidate grasping gesture set comprising joint posture and pose parameters; wherein each candidate grasping gesture in the candidate grasping gesture set corresponds to a joint posture and a pose parameter, and the Glow-CVAE conditional generation model is generated according to the point cloud data of a preset training template image and real grasping gesture information corresponding to the preset training template image; the stability of each candidate grasping gesture is determined, and a target grasping gesture is selected from all candidate grasping gestures in the candidate grasping gesture set according to the stability, and the target grasping gesture is used to grasp the grasping target, wherein the stability is determined according to the contact matching degree, surface envelope degree, grasping penetration and self-collision degree of the dexterous hand and the grasping target. The problem of limited adaptability to special-shaped parts and flexible objects and low grasping stability in related technologies is solved, and the beneficial effects of high adaptability and stability of diversified objects grasping in complex scenes are achieved.

[0110] In some embodiments, the generation module 53 further comprises:

[0111] A first fusion unit is configured to concatenate the point cloud global feature vector and the simulated tactile signal using a CVAE generation network to generate a first joint feature.

[0112] A first encoding unit is coupled to the first fusion unit and is configured to infer the first joint feature using a variational autoencoder of a Glow-CVAE conditional generation model to generate a posterior distribution parameter, wherein the posterior distribution parameter comprises a distribution parameter mean vector and a distribution parameter variance vector.

[0113] A first decoding unit is coupled to the first encoding unit and is configured to perform reparameterization sampling on the distribution parameter mean vector and the distribution parameter variance vector to generate a latent variable; after concatenating the latent variable and the point cloud global feature vector to generate a joint latent feature vector, the joint latent feature vector is decoded using a decoder of the Glow-CVA conditional generation model to generate a candidate grasping gesture set.

[0114] In some embodiments, the first fusion unit further comprises:

[0115] The feature extraction component is configured to process the point cloud global feature vector into an object feature vector corresponding to the grasping target by using a PointNet network, and extract a current posture feature vector corresponding to the dexterous hand from the simulated tactile signal by using the PointNet network.

[0116] The joint component is coupled to the feature extraction component and is configured to concatenate the object feature vector and the current posture feature vector to generate a multi-dimensional vector, extract joint distribution information from the multi-dimensional vector by a target fully connected layer of a CVAE generation network, and obtain a first joint feature.

[0117] In some embodiments, the data processing module 52 is configured to perform coordinate normalization on the point cloud data to generate normalized point cloud data, perform normalization on the normalized point cloud data to generate normalized point cloud coordinates located in a preset unit sphere, wherein one normalized point cloud coordinate represents a three-dimensional coordinate corresponding to one point cloud data, and the normalization includes the following processing: translation alignment, coordinate scale standardization; and feature extraction of all normalized point cloud coordinates by using a multi-layer perceptron corresponding to a PointNet network to obtain a corresponding point cloud global feature vector.

[0118] In some embodiments, the data processing module 52 is configured to map the tactile patch information to the dexterous hand workspace based on a Gaussian kernel function to obtain a simulated tactile signal.

[0119] In some embodiments, the stability evaluation module 54 further includes:

[0120] The first calculation unit is configured to evaluate each candidate grasping gesture based on an improved ContactNet network constructed according to a preset target function to obtain a plurality of grasping loss parameters corresponding to each candidate grasping gesture, wherein the grasping loss parameters at least include one of the following: a contact matching error parameter, a surface envelope degree penalty parameter, a grasping penetration loss parameter, and a self-collision loss, and the target function is determined based on a sub-function corresponding to the plurality of grasping loss parameters.

[0121] The first summary unit is coupled to the first calculation unit and is configured to sequentially perform normalization and weighting on all grasping loss parameters corresponding to each candidate grasping gesture to obtain a total grasping loss value corresponding to each candidate grasping gesture, wherein the stability includes the total grasping loss value.

[0122] The first selection unit is coupled to the first summary unit and is configured to select a candidate grasping gesture with the smallest total grasping loss value from all candidate grasping gestures in the candidate grasping gesture set to obtain a target grasping gesture.

[0123] In some embodiments, the first computing unit is configured to predict, based on the point cloud data corresponding to the target object, a probability of an expected contact distribution of each point in the point cloud data using the improved ContactNet network; calculate a probability of a current contact distribution of each point in the point cloud data according to the point cloud data and the candidate grasping gesture; and perform mean square error calculation based on the probability of the expected contact distribution and the probability of the current contact distribution of all points in the point cloud data to obtain a contact matching error corresponding to each candidate grasping gesture, wherein the contact matching error parameter comprises the contact matching error.

[0124] In some embodiments, the first computing unit is configured to predict, based on the point cloud data corresponding to the target object, a probability of an expected contact distribution of each point in the point cloud data using the improved ContactNet network; calculate a probability of a current contact distribution of each point in the point cloud data according to the point cloud data and the candidate grasping gesture; and perform mean square error calculation based on the probability of the expected contact distribution and the probability of the current contact distribution of all points in the point cloud data to obtain a contact matching error corresponding to each candidate grasping gesture, wherein the contact matching error parameter comprises the contact matching error.

[0125] It should be noted that each of the above modules can be a functional module or a program module, and can be implemented by software or hardware. For the modules implemented by hardware, each of the above modules can be located in the same processor; or each of the above modules can also be located in different processors in any combination.

[0126] The embodiment also provides a dexterous hand control system, including a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to perform the steps in any of the above method embodiments.

[0127] Optionally, the dexterous hand can further include a transmission device and an input and output device, wherein the transmission device is connected to the processor, and the input and output device is connected to the processor.

[0128] Optionally, in the embodiment, the processor can be configured to execute the following steps through the computer program:

[0129] S1, obtaining multi-modal perception data, wherein the multi-modal perception data includes point cloud data corresponding to a target object obtained from an RGB-D image corresponding to a grasping action, and tactile patch information of the dexterous hand on the target object.

[0130] S2, after processing the point cloud data to generate the point cloud global feature vector corresponding to the grasping target, mapping the tactile patch information to the dexterous hand workspace to generate a simulated tactile signal.

[0131] S3, processing the point cloud global feature vector and the simulated tactile signal by using a Glow-CVAE conditional generation model to generate a candidate grasping gesture set including joint poses and pose parameters; wherein each candidate grasping gesture in the candidate grasping gesture set corresponds to a joint pose and a pose parameter, and the Glow-CVAE conditional generation model is generated according to the point cloud data of the preset training template image and the real grasping gesture information corresponding to the preset training template image.

[0132] S4, determining the stability corresponding to each candidate grasping gesture, and selecting a target grasping gesture from all candidate grasping gestures in the candidate grasping gesture set according to the stability, and grasping the grasping target with the target grasping gesture, wherein the stability is determined according to the contact matching degree, surface envelope degree, grasping penetration and self-collision degree of the dexterous hand and the grasping target.

[0133] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation manners, and this embodiment will not be described here.

[0134] In addition, in combination with the dexterous hand adaptive grasping method in the above embodiments, the application embodiment can provide a storage medium for implementation. The storage medium has a computer program stored thereon; the computer program is executed by a processor to implement any one of the dexterous hand adaptive grasping methods in the above embodiments.

[0135] Those skilled in the art should understand that each technical feature of the above-described embodiments can be combined arbitrarily, and in order to make the description concise, each technical feature of the above-described embodiments is not described in all possible combinations, however, as long as the combination of these technical features does not exist contradictory, it should be considered as the scope of the description.

[0136] The above-described embodiments only express several implementation manners of the application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the application, a number of variations and improvements can be made, which are all within the scope of the application. Therefore, the patent protection scope of the application should be subject to the appended claims.

Claims

1. A dexterous hand adaptive grasping method, characterized in that, include: Acquire multimodal perception data, wherein the multimodal perception data includes: point cloud data corresponding to the grasping target obtained from the RGB-D image corresponding to the grasping action, and tactile patch information of the dexterous hand on the grasping target; After processing the point cloud data to generate the global feature vector of the point cloud corresponding to the grasping target, the tactile patch information is mapped to the dexterous hand workspace to generate simulated tactile signals. The Glow-CVAE conditional generation model is used to process the global feature vector of the point cloud and the simulated tactile signal to generate a candidate grasping gesture set including joint pose and pose parameter; wherein, each candidate grasping gesture in the candidate grasping gesture set corresponds to a joint pose and a pose parameter, and the Glow-CVAE conditional generation model is trained and generated based on the point cloud data of the preset training template image and the real grasping gesture information corresponding to the preset training template image. Determine the stability corresponding to each candidate grasping gesture, and select a target grasping gesture from all the candidate grasping gestures in the candidate grasping gesture set according to the stability, and grasp the target with the target grasping gesture. The stability is determined based on the contact matching degree, surface envelope degree, grasping penetration and self-collision degree between the dexterous hand and the grasping target. Specifically, the Glow-CVAE conditional generation model is used to process the global feature vector of the point cloud and simulated tactile signals to generate a candidate set of grasping gestures, including joint pose and positional parameters, comprising: Using a CVAE generator network, the global feature vector of the point cloud and the simulated tactile signal are concatenated to generate a first joint feature; Using the variational autoencoder of the Glow-CVAE conditional generation model, the first joint feature is inferred to generate posterior distribution parameters, wherein the posterior distribution parameters include a distribution parameter mean vector and a distribution parameter variance vector. Reparameter sampling is performed on the mean vector and variance vector of the distribution parameters to generate latent variables; After concatenating the latent variables and the global feature vector of the point cloud to generate a joint latent feature vector, the decoder of the Glow-CVAE conditional generation model is used to decode the joint latent feature vector to generate the candidate grasping gesture set. Specifically, a CVAE (Continuous Vision Amplifier) ​​network is used to concatenate the global feature vector of the point cloud with the simulated tactile signal to generate a first joint feature, including: Using the PointNet network, the global feature vector of the point cloud is processed into the object feature vector corresponding to the grasping target, and the PointNet network is used to extract the current posture feature vector corresponding to the dexterous hand from the simulated tactile signal; The object feature vector and the current pose feature vector are concatenated to generate a multidimensional vector. The joint distribution information is then extracted from the multidimensional vector through the target fully connected layer of the CVAE generation network to obtain the first joint feature.

2. The dexterous hand adaptive grasping method according to claim 1, characterized in that, The point cloud data is processed to generate a global feature vector of the point cloud corresponding to the target being captured, including: After performing coordinate normalization processing on the point cloud data to generate normalized point cloud data, the normalized point cloud data is then normalized to generate normalized point cloud coordinates located within a preset unit sphere. Here, a normalized point cloud coordinate represents the three-dimensional coordinates corresponding to a point cloud data. The normalization processing includes the following processes: translation alignment and coordinate scale standardization. Using a multilayer perceptron corresponding to the PointNet network, features are extracted from all the normalized point cloud coordinates to obtain the corresponding global feature vector of the point cloud.

3. The dexterous hand adaptive grasping method according to claim 1, characterized in that, Mapping the tactile patch information onto the dexterous hand workspace to generate simulated tactile signals includes: The tactile patch information is mapped to the dexterous hand workspace based on the Gaussian kernel function to obtain the simulated tactile signal.

4. The dexterous hand adaptive grasping method according to claim 1, characterized in that, Determine the stability corresponding to each candidate grasping gesture, and select a target grasping gesture from all candidate grasping gestures in the candidate grasping gesture set based on the stability, including: Based on the improved ContactNet network constructed according to the preset objective function, each candidate grasping gesture is evaluated to obtain multiple grasping loss parameters corresponding to each candidate grasping gesture. The grasping loss parameters include at least one of the following: contact matching error parameter, surface envelope penalty parameter, grasping penetration loss parameter, and self-collision loss. The objective function is determined based on the sub-functions corresponding to the multiple grasping loss parameters. The grasping loss parameters corresponding to each candidate grasping gesture are normalized and weighted sequentially to obtain the total grasping loss value corresponding to each candidate grasping gesture, wherein the stability includes the total grasping loss value. From all the candidate grasping gestures in the candidate grasping gesture set, select the candidate grasping gesture with the smallest total grasping loss value to obtain the target grasping gesture.

5. The dexterous hand adaptive grasping method according to claim 4, characterized in that, The improved ContactNet network is used to evaluate each candidate grab gesture, including: Based on the point cloud data corresponding to the target being captured, the improved ContactNet network is used to predict the expected contact distribution probability corresponding to each point in the point cloud data. Based on the point cloud data and the candidate grasping gestures, calculate the current contact distribution probability corresponding to each point in the point cloud data; Based on the expected contact distribution probability and the current contact distribution probability corresponding to all points in the point cloud data, a mean square error is calculated to obtain the contact matching degree error corresponding to each candidate grasping gesture, wherein the contact matching degree error parameter includes the contact matching degree error.

6. The dexterous hand adaptive grasping method according to claim 4, characterized in that, The improved ContactNet network is used to evaluate each candidate grab gesture, including: Using the improved ContactNet network, predict the optimized grasp gesture corresponding to each of the candidate grasp gestures; Calculate the symbol distance between each point in the point cloud data and the corresponding optimized grasping gesture, and based on the symbol distances corresponding to all points in the point cloud data, determine the signature distance field between the optimized grasping gesture and the grasping target, and obtain the grasping penetration loss parameter corresponding to the candidate grasping gesture.

7. A dexterous hand control system, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the dexterous hand adaptive grasping method according to any one of claims 1 to 6.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the dexterous hand adaptive grasping method as described in any one of claims 1 to 6.

Images