Dexterous hand self-adaptive grabbing method, dexterous hand control system and storage medium

Abstract

The invention relates to a dexterous hand self-adaptive grabbing method, a dexterous hand control system and a storage medium. The dexterous hand self-adaptive grabbing method comprises the steps that multi-modal sensing data is acquired; after the point cloud data is processed and a point cloud global feature vector corresponding to a grabbed target is generated, the tactile patch information is mapped to a dexterous hand working space, and a simulation tactile signal is generated; a Glow-CVAE condition generation model is utilized to process the point cloud global feature vector and the simulated tactile signal, and a candidate grabbing gesture set including joint attitude and pose parameters is generated; and the stability corresponding to each candidate grabbing gesture is determined, a target grabbing gesture is selected from the candidate grabbing gesture set according to the stability, and the grabbing target is grabbed according to the target grabbing gesture. By means of the method and device, the problems that in the prior art, adaptability to special-shaped parts and flexible objects is limited, and grabbing stability is not high are solved, and the beneficial effects of high adaptability and stability of grabbing of diversified objects in a complex scene are achieved.

Description

Technical Field

[0001] This application relates to the fields of embodied intelligence and intelligent robot control technology, and in particular to a dexterous hand adaptive grasping method, a dexterous hand control system, and a storage medium. Background Technology

[0002] Currently, sorting operations in the logistics industry are still mainly based on manual labor and traditional mechanical devices, leaving significant room for improvement in automation levels. Although leading domestic and international companies have begun exploring automation, they still face technical challenges due to the diversity of goods. Industry practice shows that robot systems equipped with adaptive grippers can improve sorting efficiency by more than 40% compared to traditional solutions. "Quality and efficiency improvement" and "flexible adaptation" have become core technical demands, driving the engineering exploration of multi-degree-of-freedom dexterous hand technology. Significant progress has been made in research on dexterous hand grasping strategies to address these needs. A dexterous hand is defined as a multi-finger, multi-degree-of-freedom robotic hand (typically a 3-5 finger structure with a total of ≥20 degrees of freedom). It adapts to objects of different shapes through various modalities such as wrapping and pinching. Its high degree-of-freedom characteristics provide the hardware foundation for grasping irregularly shaped items, but also bring computational complexity to control and planning.

[0003] In recent years, several mainstream techniques have emerged for generating grasping strategies: RGB vision-guided grasping pose estimation methods utilize semantic analysis of RGB images to estimate the graspable region and pose. Core technologies include: deep learning frameworks using CNNs to directly regress grasping pose parameters and quality scores; unsupervised adaptive learning (k-means clustering to segment foreground objects); and multi-view information fusion to improve robustness in occluded scenes. The advantages are low perception cost and fast inference speed, but the drawback is the lack of depth information leading to 3D localization errors. Grasping strategy synthesis methods based on 3D point cloud data determine stable contact positions and 6-DOF poses through point cloud geometric analysis. They possess 3D shape perception capabilities and outperform planar vision in multi-angle grasping and obstacle avoidance scenarios, but have high computational complexity, are sensitive to sensor accuracy, and require force control to handle flexible objects. The development of 6D grasping algorithms and benchmark datasets continues to drive progress in this field. Human-instructed imitation learning methods employ a "demonstration-learning" paradigm, acquiring grasping strategies through human demonstrations. The implementation path includes intuitive instruction and motion trajectory recording. Its core advantage is high sample efficiency, using supervised learning methods such as behavior cloning to train the policy model and reduce trial-and-error costs. However, it faces the "covariate shift" problem, leading to unstable performance in real-world environments. Human-manipulation trajectory-based policy learning methods extract general policies from large-scale human manipulation trajectory data, with data sources including human hand manipulation trajectory libraries and teleoperated robot trajectory data. However, these methods not only incur high data acquisition and annotation costs but also suffer from insufficient multimodal information fusion capabilities, making them ill-suited for complex scenarios.

[0004] In summary, given the increasingly diverse and complex application scenarios, existing dexterous hand grasping strategies have limited adaptability to irregularly shaped or flexible objects and low grasping stability. Therefore, there is an urgent need for an adaptive flexible grasping technology that can achieve robust grasping of diverse items. Summary of the Invention

[0005] This application provides a dexterous hand adaptive grasping method, a dexterous hand control system, and a storage medium to at least solve the problems of limited adaptability to irregularly shaped parts and flexible objects and low grasping stability in related technologies.

[0006] In a first aspect, embodiments of this application provide a dexterous hand adaptive grasping method, comprising: acquiring multimodal perception data, wherein the multimodal perception data includes: point cloud data corresponding to the grasping target obtained from an RGB-D image corresponding to the grasping action, and tactile patch information of the dexterous hand on the grasping target; after processing the point cloud data to generate a global feature vector of the point cloud corresponding to the grasping target, mapping the tactile patch information to the dexterous hand workspace to generate a simulated tactile signal; and using a Glow-CVAE conditional generation model to process the global feature vector of the point cloud and the simulated tactile signal to generate a candidate grasping gesture set including joint posture and pose parameters. In this context, each candidate grasping gesture in the candidate grasping gesture set corresponds to a joint pose and a pose parameter. The Glow-CVAE conditional generation model is trained and generated based on the point cloud data of a preset training template image and the real grasping gesture information corresponding to the preset training template image. The stability corresponding to each candidate grasping gesture is determined, and based on the stability, a target grasping gesture is selected from all the candidate grasping gestures in the candidate grasping gesture set, and the target grasping gesture is used to grasp the target. The stability is determined based on the contact matching degree, surface envelope degree, grasping penetration and self-collision degree between the dexterous hand and the grasping target.

[0007] Secondly, embodiments of this application provide a dexterous hand control system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the dexterous hand adaptive grasping method as described in the first aspect.

[0008] Thirdly, embodiments of this application provide a storage medium storing a computer program that, when executed by a processor, implements the dexterous hand adaptive grasping method as described in the first aspect above.

[0009] Compared to related technologies, the dexterous hand adaptive grasping method, dexterous hand control system, and storage medium provided in this application acquire multimodal perception data, which includes: point cloud data corresponding to the grasping target obtained from the RGB-D image corresponding to the grasping action, and tactile patch information of the dexterous hand on the grasping target; after processing the point cloud data to generate a global feature vector of the point cloud corresponding to the grasping target, the tactile patch information is mapped to the dexterous hand workspace to generate a simulated tactile signal; the Glow-CVAE conditional generation model is used to process the global feature vector of the point cloud and the simulated tactile signal to generate a candidate grasping gesture set including joint posture and pose parameters; wherein each candidate grasping gesture in the candidate grasping gesture set corresponds to one of the above. The joint pose and a pose parameter are used. The Glow-CVAE conditional generation model is trained and generated based on the point cloud data of the preset training template image and the real grasping gesture information corresponding to the preset training template image. The stability corresponding to each candidate grasping gesture is determined, and based on the stability, a target grasping gesture is selected from all the candidate grasping gestures in the candidate grasping gesture set, and the target grasping gesture is used to grasp the grasping target. The stability is determined based on the contact matching degree, surface envelope degree, grasping penetration and self-collision degree between the dexterous hand and the grasping target. This solves the problem of limited adaptability to irregular parts and flexible objects and low grasping stability in related technologies, and achieves the beneficial effect of high adaptability and stability in grasping diverse objects in complex scenes.

[0010] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a hardware structure block diagram of a terminal for a dexterous hand adaptive grasping method according to an embodiment of this application; Figure 2 This is a flowchart of a dexterous hand adaptive grasping method according to an embodiment of this application; Figure 3 This is a flowchart illustrating the construction process of the Glow-CVAE conditional generation model according to an embodiment of this application; Figure 4 This is a flowchart of a dexterous hand adaptive grasping method according to a preferred embodiment of this application; Figure 5 This is a structural block diagram of a dexterous hand control system according to an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0013] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0014] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0015] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. Taking running on a terminal as an example, Figure 1 This is a hardware structure block diagram of the terminal of the dexterous hand adaptive grasping method according to an embodiment of this application. For example... Figure 1 As shown, terminal 10 may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, terminal 10 may also include components that are larger than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0016] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the dexterous hand adaptive grasping method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0017] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0018] This embodiment provides a dexterous hand adaptive grasping method running on the aforementioned terminal. Figure 2 This is a flowchart of a dexterous hand adaptive grasping method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0019] Step S201: Obtain multimodal perception data, wherein the multimodal perception data includes: point cloud data corresponding to the grasping target obtained from the RGB-D image corresponding to the grasping action, and tactile patch information of the dexterous hand on the grasping target.

[0020] In this embodiment, by acquiring multimodal sensing data, including point cloud data collected from RGB-D images and tactile patch information of the dexterous hand on the grasping target (including the force distribution of the tactile patch), tactile and visual fusion is achieved, greatly improving the accuracy and stability of grasping. Single-modal sensing data is insufficient to cope with complex application scenarios, requiring the fusion of multiple technologies to achieve complementary advantages. Multimodal fusion enables dual capabilities of visual geometric cognition and tactile physical cognition, not only solving the problem of poor adaptability of the dexterous hand to irregularly shaped parts and soft packaging, but also ensuring the stability of grasping.

[0021] Step S202: After processing the point cloud data and generating the global feature vector of the point cloud corresponding to the grasping target, the tactile patch information is mapped to the dexterous hand workspace to generate simulated tactile signals.

[0022] In this embodiment, by processing the point cloud data, a global feature vector of the point cloud corresponding to the grasping target is generated, and the tactile patch information is mapped to the dexterous hand workspace to generate simulated tactile signals. This eliminates the differences in the size and initial posture of different objects, reduces the amount of computation and the learning difficulty, and realizes efficient grasping by the dexterous hand.

[0023] Step S203: Using the Glow-CVAE conditional generation model, the global feature vector of the point cloud and the simulated tactile signal are processed to generate a candidate grasping gesture set including joint pose and pose parameter; wherein, each candidate grasping gesture in the candidate grasping gesture set corresponds to a joint pose and a pose parameter, and the Glow-CVAE conditional generation model is trained and generated based on the point cloud data of the preset training template image and the real grasping gesture information corresponding to the preset training template image.

[0024] In this embodiment, by utilizing the Glow-CVAE conditional generation model to process the global feature vector of the point cloud and simulated tactile signals, a candidate grasping gesture set including joint pose and positional parameters is generated, which enhances the adaptability of dexterous hand grasping. Since the Glow-CVAE conditional generation model integrates the object features of the grasping target and the pose features of the dexterous hand, it achieves tactile and visual fusion, greatly improving the adaptability of grasping. Furthermore, the use of a conditional autoencoder (CVAE) to introduce conditional variables not only achieves more accurate data generation, but also enhances the robustness of the model and improves training stability by optimizing the matching of the latent distribution and prior distribution.

[0025] Step S204: Determine the stability corresponding to each candidate grasping gesture, and select the target grasping gesture from all candidate grasping gestures in the candidate grasping gesture set according to the stability, and grasp the target with the target grasping gesture. The stability is determined based on the contact matching degree, surface envelope degree, grasping penetration and self-collision degree between the dexterous hand and the grasping target.

[0026] In this embodiment, by determining the stability corresponding to each candidate grasping gesture, and selecting the target grasping gesture from all candidate grasping gestures in the candidate grasping gesture set based on the stability, and grasping the target with the target grasping gesture, the stability of the candidate grasping gesture can be evaluated by comprehensively considering contact matching degree, surface envelope degree, grasping penetration and self-collision degree. This makes it less likely for the dexterous hand to slip and fall when grasping the target object, reducing the grasping damage rate and ensuring the stability of grasping.

[0027] Through steps S201 to S204 above, multimodal perception data is acquired, including: point cloud data corresponding to the grasping target obtained from the RGB-D image corresponding to the grasping action, and tactile patch information of the dexterous hand on the grasping target; after processing the point cloud data to generate a global feature vector of the point cloud corresponding to the grasping target, the tactile patch information is mapped to the dexterous hand workspace to generate a simulated tactile signal; using the Glow-CVAE conditional generation model, the global feature vector of the point cloud and the simulated tactile signal are processed to generate a candidate grasping gesture set including joint posture and pose parameters; wherein each candidate grasping gesture in the candidate grasping gesture set corresponds to a joint posture and a pose. The parameters of the Glow-CVAE conditional generation model are generated by training on point cloud data of preset training template images and the corresponding real grasping gesture information. The stability of each candidate grasping gesture is determined, and based on the stability, a target grasping gesture is selected from all candidate grasping gestures in the candidate grasping gesture set. The target grasping gesture is then used to grasp the target. The stability is determined based on the contact matching degree, surface envelope degree, grasping penetration and self-collision degree between the dexterous hand and the grasping target. This solves the problem of limited adaptability to irregular and flexible objects and low grasping stability in related technologies, and achieves the beneficial effect of high adaptability and stability in grasping diverse objects in complex scenes.

[0028] It should be noted that in this embodiment, the stability of the multimodal perception data is determined based on the contact matching degree, surface envelope, grasping penetration, and self-collision degree between the dexterous hand and the grasping target. Other stability constraints can be added to the target optimization, such as adding force closure indicators and attraction terms to encourage the dexterous hand's fingertips to approach the surface of the grasping target, and adding joint limit regularization terms to prevent finger joints from exceeding reasonable limits. This not only eliminates more detail interference and improves the realism and practicality of grasping, but also demonstrates that the ContactNet algorithm has high computational efficiency and high adaptability to complex scenes, achieving fast and robust grasping.

[0029] In some embodiments, the Glow-CVAE conditional generation model is used to process the global feature vector of the point cloud and simulated tactile signals to generate a candidate set of grasping gestures, including joint pose and positional parameters, through the following steps:

[0030] Step 21: Using the CVAE generation network, the global feature vector of the point cloud is concatenated with the simulated tactile signal to generate the first joint feature.

[0031] Step 22: Use the variational autoencoder of the Glow-CVAE conditional generative model to infer the first joint feature and generate posterior distribution parameters, wherein the posterior distribution parameters include the distribution parameter mean vector and the distribution parameter variance vector.

[0032] Step 23: Perform reparameter sampling on the mean vector and variance vector of the distribution parameters to generate latent variables.

[0033] Step 24: After concatenating the latent variables and the global feature vector of the point cloud to generate a joint latent feature vector, the decoder of the Clow-CVA conditional generation model is used to decode the joint latent feature vector to generate a candidate grasping gesture set.

[0034] The above steps utilize the CVAE generative network to concatenate the global feature vector of the point cloud with the simulated tactile signal to generate the first joint feature. The variational autoencoder of the Glow-CVAE conditional generative model is then used to infer the first joint feature, generating posterior distribution parameters, which include the mean vector and variance vector of the distribution parameters. The mean and variance vectors of the distribution parameters are then resampled to generate latent variables. After concatenating the latent variables with the global feature vector of the point cloud to generate a joint latent feature vector, the decoder of the Clow-CVA conditional generative model decodes the joint latent feature vector to generate a candidate grasping gesture set. This allows for higher adaptability in dexterous hand grasping. The use of the conditional variational autoencoder CVAE introduces conditional variables, achieving not only more accurate data generation but also enhancing the robustness and training stability of the model by optimizing the matching between the latent and prior distributions.

[0035] In some embodiments, a CVAE generation network is used to concatenate the global feature vector of the point cloud with the simulated tactile signal to generate the first joint feature, which is achieved through the following steps:

[0036] Step 31: Using the PointNet network, the global feature vector of the point cloud is processed into the object feature vector corresponding to the grasping target, and the PointNet network is used to extract the current posture feature vector corresponding to the dexterous hand from the simulated tactile signal.

[0037] Step 32: Concatenate the object feature vector and the current pose feature vector to generate a multidimensional vector, and extract joint distribution information from the multidimensional vector through the target fully connected layer of the CVAE generation network to obtain the first joint feature.

[0038] By utilizing the PointNet network in the above steps, the global feature vector of the point cloud is processed into the object feature vector corresponding to the grasping target. The PointNet network is then used to extract the current posture feature vector corresponding to the dexterous hand from the simulated tactile signal. The object feature vector and the current posture feature vector are concatenated to generate a multi-dimensional vector. The joint distribution information is then extracted from the multi-dimensional vector through the target fully connected layer of the CVAE generator network to obtain the first joint feature. By fusing the object features of the grasping target and the posture features of the dexterous hand, the fusion of tactile and visual senses is achieved, which not only improves the adaptability of grasping but also improves grasping efficiency.

[0039] In some embodiments, the point cloud data is processed to generate a global feature vector of the point cloud corresponding to the grasping target, including the following steps:

[0040] Step 41: After performing coordinate normalization on the point cloud data to generate normalized point cloud data, normalize the normalized point cloud data to generate normalized point cloud coordinates located within a preset unit sphere. Here, a normalized point cloud coordinate represents the three-dimensional coordinates corresponding to a point cloud data. The normalization process includes the following processes: translation alignment and coordinate scale standardization.

[0041] Step 42: Using the multilayer perceptron corresponding to the PointNet network, extract features from all normalized point cloud coordinates to obtain the corresponding global feature vector of the point cloud.

[0042] After performing coordinate normalization on the point cloud data in the above steps to generate normalized point cloud data, the normalized point cloud data is then normalized to generate normalized point cloud coordinates located within a preset unit sphere. Each normalized point cloud coordinate represents the 3D coordinates corresponding to a point cloud data point. The normalization process includes the following steps: translation alignment and coordinate scale standardization. Using a multilayer perceptron corresponding to the PointNet network, features are extracted from all normalized point cloud coordinates to obtain the corresponding global feature vector of the point cloud. The normalization process eliminates differences in object size and initial pose, reducing the learning difficulty. Simultaneously, the PointNet network is concise and efficient, improving the grasping efficiency.

[0043] In some embodiments, tactile patch information is mapped to the dexterous hand workspace to generate simulated tactile signals, which is achieved through the following steps: mapping tactile patch information to the dexterous hand workspace based on a Gaussian kernel function to obtain simulated tactile signals.

[0044] In this embodiment, the Gaussian kernel function has a strong nonlinear fitting capability and can handle complex data distributions, thus making the crawling more adaptable.

[0045] In some embodiments, the stability corresponding to each candidate grasping gesture is determined, and the target grasping gesture is selected from all candidate grasping gestures in the candidate grasping gesture set based on the stability, including the following steps:

[0046] Step 51: Based on the improved ContactNet network constructed according to the preset objective function, evaluate each candidate grasping gesture to obtain multiple grasping loss parameters corresponding to each candidate grasping gesture. Among them, the grasping loss parameters include at least one of the following: contact matching degree error parameter, surface envelope degree penalty parameter, grasping penetration loss parameter, and self-collision loss. The objective function is determined based on the sub-functions corresponding to the multiple grasping loss parameters.

[0047] Step 52: Normalize and weight all the grab loss parameters corresponding to each candidate grab gesture in sequence to obtain the total grab loss value corresponding to each candidate grab gesture. The stability includes the total grab loss value.

[0048] Step 53: Select the candidate grasping gesture with the smallest total grasping loss value from all candidate grasping gestures in the candidate grasping gesture set to obtain the target grasping gesture.

[0049] The improved ContactNet network, constructed according to a preset objective function, is used in the above steps to evaluate each candidate grasping gesture, obtaining multiple grasping loss parameters corresponding to each candidate grasping gesture. These grasping loss parameters include at least one of the following: contact matching error parameter, surface envelope penalty parameter, grasping penetration loss parameter, and self-collision loss. The objective function is determined based on the sub-functions corresponding to these multiple grasping loss parameters. All grasping loss parameters corresponding to each candidate grasping gesture are then normalized and weighted sequentially to obtain the total grasping loss value for each candidate grasping gesture. Stability includes the total grasping loss value. From all candidate grasping gestures in the candidate grasping gesture set, the one with the highest total grasping loss value is selected. Small candidate grasping gestures are used to obtain the target grasping gesture. Among them, the contact matching degree error parameter is used to characterize the energy loss of the contact matching degree between the dexterous hand and the grasping target; the surface envelope degree penalty parameter is used to characterize the energy loss of the surface envelope degree of the dexterous hand relative to the grasping target; the grasping penetration loss parameter is used to characterize the degree of grasping penetration of the dexterous hand relative to the grasping target; and the self-collision loss is used to characterize the degree of self-collision of the dexterous hand. The stability evaluation of the candidate grasping gesture is comprehensively based on the contact matching degree, surface envelope degree, grasping penetration and self-collision degree, which makes it less likely for the dexterous hand to slip and fall when grasping the target object when disturbed, reduces the grasping breakage rate, improves grasping stability, and achieves high adaptability to grasping diverse objects such as irregularly shaped parts and flexible objects in complex scenarios.

[0050] In some embodiments, each candidate grab gesture is evaluated using an improved ContactNet network, achieved through the following steps:

[0051] Step 61: Based on the point cloud data corresponding to the target being captured, use the improved ContactNet network to predict the expected contact distribution probability corresponding to each point in the point cloud data.

[0052] Step 62: Based on the point cloud data and candidate grab gestures, calculate the current contact distribution probability corresponding to each point in the point cloud data.

[0053] Step 63: Based on the expected contact distribution probability and the current contact distribution probability corresponding to all points in the point cloud data, perform mean square error calculation to obtain the contact matching degree error corresponding to each candidate grasping gesture. The contact matching degree error parameter includes the contact matching degree error.

[0054] In this embodiment, PointNet processes point cloud computing in a simple and efficient manner, while the grasping posture generated by the ContactNet network has higher physical consistency and naturalness in real-world scenes, and the grasping action has high stability.

[0055] In some embodiments, an improved ContactNet network is used to evaluate each candidate grab gesture, including the following steps:

[0056] Step 71: Using the improved ContactNet network, predict the optimal grab gesture corresponding to each candidate grab gesture.

[0057] Step 72: Calculate the symbolic distance between each point in the point cloud data and the corresponding optimized grasping gesture. Based on the symbolic distances between all points in the point cloud data, determine the signature distance field between the optimized grasping gesture and the grasping target, and obtain the grasping penetration loss parameters corresponding to the candidate grasping gestures.

[0058] By utilizing the improved ContactNet network in the above steps, the optimized grasping gesture corresponding to each candidate grasping gesture is predicted; the symbolic distance between each point in the point cloud data and the corresponding optimized grasping gesture is calculated; and based on the symbolic distances of all points in the point cloud data, the sign distance field between the optimized grasping gesture and the grasping target is determined, thus obtaining the grasping penetration loss parameters corresponding to the candidate grasping gestures; the grasping posture is generated simply and efficiently, and the grasping posture is closer to real and natural, with high grasping action stability.

[0059] In some embodiments, the evaluation of each candidate grabbing gesture using an improved ContactNet network further includes the following steps: Step 81: Using the improved ContactNet network, predict the grab gesture corresponding to each candidate grab gesture.

[0060] Step 82: Select multiple target joint connection surfaces from the predicted grasping gesture, and calculate the signature distance field between two target joint connection surfaces among the multiple target joint connection surfaces, wherein the target joint connection surfaces include at least one of the following: finger connection surface, palm component connection surface.

[0061] Step 83: Select the sign distance field with the smallest sign distance field from all sign distance fields to obtain the self-collision loss corresponding to the candidate grasping gesture.

[0062] The following section will introduce the construction and training methods of the neural network model (Glow-CVAE conditional generative model). Figure 3 This is a flowchart illustrating the construction process of the Glow-CVAE conditional generation model according to an embodiment of this application, as follows: Figure 3 As shown, it includes the following steps:

[0063] Step S301: Obtain point cloud data of the preset training template image. This is typically achieved by acquiring a set of surface points from an RGB-D depth camera or sampling from a simulated environment. The acquired RGB-D images are then normalized to eliminate the influence of uneven lighting, facilitating subsequent model training. Specifically, a fixed number of N points are randomly sampled to represent the surface shape of the target object, with each point containing three-dimensional coordinates (x, y, z). The processed dataset is then divided into a training set, a test set, and a validation set.

[0064] Step S302: Construct the Glow-CVAE conditional generation model.

[0065] Step S303 involves training the constructed Glow-CVAE conditional generation model. During training, an improved YOLOv11 network structure is primarily used to train the processed dataset. To ensure the physical plausibility of hand poses during training, additional regularization terms can be introduced to penalize parameters exceeding joint limits or unreasonable hand deformations; however, in practice, plausibility is primarily ensured by relying on the actual grasping gesture information within the dataset itself.

[0066] Step S304 yields the trained Glow-CVAE conditional generation model.

[0067] The embodiments of this application will be described and illustrated below through preferred embodiments.

[0068] Figure 4 This is a flowchart of a dexterous hand adaptive grasping method according to a preferred embodiment of this application. Figure 4 As shown, the dexterous hand adaptive grasping method includes the following steps:

[0069] Step S401: Obtain point cloud data of the preset training template image. This is typically acquired from an RGB-D depth camera or sampled from a set of surface points in a simulated environment. Specifically, a fixed number of N points are randomly sampled to represent the surface shape of the target object, with each point containing three-dimensional coordinates (x, y, z).

[0070] Step S402: Construct the Glow-CVAE conditional generation model.

[0071] Step S403: Train the constructed Glow-CVAE conditional generation model to obtain the trained Glow-CVAE conditional generation model.

[0072] Step S404: Acquire multimodal perception data. This includes: point cloud data corresponding to the grasping target obtained from the RGB-D image corresponding to the grasping action, and tactile patch information of the dexterous hand on the grasping target.

[0073] Step S405: First, the point cloud data is normalized to generate normalized point cloud data; then, the normalized point cloud data is normalized to generate normalized point cloud coordinates within a preset unit sphere. Each normalized point cloud coordinate represents the 3D coordinates corresponding to a point cloud data point. The normalization process includes the following steps: translation alignment and coordinate scale standardization. To eliminate differences in object size and initial pose, the point cloud data is normalized:

[0074] First, the point cloud coordinates corresponding to the target are translated and aligned: using the geometric center of the target as a reference, the point cloud data is translated so that the center is located at the origin. Next, scale standardization is performed: the maximum distance D of all point clouds of the target relative to the origin is calculated. max The point cloud data is then scaled according to this scale so that the entire target object is normalized to a unit sphere. This can be expressed as: for point p on the object surface... i Calculate the center point ,as well as The normalized coordinates are then expressed as: This ensures that the entire target is placed within a unit sphere, where N represents the number of points collected on the target's surface, and p i Let i represent the coordinates of the i-th point, i∈[1,N].

[0075] Coordinate system alignment: Define a unified world coordinate system in the desktop scene, set the desktop plane as the XY plane and the gravity direction as the Z axis, and let the initial posture of the target be grasped conform to this global coordinate system.

[0076] Step S406: Using the multilayer perceptron corresponding to the PointNet network, features are extracted from all normalized point cloud coordinates to obtain the corresponding global feature vector of the point cloud. PointNet shares a set of multilayer perceptrons (MLPs) for each point, mapping the 3D coordinates to a high-dimensional feature space, and passing each point through several fully connected layers and nonlinear activations to form a 1024-dimensional vector. Then, the max pooling function is used to converge the points to obtain a global feature vector that is independent of the point arrangement. The feature extraction of the first point cloud global data by PointNet can be represented by the following function:

[0077] Where, 𝜓 is a point feature extraction subnetwork composed of MLP, and max represents taking the maximum value in each dimension; the resulting This is the global feature vector of the object's point cloud. This global feature condenses the geometric information of the object's shape.

[0078] Step S407: Map the tactile patch information to the dexterous hand workspace to generate simulated tactile signals. The tactile patch information is mapped to the dexterous hand workspace based on a Gaussian kernel function, using the following Gaussian function form:

[0079] Where, S i is the output signal value of the i-th tactile sensor; j is the contact point index, j∈[1,m]; m is the number of contact points; P represents the magnitude of the normal force at the nth contact point. j Let C be the coordinates of the j-th contact point. i Let i be the coordinates of the i-th tactile center; Let Variance be the variance.

[0080] Step S408: Using the CVAE generation network, the global feature vector of the point cloud and the simulated tactile signal are concatenated to generate the first joint feature. First, the PointNet network is used to process the global feature vector of the point cloud corresponding to the grasping target and the simulated tactile signal, respectively, to obtain the object feature vector of the grasping target. and the current pose feature vector of the dexterous hand The object feature vector of the grasped target and the current pose feature vector of the dexterous hand are concatenated into a 2048-dimensional vector [F]. h [F0], and by extracting information from the joint distribution, the first joint feature is obtained. This simple concatenation preserves all the information of the two modes.

[0081] Step S409: Using the variational autoencoder of the Glow-CVAE conditional generation model, the first joint feature is inferred to generate posterior distribution parameters, wherein the posterior distribution parameters include the distribution parameter mean vector and the distribution parameter variance vector.

[0082] Step S410 involves resampling the mean vector and variance vector of the distribution parameters to generate latent variables. These latent variables are then concatenated with the global feature vector of the point cloud to generate a joint latent feature vector. The decoder of the Clow-CVA conditional generation model is then used to decode the joint latent feature vector to generate a candidate grasping gesture set. In the CVAE decoder, the sampled latent variables... The object features F0 of the grasping target are concatenated to form [z;F0], which serves as the input to the decoder. Latent variables in CVAE are typically represented by low-dimensional Gaussian vectors. This embodiment selects a latent variable space with dimension |z|=64. This latent vector carries the pose information of diverse grasping gestures. Decoding network structure: The CVAE decoder is a conditional generative network that maps the latent variable z to the model parameters of the MANO model (a 3D parametric hand model). The decoding input includes latent variables and object features from conditional inputs. Specifically, the latent variable z is first concatenated with the object feature F0 to form a 1088-dimensional vector [F0;z]. Then, the parameter values ​​are progressively regressed through a multi-layer fully connected network. The decoder MLP structure is as follows: Finally, the 61-dimensional MANO model parameters are output. This includes the hand's joint angles, global translation, and global rotation (as well as possible hand shape parameters, totaling 61 degrees of freedom), By inputting a pre-defined MANO layer, the corresponding 3D mesh of the hand can be generated. .

[0083] Step S411: Based on the improved ContactNet network constructed according to the preset objective function, each candidate grasping gesture is evaluated to obtain multiple grasping loss parameters corresponding to each candidate grasping gesture. These grasping loss parameters include at least one of the following: contact matching error parameter, surface envelope penalty parameter, grasping penetration loss parameter, and self-collision loss. The objective function is determined based on the sub-functions corresponding to these multiple grasping loss parameters. ContactNet is used to evaluate and optimize the rationality of hand-object contact during the grasping generation stage. The improved ContactNet objective function integrates contact matching degree, surface envelope degree, grasping penetration, and self-collision degree to encourage fingers to correctly touch objects and avoid penetrating objects or themselves. Let the current grasping gesture be H, the object point cloud be P0, and the expected contact distribution predicted by ContactNet be c*(p), which is the target contact score of the grasping target surface point cloud. Simultaneously, the actual contact distribution c is calculated from the current hand pose geometry. H (p). The objective function includes the following terms:

[0084] Contact matching error parameter L contact Encourage finger touch on designated object areas. Compare the expected contact probability at each point on the target with the current actual contact probability, using mean squared error to measure the difference. The calculation formula is: Where c*(p) is typically obtained by applying a nonlinear mapping to the distance from the object point to the dexterous hand. The distance d is normalized to [0,1] as the contact score; L contact Punish hand gestures that deviate from the desired contact pattern.

[0085] Capture the penetration loss parameter L penetration The constraint prevents the hand from penetrating the interior of the object. The penetration depth at a point on the object is calculated using the hand's signature distance field (SDF): For each point p on the object surface, the signed distance SDF from it to the hand surface is calculated. H (p), positive outside the hand surface and negative inside. The negative distance values ​​of all points falling inside the hand are summed and squared to obtain the gripping penetration loss, calculated using the following formula: This factor increases when the hand and the object intersect, prompting the finger to move out of the object during optimization.

[0086] Self-collision loss L self To prevent fingers or palms from intersecting, different joint connection surfaces in the hand model can be selected, and the minimum distance between them can be calculated. Similarly, distance detection or geometric detection methods can be used. If some finger-palm components intrude into each other, a penalty is applied. This can be defined as follows: Iterate through pairs of links i, j in the hand that should not touch each other, accumulating the squares of their negative distances. In the implementation, simple geometry is used to approximate the various parts of the hand, and the energy L is approximated by calculating the penetration depth between each part. self Ensure that the fingers do not penetrate themselves when adjusting the gripping posture.

[0087] Other stability constraints L others In desktop scraping scenarios, it also includes a desktop plane penetration penalty L. plane This ensures the hand does not penetrate the tabletop during the grasping process. It's similar to an object penetrating the surface: if key points of the hand, such as the fingertips or the bottom of the palm, are below the height of the tabletop, a penalty is imposed on the square of the distance exceeding that. Furthermore, to ensure grasping stability, optimization objectives can include force closure indicators and attraction terms to encourage fingertips to stay close to the object surface; and joint limit regularization L can be added. joint Avoid exceeding reasonable limits.

[0088] Step S412: Normalize and weight all the grasping loss parameters corresponding to each candidate grasping gesture in sequence to obtain the total grasping loss value E corresponding to each candidate grasping gesture. grasp The calculation formula is as follows: in, L is the contact matching error weight. wrap For surface envelope penalty parameters, As the surface envelope penalty weight, To capture the weight of penetration loss, For self-collision loss weights, These are the weights for other stability constraints; the weights mentioned above are used to balance the impact of different metrics on the optimization. If other stability constraints are not considered, they are not included in the calculation. .

[0089] Step S413: Select the candidate grasping gesture with the smallest total grasping loss value from all candidate grasping gestures in the candidate grasping gesture set to obtain the target grasping gesture.

[0090] It should be noted that the steps shown in the above flowchart or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here. For example, steps S401 and S402, steps S403 and S404, and steps S406 and S407.

[0091] This embodiment also provides a dexterous hand adaptive grasping device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0092] Figure 5 This is a structural block diagram of a dexterous hand adaptive grasping device according to an embodiment of this application, such as... Figure 5 As shown, the device includes: an acquisition module 51, a data processing module 52, a generation module 53, and a stability evaluation module 54.

[0093] The acquisition module 51 is used to acquire multimodal perception data, which includes: point cloud data corresponding to the grasping target obtained from the RGB-D image corresponding to the grasping action, and tactile patch information of the dexterous hand on the grasping target.

[0094] The data processing module 52, coupled to the acquisition module 51, is used to process the point cloud data and generate the global feature vector of the point cloud corresponding to the grasping target, and then map the tactile patch information to the dexterous hand workspace to generate simulated tactile signals.

[0095] The generation module 53, coupled to the data processing module 52, is used to process the global feature vector of the point cloud and simulated tactile signals using the Glow-CVAE conditional generation model to generate a candidate grasping gesture set including joint pose and pose parameters. Each candidate grasping gesture in the candidate grasping gesture set corresponds to a joint pose and a pose parameter. The Glow-CVAE conditional generation model is trained and generated based on the point cloud data of the preset training template image and the real grasping gesture information corresponding to the preset training template image.

[0096] The stability evaluation module 54, coupled to the generation module 53, is used to determine the stability of each candidate grasping gesture, and select the target grasping gesture from all candidate grasping gestures in the candidate grasping gesture set based on the stability, and grasp the target with the target grasping gesture. The stability is determined based on the contact matching degree, surface envelope degree, grasping penetration and self-collision degree between the dexterous hand and the grasping target.

[0097] The dexterous hand adaptive grasping device provided in this application embodiment acquires multimodal perception data, which includes: point cloud data corresponding to the grasping target obtained from the RGB-D image corresponding to the grasping action, and tactile patch information of the dexterous hand on the grasping target; after processing the point cloud data to generate a global feature vector of the point cloud corresponding to the grasping target, the tactile patch information is mapped to the dexterous hand workspace to generate simulated tactile signals; using the Glow-CVAE conditional generation model, the global feature vector of the point cloud and the simulated tactile signals are processed to generate candidate grasping hands including joint posture and pose parameters. The potential set is defined as follows: each candidate grasping gesture in the candidate grasping gesture set corresponds to a joint pose and a pose parameter. The Glow-CVAE conditional generation model is trained and generated based on the point cloud data of the preset training template image and the real grasping gesture information corresponding to the preset training template image. The stability corresponding to each candidate grasping gesture is determined, and based on the stability, a target grasping gesture is selected from all candidate grasping gestures in the candidate grasping gesture set. The target grasping gesture is then used to grasp the target. The stability is determined based on the contact matching degree, surface envelope degree, grasping penetration and self-collision degree between the dexterous hand and the grasping target. This solves the problems of limited adaptability to irregularly shaped and flexible objects and low grasping stability in related technologies, and achieves the beneficial effect of high adaptability and stability in grasping diverse objects in complex scenes.

[0098] In some embodiments, the generation module 53 further includes:

[0099] The first fusion unit is used to generate a first joint feature by concatenating the global feature vector of the point cloud with the simulated tactile signal using the CVAE generation network.

[0100] The first encoding unit, coupled to the first fusion unit, is used to infer the first joint features using the variational autoencoder of the Glow-CVAE conditional generation model to generate posterior distribution parameters, wherein the posterior distribution parameters include the distribution parameter mean vector and the distribution parameter variance vector.

[0101] The first decoding unit, coupled to the first encoding unit, is used to perform reparameter sampling on the mean vector and variance vector of the distribution parameters to generate latent variables. After concatenating the latent variables and the global feature vector of the point cloud to generate a joint latent feature vector, the decoder of the Clow-CVA conditional generation model is used to decode the joint latent feature vector to generate a candidate grasping gesture set.

[0102] In some embodiments, the first fusion unit further includes:

[0103] The feature extraction component uses the PointNet network to process the global feature vector of the point cloud into the object feature vector corresponding to the grasping target, and uses the PointNet network to extract the current pose feature vector corresponding to the dexterity from the simulated tactile signal.

[0104] The joint component, coupled with the feature extraction component, is used to concatenate the object feature vector and the current pose feature vector to generate a multi-dimensional vector. The target fully connected layer of the CVAE generation network is then used to extract joint distribution information from the multi-dimensional vector to obtain the first joint feature.

[0105] In some embodiments, the data processing module 52 is used to perform coordinate normalization processing on the point cloud data to generate normalized point cloud data, and then perform normalization processing on the normalized point cloud data to generate normalized point cloud coordinates located within a preset unit sphere. Here, a normalized point cloud coordinate represents the three-dimensional coordinates corresponding to a point cloud data. The normalization processing includes the following processes: translation alignment and coordinate scale standardization; using a multilayer perceptron corresponding to the PointNet network, features are extracted from all normalized point cloud coordinates to obtain the corresponding global feature vector of the point cloud.

[0106] In some embodiments, the data processing module 52 is used to map tactile patch information to the dexterous hand workspace based on a Gaussian kernel function to obtain simulated tactile signals.

[0107] In some embodiments, the stability evaluation module 54 further includes:

[0108] The first computational unit is used to evaluate each candidate grasping gesture based on the improved ContactNet network constructed according to a preset objective function, and obtain multiple grasping loss parameters corresponding to each candidate grasping gesture. The grasping loss parameters include at least one of the following: contact matching error parameter, surface envelope penalty parameter, grasping penetration loss parameter, and self-collision loss. The objective function is determined based on the sub-functions corresponding to the multiple grasping loss parameters.

[0109] The first aggregation unit, coupled to the first calculation unit, is used to normalize and weight all the grasping loss parameters corresponding to each candidate grasping gesture in sequence to obtain the total grasping loss value corresponding to each candidate grasping gesture, wherein stability includes the total grasping loss value.

[0110] The first selection unit, coupled to the first aggregation unit, is used to select the candidate grasping gesture with the smallest total grasping loss value from all candidate grasping gestures in the candidate grasping gesture set, and obtain the target grasping gesture.

[0111] In some embodiments, the first computing unit is used to predict the expected contact distribution probability corresponding to each point in the point cloud data based on the point cloud data corresponding to the grasping target, using an improved ContactNet network; calculate the current contact distribution probability corresponding to each point in the point cloud data based on the point cloud data and candidate grasping gestures; and perform mean square error calculation based on the expected contact distribution probability and the current contact distribution probability corresponding to all points in the point cloud data to obtain the contact matching degree error corresponding to each candidate grasping gesture, wherein the contact matching degree error parameter includes the contact matching degree error.

[0112] In some embodiments, the first computing unit is used to predict the optimized grasping gesture corresponding to each candidate grasping gesture using an improved ContactNet network; calculate the symbolic distance between each point in the point cloud data and the corresponding optimized grasping gesture; and determine the signature distance field between the optimized grasping gesture and the grasping target based on the symbolic distances of all points in the point cloud data, thereby obtaining the grasping penetration loss parameter corresponding to the candidate grasping gesture. Specifically, multiple target joint connection surfaces are selected from the predicted grasping gestures, and the signature distance field between two target joint connection surfaces is calculated, wherein the target joint connection surfaces include at least one of the following: finger connection surface and palm component connection surface; the signature distance field with the smallest signature distance field is selected from all signature distance fields to obtain the self-collision loss corresponding to the candidate grasping gesture.

[0113] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0114] This embodiment also provides a dexterous hand control system, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0115] Optionally, the dexterous hand may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0116] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0117] S1, acquire multimodal perception data, which includes: point cloud data corresponding to the grasping target obtained from the RGB-D image corresponding to the grasping action, and tactile patch information of the dexterous hand on the grasping target.

[0118] S2, after processing the point cloud data and generating the global feature vector of the point cloud corresponding to the grasping target, maps the tactile patch information to the dexterous hand workspace to generate simulated tactile signals.

[0119] S3 utilizes the Glow-CVAE conditional generation model to process the global feature vector of the point cloud and simulated tactile signals to generate a candidate grasping gesture set including joint pose and pose parameters. Each candidate grasping gesture in the candidate grasping gesture set corresponds to a joint pose and a pose parameter. The Glow-CVAE conditional generation model is trained and generated based on the point cloud data of the preset training template image and the real grasping gesture information corresponding to the preset training template image.

[0120] S4. Determine the stability corresponding to each candidate grasping gesture, and select the target grasping gesture from all candidate grasping gestures in the candidate grasping gesture set according to the stability, and grasp the target with the target grasping gesture. The stability is determined based on the contact matching degree, surface envelope degree, grasping penetration and self-collision degree between the dexterous hand and the grasping target.

[0121] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0122] Furthermore, in conjunction with the dexterous hand adaptive grasping method in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the dexterous hand adaptive grasping methods in the above embodiments.

[0123] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A dexterous hand adaptive grasping method, characterized in that, include: Acquire multimodal perception data, wherein the multimodal perception data includes: point cloud data corresponding to the grasping target obtained from the RGB-D image corresponding to the grasping action, and tactile patch information of the dexterous hand on the grasping target; After processing the point cloud data to generate the global feature vector of the point cloud corresponding to the grasping target, the tactile patch information is mapped to the dexterous hand workspace to generate simulated tactile signals. The Glow-CVAE conditional generation model is used to process the global feature vector of the point cloud and the simulated tactile signal to generate a candidate grasping gesture set including joint pose and pose parameter; wherein, each candidate grasping gesture in the candidate grasping gesture set corresponds to a joint pose and a pose parameter, and the Glow-CVAE conditional generation model is trained and generated based on the point cloud data of the preset training template image and the real grasping gesture information corresponding to the preset training template image. The stability corresponding to each candidate grasping gesture is determined, and a target grasping gesture is selected from all the candidate grasping gestures in the candidate grasping gesture set according to the stability, and the target grasping gesture is used to grasp the target. The stability is determined based on the contact matching degree, surface envelope degree, grasping penetration and self-collision degree between the dexterous hand and the grasping target.

2. The dexterous hand adaptive grasping method according to claim 1, characterized in that, Using the Glow-CVAE conditional generation model, the global feature vector of the point cloud and simulated tactile signals are processed to generate a candidate set of grasping gestures, including joint pose and positional parameters, comprising: Using a CVAE generator network, the global feature vector of the point cloud and the simulated tactile signal are concatenated to generate a first joint feature; Using the variational autoencoder of the Glow-CVAE conditional generation model, the first joint feature is inferred to generate posterior distribution parameters, wherein the posterior distribution parameters include a distribution parameter mean vector and a distribution parameter variance vector. Reparameter sampling is performed on the mean vector and variance vector of the distribution parameters to generate latent variables; After concatenating the latent variables and the global feature vector of the point cloud to generate a joint latent feature vector, the decoder of the Clow-CVA conditional generation model is used to decode the joint latent feature vector to generate the candidate grasping gesture set.

3. The dexterous hand adaptive grasping method according to claim 2, characterized in that, Using a CVAE (Continuous Transformation and Image Processing) network, the global feature vector of the point cloud is concatenated with the simulated tactile signal to generate a first joint feature, including: Using the PointNet network, the global feature vector of the point cloud is processed into the object feature vector corresponding to the grasping target, and the PointNet network is used to extract the current posture feature vector corresponding to the dexterous hand from the simulated tactile signal; The object feature vector and the current pose feature vector are concatenated to generate a multidimensional vector. The joint distribution information is then extracted from the multidimensional vector through the target fully connected layer of the CVAE generation network to obtain the first joint feature.

4. The dexterous hand adaptive grasping method according to claim 1, characterized in that, The point cloud data is processed to generate a global feature vector of the point cloud corresponding to the target being captured, including: After performing coordinate normalization processing on the point cloud data to generate normalized point cloud data, the normalized point cloud data is then normalized to generate normalized point cloud coordinates located within a preset unit sphere. Here, a normalized point cloud coordinate represents the three-dimensional coordinates corresponding to a point cloud data. The normalization processing includes the following processes: translation alignment and coordinate scale standardization. Using a multilayer perceptron corresponding to the PointNet network, features are extracted from all the normalized point cloud coordinates to obtain the corresponding global feature vector of the point cloud.

5. The dexterous hand adaptive grasping method according to claim 1, characterized in that, Mapping the tactile patch information onto the dexterous hand workspace to generate simulated tactile signals includes: The tactile patch information is mapped to the dexterous hand workspace based on the Gaussian kernel function to obtain the simulated tactile signal.

6. The dexterous hand adaptive grasping method according to claim 1, characterized in that, Determine the stability corresponding to each candidate grasping gesture, and select a target grasping gesture from all candidate grasping gestures in the candidate grasping gesture set based on the stability, including: Based on the improved ContactNet network constructed according to the preset objective function, each candidate grasping gesture is evaluated to obtain multiple grasping loss parameters corresponding to each candidate grasping gesture. The grasping loss parameters include at least one of the following: contact matching error parameter, surface envelope penalty parameter, grasping penetration loss parameter, and self-collision loss. The objective function is determined based on the sub-functions corresponding to the multiple grasping loss parameters. The grasping loss parameters corresponding to each candidate grasping gesture are normalized and weighted sequentially to obtain the total grasping loss value corresponding to each candidate grasping gesture, wherein the stability includes the total grasping loss value. From all the candidate grasping gestures in the candidate grasping gesture set, select the candidate grasping gesture with the smallest total grasping loss value to obtain the target grasping gesture.

7. The dexterous hand adaptive grasping method according to claim 6, characterized in that, The improved ContactNet network is used to evaluate each candidate grab gesture, including: Based on the point cloud data corresponding to the target being captured, the improved ContactNet network is used to predict the expected contact distribution probability corresponding to each point in the point cloud data. Based on the point cloud data and the candidate grasping gestures, calculate the current contact distribution probability corresponding to each point in the point cloud data; Based on the expected contact distribution probability and the current contact distribution probability corresponding to all points in the point cloud data, a mean square error is calculated to obtain the contact matching degree error corresponding to each candidate grasping gesture, wherein the contact matching degree error parameter includes the contact matching degree error.

8. The dexterous hand adaptive grasping method according to claim 6, characterized in that, The improved ContactNet network is used to evaluate each candidate grab gesture, including: Using the improved ContactNet network, predict the optimized grasp gesture corresponding to each of the candidate grasp gestures; Calculate the symbol distance between each point in the point cloud data and the corresponding optimized grasping gesture, and based on the symbol distances corresponding to all points in the point cloud data, determine the signature distance field between the optimized grasping gesture and the grasping target, and obtain the grasping penetration loss parameter corresponding to the candidate grasping gesture.

9. A dexterous hand control system, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the dexterous hand adaptive grasping method according to any one of claims 1 to 8.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the dexterous hand adaptive grasping method as described in any one of claims 1 to 8.

Images