Adaptive grasping method for dexterous hands based on multimodal fusion imitation learning

Abstract

The present invention relates to an adaptive grasping method for dexterous hands based on multimodal fusion imitation learning, comprising: executing multimodal data collection and constructing a multimodal data set; wherein the multimodal data set includes: visual images, finger joint torques and finger joint angles corresponding to the grasping actions of the dexterous hands; constructing an adaptive grasping model for controlling the dexterous hands based on the multimodal data set; for a grasping target, the adaptive grasping model outputs action instructions for the dexterous hands to execute in order to adaptively grasp the grasping target. The adaptive grasping method for dexterous hands proposed by the present invention is based on three modal data of vision, finger joint angles and finger joint torques, and has a higher success rate and stronger adaptability to object types than existing dexterous hand grasping algorithms.

Description

Technical Field

[0001] The present invention relates to the field of dexterous hand control design, and in particular to a dexterous hand adaptive grasping method based on multi-modal fusion imitation learning. Background Art

[0002] Humanoid dexterous hands have extremely high dexterity and humanoid structural features, and are able to perform delicate manipulations such as tool handling and various hand manipulation tasks. Among all the contact-rich tasks of dexterous hands, reliable and general grasping tasks remain unsolved, especially when deformable objects are taken into account. Traditional methods encounter many problems such as predicting contact locations on unstructured surfaces, nonlinear interaction modeling, and finger grasping, which are challenging to obtain deterministic solutions. Therefore, reinforcement learning and imitation learning methods have been widely used in this field.

[0003] Reinforcement learning requires carefully designed reward functions and extensive interactions with the environment to improve performance. Imitation learning requires collecting a large amount of human demonstration data and using this data to train the model. Among these methods, physical simulation engines are widely used. However, it is very difficult to simultaneously simulate dexterous hands, deformable objects, and nonlinear interactions between them in a simulation environment. When the strategy that performs well in a simulation environment is transplanted to a real robotic system, the difference between the simulation environment and the real environment often leads to a significant drop in performance. At the same time, some previous work (aloha, umi) has verified the feasibility of directly collecting demonstration data and training robots to replicate human operations in the real world.

[0004] Referring to the information processing mechanism of human grasping behavior, humans use their eyes to observe the position, shape, and size of an object before grasping it, and estimate its type and hardness based on existing knowledge. During the grasping process, humans adjust their gestures based on deformation and tactile feedback. Similar to human multimodal perception and control strategies, dexterous hands also need to integrate heterogeneous information from multiple sources during operation to achieve adaptive control. However, the effective fusion of heterogeneous modal data still faces huge challenges. Summary of the invention

[0005] The purpose of the present invention is to provide a dexterous hand adaptive grasping method based on multimodal fusion imitation learning.

[0006] To achieve the above-mentioned object of the invention, the present invention provides a dexterous hand adaptive grasping method based on multimodal fusion imitation learning, comprising the following steps:

[0007] S1. Execute multimodal data collection and construct a multimodal data set; wherein the multimodal data set includes: visual images, finger joint torques and finger joint angles corresponding to the grasping action of the dexterous hand;

[0008] S2. Constructing an adaptive grasping model for controlling a dexterous hand based on the multimodal dataset; comprising:

[0009] S21. Performing feature extraction on the multimodal data set to obtain multiple groups of feature vectors, wherein the multiple groups of feature vectors include: a first feature vector corresponding to the visual image, a second feature vector corresponding to the finger joint torque, and a third feature vector corresponding to the finger joint angle;

[0010] S22. Perform multimodal fusion on the plurality of groups of feature vectors belonging to the continuous state to obtain multimodal features of the continuous state;

[0011] S23. Inputting the multimodal features of the continuous state into a multilayer perceptron to obtain an adaptive grasping action output of the dexterous hand, and a feature extraction neural network for extracting features from the multimodal data set, and a multimodal fusion network for performing multimodal fusion and the multilayer perceptron to form an adaptive grasping model;

[0012] S3. With respect to a grasping target, the adaptive grasping model outputs action instructions for the dexterous hand to execute so as to adaptively grasp the grasping target.

[0013] According to one aspect of the present invention, in step S1, in the step of performing multimodal data collection and constructing a multimodal data set, the finger joint torques include: little finger flexion torque, ring finger flexion torque, middle finger flexion torque, index finger flexion torque, thumb flexion torque and thumb swing torque;

[0014] The finger joint angles include: a little finger bending angle, a ring finger bending angle, a middle finger bending angle, an index finger bending angle, a thumb bending angle and a thumb swing angle.

[0015] According to one aspect of the present invention, in step S21, in the step of extracting features from the multimodal data set to obtain multiple sets of feature vectors, the feature vectors are D -dimensional feature vector.

[0016] According to one aspect of the present invention, in step S21, the step of extracting features from the multimodal data set to obtain multiple groups of feature vectors includes:

[0017] S211. Acquire the visual images at consecutive moments and perform time position coding on the visual images, and perform feature extraction based on the time position coded visual images to obtain the first feature vector;

[0018] S212. Acquire the finger joint torques at consecutive moments, normalize the spatial relationship of the finger joint torques by using one-hot encoding, perform temporal position encoding on the normalized finger joint torques, and perform feature extraction on the spatially normalized and temporally position encoded finger joint torques based on a double-layer multilayer perceptron to obtain the second feature vector;

[0019] S213. Obtain the finger joint angles at continuous moments, normalize the spatial relationship of the finger joint angles using one-hot encoding, perform temporal position encoding on the normalized finger joint angles, and perform feature extraction on the finger joint angles after spatial relationship normalization and temporal position encoding based on a double-layer multilayer perceptron to obtain the third feature vector.

[0020] According to one aspect of the present invention, in step S211, in the step of temporally encoding the visual image, the visual image is temporally encoded based on the acquisition time of the visual image, and the visual image is positionally encoded based on sine and cosine position encoding;

[0021] In step S212, in the step of performing time and position coding on the normalized finger joint torque, the finger joint torque is time coded based on the acquisition time of the finger joint torque, and the finger joint torque is position coded based on sine and cosine position coding;

[0022] In step S213, in the step of performing time and position encoding on the normalized finger joint angles, the finger joint angles are time-encoded based on the acquisition time of the finger joint angles, and the finger joint angles are position-encoded based on sine and cosine position encoding methods.

[0023] According to one aspect of the present invention, in step S22, in the step of performing multimodal fusion on multiple groups of feature vectors belonging to continuous states to obtain multimodal features, the modal fusion network used for multimodal fusion is constructed using a multi-head attention mechanism to generate multimodal features with continuous states from multiple groups of feature vectors under the control of the multi-head attention mechanism.

[0024] According to one aspect of the present invention, the multi-head attention mechanism is expressed as:

[0025] ;

[0026] ;

[0027] ;

[0028] in, represents the query of the attention head, represents the key of the attention head, represents the value of the attention head, represents a concatenation matrix of the first eigenvector, the second eigenvector and the third eigenvector, , represents the number of attention heads, is the number of the attention head; represents the linear transformation matrix corresponding to the query, key, and value of a single attention head in the attention mechanism, and Represent the number and dimension of input features respectively, is a parameter matrix whose parameters are iteratively updated during the training of the adaptive grasping model.

[0029] According to one aspect of the present invention, the multimodal features of the continuous state are used to obtain the output of the adaptive grasping action of the dexterous hand through a multilayer perceptron, and a feature extraction neural network is used to extract features from the multimodal data set. In the step of forming an adaptive grasping model using a multimodal fusion network and the multilayer perceptron for multimodal fusion, the generation of the adaptive grasping model is guided based on a loss function, wherein the loss function is expressed as:

[0030] ;

[0031] ;

[0032] in, represents the angle loss function, represents the torque loss function, Indicates the finger joint angle, represents the predicted finger joint angle, represents the finger joint torque, represents the predicted finger joint torque, Indicates the adjustment threshold of the finger joint angle, Indicates the adjustment threshold of the finger joint torque.

[0033] According to one scheme of the present invention, the adaptive grasping algorithm of the dexterous hand proposed in the present invention is based on three modal data: vision, finger joint angle, and finger joint torque. Compared with the existing dexterous hand grasping algorithm, it has a higher success rate and stronger adaptability to object types, providing important guidance for the subsequent research and development of dexterous hand grasping algorithms.

[0034] According to one solution of the present invention, the adaptive grasping algorithm of the dexterous hand of the present invention refers to human grasping experience and is based on the fusion of collected multimodal data, thereby effectively realizing accurate grasping of different objects, especially for softer objects, which can minimize the deformation of the object while maintaining stable and reliable grasping of the object.

[0035] According to one solution of the present invention, the present invention can effectively capture the complex relationship between visual images, finger joint angles and torque information by adopting the fusion of multimodal data, enhance the representation ability of complex tasks, and through the multi-head attention mechanism, realize the compensation of one modality for another different modality at the same time step, so as to effectively ensure the accuracy and reliability of object grasping. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a diagram schematically showing the steps of a method for adaptive grasping of a dexterous hand according to an embodiment of the present invention;

[0037] Figure 2 is a diagram schematically showing a dexterous hand model according to an embodiment of the present invention;

[0038] Figure 3 It is a diagram schematically showing a multi-head self-attention mechanism network model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] When describing the embodiments of the present invention, the orientation or positional relationship expressed by the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or positional relationship shown in the relevant drawings and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the above terms should not be understood as limiting the present invention.

[0041] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments cannot be described one by one here, but the embodiments of the present invention are not therefore limited to the following embodiments.

[0042] Combination Figure 1 and Figure 2As shown, according to one embodiment of the present invention, a dexterous hand adaptive grasping method based on multimodal fusion imitation learning of the present invention may involve three key sensory modalities in the grasping process: visual image , finger joint angle and finger joint torque The corresponding dexterous hand grasping action instruction is (i.e., the desired finger joint angle), thus, the adaptive grasping process of the dexterous hand of the present invention can be described as a mapping from state variables to action instructions: Furthermore, the present invention specifically comprises the following steps:

[0043] S1. Execute multimodal data collection and construct a multimodal data set; wherein the multimodal data set includes: visual images, finger joint torques, and finger joint angles corresponding to the grasping action of the dexterous hand;

[0044] S2. Build an adaptive grasping model for controlling dexterous hands based on multimodal datasets;

[0045] S3. With respect to the grasping target, the adaptive grasping model outputs action instructions for the dexterous hand to execute in order to adaptively grasp the target.

[0046] Combination Figure 1 and Figure 2 As shown, according to an embodiment of the present invention, in step S1, in the step of performing the collection of multimodal data, an immersive grasping demonstration system (Immersive Grasping Demonstration System) is combined with an RGBD camera as a device for multimodal data collection. In this embodiment, the immersive grasping demonstration system includes: an exoskeleton data glove and an anthropomorphic dexterous hand. Specifically, the exoskeleton data glove uses a high-precision sensor to capture the operator's hand movements in real time, and converts these data into precise action instructions for the anthropomorphic dexterous hand. The anthropomorphic dexterous hand transmits joint torque to the exoskeleton data glove, and adjusts the resistance feedback to the bending of the operator's fingers according to the torque. During the interaction between the anthropomorphic dexterous hand and the exoskeleton data glove, the anthropomorphic dexterous hand is imaged based on the RGBD camera to obtain the corresponding visual image; specifically, the RGBD camera can use the Intel RealSense D435i camera to obtain the corresponding visual image with a resolution of 640×480 and a frequency of 30Hz.

[0047] According to an embodiment of the present invention, in step S1, in the step of collecting multimodal data and constructing a multimodal data set, the finger joint torque includes: little finger bending torque, ring finger bending torque, middle finger bending torque, index finger bending torque, thumb bending torque and thumb swing torque; specifically, the finger joint torque can be expressed as:

[0048] ;

[0049] in, They represent the little finger flexion torque, ring finger flexion torque, middle finger flexion torque, index finger flexion torque, thumb flexion torque and thumb swing torque respectively.

[0050] In this embodiment, the finger joint angles include: the little finger bending angle, the ring finger bending angle, the middle finger bending angle, the index finger bending angle, the thumb bending angle and the thumb swing angle; specifically, the finger joint angles can be expressed as:

[0051] ;

[0052] in, They respectively represent the little finger bending angle, ring finger bending angle, middle finger bending angle, index finger bending angle, thumb bending angle and thumb swing angle.

[0053] Combination Figure 1 , Figure 2 and Figure 3 As shown, according to one embodiment of the present invention, in step S2, the step of constructing an adaptive grasping model for controlling a dexterous hand based on a multimodal data set includes:

[0054] S21. Perform feature extraction on the multimodal data set to obtain multiple sets of feature vectors, wherein the multiple sets of feature vectors include: a first feature vector corresponding to the visual image, a second feature vector corresponding to the finger joint torque, and a third feature vector corresponding to the finger joint angle; in this embodiment, the feature vector is D In this embodiment, the step of extracting features from a multimodal data set to obtain multiple sets of feature vectors includes:

[0055] S211. Obtain visual images at consecutive moments and perform temporal position encoding on the visual images, perform feature extraction based on the temporal position encoded visual images to obtain a first feature vector; in this embodiment, in the step of temporal position encoding on the visual images, the visual images are temporally encoded based on the acquisition time of the visual images, and the visual images are position encoded based on sine and cosine position encoding methods; in this embodiment, in the step of feature extraction based on the temporal position encoded visual images, ResNet-18 network is used to extract features from the visual images to obtain a first feature vector, which is expressed as: .

[0056] S212. Obtain the finger joint torques at continuous moments, use one-hot encoding to normalize the spatial relationship of the finger joint torques, and perform time position encoding on the normalized finger joint torques, and extract features of the finger joint torques after the spatial relationship is normalized and time position encoded based on a double-layer multilayer perceptron to obtain a second feature vector; in this embodiment, since there is a fixed order between the fingers, one-hot encoding is used to normalize the spatial relationship between the states of the finger joint torques corresponding to the fingers with a fixed order. Further, in the step of performing time position encoding on the normalized finger joint torques, the acquisition time of the finger joint torques is used to perform time encoding on the finger joint torques, and the finger joint torques are position encoded based on sine and cosine position encoding; further, the finger joint torques after the spatial relationship is normalized and time position encoded are feature extracted based on a double-layer multilayer perceptron (MLP) to obtain a second feature vector, which is expressed as: .

[0057] S213. Obtain the finger joint angles at continuous moments, use one-hot encoding to normalize the spatial relationship of the finger joint angles, and perform temporal position encoding on the normalized finger joint angles, and extract features of the finger joint angles after the spatial relationship is normalized and temporal position encoding based on a double-layer multi-layer perceptron to obtain a third feature vector; in this embodiment, since there is a fixed order between the fingers, one-hot encoding is used to normalize the spatial relationship between the states of the finger joint angles corresponding to the fingers with a fixed order. Further, in the step of temporal position encoding of the normalized finger joint angles, the acquisition time of the finger joint angles is used to temporally encode the finger joint angles, and the finger joint angles are position-encoded based on sine and cosine position encoding; further, the finger joint torque after the spatial relationship is normalized and temporal position encoding is extracted based on a double-layer multi-layer perceptron (MLP) to obtain a third feature vector, which is expressed as: .

[0058] S22. Perform multimodal fusion on multiple groups of feature vectors belonging to the continuous state to obtain multimodal features of the continuous state; in this embodiment, the first feature vector obtained in the above step , the second eigenvector , the third eigenvector Splicing to form a unified matrix , which can be specifically expressed as . The matrix Input to the modal fusion network to achieve the corresponding fusion effect. The modal fusion network is composed of a multi-head attention mechanism. For this purpose, for the matrixF The multi-head attention mechanism can be used to obtain the multi-modal features of continuous states. The query, key, and value vectors of a single attention head in the multi-head self-attention mechanism are initialized as Q = K = V = F . Further, a single attention head is calculated head i The output of H The outputs of the attention heads are connected to obtain enhanced multimodal features. M . By using the self-attention mechanism to associate different positions of a single sequence to calculate the attention mechanism of the representation of the same sequence, the multi-head attention mechanism can capture and extract more features from multiple dimensions, obtain different Self-Attention Scores from multiple "heads", and improve the model performance. To this end, in this embodiment, the multi-head attention mechanism is expressed as:

[0059] ;

[0060] ;

[0061] ;

[0062] in, represents the query of the attention head, The key representing the attention head, represents the value of the attention head, represents the concatenation matrix of the first eigenvector, the second eigenvector, and the third eigenvector, , represents the number of attention heads, is the number of the attention head; Represents the linear transformation matrix corresponding to the query, key, and value of a single attention head in the attention mechanism, and Represent the number and dimension of input features respectively, is a parameter matrix whose parameters are iteratively updated during the training of the adaptive grasping model.

[0063] S23. Based on the multimodal features of the continuous state, the multi-layer perceptron is input to obtain the adaptive grasping action output of the dexterous hand, and the feature extraction neural network for feature extraction of the multimodal data set, and the multimodal fusion network and the multi-layer perceptron for multimodal fusion constitute an adaptive grasping model; in this embodiment, the adaptive grasping action output is obtained based on the multimodal features of the continuous state and the multi-layer perceptron is used; the enhanced multimodal features M Input to the multi-layer perceptron MLP to obtain the predicted finger joint angles of the dexterous hand at the next moment and finger joint torque Adaptive grasping action; thus, based on the aforementioned series of all neural networks and their parameters to achieve adaptive grasping of dexterous hands constitute an adaptive grasping model.

[0064] According to one embodiment of the present invention, the multimodal features of the continuous state are used to obtain the output of the adaptive grasping action of the dexterous hand through a multilayer perceptron, and a feature extraction neural network is used to extract features of the multimodal data set. In the step of forming an adaptive grasping model using a multimodal fusion network and a multilayer perceptron for multimodal fusion, the generation of the adaptive grasping model is guided based on a loss function, wherein the loss function is expressed as:

[0065] ;

[0066] ;

[0067] in, represents the angle loss function, represents the torque loss function, Indicates the finger joint angle, represents the predicted finger joint angle, represents the finger joint torque, represents the predicted finger joint torque, Indicates the adjustment threshold of the finger joint angle, Indicates the adjustment threshold of the finger joint torque.

[0068] The above contents are merely examples of specific solutions of the present invention. For devices and structures not described in detail therein, it should be understood that they can be implemented by adopting general devices and general methods available in the art.

[0069] The above is only one solution of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. Adaptive grasping method for dexterous hands based on multimodal fusion imitation learning, It is characterized in that The following steps are involved: S1. Execute multimodal data collection and construct a multimodal data set; wherein the multimodal data set includes: visual images, finger joint torques and finger joint angles corresponding to the grasping action of the dexterous hand; S2. Constructing an adaptive grasping model for controlling a dexterous hand based on the multimodal dataset; comprising: S21. Perform feature extraction on the multimodal data set to obtain multiple groups of feature vectors, wherein the multiple groups of feature vectors include: a first feature vector corresponding to the visual image, a second feature vector corresponding to the finger joint torque, and a third feature vector corresponding to the finger joint angle; which includes: S211. Acquire the visual images at consecutive moments and perform time position coding on the visual images, and perform feature extraction based on the time position coded visual images to obtain the first feature vector; S212. Acquire the finger joint torques at consecutive moments, normalize the spatial relationship of the finger joint torques by using one-hot encoding, perform temporal position encoding on the normalized finger joint torques, and perform feature extraction on the spatially normalized and temporally position encoded finger joint torques based on a double-layer multilayer perceptron to obtain the second feature vector; S213. Acquire the finger joint angles at consecutive moments, normalize the spatial relationship of the finger joint angles by using one-hot encoding, perform temporal position encoding on the normalized finger joint angles, and perform feature extraction on the spatially normalized and temporally position encoded finger joint angles based on a double-layer multilayer perceptron to obtain the third feature vector; S22. Perform multimodal fusion on the plurality of groups of feature vectors belonging to the continuous state to obtain multimodal features of the continuous state; S23. Inputting the multimodal features of the continuous state into a multilayer perceptron to obtain an adaptive grasping action output of the dexterous hand, and a feature extraction neural network for extracting features from the multimodal data set, and a multimodal fusion network for performing multimodal fusion and the multilayer perceptron to form an adaptive grasping model; S3. With respect to a grasping target, the adaptive grasping model outputs action instructions for the dexterous hand to execute so as to adaptively grasp the grasping target.

2. The adaptive grasping method of the dexterous hand according to claim 1, It is characterized in that In step S1, in the step of collecting multimodal data and constructing a multimodal data set, the finger joint torques include: little finger bending torque, ring finger bending torque, middle finger bending torque, index finger bending torque, thumb bending torque and thumb swing torque; The finger joint angles include: a little finger bending angle, a ring finger bending angle, a middle finger bending angle, an index finger bending angle, a thumb bending angle and a thumb swing angle.

3. The adaptive grasping method of the dexterous hand according to claim 2, It is characterized in that In step S21, in the step of extracting features from the multimodal data set to obtain multiple sets of feature vectors, the feature vectors are D -dimensional feature vector.

4. The adaptive grasping method of dexterous hands according to claim 1, 2 or 3, It is characterized in that In step S211, in the step of temporally encoding the visual image, temporally encoding the visual image is performed based on the acquisition time of the visual image, and positionally encoding the visual image is performed based on sine and cosine position encoding methods; In step S212, in the step of performing time and position coding on the normalized finger joint torque, the finger joint torque is time coded based on the acquisition time of the finger joint torque, and the finger joint torque is position coded based on sine and cosine position coding; In step S213, in the step of performing time and position encoding on the normalized finger joint angles, the finger joint angles are time-encoded based on the acquisition time of the finger joint angles, and the finger joint angles are position-encoded based on sine and cosine position encoding methods.

5. The adaptive grasping method of the dexterous hand according to claim 4, It is characterized in that In step S22, in the step of performing multimodal fusion on the multiple groups of feature vectors belonging to the continuous state to obtain multimodal features, the modal fusion network used for multimodal fusion is constructed with a multi-head attention mechanism to generate multimodal features with continuous states from the multiple groups of feature vectors belonging to the continuous state under the control of the multi-head attention mechanism.

6. The adaptive grasping method of the dexterous hand according to claim 5, It is characterized in that The multi-head attention mechanism is expressed as: in, Q n represents the query of the attention head, K n The key representing the attention head, V n represents the value of the attention head, F represents a concatenation matrix of the first eigenvector, the second eigenvector and the third eigenvector, , represents the number of attention heads, is the number of the attention head; represents the linear transformation matrix corresponding to the query, key, and value of a single attention head in the attention mechanism, and Respectively represent the number and dimension of input features, M represents the enhanced multimodal features, is a parameter matrix whose parameters are iteratively updated during the training of the adaptive grasping model.

7. The adaptive grasping method of the dexterous hand according to claim 6, It is characterized in that Based on the multimodal features of the continuous state, the output of the adaptive grasping action of the dexterous hand is obtained through a multilayer perceptron, and a feature extraction neural network is used to extract features from the multimodal data set. In the step of forming an adaptive grasping model with a multimodal fusion network and the multilayer perceptron for multimodal fusion, the generation of the adaptive grasping model is guided based on a loss function, wherein the loss function is expressed as: in, L A represents the angle loss function, L T represents the torque loss function, X A Indicates the finger joint angle, represents the predicted finger joint angle, X T represents the finger joint torque, represents the predicted finger joint torque, Indicates the adjustment threshold of the finger joint angle, Indicates the adjustment threshold of the finger joint torque.

Images