A method for classifying a grasping state based on visual-haptic learning
By introducing the Transformer model and optical flow method, the accuracy problem of grasping state recognition of deformable objects was solved, achieving higher multimodal recognition accuracy and real-time grasping adjustment, thus ensuring the stability and safety of the object.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING UNIV OF TECH
- Filing Date
- 2024-08-23
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies struggle to accurately identify the grasping state of deformable objects, especially when multimodal information is insufficient, leading to unstable grasping or object damage. Traditional methods, such as wavelet fuzzy entropy algorithms, suffer from insufficient dataset expansion on deformable objects and unclear tactile image features, affecting the recognition accuracy of multimodal models.
The Transformer model is introduced to process tactile modal information, and optical flow is combined to guide the model to learn grasping features. By constructing an optical flow dataset and a multimodal end-to-end model, the recognition accuracy of tactile images is improved.
By capturing subtle motions and dynamic changes using optical flow, the accuracy of multimodal data fusion is improved, enabling real-time feedback and adjustment of the grasping strategy, thus avoiding slippage and damage to deformable objects.
Smart Images
Figure CN119036445B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of robot visual-tactile perception, specifically involving a method for classifying grasping stable states based on end-to-end model learning. Background Technology
[0002] Intelligent grasping robots have wide applications in industry and service sectors. During operation, judging the gripping state of the robot arm's end effector is crucial. The stability of grasping deformable objects, in particular, is a current research hotspot and challenge. When performing grasping operations, not only the stability of the grasped object but also its safety must be considered to prevent damage. Insufficient gripping force or incorrect gripping position are causes of gripping instability, and these issues have been studied extensively. Detecting slippage can help the robot adjust its gripping force and strategy to avoid grasping failures.
[0003] Traditional mechanical grasping often struggles to accurately capture the state of objects, easily leading to deformation or damage. Previous work used wavelet fuzzy entropy algorithms to classify grasping states, but limitations such as missing information from the TS2520 array-type tactile sensor hindered the expansion of the dataset to deformable objects. This method combines multimodal information from a visual-tactile sensor and an external camera to extend the dataset to everyday deformable objects (e.g., bread), and utilizes an end-to-end multimodal model to learn and classify different grasping states. During actual grasping using the visual-tactile sensor, it was found that when the robotic arm's end effector grasps deformable objects with relatively small forces, the acquired tactile image features are not prominent, leading to a potential problem of insufficient information in the multimodal model's learning of tactile images. Although the model can integrate information from external camera visual images to compensate for the lack of tactile image information, the importance of tactile information in the grasping process, especially in capturing the subtle slippage tendency of deformable objects, cannot be replaced by visual information. Therefore, further improving the accuracy of tactile modality recognition becomes a crucial issue for enhancing multimodal performance. Summary of the Invention
[0004] To address the aforementioned issues, the present invention employs a method for classifying grasping states based on visual-tactile learning. This method improves the accuracy of tactile modality recognition in two aspects.
[0005] (1) The Transformer model is introduced to process tactile modal information. Compared with traditional convolutional neural networks (CNN) and long short-term memory networks (LSTM), the Transformer model has a stronger ability to capture long-term dependencies and can process spatial and temporal features at the same time.
[0006] (2) Optical flow was introduced to guide the model in learning different grasping features when processing tactile images. Optical flow is a technique for calculating pixel motion in an image sequence and is widely used in the field of computer vision. It estimates the movement of an object between two frames by analyzing the brightness changes of each pixel in the image. Therefore, when processing tactile images, optical flow can capture subtle movements and changes, thereby effectively identifying dynamic information such as object slippage.
[0007] The specific implementation process includes the following steps:
[0008] (1) Construct an optical flow dataset OFB-6 for deformable objects (this invention uses 6 different types of bread as objects) in three states: overstable, fully stable, and unstable. Here, "overstable" means that the object undergoes slight or obvious deformation while being held; "fully stable" means that the object does not have obvious deformation and can be stably gripped without any signs of slipping; "unstable" means that the object has a tendency to slip or completely detaches from the gripper.
[0009] (2) Benchmark tests were conducted using publicly available datasets. The effectiveness of introducing the Transformer model to process multimodal information was verified by comparing the recognition accuracy of Transformer models (such as TimeSformer and ViViT) and convolutional recurrent neural networks CNN+LSTM in three modalities (visual only, tactile only, and combined visual-tactile).
[0010] (3) The effectiveness of the optical flow-guided model learning proposed in this invention is verified by the improvement in recognition accuracy of the model before and after adding optical flow.
[0011] In this invention, the introduction of optical flow has the following significant advantages for the model to learn tactile image slip information:
[0012] 1) Detecting subtle movements: Optical flow can detect subtle movements and deformations on the surface of an object, which is especially important when identifying object slippage. For deformable objects, tiny slippage may indicate a failure to grasp, but optical flow can detect these subtle changes in advance, allowing for timely adjustments to the grasping strategy.
[0013] 2) Improved recognition accuracy: By introducing optical flow, the model can more accurately capture dynamic changes in tactile images. This dynamic information is crucial for distinguishing different grasping states (such as overstable, fully stable, and unstable).
[0014] 3) Enhanced multimodal data fusion: Optical flow can not only process tactile images but also combine them with motion information from visual images, providing richer input data. By fusing visual and tactile information, the model can more comprehensively understand the physical changes during the grasping process, improving overall grasping performance.
[0015] 4) Real-time feedback and adjustment: Optical flow is computationally fast and suitable for real-time applications. During the grasping process, the model can detect the object's slippage trend in real time and adjust the grasping force and position accordingly to avoid grasping failure. This is especially important for processing different types of deformable objects on an assembly line. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of the scene setup and overall system in this invention.
[0017] Figure 2 This is a schematic diagram of the multimodal end-to-end model Transformer used in this invention.
[0018] Figure 3 It is a comparison image of tactile images before and after the addition of optical flow.
[0019] Figure 4 These are schematic diagrams of the six types of bread used in the experiment. Detailed Implementation
[0020] The present invention will be further described below with reference to the accompanying drawings.
[0021] Figure 1 The experimental setup was demonstrated. The invention used a 6-DOF AUBO-i5 arm equipped with a ROBOTIQ 2F-140 parallel gripper to perform a grasping experiment. Each end of the parallel gripper was equipped with a Gelsightmini visual-tactile sensor to capture minute sliding trends of objects. However, only one sensor was used in this invention to demonstrate a minimal system setup.
[0022] The red box shows the internal structure of the visual-tactile sensor, which mainly consists of three parts: soft silicone with a reflective film, three colored LEDs, and a webcam. The soft silicone is illuminated by red, green, and blue (R, G, B) light from three different directions. When the soft silicone comes into contact with an object, its deformation is captured by the built-in webcam. The visual-tactile sensor elevates tactile perception to the "pixel level," with a resolution of up to 640*480, and can acquire multimodal tactile information. The blue box shows the external RealSense D435 camera, which is placed at the front of the robot arm to capture the global state information of the object.
[0023] Figure 2This is a schematic diagram of the Transformer multimodal end-to-end model used in this invention. A Transformer layer is as follows: Figure 2 As shown in the left sub-figure, this layer consists of a self-attention layer and a fully connected layer (MLP). The MLP layer contains two linear transformations and a GeLU activation function, performing the same processing on each vector position. To stack Transformer layers for deeper encoding structures, the MLP layer does not change the size of the vectors. Furthermore, before and after the self-attention and MLP layers, there is a layer normalization layer (LayerNorm) and a residual connection (Add), respectively.
[0024] As shown in the right-hand sub-figure, the output of the current layer serves as the input to the next layer. Before the first Transformer layer, all input vectors are linearly embedded, followed by position embedding, where the elements represent the position of each vector to preserve useful sequence knowledge. The self-attention mechanism allows all inputs to interact and identifies inputs that require more attention, which is their main advantage over CNN+LSTM models.
[0025] Specifically, this self-attention mechanism can be described as transforming each element of the input sequence (e.g., each patch of an image) into three vectors: a query vector Q, a key vector K, and a value vector V. For a single self-attention block or a single head, Q, K, and V can be obtained by using the same input matrix. (Each row of X corresponds to a size d) x The input vector, where d x The dimensions of K are projected onto Q, K, and V to calculate the result, as shown in formula (1):
[0026] Q = XW Q K = XW K V = XW V (1)
[0027] in, Let be three trainable parameter matrices, and d k =d v =d x .
[0028] The output of the self-attention mechanism is obtained through formula (2), which represents the weighted sum of the value vector V. The weights are assigned according to the compatibility function of the query vector Q and the corresponding key vector K under the same vector index. Formula (3) represents another trainable parameter matrix. The intermediate results are projected into a new matrix with the same dimensions as X.
[0029]
[0030] SingleHead(X) = Attention(Q,K,V)W O (3)
[0031] In practice, to enable the model to handle information from combinations of different input space representations and enhance fitting performance, a multi-head attention mechanism is employed. For the same input matrix X, multiple different sets of W are defined. i Q W i K W i V When i∈[1,h], the Q, K, and V matrices are weighted with different weight sets W. i Q W i K W i V The Transformer model is projected h times, and each group learns different parameters. This mechanism enables the Transformer model to represent features more effectively and improves performance.
[0032] Figure 3 The image shows a comparison of tactile images of the same object before and after adding optical flow in overly stable, perfectly stable, and unstable states. The comparison reveals that in the unstable state, for deformable objects like bread, the texture features mapped by the sensor are weak, making it difficult for the Transformer model to extract these features. After adding optical flow, although the texture features are not obvious, the optical flow can reflect the object's motion trend, providing some guidance for the model to learn the object's sliding tendency.
[0033] In the experiments of this invention, grasping and lifting experiments were conducted on six types of bread commonly found in daily life, such as... Figure 4 As shown. The clamping force and position are different for each object. Specifically, for each object, several grasping and lifting attempts were first performed. Currently, since the sensor has not been calibrated in three dimensions, the clamping force of the parallel gripper was mapped to the width. Nine gripping force candidates were selected from a certain width range, including three overly stable gripping forces, three completely stable gripping forces, and three unstable gripping forces, as well as two gripping position candidates, one near the object's center and one far from the object's center. Approximately 30 experiments were performed for each object.
[0034] Throughout the experiment, automated data acquisition was employed. The parallel gripper grasped the object at a speed of 15 mm / s each time, and after 2 seconds, the robotic arm lifted it to a height of 5 cm at a speed of 5 cm / s. At the start of the lifting process, tactile data and RGB data from an external camera were recorded at 60 Hz and 640×480 resolution to ensure synchronized acquisition. Labels were assigned based on the object's grasping state from initial grasp to subsequent lifting. Slippage or drop was marked as an unstable state; stable lifting without significant deformation was marked as a fully stable state; and overstable lifting with slight or significant deformation was marked as an overstable state.
[0035] exist Figure 2 In the algorithm schematic, each sensor uses a sequence of 14 consecutive frames as input. The time when the arm begins to lift is set as the reference time T(0), and the first image in the input is selected as the image captured at time T(-6) 6 frames before T(0) to ensure that the input sequence includes images of the static grasping of the object. The remaining images are consecutive images starting from time T(0).
[0036] In the dataset creation process, data augmentation was employed to increase the data volume. Considering potential scenarios in real-world experimental environments, methods such as image shifting, rotation, brightness variations, and the addition of Gaussian noise were used. Five-fold cross-validation was used to train the model to mitigate overfitting. This invention uses five types of bread as the training set and one type as the test set, averaging the final detection accuracy on the test set. For each model, performance was analyzed using three different data source inputs (visual only, tactile only, and a combination of visual and tactile data).
[0037] For the CNN+LSTM model, since ResNet18 has fewer parameters and does not require loading a pre-trained model, ResNet18 was chosen as the CNN architecture. During training, cross-entropy was used as the loss function, and the Adam optimizer was applied. For the two Transformer models (TimeSformer and ViViT), pre-trained models were used for fine-tuning, with the input embedding size, number of layers, and number of attention heads set to 256, 8, and 16, respectively. Experimental results are shown in Tables 1 and 2.
[0038] Table 1 (Without optical flow)
[0039]
[0040]
[0041] Table 2 (Incorporating Optical Flow)
[0042] CNN+LSTM TimeSformer ViViT Vision-only 77.68％(0.33％) 85.68％(0.63％) 85.88％(0.79％) Tactile-only 83.46％(0.74％) 83.76％(0.48％) 84.56％(0.48％) Vision & Tactile 85.38％(0.54％) 89.48％(0.32％) 88.38％(0.51％)
[0043] As shown in Table 1, the Transformer model provides more accurate classification results. For the three models, the multimodal performance is superior to the single-modal performance, indicating that multimodal input provides better cues for slip detection. Comparing Tables 1 and 2, adding optical flow to haptic images outperforms the model without optical flow, demonstrating that introducing optical flow is more effective in capturing the dynamic changes of objects.
Claims
1. A method for classifying grasping states based on visual-tactile learning, comprising an experimental scenario for which a 6-DOF robotic arm equipped with a parallel gripper was used for grasping experiments. Each end of the parallel gripper is equipped with a visual-tactile sensor to capture minute sliding tendencies of the object; characterized in that... Includes the following steps: (1) Construct an optical flow dataset OFB-6 for deformable objects in three states: overstable, fully stable and unstable. Overstable means that the object undergoes slight or obvious deformation during gripping. Fully stable means that the object does not undergo obvious deformation and can be stably gripped without any signs of slipping. Unstable means that the object has a tendency to slip or completely detaches from the gripper. (2) Benchmark tests were conducted using publicly available datasets. The effectiveness of introducing the Transformer model to process multimodal information was verified by comparing the recognition accuracy of the Transformer model and the convolutional recurrent neural network CNN+LSTM in three modalities through experiments. (3) The effectiveness of the proposed optical flow-guided Transformer model learning is verified by the improvement in recognition accuracy of the Transformer model before and after adding optical flow; The visual-tactile sensor consists of three parts: soft silicone with a reflective film, three LEDs, and a webcam. The soft silicone is illuminated by LEDs in red, green, and blue directions. When the soft silicone comes into contact with an object, its deformation is captured by the built-in webcam. The visual-tactile sensor enhances tactile perception to the "pixel level" and can acquire multimodal tactile information. The external camera is placed at the front of the 6-DOF robotic arm to capture the global state information of the object. In the multimodal end-to-end Transformer model, a Transformer layer contains a self-attention layer and a fully connected MLP layer; the fully connected MLP layer contains two linear transformations and a GeLU activation function, performing the same processing on each vector position; to stack Transformer layers to obtain a deeper encoding structure, the MLP layer does not change the size of the vector; before and after the attention layer (Self-Attention) and the MLP layer, there is a LayerNorm and a residual connection (Add), respectively; The output of the current layer serves as the input to the next layer; before the first Transformer layer, all input vectors are linearly embedded, and then positional embeddings are added, where the elements represent the position of each vector to preserve useful sequence knowledge. The self-attention mechanism allows all inputs to interact with each other and identifies inputs that require more attention. The self-attention mechanism is described as transforming each element of the input sequence into three vectors: a query vector Q, a key vector K, and a value vector V. For a single self-attention block or a single head, Q, K, and V are obtained by using the same input matrix. Project onto Q, K, V for calculation. Each row corresponds to a size of The input vector, where Let K be the dimension size; as shown in formula (1): （1） in, , , There are three trainable parameter matrices, and ; The data from each visual-tactile sensor is taken as a sequence of 14 consecutive frames as input; the time when the 6-DOF robot arm starts to lift is set as the reference time T(0), and the first image of the input is selected as the image captured at time T(-6) 6 frames before T(0) to ensure that the input sequence includes the image when the object is statically grasped, and the remaining images are consecutive images after time T(0); During the dataset creation process, data augmentation was adopted to increase the amount of data. In combination with the possible situations in the actual experimental environment, image shifting, rotation, brightness changes and Gaussian noise were used. The Transformer model was trained by five-fold cross-validation to reduce overfitting. Five types of bread were used as the training set and one type of bread was used as the test set. The final detection accuracy on the test set was averaged. For the CNN+LSTM model, ResNet18 was chosen as the CNN architecture. During training, cross-entropy was used as the loss function, and the Adam optimizer was applied. For the two Transformer models, fine-tuning was performed through pre-training, with the input embedding size, number of layers, and number of attention heads set to 256, 8, and 16, respectively.
2. The method for classifying grasping states based on visual-tactile learning according to claim 1, characterized in that, The output of the self-attention mechanism is obtained through formula (2), which represents the weighted sum of the value vector V. The weights are assigned according to the compatibility function of the query vector Q and the corresponding key vector K under the same vector index. Formula (3) represents another trainable parameter matrix. Project the intermediate results onto a [parameter / object]. A new matrix with the same dimensions; （2） （3） Employing a multi-head attention mechanism, for the same input matrix Define multiple different sets , , ,when At that time, the Q, K, and V matrices are weighted with different sets of weights. , , Projecting h times, each group will learn different parameters.