Grabbing state classification method based on visual tactile learning
By introducing the Transformer model and optical flow method into robot visual haptic perception, the problem of insufficient tactile information in traditional mechanical grasping is solved, the accuracy of gripping state recognition of deformable objects is improved, multimodal data fusion is enhanced, and the More efficient crawling performance.
Patent Information
- Application Number
- CN202411164412.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-23
AI Technical Summary
Traditional mechanical grasping is difficult to accurately capture the stable state of deformable objects, especially due to insufficient tactile information, which leads to insufficient information in the learning process of multimodal models, which affects the grab performance.
The Transformer model is introduced to process tactile modal information, and combines the optical flow method to capture subtle movements and changes of objects, guiding the model to learn different grasping features and improve the accuracy of multimodal recognition.
The object slip information is captured through the optical flow method, which improves the accuracy of the identification of the grasping state and enhances multimodal data fusion. It is suitable for real-time applications and can adjust the grab strategy in a timely manner to avoid grab failure.
Smart Images

Figure BDA0005007414090000044 
Figure BDA0005007414090000051 
Figure BDA0005007414090000061
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of robot visual-tactile perception, and in particular relates to a method for grasping stable state classification based on end-to-end model learning. Background Art
[0002] Intelligent grasping and operating robots are widely used in industries such as industry and service. During operation, it is crucial to judge the grasping state of the gripper at the end of the robot arm. In particular, the grasping stability of deformable objects is a hot topic and difficulty in current research. When performing grasping operations, not only the stability of the grasped object should be considered, but also the safety of the grasped object to prevent damage to the object. Insufficient clamping force or incorrect clamping position are both causes of unstable clamping, and people have conducted long-term research on this. Detecting slip can help robots adjust their grasping force and strategy to avoid grasping failures.
[0003] Traditional mechanical grasping often fails to accurately capture the state of an object, which can easily cause deformation or damage to the object. In previous work, we used the wavelet fuzzy entropy algorithm to classify the grasping state, but it was difficult to expand the data set to deformable objects due to the lack of information from the array tactile sensor TS2520. This method combines the multimodal information of visual tactile sensors and external cameras, expands the data set to deformable objects in daily life (such as bread), and uses an end-to-end multimodal model to learn to classify different grasping states. In the process of actual grasping using visual tactile sensors, it was found that when the end effector of the robot grasped a deformable object with a small force, the features of the collected tactile image were not obvious, which led to the potential problem of insufficient information in the multimodal model in the process of learning tactile images. Although the model can use the visual images of the external camera to integrate information to make up for the problem of insufficient tactile image information, the importance of tactile information in the grasping process, especially in capturing the slight sliding trend of deformable objects, cannot be replaced by visual information. Therefore, how to further improve the accuracy of tactile modality recognition has become an important issue in improving multimodal performance. Summary of the invention
[0004] In view of the above problems, the technical solution adopted by the present invention is a method for grasping state classification based on visual-tactile learning. The method is improved in two aspects to improve the accuracy of tactile modality recognition.
[0005] (1) The Transformer model is introduced to process tactile modal information. Compared with traditional convolutional neural networks (CNN) and long short-term memory networks (LSTM), the Transformer model has a stronger ability to capture long-term dependencies and can process spatial and temporal features simultaneously.
[0006] (2) When processing tactile images, the optical flow method is introduced to guide the model to learn different grasping features. The optical flow method is a technology that calculates the movement of pixels in an image sequence. It is widely used in the field of computer vision. It estimates the movement of an object between two frames by analyzing the brightness changes of each pixel in the image. Therefore, when processing tactile images, the optical flow method can capture subtle movements and changes, and effectively identify dynamic information such as object slippage.
[0007] The specific implementation process includes the following steps:
[0008] (1) Construct an optical flow dataset OFB-6 of deformable objects (the present invention uses 6 different types of bread as the objects) in three states: overstable, fully stable, and unstable. Here, “overstable” means that the object undergoes slight or obvious deformation during clamping; “fully stable” means that the object has no obvious deformation and can be grasped stably without any signs of slipping; “unstable” means that the object has a tendency to slip or is completely out of the gripper.
[0009] (2) We use public datasets for benchmarking and experimentally compare the recognition accuracy of Transformer models (such as TimeSformer and ViViT) and convolutional recurrent neural networks (CNN+LSTM) in three modalities (vision only, touch only, and vision and touch combined) to verify the effectiveness of introducing the Transformer model to process multimodal information.
[0010] (3) The effectiveness of the optical flow-guided model learning proposed in the present invention is verified by the improvement in recognition accuracy of the model before and after adding optical flow.
[0011] In the present invention, the introduction of the optical flow method has the following significant benefits for the model to learn the tactile image sliding information:
[0012] 1) Capturing subtle movements: Optical flow can capture subtle movements and deformations on the surface of an object, which is particularly important when identifying object slippage. For deformable objects, tiny slippage may indicate grasping failure, and optical flow can detect these tiny changes in advance, thereby adjusting the grasping strategy in time.
[0013] 2) Improve recognition accuracy: By introducing the optical flow method, the model can more accurately capture the dynamic changes in the tactile image. This dynamic information is crucial for distinguishing different grasping states (such as over-stable, fully stable, unstable).
[0014] 3) Enhanced multimodal data fusion: The optical flow method can not only process tactile images, but also combine with motion information in visual images to provide richer input data. By fusing visual and tactile information, the model can more comprehensively understand the physical changes during the grasping process and improve the overall grasping performance.
[0015] 4) Real-time feedback and adjustment: The optical flow method is fast and suitable for real-time applications. During the grasping process, the model can instantly detect the object's slipping tendency and adjust the grasping force and grasping position in time to avoid grasping failure. This is especially important for processing different types of deformable objects on the assembly line. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 It is a schematic diagram of the scene construction and the overall system in the present invention.
[0014] Figure 2 This is a schematic diagram of the multimodal end-to-end model Transformer used in the present invention.
[0015] Figure 3 This is a comparison of the tactile image before and after adding optical flow.
[0016] Figure 4 This is a schematic diagram of the six types of bread used in the experiment. DETAILED DESCRIPTION
[0017] The present invention is further described below in conjunction with the accompanying drawings.
[0018] Figure 1 The construction of the experimental scene is demonstrated. The present invention uses a 6-DOF AUBO-i5 arm equipped with a ROBOTIQ 2F-140 parallel gripper to carry out a grasping experiment. A visual tactile sensor Gel Sightmini is installed at each end of the parallel gripper to capture the slight sliding tendency of the object, but only one is used in the present invention to demonstrate the minimum system setup.
[0019] The red frame shows the internal display of the visual tactile sensor, which mainly consists of three parts: soft silicone with a reflective film, three colored LEDs and a webcam. The soft silicone is illuminated by light from three different directions: red, green and blue (R, G, B). When the soft silicone comes into contact with an object, the deformation of the soft silicone is captured by the built-in webcam. The visual tactile sensor improves the tactile perception capability to the "pixel level", with a resolution of up to 640*480, and can obtain multimodal tactile information. The blue frame shows the external camera RealSense D435, which is placed in front of the robot arm to capture the global state information of the object.
[0020] Figure 2is a schematic diagram of the multimodal end-to-end model Transformer used in the present invention. A Transformer layer is as follows Figure 2 As shown in the (left sub-figure), it contains a self-attention layer (Self-Attention) and a fully connected layer (MLP). The MLP layer contains two linear transformations and a GeLU activation function, and performs the same processing on each vector position. In order to stack the Transformer layer to obtain a deeper encoding structure, the MLP layer does not change the size of the vector. In addition, there is a layer normalization (LayerNorm) and a residual connection (Add) before and after the attention layer Self-Attention and the MLP layer, respectively.
[0021] As shown in the right sub-figure, the output of the current layer is used as the input of the next layer. Before the first Transformer layer, all input vectors will be linearly embedded (Linear Embedding), and then position embedding (PositionEmbedding) will be added, where the elements represent the position of each vector to retain useful sequence knowledge. The self-attention mechanism allows all inputs to interact with each other and identify inputs that require more attention, which is their main advantage over CNN+LSTM models.
[0022] Specifically, this self-attention mechanism can be described as converting each element of the input sequence (such as each patch of an image) into three vectors, namely the query vector Q (Query), the key vector K (Key), and the value vector V (Value). For a single self-attention block or a single head, Q, K, and V can be obtained by converting the same input matrix (Each row of X corresponds to a x The input vector of d x is the dimension size of K) projected to Q, K, V for calculation, as shown in formula (1): Q=XW Q ,K=XW K ,V=XW V (1)
[0023] in, are three trainable parameter matrices, and d k =d v =d x .
[0024] The output of the self-attention mechanism is obtained by formula (2), which represents the weighted sum of the value vector V, whose weights are assigned according to the compatibility function between the query vector Q and the corresponding key vector K at the same vector index. Formula (3) represents another trainable parameter matrix Project the intermediate results into a new matrix of the same dimension as X. SingleHead(X) = Attention(Q,K,V)W O (3)
[0025] In practice, in order to enable the model to process information from different input space representation combinations and enhance fitting performance, a multi-head attention mechanism is used. For the same input matrix X, define multiple groups of different W i Q , W i K , W i V , when i∈[1,h], the Q, K, and V matrices are transformed with different weight sets W i Q , W i K , W i V After h projections, each group will learn different parameters. This mechanism enables the Transformer model to more effectively represent features and enhance performance.
[0026] Figure 3 The figure shows the comparison of tactile images of the same object before and after adding optical flow in overstable, completely stable and unstable states. Through comparison, it can be found that in the unstable state, for deformable objects such as bread, the texture features of the object mapped by the sensor are weak, and it is not easy for the Transformer model to extract features. After adding optical flow, although the texture features of the object are not obvious, the movement trend of the object can be reflected through the optical flow, which has a certain guiding role in the model learning the sliding trend of the object.
[0027] In the experiment of the present invention, six kinds of bread in daily life were subjected to grasping and lifting experiments, such as Figure 4 As shown. The gripping force and position of each object are different. Specifically, for each object, several attempts to grasp and lift are first made. At this stage, since there is no three-dimensional force calibration for the sensor, the gripping force of the parallel gripper is corresponded to the width size. Nine grasping force candidates are selected from a certain width range, including three over-stable grasping forces, three completely stable grasping forces and three unstable grasping forces, as well as two grasping position candidates, including close to the center of the object and far from the center of the object. About 30 experiments were conducted on each object.
[0028] During the entire experiment, an automated data collection method was used. The parallel gripper grasped the object at a grasping speed of 15 mm / s each time. After 2 seconds, the robotic arm lifted the object to a height of 5 cm at a speed of 5 cm / s. At the beginning of the lift, the tactile data and the RGB data of the external camera were recorded at 60 Hz and 640 × 480 resolution to ensure the same frequency of acquisition. Labels were assigned by observing the grasping state of the object from grasping to lifting. If slipping or falling occurs, it is marked as an unstable state; if the object can be lifted stably without obvious deformation, it is marked as a completely stable state; if the object is lifted stably but slight or more severe deformation occurs, it is marked as an overstable state.
[0029] exist Figure 2 In the algorithm schematic, the data of each sensor is input in a sequence of 14 consecutive frames. The time when the arm starts to lift is set as the reference time T(0), and the first input image is selected as the image captured at time T(-6) before T(0) to ensure that the input sequence contains images of static grasping of objects, and the remaining images are continuous images starting from time T(0).
[0030] In the process of data set preparation, in order to increase the size of data, data enhancement was adopted. In combination with the possible situations in the actual experimental environment, methods such as image shift, rotation, light and dark changes, and adding Gaussian noise were used, and the model was trained by five-fold cross validation to reduce the impact of overfitting. The present invention uses 5 kinds of bread as training sets and 1 kind of bread as test sets, and takes the average of the final detection accuracy on the test sets. For each model, three different data source inputs (only vision, only touch, and visual and tactile combination) are used to analyze the performance.
[0031] For the CNN+LSTM model, since ResNet18 has fewer parameters and does not need to load a pre-trained model, ResNet18 is selected as the CNN architecture. During the training process, cross entropy is used as the loss function and the Adam optimizer is applied. For the two Transformer models (TimeSformer and ViViT), the pre-trained models are used for fine-tuning, and their input embedding size, number of layers, and number of attention heads are set to 256, 8, and 16 respectively. The experimental results are shown in Tables 1 and 2. Table 1 (without adding optical flow) Table 2 (adding optical flow) CNN+LSTM TimeSformer ViViT Vision-only 77.68%(0.33%) 85.68%(0.63%) 85.88%(0.79%) Tactile-only 83.46%(0.74%) 83.76%(0.48%) 84.56%(0.48%) Vision&Tactile 85.38%(0.54%) 89.48%(0.32%) 88.38%(0.51%)
[0032] As can be seen from Table 1, the Transformer model can provide more accurate classification results. For the three models, the performance in the multimodal case is better than that in the single-modal case, indicating that multimodal input provides better clues for slip detection. Comparing Table 1 and Table 2, the performance of adding optical flow to the tactile image is better than that without adding optical flow, indicating that the introduction of optical flow has a better effect on the model to capture the dynamic changes of objects.
Claims
1. A method for grasping state classification based on visual-tactile learning. An experimental scenario for this method is built. A grasping experiment is carried out using a 6-DOF robot arm equipped with a parallel gripper. A visual-tactile sensor is installed at each end of the parallel gripper to capture the slight sliding tendency of the object. The method is characterized in that: The steps include: (1) Construct an optical flow dataset OFB-6 of deformable objects in three states: overstable, completely stable, and unstable. Overstable means that the object undergoes slight or obvious deformation during gripping. Completely stable means that the object has no obvious deformation and can be gripped stably without any signs of slipping. Unstable means that the object has a tendency to slip or completely escapes from the gripper. (2) We use public datasets to conduct benchmark tests and experimentally compare the recognition accuracy of the Transformer model and the convolutional recurrent neural network (CNN+LSTM) in three modalities to verify the effectiveness of introducing the Transformer model to process multimodal information. (3) The improvement in recognition accuracy of the Transformer model before and after adding optical flow verifies the effectiveness of the proposed optical flow-guided Transformer model learning.
2. The method for grasping state classification based on visual-tactile learning according to claim 1, characterized in that: The visual-tactile sensor consists of three parts: soft silicone with a reflective film, three LEDs and a webcam; the soft silicone is illuminated by light from three LEDs in different directions, red, green and blue. When the soft silicone comes into contact with an object, the deformation of the soft silicone is captured by the built-in webcam; the visual-tactile sensor improves the tactile perception ability to the "pixel level" and can obtain multimodal tactile information; the external camera is placed at the front of the 6-DOF robot arm to capture the global state information of the object.
3. The method for grasping state classification based on visual-tactile learning according to claim 1, characterized in that: In the multimodal end-to-end model Transformer, a Transformer layer contains a self-attention layer and a fully connected MLP layer; the fully connected MLP layer contains two linear transformations and a GeLU activation function, and performs the same processing on each vector position; to stack Transformer layers to obtain a deeper encoding structure, the MLP layer does not change the size of the vector; before and after the attention layer Self-Attention and the MLP layer, there is a layer normalization LayerNorm and a residual connection Add, respectively.
4. The method for grasping state classification based on visual-tactile learning according to claim 1, characterized in that: The output of the current layer serves as the input of the next layer; before the first Transformer layer, all input vectors will be linearly embedded, and then position embedding is added, where the elements represent the position of each vector to retain useful sequence knowledge, and the self-attention mechanism allows all inputs to interact with each other and identify inputs that need more attention.
5. The method for grasping state classification based on visual-tactile learning according to claim 1, characterized in that: The self-attention mechanism is described as converting each element of the input sequence into three vectors, namely the query vector Q, the key vector K and the value vector V; for a single self-attention block or a single head, Q, K, V are obtained by converting the same input matrix Projected to Q, K, V to calculate, each row of X corresponds to a size d x The input vector of d x is the dimension size of K; as shown in formula (1): Q=XW Q ,K=XW K ,V=XW V (1) in, are three trainable parameter matrices, and d k =d v =d x .
6. The method for grasping state classification based on visual-tactile learning according to claim 1, characterized in that: The output of the self-attention mechanism is obtained by formula (2), which represents the weighted sum of the value vector V, whose weight is assigned according to the compatibility function between the query vector Q and the corresponding key vector K under the same vector index; formula (3) represents another trainable parameter matrix Project the intermediate result into a new matrix with the same dimension as X; SingleHead(X) = Attention(Q,K,V)W O (3) Using a multi-head attention mechanism, for the same input matrix X, define multiple different sets of W i Q , W i K , W i V , when i∈[1,h], the Q, K, and V matrices are transformed with different weight sets W i Q , W i K , W i V After h projections, each group will learn different parameters.
7. The method for grasping state classification based on visual-tactile learning according to claim 1, characterized in that: The data of each visual-tactile sensor uses a sequence of 14 consecutive frames as input; the time when the 6-DOF robot arm starts lifting is set as the reference time T(0), and the first input image is selected as the image captured at time T(-6) before T(0) to ensure that the input sequence contains images of static grasping of objects, and the remaining images are continuous images starting from time T(0).
8. The method for grasping state classification based on visual-tactile learning according to claim 1, characterized in that: In the process of dataset preparation, data enhancement was adopted to increase the data size. Combined with the situations that may occur in the actual experimental environment, image shift, rotation, brightness change and Gaussian noise addition methods were used, and the Transformer model was trained through five-fold cross validation to reduce overfitting. Five kinds of bread were used as training sets and one kind of bread was used as test set, and the final detection accuracy on the test set was averaged.
9. The method for grasping state classification based on visual-tactile learning according to claim 1, characterized in that: For the CNN+LSTM model, ResNet18 was selected as the CNN architecture. During the training process, cross entropy was used as the loss function, and the Adam optimizer was applied. For the two Transformer models, they were fine-tuned through pre-training, and their input embedding size, number of layers, and number of attention heads were set to 256, 8, and 16, respectively.
Citation Information
Patent Citations
Mechanical arm autonomous grabbing method based on visual touch fusion under weak rigidity characteristic condition
CN115431279A
Grabbing stability evaluation method based on visual touch fusion perception and multi-modal space-time convolution
CN116945170A
Visual tactile sensor and tactile perception method
CN117182945A
Litchi soft claw end effector based on visual touch fusion and picking method
CN117445031A
Two-finger clamp deformable object fine grabbing system based on tactile feedback
CN117656039A
Cited By
Robot manipulator grabbing and butt joint method and equipment, medium and product
CN119681881A
