A weak stiffness object grasping method based on VTF visual-tactile information interaction

Abstract

The application discloses a weak stiffness object grabbing method based on VTF visual-tactile information interaction, and the method comprises the following steps: acquiring image information and tactile signals of an object to be grabbed; performing feature extraction and fusion processing on the image information and the tactile signals, and outputting a grabbing state recognition result; and grabbing the object to be grabbed by a robot according to the grabbing state recognition result. By using the application, the grabbing state can be further refined by interacting and fusing tactile key point feature information and visual feature information in stages, so that the grabbing effect of a mechanical arm is improved. The application can be widely applied to the technical field of machine fusion perception.

Description

Technical Field

[0001] This invention relates to the field of machine fusion perception technology, and in particular to a method for grasping weakly rigid objects based on VTF visual-tactile information interaction. Background Technology

[0002] With the diversification of grasping tasks, the demand for various dexterous maneuvers in robots is also increasing. Robot grasping capabilities are receiving increasing attention. Stable grasping is the foundation for various dexterous maneuvers in robots. The variability and slippage of targets with weak stiffness characteristics remain challenges in the grasping process. Failure to address these issues may lead to target breakage or detachment during grasping. For targets with weak stiffness characteristics, simply detecting slippage during the grasping process is insufficient, because the grasped target's structure is easily deformed during grasping. Excessive deformation of the target signifies task failure for grasping tasks. Therefore, based on slippage detection... In addition, the uncertain deformation of the grasped target also needs to be considered. In recent years, researchers have applied deep learning methods to the evaluation of robot grasping targets, using convolutional neural networks to predict the state of the grasped target. However, the prediction method also has shortcomings. First, there is the problem that the stability of visual grasping of targets with weak stiffness is difficult to evaluate. Second, the grasping state category of targets with weak stiffness is too simple, only judging whether the target will slip during grasping, without considering the uncertain deformation of targets with weak stiffness. More importantly, there is the problem of insufficient fusion of visual information and tactile global information, and insufficient consideration of the fusion of visual features and tactile key point features. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention aims to provide a method for grasping weakly rigid objects based on VTF (Vision-Tactile Information Interaction). This method can further refine the recognition and grasping state by interacting and fusing tactile key point feature information with visual feature information in stages, thereby improving the grasping effect of the robotic arm.

[0004] The first technical solution adopted in this invention is: a method for grasping weakly rigid objects based on VTF visual-tactile information interaction, comprising the following steps:

[0005] Acquire image information and tactile signals of the object to be grasped;

[0006] The image information and tactile signals are combined and processed to extract and fuse features, and the result of grasping state recognition is output.

[0007] The robot grasps the object to be grasped based on the grasping status recognition result.

[0008] Furthermore, the step of acquiring the image information and tactile signals of the object to be grasped specifically includes:

[0009] The image of the object to be grasped is obtained by taking a picture of the object with a camera.

[0010] Obtain corresponding image information based on the image of the object to be grasped;

[0011] The tactile signals of the image of the object to be grasped are obtained through a tactile sensor.

[0012] Furthermore, the step of performing feature extraction and fusion processing on image information and tactile signals to output the grasping state recognition result specifically includes:

[0013] The visual feature extraction module performs feature extraction processing on the image information to obtain visual modality information;

[0014] The tactile feature extraction module extracts features from the tactile signals to obtain tactile modal information.

[0015] The visual modality information and the tactile modality information are fused by the visual-tactile feature fusion module to obtain the grasping state recognition result.

[0016] Furthermore, the step of extracting features from image information using the visual feature extraction module to obtain visual modality information specifically includes:

[0017] Image information is input to a visual feature extraction module, which includes a convolutional layer, a regularization layer, a max pooling layer, and a visual residual block.

[0018] The visual residual block includes a first visual residual block, a second visual residual block, a third visual residual block, and a fourth visual residual block;

[0019] Image information is processed by convolutional layers to obtain a feature-extracted image.

[0020] The feature extraction image is regularized based on the regularization layer to obtain the regularized feature image;

[0021] The regularized feature image is downsampled using a max pooling layer to obtain a downsampled feature image.

[0022] Residual analysis is performed on the downsampled feature image based on the visual residual block to obtain visual modal information, which includes first visual modal information, second visual modal information, third visual modal information and fourth visual modal information.

[0023] Furthermore, the step of extracting features from the tactile signal using the tactile feature extraction module to obtain tactile modal information specifically includes:

[0024] The tactile signal is split into segments to obtain split tactile signal information, which includes first tactile signal information, second tactile signal information, third tactile signal information and fourth tactile signal information;

[0025] The split tactile signal information is input into the tactile feature extraction module, which includes a first tactile residual block, a second tactile residual block, a third tactile residual block, and a fourth tactile residual block.

[0026] The first tactile signal information is convolved based on the first tactile residual block to obtain the first tactile modal information. The first tactile residual block includes a convolutional layer.

[0027] Based on the second tactile residual block, the third tactile residual block, and the fourth tactile residual block, convolution and fusion processing are performed on the second tactile signal information, the third tactile signal information, and the fourth tactile signal information, respectively, to obtain the second tactile modal information, the third tactile modal information, and the fourth tactile modal information;

[0028] The second, third, and fourth tactile residual blocks each include two convolutional layers;

[0029] The first tactile modal information, the second tactile modal information, the third tactile modal information, and the fourth tactile modal information are integrated to obtain the tactile modal information.

[0030] Furthermore, the step of fusing visual modal information and tactile modal information through the visual-tactile feature fusion module to obtain the grasping state recognition result specifically includes:

[0031] Visual modal information and tactile modal information are transmitted to the visual-tactile feature fusion module;

[0032] The visual modality information and tactile modality information are processed by global average pooling based on the Fsq function to obtain the visual modality pooling value and the tactile modality pooling value.

[0033] The visual modal pooling values ​​and tactile modal pooling values ​​are fully connected based on the Fex function to obtain the visual modal fully connected values ​​and the tactile modal fully connected values.

[0034] The visual modality fully connected value and the tactile modality fully connected value are processed by dot product operation based on the Fsigmoid function to obtain the visual modality feature value and the tactile modality feature value;

[0035] The visual modality feature values ​​and tactile modality feature values ​​are fused by a fully connected layer to obtain the grasping state recognition result.

[0036] Furthermore, the calculation formula for the global average pooling operation of the Fsq function on visual modal information and tactile modal information is as follows:

[0037]

[0038]

[0039] In the above formula, U Vn and U Tn S represents visual modal information and tactile modal information, respectively. Vn and S Tn represents the pooling value of the visual modality and the pooling value of the tactile modality, respectively; c represents the layer number of the visual or tactile feature tensor; and H and W represent the length and width of the tensor, respectively.

[0040] Furthermore, the calculation formula for the fully connected processing of the visual modal pooling value and the tactile modal pooling value by the Fex function is as follows:

[0041] Zn=W[S Vn ,S Tn ]+b

[0042] E Vn =W Vn Zn+b Vn

[0043] E Tn =W Tn Zn+b Tn

[0044] In the above formula, Zn represents the excitation signal, and E Vn and E Tn W represents the fully connected value of the visual modality and the fully connected value of the tactile modality, respectively. Vn W represents the weights of visual features. Tn The weights representing tactile features, b Vn Indicates visual feature bias, b Tn 'b' represents the bias of tactile features, and 'b' represents the fused visual-tactile bias.

[0045] Furthermore, the calculation formula for the Fsigmoid function performing a dot product operation on the visual modality fully connected value and the tactile modality fully connected value is as follows:

[0046] U' Vn =σ(E' Vn )⊙U Vn

[0047] U' Tn =σ(E' Tn )⊙U Tn

[0048] In the above formula, U' Vn and U' Tn Represents the visual modality feature value and the tactile modality feature value, σ represents the Fsigmoid function, and E' Vn and E' Tn U represents the excitation factors for vision and touch, respectively. Vn and U Tn These represent the original visual and tactile features, respectively.

[0049] Furthermore, the grasping state recognition results specifically include non-contact state, appropriate contact state, excessive contact state, no slippage state, and slippage state.

[0050] The beneficial effects of the method of the present invention are as follows: The present invention extracts the image information of the object to be detected through a visual feature extraction module and a tactile feature extraction module, and further fuses the image information of the object to be detected through a visual-tactile feature fusion module (VTF). This can address the problem of insufficient fusion between visual information and global tactile information by interactively fusing tactile key point feature information with visual feature information in stages to obtain the grasping state recognition result. The grasping state recognition result expands the two categories of slip and non-slip to five categories: no contact, appropriate contact, excessive contact, no slip, and slip. Attached Figure Description

[0051] Figure 1 This is a flowchart of the steps of a weakly stiff object grasping method based on VTF visual-tactile information interaction according to the present invention.

[0052] Figure 2 This is a schematic diagram of the phased fusion process of visual features and key point tactile features in this invention;

[0053] Figure 3 This is a schematic diagram of the structure of the visual residual block for visual feature extraction according to the present invention;

[0054] Figure 4 This is a schematic diagram of the visual-tactile fusion operation of the present invention;

[0055] Figure 5 This is a schematic diagram of the structure for sorting different inputs of tactile information according to the present invention. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0057] Reference Figure 1This invention provides a method for grasping weakly stiff objects based on VTF (Vision-Tactile Information Interaction), the method comprising the following steps:

[0058] S1. Acquire image information and tactile signals of the object to be grasped;

[0059] Specifically, image information is acquired through a 1080p resolution camera, and tactile signals are obtained through a 4x4 array of tactile sensors. One frame of image signal is (224x224x3) (length x width x number of channels), and one frame of tactile signal is (4x4x3). The tactile sensor has a total of sixteen points, and each point can feed back three values, corresponding to the magnitude of force in the x, y, and z directions.

[0060] S2. Perform feature extraction and fusion processing on image information and tactile signals, and output the grasping state recognition result;

[0061] Specifically, the result of grasping is to input the acquired image signals and tactile signals into, respectively, a system such as... Figure 2 In the network shown, the output Y corresponds to one of five states, where Y represents the grasping state of the grasped object. 0, 1, 2, 3, and 4 represent the non-contact state, appropriate contact state, excessive contact state, no slip state, and slip state, respectively.

[0062] S21. Visual modal information is obtained by performing feature extraction processing on the image information through the visual feature extraction module.

[0063] Specifically, refer to Figure 3As shown, visual features are extracted by convolutional layers, visual residual blocks 1, 2, 3, and 4 (i.e., the first, second, third, and fourth visual residual blocks), and the convolutional layer is a single 2D convolution with the following parameters: (3 input channels, 64 output channels, kernel size 7, stride 2, padding 3, followed by a regularization layer and a max-pooling layer with kernel size 3, stride 2, and padding 1). The specific operation of the convolutional layer is determined by four quantities: n filters, F receptive field, P zero-padding, and S stride. The input image is assumed to be W×W. The input feature is W×W×n (length × width × number of channels). After processing by the convolutional layer, it becomes W1×W1×n (length × width × number of channels). The convolution formula is: W1=(W-F+2P) / S+1; The regularization layer is Batch-Normalization. The configuration of the max pooling layer as a pooling operation is determined by the following four quantities: the number of output channels is m, the side length of the convolutional kernel is F, zero padding is P, and the stride is S. Assuming the input feature is W×W×n (length × width × number of channels), after processing by the max pooling layer, it becomes W2×W2×m (length × width × number of channels). The max pooling formula is: W2=(W-F+2P) / S+1;

[0064] The visual image changes from (224x224x3) to (56x56x64), and then passes through 4 visual residual block modules, such as... Figure 3 As shown in the figure, k represents the size of the convolution kernel, s represents the stride in the convolution process, p represents padding, c represents the output channel of the convolution operation, BN represents the regularization operation, and RL represents the ReLU function.

[0065] S22. The tactile signal is processed by the tactile feature extraction module to obtain tactile modal information;

[0066] Specifically, the size of the tactile information in a frame is (4x4x3). Simply using tactile information for convolution may not result in sufficient fusion with visual information. This invention proposes a new sorting method for inputting the corresponding network, such as... Figure 5 As shown, the tactile information (4x4x3) is divided into four (3x3x3) inputs, and the corresponding tactile information is enclosed in wireframes;

[0067] The following are the main parameters of the four tactile residual blocks, with specific parameters as follows:

[0068] The haptic residual block 1 is the convolutional layer 1 inside the first haptic residual block. It has 3 kernels, a stride of 1, padding of 1, 3 input channels, and 64 output channels.

[0069] Haptic residual block 2, which is convolutional layer 2 within the second haptic residual block, has a kernel of 3, a stride of 1, padding of 1, 64 input channels, and 125 output channels. Convolutional layer 3 within haptic residual block 2 also has a kernel of 3, a stride of 1, padding of 1, 3 input channels, and 3 output channels. The symbol represents directly splicing two channels (i.e., 125 + 3 = 128 channels);

[0070] Haptic residual block 3, also known as convolutional layer 4 within the third haptic residual block, has a kernel size of 3, a stride of 1, padding of 1, 128 input channels, and 253 output channels. Convolutional layer 5 within haptic residual block 3 also has a kernel size of 3, a stride of 1, padding of 1, 3 input channels, and 3 output channels. The symbol represents directly splicing two channels (i.e., 253 + 3 = 256 channels);

[0071] The haptic residual block 4, also known as the fourth haptic residual block, contains convolutional layer 6 with a kernel of 3, a stride of 1, padding of 1, 256 input channels, and 512 output channels. The haptic residual block 4 also contains convolutional layer 7 with a kernel of 3, a stride of 1, padding of 1, 3 input channels, and 3 output channels. The symbol represents directly splicing two channels (i.e., 509 + 3 = 512 channels).

[0072] S23. The visual modal information and tactile modal information are fused through the visual-tactile feature fusion module to obtain the grasping state recognition result.

[0073] Specifically, the main function of the visual-tactile feature fusion module is to interact two different information modes, visual and tactile, and then perform a dot product operation between the interacted information and the features extracted at each stage to obtain the interacted visual-tactile features. The VTF fusion method proposed in this invention can well characterize the differences and correlations between the two modalities. The specific operation is as follows: Figure 4 As shown;

[0074] U Vn (c) and U Tn (c) The extracted features are first processed by the Fsq(·) function. The Fsq(·) function mainly performs a global average pooling operation on the extracted tensors, transforming the tensor size (HxWxC) into (1x1xC) through global pooling, and then obtaining S. Vn and S Tn The specific formula is as follows:

[0075]

[0076]

[0077] In the above formula, U Vn and U Tn S represents visual modal information and tactile modal information, respectively. Vn and S Tn represents the pooling value of the visual modality and the pooling value of the tactile modality, respectively; c represents the layer of the visual or tactile feature tensor; H and W represent the length and width of the tensor, respectively.

[0078] In the above formula, c (belonging to 1, 2, 3...C) represents the layer of the visual or tactile feature tensor, H and W represent the length and width of the tensor, and U... Vn (c) and U Tn (c) represent the nth layer of features of the tensor, ∑U Vn (c) and ∑U Tn (c) represents a globally weighted numerical representation of the c-th layer features of the tensor, S Vn (c) represents the global average pooling value of the c-th layer of visual features, S Tn (c) represents the global average pooling value of the c-th layer of tactile features, where n is 1, 2, or 3;

[0079] S Vn and S Tn After concatenation, a joint excitation signal Zn is obtained. Zn then passes through two independent fully connected layers (Fex(·) function) to obtain E. V and E T The formula is shown below:

[0080] Zn=W[S Vn ,S Tn ]+b

[0081] E Vn =W Vn Zn+b Vn

[0082] E Tn =W Tn Zn+b Tn

[0083] In the above formula, Zn represents the excitation signal, and E Vn and E Tn W represents the fully connected value of the visual modality and the fully connected value of the tactile modality, respectively. Vn W represents the weights of visual features. Tn The weights representing tactile features, b Vn Indicates visual feature bias, b Tn 'b' represents the bias of tactile features, and 'b' represents the fused visual-tactile bias.

[0084] E Vn and E TnAfter passing through the Fsigmode(·) function (σ), the new feature U' is obtained by performing a dot product operation (denoted by the symbol ⊙) with the original feature. Vn and U' Tn The specific formula is as follows:

[0085] U' Vn =σ(E' Vn )⊙U Vn

[0086] U' Tn =σ(E' Tn )⊙U Tn

[0087] In the above formula, U' Vn and U' Tn Represents the visual modality feature value and the tactile modality feature value, σ represents the Fsigmoid function, and E' Vn and E' Tn U represents the excitation factors for vision and touch, respectively. Vn and U Tn These represent the original visual and tactile features, respectively.

[0088] S3. The robot grasps the object to be grasped based on the grasping state recognition result.

[0089] Specifically, after the output Y is executed, the next step after obtaining the grasping state is to control the width and force of the two-finger gripper, as follows:

[0090] When the state is non-contact or sliding, the width of the gripper decreases and the gripping force increases;

[0091] When the state is appropriate and there is no slippage, the width and force of the gripper will remain unchanged;

[0092] When the state is deformed, the width of the gripper increases and the gripping force decreases.

[0093] Furthermore, it should be noted that the proposed method for predicting deformation and slippage of weakly stiff target objects eliminates the need for rewriting programs to adaptively grip weakly stiff target objects. This is because the method integrates adaptive, moderate, and non-slippage gripping of the target object. The final output of the gripping state of the weakly stiff target object is obtained in the first problem, and the gripper's gripping state is then automatically adjusted. In this process, a closed-loop adaptive control is formed, from sensing the state of the gripping target object to automatically adjusting the width and force of the gripper to control the uncertain deformation and slippage of the target object. This process eliminates the need for traditional gripping threshold control, and the method can be transferred to another gripping environment and task to achieve programming-free control.

[0094] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A weak-rigidity object grasping method based on VTF visual-tactile information interaction, characterized in that, The method comprises the following steps: obtaining image information and tactile signal of an object to be grasped; inputting the image information into a visual feature extraction module, the visual feature extraction module comprising a convolution layer, a regularization layer, a max-pooling layer and a visual residual block; the visual residual block comprises a first visual residual block, a second visual residual block, a third visual residual block and a fourth visual residual block; performing convolution processing on the image information based on the convolution layer to obtain a feature extraction image; performing regularization processing on the feature extraction image based on the regularization layer to obtain a regularized feature image; performing down-sampling processing on the regularized feature image based on the max-pooling layer to obtain a down-sampled feature image; performing residual analysis on the down-sampled feature image based on the visual residual block to obtain visual modality information, the visual modality information comprising first visual modality information, second visual modality information, third visual modality information and fourth visual modality information; performing splitting processing on the tactile signal to obtain split tactile signal information, the split tactile signal information comprising first tactile signal information, second tactile signal information, third tactile signal information and fourth tactile signal information; inputting the split tactile signal information into a tactile feature extraction module, the tactile feature extraction module comprising a first tactile residual block, a second tactile residual block, a third tactile residual block and a fourth tactile residual block; performing convolution processing on the first tactile signal information based on the first tactile residual block to obtain first tactile modality information, the first tactile residual block comprising one convolution layer; performing convolution and fusion processing on the second tactile signal information, the third tactile signal information and the fourth tactile signal information based on the second tactile residual block, the third tactile residual block and the fourth tactile residual block respectively to obtain second tactile modality information, third tactile modality information and fourth tactile modality information; the second tactile residual block, the third tactile residual block and the fourth tactile residual block each comprise two convolution layers; integrating the first tactile modality information, the second tactile modality information, the third tactile modality information and the fourth tactile modality information to obtain tactile modality information; transmitting the visual modality information and the tactile modality information to a visual-tactile feature fusion module; performing global average pooling operation processing on the visual modality information and the tactile modality information based on a Fsq function to obtain visual modality pooling values and tactile modality pooling values; performing full connection processing on the visual modality pooling values and the tactile modality pooling values based on a Fex function to obtain visual modality full connection values and tactile modality full connection values; performing point multiplication operation processing on the visual modality full connection values and the tactile modality full connection values based on a Fsigmoid function to obtain visual modality feature values and tactile modality feature values; performing fusion processing on the visual modality feature values and the tactile modality feature values through a full connection layer to obtain a grasping state recognition result; a robot grasps the object to be grasped according to the grasping state recognition result.

2. The weak-rigidity object grasping method based on VTF visual-tactile information interaction according to claim 1, characterized in that, The step of obtaining image information and tactile signal of an object to be grasped specifically comprises: capturing the object to be grasped through a camera to obtain an image of the object to be grasped; obtaining corresponding image information based on the image of the object to be grasped; A tactile signal of an image of the object to be grabbed is acquired by a tactile sensor.

3. The weak-rigidity object grasping method based on VTF visual-haptic information interaction according to claim 2, characterized in that, A calculation formula of the Fsq function for performing a global average pooling operation on the visual modal information and the tactile modal information is as follows: ； In the above formulae, and denote visual modality information and haptic modality information, respectively, and denote visual modality pooled values and haptic modality pooled values, respectively, denotes the layer number of the visual or haptic feature tensor, and denote the length and width of the tensor, respectively.

4. The weak-rigidity object grasping method based on VTF visual-haptic information interaction according to claim 3, characterized in that, A calculation formula of the Fex function for performing a full connection operation on the visual modal pooling value and the tactile modal pooling value is as follows: ； In the above formula, represents the excitation signal, and respectively represent the visual modality full connection value and the tactile modality full connection value, represents the weight of the visual feature, represents the weight of the tactile feature, represents the bias of the visual feature, represents the bias of the tactile feature, represents the fused visual-tactile bias.

5. The weak-rigidity object grasping method based on VTF visual-tactile information interaction according to claim 4, characterized in that, A calculation formula of the Fsigmoid function for performing a point multiplication operation on the visual modal full connection value and the tactile modal full connection value is as follows: ； In the above formulae, and denote the visual and haptic modality feature values, denotes the Fsigmoid function, and denote the visual and haptic excitation factors, and denote the raw visual and haptic features.

6. The weak-rigidity object grasping method based on VTF visual-tactile information interaction according to claim 5, characterized in that, The grabbing state recognition result specifically includes a non-contact state, a proper contact state, an excessive contact state, a non-slippage state, and a slippage state.

Images