Robotic arm autonomous grabbing method based on vision and touch fusion under weak-rigidity characteristic condition

Abstract

Disclosed in the present invention is a robotic arm autonomous grabbing method based on vision and touch fusion under a weak-rigidity characteristic condition. The method comprises: acquiring a visual image and touch data, and performing data preprocessing, to obtain visual information and touch information; performing fusion processing on the visual information and the touch information to obtain feature information; performing classification processing on the feature information on the basis of a long short-term memory neural network, to obtain a classification result; and according to the classification result, monitoring and controlling the state of a robotic arm autonomously grabbing an object having a weak-rigidity characteristic. By using the present invention, the state of a robotic arm grabbing an object can be mastered more accurately. The present robotic arm autonomous grabbing method based on vision and touch fusion under a weak-rigidity characteristic condition can be widely applied to the technical field of robot control.

Description

Autonomous grasping method of robotic arm based on vision-touch fusion under weak stiffness conditions Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to an autonomous grasping method of a robotic arm based on vision-touch fusion under conditions of weak stiffness characteristics. Background Art

[0002] With the rapid development of sensor technology and deep learning technology, the application of sensors and deep learning networks to robots has also made great progress. Among them, robot target grasping mainly uses vision and touch in the task. Many current tactile fusion systems will use various grasping experiments to test the quality of the perception system. However, for objects to be grasped under conditions of variability or weak stiffness characteristics, the existing technology is more about judging whether the grasped object will slip, and there is no good judgment on the stiffness and shape of the grasped object. In addition, the existing technology has problems such as insufficient consideration of the feature fusion of visual technology and tactile technology for robot grasping objects, which makes it difficult for existing robot manipulators to grasp objects more friendly and compliantly.

[0003] Summary of the Invention

[0004] In order to solve the above technical problems, the purpose of the present invention is to provide an autonomous grasping method of a robotic arm based on vision-touch fusion under conditions of weak stiffness characteristics, which can more accurately grasp the state of the robotic arm grasping the object.

[0005] The first technical solution adopted by the present invention is: a robotic arm autonomous grasping method based on vision-touch fusion under weak stiffness characteristics, comprising the following steps:

[0006] Acquire visual images and tactile data and perform data preprocessing to obtain visual information and tactile information;

[0007] Fuse visual information and tactile information to obtain feature information;

[0008] Classify the feature information based on the long short-term memory neural network to obtain the classification results;

[0009] Based on the classification results, the state of the robotic arm in autonomously grasping objects with weak stiffness characteristics is monitored and controlled.

[0010] Furthermore, the step of acquiring visual images and tactile data and performing data preprocessing to obtain visual information and tactile information specifically includes:

[0011] Acquiring visual images and tactile data from the end of the robot's mechanical arm, wherein the visual images are obtained by a visual camera installed at the end of the robot's mechanical arm, and the tactile data are obtained by a tactile sensor installed at the end of the robot's mechanical arm;

[0012] Inputting the visual image and tactile data into a feature extraction module for feature extraction processing, wherein the feature extraction module includes a CNN convolutional neural network model and a CL convolutional neural network model;

[0013] Based on the CNN convolutional neural network model, visual features of the visual image are extracted and processed to obtain visual information;

[0014] The tactile data is processed by convolution operation based on the CL convolutional neural network model to obtain tactile information.

[0015] Furthermore, the expression of the feature extraction process of the feature extraction module is as follows: S(X V n)=X V nX V 0 M V n=E(S(X V n)),M T n=CL(X T n)

[0016] In the above formula, E(·) represents the feature extraction operation of the CNN convolutional neural network model, CL(·) represents the convolution operation of the CL convolutional neural network model, S(·) represents the difference processing of the visual image, and X V Indicates that the visual image is collected, X V 0 represents the collected visual image frame 0, M V Represents visual information, M T represents the tactile information, H, W, and C represent the length, width, and number of channels of the feature map, and n represents the number of frames of visual and tactile information.

[0017] Furthermore, the step of fusing the visual information and the tactile information to obtain feature information specifically includes:

[0018] Slice and reconstruct the visual and tactile information to obtain preliminary feature information;

[0019] Based on the SE attention mechanism network model, the preliminary feature information is assigned weight parameters and evaluated to obtain feature information.

[0020] Furthermore, the step of slicing and recombining the visual information and the tactile information to obtain preliminary feature information specifically includes:

[0021] Encoding the visual information and the tactile information according to a preset spatial position to obtain a feature slice vector of the visual information and a feature slice vector of the tactile information;

[0022] Recombining the feature slice vector of the visual information and the feature slice vector of the tactile information to obtain a recombined slice vector;

[0023] Perform feature extraction on the reorganized slice vector to obtain preliminary feature information.

[0024] Furthermore, the step of assigning weight parameters to the preliminary feature information and performing evaluation processing based on the SE attention mechanism network model to obtain feature information specifically includes:

[0025] Input the preliminary feature information into the SE attention mechanism network model, wherein the SE attention mechanism network model includes a pooling layer, a perceptron layer, and a layer;

[0026] Based on the pooling layer of the SE attention mechanism network model, the preliminary feature information is globally averaged pooled to obtain global feature information;

[0027] Based on the perceptron layer of the SE attention mechanism network model, the global feature information is processed by nonlinear feature transformation to obtain the weight parameters of the global feature information;

[0028] Based on the layer of the SE attention mechanism network model, the weight parameters of the global feature information are recalibrated to obtain feature information.

[0029] Furthermore, the step of classifying the feature information based on the long short-term memory neural network to obtain the classification result specifically includes:

[0030] Inputting feature information into a long short-term memory neural network, wherein the long short-term memory neural network includes an encoding layer, a fully connected layer, and a classification layer;

[0031] The encoding layer based on the long short-term memory neural network encodes the feature information to obtain the feature vector;

[0032] The fully connected layer of the long short-term memory neural network is used to reduce the dimension of the feature vector to obtain the reduced dimension feature vector;

[0033] The classification layer based on the long short-term memory neural network classifies the feature vector after dimensionality reduction to obtain the classification result.

[0034] The beneficial effects of the method of the present invention are as follows: in the process of grasping objects at the end of the robot manipulator arm, the present invention introduces visual information and tactile information, further fuses the visual information and tactile information, slices and reorganizes the feature information of the visual information and tactile information, and recalibrates it based on the weight of each information. This can more accurately grasp the state of the manipulator arm in the process of grasping objects, and finally classifies the state of the manipulator arm in the grasping process through the LSTM neural network, so as to timely predict whether deformation and slippage will occur in the process of the manipulator arm grasping objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] FIG1 is a flowchart of the steps of the autonomous grasping method of a robotic arm based on vision-touch fusion under the condition of weak stiffness characteristics of the present invention;

[0036] FIG2 is a schematic diagram of the overall framework based on Resnet50+SE+LSTM proposed in the present invention;

[0037] FIG3 is a schematic diagram of a conventional fusion operation of visual and tactile information;

[0038] FIG4 is a schematic diagram of slicing and reorganizing visual tactile information according to the present invention;

[0039] FIG5 is a flow chart of the autonomous grasping method of a robotic arm based on vision-touch fusion according to the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0041] 1, 2 and 5, the present invention provides a method for autonomous grasping of a robotic arm based on visual-touch fusion under conditions of weak stiffness characteristics, the method comprising the following steps:

[0042] S1. Acquire visual images and tactile data and perform data preprocessing to obtain visual information and tactile information;

[0043] Specifically, the visual image and tactile data are obtained, wherein the process of obtaining the visual information is to obtain the image difference by subtracting the image of the 0th frame (representing the initialization visual information of the gripper) from the collected multi-frame image (3x224x224), and then extract the features of the modified Resnet50 network to obtain the (256x4x4) visual features. The process of obtaining the tactile information is to obtain the (256x4x4) tactile features by passing the collected multi-frame tactile data (3x4x4) through a convolution layer (in_channels=3, out_channels=256, kernel_size=3, stride=1, padding=1). The visual image is obtained by the visual camera installed at the end of the robot's mechanical arm, and the tactile data is obtained by the tactile sensor installed at the end of the robot's mechanical arm. Further, the visual feature M V Extracted by Resnet50, the corresponding image collected is subtracted from the image of frame 0 (the initial image of the visual sensor) as input, that is, the visual camera will collect multiple frames of images as input, frame 0 is used as the initial visual information, and each collected image must be subtracted from the first collected image. For the tactile sensor, the output of the 4x4x3 tactile data after a layer of volume (CL) (in_channels = 3, out_channels = 256, kernel_size = 3, stride = 1, padding = 1) is used as the tactile feature M T , the purpose is to increase the number of channels of tactile information to facilitate the later feature fusion, wherein the feature extraction process of the feature extraction module is expressed as follows: S(X V n)=X V nX V 0 M V n=E(S(X V n)),M T n=CL(X T n)

[0044] In the above formula, E(·) represents the feature extraction operation of the CNN convolutional neural network model, CL(·) represents the convolution operation of the CL convolutional neural network model, S(·) represents the difference processing of the visual image, and X V Indicates that the visual image is collected, X V 0 represents the collected visual image frame 0, M V Represents visual information, M T represents the tactile information, H, W, and C represent the length, width, and number of channels of the feature map, and n represents the number of frames of visual and tactile information.

[0045] S2, fusing the visual information and the tactile information to obtain feature information;

[0046] Specifically, the main function of visual-tactile feature fusion is to create a learnable representation based on visual and tactile features. Currently, most visual and tactile information fusion methods use simple feature splicing, as shown in Figure 3. However, this fusion method cannot well represent the differences and correlations between the two modalities. Due to its simplicity, direct fusion is still the mainstream method for visual-tactile modality fusion.

[0047] The fusion method proposed in this solution is different from the direct fusion method. The feature extraction is divided into two stages. The first is to perform "slicing and reorganization" on the visual and tactile features, reordering different modes of the same state to obtain early features. Secondly, the SE attention mechanism is performed on the reorganized early features to obtain the final learnable features.

[0048] S21, slicing and recombining the visual information and the tactile information to obtain preliminary feature information;

[0049] Specifically, the initial visual-tactile information fusion, given the visual information M V and tactile information M T , based on them to build early visual tactile fusion Its expression is as follows:

[0050] In the above formula, Represents a slice reassembly operation, Indicates preliminary feature information;

[0051] Referring to Figure 4, let p and q be features M V and M T The spatial position in ,p,q∈{1,2,…,H×W}, v in Figure 4 p Represents the feature M at spatial position p V Similarly, t q Represents the feature M at the spatial position q T Therefore, the features that define the early fusion of spatial positions p and q are as follows: L pq =Concate(v p ,t q )

[0052] In the above formula, Concate(·) represents the concatenation operation of two slice vectors;

[0053] The initial fusion feature vector pair M V At specific positions p and M T The specific position q is encoded, and the encoded features The total number of channels is C V +C T , the length and width are both H V ×H T , therefore, the preliminary features are expressed as

[0054] S22, assigning weight parameters to the preliminary feature information based on the SE attention mechanism network model and evaluating it to obtain feature information;

[0055] Specifically, each channel of the initial fused feature block may contain a lot of redundant information, which is further enhanced and extracted by the SE attention mechanism to facilitate the evaluation of the grasping task. It adds different weight parameter information to each channel of the original feature map by squeezing and excitation. The operation expression of the SE attention mechanism network model is as follows:

[0056] In the above formula, SE(·) represents the operation function of the SE attention mechanism network model;

[0057] The SE attention mechanism network model includes a Squeeze part, whose operation is to first compress the reorganized features into 1 real number through global average pooling, and change the feature map (16x16x512) to (1x1x512); the Excitation part, whose operation is to generate a weight value for each channel of the feature, and build the correlation between the channels through the full connection of two layers, and (1x1x512) becomes (1x1x512); the Scale part, whose operation is to multiply the normalized weight obtained above by the weight coefficient channel by channel, that is, (16x16x512)*(1x1x512) becomes (16x16x512);

[0058] The SE attention mechanism first pays attention to the early Squeeze (B, C, H, W) through the global average pooling F sq (B, C, 1, 1), (B, C, 1, 1) is the weight extracted from each channel itself, and then passed through F ex (·)Module(F ex Specifically, FC+RELU+FC+Sigmoid) is used to obtain the weights of attention of different channels (B, C, 1, 1), and then the weights of attention of different channels are applied to Each channel is multiplied by the corresponding weight on each channel, that is, F scale (·)Module gets

[0059] S3. Classify the feature information based on the long short-term memory neural network to obtain the classification result.

[0060] Specifically, for the LSTM module, we chose to use the Long Short-Term Memory (LSTM) network. LSTM has been widely used to process video data, which is very similar to our task. After SE attention mechanism, we get After passing through a fully connected layer, 64 visual-tactile cross-modal features can be obtained. These 64 features are used as input to the LSTM. The LSTM has two hidden layers, both with 64 hidden layers. The last part of the model is the classification module, which classifies the output of the LSTM module as input and outputs the classification result. The processing expression of the long short-term memory neural network is as follows:

[0061] In the above formula, FC(·) represents a fully connected layer, LSTM(·) represents an LSTM with two hidden layers, and Y represents the classification result, where Y = 0, 1, 2, 3, and 4 represent the results of no contact, appropriate contact, excessive contact, no slip, and slip, respectively.

[0062] Furthermore, after obtaining the results, the next step is to control the width and strength of the two-finger gripper. Specifically, when the state is non-contact or slipping, the width of the gripper becomes smaller and the strength of the gripper becomes larger; when the state is appropriate and there is no slipping, the width and strength of the gripper will remain unchanged; when the state is deformed, the width of the gripper becomes larger and the strength of the gripper becomes smaller.

[0063] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. An autonomous grasping method of a robotic arm based on visual-touch fusion under weak stiffness characteristics, characterized in that: The following steps are involved: Acquire a visual image and tactile data of the end of the robot's mechanical arm, wherein the visual image is obtained by a visual camera installed at the end of the robot's mechanical arm, and the tactile data is obtained by a tactile sensor installed at the end of the robot's mechanical arm; Inputting the visual image and the tactile data into a feature extraction module for feature extraction processing, wherein the feature extraction module includes a CNN convolutional neural network model and a CL convolutional neural network model; Based on the CNN convolutional neural network model, visual features of the visual image are extracted and processed to obtain visual information; Based on the CL convolutional neural network model, the tactile data is processed by convolution operation to obtain tactile information; Encoding the visual information and the tactile information according to a preset spatial position to obtain a feature slice vector of the visual information and a feature slice vector of the tactile information; Recombining the feature slice vector of the visual information and the feature slice vector of the tactile information to obtain a recombined slice vector; Perform feature extraction on the reorganized slice vector to obtain preliminary feature information, which specifically includes: Initial visual-tactile information fusion, given the visual information M V and tactile information M T , based on them to build early visual-tactile fusion Its expression is as follows: In the above formula, Represents a slice reorganization operation, Indicates preliminary feature information; Let p, q be the features M V and M T The spatial position in, p, q∈{1,2,…,H×W}, v p Represents the feature M at spatial position p V The vector of q Represents the feature M at the spatial position q T Therefore, the features that define the early fusion of spatial positions p and q are as follows: L pq =Concate(v p ,t q ) In the above formula, Concate(·) represents the concatenation operation of two slice vectors; The initial fusion feature vector pair M V The specific position p and M T The specific position q is encoded in the , and the encoded features The total number of channels is C V +C T , the length and width are H V ×H T , W V ×W T , therefore, the preliminary features are expressed as Based on the SE attention mechanism network model, the preliminary feature information is assigned weight parameters and evaluated to obtain feature information; The feature information is classified based on the long short-term memory neural network to obtain a classification result. Includes no contact, proper contact, excessive contact, no slip and slip; According to the classification results, the state of the robot arm in autonomously grasping objects with weak stiffness characteristics is monitored and controlled.

2. The method for autonomous grasping of a robot arm based on visual-touch fusion under the condition of weak stiffness characteristics according to claim 1 is characterized in that: The expression of the feature extraction process of the feature extraction module is as follows: S(X V n)=X V nX V 0 M V n=E(S(X V n)),M T n=CL(X T n) In the above formula, E(·) represents the feature extraction operation of the CNN convolutional neural network model, CL(·) represents the convolution operation of the CL convolutional neural network model, S(·) represents the difference processing of the visual image, and X V Indicates that the visual image is collected, X V 0 represents the 0th frame of visual image collected, M V Represents visual information, M T represents tactile information, H, W and C represent the length, width and number of channels of the feature map, and n represents the number of frames of visual and tactile information.

3. The method for autonomous grasping of a robot arm based on visual-touch fusion under the condition of weak stiffness characteristics according to claim 2 is characterized in that: The step of assigning weight parameters to the preliminary feature information based on the SE attention mechanism network model and evaluating the feature information to obtain the feature information specifically includes: Inputting preliminary feature information into the SE attention mechanism network model, wherein the SE attention mechanism network model includes a Squeeze layer, an Excitation layer, and a Scale layer; Based on the Squeeze layer of the SE attention mechanism network model, the preliminary feature information is processed by global average pooling to obtain global feature information; Based on the Excitation layer of the SE attention mechanism network model, the global feature information is processed by nonlinear feature transformation to obtain the weight parameters of the global feature information; Based on the Scale layer of the SE attention mechanism network model, the weight parameters of the global feature information are recalibrated to obtain the feature information.

4. The method for autonomous grasping of a robot arm based on visual-touch fusion under the condition of weak stiffness characteristics according to claim 3 is characterized in that: The step of classifying the feature information based on the long short-term memory neural network to obtain the classification result specifically includes: Inputting feature information into a long short-term memory neural network, wherein the long short-term memory neural network includes an encoding layer, a fully connected layer, and a classification layer; The encoding layer based on the long short-term memory neural network encodes the feature information to obtain a feature vector; Based on the fully connected layer of the long short-term memory neural network, the feature vector is reduced in dimension to obtain the feature vector after dimension reduction; The classification layer based on the long short-term memory neural network classifies the feature vector after dimension reduction to obtain the classification result.

Images