Method for evaluating grasping stability based on visual-tactile fusion perception and multi-modal spatio-temporal convolution

Abstract

The present application belongs to the technical field of robot intelligent grasping, and is a grasping stability evaluation method based on visual and tactile fusion perception and multi-modal spatio-temporal convolution, comprising the following steps: step A01: establishing a robot simulation grasping platform based on Pybullet and TACTO; step A02: collecting virtual grasping and visual and tactile images according to the robot simulation grasping platform established in step A01 to obtain a multi-modal grasping dataset; step A03: establishing a static visual and tactile grasping stability analysis model; step A04: collecting dynamic prior data according to the simulation grasping platform built in step A01 to obtain a simulation grasping object dataset; step A05: constructing a spatio-temporal convolution grasping stability evaluation model; and step A06: training and verifying a dynamic visual and tactile fusion model. Through the mutual complementation of tactile modal information and visual modal information, the accuracy and stability of grasping are improved. Grasping stability evaluation helps to improve the grasping success rate and work efficiency of robots, and plays an important role in the efficient and successful implementation of robot grasping tasks.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robotic grasping technology, specifically involving a grasping stability evaluation method based on visual-touch fusion perception and multimodal spatiotemporal convolution. Background Technology

[0002] Currently, robots need to maintain a stable grasping posture and clamping force when facing interference or errors during the grasping process, in order to ensure the success rate and stability of grasping.

[0003] Current methods for analyzing grasping stability can be broadly categorized into two types: analytical methods and data-driven methods. Analytical methods rely on known physical models of the object, environment, and gripper to construct the grasp and infer its mass, ensuring that the object satisfies force and shape closure. However, analytical methods depend on accurate state estimation of the object's posture and correct modeling assumptions, resulting in poor prediction of grasping stability in real-world environments. Data-driven methods can effectively plan the grasping position from image or point cloud data and can reason about grasping configurations for various objects in stacked environments. However, methods relying on visual modalities are often open-loop, meaning that the robot's motion state remains unchanged after the grasping configuration is determined, making it highly susceptible to unexpected events. To achieve effective closed-loop methods for robot grasping and thus stable grasping, extensive research has incorporated tactile perception into robots.

[0004] In summary, vision cannot fully perceive the contact surface characteristics of the object being grasped, changes in the object's pose, and changes in contact force; tactile methods often require multiple attempts, making it difficult to achieve efficient grasping tasks.

[0005] Based on this, we researched and developed a grasping stability evaluation method based on visual-touch fusion perception and multimodal spatiotemporal convolution. Summary of the Invention

[0006] This invention provides a grasping stability evaluation method based on visual-tactile fusion perception and multimodal spatiotemporal convolution to solve the problems of low accuracy of visual perception and poor stability of tactile perception in existing grasping stability evaluation methods. At the same time, it provides a method that can be used as a reference for achieving high accuracy and stability evaluation of robot grasping stability.

[0007] This invention is achieved through the following technical solution:

[0008] A method for evaluating the stability of grasping based on visual-touch fusion perception and multimodal spatiotemporal convolution includes the following steps:

[0009] Step A01: Establish a robot simulation grasping platform based on Pybullet and TACTO;

[0010] Step A02: Collect virtual grasping and visual-touch images based on the robot simulation grasping platform established in Step A01 to obtain a multimodal grasping dataset;

[0011] Step A03: Establish a static visual-touch grasping stability analysis model;

[0012] Step A04: Collect dynamic prior data based on the simulation grasping platform built in step A01 to obtain the simulation grasping object dataset;

[0013] Step A05: Construct a spatiotemporal convolutional grasping stability evaluation model;

[0014] Step A06: Train and validate the dynamic visual-touch fusion model.

[0015] Optionally, step A01 involves establishing a robot simulation grasping platform based on Pybullet and TACTO, including the following steps:

[0016] Step A011: Simulate and configure the corresponding tactile sensor based on the visual sensor simulator TACTO;

[0017] Step A012: Use Pybullet to simulate the physical behavior of the robot grasping;

[0018] Step A013: Build a grasping scenario in a simulation environment and establish a robot simulation grasping platform. The grasping scenario includes models of a robotic arm, gripper, camera, grasped object, and tactile sensor.

[0019] Optionally, in step A02, virtual grasping and visual-touch images are collected based on the robot simulation grasping platform established in step A01 to obtain a multimodal grasping dataset, including the following steps:

[0020] Step A021: Obtain the simulated grasping object of the EGAD model;

[0021] Step A022: Collect virtual grasping and visual / touch images based on the robot simulation grasping platform;

[0022] Step A023: Load the model of the robot simulation grasping platform, adjust the state of the simulated grasping object, and obtain the data collected by the robot simulation grasping platform to simulate grasping actions.

[0023] Optionally, step A022 involves acquiring virtual grasping and visual / touch images based on the robot simulation grasping platform, including the following steps:

[0024] Step A0221: Initialize Pybullet, create a renderer, and create a light sensor and a haptic sensor in the renderer;

[0025] Step A0222: Load and parse the URDF file, add the mesh file, and finally add it to the renderer;

[0026] Step A0222: Use Pybullet for step simulation, TACTO to synchronize the pose of the simulated grasped object, and finally synchronize the pose of the simulated grasped object to the renderer.

[0027] Optionally, step A023: Loading the model of the robot simulation grasping platform, adjusting the state of the simulated grasping object, and obtaining the data collected by the robot simulation grasping platform's simulated grasping actions, includes the following steps:

[0028] Step A0231: After loading the model of the robot simulation grasping platform, the robot gripper is initialized to the open state. The simulation grasping state is adjusted by first placing the simulated grasping object in the initial position, and then the simulated grasping object falls freely to the ground.

[0029] Step A0232: After the simulated grasped object's posture stabilizes, drive the motor to move the robot gripper to the predetermined position. Then, the robot gripper closes. After the robot gripper stabilizes, use Pybullet to record the current camera's viewpoint, the contact normal force information between the light source sensor, the tactile sensor, and the simulated grasped object, as well as the current pose information of the simulated grasped object. Collect the current RGB image, depth image, and contact force data, and then perform DIGIT image rendering. Collect the RGB image and depth image from the light source sensor and the tactile sensor. Then, the robotic arm rises at a constant speed of 20cm. After the robotic arm stabilizes, obtain the pose information of the simulated grasped object at the current moment. If the height difference along the z-axis from the initial moment is less than a threshold, it is confirmed as a grasping failure, and the grasping label for this set of data is recorded as 0; otherwise, it is confirmed as a grasping success, and the grasping label for this set of data is recorded as 1.

[0030] Step A0233: During the grasping process, the robot gripper updates the position and orientation information of the simulated grasped object. The driving position of the robot arm is increased with random perturbation around the centroid of the grasped simulated grasped object, and random rotation around the z-axis is added to the vertical grasping. The gripping force of the robot gripper is randomly updated. The gripping force of the robot gripper is in the range of 5 to 20 N.

[0031] Optionally, step A03 establishes a static visual-touch gripping stability analysis model, including the following steps:

[0032] A031: The ResNet-18 model pre-trained on ImageNet is used as the feature extractor of the input image, and the one-dimensional vector output by the ResNet-18 model is concatenated. Two fully connected layers are used to predict whether the simulated grasped object will fall after the grasping action is completed.

[0033] A032: Training and validation of the network model, using a 5-fold cross-validation dataset for evaluation. The ratio of training to test sets is 8:2. The network model is trained using the Adam optimizer with a learning rate of 5e. -4 The batch size was set to 32 and the epoch size to 10; both the visual and tactile images were scaled and cropped to 224x224.

[0034] Optionally, step A04 collects dynamic prior data based on the simulation grasping platform built in step A01 to obtain a simulation grasping object dataset. Specifically, when collecting dynamic prior data, the lifting speed of the robotic arm, camera resolution, frame rate, and DIGIT acquisition resolution and frame rate are set. After the robotic arm gripper is stably closed, data is collected for 1 second, and the RGB image, left DIGIT image, right DIGIT image, gripping force, and grasping label at the time of data collection are recorded to obtain a simulation grasping object dataset during the basic lifting of the robotic arm.

[0035] Optionally, step A05: Constructing a spatiotemporal convolutional grasping stability evaluation model, including the following steps:

[0036] A051: Input visual image sequence (V1, V2, ..., V...) n ) and tactile image sequences (T1, T2, ..., T n ), establish a deep neural network model, and confirm the sequence F(V1, V2, ... V n T1, T2, ... T n Whether an unstable sliding has occurred, where n is the sequence length and F is the classifier. When F outputs 0, the current time step is in a sliding state; when F outputs 1, the current time step is in a stable state.

[0037] A052: The classifier F is fitted by the deep neural network model. The deep neural network model takes visual image sequences and tactile image sequences as inputs and sliding probabilities as outputs. The network framework of the deep neural network model includes a spatial feature extraction module, a temporal feature extraction module, and a multilayer perceptron classifier, thereby obtaining a spatiotemporal convolutional grasping stability evaluation model.

[0038] Optionally, step A052 includes the following steps:

[0039] Step A0521: Construct a spatial feature extraction module based on the ResNet-50 network to extract two-dimensional spatial features from the video of the simulated object grasping dataset;

[0040] Step A0522: Construct a temporal feature extraction module using a Longformer structure with an attention mechanism;

[0041] Step A0523: The multilayer perceptron classifier outputs the sliding probability. The multilayer perceptron classifier includes GELU, layer normalization, and Dropout.

[0042] Optionally, step A06: Training and validating the dynamic vision-touch fusion model includes the following steps:

[0043] Step A061: Divide the simulated grasping object dataset into a training set and a validation set. During training, the visual modal information is first cropped. In the grasping stability evaluation task, the visual modal information includes the end-execution module and the state of the simulated grasping object being grasped. The visual modal information is located in the lower middle part of the camera image.

[0044] Step A062: During training, the key information of the visual modality is cropped to 600x320x3 and scaled down to 256x256x3. The tactile image includes left finger data and right finger data. The tactile image is stitched along the width direction and then rotated 90° counterclockwise to merge into a 480x320x3 image. The visual and tactile images are then uniformly scaled down to 224x224x3.

[0045] Step 063: The model is trained using the Adam optimizer and gradient backpropagation. The time series length of the visual images is 8 frames, and the batch size is set to 2 during training and validation. Data augmentation is performed on the training data, and 8 frames are randomly selected from the video sequence. Pre-training enhancement is used to accelerate the training process, with an epoch set to 20 and a learning rate set to 10. -3 The attention window size is set to 8, and the dropout probability is set to 0.1.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] The grasping stability evaluation method based on visual-touch fusion perception and multimodal spatiotemporal convolution of the present invention has the following advantages: The present invention solves the problems of low accuracy of visual perception and poor stability of tactile perception in existing grasping stability evaluation methods, and provides a reference method for achieving high accuracy and stability evaluation of robot grasping stability. Attached Figure Description

[0048] Figure 1 This is a software architecture diagram for TACTO;

[0049] Figure 2 A workflow diagram for TACTO;

[0050] Figure 3 Validate the dataset for EGAD;

[0051] Figure 4 Example image of DIGIT simulation;

[0052] Figure 5 This is a diagram of a network model for evaluating the stability of visual-touch fusion grasping based on static images.

[0053] Figure 6 This is a network architecture diagram for video-based crawling stability evaluation.

[0054] Figure 7 The diagram shows a network model for evaluating the stability of video-based web scraping.

[0055] Figure 8 Attention pattern diagram;

[0056] Figure 9 The training accuracy curve of the spatiotemporal convolutional network is shown in the figure.

[0057] Figure 10 A training set loss curve for a spatiotemporal convolutional network;

[0058] Figure 11 The loss curves for the training and validation sets of the spatiotemporal convolutional network are shown.

[0059] Figure 12 This is a flowchart illustrating the technical solution of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments. The illustrative embodiments and descriptions of this invention are only for explaining this invention and are not intended to limit this invention.

[0061] Example:

[0062] like Figure 1-12 As shown, the grasping stability evaluation method based on visual-touch fusion perception and multimodal spatiotemporal convolution includes the following steps:

[0063] Step A01: Establish a robot simulation grasping platform based on Pybullet and TACTO;

[0064] Step A02: Collect virtual grasping and visual-touch images based on the robot simulation grasping platform established in Step A01 to obtain a multimodal grasping dataset;

[0065] Step A03: Establish a static visual-touch grasping stability analysis model;

[0066] Step A04: Collect dynamic prior data based on the simulation grasping platform built in step A01 to obtain the simulation grasping object dataset;

[0067] Step A05: Construct a spatiotemporal convolutional grasping stability evaluation model;

[0068] Step A06: Train and validate the dynamic visual-touch fusion model.

[0069] Step A01, which involves establishing a robot simulation grasping platform based on Pybullet and TACTO, includes the following steps:

[0070] Step A011: Based on the vision sensor simulator TACTO, simulate and configure the corresponding tactile sensor. TACTO is a vision-based vision sensor simulator, and the vision sensor can be DIGIT or OmniTact.

[0071] Step A012: Apply Pybullet to simulate the physical behavior of robot grasping, so as to improve the accuracy of collision detection and dynamic model rendering;

[0072] Step A013: Build a grasping scenario in a simulated environment and establish a robot simulation grasping platform. The grasping scenario of the robot simulation grasping platform includes a robotic arm, the robotic arm's gripper, a camera, the object to be grasped, and a tactile sensor model. The object to be grasped can be understood as a simulated object to be grasped.

[0073] In step A02, virtual grasping and visual-touch images are collected based on the robot simulation grasping platform established in step A01 to obtain a multimodal grasping dataset, including the following steps:

[0074] Step A021: Obtain the simulated grasping object of the EGAD model;

[0075] Step A022: Collect virtual grasping and visual / touch images based on the robot simulation grasping platform;

[0076] Step A023: Load the model of the robot simulation grasping platform, adjust the state of the simulated grasping object, and obtain the data collected by the robot simulation grasping platform to simulate grasping actions.

[0077] Step A021: Obtain the simulated grasping object of the EGAD model. The simulated grasping object is selected from the simulated grasping object of EGAD, which is the Evolved Grasping Analysis Dataset. There are a total of 2330 models in EGAD.

[0078] Optionally, step A022 involves acquiring virtual grasping and visual / touch images based on the robot simulation grasping platform, including the following steps:

[0079] Step A0221: Initialize Pybullet, create a renderer, and create a light sensor and a haptic sensor in the renderer;

[0080] Step A0222: Load and parse the URDF file, add the mesh file, and finally add it to the renderer;

[0081] Step A0222: Use Pybullet for step simulation, TACTO to synchronize the pose of the simulated grasped object, and finally synchronize the pose of the simulated grasped object to the renderer.

[0082] Optionally, step A023: Loading the model of the robot simulation grasping platform, adjusting the state of the simulated grasping object, and obtaining the data collected by the robot simulation grasping platform's simulated grasping actions, includes the following steps:

[0083] Step A0231: After loading the model of the robot simulation grasping platform, the robot gripper is initialized to the open state. The simulation grasping state is adjusted by first placing the simulated grasping object in the initial position, and then the simulated grasping object falls freely to the ground.

[0084] Step A0232: After the simulated object's posture stabilizes, drive the motor to move the robot gripper to the predetermined position. Then, the gripper closes. Once the gripper's holding state is stable, use Pybullet to record the current camera's viewpoint, the contact normal force between the light sensor, tactile sensor, and the simulated object, as well as the current pose of the simulated object. Acquire the current RGB image, depth image, and contact force data. Then, perform DIGIT image rendering, acquiring the RGB and depth images from the light sensor and tactile sensor. The robotic arm then rises uniformly by 20cm. After the robotic arm stabilizes, acquire the pose information of the simulated object at the current moment. If the height difference along the z-axis from the initial moment is less than a threshold, the grasp is considered a failure, and the grasp label for this set of data is recorded as 0; otherwise, the grasp is considered successful, and the grasp label for this set of data is recorded as 1. The acquired image data is as follows: Figure 4 As shown;

[0085] Step A0233: During the grasping process, the robot gripper updates the position and orientation information of the simulated grasped object. The driving position of the robot arm is increased with random perturbation around the centroid of the grasped simulated grasped object, and random rotation around the z-axis is added to the vertical grasping. The gripping force of the robot gripper is randomly updated. The gripping force of the robot gripper is in the range of 5 to 20 N.

[0086] Optionally, step A03 establishes a static visual-touch gripping stability analysis model, including the following steps:

[0087] A031: As Figure 5As shown, a ResNet-18 model pre-trained on ImageNet is used as the feature extractor for the input image, and the one-dimensional vector output by the ResNet-18 model is concatenated. Two fully connected layers are used to predict whether the simulated grasped object will fall after the grasping action is completed. Specifically, the two fully connected layers are fully connected hidden layers with dimensions of 512 and 256, respectively.

[0088] A032: Training and validation of the network model, using a 5-fold cross-validation dataset for evaluation. The ratio of training to test sets is 8:2. The network model is trained using the Adam optimizer with a learning rate of 5e. -4 The batch size was set to 32 and the epoch size to 10. Both the visual and tactile images were scaled and cropped to 224x224 to meet the input format requirements of the network model.

[0089] In this embodiment, virtual grasping and visual-tactile images are collected through the robot simulation grasping platform established in step A01 in step 2 (step A02) to obtain a multimodal grasping dataset. The obtained multimodal grasping dataset includes 49,000 data sets. Each data set includes a visual RGB image, a depth image, a left-finger tactile RGB image, a depth image, a right-finger tactile RGB image, a depth image, bi-finger normal contact force, gripper holding force, and a grasping stability label (i.e., a grasping label).

[0090] Optionally, step A04 involves collecting dynamic prior data based on the simulation grasping platform established in step A01 to obtain a simulation grasping object dataset. Specifically, when collecting the dynamic prior data, the lifting speed of the robotic arm is set to 3mm / s, the camera resolution to 640x480, and the frame rate to 30FPS. The acquisition resolution of the DIGIT camera is set to 320x240, and the frame rate to 30FPS. After the robotic arm gripper is stably closed, 1 second is captured, recording the RGB image, left DIGIT image, right DIGIT image, gripping force, and grasping label at the time of data acquisition, thus obtaining a simulation grasping object dataset during the basic lifting period of the robotic arm. For each object in the object dataset, 100 sets of grasping videos are collected.

[0091] FPS stands for Frames per Second.

[0092] Step A05: Constructing a spatiotemporal convolutional grasping stability evaluation model includes the following steps:

[0093] A051: Define the grasping stability analysis task as a binary classification problem, with the input visual image sequence (V1, V2, ..., V...). n ) and tactile image sequences (T1, T2, ..., T n), establish a deep neural network model, and confirm the sequence F(V1, V2, ... V n T1, T2, ... T n Whether an unstable sliding has occurred, where n is the sequence length and F is the classifier. When F outputs 0, the current time step is in a sliding state; when F outputs 1, the current time step is in a stable state.

[0094] A052: The classifier F is fitted by the deep neural network model. The deep neural network model takes visual image sequences and tactile image sequences as inputs and sliding probabilities as outputs. The network framework of the deep neural network model includes a spatial feature extraction module, a temporal feature extraction module, and a multilayer perceptron classifier, thereby obtaining a spatiotemporal convolutional grasping stability evaluation model.

[0095] Among them, the multilayer perceptron classifier is an MLP classifier, which stands for Multilayer Perceptron classifier.

[0096] Optionally, step A052 includes the following steps:

[0097] Step A0521: Construct a spatial feature extraction module based on the ResNet-50 network to extract two-dimensional spatial features from the video of the simulated object grasping dataset;

[0098] Step A0522: Construct a temporal feature extraction module using a Longformer structure with an attention mechanism;

[0099] Step A0523: The multilayer perceptron classifier outputs the sliding probability. The multilayer perceptron classifier includes GELU, layer normalization, and Dropout.

[0100] Step A0522 specifically involves: (e.g.) Figure 7 As shown, a Longformer architecture, which improves upon the traditional attention mechanism of Transformer, is used to construct the temporal feature extraction module. For each token, local attention is calculated only for nearby tokens within a fixed window size, and a small amount of global attention can be calculated based on the specific task. Figure 8 As shown, when the lengths of sequences Q and K are L, in the traditional Transformer self-attention, it is necessary to perform row-by-row dot product operations on Q and K, thus QK T ∈R L×L Its complexity is O(L). 2In Longformer's attention mechanism, for any row q in Q, only its dot product with the [qw / 2, q+w / 2] window in K needs to be performed, with a complexity of O(L×w). Here, w is the window length, whose value is independent of L and much smaller than L.

[0101] Step A0523: GELU stands for Gaussian Error Linear Unit. GELU is defined by the following formula:

[0102]

[0103] Optionally, step A06: Training and validating the dynamic vision-touch fusion model includes the following steps:

[0104] Step A061: Divide the simulated grasping object dataset into a training set and a validation set in a ratio of 8:2. During training, the visual modal information is first cropped. In the grasping stability evaluation task, the visual modal information includes the end-execution module and the state of the simulated grasping object being grasped. The visual modal information is located in the lower middle part of the camera image.

[0105] In step A061, when the visual modal information is cropped during training, since the camera is used as both a signal input source for grasp detection and a modal input for grasp stability evaluation, the information in different parts of the image space needs to meet the requirements of different tasks. In the grasp stability evaluation task, the visual modal information mainly includes the end effector and the state of the grasped object. This information is mainly concentrated in the lower middle part of the camera image.

[0106] Step A062: During training, the key information of the visual modality is cropped to 600x320x3 and scaled down to 256x256x3. The tactile image includes left finger data and right finger data. The tactile image is stitched along the width direction and then rotated 90° counterclockwise to merge into a 480x320x3 image. The visual and tactile images are then uniformly scaled down to 224x224x3.

[0107] Step 063: The model is trained using the Adam optimizer and gradient backpropagation. The time series length of the visual images is 8 frames, and the batch size is set to 2 during training and validation. Data augmentation is performed on the training data, and 8 frames are randomly selected from the video sequence. Pre-training enhancement is used to accelerate the training process, with an epoch set to 20 and a learning rate set to 10. -3 The attention window size is set to 8, and the dropout probability is set to 0.1.

[0108] The grasping stability evaluation method proposed in this embodiment, based on visual-touch fusion perception and multimodal spatiotemporal convolution, is used to measure the performance of the grasping stability evaluation method. The main comparison is made with the impact of different types of input modes and different mixed modes on the grasping stability analysis, as shown in Table 1 below:

[0109]

[0110] The meanings of the symbols in Table 1 are explained below:

[0111] (1) TCL: Tactile-Color-Left, i.e., left-hand tactile RGB image;

[0112] (2) TDL: Tactile-Depth-Left, i.e., left finger tactile depth image;

[0113] (3) TCR: Tactile-Color-Right, i.e., the RGB image of right-hand tactile perception;

[0114] (4) TDR: Tactile-Depth-Right, i.e., right finger tactile depth image;

[0115] (5) VC: Vision-Color, i.e., visual RGB image;

[0116] (6) VD: Vision-Depth, i.e., visual depth image;

[0117] (7)ALL: Includes visual RGB images, Depth images, and left and right finger tactile RGB images and Depth images.

[0118] As can be seen, the grasping stability evaluation method based on visual-touch fusion perception and multimodal spatiotemporal convolution proposed in this embodiment has the following results: the grasping stability evaluation accuracy is: vision and two-finger touch > two-finger touch > single-finger touch > vision.

[0119] Relying solely on vision to judge grasping stability has a low success rate but is relatively stable, with average accuracies of 69.47% and 76.69% for depth and color images, respectively. Learning stable grasping from tactile feedback yields higher accuracy, with two-finger tactile feedback significantly outperforming single-finger tactile feedback, and the detection performance using tactile color images as input outperforming depth images, achieving a maximum accuracy of 85.01%, an 8.32% improvement over the single visual modality. However, the accuracy fluctuates greatly and exhibits poor stability. The proposed vision-tactile fusion method further improves detection performance based on tactile perception, with a significantly smaller variance than the tactile modality detection performance, indicating better stability.

[0120] The performance evaluation of the grasping stability assessment method based on view-touch fusion and spatiotemporal convolution proposed in this embodiment shows that the grasping stability analysis network proposed in this invention achieves an accuracy of 95.80% on the view-touch grasping dataset, with 51.5M parameters and an average inference time of 279.01ms. Its training and validation curves are shown below. Figure 9 , Figure 10 and Figure 11 As shown.

[0121] Finally, the effects of different spatial modules (ResNet-50, ResNet-101, VGG-16) and different temporal attention modules (Longformer, LSTM) on network training results were compared. The LSTM was set to two layers with 64 hidden nodes. The performance comparison of different convolutional networks is shown in Table 2 below.

[0122] Model Parameters Accuracy (%) Inference time (ms) ResNet-50--Longformer 51,472,450 95.80 279.01 ResNet-101--Longformer 70,464,578 96.33 431.83 VGG-16--Longformer 165,379,138 94.17 356.73 ResNet-50--LSTM 23,803,650 92.20 201.23 ResNet-101--LSTM 42,795,778 93.33 459.83 VGG-16--LSTM 134,826,754 91.67 215.37

[0123] As shown in the table above, models using Longformer for temporal attention encoding generally achieve higher accuracy than those using LSTM, but have significantly more parameters and longer inference times. In the spatial module, ResNet-101 outperforms ResNet-50 and VGG-16, but has the longest average inference time. The grasping stability evaluation method based on view-touch fusion and spatiotemporal convolution proposed in this invention best meets the accuracy and real-time requirements of robot grasping stability analysis.

[0124] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for grasp stability evaluation based on visual-tactile fusion perception and multi-modal spatio-temporal convolution, characterized in that: Comprising the following steps: Step A01: Establish a robot simulation grasping platform based on Pybullet and TACTO; Step A02: Collect virtual grasping and tactile images according to the robot simulation grasping platform established in step A01 to obtain a multi-modal grasping dataset, comprising the following steps: Step A021: Obtain a simulation grasping object of the EGAD model; Step A022: Collect virtual grasping and tactile images according to the robot simulation grasping platform; Step A023: Load the model of the robot simulation grasping platform, adjust the state of the simulation grasping object, and obtain simulation grasping action collection data of the robot simulation grasping platform; Step A03: Establish a static tactile grasping stability analysis model; Step A04: Collect dynamic prior data according to the simulation grasping platform built in step A01 to obtain a simulation grasping object dataset; Step A05: Construct a spatio-temporal convolution grasping stability evaluation model; Step A06: Train and validate the dynamic tactile fusion model; Step A022 Collect virtual grasping and tactile images according to the robot simulation grasping platform, comprising the following steps: Step A0221: Initialize Pybullet, create a renderer, and create a light sensor and a tactile sensor in the renderer; Step A0222: Load and parse the URDF file, add the mesh file, and finally add it to the renderer; Step A0222: Step simulation is performed using Pybullet, and TACTO synchronizes the pose of the simulation grasping object, and finally the pose of the simulation grasping object is synchronized to the renderer; Step A023: Load the model of the robot simulation grasping platform, adjust the state of the simulation grasping object, and obtain simulation grasping action collection data of the robot simulation grasping platform, comprising the following steps: Step A0231: After loading the model of the robot simulation grasping platform, the robot gripper is initialized to an open state, and the state of the simulation grasping object is adjusted, first the simulation grasping object is placed in the initial position, and then the simulation grasping object is allowed to fall freely to the ground; Step A0232: After the pose of the simulation grasping object is stable, drive the motor to move the robot gripper to the predetermined position, then close the robot gripper, and after the robot gripper is stable, record the current camera view, the contact normal force information between the light sensor and the tactile sensor and the simulation grasping object, and the current grasped simulation grasping object pose information using Pybullet, collect the current RGB image, Depth image and contact force data, then perform DIGIT image rendering, collect the RGB image and Depth image of the light sensor and tactile sensor, then the robot arm rises at a constant speed by 20cm, and after the robot arm is stable, the current simulation grasping object pose information is obtained, if the height difference along the z-axis direction with the initial time is less than a threshold, it is confirmed that the grasping fails, and the grasping label of this group of data is recorded as 0; otherwise, it is confirmed that the grasping is successful, and the grasping label of this group of data is recorded as 1; Step A0233: The robot gripper updates the pose information of the simulated grasped object during the grasping process. The driving position of the robot arm is increased with random perturbations around the center of mass of the simulated grasped object, and random rotation around the z-axis is added to the vertical grasping. The gripping force of the robot gripper is randomly updated, and the gripping force of the robot gripper is in the range of 5-20 N.

2. The grasp stability evaluation method based on visual-haptic fusion perception and multi-modal spatio-temporal convolution according to claim 1, characterized in that: Step A01: A robot simulation grasping platform is established based on Pybullet and TACTO, including the following steps: Step A011: Based on the visual sensor simulator TACTO, the corresponding tactile sensor is simulated and configured; Step A012: Apply Pybullet to simulate the physical behavior of robot grasping; Step A013: Build a grasping scene in a simulation environment to establish a robot simulation grasping platform. The grasping scene includes a robot arm, a gripper, a camera, a grasped object, and a tactile sensor model.

3. The method of claim 1, wherein the method further comprises: determining a grasp stability score for each of the plurality of grasp poses based on the fusion of the tactile perception and the multi-modal spatio-temporal convolution. Step A03: Establish a static visual-tactile grasping stability analysis model, including the following steps: A031: Use the pre-trained ResNet-18 model on ImageNet as a feature extractor for input images. The one-dimensional vector output by the ResNet-18 model is concatenated, and two fully connected layers are used to predict whether the simulated grasped object will fall after the grasping action is completed; A032: Train and validate the network model, adopt 5-fold cross-validation evaluation dataset, wherein the ratio of training set to test set is 8:2, the Adam optimizer is adopted to train the network model, the learning rate is 5e -4 , the batch is set to 32, the Epoch is set to 10; the visual image and the tactile image are scaled and cropped to 224x224.

4. The grasp stability evaluation method of claim 1, wherein: Step A04: Collect dynamic prior data from the simulation grasping platform established in step A01 to obtain a simulation grasped object dataset. Specifically, when collecting dynamic prior data, set the lifting speed of the robot arm, the resolution and frame rate of the camera, and the acquisition resolution and frame rate of the DIGIT. After the robot arm gripper is stably closed, collect data for 1s, record the RGB image, left DIGIT image, right DIGIT image, gripping force, and grasping label at the time of data collection, and obtain the simulation grasped object dataset during the basic lifting of the simulated robot arm.

5. The grasp stability evaluation method of claim 1, wherein: Step A05: Construct a spatio-temporal convolution grasping stability evaluation model, including the following steps: A051: input a visual image sequence and a tactile image sequence , establish a deep neural network model, confirm whether unstable sliding occurs in the sequence , wherein n is the length of the sequence, F is a classifier, when F the output is 0, the current time is a sliding state, and when F the output is 1, the current time is a stable state A052: fitting a classifier by the deep neural network model F The deep neural network model takes a visual image sequence and a tactile image sequence as input and outputs a sliding probability. The network framework of the deep neural network model comprises a spatial feature extraction module, a temporal feature extraction module, and a multi-layer perception classifier, thereby obtaining a spatio-temporal convolutional grasping stability evaluation model.

6. The grasp stability evaluation method of claim 5, wherein: Step A052 includes the following steps: Step A0521: Construct a spatial feature extraction module based on the ResNet-50 network to extract two-dimensional spatial features of the video in the simulation grasped object dataset; Step A0522: Construct a time feature extraction module using the Longformer structure with attention mechanism; Step A0523: The multi-layer perceptron classifier outputs a sliding probability, which includes GELU, layer normalization, and Dropout.

7. The grasp stability evaluation method of claim 1, wherein: Step A06: Train and validate the dynamic visual-tactile fusion model, including the following steps: Step A061: Divide the simulation grasped object dataset into a training set and a validation set. During training, first crop the visual modality information. In the grasping stability evaluation task, the visual modality information includes the end-of-life module and the state of the grasped simulation grasped object. The visual modality information is located in the lower part of the camera imaging. Step A062: During training, the key information of the visual modality information is cropped to 600x320x3, scaled to 256x256x3, the tactile image includes left finger data and right finger data, the tactile image is spliced along the width direction, then rotated 90° counterclockwise, combined into an image of 480x320x3, and the visual-tactile image is uniformly scaled to 224x224x3; Step 063: The training model uses the Adam optimizer and gradient backpropagation, the touch image time series length is 8, the batchsize is set to 2 during training and verification, the training data is data augmented, and 8 frames of video sequences are randomly selected; the pre-training enhanced acceleration training process is adopted, the Epoch is set to 20, the learning rate is set to 10 -3 , the attention window size is set to 8, and the Dropout probability is set to 0.1.

Images