A robot grasping pose optimization method based on visual-haptic fusion

Abstract

The application provides a robot grasping pose optimization method based on visual-haptic fusion, and belongs to the field of robot grasping pose estimation. The method first predicts an initial grasping pose based on a target object sparse point cloud obtained by visual initial observation segmentation, then the robot plans to execute the grasping to obtain haptic data of the object surface, the target object point cloud information in the region where the initial grasping pose is located is supplemented based on the haptic data with local information of the object, and finally the initial grasping pose is optimized according to the supplemented point cloud, so that more stable robot grasping is realized.

Description

Technical Field

[0001] This invention belongs to the field of robotic grasping pose estimation, specifically fusing depth camera data with tactile sensor data to guide robots in predicting the optimal grasping pose of target objects and completing grasping tasks in complex scenarios. Specifically, it provides a robotic grasping pose optimization method based on visual-tactile fusion. Background Art

[0002] The robotic grasping pose estimation task involves autonomously determining the position and posture of a hand or grasping tool on a known or unknown target object using its own sensors or external information (such as vision, tactile feedback, and joint force feedback). This task can be divided into three parts: object recognition and localization, grasping pose estimation, and grasp execution. The robot needs to identify the target object and accurately locate its position in space. It then accurately estimates the grasping pose based on factors such as the object's shape and material. Based on the estimated results, it then plans and completes the grasp by considering various constraints, such as contact force constraints, kinematic reachability constraints, and dynamic constraints.

[0003] Vision-based grasping pose estimation methods have been widely studied and applied in fields such as automated production, service robotics, and hazardous environment operations. In low-noise environments, these vision-guided methods can provide stable and high-quality grasping pose estimates. However, in real-world environments, the grasping success rate and reliability of the grasping poses predicted by these methods can be significantly reduced due to limitations such as sensor accuracy, varying lighting conditions, interference from surface reflections, and the lack of texture features. Therefore, improving the robustness of grasping pose estimation in complex environments is a challenge currently being addressed by researchers.

[0004] A vision-based tactile sensor uses an elastic material overlaid on a camera lens. When the sensor contacts an object, the elastic material deforms to conform to the object's contours. A built-in multi-color LED lighting system illuminates the contact surface, while an embedded camera records changes in the gel surface. Software algorithms analyze color and shape changes in the image to accurately reconstruct the three-dimensional geometry, texture, and hardness of the contacted portion of the object. Depth cameras simultaneously provide high-quality color images and depth information, providing a macroscopic view of the environment and target object. In recent years, methods based on visual-tactile fusion have achieved numerous results in the field of robotic grasping, such as grasping force control based on visual-tactile feedback, multi-finger dexterous manipulation, and posture tracking. By combining visual and tactile perception, robots can integrate large-scale spatial information and local contact information about the target object, reducing the impact of environmental factors such as varying lighting conditions, interference from reflective surfaces, and missing texture information, thereby improving the robot's understanding of the environment and its ability to interact with it. Summary of the Invention

[0005] This invention provides a robotic grasping pose optimization method based on visual-tactile fusion. The goal is to predict and optimize a stable grasping pose by integrating spatial information around the target object with local information near the grasping point. This method first predicts an initial grasping pose based on a sparse point cloud of the target object obtained through initial visual observation and segmentation. The robot then plans and executes this grasp to obtain tactile data from the object's surface. The target object point cloud information in the area of ​​the initial grasping pose is supplemented using tactile data containing local object information. Finally, the initial grasping pose is optimized based on the supplemented point cloud to achieve more stable robotic grasping.

[0006] The technical solution of the present invention is:

[0007] A robot grasping posture optimization method based on visual-tactile fusion, the steps are as follows:

[0008] Step 1: Load the robot visual and tactile information acquisition and grasping training test environment

[0009] The simulation experiment environment was built on the PyBullet simulation platform. The experimental robot used a UR10e robotic arm and a Robotiq-140 gripper to perform grasping operations. The experimental robot was equipped with a depth camera and two Digit tactile sensors to collect raw visual and tactile information. Objects were selected from the YCB dataset as grasping targets, and grasping test scenarios of varying difficulty were designed, including single-object, multi-object, and stacked multi-object grasping.

[0010] Real-world experimental environment: The experimental robot used a UR10e robotic arm and a Robotiq-140 gripper to perform grasping operations. A tactile sensor bracket replaced the Robotiq-140 gripper end cap, and the experimental robot was equipped with a realsense D435i depth camera and a Gelsight-Mini tactile sensor. Robot motion control and depth camera and tactile data collection were performed based on ROS (Robot Operating System).

[0011] Step 2: Tactile local point cloud prediction module training

[0012] In the training phase, we firstly calculate the sparse point cloud S obtained by visual observation. ori Apply PointNetGPD algorithm to generate grasping pose P ori, operate the robot to perform grasping, collect tactile sensor data and downsample to obtain 3-channel RGB data with a length and width of 64; when operating the robot to grasp the object to collect tactile raw data, for each contact operation, a tactile image with more than 20% pixel points changing is considered a successful acquisition. After successful acquisition, the surface of the object contact point is intercepted in the simulation to obtain a local point cloud as a reference, and the current position P of the center point of the tactile sensor in the world coordinate system is recorded c .

[0013] The tactile local point cloud prediction module uses a convolutional neural network (CNN) as the feature encoder of the tactile image, receives 3-channel tactile RGB data as the original input, and outputs a feature vector of length 128. The feature vector is regularized by a multi-layer perceptron (MLP) to obtain 25 grid vertex positions, and is then converted into the position of the center point of the tactile sensor P. c Convert the 25 mesh vertices to the tactile sensor coordinate system and uniformly sample the local tactile point cloud S on the mesh. tac As the final output, the overall loss function is shown in formula (1).

[0014]

[0015] The loss function amplifies the sampled point cloud P by calculating the coefficient r The average nearest square distance between the reference point cloud Q is obtained, where α represents the calculation coefficient and R represents the total number of uniform sampling. Represents the sampling point cloud P r The point in Represents a point in the reference point cloud Q.

[0016] Step 3: Local point cloud completion module training

[0017] The local point cloud completion module uses the PointNet++ network to train a global encoder and a local encoder to encode the global features of the point cloud and the local tactile features, respectively. After fusing the global and local tactile features, the decoder is used to decode the completed point cloud.

[0018] In the training phase, we firstly calculate the sparse point cloud S obtained by visual observation. ori Apply PointNetGPD algorithm to generate grasping pose P ori , operate the robot to perform grasping, collect tactile sensor data and input it into the point cloud prediction module in step 2 to obtain the local tactile point cloud S tac The object’s overall reference point cloud S is obtained based on object model sampling and synthesis. gt . Use a global encoder to encode the sparse point cloud S ori , get the global features; the local encoder encodes the tactile point cloud S tac, and obtain the tactile local features; in order to make the model attention weight biased towards the tactile local features, the global features and the tactile local features are fused through the weighted fusion module, and then the MLP is used for decoding to obtain the completed point cloud S fil .

[0019] The loss function includes the reconstruction loss function L rec , feature consistency loss function L feat And the local structure loss function L local The overall loss function L obtained by weighted addition of the above loss functions is total .

[0020] Reconstruction loss function L rec According to the reference point cloud S gt and the completed point cloud S fil The average nearest square distance is calculated as shown in formula (2).

[0021]

[0022] Feature consistency loss function L feat To ensure that the sparse point cloud S ori With tactile point cloud S tac The features extracted from the point cloud are consistent. The PointNet++ network is designed to encode the input point cloud into high-dimensional features F1 that retain more spatial details and low-dimensional features F2 that contain more semantic information. High-dimensional features with sparse point clouds The similarity of is shown in formula (3). Using C2 to calculate the low-dimensional features of the tactile point cloud and sparse point cloud low-dimensional features The similarity of L is shown in formula (4). feat Minimize the difference between C1, C2 and the unit matrix to ensure that the extracted features are as consistent as possible, as shown in formula (5).

[0023]

[0024] Among them, |·|2 represents the L2 norm, |·| F represents the Frobenius norm,

[0025] Local structure loss function L local Use KNN to calculate the local difference and evenly select N coordinate points c near the center point of the tactile sensor that currently generates tactile perception. i As the center point, in the completed point cloud S fil With the reference point cloud S gt KNN sampling is used to obtain point pairs p j 、g jAnd calculate the distance, as shown in formula (6).

[0026]

[0027] in represents the KNN sampling result point set, and K represents the number of nearest neighbors of each center, as shown in formula (7).

[0028]

[0029] Overall loss function L total As shown in formula (8).

[0030] L total =αL rec +βL feat +γL local (8)

[0031] Among them, α regulates L rec Weight, β adjusts L feat Weight, γ adjusts L local Weight.

[0032] Step 4: Grasp the pose optimization module

[0033] The completed point cloud S obtained in step 3 fil , and the initial grasping pose P ori As input, the initial grasp pose P ori A bounding box is defined near the center of the gripper to segment the point cloud and generate a mask. The PointNetGPD method is used to predict a set of alternative grasping poses. The grasping score and the deviation from the initial pose are comprehensively calculated among the alternative grasping poses to output the optimal grasping pose P. opt .

[0034] Beneficial effects of the present invention:

[0035] (1) Optimizing grasping posture using local tactile information

[0036] Vision-based RGBD observation data can provide global information about the target object, including its geometry, spatial position, and distribution in the environment. RGBD data has limited ability to represent the subtle structure of an object's surface, which can lead to errors or failures when the robot performs grasping operations. Vision-based tactile sensors provide data that accurately reflects the shape, texture, and surface characteristics of the object being touched. High-resolution tactile information can enhance the robot's understanding of the local features of the target object. By combining tactile feedback with visual perception, the robot can more accurately assess the physical properties of the object and adjust its grasping strategy accordingly.

[0037] (2) Combined with research on grasping affordance

[0038] Affordance refers to the opportunities or possibilities in an environment—the function or use that an object or scene element possesses for a specific subject. Grasping affordance research helps robots understand which parts of a target object are suitable for grasping. Grasping pose optimization methods based on visual-tactile fusion can generate stable and reliable grasping poses based on the local regions derived from affordance research, refine affordance judgments, and provide a richer perception model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the solution of the present invention.

[0040] Figure 2 This is the network model structure of the present invention. DETAILED DESCRIPTION

[0041] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0042] like Figure 1 As shown, a robot grasping posture optimization method based on visual-tactile fusion includes robot visual-tactile information collection and grasping training and testing environment construction, training data collection and preprocessing, tactile local point cloud prediction module training, local point cloud completion module training, grasping posture generation, grasping posture optimization and grasping execution.

[0043] The simulation environment, built on the PyBullet simulation platform, utilizes a URDF model of a spliced ​​UR10e robotic arm, a Robotiq-140 gripper, and a Digit tactile sensor, along with a depth camera that imports the intrinsic parameters of a real-world RealSense D435i camera, to create a training environment for robotic visual-tactile information collection and grasping. 8,000 tactile image data with local point cloud labels were collected in the simulation environment for training the tactile local point cloud prediction module. Twelve objects were selected from the YCB dataset and imported into the simulation environment. Using a method where the grasp pose was predicted and fine-tuned before acquisition, 2,400 sets of data were collected for each object, for a total of 28,800 sets of sample data consisting of initial grasp poses, sparse point clouds, and tactile images for training the point cloud completion module.

[0044] A real-world environment was used to test grasping effectiveness. A UR10e robotic arm and a Robotiq-140 gripper were used as the experimental robots. A tactile sensor bracket replaced the end cap of the Robotiq-140 gripper. The robot was equipped with a RealSense D435i depth camera and a Gelsight-Mini tactile sensor. Sensor perception data collection and robot motion control were achieved based on ROS communication.

[0045] like Figure 2As shown in the figure, the network workflow is as follows: the depth camera obtains the initial sparse point cloud, PointNetGPD is used on the sparse point cloud to predict the initial grasping pose, the robot is manipulated to complete the grasping, the tactile sensor data is sampled and downsampled to obtain 3-channel RGB data with a length and width of 64, which is sent to the CNN network for encoding to obtain 3-channel grid vertex position data with a length of 25, 1000 points are uniformly sampled in the grid to obtain tactile point cloud data, and the initial grasping pose and sparse point cloud data are used as input to the point cloud completion module to obtain the completed point cloud, and then the PointNetGPD algorithm is applied to generate the optimized grasping pose with a higher score as the final result.

[0046] About tactile local point cloud prediction module training:

[0047] Since it is difficult to directly obtain the local point cloud of the contact part in a real environment, it is necessary to pre-train a tactile local point cloud prediction module to convert the tactile image into a local point cloud.

[0048] In the training phase, we firstly calculate the sparse point cloud S obtained by visual observation. ori Apply PointNetGPD algorithm to generate grasping pose P ori , operate the robot to perform grasping, collect tactile sensor data and downsample to obtain 3-channel RGB data with a length and width of 64; when operating the robot to grasp the object to collect tactile raw data, for each contact operation, a tactile image with more than 20% pixel points changing is considered a successful acquisition. After successful acquisition, the surface of the object contact point is intercepted in the simulation to obtain a local point cloud as a reference, and the current position P of the tactile sensor center point in the world coordinate system is recorded c .

[0049] The tactile local point cloud prediction module uses a convolutional neural network (CNN) as the feature encoder of the tactile image, receives 3-channel tactile RGB data as the original input, and outputs a feature vector of length 128. The feature vector is regularized by a multi-layer perceptron (MLP) to obtain 25 grid vertex positions, and is then converted into the position of the center point of the tactile sensor P. c Convert the 25 mesh vertices to the tactile sensor coordinate system and uniformly sample the local tactile point cloud S on the mesh. tac As the final output, the overall loss function is shown in formula (1).

[0050]

[0051] The loss function amplifies the sampled point cloud P by calculating the coefficient r The average nearest square distance between the reference point cloud Q is obtained, where α represents the calculation coefficient and R represents the total number of uniform sampling. Represents the sampling point cloud P rThe point in Represents a point in the reference point cloud Q.

[0052] About the local point cloud completion module training:

[0053] The local point cloud completion module uses the PointNet++ network to train a global encoder and a local encoder to encode the global features of the point cloud and the local tactile features, respectively. After fusing the global and local tactile features, the decoder is used to decode the completed point cloud.

[0054] In the training phase, we firstly calculate the sparse point cloud S obtained by visual observation. ori Apply PointNetGPD algorithm to generate grasping pose P ori , operate the robot to perform grasping, collect tactile sensor data and input it into the point cloud prediction module in step 2 to obtain the local tactile point cloud S tac The object’s overall reference point cloud S is obtained based on object model sampling and synthesis. gt . Use a global encoder to encode the sparse point cloud S ori , get the global features; the local encoder encodes the tactile point cloud S tac , and obtain the tactile local features; in order to make the model attention weight biased towards the tactile local features, the global features and the tactile local features are fused through the weighted fusion module, and then the MLP is used for decoding to obtain the completed point cloud S fil .

[0055] The loss function includes the reconstruction loss function L rec , feature consistency loss function L feat And the local structure loss function L local The overall loss function L obtained by weighted addition of the above loss functions is total .

[0056] Reconstruction loss function L rec According to the reference point cloud S gt and the completed point cloud S fil The average nearest square distance is calculated as shown in formula (2).

[0057]

[0058] Feature consistency loss function L feat To ensure that the sparse point cloud S ori With tactile point cloud S tac The features extracted from the point cloud are consistent. The PointNet++ network is designed to encode the input point cloud into high-dimensional features F1 that retain more spatial details and low-dimensional features F2 that contain more semantic information. High-dimensional features with sparse point clouds The similarity of is shown in formula (3). Using C2 to calculate the low-dimensional features of the tactile point cloud and sparse point cloud low-dimensional features The similarity of L is shown in formula (4). feat Minimize the difference between C1, C2 and the unit matrix to ensure that the extracted features are as consistent as possible, as shown in formula (5).

[0059]

[0060] Among them, |·|2 represents the L2 norm, |·| F represents the Frobenius norm,

[0061] Local structure loss function L local Use KNN to calculate the local difference and evenly select N coordinate points c near the center point of the tactile sensor that currently generates tactile perception. i As the center point, in the completed point cloud S fil With the reference point cloud S gt KNN sampling is used to obtain point pairs p j 、g j And calculate the distance, as shown in formula (6).

[0062]

[0063] in represents the KNN sampling result point set, and K represents the number of nearest neighbors of each center, as shown in formula (7).

[0064]

[0065] Overall loss function L total As shown in formula (8).

[0066] L total =αL rec +βL feat +γL local (8)

[0067] Among them, α regulates L rec Weight, β adjusts L feat Weight, γ adjusts L local Weight.

[0068] About the grasping pose optimization module:

[0069] The completed point cloud S obtained in step 3 fil , and the initial grasping pose P ori As input, the initial grasp pose P oriA bounding box is defined near the center of the gripper to segment the point cloud and generate a mask. The PointNetGPD method is used to predict a set of alternative grasping poses. The grasping score and the deviation from the initial pose are comprehensively calculated among the alternative grasping poses to output the optimal grasping pose P. opt .

Claims

1. A robot grasping posture optimization method based on visual-tactile fusion, characterized in that: Here are the steps: Step 1: Load the robot visual and tactile information acquisition and grasping training test environment; Step 2: Training of tactile local point cloud prediction module; Step 3: Local point cloud completion module training; The step 3 is specifically as follows: The local point cloud completion module uses the PointNet++ network to train a global encoder and a local encoder to encode the global features and local tactile features of the point cloud, respectively. After fusing the global features with the local tactile features, the decoder is used to decode the completed point cloud. In the training phase, we first use the sparse point cloud obtained from visual observations to Apply the PointNetGPD algorithm to generate the grasping pose , operate the robot to perform grasping, collect tactile sensor data and input it into the point cloud prediction module in step 2 to obtain the local tactile point cloud ; Based on the object model sampling synthesis, the object's overall reference point cloud is obtained ; Encode sparse point clouds using a global encoder , get global features; local encoder encodes tactile point cloud , and obtain the tactile local features; in order to make the model attention weight biased towards the tactile local features, the weighted fusion module is used to complete the fusion of global features and tactile local features, and then MLP is used for decoding to obtain the completed point cloud ; The loss function includes the reconstruction loss function , feature consistency loss function and local structure loss function ; The overall loss function obtained by weighted addition of the above loss functions ; Reconstruction loss function Based on the benchmark point cloud and complete point cloud The average nearest square distance is calculated as shown in formula (2); ； Feature consistency loss function To ensure that the sparse point cloud and tactile point clouds The extracted features are consistent, and the PointNet++ network is designed to encode the input point cloud into high-dimensional features that retain more spatial details. and low-dimensional features that contain more semantic information ;use Computing high-dimensional features in tactile point clouds High-dimensional features with sparse point clouds The similarity of is shown in formula (3); using Computing low-dimensional features in tactile point clouds and sparse point cloud low-dimensional features The similarity of is shown in formula (4). minimize 、 The difference from the identity matrix ensures that the extracted features are as consistent as possible, as shown in Equation (5); ； ； ； in, represents the L2 norm, represents the Frobenius norm, Local structure loss function Use KNN to calculate the local difference and evenly select N coordinate points near the center point of the tactile sensor that currently produces tactile perception. As the center point, in the completion point cloud With the benchmark point cloud KNN sampling is used to obtain point pairs 、 And calculate the distance, as shown in formula (6); ； in Represents the KNN sampling result point set, represents the number of nearest neighbors of each center, as shown in formula (7); ； Overall loss function As shown in formula (8); ； in, adjust weight, adjust weight, adjust weight; Step 4: Grasp the pose optimization module.

2. The robot grasping posture optimization method based on visual-tactile fusion according to claim 1 is characterized in that: The step 1 is specifically as follows: The simulation experiment environment is built on the PyBullet simulation platform. A robotic arm and gripper are used as the experimental robot to perform grasping operations. The experimental robot is equipped with a depth camera and two Digit tactile sensors to collect raw visual and tactile information. Objects are selected from the YCB dataset as grasping targets, and grasping test scenarios of varying difficulty are designed, including single-object, multi-object, and multi-object stacking. Real experimental environment: Use a robotic arm and gripper as the experimental robot to complete grasping operations, use a tactile sensor bracket to replace the end of the gripper, and install a depth camera and tactile sensor on the experimental robot; complete robot motion control and depth camera data and tactile data collection based on ROS.

3. The robot grasping posture optimization method based on visual-tactile fusion according to claim 1 is characterized in that: The step 2 is specifically as follows: In the training phase, we first use the sparse point cloud obtained from visual observations to Apply the PointNetGPD algorithm to generate the grasping pose , operate the robot to perform grasping, collect tactile sensor data and downsample it to obtain 3-channel RGB data with a length and width of 64; when operating the robot to grasp the object to collect tactile raw data, for each contact operation, a tactile image with more than 20% pixel points changing is considered a successful acquisition. After successful acquisition, the surface of the object contact point is intercepted in the simulation to obtain a local point cloud as a reference, and the position of the current tactile sensor center point in the world coordinate system is recorded ; The tactile local point cloud prediction module uses a convolutional neural network (CNN) as the feature encoder of the tactile image, receives 3-channel tactile RGB data as the original input, and outputs a feature vector of length 128. The feature vector is regularized by a multi-layer perceptron (MLP) to obtain 25 grid vertex positions, and is then converted into the position of the center point of the tactile sensor. Convert 25 mesh vertices to the tactile sensor coordinate system and evenly sample the local tactile point cloud on the mesh As the final output; the overall loss function is shown in formula (1); ； The loss function amplifies the sampled point cloud by calculating the coefficient With the benchmark point cloud The average nearest square distance between represents the calculation coefficient, represents the total number of uniform sampling times, Representative sampling point cloud The point in Representative benchmark point cloud The point in .

4. The robot grasping posture optimization method based on visual-tactile fusion according to claim 1 is characterized in that: The step 4 is specifically as follows: the completed point cloud obtained in step 3 is , and the initial grasping pose As input, the initial grasp pose Define a bounding box near the center of the gripper to segment the point cloud and generate a mask. Apply the PointNetGPD method to predict a set of alternative grasping poses. Among the alternative grasping poses, comprehensively calculate the grasping score and the deviation from the initial pose to output the optimal grasping pose. .

Images