Visual and tactile fusion-based robot grabbing pose optimization method

Abstract

The invention provides a robot grabbing pose optimization method based on visual and tactile fusion, and belongs to the field of robot grabbing pose estimation. According to the method, firstly, an initial grabbing pose is predicted based on target object sparse point cloud obtained through vision initial observation segmentation, then a robot plans and executes grabbing to obtain tactile data of the surface of an object, and target object point cloud information of an area where the initial grabbing pose is located is complemented based on the tactile data with object local information; and finally, the initial grabbing pose is optimized according to the complemented point cloud, so that more stable robot grabbing is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of robot grasping posture estimation, and in particular, relates to fusing depth camera data with tactile sensor data to guide robots to predict the optimal grasping posture of target objects in complex scenes and complete grasping tasks. Specifically, it is a robot grasping posture optimization method based on visual-tactile fusion. Background Art

[0002] The robot grasping pose estimation task refers to the robot autonomously determining the position and pose of the hand or grasping tool on a known or unknown target object through its own sensors or external information (such as visual system, tactile feedback, joint force feedback, etc.). This task can be divided into three parts: object recognition and positioning, grasping pose estimation, and grasping execution. The robot needs to identify the target object and accurately locate its position in space, and perform accurate grasping pose estimation based on factors such as the shape and material of the object. Then, based on the estimation results, the robot plans and completes the grasping by considering different constraints such as contact force constraints, kinematic reachability constraints, and dynamic constraints.

[0003] At present, vision-based grasping pose estimation methods have been widely studied and applied in fields such as automated production, service robots, and hazardous environment operations. In low-noise environments, these vision-guided methods can provide stable and high-quality grasping pose estimation. However, in actual environments, the grasping success rate and reliability of the grasping poses predicted by such methods will be significantly reduced due to factors such as sensor accuracy, changing lighting conditions, object surface reflection interference, and lack of texture features. Therefore, how to improve the robustness of robot grasping pose estimation in complex environments is a challenge that researchers are currently committed to solving.

[0004] The vision-based tactile sensor covers the camera lens with elastic material. When the sensor contacts the object, the elastic material deforms to adapt to the contour of the object. The built-in multi-color LED lighting system illuminates the contact surface, and the embedded camera records the changes in the gel surface. By analyzing the color and shape changes in the image through software algorithms, the three-dimensional geometry, texture, hardness and other information of the contacted part of the object can be accurately reconstructed. The depth camera can provide high-quality color images and depth information at the same time, giving a macroscopic observation of the environment and the target object. In recent years, methods based on visual and tactile fusion have achieved many results in the field of robot grasping, such as grasping force control based on visual and tactile feedback, multi-finger dexterous operation, and posture tracking. Combining the two modal information of visual perception and tactile perception, the robot can integrate the large-scale spatial information and local contact information of the target object, reduce the impact of environmental factors caused by changes in lighting conditions, interference from reflective surfaces, and lack of texture information, thereby improving the robot's understanding and interaction capabilities with the environment. Summary of the invention

[0005] The present invention provides a robot grasping posture optimization method based on visual-tactile fusion, the purpose of which is to predict and optimize a stable grasping posture by integrating the spatial information around the target object and the local information near the grasping point. The method first predicts an initial grasping posture based on the sparse point cloud of the target object obtained by initial visual observation segmentation, and then the robot plans and executes this grasping to obtain the tactile data of the object surface, and then completes the target object point cloud information in the area where the initial grasping posture is located based on the tactile data with local information of the object, and finally optimizes the initial grasping posture according to the completed point cloud to achieve more stable robot grasping.

[0006] The technical solution of the present invention is:

[0007] A robot grasping posture optimization method based on visual-tactile fusion, the steps are as follows:

[0008] Step 1: Load the robot visual and tactile information collection and grasping training test environment

[0009] Simulation experiment environment: Based on the PyBullet simulation platform, the UR10e robot arm and the Robotiq-140 gripper are used as experimental robots to complete the grasping operation. The experimental robot is equipped with a depth camera and two Digit tactile sensors to collect raw visual and tactile information. Some objects are selected from the YCB dataset as grasping targets, and grasping test scenarios of different difficulty levels are designed for single objects, multiple objects, and multiple objects stacked.

[0010] Real experimental environment: UR10e robotic arm and Robotiq-140 gripper are used as experimental robots to complete the grasping operation. The end of the Robotiq-140 gripper is replaced with a tactile sensor bracket, and the realsenseD435i depth camera and Gelsight-Mini tactile sensor are installed on the experimental robot. Robot motion control and depth camera data and tactile data collection are completed based on ROS (Robot Operating System).

[0011] Step 2: Training of tactile local point cloud prediction module

[0012] In the training phase, we first use the sparse point cloud S obtained from visual observations. ori PointNetGPD algorithm is applied to generate the grasping pose P ori, operate the robot to perform grasping, collect tactile sensor data and obtain 3-channel RGB data with a length and width of 64 after downsampling; when operating the robot to grasp objects and collect tactile raw data, for each contact operation, a tactile image with more than 20% pixel points changing is successfully collected. After successful collection, the surface of the object contact point is intercepted in the simulation to obtain a local point cloud as a reference, and the current position P of the center point of the tactile sensor in the world coordinate system is recorded c .

[0013] The tactile local point cloud prediction module uses a convolutional neural network (CNN) as the feature encoder of the tactile image, receives 3-channel tactile RGB data as the original input, and outputs a feature vector of length 128. The feature vector is regularized by a multi-layer perceptron (MLP) to obtain 25 grid vertex positions, and is then converted into a grid according to the position P of the center point of the tactile sensor. c Transform the 25 mesh vertices into the tactile sensor coordinate system and uniformly sample the local tactile point cloud S on the mesh. tac As the final output, the overall loss function is shown in formula (1).

[0014]

[0015] The loss function enlarges the sampled point cloud P by calculating the coefficient r The average nearest square distance between the reference point cloud Q is obtained, where α represents the calculation coefficient and R represents the total number of uniform sampling. Represents the sampling point cloud P r The point in Represents a point in the reference point cloud Q.

[0016] Step 3: Local point cloud completion module training

[0017] The local point cloud completion module trains a global encoder and a local encoder based on the PointNet++ network to encode the global features and local tactile features of the point cloud respectively. After fusing the global features and local tactile features, the decoder is used to decode and obtain the completed point cloud.

[0018] In the training phase, we first use the sparse point cloud S obtained from visual observations. ori PointNetGPD algorithm is applied to generate the grasping pose P ori , operate the robot to perform grasping, collect tactile sensor data and input it into the point cloud prediction module in step 2 to obtain the local tactile point cloud S tac The object’s overall reference point cloud S is obtained based on object model sampling and synthesis. gt . Use a global encoder to encode the sparse point cloud S ori , get the global features; the local encoder encodes the tactile point cloud S tac, and obtain the local tactile features; in order to make the model attention weight biased towards the local tactile features, the global features and the local tactile features are fused through the weighted fusion module, and then the MLP is used for decoding to obtain the completed point cloud S fil .

[0019] The loss function includes the reconstruction loss function L rec , feature consistency loss function L feat And the local structure loss function L local The overall loss function L obtained by weighted addition of the above loss functions total .

[0020] Reconstruction loss function L rec According to the reference point cloud S gt And the completed point cloud S fil The average nearest square distance is calculated as shown in formula (2).

[0021]

[0022] Feature consistency loss function L feat To ensure that in the sparse point cloud S ori With tactile point cloud S tac The features extracted from the tactile point cloud are consistent. The PointNet++ network is designed to encode the input point cloud into high-dimensional features F1 that retain more spatial details and low-dimensional features F2 that contain more semantic information. C1 is used to calculate the high-dimensional features of the tactile point cloud. High-dimensional features with sparse point clouds The similarity of is shown in formula (3). Using C2 to calculate the low-dimensional features of the tactile point cloud With sparse point cloud low-dimensional features The similarity of L is shown in formula (4). feat Minimize the difference between C1, C2 and the unit matrix to ensure that the extracted features are as consistent as possible, as shown in formula (5).

[0023]

[0024] Among them, |·|2 represents the L2 norm, |·| F represents the Frobenius norm,

[0025] Local structure loss function L local Use KNN to calculate the local difference and evenly select N coordinate points c near the center point of the tactile sensor that currently produces tactile perception. i As the center point, in the completed point cloud S fil With the reference point cloud S gt KNN sampling is used to obtain point pairs p j , g jAnd calculate the distance, as shown in formula (6).

[0026]

[0027] in represents the KNN sampling result point set, and K represents the number of nearest neighbors of each center, as shown in formula (7).

[0028]

[0029] The overall loss function L total As shown in formula (8).

[0030] L total =αL rec +βL feat +γL local (8)

[0031] Among them, α regulates L rec Weight, β adjusts L feat Weight, γ adjusts L local Weight.

[0032] Step 4: Grasp the pose optimization module

[0033] The completed point cloud S obtained in step 3 fil , and the initial grasping posture P ori As input, take the initial grasping pose P ori A bounding box is defined near the center of the gripper to segment the point cloud and generate a mask. The PointNetGPD method is used to predict a set of candidate grasping poses. The grasping score and the deviation from the initial pose are comprehensively calculated among the candidate grasping poses to output the optimal grasping pose P. opt .

[0034] Beneficial effects of the present invention:

[0035] (1) Optimizing grasping posture using local tactile information

[0036] Vision-based RGBD observation data can provide global information about the target object, including its geometry, spatial position, and distribution in the environment. RGBD data has limited ability to represent the fine structure of the object's surface, which may cause errors or failures when the robot performs grasping operations. The data provided by vision-based tactile sensors can accurately reflect the shape, texture, and surface characteristics of the contacted part of the object. High-resolution tactile information can enhance the robot's understanding of the local features of the target object. By combining tactile feedback with visual perception, the robot can more accurately assess the physical properties of the object and adjust its grasping strategy accordingly.

[0037] (2) Combined with grasping affordance research

[0038] Affordance refers to the opportunities or possibilities in the environment, that is, the functions or uses of an object or scene element for a specific subject. Grasping affordance research helps robots understand which parts of the target object are suitable for grasping. The grasping posture optimization method based on visual-tactile fusion can generate stable and reliable grasping postures based on the local areas obtained from the affordance study, refine the affordance judgment, and provide a richer perception mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the solution of the present invention.

[0040] Figure 2 This is the network model structure of the present invention. DETAILED DESCRIPTION

[0041] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0042] like Figure 1 As shown, a robot grasping posture optimization method based on visual-tactile fusion includes robot visual-tactile information collection and grasping training and testing environment construction, training data collection and preprocessing, tactile local point cloud prediction module training, local point cloud completion module training, grasping posture generation, grasping posture optimization and grasping execution.

[0043] The simulation environment is built based on the PyBullet simulation platform. The urdf model of the spliced ​​UR10e robot arm, Robotiq-140 gripper and Digit tactile sensor and the depth camera with the real sense D435i camera internal reference are used to build the robot visual tactile information collection and grasping training environment. In the simulation environment, 8000 tactile image data with local point cloud labels are collected for the training of the tactile local point cloud prediction module. 12 objects are selected from the YCB dataset and imported into the simulation environment. The method of predicting the grasping posture and fine-tuning the collection is adopted. 2400 groups of sample data consisting of initial grasping posture, sparse point cloud, and tactile image are collected for each object, and a total of 28,800 groups of sample data consisting of initial grasping posture, sparse point cloud, and tactile image are used for point cloud completion module training.

[0044] The real environment is used to test the grasping effect. The UR10e robotic arm and the Robotiq-140 gripper are used as experimental robots. The tactile sensor bracket is used to replace the end of the Robotiq-140 gripper. The robot is equipped with a realsense D435i depth camera and a Gelsight-Mini tactile sensor. Sensor perception data collection and robot motion control are realized based on ROS communication.

[0045] like Figure 2As shown in the figure, the workflow of the network is as follows: the depth camera obtains the initial sparse point cloud, PointNetGPD is used on the sparse point cloud to predict the initial grasping pose, the robot is manipulated to complete the grasping, the tactile sensor data is sampled and downsampled to obtain 3-channel RGB data with a length and width of 64, which is sent to the CNN network for encoding to obtain the grid vertex position data with a 3-channel length of 25, 1000 points are uniformly sampled in the grid to obtain tactile point cloud data, and the initial grasping pose and sparse point cloud data are sent as input to the point cloud completion module to obtain the completed point cloud, and then the PointNetGPD algorithm is applied to generate the optimized grasping pose with a higher score as the final result.

[0046] About the tactile local point cloud prediction module training:

[0047] Since it is difficult to directly obtain the local point cloud of the contact part in the real environment, it is necessary to pre-train a tactile local point cloud prediction module to convert the tactile image into a local point cloud.

[0048] In the training phase, we first use the sparse point cloud S obtained from visual observations. ori PointNetGPD algorithm is applied to generate the grasping pose P ori , operate the robot to perform grasping, collect tactile sensor data and obtain 3-channel RGB data with a length and width of 64 after downsampling; when operating the robot to grasp objects and collect tactile raw data, for each contact operation, a tactile image with more than 20% pixel points changing is successfully collected. After successful collection, the surface of the object contact point is intercepted in the simulation to obtain a local point cloud as a reference, and the current position P of the center point of the tactile sensor in the world coordinate system is recorded c .

[0049] The tactile local point cloud prediction module uses a convolutional neural network (CNN) as the feature encoder of the tactile image, receives 3-channel tactile RGB data as the original input, and outputs a feature vector of length 128. The feature vector is regularized by a multi-layer perceptron (MLP) to obtain 25 grid vertex positions, and is then converted into a grid according to the position P of the center point of the tactile sensor. c Transform the 25 mesh vertices into the tactile sensor coordinate system and uniformly sample the local tactile point cloud S on the mesh. tac As the final output, the overall loss function is shown in formula (1).

[0050]

[0051] The loss function enlarges the sampled point cloud P by calculating the coefficient r The average nearest square distance between the reference point cloud Q is obtained, where α represents the calculation coefficient and R represents the total number of uniform sampling. Represents the sampling point cloud P rThe point in Represents a point in the reference point cloud Q.

[0052] About the local point cloud completion module training:

[0053] The local point cloud completion module trains a global encoder and a local encoder based on the PointNet++ network to encode the global features and local tactile features of the point cloud respectively. After fusing the global features and local tactile features, the decoder is used to decode and obtain the completed point cloud.

[0054] In the training phase, we first use the sparse point cloud S obtained from visual observations. ori PointNetGPD algorithm is applied to generate the grasping pose P ori , operate the robot to perform grasping, collect tactile sensor data and input it into the point cloud prediction module in step 2 to obtain the local tactile point cloud S tac The object’s overall reference point cloud S is obtained based on object model sampling and synthesis. gt . Use a global encoder to encode the sparse point cloud S ori , get the global features; the local encoder encodes the tactile point cloud S tac , and obtain the local tactile features; in order to make the model attention weight biased towards the local tactile features, the global features and the local tactile features are fused through the weighted fusion module, and then the MLP is used for decoding to obtain the completed point cloud S fil .

[0055] The loss function includes the reconstruction loss function L rec , feature consistency loss function L feat And the local structure loss function L local The overall loss function L obtained by weighted addition of the above loss functions total .

[0056] Reconstruction loss function L rec According to the reference point cloud S gt And the completed point cloud S fil The average nearest square distance is calculated as shown in formula (2).

[0057]

[0058] Feature consistency loss function L feat To ensure that in the sparse point cloud S ori With tactile point cloud S tac The features extracted from the tactile point cloud are consistent. The PointNet++ network is designed to encode the input point cloud into high-dimensional features F1 that retain more spatial details and low-dimensional features F2 that contain more semantic information. C1 is used to calculate the high-dimensional features of the tactile point cloud. High-dimensional features with sparse point clouds The similarity of is shown in formula (3). Using C2 to calculate the low-dimensional features of the tactile point cloud With sparse point cloud low-dimensional features The similarity of L is shown in formula (4). feat Minimize the difference between C1, C2 and the unit matrix to ensure that the extracted features are as consistent as possible, as shown in formula (5).

[0059]

[0060] Among them, |·|2 represents the L2 norm, |·| F represents the Frobenius norm,

[0061] Local structure loss function L local Use KNN to calculate the local difference and evenly select N coordinate points c near the center point of the tactile sensor that currently produces tactile perception. i As the center point, in the completed point cloud S fil With the reference point cloud S gt KNN sampling is used to obtain point pairs p j , g j And calculate the distance, as shown in formula (6).

[0062]

[0063] in represents the KNN sampling result point set, and K represents the number of nearest neighbors of each center, as shown in formula (7).

[0064]

[0065] The overall loss function L total As shown in formula (8).

[0066] L total =αL rec +βL feat +γL local (8)

[0067] Among them, α regulates L rec Weight, β adjusts L feat Weight, γ adjusts L local Weight.

[0068] About the grasping posture optimization module:

[0069] The completed point cloud S obtained in step 3 fil , and the initial grasping posture P ori As input, take the initial grasping pose P oriA bounding box is defined near the center of the gripper to segment the point cloud and generate a mask. The PointNetGPD method is used to predict a set of candidate grasping poses. The grasping score and the deviation from the initial pose are comprehensively calculated among the candidate grasping poses to output the optimal grasping pose P. opt .

Claims

1. A robot grasping posture optimization method based on visual-tactile fusion, characterized in that: Here are the steps: Step 1: Load the robot visual and tactile information collection and grasping training test environment; Step 2: Training of tactile local point cloud prediction module; Step 3: Local point cloud completion module training; Step 4: Grasp the posture optimization module.

2. The robot grasping posture optimization method based on visual-tactile fusion according to claim 1 is characterized in that: The step 1 is specifically as follows: Simulation experiment environment: Based on the PyBullet simulation platform, the UR10e robotic arm and the Robotiq-140 gripper are used as experimental robots to complete the grasping operation. The experimental robot is equipped with a depth camera and two Digit tactile sensors to collect raw visual and tactile information. Some objects are selected from the YCB dataset as grasping targets, and grasping test scenarios of different difficulty levels are designed for single objects, multiple objects, and multiple objects stacked. Real experimental environment: Use UR10e robotic arm and Robotiq-140 gripper as experimental robots to complete grasping operations, use tactile sensor bracket to replace the end of Robotiq-140 gripper, and install realsense D435i depth camera and Gelsight-Mini tactile sensor on the experimental robot; complete robot motion control and depth camera data and tactile data collection based on ROS.

3. The robot grasping posture optimization method based on visual-tactile fusion according to claim 1 is characterized in that: The step 2 is specifically as follows: In the training phase, we first use the sparse point cloud S obtained from visual observations. ori PointNetGPD algorithm is applied to generate the grasping pose P ori , operate the robot to perform grasping, collect tactile sensor data and obtain 3-channel RGB data with a length and width of 64 after downsampling; when operating the robot to grasp objects and collect tactile raw data, for each contact operation, a tactile image with more than 20% pixel points changing is successfully collected. After successful collection, the surface of the object contact point is intercepted in the simulation to obtain a local point cloud as a reference, and the current position P of the center point of the tactile sensor in the world coordinate system is recorded c ; The tactile local point cloud prediction module uses a convolutional neural network (CNN) as the feature encoder of the tactile image, receives 3-channel tactile RGB data as the original input, and outputs a feature vector of length 128. The feature vector is regularized by a multi-layer perceptron (MLP) to obtain 25 grid vertex positions, and is then converted into a grid according to the position P of the center point of the tactile sensor. c Transform the 25 mesh vertices into the tactile sensor coordinate system and uniformly sample the local tactile point cloud S on the mesh. tac As the final output; the overall loss function is shown in formula (1); The loss function enlarges the sampled point cloud P by calculating the coefficient r The average nearest square distance between the reference point cloud Q is obtained, where α represents the calculation coefficient and R represents the total number of uniform sampling. Represents the sampling point cloud P r The point in Represents a point in the reference point cloud Q.

4. The robot grasping posture optimization method based on visual-tactile fusion according to claim 1 is characterized in that: The step 3 is specifically as follows: The local point cloud completion module trains a global encoder and a local encoder based on the PointNet++ network to encode the global features and local tactile features of the point cloud respectively. After fusing the global features and local tactile features, the decoder is used to decode and obtain the completed point cloud. In the training phase, we first use the sparse point cloud S obtained from visual observations. ori PointNetGPD algorithm is applied to generate the grasping pose P ori , operate the robot to perform grasping, collect tactile sensor data and input it into the point cloud prediction module in step 2 to obtain the local tactile point cloud S tac ; Based on the object model sampling synthesis, the object's overall reference point cloud S is obtained gt ; Encode the sparse point cloud S using a global encoder ori , get the global features; the local encoder encodes the tactile point cloud S tac , and obtain the local tactile features; in order to make the model attention weight biased towards the local tactile features, the global features and the local tactile features are fused through the weighted fusion module, and then the MLP is used for decoding to obtain the completed point cloud S fil ; The loss function includes the reconstruction loss function L rec , feature consistency loss function L feat And the local structure loss function L local ; The overall loss function L obtained by weighted addition of the above loss functions total ; Reconstruction loss function L rec According to the reference point cloud S gt And the completed point cloud S fil The average nearest square distance is calculated as shown in formula (2); Feature consistency loss function L feat To ensure that in the sparse point cloud S ori With tactile point cloud S tac The features extracted from the tactile point cloud are consistent. The PointNet++ network is designed to encode the input point cloud into high-dimensional features F1 that retain more spatial details and low-dimensional features F2 that contain more semantic information. C1 is used to calculate the high-dimensional features of the tactile point cloud. High-dimensional features with sparse point clouds The similarity of is shown in formula (3); C2 is used to calculate the low-dimensional features of the tactile point cloud With sparse point cloud low-dimensional features The similarity of L is shown in formula (4). feat Minimize the difference between C1, C2 and the unit matrix to ensure that the extracted features are as consistent as possible, as shown in formula (5); Among them, |·|2 represents the L2 norm, |·| F represents the Frobenius norm, Local structure loss function L local Use KNN to calculate the local difference and evenly select N coordinate points c near the center point of the tactile sensor that currently produces tactile perception. i As the center point, in the completed point cloud S fil With the reference point cloud S gt KNN sampling is used to obtain point pairs p j , g j And calculate the distance, as shown in formula (6); in represents the KNN sampling result point set, K represents the number of nearest neighbors of each center, as shown in formula (7); The overall loss function L total As shown in formula (8); L total =αL rec +βL feat +γL local (8) Among them, α regulates L rec Weight, β adjusts L feat Weight, γ adjusts L local Weight.

5. The robot grasping posture optimization method based on visual-tactile fusion according to claim 1 is characterized in that: The step 4 is specifically as follows: the completed point cloud S obtained in step 3 is fil , and the initial grasping posture P ori As input, take the initial grasping pose P ori A bounding box is defined near the center of the gripper to segment the point cloud and generate a mask. The PointNetGPD method is used to predict a set of candidate grasping poses. The grasping score and the deviation from the initial pose are comprehensively calculated among the candidate grasping poses to output the optimal grasping pose P. opt .

Images