Transparent object six-degree-of-freedom grabbing method based on RGB-D multi-mode information fusion
By using RGB-D multimodal information fusion and the SE-DFNet and AnyGrasp algorithms, the problems of inaccurate depth acquisition of transparent objects and poor flexibility of traditional grasping schemes are solved, and high-precision six-degree-of-freedom grasping is achieved.
Patent Information
- Application Number
- CN202511154185.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-18
- Publication Date
- 2025-11-04
AI Technical Summary
Existing technologies struggle to effectively perceive and stably grasp transparent objects, particularly due to inaccurate depth information acquisition and the lack of flexibility in traditional grasping methods.
A method based on RGB-D multimodal information fusion is adopted. The depth map is corrected by the SE-DFNet depth completion network, and the AnyGrasp algorithm is combined to estimate the six-degree-of-freedom grasping pose. The grasping operation is then performed by a robotic arm.
It significantly improves the perception, grasping flexibility and stability of transparent objects, enabling dexterous manipulation of a variety of transparent objects.
Smart Images

Figure CN120886257A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer vision and robot grasping, and particularly relates to a six-degree-of-freedom grasping method for transparent objects based on RGB-D multi-modal information fusion. BACKGROUND
[0002] Transparent objects are ubiquitous in daily life, such as water cups and fish tanks in the home environment, and beakers and test tubes commonly used in scientific research. At present, robots have entered all aspects of human life, and it has become an important trend in various industries to use robots to replace human labor to complete some repetitive and dangerous work. Therefore, whether a robot can operate transparent objects as easily as a human being is a problem worthy of attention. The problem of "easily operating transparent objects" involves two key points: on the one hand, the robot needs to perceive the transparent object, and the transparent object is a weak-textured object that is relatively invisible in the scene, so it is difficult for the robot to find the transparent object only according to the RGB photo obtained by the sensor; on the other hand, the grasping of the robot on the object should have flexibility, stability and universality, which requires a high-degree-of-freedom grasping pose detection algorithm that complies with physical constraints and has strong generalization.
[0003] In the aspect of transparent object perception, early research was implemented by mining information in two-dimensional images, such as focusing on the highlight points generated by the mirror reflection of the surface of the transparent object, or using the blur and distortion effect caused by the transparent object on the background to detect the edge of the transparent object, but the accuracy of these methods is low and misjudgment is easy to occur. With the popularity of RGB-D sensors, research in this direction also introduces the depth modality on the basis of two-dimensional RGB images, and uses the depth value to assist in determining whether the corresponding pixel point is the background or the transparent object. However, after introducing the depth modality, there is still a problem to be solved: the working mechanism of most depth cameras is to emit infrared light to the object and observe the diffuse reflection of the light back, but the transparent object will cause mirror reflection, resulting in that the light does not return to the camera, or the light transmits through the transparent object and reflects back to the camera from the background, which will cause inaccurate depth information, so the depth information needs to be corrected, and this process is called depth completion. The result of depth completion is not only used to help determine whether a pixel point exists a transparent object, but also participates in the positioning of the transparent object in the three-dimensional space in the downstream pose estimation link. Depth completion is usually completed by a depth neural network, and the current well-known depth completion models include ClearGrasp, LIDF-Refine, DFNet, etc., but the accuracy of the model depth completion still needs to be improved.
[0004] In the aspect of robot grasping objects, the 2D plane grasping is widely used at present, such as GR-ConvNet, EfficientGrasp and other algorithms, which limits the robot to move the gripper vertically downward from above the workbench, and cannot grasp from the side, so the grasping scheme is single and has poor flexibility. Therefore, a more flexible six-degree-of-freedom grasping is adopted, so that the gripper has the freedom of translation along the x, y and z axes and rotation around the x, y and z axes, which is more in line with the requirements of the robot to dexterously manipulate real-world objects. Secondly, the grasping scheme should ensure stable grasping of objects, that is, to find reasonable grasping points and grasping angles that meet experience and physical analysis, and to avoid empty grasping, slipping and being blocked by other objects. In addition, the grasping algorithm should be widely used on various transparent objects and can be universally applicable to objects of different sizes, shapes and surface materials.
[0005] In summary, there is an urgent need for a flexible, stable and widely used six-degree-of-freedom grasping pose estimation algorithm to support dexterous manipulation in complex scenes. SUMMARY
[0006] The present application aims to solve the problems in the prior art and provides a six-degree-of-freedom grasping method for transparent objects based on RGB-D multi-modal information fusion, which significantly improves the perception ability of the system for transparent objects, solves the problem of poor depth acquisition effect of existing RGB-D sensors for transparent objects, and improves the adaptability and robustness of the system when processing various transparent objects.
[0007] To achieve the above-mentioned purpose, the following technical solutions are adopted:
[0008] A six-degree-of-freedom grasping method for transparent objects based on RGB-D multi-modal information fusion, comprising the steps of:
[0009] S1, input the original RGB image and depth map of the transparent object collected by the RGB-D camera into the end-to-end depth completion network SE-DFNet, and correct the depth map by the end-to-end depth completion network SE-DFNet.
[0010] Preferably, the network is specifically composed as follows:
[0011] The SE-DFNet network as a whole adopts the U-Net architecture based on the encoding-decoding structure. First, the input original depth map is subjected to multi-level down-sampling in order to strengthen the model's understanding of the features of the transparent object at different scales. The RGB image, the original depth map and the different scale depth maps after down-sampling are transmitted into the encoder, and the encoding part is composed of multiple CDCD (Conv-Dense-Conv-Downsample) modules, which improves the feature expression ability and the utilization rate of the original features by stacking densely connected convolutional layers.
[0012] After each CDCD module of the encoder, a SE channel attention module is introduced to dynamically weight the channel dimension of the feature maps at different levels. The module performs global average pooling on the spatial dimension of each channel through the "Squeeze" operation to extract the global context information at the channel level. Then, through the "Excitation" operation, a fully connected network is used to generate the weight coefficient of each channel, and the Sigmoid function is used for normalization. In this way, the network can adaptively enhance the extraction of key features according to the actual contribution of each channel in the RGB image and the depth map, and suppress redundant or interfering information. In this way, the network realizes the layer-by-layer optimization of the joint features of the RGB image and the depth map during the encoding process. The SE module dynamically adjusts the channel weight after feature extraction at each scale, strengthens the complementary relationship between multi-modal information, and thus provides more effective feature representation for the subsequent decoding process.
[0013] The decoding part adopts the CDCU (Conv-Dense-Conv-Upsample) module to restore the image resolution by means of pixel-level upsampling mechanism (DUC, Dense Upsampling Convolution), realizing high-precision depth reconstruction. The network realizes multi-scale feature extraction and analysis through depth map downsampling-upsampling, and optimizes the depth error loss and surface normal smoothness loss in the training stage, so that the network can accurately correct the depth of the transparent area.
[0014] S2, according to the internal parameters of the RGB-D camera, combine the RGB image and the completed depth map to form a scene point cloud, and then input it into the 6D grasp pose estimation algorithm AnyGrasp based on deep neural network to obtain a candidate set of grasp poses, and score each grasp scheme in the candidate set of grasp poses; wherein each grasp scheme has seven degrees of freedom, defined as G=[R t w], R is the rotation matrix of the gripper of the robot arm, t is the center position of the gripper of the robot arm, and w is the opening width of the gripper of the robot arm.
[0015] Preferably, the conversion method of the scene point cloud comprises:
[0016] The camera calibration function package of ROS is used to calibrate the camera with the help of a chessboard calibration board to obtain the camera intrinsic matrix K:
[0017]
[0018] wherein f x , f y is the focal length, c x , c y is the principal point coordinate.
[0019] For each pixel coordinate (u, v) in the image and its corresponding depth value d, the point as a point in the point cloud, the three-dimensional coordinates x in the camera coordinate system are i = [x, y, z] T Each component is calculated as follows:
[0020]
[0021] According to the above relationship, the RGB-D image is converted into a point cloud containing N points, and input into the AnyGrasp network.
[0022] Further, after inputting the scene point cloud into the 6D grasp pose estimation algorithm AnyGrasp based on a deep neural network, first, the global and local features of the point cloud are extracted through the ResUNet backbone network, and the point-level graspability score is generated by a multi-layer perception MLP, representing the grasp possibility of each point.
[0023] According to the score, M candidate points with higher grasp possibility are obtained through farthest point sampling FPS. The MLP judges the grasp possibility from V views (i.e. grasp directions) for each candidate point, generating view-level graspability scores, where M and V are both natural numbers greater than zero.
[0024] The AnyGrasp network divides the M candidate points into different cylindrical regions according to their positions and view directions, each cylindrical region has the same size and contains K candidate points, and K is a natural number greater than zero. In each cylindrical group, different jaw plane rotation angles and jaw depths are searched to further sample the grasp pose, and the score of the grasp pose and the gripper width are output by the MLP, thereby obtaining multiple grasp schemes, and the score s grasp output by AnyGrasp for each grasp scheme is recorded.
[0025] Further, considering that reliable grasp points are often close to the center of gravity of the object, a COG perception mechanism is adopted, and a stability score s stable is introduced to measure the vertical distance ratio between the center of gravity of the object and the contact surface of the gripper. The final score s final of each grasp scheme is defined as:
[0026] s final = s grasp · (1-s stable )
[0027] The higher the score, the more feasible and stable the grasp is.
[0028] S3, select the grasp scheme with the highest score, deploy the grasp scheme on the robotic arm, and execute the grasp of the transparent object.
[0029] Preferably, the specific steps of deploying the grasping scheme to the robot arm include:
[0030] S31, hand-eye calibration: the eye-in-hand structure is adopted, that is, the camera is fixedly installed on the end gripper of the robot arm. The purpose of hand-eye calibration is to solve the transformation T from the camera coordinate system {C} to the end gripper coordinate system {E} of the robot arm EC The process adopts the camera-calibration function package of ROS, and is completed with the aid of a checkerboard calibration board.
[0031] S32, grasping pose transformation: the highest-scoring predicted grasping scheme G = [R t w] is converted from the camera coordinate system to the robot arm base coordinate system to obtain G' = [R' t' w], wherein [R' t'] is a 6D grasping pose estimate. On the premise that the end effector relative to the base coordinate system pose T can be calculated by forward kinematics of the robot arm BE The [R' t'] calculation method is:
[0032]
[0033] S33, motion planning and execution: taking [R' t'] as the target grasping pose, the joint variable values corresponding to the target pose are solved through inverse kinematics of the robot arm, the robot arm motion planning module MoveIt is called to generate joint space path planning and trajectory planning of the gripper from the current state to the grasping pose, and control instructions are generated and sent accordingly to move the gripper. The gripper opening width is adjusted to w, and the gripper closing is performed to complete the grasping action.
[0034] Compared with the prior art, the present application has the following advantages:
[0035] 1. The present application introduces depth modal information, combines RGB images and depth maps for joint perception, significantly improves the perception ability of the system for transparent objects, and provides a more reliable basis for subsequent spatial grasping pose estimation.
[0036] 2. The present application proposes a depth completion method for transparent objects, efficiently combines the information of RGB and depth channels using the channel attention mechanism, uses the RGB image to guide the analysis of the depth map, effectively repairs the missing points in the depth map of the transparent object, improves the integrity and accuracy of the depth map, and solves the problem of poor depth acquisition effect of existing RGB-D sensors for transparent objects.
[0037] 3. The present application adopts a six-degree-of-freedom grasping pose estimation method, which has higher operational flexibility compared to low-degree-of-freedom grasping strategies. An end-to-end deep neural network is used to generate grasping poses based on a sampling and scoring mechanism, and the center of gravity position is included in the scoring system to improve the adaptability and robustness of the system in handling various transparent objects.
[0038] 4. The application constructs a real mechanical arm grabbing system, integrates depth completion, grabbing detection, mechanical arm motion planning and control, proves the feasibility of the used algorithm, and provides a complete reference scheme for transparent object processing tasks in actual production and service scenarios. BRIEF DESCRIPTION OF DRAWINGS
[0039] Figure 1 is a flowchart of the application.
[0040] Figure 2 is a structural diagram of the SE-DFNet neural network proposed by the application.
[0041] Figure 3 is an application effect diagram of the depth completion algorithm of the embodiment of the application.
[0042] Figure 4 is a node communication diagram when the application is deployed on the ROS platform.
[0043] Figure 5 is a mechanical arm grabbing transparent object real experiment effect diagram of the embodiment of the application. DETAILED DESCRIPTION
[0044] The application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not to limit the scope of the application. In addition, it should be understood that after reading the content taught by the application, those skilled in the art can make various modifications or modifications to the application, and these equivalent forms also fall within the scope defined by the application.
[0045] The application provides a transparent object six-degree-of-freedom grabbing method based on RGB-D multi-modal information fusion, which contains three stages:
[0046] 1) Depth completion link, use an RGB-D camera to shoot an RGB image and a depth image of a transparent object, and use a SE-DFNet depth completion network to correct and process the depth image.
[0047] 2) 6D grabbing pose estimation link, combine the RGB two-dimensional image information and the depth information into a point cloud according to the camera internal parameter, and use an AnyGrasp 6D grabbing pose estimation algorithm to obtain the pose of the gripper grabbing the transparent object in the camera coordinate system.
[0048] 3) Seven-degree-of-freedom mechanical arm grabbing system is built. The algorithms are integrated and deployed on a real mechanical arm, and the coordinate system conversion, motion planning and control instruction sending are performed in turn to grab the real transparent object.
[0049] Embodiment: as Figure 1As shown, the present application is a kind of six degrees of freedom transparent object grasping method based on RGB-D multi-modal information fusion, comprising the following steps:
[0050] S1, the original RGB image and depth map collected by the RGB-D camera are input into the end-to-end depth completion network SE-DFNet.The network adopts the coding-decoding structure similar to U-Net, and performs down-sampling and up-sampling operations in the coding-decoding stage respectively, to gradually extract multi-scale features for reasoning, and finally restore high-resolution output.The schematic diagram of the network is as shown in Figure 2 As shown, the specific structure is as follows:
[0051] In the coding stage, the input layer of the network accepts the RGB color picture and depth map collected by the RGB-D camera, a total of 4 channels, and performs the following coding based on down-sampling.First, the original color picture and depth map are processed through a convolution down-sampling module.Next, the original depth map is separately reduced in size by using the nearest neighbor interpolation method, and is input into three CDCD (Conv-Dense-Conv-Downsample) modules, which will further use convolution operation to down-sample and reduce the length and width of the depth map by half.The first CDCD module takes the RGB and depth coupled image processed by the initial stage convolution down-sampling and the reduced size depth map as input, and has two outputs of skip path and normal path coexisting: the skip path uses the CDC (Conv-Dense-Conv) module, maintains high resolution, and fuses multiple layer features to lay the foundation for subsequent decoding;the normal path normally uses the CDCD module, and reduces the length and width of the image by half while extracting features.The second CDCD module takes the above reduced image and the original depth Figure Two map as input, and also has two outputs of skip and normal, and the third CDCD module is the same.
[0052] To enhance the extraction effect of multi-modal and multi-scale features, an SE channel attention module is introduced after each CDCD module, which can set different weights for multiple feature channels and dynamically adjust these weights to enhance the extraction ability of the entire network for key features.
[0053] For the multi-channel image input into the SE module After convolution operation, its shape changes to HxWxC, where C is the number of channels.This step realizes the mapping:
[0054]
[0055] For the obtained feature map U, the SE module sequentially performs two operations of global information embedding (Squeeze) and adaptive recalibration (Excitation) to give different weights to each channel of the feature map U.
[0056] In the global information embedding (Squeeze) stage, the SE module extracts global semantic information for each of the C channels of the feature map U through a global average pooling operation, while reducing the original spatial dimension of HxW, and compresses each channel into a scalar z c After compression, a feature vector F sq (U) with a shape of 1x1xC can be obtained. Here, the specific calculation formula of z c is as follows:
[0057]
[0058] In the adaptive recalibration (Excitation) stage, the module learns dynamic channel weights through a neural network, and then re-evaluates the relative importance of each channel. This step uses a two-layer fully connected neural network, where the first layer reduces the input of the C channels to C / r channels (r is the compression ratio) and outputs after ReLU activation function; the second layer restores the channel number to C and outputs after Sigmoid activation function normalization. The output s of this step has a shape of 1x1xC, and each of the C channels represents the importance of each channel of the feature map U. The specific calculation formula of s is as follows:
[0059] s=F ex (z,W)=Sigmoid(g(z,W))=Sigmoid(W2·ReLU(W1z))
[0060] In the formula, z is the global information obtained in the Squeeze stage, and W1 and W2 are the weights of the two fully connected layers.
[0061] After obtaining the channel weight distribution s, s is multiplied with the feature map U channel by channel to calibrate the importance of each channel, and an output feature map with a shape of HxWxC can be obtained. The calculation formula of each channel
[0062]
[0063] The decoding part of the network is mainly up-sampling, and symmetric logic is used in the encoding stage. The output results of the four CDC modules are input into the four CDCU (Conv-Dense-Conv-Upsample) modules in descending order of scale size, and the output of the lower layer CDCU is used as one of the inputs of the higher level CDCU module. After four times of cascaded CDCU module operations, a completed depth map with the same scale as the original image can be obtained. CDCU realizes up-sampling through the method of pixel rearrangement after convolution, restores the spatial resolution of the feature map from low resolution to high resolution, that is, first increase the channel number of the feature map through convolution, and then "rearrange" these increased channels into a larger spatial size through pixel rearrangement (PixelShuffle) to complete the up-sampling. By using the CDCU (Conv-Dense-Conv-Upsample) module and the pixel-level up-sampling mechanism (DUC, Dense Upsampling Convolution), the image resolution is restored, and high-precision depth reconstruction is realized.
[0064] The training of the network is based on the large-scale real-world transparent object RGB-D image dataset TransCG. In the training stage, the depth error loss and the surface normal smoothness loss are jointly optimized, so that the network can accurately correct the depth of the transparent region. The specific loss function Λ is:
[0065]
[0066] In the formula, Δ * is the predicted depth and the depth real value, is the unit gradient of the predicted depth map and the real depth map in the horizontal direction, is the unit gradient of the predicted depth map and the real depth map in the vertical direction, and β is a weight specified by a person. The calculated L d is the mean square error of the predicted depth relative to the depth real value, and L s is the cosine distance loss based on the surface normal, which is used to measure the similarity between the surface normal direction of the predicted depth map and the surface normal direction of the real depth map. L d and L s respectively punish the depth reconstruction error from the depth value difference and the surface normal proximity, so that the model can infer the depth close to the real value. The effect of depth completion is shown in Figure 3 , and it can be seen that the algorithm completes a large number of missing points in the original depth map.
[0067] S2, according to the camera intrinsic parameter, the RGB image and the completed depth map are combined to form a scene point cloud, which is input into the 6D grasp pose estimation algorithm AnyGrasp based on the depth neural network to obtain a candidate set of grasp poses. Specifically, it includes:
[0068] With the camera-calibration function package of ROS, the camera is calibrated by means of a chessboard calibration board, and the focal length and optical center coordinate parameters of the RGB-D camera are obtained in the form of an intrinsic matrix K:
[0069]
[0070] where f x , f y is the focal length, and c x , c y is the principal point (optical center) coordinate.
[0071] For each pixel coordinate (u, v) in the image and its corresponding depth value d, when the pixel point is a point in the point cloud, the three-dimensional coordinates x i = [x, y, z] T in the camera coordinate system are calculated as follows:
[0072]
[0073] According to the above relationship, the RGB-D image is converted into a point cloud containing N points, which is input into the AnyGrasp end-to-end network. The AnyGrasp network is divided into two stages. In the first stage, the graspable points are screened according to the graspability score (graspness), and in the second stage, the specific grasp scheme is predicted for the graspable points.
[0074] Firstly, in the first stage, the global and local features of the point cloud are extracted by the ResUNet backbone network, and a C-dimensional feature vector of each point is output, with a dimension of N x (3+C), where 3 is the three-dimensional coordinates of the point, and C is the C-dimensional feature vector. The multi-layer perceptron (MLP) is used to infer the graspability (graspness) according to the geometric features (such as surface normal, local concave-convexness), and a point-level graspability score is generated, which represents the grasping possibility of each point.
[0075] According to the score, M seed candidate points with higher grasping possibility are obtained by farthest point sampling (FPS), with a dimension of M x (3+C). The MLP is used to evaluate the grasping possibility of each seed point from V views (i.e. grasping directions), and a view-level graspability score is generated, with a dimension of M x (V+C).
[0076] Next, the positions, features and their view-level graspability scores of M seed points are input into the two-stage of the network. The Anygrasp network divides M seed points into different cylindrical regions according to their positions and the best view direction, each cylindrical region has the same size and contains K points. In each cylindrical group, further sampling of grasp poses is performed by searching different jaw plane rotation angles and jaw depths, and the scores of the grasp poses and the gripper width are output by the MLP, with the output dimension being M x (A x D x 2), where A is the number of in-plane rotation angles, D is the number of jaw depths, and 2 represents the grasp pose score and the jaw width. A plurality of grasp schemes can be obtained from the above process, each grasp scheme having seven degrees of freedom, defined as G = [R t w], where R is a jaw rotation matrix, t is a jaw center position, and w is a jaw opening width. The score output by the network for each grasp scheme is denoted as s grasp This score is represented by the friction coefficient:
[0077]
[0078] where μ i According to force closure analysis and collision detection, the grasp scheme that fails in force closure or has collision is an invalid grasp, and otherwise it is an effective grasp.
[0079] Considering that reliable grasp points are often close to the object's center of gravity, a COG center of gravity perception mechanism is adopted, and a stability score s stable is introduced to measure the vertical distance ratio between the object's center of gravity and the jaw contact surface. The final score s final of each grasp scheme is defined as:
[0080] s final = s grasp ·(1-s stable )
[0081] The higher this score is, the more feasible and stable the grasp is.
[0082] S3, select the grasp scheme with the highest s final , input the corresponding 6D grasp pose and jaw opening width into the robot operation module, and execute the grasp. Specifically, it includes:
[0083] Hand-eye calibration: The eye-in-hand structure is adopted, i.e. the camera is fixedly installed on the end gripper of the robot arm. The purpose of the hand-eye calibration is to solve the transformation from the camera coordinate system to the end gripper coordinate system of the robot arm. In the calibration process, the base of the robot arm and the calibration board are fixed and do not move, and the base coordinate system and the calibration board coordinate system established thereon also remain unchanged. Since the camera moves with the gripper, the camera coordinate system and the end coordinate system of the robot arm change multiple times, but their relative poses remain unchanged. The hand-eye calibration involves the base coordinate system (also referred to as the base coordinate system) of the robot arm, the end coordinate system of the robot arm, the camera coordinate system, and the calibration board coordinate system.
[0084] Let the hand-eye calibration matrix to be solved be X, take a point P0 on the calibration board, and convert P0 into a point P1 in the camera coordinate system according to the external parameter T1 of the camera relative to the calibration board. The external parameter can be obtained by means of the ROS camera-calibration function package and the chessboard calibration board. The point P1 in the camera coordinate system is transformed into a point P2 in the end coordinate system of the robot arm according to the hand-eye calibration matrix X. The transformation matrix T from the end gripper coordinate system to the base coordinate system can be easily calculated from the modeling parameters of the robot arm itself. BE Thus, the corresponding point P3 of P0 in the base coordinate system is obtained, i.e.
[0085] T BE XT1P0=P3
[0086] When the camera moves with the gripper, the coordinate values P0 and P2 do not change, while P1 and P2 change, and the transformation matrix T from the end gripper coordinate system to the base coordinate system and the external parameter of the camera relative to the calibration board also change. At this time, the transformation relationship of P0 and P3 is:
[0087] T′ BE XT1′P0=P3
[0088] In the formula, T′ BE T′ can be obtained by re-calibration of the camera.
[0089] Solving the above two equations can obtain: T BE XT1P0=T BE 'XT1'P0
[0090] Eliminate P0 and perform transformation to obtain: XT1'T1 -1 =T BE ' -1 T BE X
[0091] The above formula is in the form of XA=BX, where A=T1'T1 -1 and B=T BE ' -1 TBE It is known that the hand-eye calibration matrix X can be solved.
[0092] Grasp pose transformation: convert the top-scoring predicted grasp G = [R t w] from the camera coordinate system to the robot base coordinate system to obtain G' = [R' t' w], where [R' t'] is the 6D grasp pose estimate that can be directly used for robot grasping. Under the premise that the end-effector pose T BE (i.e., the hand-eye calibration matrix X), the [R' t'] calculation method is:
[0093]
[0094] Motion planning and execution: take [R' t'] as the target grasp pose, solve the joint variable values corresponding to the target pose through inverse kinematics of the robot, call the robot motion planning module MoveIt to generate the joint space path planning of the gripper from the current state to the grasp pose, and perform trajectory planning for each joint variable to obtain the expected joint angle, motion speed and acceleration at each time, according to which control commands are generated and sent to move the gripper. Adjust the gripper opening width to w, and perform the gripper closing to complete the grasping action.
[0095] The above functions are realized based on the ROS platform. Depth completion and 6D grasp pose estimation are integrated into a vision node, which acquires RGB-D images through the Inter RealSense D435i camera, then performs depth completion and grasp pose estimation, and passes the estimated pose to the execution node for coordinate transformation, motion planning and robot control. The Franka Emika Panda seven-degree-of-freedom robot executes the grasping task. The system is built using the service communication mechanism of ROS, in which the depth completion and pose estimation node is the server (Service) and the execution node is the client (Client). The execution node can actively request services from the vision node to obtain grasp pose information after the robot and the camera are reset to the initial pose. The execution node transforms the grasp pose in the camera coordinate system to the base coordinate system according to the results of hand-eye calibration, and solves the angles of each joint of the seven-degree-of-freedom robot during grasping through inverse kinematics. For each joint variable, path planning and trajectory planning are performed in turn to obtain the expected joint angle, motion speed and acceleration at each time, which are converted into motion commands through the libfranka programming interface and sent to the robot for execution. Precise and fast feedback control is achieved through the built-in PD controller. The architecture of this part is shown in Figure 4
[0096] Different materials, different sizes, different shapes, and transparent objects that do not appear in the training set used by AnyGrasp are selected for testing the grasping effect of the above system. When grasping, the robot arm is reset to the initial pose, and the transparent object to be tested is placed in the field of view of the RGB-D camera at this pose. Depth completion and pose estimation can be started, and then grasping can be performed. After grasping the object, the robot arm transports the object to the designated position and releases the gripper to drop the object. As shown in Figure 5
[0097] For each object, 50 grasping tests are performed, the total number of grasping poses generated by the network is averaged, the highest score of the grasping pose and the grasping success rate in 50 grasping attempts are counted, and the statistical results are shown in Table 1 as follows:
[0098] Table 1 Comparison of grasping success rate
[0099] Item Average number of grasp poses Grasp pose with highest score Success rate of grasps (%) Mineral water bottle 4.2 0.978 98 Plastic bottle (large) 3.9 0.942 94 Plastic bottle (small) 3.6 0.893 92 Thin glass bottle 3.5 0.827 92 Plastic cup (large) 2.8 0.685 90 Plastic cup (small) 2.2 0.643 90
[0100] As can be seen from the implementation results, the more the average number of grasping poses and the higher the highest score of the grasping pose, the more confident AnyGrasp algorithm is in grasping the input point cloud, and the higher the success rate in actual grasping. This shows that the score inferred by AnyGrasp has a strong corresponding relationship with the actual execution effect on the physical object, and reflects the rationality of the design and training of AnyGrasp network. On the other hand, the system shows a high grasping success rate for various transparent objects selected for the experiment, indicating that the system can achieve stable grasping of transparent objects and has good generalization.
[0101] In summary, the present application comprehensively utilizes RGB-D multi-modal information, corrects the depth perception of transparent objects by introducing a depth completion network, realizes accurate six-degree-of-freedom grasping pose estimation using AnyGrasp algorithm, and deploys the grasping result to a physical robot arm system to complete the grasping operation, forming a complete transparent object grasping process. The present application improves the perception accuracy of transparent objects while realizing the flexibility, stability and universality of grasping.
Claims
1. A method for grasping transparent objects with six degrees of freedom based on RGB-D multimodal information fusion, characterized in that, Including the following steps: S1. Input the original RGB image and depth map of the transparent object captured by the RGB-D camera into the end-to-end depth completion network SE-DFNet, and the end-to-end depth completion network SE-DFNet will correct the depth map. S2. Based on the intrinsic parameters of the RGB-D camera, the RGB image and the completed depth image are combined to form a scene point cloud. Then, the point cloud is input into AnyGrasp, a 6D grasping pose estimation algorithm based on deep neural networks, to obtain a grasping pose candidate set. Each grasping scheme in the grasping pose candidate set is scored. Each grasping scheme has seven degrees of freedom, defined as G = [R tw], where R is the rotation matrix of the gripper of the robotic arm, t is the center position of the gripper of the robotic arm, and w is the opening width of the gripper of the robotic arm. S3. Select the highest-scoring grasping scheme, deploy the grasping scheme on the robotic arm, and perform the grasping of the transparent object.
2. The method for six-degree-of-freedom grasping of transparent objects based on RGB-D multimodal information fusion according to claim 1, characterized in that, The end-to-end deep completion network SE-DFNet in step S1 adopts the U-Net architecture, based on the encoder-decoder structure. The encoder part consists of multiple CDCD modules. An SE channel attention module is introduced after each CDCD module of the encoder to dynamically weight the feature maps of different levels according to the channel dimension. The decoder part adopts the CDCU module to restore the image resolution with the help of the pixel-level upsampling mechanism.
3. The method for six-DOF grasping of transparent objects based on RGB-D multimodal information fusion according to claim 2, characterized in that, The end-to-end depth completion network SE-DFNet first performs multi-level downsampling on the input original depth map, and then feeds the RGB image, the original depth map, and the downsampled depth maps at different scales into the encoder.
4. The method for six-DOF grasping of transparent objects based on RGB-D multimodal information fusion according to claim 2, characterized in that, The SE channel attention module first performs global average pooling on the spatial dimension of each channel through the "Squeeze" operation to extract global context information at the channel level; then, it uses the "Excitation" operation to generate weight coefficients for each channel using a fully connected network and normalizes them using the Sigmoid function.
5. The method for six-degree-of-freedom grasping of transparent objects based on RGB-D multimodal information fusion according to claim 1, characterized in that, In step S2, the method for converting the scene point cloud includes: First, using the ROS camera-calibration package and a checkerboard calibration board, the camera is calibrated to obtain the camera intrinsic parameter matrix K. Among them, f x f y c is the focal length. x c y Principal point coordinates; For each pixel coordinate point (u, v) and its corresponding depth value d in the image, when this point is considered as a point in the point cloud, its three-dimensional coordinates x in the camera coordinate system are... i =[x,y,z] T The calculations for each component are as follows: Then, based on the above relationship, the RGB-D image is converted into a point cloud containing N points.
6. The method for six-DOF grasping of transparent objects based on RGB-D multimodal information fusion according to claim 1, characterized in that, In step S2, after the scene point cloud is input into AnyGrasp, a 6D grasping pose estimation algorithm based on deep neural networks, the global and local features of the point cloud are first extracted through the ResUNet backbone network. A multilayer perceptron (MLP) is then used to generate point-level graspability scores to represent the graspability probability of each point. Based on the scores, the farthest point sampling (FPS) is used to obtain M candidate points with high graspability. The MLP is then used to evaluate the graspability of each candidate point from V views to generate a view-level graspability score.
7. The method for six-DOF grasping of transparent objects based on RGB-D multimodal information fusion according to claim 6, characterized in that, In step S2, the 6D grasping pose estimation algorithm AnyGrasp assigns M candidate points to different cylindrical regions based on their position and view orientation. Each cylindrical region is the same size and contains K candidate points. Within each cylindrical group, the grasping posture is further sampled by searching for different in-plane rotation angles and gripper depths. The MLP outputs the grasping posture score and gripper width, thereby obtaining multiple grasping schemes.
8. The method for six-DOF grasping of transparent objects based on RGB-D multimodal information fusion according to claim 1, characterized in that, In step S2, when scoring each grasping scheme in the grasping pose candidate set, a COG-aware mechanism is adopted, introducing a stability score s. stable Stability rating s stable To measure the ratio of the vertical distance between the object's center of gravity and the gripper's contact surface, the final score s for each gripping scheme is calculated. final Defined as: s final =s grasp ·(1-s stable ) In Anygrasp, the score assigned to each crawling scheme by the MLP is denoted as s. grasp .
9. A method for six-DOF grasping of transparent objects based on RGB-D multimodal information fusion according to claim 1, characterized in that, Step S3, which involves deploying the grasping scheme onto the robotic arm, includes: S31. Perform hand-eye calibration using the ROS camera-calibration package and a checkerboard calibration board to solve the transformation from the camera coordinate system to the coordinate system of the robotic arm's end gripper. S32. Complete the grasping posture transformation, transforming the highest-scoring grasping scheme G=[R tw] from the camera coordinate system to the robot arm base coordinate system, obtaining G′=[R′t′w], where [R′t′] is the 6D grasping pose estimation, and the calculation method of [R′t′] is as follows: In the formula, T BE Let T be the transformation matrix from the coordinate system of the end effector gripper to the base coordinate system of the robot arm. EC This is the transformation matrix from the camera coordinate system to the coordinate system of the robotic arm's end effector gripper. S33. Motion planning and execution: Using [R′t′] as the target grasping pose, the robot arm calculates the joint variable values corresponding to the target pose through inverse kinematics. The robot arm motion planning module MoveIt is called to generate the joint space path planning and trajectory planning of the gripper from the current state to the grasping pose. Based on this, control commands are generated and sent to adjust the gripper opening width to w, move the gripper, and execute the gripper closing to complete the grasping action.
10. A method for six-DOF grasping of transparent objects based on RGB-D multimodal information fusion according to claim 9, characterized in that, In step S3, the camera is fixedly mounted on the end gripper of the robotic arm.
Citation Information
Cited By
Robot grabbing method based on geometric topology constraint field and self-adaptive calibration
CN121650013A
A Robot Grasping Method Based on Geometric Topological Constraint Field and Adaptive Calibration
CN121650013B
