Method for transparent object depth completion based robotic arm tube grabbing

Abstract

The application discloses a kind of based on transparent object depth completion's mechanical arm tube grabbing method, it is characterized in comprising the following steps: build transparent depth completion model;The point cloud after completion is projected back depth image;Utilize depth camera to obtain the color image and depth image of transparent test tube and carry out alignment;Depth completion model is transplanted to ROS development platform, and network weight parameter is loaded, and the test tube image that is obtained in real time in ROS is handled, and complete depth image is obtained;Mechanical arm obtains the position information of test tube, and the center coordinate system (x t ,y t ,z t ) of test tube top is calculated, and the position of mechanical arm end effector is grabbed, and the pose information of mechanical arm end effector in cartesian space is obtained;According to the pose information under world coordinate system obtained, combined with cubic B spline interpolation algorithm, the trajectory information of mechanical arm end gripper is obtained.This method can be more accurate to estimate the position information of transparent test tube, and quickly plan corresponding grabbing trajectory to complete grabbing.

Description

Technical Field

[0001] This invention belongs to the field of depth camera application technology, specifically a method for a robotic arm to grasp test tubes based on depth completion of transparent objects. Background Technology

[0002] In chemical analysis laboratories, the use of robots mainly involves the grasping and movement of robotic arms. However, the complex and ever-changing environment of laboratory equipment, with its diverse instruments and frequently shifting locations, significantly impacts the success rate of robotic arm grasping. Test tubes are commonly used instruments in every chemical laboratory, requiring their use in many steps of the experimental process. However, as transparent objects, test tubes possess unique visual characteristics, making it difficult for general-purpose RGB-D cameras to capture their complete depth information.

[0003] The unique visual characteristics of transparent objects, such as refraction and reflection, make it difficult for traditional RGBD cameras to accurately capture their depth information. The depth information obtained by cameras for transparent objects often reflects the depth behind the surface or is lost due to specular highlights. For automated grasping robots used in chemical laboratories, the automatic recognition method typically locates the grasping position by analyzing the depth information of each object's surface. However, the unique visual characteristics of transparent test tubes make it difficult for traditional algorithms to obtain ideal depth values, lacking flexibility and stability. With the increasing application of deep learning in image processing, it can more completely complete the depth information of transparent objects and cope with various complex and changing laboratory environments. Therefore, using deep learning methods to process data from transparent test tubes can significantly increase the success rate of robotic arm grasping. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention uses a depth camera to collect real-time depth and color information of the test tube surface. It also proposes a method for a robotic arm to grasp test tubes based on depth completion of transparent objects. The method involves processing the transparent test tube data using a depth completion network to obtain a complete depth map. The position information of the test tube is inferred from the obtained depth data, and the grasping pose information is calculated based on the 3D spatial rotation principle. Then, the trajectory of the robotic arm's end effector is planned using an interpolation algorithm, and the motion information within the joint space is obtained through inverse kinematics. Finally, the robotic arm is controlled to complete the test tube grasping process.

[0005] The method for a robotic arm to grasp test tubes based on depth completion of transparent objects is characterized by the following steps:

[0006] (1) Construct a transparent depth completion model;

[0007] (2) Project the completed point cloud back into the depth image and use it as input for training the depth completion module;

[0008] (3) Use a depth camera to acquire color and depth images of the transparent test tube and align them. The depth camera is fixed at the end of the robotic arm.

[0009] (4) The depth completion model built in step (1) is ported to the ROS development platform, and the network weight parameters saved in step (2) are loaded to process the test tube image obtained in real time in ROS to obtain a complete depth image.

[0010] (5) The robotic arm subscribes to the topics published in step (3) to obtain the position information of the test tube, and combines it with the center coordinate system (x) of the top of the test tube. t ,y t ,z t ) Calculate the position of the robotic arm end effector for grasping, and obtain the pose information of the robotic arm end effector in Cartesian space;

[0011] (6) Based on the pose information in the world coordinate system obtained in step (4), the trajectory information of the end gripper of the robotic arm is obtained by combining the cubic B-spline interpolation algorithm.

[0012] Furthermore, the transparent depth completion model includes a point cloud completion module and a depth completion module. The point cloud completion module is used for preprocessing the depth map, and the depth completion module inputs the relatively complete depth data converted from the point cloud into the depth completion module to further refine the depth information.

[0013] Furthermore, step (1) includes:

[0014] (11) Construct a point cloud completion module to back-project the depth of the transparent object onto the point cloud and estimate the correct depth image by predicting the complete point cloud shape;

[0015] (12) Use the projected sparse point cloud as the input to the point cloud completion module;

[0016] (121) For the disordered sparse point cloud in step (12), a 3D mesh is constructed to convert the disordered point cloud information into regular data that can represent the local information and structure of the point cloud.

[0017] (122) Define a Grid G ​​= <V, W>, where V and W represent the vertex set and value set of the Grid, respectively;

[0018] (123) Determine the coordinate position of the original point cloud and calculate the weight of each vertex;

[0019] (124) Use a 3D CNN encoder-decoder structure to learn the feature data required to complete the point cloud, and transform the grid into an unordered point cloud through Gridding Reverse;

[0020] (125) The point cloud is completed by creating an MLP and the coarse point cloud is further refined.

[0021] Furthermore, in step (11), the equation representing the pixel coordinates of the depth image in 3D space is:

[0022]

[0023] Where (x, y, z) is a three-dimensional coordinate point in space, (x', y') is the coordinate of a pixel in the depth map, D is the depth value, and f x ,f y For camera internal parameters;

[0024] In step (122), in w represents each vertex generated in the mesh. i This represents the weight of each vertex;

[0025] In step (123), the original point cloud is If the coordinates of a point p = (x, y, z) in a point cloud satisfy:

[0026]

[0027] Then this point is taken as vertex v i Neighborhood In this context, the weight corresponding to each vertex can be represented as:

[0028]

[0029] in

[0030] Furthermore, the depth completion module adopts an encoder-decoder architecture. The encoder part uses EfficientNet as the backbone and constructs CECD and CEC modules. The RGB and depth images are stitched together and used as the network input. The depth data is input into a single-channel sampling module for feature extraction, and the depth features of the corresponding resolution are used as the input of each CECD block and CEC block.

[0031] The decoder is a lightweight RefineNet decoder, mainly composed of the CRP module and the FUSE module. The CRP block mainly consists of a 1x1 convolutional layer and a 5x5 max pooling layer, while the FUSE block mainly consists of two 1x1 convolutional layers and an upsampling module. The feature information output by the CEC module is combined with the original depth information as the input of the decoder to recover the complete depth information.

[0032] Furthermore, step (2) includes:

[0033] (21) Through This represents the process of projecting a point cloud back into a depth image;

[0034] (22) Input the noisy depth map generated after point cloud completion and the original RGB color image into the depth completion module;

[0035] (23) The point cloud completion network is trained using Gridding Loss, and its loss function can be expressed as:

[0036]

[0037] in This represents the weight of each vertex in the grid;

[0038] The loss function of a deep completion network can be expressed as:

[0039]

[0040] Where β is the weight parameter. and D gt D represents the predicted depth and the true depth, respectively. h and D w These are the gradient vectors of the depth map D along the width and height coordinate axes, respectively.

[0041] Furthermore, in step (3), the "eye on hand" calibration method is adopted;

[0042] Clear Grasp, TODD, and TransCG were used as training datasets.

[0043] Furthermore, in step (4), after the complete depth image is acquired, the depth map is mapped to coordinate points in 3D space, and the finally obtained coordinate points are published through the ROS message mechanism.

[0044] Furthermore, step (5) includes:

[0045] (51) The center point of the opening and closing of the gripper is defined as the origin of the end coordinate system. The direction of the Z-axis is perpendicular to the gripper plane and downward. The direction of the X-axis is parallel to the gripper gripping direction and forward. The direction of the Y-axis is determined according to the right-hand rule.

[0046] (52) The completed depth map is back-projected onto 3D space, and the complete point cloud can be represented as follows: The transparent test tubes are placed on a test tube rack. The grasping pose is determined by traversing all point clouds, starting with searching for the highest point at the top of the test tube:

[0047]

[0048] For vertex p highest =(x h ,y h ,z h ), search its neighborhood N(p) highest All points within ) that satisfy the following requirements:

[0049]

[0050] Where r T Represent the radius of the test tube opening, and then search the neighborhood N(p) highest The center point (x) c ,y c ,z c This serves as the spatial position for the grippers to grasp the object.

[0051] (53) After obtaining the gripping position of the test tube, calculate the posture of the gripping point. According to the coordinate system setting in (51), the positive direction of the x-axis is the forward direction of the gripping, and the posture of the gripping coordinate system during gripping is the posture when the image is acquired.

[0052] (54) Based on the obtained test tube position information in the camera coordinate system and the transformation relationship between the camera coordinate system and the world coordinate system, the test tube coordinate position in the world coordinate system is obtained. The process can be expressed as follows:

[0053]

[0054] Furthermore, in step (6), the cubic B-spline interpolation formula is defined as follows:

[0055] C3(u)=∑P i N i,3 (u)

[0056] Where P i These are the control points of the spline curve, P i Set as the initial point and the gripping point of the test tube, N i,3These are the basis functions of a cubic spline curve, and their equations can be solved using recursive formulas:

[0057]

[0058] Where k is the degree of the curve;

[0059] The obtained trajectory information is added with timestamps, velocity and acceleration values ​​to obtain a complete motion trajectory, and then converted into motion information in joint space through inverse kinematics, which is sent to the robotic arm control module to complete the test tube grasping task.

[0060] Compared with the prior art, the present invention has the following advantages:

[0061] This invention utilizes an RGBD sensor to collect depth and RGB information from the surface of a transparent test tube. An improved U-Net architecture depth completion model is then used to complete the depth of the transparent test tube. Based on the complete depth information, the grasping pose of the test tube is calculated. Finally, a trajectory planning algorithm is used to obtain the grasping trajectory of the robotic arm. This method can accurately estimate the position information of the transparent test tube and quickly plan the corresponding grasping trajectory to complete the grasping process. Attached Figure Description

[0062] Figure 1 A schematic diagram of a transparent depth completion model. Detailed Implementation

[0063] The invention will now be further described with reference to the accompanying drawings.

[0064] like Figure 1 As shown, a method for a robotic arm to grasp a test tube based on depth completion of a transparent object includes:

[0065] Step 1: Construct a transparent depth completion model as shown in the figure. First, build a point cloud completion module to backproject the depth of the transparent object onto the point cloud. Estimate the correct depth image by predicting the complete point cloud shape. The process of converting depth map pixels to 3D spatial coordinates can be represented as:

[0066]

[0067] Where (x, y, z) is a three-dimensional coordinate point in space, (x', y') is the coordinate of a pixel in the depth map, D is the depth value, and f x ,f y This refers to the camera's internal parameters.

[0068] The projected sparse point cloud will serve as input to the point cloud completion module. Inspired by the point cloud processing module proposed in GRNet, for the input sparse and disordered point cloud, a 3D mesh is first constructed to convert the disordered point cloud information into regular data that can represent the local information and structure of the point cloud. A Grid G ​​= <V, W> is defined, where... These represent the vertex set and value set of the Grid, respectively. For each vertex generated in the mesh, the value represents the weight of that vertex. (Original point cloud) If the coordinates of a point p = (x, y, z) in a point cloud satisfy:

[0069]

[0070] Then this point is taken as vertex v i Neighborhood One point in the equation. Then the weight corresponding to each vertex can be expressed as:

[0071]

[0072] in Then, a 3D CNN encoder-decoder architecture is used to learn the feature data needed to complete the point cloud, and the grid is transformed into an unordered point cloud by Gridding Reverse, with coordinates being the weighted sum of eight vertices. Finally, an MLP is created to complete the point cloud and further refine the coarse point cloud.

[0073] The transparent depth completion model mainly consists of two parts: a point cloud completion (GRNet) module and a depth completion module. The first part preprocesses the depth map, backprojecting the original input depth map into 3D space and using it as input to the GRNet module. The GRNet module then predicts the complete point cloud to estimate the correct depth information. The second part, the more complete depth data derived from the point cloud, is input into the depth completion module to further refine the depth information.

[0074] The depth completion module adopts an encoder-decoder architecture. The encoder uses EfficientNet as the backbone and constructs Conv-Efficient-Conv-Downsample (CECD module) and Conv-Efficient-Conv (CEC module). The RGB and depth images are concatenated and used as the network input. To preserve the original depth information, the depth data is input into a single-channel sampling module for feature extraction, and the depth features corresponding to the resolution are used as the input for each CECD block and CEC block.

[0075] The decoder section employs a lightweight RefineNet decoder, primarily composed of a CRP module and a FUSE module. The CRP module mainly consists of a 1x1 convolutional layer and a 5x5 max-pooling layer, while the FUSE module mainly consists of two 1x1 convolutional layers and an upsampling module. The feature information output by the CEC module is combined with the original depth information as input to the decoder to reconstruct the complete depth information.

[0076] Step 2: Project the completed point cloud back onto the depth image. This process can be represented as follows:

[0077]

[0078] The noisy depth map generated after point cloud completion is then input into the depth completion module along with the original RGB color image. The depth completion module uses an encoder-decoder architecture, employing EfficientNet as the backbone and customizing Conv-Efficient-Conv-Downsample (CECD block) and Conv-Efficient-Conv (CEC block) for feature extraction. A lightweight RefineNet decoder is used for upsampling. The decoder mainly consists of a CRP module composed of 1x1 convolutional layers and 5x5 max-pooling layers, as well as two 1x1 convolutional layers.

[0079] The FUSE module, composed of layers, improves the performance of the algorithm while ensuring accuracy.

[0080] The point cloud completion network is trained using Gridding Loss, and its loss function can be expressed as:

[0081]

[0082] in Representing the weight of each vertex in the grid, the loss function of the depth completion network can be expressed as:

[0083]

[0084] Where β is the weight parameter. and D gt D represents the predicted depth and the true depth, respectively. h and D w These are the gradient vectors of the depth map D along the width and height coordinate axes, respectively.

[0085] Step 3: Acquire and align color and depth images of the transparent test tube using a depth camera. The camera is fixed to the end of the robotic arm, employing an "eye-on-hand" calibration method. This means that hand-eye calibration is required to establish the transformation relationship between the camera coordinate system and the robotic arm's base coordinate system. While keeping the calibration plate within the camera's field of view, the robotic arm is moved to multiple different poses to obtain motion samples from multiple points for calculating the transformation matrix. The datasets used for training are ClearGrasp, TODD, and TransCG. In other words, the ClearGrasp, TODD, and TransCG datasets are used for joint training.

[0086] Step 4: Port the depth completion model built in Step 1 to the ROS development platform, load the network weight parameters saved in Step 2, process the test tube image obtained in real time in ROS to obtain a complete depth image; then map the depth map to coordinate points in 3D space, and publish the finally obtained coordinate points through the ROS message mechanism.

[0087] Step 5: The robotic arm subscribes to the topic published in Step 3 to obtain the position information of the test tube, and combines it with the center coordinate system of the top of the test tube (x... t ,y t ,z t The position of the robotic arm's end effector for grasping is calculated to obtain the pose information of the end effector in Cartesian space; the specific calculation process is as follows:

[0088] a) The end effector of the robotic arm is a PGI parallel electric gripper. The center point of the gripper opening and closing is defined as the origin of the end coordinate system. The direction of the Z-axis is perpendicular to the gripper plane downwards, and the direction of the X-axis is parallel to the gripper gripping direction forwards. The direction of the Y-axis is determined according to the right-hand rule.

[0089] b) Backproject the completed depth map back into 3D space. The complete point cloud can be represented as follows: The transparent test tubes are placed on a test tube rack. We determine the grasping pose by traversing all the point clouds, starting by searching for the highest point on the top of the test tube:

[0090]

[0091] For vertex p highest =(x h ,y h ,z h ), search its neighborhood N(p) highest All points within ) that satisfy the following requirements:

[0092]

[0093] Where rT This represents the radius of the test tube opening. Then, the neighborhood N(p) is searched. highest The center point (x) c ,y c ,z c () serves as the spatial position for the gripper to grasp.

[0094] c) After obtaining the gripping position of the test tube, it is also necessary to calculate the orientation of the gripper at the gripping point. According to the coordinate system settings in a), the positive direction of the x-axis is the forward direction of the gripper gripping. Since the z-axis of the camera coordinate system and the x-axis of the gripper point in the same direction, the orientation of the gripper coordinate system during gripping is equal to the orientation when acquiring the image.

[0095] d) Based on the obtained test tube position information in the camera coordinate system and the transformation relationship between the camera coordinate system and the world coordinate system, the test tube coordinate position in the world coordinate system can be obtained. The process can be expressed as follows:

[0096]

[0097] Step Six: Based on the pose information in the world coordinate system obtained in Step Four, the trajectory information of the robotic arm's end effector gripper is obtained using the cubic B-spline interpolation algorithm. The cubic B-spline interpolation formula is defined as follows:

[0098] C3(u)=∑P i N i,3 (u)

[0099] Where P i These are the control points of the spline curve; here they are set as the initial point and the gripping point of the test tube. N i,3 These are the basis functions of a cubic spline curve, and their equations can be solved using recursive formulas:

[0100]

[0101] Where k is the degree of the curve.

[0102] Finally, timestamps, velocity, and acceleration values ​​are added to the obtained trajectory information to obtain the complete motion trajectory. The motion information in joint space is then converted through inverse kinematics and sent to the robotic arm control module to complete the test tube grasping task.

[0103] This invention utilizes an RGBD sensor to collect depth and RGB information from the surface of a transparent test tube. An improved U-Net architecture depth completion model is then used to complete the depth of the transparent test tube. Based on the complete depth information, the grasping pose of the test tube is calculated. Finally, a trajectory planning algorithm is used to obtain the grasping trajectory of the robotic arm. This method can accurately estimate the position information of the transparent test tube and quickly plan the corresponding grasping trajectory to complete the grasping process.

Claims

1. A method for a robot arm to grab a test tube based on transparent object depth completion, characterized in that, The method comprises the following steps: (1) building a transparent depth completion model; (2) projecting the completed point cloud back to the depth image as the input of the depth completion module for training; the depth completion module adopts an encoder-decoder architecture, the encoder part adopts EfficientNet as the backbone and constructs a CECD module and a CEC module, the RGB and depth images are spliced as the input of the network, the depth data is input into a single-channel sampling module for feature extraction, and the depth features corresponding to the resolution are taken as the input of each CECD block and CEC block; the decoder is a lightweight RefineNet decoder mainly composed of a CRP module and a FUSE module, wherein the CRP block is mainly composed of a 1x1 size convolution layer and a 5x5 size maximum pooling layer, the FUSE block is mainly composed of two 1x1 size convolution layers and an up-sampling module, the feature information output by the CEC module is combined with the original depth information as the input of the decoder, and complete depth information is restored; (21) by representing a process of projecting a point cloud back to a depth image; (22) inputting the noisy depth image generated after the point cloud is completed and the original RGB color image into the depth completion module; (23) training the point cloud completion network using Gridding Loss, and the loss function can be expressed as: ， wherein, represents the weight of each vertex in the grid; the loss function of the depth completion network can be represented as: ， where β is a weight parameter, and D gt denote the predicted and true depth, respectively, D h and D w are the gradient vectors of the depth map D along the width and height coordinate axes, respectively; (3) acquiring color images and depth images of the transparent test tube by using a depth camera and aligning the images, wherein the depth camera is fixed at the end of a mechanical arm; (4) transplanting the depth completion model built in step (1) to a ROS development platform, loading the network weight parameters saved in step (2), and processing the test tube images acquired in real time in ROS to obtain complete depth images; (5) The mechanical arm subscribes to the topic published in step (3) to obtain the position information of the test tube, combines the center coordinate system (x t , y t , z t ) of the top of the test tube to calculate the position of the end effector of the mechanical arm for grabbing, and obtains the pose information of the end effector of the mechanical arm in the Cartesian space; (6) obtaining the trajectory information of the mechanical arm gripper according to the pose information in the world coordinate system obtained in step (4) and combining a cubic B-spline interpolation algorithm; the definition formula of the cubic B-spline interpolation is as follows: C3(u) = ∑P i N i,3 (u) where P i is the control point of the spline curve, P i is set as the initial point and the grabbing point of the test tube, N i ,3 is the base function of the cubic spline curve, and the equation thereof can be solved through a recursive formula: ， wherein k is the order of the curve; adding time stamps, speed and acceleration values to the obtained trajectory information to obtain complete motion trajectories, and converting the motion information to joint space through inverse kinematics to send the motion information to a mechanical arm control module to complete the test tube grabbing task.

2. The method of claim 1, wherein, The transparent depth completion model comprises a point cloud completion module and a depth completion module, the point cloud completion module is used for preprocessing of a depth image, and the depth completion module inputs relatively complete depth data converted from a point cloud into the depth completion module to further refine the depth information.

3. The method of claim 2, wherein The step (1) comprises: (11) constructing a point cloud completion module, projecting the depth of the transparent object back to the point cloud, and estimating the correct depth image by predicting the complete point cloud shape; (12) taking the projected sparse point cloud as the input of the point cloud completion module; (121) constructing a 3D grid to convert the disordered point cloud information into regular data capable of representing the local information and structure of the point cloud for the disordered sparse point cloud in step (12); (122) defining a Grid G=<V,W>, wherein V and W represent the vertex set and value set of the Grid respectively; (123) judging the coordinate position of the original point cloud, and calculating the weight value corresponding to each vertex; (124) Using a 3D CNN encoder-decoder structure to learn the feature data required to complete the point cloud, and converting the Grid to an unordered point cloud through Gridding Reverse; (125) Complete the point cloud by creating an MLP, and further refine the rough point cloud.

4. The method of claim 3, wherein, In the step (11), the pixel point of the depth image is represented by the equation of 3D space coordinates: ， Wherein, (x, y, z) is a three-dimensional coordinate point in space, (x', y') is a pixel point coordinate in the depth map, D is the depth value, f x ,f y is the camera intrinsic parameter; In step (122), wherein, represents each vertex generated in the grid, w i represents the weight of each vertex; In step (123), the original point cloud is , if the coordinate position of a point p = (x, y, z) in the point cloud satisfies: ， then the point is taken as vertex v i Neighborhood N vi The weight corresponding to each vertex is represented as: ， wherein .

5. The method of claim 1, wherein, In the step (3), the "eye on hand" calibration method is adopted; Clear Grasp, TODD and TransCG are used as training data sets.

6. The method of claim 1, wherein, In step (4), after obtaining the complete depth image, the depth image is mapped to the coordinate point in the 3D space, and the finally obtained coordinate point is published through the ROS message mechanism.

7. The method of claim 1, wherein, Step (5) includes: (51) The center point of the opening and closing of the gripper is defined as the origin of the end coordinate system, the downward direction perpendicular to the gripper plane is defined as the Z-axis direction, the forward direction parallel to the gripper grabbing direction is defined as the X-axis direction, and the Y-axis direction is determined according to the right-hand rule; (52) The completed depth map is projected back to the 3D space again, and the complete point cloud can be represented as: The transparent test tube is placed on the test tube rack, and the pose of the grasp is located by traversing all the point clouds. First, search for the highest point on the top of the test tube: ， For a vertex phighest= (x h ,y h ,z h ), search all points in its neighborhood N(p highest ) that satisfy the following requirements: ， wherein r T represents the radius of the test tube mouth, and then searches for the center point (x highest , y c , z c ) of the neighborhood N(p c ) as the spatial position of the gripper grabbing; (53) After obtaining the grabbing position of the test tube, the pose of the grabbing point gripper is calculated, and according to the coordinate system setting in (51), the positive direction of the x-axis is the forward direction of the gripper grabbing, and the pose of the gripper coordinate system during grabbing is the pose during image acquisition; (54) According to the obtained test tube position information in the camera coordinate system and the conversion relationship between the camera coordinate system and the world coordinate system, the test tube coordinate position in the world coordinate system is obtained, and the process can be represented as: 。

Images