A pose optimization method for robot three-dimensional visual grasping

Abstract

The application discloses a posture optimization method for robot three-dimensional visual grabbing, and comprises the following steps: S1, obtaining an estimated posture of a grabbed object in a camera coordinate system; S2, transforming the estimated posture of the grabbed object in the camera coordinate system into a world coordinate system to obtain an estimated posture of the grabbed object in the world coordinate system; and S3, performing grabbing posture optimization according to an actual working environment. The method can set an optimization direction of the posture estimation, reduce the instability of the posture estimation of the object with a symmetrical shape caused by a point cloud registration method, and effectively improve the success rate of grabbing planning.

Description

Technical Field

[0001] This invention pertains to a method for locating workpieces in robot sorting operations, specifically a method for optimizing the gripping posture of a robot during the gripping process using a gripper. Background Technology

[0002] In robotic sorting and grasping operations, there are many workpieces or everyday items with symmetrical cylindrical shapes, such as various types of shafts, sleeves, beverage bottles, connecting pipes, etc. Figure 1 The image shows an example of a cylindrical part to be gripped. This cylindrical object can be gripped from multiple angles, making it ideal for use with robotic grippers.

[0003] Robots typically employ vision technology for assisted localization when grasping such objects, thereby determining the grasping pose. Traditional industrial 2D vision technology requires pre-designing the grasping station, determining the image angle and distance, setting an image template for the object to be grasped, locating the target part in the image, and then mapping the 2D information into 3D data based on scene parameters. While this method is simple in principle, it places high demands on the design of the work unit and the placement of the grasped object, making it unsuitable for rapid deployment and migration of functions. 3D vision-assisted localization technology can be divided into model-based and model-free methods. The model-free method uses an optimized network based on 3D point cloud information to detect the grasping part, selects the grasping posture with the highest grasping quality index, and obtains the position and posture of the workpiece grasping part in the robot coordinate system through coordinate system transformation, thereby planning the robot's gripper posture. This method allows the robot to grasp different types of objects in unstructured environments; however, the optimized network training process is complex, the model has poor anti-interference capabilities, and it is difficult to classify categories. Model-based assisted localization (MAL) techniques typically begin by using image information to initially locate the target workpiece. Then, depth information is combined to calculate the workpiece's point cloud data, which is then registered with an existing 3D workpiece model to obtain the workpiece's 3D pose information. This method eliminates the need for specialized workstations, is suitable for various working environments, avoids complex debugging processes, and allows for rapid deployment. However, because cylindrical workpieces can be grasped from multiple angles, such as... Figure 2 As shown, translation and rotation along the cylinder's rotation axis can yield countless grasping poses. However, the 3D pose information obtained through registration is only one of these countless poses and is not necessarily the optimal grasping pose, which often leads to the failure of grasping planning. Summary of the Invention

[0004] To address the problem of posture selection in 3D vision grasping based on known models, especially for grasping cylindrical objects, and to improve the success rate of robot 3D vision-assisted grasping planning, a posture optimization method for robot 3D vision grasping is proposed.

[0005] A posture optimization method for robot 3D vision grasping, characterized by the following steps:

[0006] S1. Obtain the estimated pose of the object being captured in the camera coordinate system;

[0007] S2. Transform the estimated pose of the object being grabbed in the camera coordinate system to the world coordinate system to obtain the estimated pose of the object being grabbed in the world coordinate system.

[0008] S3. Optimize the grabbing posture based on the actual working environment.

[0009] Step S1 includes:

[0010] S11. Use the depth camera at the end of the machine to capture a three-dimensional model of the target object to be grasped, or create a three-dimensional model of the target object to be grasped based on its actual size.

[0011] S12. Import the two-dimensional information of the model into the virtual working scene and segment it to establish the image segmentation dataset of the model; the import into the virtual working scene includes: processing the surface texture of the model and enriching the background elements;

[0012] S13. Use a neural network to train the image segmentation dataset to obtain the grabbing position of the two-dimensional image of the object to be grabbed.

[0013] S14. Add third-dimensional depth information based on the coordinate position of the original two-dimensional image to obtain the three-dimensional point cloud cluster of the actual target object.

[0014] S15. Set the sampling rate to sample the 3D model of the target object to obtain the sampling point cloud;

[0015] S16. Perform point cloud registration between the sampled point cloud and the target object's 3D point cloud cluster to obtain the estimated pose of the captured object in the camera coordinate system.

[0016] The three-dimensional model includes two-dimensional image pixel information and third-dimensional depth information.

[0017] S2 includes: multiplying the three pose transformation matrices sequentially to obtain the initial pose T of the target object in the world coordinate system.

[0018] T = T1 * T2 * T3 (1)

[0019] Among them, the coordinate transformation matrix of the end effector gripper relative to the robot base is T1; after hand-eye calibration, the homogeneous transformation matrix of the robot end effector camera relative to the robot end effector gripper is T2; and the estimated pose of the grasped object relative to the robot end effector camera is T3.

[0020] The S3 optimizes the grasping posture based on the actual working environment and needs to meet the following parameter requirements:

[0021] a. If the 3D camera is mounted on the robot's end effector, set the optimal direction for the camera's orientation to minimize the camera's rotation angle;

[0022] b. If the gripping process involves picking up the workpiece on a workbench or in a basket, then the optimal direction for the extension and retraction of the gripper should be set from top to bottom using the robot's end effector gripper.

[0023] c. For cylindrical workpieces, the gripping center point perpendicular to the rotation axis is the alternative gripping posture.

[0024] The S3 optimizes the grasping posture based on the actual working environment, including the following steps:

[0025] S31. Calculate the rotation angle of the Zo axis relative to the z axis; if it is less than 10° or greater than 170°, the coordinate system {Xo,Yo,Zo} should be rotated 90° around the Yo axis so that the Xo axis faces the negative z axis, and proceed to the next step; otherwise, the coordinate system {Xo,Yo,Zo} should be rotated around the Zo axis so that the Xo axis direction is parallel to the negative z axis direction in the {x,y,z} coordinate system, so that the target gripper can grasp from top to bottom;

[0026] S32. Calculate the optimized rotation angle of the Zo axis relative to the x axis; if it is less than 90°, further rotate the coordinate system {Xo,Yo,Zo} around the Xo axis by 180° so that the rotation distance of the end camera C relative to the negative x-axis is small, then end; otherwise, end.

[0027] The present invention has the following beneficial effects and advantages:

[0028] The method described in this invention optimizes the estimated pose of an object, allowing for the setting of an optimization direction for pose estimation. This reduces the instability in the estimated pose of symmetrical objects caused by point cloud registration methods, effectively improving the success rate of grasping planning. Attached Figure Description

[0029] Figure 1 This is an example of a cylindrical part to be grasped and a schematic diagram of the possible grasping postures.

[0030] Figure 2 Flowchart for obtaining the estimated pose of the object being captured.

[0031] Figure 3 To capture point cloud data of parts read by the robot in the environment.

[0032] Figure 4 Flowchart for attitude optimization. Detailed Implementation

[0033] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0035] Figure 1 This is an example of a cylindrical part to be grasped and a schematic diagram of the possible grasping postures; the object on the left is in a horizontal posture with the cylinder's rotation axis parallel to the table, and the object on the right is in a vertical posture with the cylinder's rotation axis perpendicular to the table. The grasping angle can be parallel to or perpendicular to the cylinder's rotation axis, among other grasping postures.

[0036] Figure 2 Flowchart for obtaining the estimated pose of the object being captured.

[0037] Figure 3 To capture the point cloud data of the parts read by the robot in the grasping environment, A is a schematic diagram of the measured point cloud of the two grasped objects, B is a schematic position of the robot's end effector gripper, C is a schematic position of the robot's end effector camera, {x,y,z} is the absolute coordinate system, {X,Y,Z} is the transformed pose of the robot's end effector gripper relative to the absolute coordinate system, and {Xo,Yo,Zo} is the assumed transformed pose of the grasped object relative to the absolute coordinate system.

[0038] Figure 4 This is a flowchart for attitude optimization, serving as a general guide for grasping attitude optimization.

[0039] This method is based on 3D vision-assisted localization technology using a known model. After confirming the pose of the object to be grasped, it optimizes the grasping posture based on the characteristic that symmetrical objects such as cylindrical shapes have an infinite number of grasping poses. The specific implementation process is as follows:

[0040] 1. Prepare model data and obtain the estimated pose of the captured object through the trained image segmentation model and point cloud registration.

[0041] As attached Figure 2As shown, a 3D model of the target object is first established. This 3D model is generated by modifying an existing CAD model file of the target object, reconstructing the mesh, adding textures and surface maps, etc., to create a more realistic representation in the virtual scene. This model has two uses: first, it can be imported into the virtual work scene to build an image segmentation dataset for training artificial neural networks; second, it can be used to sample spatial data points of the 3D model according to certain resolution parameters to obtain model point cloud data, which is then used for model registration with the actual collected object point cloud. In the first use, when building the virtual scene, the background elements should be enriched to closely resemble the actual work environment. Multi-view rendering is performed in the simulation environment to generate an image dataset for training object recognition and segmentation algorithms. An image segmentation deep learning network is then used for training, allowing the acquisition of the marked position information of the captured object in the 2D image during practical applications. In the second application of the 3D model, the background point cloud data in the 3D working scene can be removed by using point cloud processing tools based on the position of the object marker in the 2D color image, to obtain the actual point cloud cluster of the target object, and then the actual point cloud cluster can be registered with the point cloud obtained by model sampling to obtain the estimated pose of the captured object.

[0042] 2. Perform coordinate transformation to change the estimated pose of the object to be captured from the camera coordinate system to the world coordinate system.

[0043] by Figure 3 For example, a 3D camera C is installed at the robot's end effector. The estimated pose of the object A obtained in the above steps is a calculation result in a coordinate system with the robot's end effector camera C as the relative origin. However, the robot's end effector gripper B performs the grasping operation. Therefore, it is necessary to transform the estimated pose to a world coordinate system suitable for the robot's inverse kinematics, motion planning, and end effector gripping operation. The principle of this coordinate transformation can be described as follows:

[0044] Let T1 be the coordinate transformation matrix of the end effector gripper relative to the robot base, based on the robot's kinematics calculations; T2 be the homogeneous transformation matrix of the robot's end effector camera relative to the robot's end effector gripper after hand-eye calibration; and T3 be the homogeneous transformation matrix of the estimated pose of the grasped object obtained in the previous step relative to the robot's end effector camera. Multiplying these three pose transformation matrices sequentially yields the initial pose T of the grasped object, as shown in the following equation:

[0045] T = T1 * T2 * T3 (1)

[0046] The initial pose T is the estimated pose of object A relative to the world coordinate system, which is {Xo,Yo,Zo}.

[0047] 3. Optimize the grasping posture by determining optimization parameters based on the actual working environment to improve the success rate of grasping planning.

[0048] The working environment for workpiece sorting and gripping varies greatly, and specific optimization parameters can be determined based on the actual situation. However, the following principles can be summarized in order of priority from high to low:

[0049] If a 3D camera is mounted at the end of a robot, it is more advantageous to minimize the camera's rotation angle to prevent cable tangling; that is, the optimal direction of the camera's orientation can be set.

[0050] Grabbing work typically involves gripping workpieces on a workbench or in a basket, so it is more advantageous for the robot's end effector gripper to grip from top to bottom, meaning the optimal direction for gripper extension and retraction can be set.

[0051] For cylindrical workpieces, it is more advantageous to grip the center point perpendicular to the rotation axis, which means that optimization weights can be set for the alternative gripping poses.

[0052] Based on the above three principles, the following gripping posture optimization scheme can be designed.

[0053] Set up an appendix Figure 3 The world coordinate system shown is the robot base coordinate system {x, y, z}. The transformed pose of the robot's end effector gripper relative to the world coordinate system is the {X, Y, Z} coordinate system. The estimated pose of object A relative to the world coordinate system is the {Xo, Yo, Zo} coordinate system. Controlling the robot's movement so that the transformed pose {X, Y, Z} of gripper B completely coincides with the pose {Xo, Yo, Zo} of object A allows for a grasping operation. However, due to the presence of the worktable and the material basket, this posture inevitably leads to a collision between the robot and the worktable, ultimately causing the grasping plan to fail. Therefore, the following optimization steps are required:

[0054] Step 1: Calculate the rotation angle of the Zo axis relative to the z axis. If it is too close to the z axis (less than 10° or greater than 170°), meaning the rotation axis of the cylindrical object is close to being parallel to the z axis, then there is a risk of collision if the grasping is performed according to the principle in Step 2. The coordinate system {Xo,Yo,Zo} should be rotated 90° around the Yo axis so that the Xo axis points downwards (i.e., the negative direction of the z axis), satisfying the grasping principle in Step 2. After rotation, proceed directly to Step 2 for optimization. Otherwise, if the Zo axis has a certain angle with the z axis, the {Xo,Yo,Zo} coordinate system can be rotated around the Zo axis so that the Xo axis points downwards (i.e., parallel to the negative direction of the z axis in the {x,y,z} coordinate system). This allows the gripper to grasp the target from top to bottom, satisfying the grasping principle in Step 2.

[0055] Step 2: Calculate the optimized rotation angle of the Zo axis relative to the x-axis. If it is less than 90°, then based on the optimization in the previous step, rotate the coordinate system {Xo,Yo,Zo} by 180° around the Xo axis to minimize the rotation distance of the end-point camera C relative to the negative x-axis, thus satisfying the optimization principle; end the process. The flowchart of the above steps is attached. Figure 4 As shown.

[0056] Specific examples:

[0057] Set in the appendix Figure 3 In the environment shown, for the attached Figure 1 The left-hand side of the object is used for grasping pose estimation and optimization.

[0058] First, according to the appendix Figure 2 The process shown is as follows: Prepare data. Based on the CAD model of the target workpiece, construct a virtual simulation environment of the work scene, obtain two-dimensional image information of the target object in the work scene from multiple perspectives, and prepare an image segmentation dataset. Train the image segmentation network model on this dataset, and the training results can achieve pixel-level labeling of the target object appearing in the image. Use this labeling information to obtain the actual point cloud data of the target object in the RGBD camera. Sample the CAD model and extract a data point at 1mm interval to obtain the sampled point cloud data of the target workpiece model. Register the actual collected point cloud data with the point cloud data obtained based on the model sampling to obtain the estimated pose of the grasped object, and calculate the relative pose of the grasped object with respect to the absolute coordinate system {x,y,z} according to formula (1), and set it as the attached position. Figure 3 in {Xo,Yo,Zo}.

[0059] Then calculate the rotation angle of the Zo axis relative to the z axis to determine if the rotation axis of the part is close to the z axis. Based on the coordinate transformation relationship, the z-axis vector... Zo axis vector Based on the relationship between vectors, the following formula can be obtained for calculating the included angle:

[0060]

[0061] If θ is less than 10° or greater than 170°, the Zo axis is considered to be too close to the z axis. In this embodiment, the angle between the Zo axis and the z axis is relatively large, so we continue to the next step of optimization.

[0062] Next, the coordinate system {Xo, Yo, Zo} is rotated around the Zo axis so that the Xo axis points downwards, i.e., the Xo axis is rotated to the same plane as the Zo and Z axes. This allows the gripper to grasp from top to bottom, avoiding unnecessary collisions and improving the success rate of the grasping plan. Since Yo is coplanar and perpendicular to Xo and Zo, the rotation angle can be obtained by calculating the normal vector of the coplanar Zo and Z axes and then using the angle between the normal vector and Yo. Let the normal vector of the coplanar Zo and Z axes be... but

[0063]

[0064] Therefore, rotating around the Zo axis, such that the downward rotation angle σ in the Xo axis direction is...

[0065]

[0066] Rotation direction according to The sign of the value determines the result.

[0067] Finally, the optimized rotation angle of the Zo axis relative to the x-axis is calculated. If it is less than 90°, a further 180° rotation around the Xo axis is performed based on the previous optimization, minimizing the rotation distance of the end-camera C relative to the negative x-axis direction. The formula for calculating the rotation angle is as follows:

[0068]

[0069] In this example, the Zo axis has a small rotation angle relative to the x axis, so it is necessary to rotate it another 180° around the Xo axis so that the camera faces the negative direction of the x axis when capturing.

[0070] After the above optimization calculations, the attached... Figure 3 The grasping pose of the object on the left can be used for robot grasping motion planning, enabling rapid grasping of parts.

[0071] Innovation: This invention designs a pose generation and optimization method for 3D visual grasping. By proposing optimization principles and a three-step rotation optimization process for registration pose, it achieves the effects of setting the optimization direction of pose estimation, solving the instability of point cloud registration, and effectively improving the success rate of grasping planning.

[0072] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should be considered within the scope of protection of the present invention.

Claims

1. A posture optimization method for robot 3D vision grasping, characterized in that, Includes the following steps: S1. Obtain the estimated pose of the object being captured in the camera coordinate system; S2. Transform the estimated pose of the object being grabbed in the camera coordinate system to the world coordinate system to obtain the estimated pose of the object being grabbed in the world coordinate system. S3. Optimize the grasping posture based on the actual working environment; The S3 optimizes the grasping posture based on the actual working environment and needs to meet the following parameter requirements: a. The camera is mounted at the end of the robot, and the optimal direction of the camera orientation is set to minimize the camera rotation angle; b. The gripping process involves gripping the workpiece on the workbench or in the basket, with the optimal direction of the gripper's extension and retraction set from top to bottom. c. For cylindrical workpieces, the gripping center point perpendicular to the rotation axis is the alternative gripping posture; The S3 optimization of the grasping posture based on the actual working environment also includes the following steps: S31. Calculate the rotation angle of the Zo axis relative to the z axis; if it is less than 10° or greater than 170°, the coordinate system {Xo,Yo,Zo} should be rotated 90° around the Yo axis so that the Xo axis faces the negative z axis, and proceed to the next step; otherwise, the coordinate system {Xo,Yo,Zo} should be rotated around the Zo axis so that the Xo axis is parallel to the negative z axis in the {x,y,z} coordinate system, so that the gripper can grasp from top to bottom; S32. Calculate the optimized rotation angle of the Zo axis relative to the x-axis; if it is less than 90°, further rotate the coordinate system {Xo,Yo,Zo} around the Xo axis by 180° to make the camera rotate a smaller distance relative to the negative x-axis, then end; otherwise, end. Where {X,Y,Z} represents the transformed pose of the robot gripper relative to the absolute coordinate system, and {Xo,Yo,Zo} represents the estimated transformed pose of the assumed grasped object relative to the absolute coordinate system.

2. The posture optimization method for robot 3D vision grasping according to claim 1, characterized in that, Step S1 includes: S11. Use the camera at the end of the machine to capture a 3D model of the object being grasped, or create a 3D model of the object being grasped based on its actual size. S12. Import the two-dimensional information of the model into the virtual working scene, and perform segmentation to establish the image segmentation dataset of the model; S13. Use a neural network to train the image segmentation dataset to obtain the grabbing position of the two-dimensional image of the object to be grabbed. S14. Add third-dimensional depth information based on the coordinate position of the original two-dimensional image to obtain the three-dimensional point cloud cluster of the actual target object; S15. Set the sampling rate to sample the 3D model of the object being captured and obtain the sampling point cloud; S16. Perform point cloud registration between the sampled point cloud and the actual target object's 3D point cloud cluster to obtain the estimated pose of the captured object in the camera coordinate system.

3. The posture optimization method for robot 3D vision grasping according to claim 2, characterized in that, The three-dimensional model includes two-dimensional image pixel information and third-dimensional depth information.

4. The posture optimization method for robot 3D vision grasping according to claim 1, characterized in that, S2 includes: multiplying the three pose transformation matrices sequentially to obtain the initial pose T of the actual target object being grasped in the world coordinate system. T=T1 T2 T3 (1) Among them, the coordinate transformation matrix of the gripper relative to the robot base is T1; after hand-eye calibration, the homogeneous transformation matrix of the robot's camera relative to the gripper is T2; and the homogeneous transformation matrix of the estimated pose of the grasped object relative to the robot's camera is T3.

Images