A method for guiding a gripping member based on a stationary point cloud sensor

Abstract

The application provides a guiding and grabbing method of a fixed point cloud sensor, comprising the following steps: acquiring a part point cloud; if it is not the first time of grabbing, correcting the part point cloud by using a compensation parameter; then converting the corrected part point cloud to a base coordinate system of a robot; the robot advances to a preparation position according to a preset standard grabbing track, adjusts an end pose of the robot, and forwards to grab the part; after grabbing the part, the robot retreats to the preparation position, and carries out feeding or discharging; wherein one or more three-dimensional targets are fixed at the end of the robot; a calibration process is carried out at the preparation position, and the compensation parameter is acquired, which is used for correcting the part point cloud in the next time of grabbing; by correcting the part point cloud, the accuracy of the part point cloud acquired each time is consistent with the accuracy when the sensor is initially calibrated, the accuracy calibration of various types of point cloud sensors is applicable, the accuracy of visual guiding and grabbing is effectively improved, and grabbing failure is avoided.

Description

Technical Field

[0001] This invention relates to the field of visually guided object grasping, and more specifically to a guided object grasping method based on a fixed point cloud sensor. Background Technology

[0002] With the maturity of 3D point cloud acquisition technology, using point cloud sensors to guide robots in grasping parts is gradually becoming the mainstream method. In this method, point cloud sensors acquire 3D point clouds of the part surface, determine the position of the part to be grasped (such as randomly placed workpieces), and a gripper is installed at the end of the robot to guide and achieve automated grasping of the workpiece using point cloud information.

[0003] Currently, there are various sensor technologies for acquiring 3D point clouds, including grating projection, laser galvanometers, and laser line scanning. In grating projection, the projection unit's performance becomes unstable with temperature changes, leading to fluctuations in the projected position. In laser galvanometers and laser line scanning, prolonged movement of the laser's motion unit and changes in external temperature can also cause performance degradation in these methods, resulting in projection position deviations.

[0004] Inaccuracies in the projection position will ultimately be reflected in the 3D point cloud calculation. That is, the acquired point cloud will differ from the point cloud acquired during initial calibration due to rotation, translation, and scaling. This results in a positional deviation in the point cloud image of the part to be grasped acquired by the point cloud sensor after a period of operation (the exact duration is unknown). Using this deviated point cloud to guide the robot in grasping the part will lead to inaccurate grasping posture and prevent the robot from correctly grasping the workpiece. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a guided part-grabbing method based on a fixed point cloud sensor. This method performs real-time accuracy calibration on the point cloud sensor, and by correcting the part point cloud, ensures that the accuracy of the part point cloud acquired each time is consistent with the accuracy during the initial calibration of the sensor.

[0006] It features easy implementation, short processing time, and high accuracy, and is suitable for precision calibration of various types of point cloud sensors, effectively improving the accuracy of vision-guided object grasping and avoiding grasping failure.

[0007] The technical solution is as follows:

[0008] A guided part-grabbing method based on a fixed point cloud sensor, wherein the point cloud sensor is fixed in the workstation to guide the robot to grasp the part; the part is placed one by one or in batches within the field of view of the point cloud sensor by a mobile device.

[0009] Includes the following steps:

[0010] 1) The point cloud sensor acquires the part point cloud. If it is the first time the part is gripped, then: directly combine the pre-calibrated relationship between the sensor coordinate system and the robot base coordinate system to transform the part point cloud into the robot's base coordinate system.

[0011] Otherwise: retrieve the compensation parameters stored during the last part grabbing, use the compensation parameters to correct the part point cloud; then, combine the pre-calibrated relationship between the sensor coordinate system and the robot base coordinate system, and transform the corrected part point cloud into the robot's base coordinate system.

[0012] 2) The robot moves forward to above the part according to the pre-set standard gripping trajectory. At this time, the robot end effector is within the field of view of the point cloud sensor, and this position is recorded as the preparation position. At the preparation position, the robot controller adjusts the pose of the robot end effector according to the pose change matrix between the part point cloud in the base coordinate system and the pre-stored standard part point cloud. Based on the adjusted end effector pose, the robot moves forward to grip the part. After gripping the part, the robot returns to the preparation position and then performs loading or unloading according to the pre-set standard placement trajectory.

[0013] The robot end effector is fixed with one or more three-dimensional targets, which are spheres or regular polyhedra.

[0014] When the robot moves forward to the preparatory position, or moves back to the preparatory position, a calibration process is performed to obtain the compensation parameters;

[0015] The calibration process includes the following steps:

[0016] ① The point cloud sensor acquires point clouds of at least three stereo targets located at different positions, fits the geometric center coordinates of the stereo targets based on the point clouds, and stores the fitted coordinates into the compensation coordinate set.

[0017] ② Based on the one-to-one correspondence between the compensation coordinate set and each coordinate in the pre-stored calibration coordinate set, the compensation parameters are calculated, including the scaling factor, rotation matrix, and translation matrix;

[0018] ③ Store compensation parameters for correcting the part point cloud during the next part grabbing.

[0019] Preferably, in step 1), if the current part grab is not the first part grab, and the time interval between the current part grab and the previous part grab is greater than a preset value, then a supplementary calibration process is performed to re-acquire the compensation parameters, and the compensation parameters stored in the previous part grab are replaced with the newly acquired compensation parameters, and the part point cloud is corrected using the newly acquired compensation parameters.

[0020] The steps of the supplementary calibration process are as follows:

[0021] The robot moves to the prepared position according to the pre-set standard gripping trajectory;

[0022] Perform steps ① and ②.

[0023] Furthermore, in step ①, the point cloud sensor acquires point clouds of at least three stereo targets located at different positions in the following two ways:

[0024] Method A: At the preparatory position, the robot runs a pre-set calibration trajectory and moves the stereo target within the field of view of the point cloud sensor. During the movement, the point cloud sensor acquires point clouds of the stereo target at at least three different positions.

[0025] At this point, in step ②, the calibration coordinate set is: the geometric center coordinates of the stereo target at different spatial locations obtained by the point cloud sensor when the calibration trajectory is taught.

[0026] The calibration trajectory is as follows: the robot end effector starts from the preparatory position, moves and stops multiple times within the field of view of the point cloud sensor, and finally returns to the preparatory position.

[0027] Method B: When the number of 3-dimensional targets fixed at the robot's end effector is greater than or equal to 3; in step ①, the point cloud sensor acquires the point clouds of at least three 3-dimensional targets located at different positions in the following way:

[0028] At the preparatory position, the robot end effector remains stationary, and the point cloud sensor directly acquires the point cloud of different three-dimensional target surfaces.

[0029] At this point, in step ②, the calibration coordinate set is: the geometric center coordinates of different 3D targets obtained by the point cloud sensor when the teaching robot follows the standard grasping trajectory / standard placement trajectory.

[0030] Furthermore, the pre-stored standard part point cloud is the part point cloud acquired by the point cloud sensor when the teaching robot is tracing the standard part grasping trajectory.

[0031] Furthermore, in step ②, the formula for calculating the compensation parameters is as follows:

[0032]

[0033] Among them, (x i y i , z i (x) represents the i-th coordinate in the calibration coordinate set. i ′ y i ′ , z i ′) represents the i-th coordinate in the compensation coordinate set, and K, R, and T represent the scaling factor, rotation matrix, and translation matrix to be solved, respectively.

[0034] Furthermore, the point cloud sensor includes a camera and a projection device, wherein the projection device is a projector, a laser galvanometer device, or a laser line scanner.

[0035] The method of the present invention has the following characteristics:

[0036] ① This method can perform accuracy calibration of the point cloud sensor while the robot is grasping the part, realizing a new approach to accuracy control of the point cloud sensor. By using a target to obtain offset compensation parameters, the point cloud of the part collected by the point cloud sensor is corrected in real time, the accuracy is calibrated, and accuracy loss is reduced. The corrected point cloud of the part can ensure that the point cloud accuracy is consistent with that of the sensor during calibration, reducing the accuracy loss caused by sensor projection deviation.

[0037] ② No additional space is required for the point cloud sensor's field of view. When the field of view of the point cloud sensor is small or the part to be grasped is large, the field of view of the vision sensor is often filled by the part. At this time, there is no extra space in the workstation to place the stereo target. This method directly installs the stereo target at the end of the robot without occupying the workstation space. At the same time, it utilizes the high repeatability of the robot to ensure that the position of the stereo target is consistent each time. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the calibration process as the robot moves to the preparatory position.

[0039] Figure 2 This is a flowchart illustrating the calibration process when the robot returns to the preparatory position. Detailed Implementation

[0040] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] A guided part-grabbing method based on a fixed point cloud sensor, wherein the point cloud sensor is fixed in the workstation to guide the robot to grasp the part; the part is placed one by one or in batches within the field of view of the point cloud sensor by a mobile device.

[0042] Includes the following steps:

[0043] 1) The point cloud sensor acquires the part point cloud. If it is the first time the part is gripped, then: directly combine the pre-calibrated relationship between the sensor coordinate system and the robot base coordinate system to transform the part point cloud into the robot's base coordinate system.

[0044] Otherwise: retrieve the compensation parameters stored during the last part grabbing, use the compensation parameters to correct the part point cloud; then, combine the pre-calibrated relationship between the sensor coordinate system and the robot base coordinate system, and transform the corrected part point cloud into the robot's base coordinate system.

[0045] 2) The robot moves forward to above the part according to the pre-set standard gripping trajectory. At this time, the robot end effector is within the field of view of the point cloud sensor, and this position is recorded as the preparation position. At the preparation position, the robot controller adjusts the pose of the robot end effector according to the pose change matrix between the part point cloud in the base coordinate system and the pre-stored standard part point cloud. Based on the adjusted end effector pose, the robot moves forward to grip the part. After gripping the part, the robot returns to the preparation position and then performs loading or unloading according to the pre-set standard placement trajectory.

[0046] The robot end effector is fixed with one or more three-dimensional targets, which are spheres or regular polyhedra.

[0047] When the robot moves forward to the preparatory position, or moves back to the preparatory position, a calibration process is performed to obtain the compensation parameters;

[0048] In practice, during a single gripping operation, the robot is in the ready position twice. It can autonomously choose to perform the calibration process during one of these ready positions; for example, such as... Figure 1 This demonstrates a gripping process where the robot advances to a pre-positioned location and performs calibration; such as... Figure 2 The demonstration showed a gripping process where the robot retracts to a preparatory position and performs calibration.

[0049] The calibration process includes the following steps:

[0050] ① The point cloud sensor acquires point clouds of at least three stereo targets located at different positions, fits the geometric center coordinates of the stereo targets based on the point clouds, and stores the fitted coordinates into the compensation coordinate set.

[0051] ② Based on the one-to-one correspondence between the compensation coordinate set and each coordinate in the pre-stored calibration coordinate set, the compensation parameters are calculated, including the scaling factor, rotation matrix, and translation matrix;

[0052] ③ Store compensation parameters for correcting the part point cloud during the next part grabbing.

[0053] To prevent the time span between the current capture and the previous capture from being too large, which could lead to inaccurate compensation parameters, the following processing is also performed:

[0054] In step 1), if this grab is not the first grab and the time interval between the grab and the previous grab is greater than a preset value, then a supplementary calibration process is performed to re-acquire the compensation parameters, replace the compensation parameters stored in the previous grab with the newly acquired compensation parameters, and correct the part point cloud with the newly acquired compensation parameters.

[0055] The steps of the supplementary calibration process are as follows:

[0056] The robot moves to the prepared position according to the pre-set standard gripping trajectory;

[0057] Perform steps ① and ②.

[0058] Specifically, the preset value ranges from 10 minutes to 30 minutes.

[0059] In detail, in step ①, the point cloud sensor acquires point clouds of at least three stereo targets located at different positions in two ways (method A and method B):

[0060] Method A includes the following steps:

[0061] At the preparatory position, the robot runs a pre-set calibration trajectory, moving the stereo target within the field of view of the point cloud sensor. During the movement, the point cloud sensor acquires point clouds of the stereo target at at least three different locations.

[0062] At this point, in step ②, the calibration coordinate set is: the geometric center coordinates of the stereo target at different spatial locations obtained by the point cloud sensor when the calibration trajectory is taught.

[0063] The calibration trajectory is as follows: the robot end effector starts from the preparatory position, moves and stops multiple times within the field of view of the point cloud sensor, and finally returns to the preparatory position.

[0064] Method B, for cases where the number of 3 or more 3D targets are fixed at the robot's end effector, involves the following steps:

[0065] At the preparatory position, the robot end effector remains stationary, and the point cloud sensor directly acquires the point cloud of different three-dimensional target surfaces.

[0066] At this point, in step ②, the calibration coordinate set is: the geometric center coordinates of different 3D targets obtained by the point cloud sensor when the teaching robot follows the standard grasping trajectory / standard placement trajectory.

[0067] More specifically, the pre-stored standard part point cloud is the part point cloud acquired by the point cloud sensor when the teaching robot is tracing a standard part grasping trajectory.

[0068] In step ②, the formula for calculating the compensation parameters is as follows:

[0069]

[0070] Among them, (x i y i , z i (x) represents the i-th coordinate in the calibration coordinate set. i ′ y i ′ , z i ′ ) represents the i-th coordinate in the compensation coordinate set, and K, R, and T represent the scaling factor, rotation matrix, and translation matrix to be solved, respectively.

[0071] The method provided in this embodiment can be applied to various types of point cloud sensors, such as point cloud sensors including cameras and projection devices, where the projection device is a projector, a laser galvanometer device, or a laser line scanner.

[0072] The foregoing description of specific exemplary embodiments of the present invention is for illustrative and descriptive purposes. It is not intended to be exhaustive, nor to limit the invention to the precise forms disclosed; obviously, many changes and variations are possible in accordance with the foregoing teachings. The exemplary embodiments were chosen and described to explain the specific principles of the invention and its practical application, thereby enabling others skilled in the art to implement and utilize various exemplary embodiments of the invention, as well as their different alternatives and modifications. The scope of the invention is intended to be defined by the appended claims and their equivalents.

Claims

1. A method of guiding a gripper based on a stationary point cloud sensor, the point cloud sensor being fixed in a station for guiding a robot to pick a part; The parts are placed in the field of view of the point cloud sensor one by one or in batches through the mobile device; It is characterized by comprising the following steps: 1) The point cloud sensor acquires the part point cloud, if it is the first time to grab the part, then: directly combine the relationship between the sensor coordinate system and the robot base coordinate system which is calibrated in advance, and convert the part point cloud to the base coordinate system of the robot; Otherwise: call the compensation parameters stored when grabbing the part last time, modify the part point cloud by using the compensation parameters; then combine the relationship between the sensor coordinate system and the robot base coordinate system which is calibrated in advance, and convert the modified part point cloud to the base coordinate system of the robot; 2) The robot advances to above the part according to the pre-set standard part grabbing trajectory, at this time, the robot end is in the field of view of the point cloud sensor, and the position is recorded as the preparation position; at the preparation position, the robot controller adjusts the robot end pose according to the pose change matrix between the part point cloud in the base coordinate system and the pre-stored standard part point cloud, and the robot grabs the part based on the adjusted end pose; after grabbing the part, the robot retreats to the preparation position, and then carries out feeding or discharging according to the pre-set standard part placing trajectory; Wherein, the robot end is fixed with one or more three-dimensional targets, and the three-dimensional target is a sphere or a regular polyhedron; When the robot advances to the preparation position or retreats to the preparation position, a calibration process is carried out to acquire the compensation parameters; The calibration process comprises the following steps: ①The point cloud sensor acquires the point cloud of at least three three-dimensional targets located at different positions, and fits the geometric center coordinates of the three-dimensional targets based on the point cloud, and stores the fitted coordinates into a compensation coordinate set; ②Based on the one-to-one correspondence between the compensation coordinate set and the pre-stored calibration coordinate set, the compensation parameters are solved, and the compensation parameters include scaling factor, rotation matrix and translation matrix; ③The compensation parameters are stored for modifying the part point cloud when grabbing the part next time.

2. The method of guiding a gripping member according to claim 1, wherein: In step 1), if the current part grabbing is not the first time, and the time interval between the current part grabbing and the last part grabbing is greater than a pre-set value, then a supplementary calibration process is carried out to re-acquire the compensation parameters, replace the compensation parameters stored in the last part grabbing with the newly acquired compensation parameters, and modify the part point cloud by using the newly acquired compensation parameters; Wherein, the steps of the supplementary calibration process are as follows: The robot advances to the preparation position according to the pre-set standard part grabbing trajectory; Steps ① and ② are executed.

3. The method of guiding a gripping member according to claim 1, wherein: In step ①, the point cloud sensor acquires the point cloud of at least three three-dimensional targets located at different positions in the following way: At the preparation position, the robot runs the pre-set calibration trajectory to drive the three-dimensional target to move in the field of view of the point cloud sensor, and the point cloud sensor acquires the point cloud of the three-dimensional target at at least three different positions during the movement.

4. The method of guiding a gripping member according to claim 3, wherein: In step ②, the calibration coordinate set is the geometric center coordinates of the three-dimensional target at different spatial positions acquired by the point cloud sensor when the calibration trajectory is taught.

5. The method of guiding a gripping member according to claim 3, wherein: The calibration trajectory is that the robot end effector starts from the preparation position, moves and stops multiple times within the field of view of the point cloud sensor, and finally returns to the preparation position.

6. The method of guiding a gripping member according to claim 1, wherein: When the number of solid targets fixed by the robot end effector is greater than or equal to 3, the point cloud sensor acquires the point clouds of at least three solid targets at different positions in the following manner in step ①: At the preparation position, the robot end effector remains stationary, and the point cloud sensor directly acquires the point clouds of the surfaces of different solid targets.

7. The method of guiding a gripping member according to claim 6, wherein: In step ②, the set of calibration coordinates is the geometric center coordinates of different solid targets acquired by the point cloud sensor when the standard part picking trajectory of the teaching robot is demonstrated.

8. The method of guiding a gripping member according to claim 1, wherein: The pre-stored standard part point cloud is the part point cloud acquired by the point cloud sensor when the standard part picking trajectory of the teaching robot is demonstrated.

9. The method of guiding a gripping member according to claim 1, wherein: In step ②, the formula for solving the compensation parameters is as follows: wherein (x i , y i , z i ) represents the i-th coordinate in the calibration coordinate set, (x i ′ , y i ′ , z i ′ ) represents the i-th coordinate in the compensation coordinate set, K, R, T represent the scale factor, the rotation matrix, and the translation matrix to be solved, respectively.

10. The method of guiding a gripping member according to claim 1, wherein: The point cloud sensor includes a camera and a projection device, which is a projector, a laser galvanometer device, or a laser line scanning device.

Images