Mechanical arm correction method and system based on image processing

CN119057766APending Publication Date: 2024-12-03DELTA ELECTRONICS INC(CN)
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202310631217.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

Smart Images

  • Figure CN119057766A_ABST
    Figure CN119057766A_ABST
Patent Text Reader

Abstract

A mechanical arm correction method based on image processing is used for correcting a mechanical arm and comprises the steps that the visiting sequence of at least one correction point on a correction plate is set; and according to the visiting sequence, driving the mechanical arm to enable a photographing device at the tail end of the mechanical arm to move to at least three photographing positions, and respectively photographing at least one image at the at least three photographing positions through the photographing device, wherein the step of generating at least one image at each of the at least three shooting positions by the shooting device comprises the following steps of: generating at least one image after identifying that the shooting range of the shooting device covers at least one correction point; and calculating vector information of at least one correction point according to the at least one image so as to adjust the mechanical arm. The invention further relates to a mechanical arm correction system based on image processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a processing technology, and more particularly to a robot arm calibration method and system based on image processing. Background Art

[0002] With the advancement of industrial technology, various automation devices have been widely developed for application in life and industry. Generally speaking, a robot arm is an important component of an automation device. Although the stability of processing by a robot arm is much greater than that of manual labor, due to the large number of joints in the robot arm, the accumulation and transmission of multiple errors (for example, errors generated by the transmission mechanism or belt during adjustment) cause the accuracy of the robot arm to be generally low. How to correct these errors to improve the accuracy of robot arm processing is a problem that technicians in this field are eager to solve. Summary of the invention

[0003] The present disclosure proposes a robot arm calibration method based on image processing, which is used to calibrate the robot arm, including: setting a visiting order of at least one calibration point on a calibration plate; and driving the robot arm according to the visiting order so that a photographic device on a robot arm end of the robot arm moves to at least three shooting positions, and the photographic device is used to respectively shoot at least one image at the at least three shooting positions, and the at least one image has a center point. The step of generating at least one image at the at least three shooting positions by the photographic device includes: generating at least one image after identifying that the shooting range of the photographic device covers at least one calibration point; and calculating vector information of at least one calibration point from at least one image to adjust the robot arm.

[0004] The present disclosure also proposes a robot arm calibration system based on image processing, including a calibration plate, a robot arm, a photographic device and a controller. The calibration plate includes at least one calibration point. The photographic device is disposed at the end of the robot arm of the robot arm to photograph the calibration plate. The controller is connected to the robot arm and is used to control the robot arm to perform the following operations: setting a visiting sequence of at least one calibration point on the calibration plate; and driving the robot arm according to the visiting sequence so that a photographic device on the end of a robot arm of the robot arm moves to at least three shooting positions, and at least one image is photographed at the at least three shooting positions by the photographic device, and at least one image has a center point respectively, wherein the step of generating at least one image by the photographic device at at least three shooting positions includes: generating at least one image after identifying that the shooting range of the photographic device covers at least one calibration point; and calculating the vector information of at least one calibration point from at least one image to adjust the robot arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1A schematic diagram of a robotic arm correction system based on image processing according to some embodiments of the present disclosure is shown.

[0006] Figure 2A A flowchart of a robotic arm calibration method based on image processing in some embodiments of the present disclosure is shown.

[0007] Figure 2B Showing some embodiments of the present disclosure Figure 2A Flow chart of further steps of the image processing based robotic arm calibration method.

[0008] Figure 2C Showing some embodiments of the present disclosure Figure 2B Flow chart of further steps of the image processing based robotic arm calibration method.

[0009] Figure 3 A schematic diagram showing photographing a correction plate at at least three photographing positions in some embodiments of the present disclosure is shown.

[0010] Figure 4A , Figure 4B , Figure 4C A schematic diagram illustrating at least one image in some embodiments.

[0011] Figure 5 A schematic diagram showing multiple joints and the end of a robotic arm in some embodiments of the present disclosure.

[0012] Description of reference numerals:

[0013] 1: Robotic arm calibration system based on image processing

[0014] 11: Robotic Arm

[0015] 111: End of the robotic arm

[0016] 12: Photographic Installation

[0017] 13: Calibration plate

[0018] 14: Controller

[0019] p~r: calibration point

[0020] ΔC p,r , ΔC p,q : The second vector

[0021] S210~S240: Steps

[0022] VSp, VSq, VSr: Shooting range

[0023] IMGp, IMGq, IMGr: Image

[0024] Op, Oq, Or: center point

[0025] ΔV p , ΔV q , ΔV r : 2D pixel difference

[0026] B: Base

[0027] J1~J3: Joints

[0028] LK1~LK4: Connection part

[0029] a1~a2: length

[0030] q1, q2, q4: axis angle

[0031] w: horizontal length

[0032] q3, h: vertical offset

[0033] Z0, Z1, Z2, Z3, Z4, Z E , X0, X1, X2, X3, X4, X E : Coordinate axis DETAILED DESCRIPTION

[0034] During the operation of the robot arm, the origin of the end of the robot arm may be offset due to various unexpected conditions (for example, offset caused by assembly or manufacturing, offset caused by power failure or external impact, offset caused by the transmission mechanism or belt during adjustment, etc.). In view of this, the present disclosure proposes a non-contact robot arm calibration method and system based on image processing, which uses the calibration points on the calibration plate to calculate the distance between the calibration points, and uses these distances to calculate the length offsets of the multiple connecting rods of the robot arm and the axis angle offsets of the multiple reference points of the robot arm, thereby correcting the offset of the origin of the end of the robot arm.

[0035] Figure 1 Schematic diagram of a robotic arm calibration system 1 based on image processing according to some embodiments of the present disclosure is shown. Figure 1 As shown, the robotic arm calibration system 1 based on image processing includes a robotic arm 11 , a photographing device 12 , a calibration plate 13 and a controller 14 .

[0036] Furthermore, the photographic device 12 is disposed at the position of the end 111 (i.e., the flange) of the robotic arm 11. The robotic arm 11 drives the photographic device 12 to move to at least three shooting positions, and the photographic device 12 shoots the correction plate 13 at at least three shooting positions to generate at least one image respectively. In other words, at each shooting position, the photographic device 12 shoots an image of the correction plate 13. Further, the photographic device 12 mainly obtains images in a two-dimensional space of the actual space, and performs judgment and analysis based on the images. In other words, the overall field of view of the photographic device 12 can be a two-dimensional space.

[0037] In some embodiments, the correspondence between the pixel distance (i.e., the distance between pixels) in the image captured by the photographic device 12 and the distance in the actual space can be pre-stored in the controller 14 or other remote storage devices (e.g., a remote database or a cloud hard drive, etc.). In other words, the absolute distance in the actual space (i.e., the real distance in the real field) can be directly calculated from the distance in the image captured by the photographic device 12 according to the above correspondence. In some embodiments, the pixel position of the center point of the image (i.e., the coordinates in the X and Y directions in the image) will be adjacent to the pixel position of one of the calibration points in the image. In some embodiments, the photographic device 12 can be implemented by various image acquirers, such as a camera or a photosensitive device, etc., to acquire an image of the calibration plate 13 within a preset shooting range (field of view, FoV).

[0038] Furthermore, the robot arm 11 includes a plurality of links and a plurality of reference points between these links. In some embodiments, the robot arm 11 may be, for example, but not limited to, a six-axis robot arm or a four-axis SCARA robot arm. In some embodiments, the plurality of reference points may include a plurality of joints on the robot arm 11 and a robot arm end 111. In some embodiments, the lengths of the plurality of links between the plurality of joints may also be pre-set to be stored in the controller 14 or other remote storage device. In some embodiments, each of the plurality of joints may have a sensor (not shown) to rotate the shaft angle of each of the plurality of joints. In some embodiments, when the robot arm 11 is a six-axis robot arm, each of the plurality of joints may rotate in a three-axis direction (X, Y, Z directions in the real field). In some embodiments, when the robot arm 11 is a four-axis SCARA robot arm, each of the plurality of joints may rotate in a vertical single-axis direction (Z direction in the real field).

[0039] Furthermore, the correction plate 13 has at least one correction point p~r. In some embodiments, the correction points p~r are respectively in the shooting range of the photographic device 12 at at least three shooting positions. In some embodiments, the shooting positions at which the photographic device 12 can shoot the correction points p~r can be pre-set, and the absolute coordinates of these shooting positions in the actual space (i.e., the cartesian coordinates in the real field) are also stored in the controller 14 or other remote storage devices. In some embodiments, the normal vector N of the entire plane of the correction plate 13 is parallel to the Z direction in the real field. It is worth noting that the shooting range of the photographic device 12 at at least three shooting positions only needs to cover the correction points p~r, and there is no need to align the center point of the shooting range with the correction points p~r. This approach will reduce the error generated by the mechanical arm 11 when fine-tuning (for example, the error generated by the transmission mechanism or belt when adjusting) and save the time spent on fine-tuning. In addition, although three correction points p~r are taken as an example here, in actual applications, the number of correction points can also be any other positive integer greater than 3.

[0040] It is worth noting that the transformation matrix between the robot arm coordinate system used by the base of the robot arm 11 and the absolute coordinate system in the real space (i.e., the homogeneous transformation matrix) is also pre-set to be stored in the controller 14 or other remote storage device. In some embodiments, the correction plate 13 can be any checkerboard or grid correction plate for calibrating the robot arm 11. In some embodiments, the correction points p~r can be marks of various different patterns on the correction plate 13 (for example, marks of patterns of triangles, stars or circular centers). In some embodiments, the absolute coordinates of the correction points p~r in the real space can also be pre-stored in the controller 14 or other remote storage device.

[0041] Furthermore, the controller 14 is connected to the robot arm 11 and controls to drive the robot arm 11 to move the photographing device 12. In some embodiments, the controller 14 can control to drive the robot arm 11 to move the photographing device 12 to the shooting position according to the respective lengths of the multiple links of the robot arm 11 and the respective axis angles of the multiple reference points between the links. In some embodiments, the controller 14 can be implemented by a processing circuit or a dedicated logic circuit, such as a processor, a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0042] The operation of the robotic arm calibration system 1 based on image processing will be further described below. Figure 2A , Figure 2A A flowchart of a robotic arm calibration method based on image processing in some embodiments of the present disclosure is shown. Figure 1 The components in the image processing-based robotic arm calibration system 1 are used to execute steps S210 to S240 in the image processing-based robotic arm calibration method.

[0043] like Figure 2A As shown, first, in step S210, the controller 14 sets the visiting order of at least one calibration point p~r on the calibration plate 13. In some embodiments, the visiting order can be pre-stored in the controller 14. In some embodiments, the visiting order can indicate: visiting at least one calibration point p~r in sequence from near to far from the initial position of the robot arm 11. For example, assuming that the calibration point p is the closest position to the initial position of the robot arm 11, and the calibration point r is the farthest position from the initial position of the robot arm 11, the robot arm 11 can be moved to the position adjacent to the calibration points p~r in sequence, and the calibration points p~r can be respectively in the shooting range of the photographic device 12 at these three shooting positions.

[0044] In some embodiments, the visiting sequence may indicate: arbitrarily selecting a first calibration point from at least one calibration point p-r as the initial position, and sequentially visiting a second calibration point and a third calibration point from at least one calibration point according to the shortest path. In some embodiments, a shortest path algorithm (e.g., Dijkstra algorithm or shortest path faster algorithm (SPFA)) may be used to generate the shortest path. In some embodiments, the visiting sequence may indicate: arbitrarily selecting a first calibration point from at least one calibration point p-r as the initial position, and returning to the initial position after visiting a second calibration point from at least one calibration point p-r, and then visiting a third calibration point from at least one calibration point p-r.

[0045] Furthermore, if Figure 2A As shown, in step S220, the robotic arm 11 is driven according to the visiting sequence so that the photographic device 12 on the robotic arm end 111 of the robotic arm 11 moves to at least three shooting positions, and at least one image is respectively captured at the at least three shooting positions by the photographic device 12. The at least one image has a center point. In some embodiments, the visiting sequence of the photographic device 12 moving to the at least three shooting positions can be pre-set by the user and stored in the controller 14 or other remote storage device.

[0046] The following is a practical example of photographing the correction plate 13 at at least three photographing positions. Figure 3 , Figure 3 FIG. 1 is a schematic diagram showing photographing the correction plate 13 at at least three photographing positions in some embodiments of the present disclosure. Figure 3 As shown, it is assumed that the photographic device 12 can be moved to an initial position in advance, the correction point closest to the initial position is the correction point q, and the correction point farthest from the initial position is the correction point r. At this time, the correction points q~r are not in the shooting range VSo of the photographic device. Then, the photographic device 12 can be moved to the first shooting position. At this time, the correction point p is in the shooting range VSp of the photographic device 12, and the photographic device 12 will shoot the correction plate 13 at the first shooting position. In addition, after shooting at the first shooting position is completed, the photographic device 12 can be moved back to the initial position. In other embodiments, the initial position can also be directly set as the position of the correction point p, and the correction point r and the correction point q can be visited in sequence.

[0047] Next, the photographic device 12 can be moved to the second shooting position. At this time, the correction point q is within the shooting range VSq of the photographic device 12, and the photographic device 12 will shoot the correction plate 13 at the second shooting position. In addition, after shooting at the second shooting position is completed, the photographic device 12 can be moved back to the initial position. Finally, the photographic device 12 can be moved to the third shooting position. At this time, the correction point r is within the shooting range VSr of the photographic device 12, and the photographic device 12 will shoot the correction plate 13 at the third shooting position. In addition, after shooting at the third shooting position is completed, the photographic device 12 can be moved back to the initial position.

[0048] The following will further Figure 2A The operation of step S220 in FIG. Figure 2B , Figure 2B Showing some embodiments of the present disclosure Figure 2A Flow chart of further steps of the image processing based robotic arm calibration method. Figure 1 The components of the image processing-based robot arm calibration system 1 are used to perform Figure 2A Step S220 in the further steps S221 to S222. Figure 2B As shown, first, Figure 2B As shown, in step S221, the controller 14 generates at least one image after recognizing that the shooting range of the camera 12 covers at least one calibration point p-r. In other words, an image will be generated as long as the shooting range of the camera 12 covers the calibration point.

[0049] Furthermore, in step S222, vector information of at least one calibration point p-r is calculated from at least one image to adjust the robot arm 11. In other words, the robot arm can be calibrated and adjusted based on the vector information of the calibration points p-r calculated from the image. In some embodiments, the vector information may include a plurality of first vectors and a plurality of second vectors.

[0050] Furthermore, the following embodiments will further illustrate the generation of vector information and Figure 2B The operation of step S222 in is described below. Figure 2C , Figure 2C Showing some embodiments of the present disclosure Figure 2B Flow chart of further steps of the image processing based robotic arm calibration method. Figure 1 The components of the image processing-based robot arm calibration system 1 are used to perform Figure 2C Step S222 in the further step S222A to S222C. Figure 2C As shown, first, Figure 2C As shown, in step S222A, the controller 14 calculates a plurality of first vectors between the positions of the calibration points p-r in at least one image and the center point of each of the at least one image, and calculates a plurality of vector differences between the plurality of first vectors. In some embodiments, the controller 14 may identify a two-dimensional pixel difference between the pixel position of the calibration point in each of the at least one image and the pixel position of the center point in each of the at least one image (for example, the two-dimensional pixel difference between the pixel position of the calibration point in the image and the pixel position of the center point is 3 pixels in the X direction and 2 pixels in the Y direction). Then, the controller 14 may convert the plurality of two-dimensional pixel differences identified from the at least one image into a plurality of first vectors in the real space (for example, 3 pixels in the X direction and 2 pixels in the Y direction in the image are converted into a first vector, and the first vector is composed of 3 cm in the X direction, 2 cm in the Y direction, and 0 cm in the Z direction in the real space).

[0051] The following is an actual example to illustrate the first vector. FIG. 4A to FIG. 4C , FIG. 4A to FIG. 4C Schematic diagram showing at least one image IMGp, IMGq, IMGr in some embodiments. FIG. 4A to FIG. 4C , the images IMGp, IMGq, IMGr have multiple center points Op, Oq, Or. Thus, multiple two-dimensional pixel differences ΔV between the correction points p~r in the images IMGp, IMGq, IMGr and the center points Op, Oq, Or can be calculated. p , ΔV q , ΔV r , where the two-dimensional pixel difference ΔV p , ΔV q , ΔVr Both include the X-direction component and the Y-direction component of the pixel number. p , ΔV q , ΔV r The vectors are converted into a plurality of first vectors in the real space, and a plurality of vector differences between the plurality of first vectors are calculated.

[0052] Furthermore, if Figure 2C As shown, in step S222B, the controller 14 calculates a plurality of second vectors between the calibration points p-r on the calibration plate 13, and obtains a plurality of axis angles of a plurality of reference points on the robot arm 11 when the photographing device 12 is at least three shooting positions. In some embodiments, the controller 14 may calculate a plurality of second vectors between the absolute coordinates of the calibration points p-r in the real space.

[0053] The following is an actual example to illustrate the second vector. Figure 1 As shown, the second vector ΔC of the three-dimensional distance difference between the correction points p and r can be calculated according to the absolute coordinates of the stored correction points p~r in the actual space. p,r and the second vector ΔC of the three-dimensional distance difference between the correction points p and q p,q , where the second vector ΔC p,r , ΔC p,q They are all 3×1 vectors, and the elements in the vectors are the distance differences in the X, Y, and Z directions respectively.

[0054] Furthermore, if Figure 2C As shown, in step S222C, the controller 14 calculates the length offset vectors of the multiple links on the robot arm 11 and the rotation offset vectors of the multiple reference points on the robot arm 11 according to the multiple vector differences, the multiple second vectors and the multiple axis angles, so as to calibrate the robot arm 11 using the length offset vector and the rotation offset vector. In some embodiments, the axis angle may be the angle at which the multiple reference points on the robot arm 11 rotate when the camera 12 is in at least three shooting positions. In some embodiments, the length offset vector may include the length offset of each of the multiple links, and the rotation offset vector may include the axis angle offset of each of the multiple reference points. In this way, the length of each of the multiple links can be adjusted using the length offset of each of the multiple links, and the axis angle offset of each of the multiple reference points can be adjusted using the axis angle offset of each of the multiple rotations.

[0055] In some embodiments, the controller 14 may generate a prediction function of the robot arm 11 using inverse kinematics, and calculate a plurality of predicted positions when the camera 12 moves to at least three shooting positions according to a plurality of axis angles and the prediction function, wherein the prediction function uses a plurality of lengths of a plurality of links and a plurality of axis angles of a plurality of reference points as variables. Then, the controller 14 may calculate a length offset vector and a rotation offset vector using the prediction function according to a plurality of vector differences, a plurality of second vectors, and a plurality of predicted positions.

[0056] The following is an actual example to illustrate the prediction function. Figure 5 , Figure 5 Schematic diagram showing multiple joints J1-J3 and a robot arm end 111 in some embodiments of the present disclosure. Figure 5 As shown, the robot 11 is a four-axis SCARA robot, which includes multiple joints J1~J3 and a robot end 111, wherein the joint J1 is connected to the base B of the robot 11, the axis angles of rotation of the joints J1~J3 are q1, q2, and q4 respectively, the length of the connection part LK1 between the joints J1~J2 is a1, the length of the connection part LK2 between the joints J2~J3 is a2, the vertical offset between the joints J2~J3 is q3 (that is, the length of the connection part LK3), the joint J3 and the robot end 111 are connected by the connection part LK4, the length of the connection part LK4 in the horizontal direction is W, and the vertical offset between the joint J3 and the robot end 111 is h (that is, the length of the connection part LK4 in the vertical direction).

[0057] In detail, due to Figure 5 In order to observe the side view of the robot arm in the Y direction, for the sake of convenience, the Y axis can be ignored below (because the Y direction of all coordinate systems is the same) to illustrate the axis angle of the above rotation. The origin and coordinate axis of the robot arm coordinate system of base B overlap with the origin and coordinate axis of the coordinate system of joint J1 (that is, the origin overlaps at joint J1, the coordinate axes Z0 and Z1 overlap, and the coordinate axes X0 and X1 overlap). At this time, the coordinate axis X1 can be rotated counterclockwise by the axis angle q1 with the coordinate axes Z0 and Z1 as the center, and the coordinate axis X2 can be rotated counterclockwise by the axis angle q2 with the coordinate axis Z2 of the coordinate system of joint J2 as the center. Next, the origin and coordinate axes of the coordinate system at the end of the connection part LK3 overlap with the origin and coordinate axes of the coordinate system of the joint J3 (that is, the origin overlaps at the joint J3, the coordinate axes Z3 and Z4 overlap, and the coordinate axes X3 and X4 overlap). At this time, the coordinate axis X4 can be rotated counterclockwise by an axis angle q4 with the coordinate axes Z3 and Z4 as the center.

[0058] Next, based on the structure and parameters of the above-mentioned robot arm 11, multiple transformation matrices between the robot arm coordinate system of the base B, the coordinate system of the joints J1~J2, the coordinate system of the end of the connection part LK3, the coordinate system of the joint J3 and the arm end coordinate system of the robot arm end 111 can be calculated. These transformation matrices will be expressed by the following formulas (1)~(5).

[0059]

[0060]

[0061]

[0062]

[0063]

[0064] in is the transformation matrix between the robot arm coordinate system of base B and the coordinate system of joint J1, is the transformation matrix between the coordinate system of joint J1 and the coordinate system of joint J2, is the transformation matrix between the coordinate system of the joint J2 and the coordinate system of the end of the connection part LK3, is the transformation matrix between the coordinate system of the end of the connection part LK3 and the coordinate system of the joint J3, is the transformation matrix between the coordinate system of the joint J3 and the arm end coordinate system of the robot end 111, is the rotation matrix among these transformation matrices, is the displacement matrix in these transformation matrices. Thus, the function of the following formula (6) can be generated from the transformation matrix.

[0065]

[0066] Among them A E (θ R ) is the four-axis expression, is the displacement matrix between the robot arm coordinate system and the arm end coordinate system, which is also the above prediction function. The product of x 、p y 、p z are the distance differences in the X, Y, and Z directions between the origin of the robot arm coordinate system and the origin of the arm end coordinate system, θ R It is the angle difference between the Z axis of the robot arm coordinate system and the Z axis of the arm end coordinate system.

[0067] It can be seen that the prediction function uses the multiple lengths of multiple connecting rods (i.e., the above-mentioned connecting parts LK1~LK4) and the multiple axis angles of multiple reference points (i.e., the above-mentioned joints J1~J3) as variables. Once these detected lengths and these axis angles are brought in, the absolute coordinates of the end of the robot 111 in the actual space (i.e., the above-mentioned predicted position) can be predicted. It is worth noting that this absolute coordinate is predicted by the prediction function, and therefore, it may not be equivalent to the actual absolute coordinates of the end of the robot 111 in the actual space (it may be because the connecting rods and joints of the end of the robot 111 are offset after the robot 11 has been used for a long time).

[0068] Under ideal conditions, when the controller 14 controls to drive the photographic device 12 on the robotic arm 11 to move to 2 or more shooting positions according to the set shooting positions, the difference between the actual positions of the photographic device 12 at each 2 shooting positions minus the difference between the predicted positions calculated by the photographic device 12 at each 2 shooting positions will be equal to 0.

[0069] The following Figure 1 as well as FIG. 4A to FIG. 4B To explain, such as Figure 1 as well as FIG. 4A to FIG. 4B As shown, there are three shooting positions at this time, and the second vector between the correction points p and q and the second vector between the correction points p and r are ΔC p,r , ΔC p,q Assume that the two-dimensional pixel difference ΔV in images IMGp and IMGq can be p , ΔV q Generate multiple vector differences ΔV in real space p,q , which can be obtained by the two-dimensional pixel difference ΔV in images IMGp and IMGr p , Δv r Generate multiple vector differences ΔV in real space p,r , at this time ΔC p,q , ΔV p,q The sum of is the difference between the actual position of the photographing device 12 at the shooting position corresponding to the calibration points p and q, and ΔC p,r , ΔV p,r The sum of is the difference between the actual position of the photographing device 12 at the shooting position corresponding to the calibration points p and r. Thus, the above difference can be expressed by the following formulas (7) to (8).

[0070]

[0071]

[0072] in is the prediction function obtained above (i.e., ), is the predicted position of the robot arm end 111 when the photographing device 12 is at the photographing position corresponding to the calibration point q, θ q is the sum of multiple axis angles when the photographic device 12 is at the photographing position corresponding to the calibration point q, is the predicted position of the robot arm end 111 when the photographing device 12 is at the photographing position corresponding to the calibration point p, θ p is the sum of multiple axis angles when the photographic device 12 is at the shooting position corresponding to the calibration point p, is the predicted position of the robot arm end 111 when the photographing device 12 is at the photographing position corresponding to the calibration point r, θ r It is the sum of multiple axis angles when the photographing device 12 is at the photographing position corresponding to the calibration point r.

[0073] Next, two 3×1 vectors can be generated from the above formulas (7) and (8). Ideally, these two vectors should be zero vectors. However, due to the errors in length or axis angle caused by manufacturing or assembly, these two vectors may not be 0. The following will explain how to calculate the errors in length or axis angle caused by manufacturing or assembly.

[0074] In some embodiments, the controller 14 may calculate partial derivatives of multiple lengths and multiple axis angles under the prediction function to generate a Jacobian matrix. Then, the controller 14 may calculate a length offset vector and a rotation offset vector based on multiple vector differences, multiple second vectors, multiple predicted positions, the Jacobian matrix, and multiple axis angles. Then, the controller 14 may use the length offset vector and the rotation offset vector to correct multiple lengths of multiple links and multiple axis angles of multiple rotation axes on the robot arm 11.

[0075] The following is a practical example to illustrate the calculation of the length offset vector and the rotation offset vector.

[0076] As shown in the following formula (9).

[0077]

[0078] Where J is the The calculated Jacobian matrix, J q is the matrix when J is brought into the multiple lengths and multiple axis angles of the photographing device 12 at the shooting position corresponding to the calibration point q, J p is the matrix when J is brought into the multiple lengths and multiple axis angles of the photographing device 12 at the shooting position corresponding to the calibration point p, J rJ is the matrix when the photographic device 12 is at multiple lengths and multiple axis angles in the shooting position corresponding to the correction point r, Δa is the length offset vector composed of the length offsets of each of the multiple links on the robotic arm 11, and Δθ is the rotation offset vector composed of the axis angle offsets of each of the multiple reference points on the robotic arm 11.

[0079] Thus, the length offset vector and the rotation offset vector can be calculated by the simultaneous equations of formula (9), and the length offset vector and the rotation offset vector are used to adjust the lengths of the multiple links on the robot arm 11 and the axis angles of the multiple reference points.

[0080] In some embodiments, the controller 14 may determine whether the difference between the actual position of the camera 12 at each of the two shooting positions minus the difference between the predicted position calculated by the camera 12 at each of the two shooting positions is less than a convergence value, wherein the convergence value may be preset by the user. Then, when it is not less than the convergence value, the above-mentioned simultaneous equations will continue to be used to calculate the length offset vector and the rotation offset vector to calibrate the robot arm 11. Otherwise, the length offset vector and the rotation offset vector will not be calculated to end the robot arm calibration method based on image processing disclosed in the present invention.

[0081] Through the above steps, the length offset vector and the rotation offset vector can be calculated using a non-contact image shooting method. In addition, when it is not less than the convergence value, a new length offset vector and a new rotation offset vector can be continuously calculated to correct the robot arm 11 until it is less than the convergence value. Through such a continuous correction method, the offset of the origin of the robot arm end 111 of the robot arm 11 can be corrected.

[0082] In summary, the present disclosure provides a robot arm calibration method based on image processing, which utilizes the calibration points on the calibration plate to calculate the distance between the calibration points, and utilizes these distances to calculate the length offsets of each of the multiple connecting rods of the robot arm, and the axis angle offsets of each of the multiple reference points of the robot arm, and utilizes the length offsets of each of the multiple connecting rods and the axis angle offsets of each of the multiple reference points to achieve the origin calibration of the robot arm. Therefore, during the operation of the robot arm, if it is uncertain whether the origin of the robot arm has been offset, the robot arm can be calibrated in real time in the working environment of the robot arm, and there is no need to re-teach the points after the calibration is completed. In this way, the time and cost required for calibration can be saved, and the working efficiency of the robot arm can be greatly improved. In addition, since only the camera at the end of the robot arm is used to shoot the calibration plate, when the robot arm is calibrated, there is no need to install any additional tools before calibration, thereby saving the calibration process and time, and indirectly improving the working efficiency of the robot arm.

Claims

1. A robotic arm calibration method based on image processing, for calibrating a robotic arm, comprising: Setting a visiting sequence of at least one calibration point on a calibration board; as well as According to the visiting sequence, the robot arm is driven to move a photographing device on a robot arm end of the robot arm to at least three shooting positions, and the photographing device is used to respectively shoot at least one image at the at least three shooting positions, and the at least one image has a center point. The step of respectively generating the at least one image at the at least three shooting positions by the photographing device includes: After recognizing that the shooting range of the photographic device covers at least one calibration point, the at least one image is generated; as well as Vector information of the at least one calibration point is calculated from the at least one image to adjust the robot arm.

2. The robot arm calibration method based on image processing as claimed in claim 1, wherein the vector information includes a plurality of first vectors and a plurality of second vectors, and the step of calculating the vector information of the at least one calibration point from the at least one image to adjust the robot arm comprises: Calculating a plurality of first vectors between the position of the at least one calibration point in the images and the center points of the images, and calculating a plurality of vector differences between the first vectors; calculating a plurality of second vectors between the at least one calibration point on the calibration plate; and A length offset vector of multiple links on the robot arm and a rotation offset vector of multiple reference points on the robot arm are calculated according to the vector differences, the second vectors and the axis angles, so as to calibrate the robot arm using the length offset vector and the rotation offset vector.

3. The robotic arm calibration method based on image processing as claimed in claim 2, wherein the step of calculating the first vectors between the position of the at least one calibration point in the images and the center points of the images comprises: identifying a one- and two-dimensional pixel difference between a pixel position of a calibration point in each of the images and a pixel position of a center point in each of the images; as well as The two-dimensional pixel differences identified from the images are respectively converted into first vectors in a real space.

4. The robotic arm calibration method based on image processing as claimed in claim 2, wherein the length offset vector may include the length offsets of each of the links, the rotation offset vector may include the axis angle offsets of each of the reference points, and the step of calculating the second vectors between the at least one calibration point comprises: The second vectors between the absolute coordinates of the at least one calibration point of the calibration plate in a real space are calculated.

5. The robot arm calibration method based on image processing as claimed in claim 2, wherein the step of calculating the length offset vectors of the links on the robot arm and the rotation offset vectors of the reference points on the robot arm according to the vector differences, the second vectors and the axis angles, and calibrating the robot arm using the length offset vector and the rotation offset vector comprises: Generate a prediction function of the robot arm using forward kinematics, and calculate a plurality of predicted positions of the camera when the camera moves to the shooting positions according to the axis angles and the prediction function, wherein the prediction function uses the lengths of the connecting rods and the axis angles of the reference points as variables; and The length offset vector and the rotation offset vector are calculated using the prediction function according to the vector differences, the second vectors and the predicted positions.

6. The robot arm calibration method based on image processing as claimed in claim 5, wherein the step of calculating the length offset vectors of the links on the robot arm and the rotation offset vectors of the reference points on the robot arm according to the vector differences, the second vectors and the axis angles, and calibrating the robot arm using the length offset vector and the rotation offset vector comprises: Calculating partial derivatives of the lengths and partial derivatives of the axis angles under the prediction function to generate a Jacobian matrix; Calculating the length offset vector and the rotation offset vector according to the vector differences, the second vectors, the predicted positions, the Jacobian matrix, and the axis angles; and The length offset vector and the rotation offset vector are used to correct the lengths of the connecting rods and the axis angles of the rotation axes on the robot arm. 7 . The robotic arm calibration method based on image processing as claimed in claim 1 , wherein the visiting sequence indicates: starting from the robotic arm to the at least one calibration point.

8. The image processing-based robotic arm calibration method as described in claim 1, wherein the visit sequence indicates: a position of a first calibration point is arbitrarily selected from the at least one calibration point as an initial position, and a second calibration point and a third calibration point among the at least one calibration point are visited in sequence according to a shortest path.

9. The image processing-based robotic arm calibration method as described in claim 1, wherein the visiting sequence indicates: arbitrarily selecting the position of a first calibration point from the at least one calibration point as the initial position, and returning to the initial position after visiting a second calibration point from the at least one calibration point, and then visiting a third calibration point from the at least one calibration point.

10. A robotic arm calibration system based on image processing, comprising: A calibration plate including at least one calibration point; a robotic arm; a photographing device, disposed at the end of a mechanical arm of the mechanical arm, for photographing the calibration plate; A controller is connected to the robot arm and is used to control the robot arm to perform the following operations: Setting a visiting order of the at least one calibration point; According to the visiting sequence, the robotic arm is driven to move the photographic device at the end of the robotic arm to at least three shooting positions, and the photographic device is used to respectively shoot at least one image at the at least three shooting positions, and the at least one image has a center point. The step of using the photographic device to respectively generate the at least one image at the at least three shooting positions comprises: After recognizing that the shooting range of the photographic device covers the at least one calibration point, the at least one image is generated; as well as Vector information of the at least one calibration point is calculated from the at least one image to adjust the robot arm.

11. The robotic arm calibration system based on image processing as claimed in claim 10, wherein the vector information comprises a plurality of first vectors and a plurality of second vectors, wherein calculating the vector information of the at least one calibration point from the at least one image to adjust the operation of the robotic arm comprises: Calculating a plurality of first vectors between the position of the at least one calibration point in the images and the center points of the images, and calculating a plurality of vector differences between the first vectors; calculating a plurality of second vectors between the at least one calibration point on the calibration plate; and A length offset vector of multiple links on the robot arm and a rotation offset vector of multiple reference points on the robot arm are calculated according to the vector differences, the second vectors and the axis angles, so as to calibrate the robot arm using the length offset vector and the rotation offset vector.

12. The robotic arm calibration system based on image processing as claimed in claim 11, wherein the controller further performs the following steps: identifying a one- and two-dimensional pixel difference between a pixel position of a calibration point in each of the images and a pixel position of a center point in each of the images; and The two-dimensional pixel differences identified from the images are respectively converted into first vectors in a real space.

13. The image processing based robot arm calibration system as claimed in claim 11, wherein the length offset vector may include the length offsets of each of the links, the rotation offset vector may include the axis angle offsets of each of the reference points, and the controller further performs the following steps: The second vectors between the absolute coordinates of the at least one calibration point of the calibration plate in a real space are calculated.

14. The robotic arm calibration system based on image processing as claimed in claim 11, wherein the controller further performs the following steps: Generate a prediction function of the robot arm using forward kinematics, and calculate a plurality of predicted positions of the camera when the camera moves to the shooting positions according to the axis angles and the prediction function, wherein the prediction function uses the lengths of the connecting rods and the axis angles of the reference points as variables; and The length offset vector and the rotation offset vector are calculated using the prediction function according to the vector differences, the second vectors and the predicted positions.

15. The robotic arm calibration system based on image processing as claimed in claim 14, wherein the controller further performs the following steps: Calculating partial derivatives of the lengths and partial derivatives of the axis angles under the prediction function to generate a Jacobian matrix; Calculating the length offset vector and the rotation offset vector according to the vector differences, the second vectors, the predicted positions, the Jacobian matrix, and the axis angles; and The length offset vector and the rotation offset vector are used to correct the lengths of the connecting rods and the axis angles of the rotation axes on the robot arm. 16 . The robotic arm calibration system based on image processing as claimed in claim 10 , wherein the visiting sequence indicates: visiting the at least one calibration point in sequence from near to far from an initial position of the robotic arm.

17. The image processing-based robotic arm calibration system as described in claim 10, wherein the visit sequence indicates: a position of a first calibration point is arbitrarily selected from the at least one calibration point as an initial position, and a second calibration point and a third calibration point among the at least one calibration point are visited in sequence according to a shortest path.

18. A robotic arm calibration system based on image processing as described in claim 10, wherein the visiting sequence indicates: a first calibration point is arbitrarily selected from the at least one calibration point as an initial position, and after visiting a second calibration point among the at least one calibration point, the robot returns to the initial position, and then visits a third calibration point among the at least one calibration point.

Citation Information

Patent Citations

  • Robot hand-eye calibration method and device and robot

    CN111791227A

  • Robot parameter calibration method, device and equipment and storage medium

    CN113459084A

  • Memory device and memory system for executing test

    KR1020220118266A