Method for estimating the position of an imaging device

Abstract

To provide a position estimation method of an imaging device which can reliably estimate a position of an imaging device relative to a TCP of a robot.SOLUTION: A position estimation method of an imaging device according to the present invention acquires a first image by imaging a calibration plate 114 by a camera 110, generates a coordinate axis of a calibration plate reference from the first image, rotates a joint of a robot arm 106 at a minute angle on three coordinate planes of an xy plane, a yz plane and an xz plane prescribed with the coordinate axis, acquires a second image by imaging the calibration plate again by the camera, obtains a movement amount of the camera relative to the calibration plate before and after rotation of the robot arm from the first image and the second image, calculates distances from the TCP to the camera in the three coordinate planes on the basis of the movement amount of the camera and the rotation angle, converts the distances acquired in the three coordinate planes into three-dimensional coordinates, and estimates a position of the camera relative to the TCP.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a method for estimating the position of an imaging device with respect to the TCP of a robot.

Background Art

[0002] As an example, in a production site such as a factory, a robot having a robot arm is used. The robot arm has an electric motor which is an actuator provided at each joint thereof. The electric motor is driven by a drive signal to realize the target operation of the robot arm. Further, an end effector such as a finger for gripping a workpiece which is a gripping target is attached to the tip of the robot arm.

[0003] In addition, an imaging device such as a camera may be attached to the robot. The robot can capture an image of the end effector and the workpiece by the imaging device, and based on this image, control various operations, thereby enabling the end effector to perform precise operations.

[0004] Therefore, in the robot, it is necessary to control various operations after grasping the position of the imaging device with respect to the tool center point (TCP) which is the coordinate center of the flange (robot tip). Hereinafter, estimating the position of the imaging device with respect to the TCP (robot tip coordinate center) of the robot is referred to as calibration.

[0005] Patent Document 1 describes a robot system including a robot having a movable part including a plurality of arms and a control device having a control part for controlling the operation of the robot. An imaging part is provided on the arm.

[0006] In this robot system, the control unit first drives the movable part to move the imaging unit to at least two locations (in three dimensions) without changing the orientation of the imaging unit. Next, the control unit calculates a conversion coefficient between the imaging unit's coordinate system and the robot coordinate system based on the coordinates of the imaging unit in the coordinate system at at least two locations and the coordinates of the robot coordinate system at at least two locations.

[0007] The control unit then calculates the offset of the imaging unit relative to the arm on which the imaging unit is installed, thereby determining the mounting position of the imaging unit relative to the movable part. In this way, the robot system of Patent Document 1 estimates the mounting position of the imaging device relative to the robot tip after calculating the conversion coefficient between the camera coordinate system and the robot coordinate system.

[0008] Patent Document 2 describes an automated hand-eye calibration for a robot motion vision system. In this automated hand-eye calibration, the position of a calibration plate is measured before and after the robot is rotated, and the position of the imaging device attached to the robot is estimated based on the position of this calibration plate. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Japanese Patent Publication No. 2018-094653 [Patent Document 2] Japanese Patent Publication No. 2020-116734 [Overview of the project] [Problems that the invention aims to solve]

[0010] In the technologies described in Patent Documents 1 and 2, the position of the imaging device relative to the TCP (Telescope Point) of the robot's tip coordinate center is estimated by acquiring three-dimensional positional information of the imaging device before and after the rotation of the robot's arm, and then solving the inverse of the 3x3 rotation matrix representing the rotation of the robot's arm. This process is complex. Furthermore, depending on the direction of movement of the imaging device, for example, if it rotates only around the X-axis, only around the Y-axis, or only around the Z-axis, the inverse matrix cannot be calculated, and the position of the imaging device cannot be estimated.

[0011] In view of these problems, the present invention aims to provide a method for estimating the position of an imaging device that can reliably estimate the position of the imaging device relative to the TCP of the robot tip coordinate center. [Means for solving the problem]

[0012] To solve the above problems, a typical configuration of the imaging device position estimation method according to the present invention is an imaging device position estimation method for estimating the position of an imaging device attached to a robot relative to the TCP of a robot, characterized in that the imaging device images a calibration plate to acquire a first image, generates a coordinate axis based on the calibration plate from the first image, rotates the joints of the robot by small angles in three coordinate planes defined by the coordinate axes (xy plane, yz plane, and xz plane), after rotating the robot joints, images the calibration plate again with the imaging device to acquire a second image, determines the movement of the imaging device relative to the calibration plate before and after the rotation of the robot joints from the first and second images, calculates the distance from the TCP to the imaging device in each of the three coordinate planes based on the movement of the imaging device and the angle of rotation, converts the distances obtained in the three coordinate planes into three-dimensional coordinates, and estimates the position of the imaging device relative to the TCP.

[0013] In the above configuration, a coordinate axis based on the calibration plate is generated from the first image obtained by imaging the calibration plate, and the robot's joints are rotated by small angles within the three coordinate planes defined by this axis: the xy plane, the yz plane, and the xz plane. Therefore, the rotations within the three coordinate planes can be represented by a 2x2 rotation matrix, and since the angle of rotation is small, a rotation matrix approximating trigonometric functions can be used.

[0014] Furthermore, the movement of the imaging device relative to the calibration plate before and after the rotation of the robot's joints can be determined from the first image acquired before the robot's joints are rotated and the second image acquired after the robot's joints are rotated. Therefore, the distance from the TCP to the imaging device in the three coordinate planes can be calculated based on the movement of the imaging device and the rotation matrix. That is, the distance in three-dimensional coordinates is calculated by decomposing it into three coordinate planes: the xy plane, the yz plane, and the zx plane. Here, since the rotation matrix is ​​an approximated 2x2 rotation matrix, the inverse matrix can be reliably solved, and thus the above distance can be reliably obtained in the three coordinate planes.

[0015] Furthermore, by, for example, averaging the above distances in the three coordinate planes, this distance can be converted into a three-dimensional coordinate system. In this way, the position of the imaging device relative to the robot's TCP can be reliably estimated. [Effects of the Invention]

[0016] According to the present invention, it is possible to provide a method for estimating the position of an imaging device that can reliably estimate the position of the imaging device relative to the TCP of a robot. [Brief explanation of the drawing]

[0017] [Figure 1] This figure illustrates a robot system to which the position estimation method for an imaging device according to an embodiment of the present invention is applied. [Figure 2]It is a block diagram showing the functions of the position estimation device and the robot control device in FIG. 1. [Figure 3] It is a flowchart showing the operation of a robot system to which the position estimation device in FIG. 1 is applied. [Figure 4] It is a diagram for explaining the process shown in the flowchart of FIG. 3.

Embodiments for Carrying Out the Invention

[0018] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Dimensions, materials, and other specific numerical values shown in such embodiments are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In the present specification and drawings, elements having substantially the same functions and configurations are denoted by the same reference numerals to omit redundant explanations, and elements not directly related to the present invention are not shown.

[0019] FIG. 1 is a diagram for explaining a robot system 100 to which the position estimation method of an imaging device in an embodiment of the present invention is applied. The robot system 100 is a system used in a production site such as a factory, and includes a robot 102 and a robot control device 104. The robot 102 has a robot arm 106. An end effector 108 is attached to the tip of the robot arm 106. The end effector 108 is, for example, a finger for gripping a predetermined workpiece as a gripping target.

[0020] The robot arm 106 is operationally controlled by the robot control device 104, and by driving an electric motor, which is an actuator provided at each joint thereof, the end effector 108 can be moved to a predetermined position. The end effector 108 can move close to the workpiece by the robot arm 106 and grip the workpiece. Also, the tool center point (TCP) shown in the figure is centered on the flange (robot tip).

[0021] The robot 102 is equipped with a camera 110 which is an imaging device. The robot 102 can capture images of the end effector 108 and the workpiece by the camera 110 to obtain images, and based on these images, control various operations, so that precise operations can be executed by the end effector 108. Therefore, in the robot system 100, it is necessary to control various operations after grasping the position (mounting position) of the camera 110 with respect to the TCP of the robot 102.

[0022] Therefore, the robot system 100 further includes a position estimation device 112, and adopts a configuration in which the robot arm 106 is rotated, and before and after the rotation, the calibration plate 114 is imaged by the camera 110. This will be specifically described below.

[0023] FIG. 2 is a block diagram showing the functions of the position estimation device 112 and the robot control device 104 in FIG. 1. FIG. 3 is a flowchart showing the operation of the robot system 100 to which the position estimation device 112 in FIG. 1 is applied. FIG. 4 is a diagram for explaining the process shown in the flowchart of FIG. 3.

[0024] In the robot system 100, first, the camera control unit 116 of the position estimation device 112 sets the initial value of the internal parameters of the camera 110 (step S100). The internal parameters are, for example, the number of pixels, cell size, and specification values of the focal length.

[0025] Next, the camera control unit 116 uses the camera 110 to image the calibration plate 114 at its initial position (step S102). In step S102, the camera control unit 116 outputs an operation command to the arm movement unit 118 of the robot control device 104 to control the robot arm 106, thereby moving the robot arm 106. This corrects the flange coordinate axis (robot tip coordinate axis A) shown in Figure 4(a) as defined by the robot control device 104, generates a coordinate axis B parallel to the coordinate axis based on the calibration plate 114, and sets the robot 102 to its initial position. The image in Figure 4(a) including coordinate axis B is the first image 120 at the initial position acquired in step S102.

[0026] In this way, step S102 generates a fixed coordinate axis B based on the calibration plate 114 from the first image 120, and this coordinate axis B defines three coordinate planes: the xy plane, the yz plane, and the xz plane (see Figure 4(b)).

[0027] Next, the camera control unit 116 rotates the robot arm 106 within the coordinate plane referenced by the calibration plate 114 using the arm movement unit 118, and then images the calibration plate 114 with the camera 110 (step S104).

[0028] Specifically, the TCP of the robot 102 is rotated (moved in an arc trajectory) by rotating the joint of the robot arm 106 by a small rotation angle around an axis parallel to the calibration plate 114, thereby imaging the calibration plate 114. Here, the image acquired in step S104 is the second image, which is the position after rotating by a small rotation angle from the initial position of the first image 120 (the small rotation position).

[0029] Next, the position estimation device 112 calculates the distance from TCP to camera 110 in the coordinate plane referenced by the calibration plate 114 (step S106). Specifically, the coordinate value calculation unit 122 of the position estimation device 112 determines the movement (travel distance) of camera 110 relative to the calibration plate 114 before and after the rotation of the joints of the robot arm 106 from the first image 120 and the second image. Furthermore, the coordinate transformation unit 124 calculates the distance from TCP to camera 110 in the coordinate plane based on the movement of camera 110 and the angle of rotation of the joints of the robot arm 106, using one of the following equations (1), (2), or (3).

number

[0030] Here, a1 and b1 in equation (1) are the movement of the camera 110 relative to the calibration plate 114 in the xy plane before and after the rotation of the joint of the robot arm 106, and are values ​​obtained by comparing the first image 120 and the second image. dx1 and dy1 are unknowns, and are the distances from TCP to camera 110 in the xy plane. θ1 is a small rotation angle in the xy plane, for example, about 1° to 5°.

[0031] As shown in equation (1), rotation in the xy-plane can be represented by a 2x2 rotation matrix. Furthermore, since the rotation angle is small, a rotation matrix can be used in which cosθ1 is approximated as "1" and sinθ1 as "θ1". The inverse of this approximated 2x2 rotation matrix can be reliably solved. This allows us to reliably obtain the distances dx1 and dy1 from TCP to camera 110 in the xy-plane.

[0032] Next, in the position estimation device 112, if the distance from TCP to camera 110 has not been calculated by the coordinate value calculation unit 122 and the coordinate transformation unit 124 in the three coordinate planes based on the calibration plate 114, namely the xy plane, yz plane, and zx plane (step S108, No), the process returns to step S104 and repeats step S106 to calculate the distance from TCP to camera 110 not only in the xy plane but also in the yz plane and zx plane using equations (2) and (3) above.

[0033] Furthermore, b2 and c1 in equation (2) are the movement of the camera 110 relative to the calibration plate 114 in the yz plane before and after the rotation of the joint of the robot arm 106, and are values ​​obtained by comparing the first image 120 and the second image. dy2 and dz1 are unknown variables, and are the distance from TCP to camera 110 in the yz plane. θ2 is a small rotation angle in the yz plane, for example, about 1° to 5°.

[0034] As shown in equation (2), rotation in the yz plane can be represented by a 2x2 rotation matrix. Furthermore, since the rotation angle is small, a rotation matrix can be used in which cosθ2 is approximated as "1" and sinθ2 as "θ2". The inverse of this approximated 2x2 rotation matrix can be reliably solved. This allows us to reliably obtain the distance dy2 and dz1 from TCP to camera 110 in the yz plane.

[0035] Furthermore, c2 and a2 in equation (3) are the movement of the camera 110 relative to the calibration plate 114 in the zx plane before and after the rotation of the joint of the robot arm 106, and are values ​​obtained by comparing the first image 120 and the second image. dz2 and dx2 are unknown variables and are the distance from TCP to camera 110 in the zx plane. θ3 is a small rotation angle in the zx plane, for example, about 1° to 5°.

[0036] As shown in equation (3), rotation in the zx plane can be represented by a 2x2 rotation matrix. Furthermore, since the rotation angle is small, a rotation matrix can be used in which cosθ3 is approximated as "1" and sinθ3 as "θ3". The inverse of this approximated 2x2 rotation matrix can be reliably solved. This allows us to reliably obtain the distances dz2 and dx2 from TCP to camera 110 in the zx plane.

[0037] In other words, the coordinate value calculation unit 122 and the coordinate transformation unit 124 can calculate the distance in three-dimensional coordinates by decomposing it into three coordinate planes: the xy plane, the yz plane, and the zx plane. Therefore, even if the camera 110 rotates only around the X axis, only around the Y axis, or only around the Z axis, the inverse of the rotation matrix can be calculated, and thus the distance from TCP to camera 110 in the xy, yz, and zx planes can be reliably obtained.

[0038] Next, the coordinate value calculation unit 122 and the coordinate transformation unit 124 calculate the distance from TCP to camera 110 in the three coordinate planes based on the calibration plate 114, namely the xy plane, yz plane, and zx plane (step S108, Yes), and then the process of step S110 is performed.

[0039] In step S110, the plate position calculation unit 128 of the position estimation device 112 estimates the position of the camera 110 relative to the TCP by converting the distances (dx1, dy1), (dy2, dz1), and (dz2, dx2) from the TCP to the camera 110, which were obtained in the three coordinate planes, the xy plane, yz plane, and zx plane, into three-dimensional coordinates.

[0040] Specifically, the plate position calculation unit 128 can estimate the position (x,y,z) of the camera 110 in three-dimensional coordinates relative to the TCP as ((dx1+dx2) / 2,(dy1+dy2) / 2,(dz1+dz2) / 2) by taking, for example, the average of the two-dimensional information (dx1,dy1), (dy2,dz1), and (dz2,dx2) in the xy, yz, and zx planes shown in Figure 4(b).

[0041] Next, the position estimation device 112 calculates the calibration position (step S112) based on the position of the camera 110 in three-dimensional coordinates relative to the TCP estimated by the calibration unit 130 in step S110, and then terminates the process.

[0042] In the robot system 100 to which the position estimation device 112 is applied, rotation in three-dimensional space can be decomposed into rotation in three coordinate planes: the xy plane, the yz plane, and the zx plane. Using an approximate 2x2 rotation matrix, the distance from TCP to camera 110 in each of the three coordinate planes can be calculated. Then, by taking the average of the distances from TCP to camera 110 obtained in the three coordinate planes, this distance can be converted into three-dimensional coordinates to reliably estimate the position of camera 110 relative to TCP.

[0043] Preferred embodiments of the present invention have been described above with reference to the attached drawings, but it goes without saying that the present invention is not limited to these examples. It will be obvious to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these will naturally also fall within the technical scope of the present invention. [Industrial applicability]

[0044] This invention can be used as a method for estimating the position of an imaging device relative to a robot's TCP (Telescope Component). [Explanation of symbols]

[0045] 100...Robot system, 102...Robot, 104...Robot control device, 106...Robot arm, 108...End effector, 110...Camera, 112...Position estimation device, 114...Calibration plate, 116...Camera control unit, 118...Arm movement unit, 120...First image, 122...Coordinate value calculation unit, 124...Coordinate transformation unit, 126...Estimated value calculation unit, 128...Plate position calculation unit, 130...Calibration unit

Claims

[Claim 1] A method for estimating the position of an imaging device attached to a robot with respect to the TCP of the robot, The imaging device captures an image of the calibration plate to obtain a first image. A coordinate axis is generated from the first image with the calibration plate as the reference, and the joints of the robot are rotated by a small angle θ in the three coordinate planes defined by the coordinate axis: the xy plane, the yz plane, and the xz plane. The rotations in the three coordinate planes are represented by 2x2 rotation matrices in which cosθ is approximated as 1 and sinθ as θ. After rotating the joint of the robot, the imaging device takes another image of the calibration plate to acquire a second image. From the first and second images, the movement of the imaging device relative to the calibration plate before and after the rotation of the robot's joint is determined, and based on the movement of the imaging device and the 2x2 rotation matrix, the inverse of the 2x2 rotation matrix is ​​solved in each of the three coordinate planes to calculate the distance from the TCP to the imaging device in each of the three coordinate planes. A method for estimating the position of an imaging device, characterized by converting the distances obtained in the three coordinate planes into three-dimensional coordinates and estimating the position of the imaging device relative to the TCP.

Images