Robot control device, robot control method, and robot

Abstract

The present invention provides a robot control device that drives a robotic arm to capture images of the same object using a camera at a plurality of image capture positions, and estimates the relative position of the camera with respect to a mechanical interface by means of a combination of: a first position of the object in a robot coordinate system calculated on the basis of the angle of each arm detected by a detection unit when an image of the object is captured at a first image capture position among the plurality of image capture positions and the position of the object in an image coordinate system recognized from a captured image of the object at the first image capture position; and a second position of the object in the robot coordinate system calculated on the basis of the angle of each arm detected by the detection unit when an image of the object is captured at a second image capture position among the plurality of image capture positions and the position of the object in an image coordinate system recognized from a captured image of the object at the second image capture position.

Description

Robot control device, robot control method, and robot

[0001] This specification discloses a robot control device, a robot control method, and a robot.

[0002] A conventional vertical articulated robot includes first to sixth arms, a tool (hand) attached to the distal end surface of the sixth arm (distal arm), and a mobile camera attached to the sixth arm offset from the central axis of the sixth arm so as to be substantially parallel to the sixth arm's axis. The robot captures an image of a calibration board (calibration board) with a plurality of markers attached at a constant pitch using the mobile camera, and calculates correction parameters for converting the coordinates of an object in the mobile camera's image coordinate system to the coordinates of the object in the robot's coordinate system (see, for example, Patent Document 1). In this robot, the robot arm is driven so that the marker is positioned at the center O30 (image capture reference point O3) of the mobile camera image (first state). Next, the robot moves an axis coordinate O6 (the point where the distal end surface of the sixth arm intersects with the central axis) by a rotation angle θ1, with a line segment passing through the center O30 as the rotation axis (third state). The robot then calculates the coordinate of the imaging reference point O3 in the robot coordinate system based on the rotation angle θ1, the coordinate of the axis coordinate O6 in the robot coordinate system in the first state, and the coordinate of the axis coordinate O6 in the robot coordinate system in the third state, and calculates the offset components Δx and Δy of the imaging reference point O3 relative to the axis coordinate O6 from the calculated coordinate of the imaging reference point O3 and the coordinate of the axis coordinate O6 in the robot coordinate system in either the first state or the third state.

[0003] Japanese Patent Application Laid-Open No. 2018-1332

[0004] It has been described that the above-mentioned robot can accurately calculate the offset of a tool attached to the robot arm by calibrating a fixed camera fixed to the working area and a mobile camera attached to the arm of the vertically articulated robot. However, this configuration requires a calibration member and a hand to hold it. Furthermore, it is also necessary to calibrate the fixed camera in order to calibrate the mobile camera. Furthermore, when calibrating the cameras, it is desirable to perform the calibration with high accuracy and to minimize the operator's effort as much as possible.

[0005] A primary object of the present disclosure is to perform camera calibration simply and with good accuracy in a robot equipped with a horizontal articulated arm.

[0006] The present disclosure has adopted the following means to achieve the above-mentioned main object.

[0007] A robot control device according to the present disclosure comprises: a robot arm having a first arm capable of horizontal rotation, and a second arm capable of horizontal rotation relative to the first arm and provided with a mechanical interface; a camera attached to the second arm; a memory unit that stores a relative position of the camera relative to the mechanical interface of the second arm; and a detection unit that detects angles of each arm of the robot arm, wherein the robot control device captures an image of a workpiece with the camera, and recognizes the position of the workpiece in a robot coordinate system based on the angles of each arm detected by the detection unit when the image of the workpiece is captured, the position of the workpiece in an image coordinate system recognized from the captured image of the workpiece, and the relative position of the camera relative to the mechanical interface stored in the memory unit, The gist is that the robot arm is driven to capture images of the same object with the camera at a plurality of imaging positions, and the relative position of the camera with respect to the mechanical interface is estimated by combining a first position of the object in a robot coordinate system calculated based on the angle of each arm detected by the detection unit when the object is captured at a first imaging position of the plurality of imaging positions and the position of the object in an image coordinate system recognized from the captured image of the object at the first imaging position, and a second position of the object in the robot coordinate system calculated based on the angle of each arm detected by the detection unit when the object is captured at a second imaging position of the plurality of imaging positions and the position of the object in an image coordinate system recognized from the captured image of the object at the second imaging position.

[0008] In the robot control device disclosed herein, in a robot equipped with a horizontally articulated arm, when an operator places an object, the robot arm is driven to capture images of the same object with cameras at multiple imaging positions, and the relative position of the cameras with respect to the mechanical interface can be automatically estimated. As a result, in a robot equipped with a horizontally articulated arm, camera calibration can be performed easily and with high accuracy.

[0009] 7A is a schematic configuration diagram of a robot system; FIG. 7B is a block diagram showing the electrical connection relationship of the robot system; FIG. 7C is an explanatory diagram showing the robot coordinate system and image coordinate system of the robot system; FIG. 7D is a flowchart showing an example of calibration processing; FIG. 7E is an explanatory diagram showing the installation of a calibration plate; FIG. 7F is a configuration diagram of a calibration plate; FIG. 7A is an explanatory diagram showing the imaging of the calibration plate by a hand camera at imaging point a, and FIG. 7B is an explanatory diagram showing the imaging of the calibration plate by a hand camera at imaging point b; FIG. 8A is an explanatory diagram showing the image of the central dot relative to the field of view of the hand camera at imaging point a, and FIG. 8B is an explanatory diagram showing the image of the central dot relative to the field of view of the hand camera at imaging point b; FIG. 8F is an explanatory diagram showing the relationship between the angle of the robot arm and the position of the MI center (Xmi, Ymi) in the robot coordinate system; and FIG. 8G is an explanatory diagram showing the relationship between the position of the MI center (Xmi, Ymi) in the robot coordinate system and the position of the camera center (Xc, Yc) in the robot coordinate system. 1 is an explanatory diagram showing how the relative position (Lcx, Lcy) of the camera center with respect to the MI center is converted into the relative position (Lcx', Lcy') of the camera center with respect to the MI center in the robot coordinate system. FIG. 1 is an explanatory diagram showing the relationship between the position (Xc, Yc) of the camera center in the robot coordinate system and the position (Xv, Yv) of the center dot Dc in the image coordinate system. FIG. 1 is an explanatory diagram showing how the position (Xv, Yv) of the center dot Dc in the image coordinate system is converted into the relative position (Xv', Yv') of the center dot Dc with respect to the camera center in the robot coordinate system. FIG. 2 is an explanatory diagram showing the position (Xra, Yra) of the center dot Dc in the robot coordinate system calculated by imaging the center dot at imaging point a, and the position (Xrb, Yrb) of the center dot Dc in the robot coordinate system calculated by imaging the center dot at imaging point b.

[0010] Next, embodiments of the present disclosure will be described with reference to the drawings.

[0011] Fig. 1 is a schematic diagram of a robot system 1. Fig. 2 is a block diagram showing the electrical connections of the robot system 1.

[0012] The robot system 1 performs a predetermined operation on a workpiece W. The predetermined operation may include, for example, picking up a workpiece W (component) supplied by a workpiece supply unit 80, transporting the workpiece W to a mounting target S (substrate), and mounting the workpiece W.

[0013] 1 and 2, the robot system 1 includes a robot 10, a work camera 70, and a control device 90 that controls the robot 10. The robot 10 includes a base 11, a robot arm 20 supported by the base 11, and a hand camera 60 attached to the robot arm 20.

[0014] The base 11 is fixed to the workbench 2 and supports the base end of the robot arm 20. The robot arm 20 is a two-link horizontal articulated robot arm, and includes a first arm 21 as a first link, a first arm driver 30 that drives the first arm 21, a second arm 22 as a second link, a second arm driver 40 that drives the second arm 22, a shaft 23, and a shaft driver 50 that drives the shaft 23.

[0015] The first arm 21 has a base end connected to the base 11 via a first joint axis J1 and is configured to be rotatable (horizontally pivotable) within a horizontal plane relative to the base 11 by rotation of the first joint axis J1. The second arm 22 has a base end connected to a tip end of the first arm 21 via a second joint axis J2 and is configured to be rotatable (horizontally pivotable) within a horizontal plane relative to the first arm 21 by rotation of the second joint axis J2. The shaft 23 is connected to the tip end of the second arm 22 via a third joint axis J3 and is configured to be rotatable around the axis of the third joint axis J3 relative to the second arm 22 and to be movable up and down along the axial direction of the third joint axis J3. A mechanical interface MI is provided at the tip of the shaft 23, and a work tool is attached to the mechanical interface MI. In this embodiment, the work tool is a workpiece holder 24 for picking up and holding a workpiece W. Examples of the workpiece holding portion 24 include a suction nozzle that adsorbs the workpiece W by negative pressure, a mechanical chuck that grips the workpiece W with a pair of claws, and an electromagnetic chuck that adsorbs the workpiece W by an electromagnet.

[0016] 2, the first arm driver 30 includes a motor 32 and an encoder 34. The rotation shaft of the motor 32 is connected to the first joint shaft J1 via a reducer (not shown). The first arm driver 30 rotates the first arm 21 around the first joint shaft J1 as a fulcrum by torque transmitted from the motor 32 to the first joint shaft J1 via the reducer. The encoder 34 is attached to the rotation shaft of the motor 32 and is configured as a rotary encoder that detects the amount of rotational displacement of the motor 32.

[0017] Similar to the first arm driver 30, the second arm driver 40 includes a motor 42 and an encoder 44. The rotation shaft of the motor 42 is connected to the second joint shaft J2 via a reducer (not shown). The second arm driver 40 rotates the second arm 22 around the second joint shaft J2 as a fulcrum by torque transmitted from the motor 42 to the second joint shaft J2 via the reducer. The encoder 44 is attached to the rotation shaft of the motor 42 and is configured as a rotary encoder that detects the amount of rotational displacement of the motor 42.

[0018] As described above, the robot arm 20 is configured as a horizontally jointed robot arm. Therefore, the robot 10 has, as posture modes of the robot arm 20, a right-arm posture mode in which the robot arm 20 operates in a right-arm posture, and a left-arm posture mode in which the robot arm 20 operates in a left-arm posture.

[0019] As shown in FIG. 2 , the shaft driving unit 50 includes motors 52 a, 52 b and encoders 54 a, 54 b. The shaft 23, although not shown, is a spline shaft with spline grooves extending in the axial direction, and a spline nut is spline-fitted to the shaft 23. The rotating shaft of the motor 52 a is connected to the spline nut, and the shaft 23 rotates as the spline nut rotates due to power from the motor 52 a. The shaft driving unit 50 also includes an elevator shaft extending parallel to the shaft 23 and a support member, not shown, that is fixed to the elevator shaft and rotatably supports the shaft 23. The elevator shaft is a ball screw shaft with a helical screw groove, and a ball screw nut is threadedly engaged with the elevator shaft. The rotating shaft of the motor 52 b is connected to the ball screw nut, and the elevator shaft moves up and down as the ball screw nut rotates due to power from the motor 52 b. The shaft 23 moves up and down together with the lifting shaft as the lifting shaft moves up and down. The encoder 54a is configured as a rotary encoder that detects the amount of rotational displacement of the shaft 23. The encoder 54b is configured as a linear encoder that detects the lifting position of the shaft 23.

[0020] The hand camera 60 is attached to the tip of the second arm 22 via a bracket 61. For example, the hand camera 60 is attached so that the center of the hand camera 60 (camera center) is located on an extension line of the second arm 22 that passes from the second joint axis J2 to the center of the mechanical interface MI (MI center). The hand camera 60 captures an image of the workpiece W supplied by the workpiece supply unit 80 and outputs the captured image to the control device 90. The control device 90 processes the captured image to recognize the position of the workpiece W.

[0021] In this embodiment, the hand camera 60 is a camera whose lens can be replaced with one of different magnifications. The hand camera 60 is attached to the horizontally articulated robot arm 20, and because the robot arm 20 cannot move up and down, the field of view of the hand camera 60 cannot be changed by changing the distance (working distance) between the hand camera 60 and the workpiece W. However, by changing the magnification of the lens, the field of view of the hand camera 60 can be changed to match the workpiece W to be imaged.

[0022] The work camera 70 is installed on the work table 2 between the work supply unit 80 and the mounting target (substrate) S. The work camera 70 captures an image of the work W held by the work holding unit 24 of the robot 10 from below, and outputs the captured image to the control device 90. The control device 90 processes the captured image to determine whether the work W is being held properly by the work holding unit 24.

[0023] As shown in FIG. 2 , the control device 90 includes a CPU 91, a ROM 92 for storing a processing program, a RAM 93 as a work memory, a storage device 94 such as an HDD or SSD, and an input / output interface (not shown). The storage device 94 stores information necessary for the work, such as information about the workpiece W and information about the work tool (workpiece holder 24). The storage device 94 also stores the resolution (length per pixel of the image) of the image captured by the hand camera 60, the relative position (Lcx, Lcy) of the center of the hand camera 60 (camera center) relative to the MI center, and the posture mode of the robot arm 20 currently selected by the operator (right arm posture mode, left arm posture mode). Position signals from the encoders 34, 44, 54a, and 54b, image signals from the hand camera 60, and image signals from the work camera 70 are input to the control device 90 via the input / output interface. The control device 90 outputs control signals to the motors 32, 42, 52a, 52b, a control signal to the hand camera 60, a control signal to the work camera 70, and the like via an input / output interface.

[0024] Next, we will explain the operation of the robot system 1 configured in this manner. In particular, we will explain the operation of picking up the workpiece W at the recognized position by the workpiece holder 24 after capturing an image of the workpiece W with the hand camera 60 and performing a recognition process to recognize the position of the workpiece W.

[0025] In the recognition process, the CPU 91 of the control device 90 first controls the robot arm 20 (first arm drive unit 30 and second arm drive unit 40) so that the hand camera 60 moves above the work supply unit 80, and the hand camera 60 captures an image of the workpiece W on the work supply unit 80. Next, the CPU 91 performs image processing on the image captured by the hand camera 60 to recognize the position (Xv, Yv) [pixel] of the workpiece W in pixel units in the image coordinate system. Next, the CPU 91 converts the position (Xv, Yv) [pixel] of the workpiece W in pixel units to the position (Xv, Yv) [μm] of the workpiece W in length units (e.g., μm units). The position (Xv, Yv) [μm] of the workpiece W in length units is calculated by multiplying the position (Xv, Yv) [pixel] of the workpiece W in pixel units by the length per pixel of the image (image resolution). The CPU 91 then converts the position (Xv, Yv) of the workpiece W in length units in the image coordinate system into the position (Xr, Yr) of the workpiece W in the robot coordinate system. As shown in FIG. 3 , the image coordinate system is a two-dimensional Cartesian coordinate system defined by an Xv-axis and a Yv-axis that are parallel to the image plane and perpendicular to each other, with the origin at a reference point of the image (e.g., the center of the image). The robot coordinate system is a three-dimensional Cartesian coordinate system defined by an Xr-axis and a Yr-axis that are parallel to the horizontal direction and perpendicular to each other, and a Zr-axis that is coaxial with the first joint axis J1, with the origin at a point on the base 11 that is coaxial with the first joint axis J1.

[0026] The conversion of the position of the workpiece W from the image coordinate system to the robot coordinate system is performed as follows. That is, the CPU 91 calculates the rotation angle α (= θ1 + θ2 - 90°) of the image coordinate system (Xv axis, Yv axis) relative to the robot coordinate system (Xr axis, Yr axis) based on the angle θ1 of the first joint axis J1 and the angle θ2 of the second joint axis J2 from the encoders 34, 44 when the workpiece W is imaged by the hand camera 60. Next, the CPU 91 converts the position (Xv, Yv) of the workpiece W in the image coordinate system into a relative position (ΔXr, ΔYr) that is the position of the workpiece W relative to the center of the hand camera 60 (camera center, i.e., the origin of the image coordinate system) in the robot coordinate system using a rotation matrix that specifies the angle (-α). Next, the CPU 91 calculates the position of the MI center (Xmi, Ymi), which is the position of the MI center in the robot coordinate system, from the angle θ1 of the first joint axis J1 and the angle θ2 of the second joint axis J2 when the workpiece W is imaged by the hand camera 60, using the following equations (1) and (2) based on forward kinematics.

[0027] Xmi=L1・cos(θ1)+L2・cos(θ1+θ2)…(1) Ymi=L1・sin(θ1)+L2・sin(θ1+θ2)…(2)

[0028] The CPU 91 then calculates the position (Xc, Yc) of the camera center in the robot coordinate system from the position (Xmi, Ymi) of the MI center in the robot coordinate system using the relative position (Lcx, Lcy) of the camera center with respect to the MI center. Here, Lcx indicates the distance from the MI center to the camera center in the horizontal plane in the extension direction of the second arm 22 (see FIG. 10 ). Lxy indicates the distance from the MI center to the camera center in the orthogonal direction perpendicular to the extension direction of the second arm 22 (see FIG. 10 ). The calculation of the position (Xc, Yc) of the camera center is performed by adding the relative position of the camera center with respect to the MI center to the position (Xmi, Ymi) of the MI center in the robot coordinate system. More specifically, this is performed as follows: The CPU 91 calculates the angle β (= θ1 + θ2) of the second arm 22 with respect to the Xr axis of the robot coordinate system by adding the angle θ1 of the first joint axis J1 and the angle θ2 of the second joint axis J2. Next, the CPU 91 converts the relative position (Lcx, Lcy) of the camera center with respect to the MI center into the relative position (Lcx', Lcy') of the camera center with respect to the MI center in the robot coordinate system using a rotation matrix R(β) with the specified angle β (see FIG. 11). The CPU 91 then calculates the position (Xc, Yc) of the camera center in the robot coordinate system by adding the relative position (Lcx', Lcy') of the camera center with respect to the MI center in the robot coordinate system to the position (Xmi, Ymi) of the MI center in the robot coordinate system. After calculating the position (Xc, Yc) of the camera center in the robot coordinate system, the CPU 91 recognizes the position (Xr, Yr) of the workpiece W with respect to the camera center in the robot coordinate system by adding the position (Xc, Yc) of the camera center in the robot coordinate system to the position (Xc, Yc) of the camera center in the robot coordinate system.

[0029] When the CPU 91 recognizes the position (Xr, Yr) of the workpiece W in the robot coordinate system through this recognition process, it sets a target position for the workpiece holding unit 24 based on the position (Xr, Yr) of the workpiece W, and controls the robot arm 20 (first arm driving unit 30 and second arm driving unit 40) and the workpiece holding unit 24 so that the workpiece holding unit 24 moves to the set target position and holds the workpiece W.

[0030] Here, the relative position (Lcx, Lcy) of the hand camera 60 with respect to the MI center generally varies from the design value due to mounting errors of the hand camera 60 and dimensional errors of the bracket 61. That is, the relative position (Lcx, Lcy) of the hand camera 60 with respect to the MI center includes the design value (Lcx_0, Lcy_0) of the position related to the mounting of the hand camera 60 with respect to the second arm 22 and a variation (Lcx_e, Lcy_e) due to the variation from the design value, and can be expressed by the following equations (3) and (4).

[0031] Lcx=Lcx_0+Lcx_e …(3) Lcy=Lcy_0+Lcy_e …(4)

[0032] For this reason, in this embodiment, the robot system 1 performs calibration by capturing images of the calibration plate 100, which has been placed on the workbench 2 by an operator, at multiple imaging points a and b using the hand camera 60, calculating the variation (Lcx_e, Lcy_e) of the hand camera 60 from each captured image, and estimating the relative position (Lcx, Lcy) of the hand camera 60 with respect to the MI center. FIG. 4 is a flowchart showing an example of the calibration process executed by the CPU 91 of the control device 90. This process is executed when the operator instructs execution of the calibration after placing the calibration plate 100 on the work supply unit 80 (see FIG. 5). By placing the calibration plate 100 on the work supply unit 80, the CPU 91 can capture an image of the calibration plate 100 under the same conditions as when capturing an image of the workpiece W when picking up the workpiece W, thereby enabling the calibration process to be properly executed.

[0033] As shown in FIG. 6 , the calibration plate 100 has a plurality of dots D formed on its surface in a grid pattern at a constant (known) distance p. Of the plurality of dots D, the dot (central dot Dc) formed at the center of the calibration plate 100 is used for the calibration process. The peripheral dots D are used to measure the image resolution, which converts the position of the workpiece W in pixel units in the image coordinate system based on the image captured by the hand camera 60 described above into the position of the workpiece W in length units. The image resolution is measured as follows: The CPU 91 captures an image of the calibration plate 100 using the hand camera 60 and processes the captured image to recognize the plurality of dots D. Next, the CPU 91 measures the distance in pixels [pixels] between two adjacent dots D among the recognized plurality of dots D. The CPU 91 then calculates the image resolution by dividing the known distance between the dots in length units [mm] by the measured distance between the dots in pixel units [pixels].

[0034] As described above, the robot 10 includes a horizontally articulated robot arm 20 and has a right-arm posture mode and a left-arm posture mode. In this embodiment, the CPU 91 captures images of the calibration plate 100 at imaging points a and b in the right-arm posture mode when the robot 10 is set to the right-arm posture mode, and in the left-arm posture mode when the robot 10 is set to the left-arm posture mode. If the optical axis of the hand camera 60 is tilted relative to the vertical direction (the Zr axis in the robot coordinate system), the optical axis of the hand camera 60 will not coincide when capturing an image of an object on the worktable 2 in the right-arm posture mode and when capturing an image in the left-arm posture mode. Therefore, if the calibration plate 100 is captured in a posture mode different from the posture mode when the hand camera 60 captures an image of the workpiece W, a discrepancy will occur between the positions recognized based on the captured images. Therefore, by having the CPU 91 image the calibration plate 100 in the same posture mode as when the workpiece W is imaged with the hand camera 60, the calibration process can be performed with good accuracy even if the optical axis of the hand camera 60 is tilted with respect to the Zr axis.

[0035] The calibration process in FIG. 4 will now be described in detail.

[0036] When the calibration process is executed, the CPU 91 of the control device 90 first sets a target position of the MI center based on design values ​​(Lcx_0, Lcy_0) for the relative position of the hand camera 60 with respect to the MI center so that the hand camera 60 is located at imaging point a, and controls the robot arm 20 (the first arm driver 30 and the second arm driver 40) so that the MI center moves to the target position (S100). Next, the CPU 91 images the calibration plate 100 with the hand camera 60 at imaging point a (S102). FIG. 7A shows how the calibration plate 100 is imaged with the hand camera 60 at imaging point a, and FIG. 8A shows an example of an image of dot D, including a center dot Dc, relative to the field of view of the hand camera 60 at imaging point a. In FIG. 8A, the dashed-dotted line indicates the field of view of the hand camera 60 when the relative position (Lcx, Lcy) of the hand camera 60 with respect to the MI center is as designed, and the solid line indicates the actual field of view of the hand camera 60. Next, the CPU 91 acquires from the encoders 34, 44 the angle θ1a of the first joint axis J1 and the angle θ2a of the second joint axis J2 of the robot arm 20 when an image of the calibration plate 100 is captured at the imaging point a (S104). The CPU 91 then calculates the position of the MI center (Xmia, Ymia) in the robot coordinate system at the imaging point a using forward kinematics based on the acquired angle θ1a of the first joint axis J1 and the angle θ2a of the second joint axis J2 (S106). The relationship between the angle of the robot arm 20 (angles θ1, θ2 of the first joint axis J1 and the second joint axis J2) and the position of the MI center (Xmi, Ymi) in the robot coordinate system is shown in FIG. 9 . The relationship between the position of the MI center (Xmi, Ymi) in the robot coordinate system and the position of the camera center (Xc, Yc) in the robot coordinate system is also shown in FIG. 10 . Next, the CPU 91 processes the image of the calibration plate 100 captured in S102 to recognize the position (Xva, Yva) of the center dot Dc in the image coordinate system (S108).The process of S108 is performed by recognizing the position (Xv, Yv) [pixel] of the center dot Dc in the image coordinate system in pixel units based on the image captured by the hand camera 60, and then converting the position (Xva, Yva) [pixel] of the center dot Dc in pixel units to the position (Xva, Yva) [μm] of the center dot Dc in length units using the image resolution. Then, the CPU 91 converts the position (Xva, Yva) of the center dot Dc in the image coordinate system to the relative position (Xva', Yva') of the center dot Dc with respect to the camera center in the robot coordinate system based on the angle θ1a of the first joint axis J1 and the angle θ2a of the second joint axis J2 acquired in S104 (S110). The relationship between the position (Xc, Yc) of the camera center in the robot coordinate system and the position (Xv, Yv) of the center dot Dc in the image coordinate system is shown in FIG. 12 . 13 shows how the position (Xv, Yv) of the center dot Dc in the image coordinate system is converted into the relative position (Xv', Yv') of the center dot Dc with respect to the camera center in the robot coordinate system. As shown in the figure, the image coordinate system is rotated by an angle α (= θ1 + θ2 - 90°) with respect to the robot coordinate system. Therefore, the conversion from the position (Xva, Yva) of the center dot Dc in the image coordinate system to the relative position (Xra', Yra') of the center dot Dc with respect to the camera center in the robot coordinate system can be performed by applying a rotation matrix R(-α) specifying an angle (-α) to the position (Xva, Yva) of the center dot Dc in the image coordinate system.

[0037] Next, the CPU 91 sets a target position for the MI center based on the design values ​​(Lcx_0, Lcy_0) for the relative position of the hand camera 60 with respect to the MI center so that the hand camera 60 is located at imaging point b, and controls the robot arm 20 (the first arm driver 30 and the second arm driver 40) so that the MI center moves to the target position (S112). Next, the CPU 91 images the calibration plate 100 with the hand camera 60 at imaging point b (S114). FIG. 7B shows how the calibration plate 100 is imaged with the hand camera 60 at imaging point b, and FIG. 8B shows an example of an image of dot D, including a center dot Dc, relative to the field of view of the hand camera 60 at imaging point b. In FIG. 8B, the dashed-dotted line indicates the field of view of the hand camera 60 when the relative position of the hand camera 60 with respect to the MI center is as designed, and the solid line indicates the actual field of view of the hand camera 60. As shown in FIGS. 8A and 8B , imaging points a and b are determined so that the images of the center dot Dc in the field of view of the hand camera 60 are diagonally positioned relative to each other. This corresponds to maximizing the difference between the angle of the robot arm 20 when imaging the center dot Dc at imaging point a and the angle of the robot arm 20 when imaging the same center dot Dc at imaging point b. The reason for this determination will be described later. In this embodiment, the hand camera 60 is interchangeable with lenses of different magnifications, and the field of view of the hand camera 60 is changed according to the lens magnification. The CPU 91 sets imaging points a and b by kinematic calculation based on the setting of the field of view of the hand camera 60 according to the lens magnification so that the images of the center dot Dc in the field of view of the hand camera 60 are diagonally positioned relative to each other. The field of view of the hand camera 60 according to the lens magnification is set by, for example, inputting the magnification of the replaced lens into the control device 90 in advance when the operator replaces the lens of the hand camera 60.

[0038] Next, the CPU 91 acquires from the encoders 34, 44 the angle θ1b of the first joint axis J1 and the angle θ2b of the second joint axis J2 of the robot arm 20 when an image of the calibration plate 100 is captured at the imaging point b (S116). Then, the CPU 91 calculates the position (Xmib, Ymib) of the MI center in the robot coordinate system at the imaging point b using forward kinematics based on the acquired angle θ1b of the first joint axis J1 and the angle θ2b of the second joint axis J2 (S118). Next, similar to S108, the CPU 91 processes the captured image of the calibration plate 100 captured in S114 to recognize the position (Xvb, Yvb) of the center dot Dc in the image coordinate system (S120). Then, similar to S110, the CPU 91 converts the position (Xvb, Yvb) of the center dot Dc in the image coordinate system into the relative position (Xrb', Yrb') of the center dot Dc with respect to the camera center in the robot coordinate system based on the angle θ1b of the first joint axis J1 and the angle θ2b of the second joint axis J2 obtained in S116 (S122).

[0039] After executing the processes of S100 to S122 in this way, the CPU 91 determines whether these processes have been executed a predetermined number of times (e.g., 10 times) (S124). If the CPU 91 determines that these processes have not been executed the predetermined number of times, it returns to S100 and repeats the processes of S100 to S122. If the CPU 91 determines that these processes have been executed the predetermined number of times, it proceeds to S126. As described above, in this embodiment, the CPU 91 captures images of the calibration plate 100 a predetermined number of times at the imaging points a and b, and calculates the position (Xmia, Ymia) of the MI center in the robot coordinate system at the imaging point a, the relative position (Xra', Yra') of the center dot Dc with respect to the camera center in the robot coordinate system at the imaging point a, the position (Xmib, Ymib) of the MI center in the robot coordinate system at the imaging point b, and the relative position (Xrb', Yrb') of the center dot Dc with respect to the camera center in the robot coordinate system at the imaging point b, each predetermined number of times.

[0040] Next, the CPU 91 calculates the average values ​​of the angles θ1a and θ2a of the first joint axis J1 and the second joint axis J2 at the imaging point a obtained in S104, the average value of the position (Xmia, Ymia) of the MI center in the robot coordinate system calculated in S106, the average value of the relative position (Xra', Yra') of the center dot Dc with respect to the camera center in the robot coordinate system calculated in S110, the average values ​​of the angles θ1b and θ2b of the first joint axis J1 and the second joint axis J2 at the imaging point b obtained in S116, the position (Xmib, Ymib) of the MI center in the robot coordinate system calculated in S118, and the average value of the relative position (Xrb', Yrb') of the center dot Dc with respect to the camera center in the robot coordinate system calculated in S122 (S126). As a result, when the first arm driving unit 30 and the second arm driving unit 40 are controlled in S100 or S112 so that the robot arm 20 moves to the target position, even if there is variation in the actual position to which the arm moves relative to the target position, the position of the MI center in the robot coordinate system and the relative position of the center dot Dc with respect to the camera center in the robot coordinate system can be calculated with good accuracy.

[0041] Next, the CPU 91 uses the variation fluctuation (Lcx_e, Lcy_e) as a variable in the relative position of the camera center with respect to the MI center (Lcx_0 + Lcx_e, Lcy_0 + Lcy_e) and calculates the variation fluctuation (Lcx_e, Lcy_e) using the following relational expressions (5) to (10) such that the position (Xra, Yra) of the center dot Dc in the robot coordinate system calculated by imaging the calibration plate 100 (center dot Dc) at imaging point a is equal to the position (Xrb, Yrb) of the center dot Dc in the robot coordinate system calculated by imaging the calibration plate 100 (center dot Dc) at imaging point b (S128). Here, in equations (5) and (6), “R(θ1a+θ2a)” indicates a rotation matrix in which the angle (θ1a+θ2a) obtained by adding up the angles θ1a and θ2a of the first joint axis J1 and the second joint axis J2 at the imaging point a is used as a designated angle, and in equations (7) and (8), “R(θ1b+θ2b)” indicates a rotation matrix in which the angle (θ1a+θ2a) obtained by adding up the angles θ1a and θ2a of the first joint axis J1 and the second joint axis J2 at the imaging point b is used as a designated angle. In equations (5) and (6), "θ1a" and "θ2a" are the average values ​​of the angles θ1a and θ2a of the first joint axis J1 and the second joint axis J2 at the imaging point a calculated in S126, and in equations (7) and (8), "θ1b" and "θ2b" are the average values ​​of the angles θ1b and θ2b of the first joint axis J1 and the second joint axis J2 at the imaging point b calculated in S126. In equations (5) to (8), "Xmia", "Ymia", "Xmib", and "Ymib" are the average values ​​of the positions Xmia, Ymia, Xmib, and Ymib of the MI center in the robot coordinate system calculated in S126. In equations (5) to (8), "Xra'", "Yra'", "Xrb'", and "Yrb'" are the average values ​​of the relative positions Xra', Yra', Xrb', and Yrb' of the central dot Dc with respect to the camera center in the robot coordinate system calculated in S126.

[0042] Xra=Xmia+R(θ1a+θ2a)・(Lcx_0+Lcx_e)+Xra'...(5) Yra=Ymia+R(θ1a+θ2a)・(Lcy_0+Lcy_e)+Yra'...(6) Xrb=Xmib+R(θ1b+θ2b)・(Lcx_0+Lcx_e)+Xrb'...(7) Yrb=Ymib+R(θ1b+θ2b)・(Lcy_0+Lcy_e)+Yrb'…(8) Xra=Xrb…(9) Yra=Yrb…(10)

[0043] As shown in equations (5) and (6), the position (Xra, Yra) of the center dot Dc in the robot coordinate system is calculated by adding the relative position of the camera center to the MI center in the robot coordinate system, which is obtained by applying the rotation matrix R(θ1a+θ2a) to the relative position of the camera center to the MI center (Lcx_0+Lcx_e, Lcy_0+Lcy_e), to the position of the MI center (Xmia, Ymia), and the position (Xra', Yra') of the center dot Dc relative to the camera center in the robot coordinate system. Furthermore, as shown in equations (7) and (8), the position (Xrb, Yrb) of the center dot Dc in the robot coordinate system is calculated by adding the relative position of the camera center to the MI center in the robot coordinate system obtained by applying the rotation matrix R(θ1b+θ2b) to the relative position of the camera center to the MI center (Lcx_0+Lcx_e, Lcy_0+Lcy_e) to the position of the MI center (Xmib, Ymib), and the position (Xrb', Yrb') of the center dot Dc relative to the camera center in the robot coordinate system.

[0044] 14 shows the position (Xra, Yra) of the center dot Dc in the robot coordinate system calculated by capturing an image of the center dot Dc at imaging point a, and the position (Xrb, Yrb) of the center dot Dc in the robot coordinate system calculated by capturing an image of the center dot Dc at imaging point b. If the relative position (Lcx, Lcy) of the camera center with respect to the MI center is consistent with the design value (Lcx_0, Lcy_0), then the position (Xra, Yra) of the center dot Dc and the position (Xrb, Yrb) of the center dot Dc will match, as shown in FIG. 14. However, the relative position (Lcx, Lcy) of the camera center with respect to the MI center actually includes a variation (Lcx_e, Lcy_e) from the design value, and when the positions (Xra, Yra) and (Xrb, Yrb) of the center dot Dc in the robot coordinate system are calculated using the design value (Lcx_0, Lcy_0), the two positions will differ. Based on this, by setting up simultaneous equations with the variation fluctuation component (Lcx_e, Lcy_e) of the relative position (Lcx_0+Lcx_e, Lcy_0+Lcy_e) of the camera center with respect to the MI center as a variable so that the position (Xra, Yra) of the center dot Dc is equal to the position (Xrb, Yrb) of the center dot Dc, and solving these equations, the variation fluctuation component (Lcx_e, Lcy_e) can be calculated with good accuracy. In other words, the relative position (Lcx, Lcy) of the camera center with respect to the MI center can be calculated with good accuracy.

[0045] As described above, imaging points a and b are determined so as to maximize the difference between the angle of the robot arm 20 when the center dot Dc is imaged at imaging point a and the angle of the robot arm 20 when the same center dot Dc is imaged at imaging point b. As a result, when the relative position (Lcx, Lxy) of the camera center with respect to the MI center includes an error from the design value, a difference corresponding to the error can be generated between the position (Xra, Yra) of the center dot Dc calculated by imaging the center dot Dc at imaging point a and the position (Xrb, Yrb) of the center dot Dc calculated by imaging the center dot Dc at imaging point b, and the accuracy of calculating the variation (Lcx_e, Lcy_e) can be further improved.

[0046] After calculating the variation fluctuations (Lcx_e, Lcy_e) in this way, the CPU 91 adds the variation fluctuations (Lcx_e, Lcy_e) to the design values ​​(Lcx_0, Lcy_0) using the above-mentioned equations (3) and (4) to calculate the relative position (Lcx, Lcy) of the camera center with respect to the MI center, and registers the calculated relative position (Lcx, Lxy) in the storage device 94 (S130), thereby completing the calibration process. The operator simply places the calibration plate 100 on the workbench 2, and the robot 10 automatically images the calibration plate 100 and performs calibration, thereby enabling the calibration of the hand camera 60 to be performed simply and with good accuracy.

[0047] Here, the correspondence between the main elements of this embodiment and the main elements of the present disclosure will be described. That is, the control device 90 of this embodiment corresponds to a robot control device, the first arm 21 is an example of a first arm of the present disclosure, the second arm 22 is an example of a second arm, the hand camera 60 is an example of a camera, the storage device 94 is an example of a storage unit, and the encoders 34 and 44 are examples of a detection unit. Also, the CPU 91 of the control device 90 that executes the recognition process is an example of a recognition unit, and the CPU 91 of the control device 90 that executes the calibration process is an example of a calculation unit.

[0048] It goes without saying that the present disclosure is not limited to the above-described embodiments, and can be embodied in various forms as long as they fall within the technical scope of the present disclosure.

[0049] For example, in the above-described embodiment, the CPU 91 captures images of the calibration plate 100 at the imaging point a and the imaging point b a predetermined number of times and calculates the average of the images captured over the predetermined number of times. However, the CPU 91 may capture images of the calibration plate 100 once each at the imaging point a and the imaging point b.

[0050] In the above-described embodiment, imaging points a and b are determined so that the images of the central dot Dc are diagonally positioned relative to each other in the field of view of the hand camera 60. However, imaging points a and b may be located at any positions as long as they capture images of the same central dot Dc by the hand camera 60 at different positions.

[0051] As described above, in a robot with a horizontally articulated arm, the robot control device allows an operator to place an object on the robot arm, drive the robot arm, and capture images of the same object with cameras at multiple imaging positions, thereby automatically estimating the relative position of the cameras with respect to the mechanical interface. As a result, in a robot with a horizontally articulated arm, camera calibration can be performed easily and with high accuracy.

[0052] In the robot control device according to the present disclosure, the first position of the object in the robot coordinate system based on a point on the rotation axis of the first arm and the second position of the object in the robot coordinate system may be the same value, and the relative position of the camera with respect to the mechanical interface may be calculated and estimated by this. In this way, the relative position of the camera with respect to the mechanical interface can be calculated with high accuracy through simple calculations.

[0053] Further, in the robot control device disclosed herein, the relative position of the camera with respect to the mechanical interface includes a design value for a position related to attachment of the camera to the second arm and a variation due to variation from the design value, and the first position is calculated by coordinate transforming, into a robot coordinate system, the sum of the design value and the variation and the position of the object in an image coordinate system recognized from the image of the object at the first imaging position, based on the angle of each arm detected by the detection unit when the object is imaged at the first imaging position, and the second position is calculated by coordinate transforming, into a robot coordinate system, the sum of the design value and the variation and the position of the object in an image coordinate system recognized from the image of the object at the second imaging position, based on the angle of each arm detected by the detection unit when the object is imaged at the second imaging position, and the variation is calculated based on a combination of the first position and the second position, and the relative position of the camera with respect to the mechanical interface may be estimated. In this way, the relative position of the camera with respect to the mechanical interface can be estimated with good accuracy, taking into account the variation in the design value.

[0054] In the robot control device disclosed herein, the first and second imaging positions may be set so that an image of the object captured at the first imaging position and an image of the object captured at the second imaging position are positioned diagonally opposite each other in the field of view of the camera. This allows a large difference in the angle of the robot arm between the first and second imaging positions, thereby enabling the relative position of the camera with respect to the mechanical interface to be estimated with greater accuracy.

[0055] In this case, the camera may be replaceable with a lens of a different magnification, and the control device may set the first imaging position and the second imaging position so that the first imaging position and the second imaging position are located diagonally in the field of view of the camera based on a setting of the field of view of the camera according to the attached lens. By setting the first imaging position and the second imaging position according to the field of view that the camera can take, it is possible to make the angle difference of the robot arm large between the first imaging position and the second imaging position.

[0056] Further, in the robot control device of the present disclosure, a first operation of driving and controlling the robot arm so that the camera is located at the first image capturing position and then capturing an image of the object with the camera, and a second operation of driving and controlling the robot arm so that the camera is located at the second image capturing position and then capturing an image of the object with the camera are executed a plurality of times each, and an average value of angles of each arm detected by the detection unit when the object is captured a plurality of times in the first operation and an average value of positions of the object in an image coordinate system recognized from a plurality of captured images when the object is captured a plurality of times in the first operation are calculated, and and calculating an average value of the angles of each arm detected by a detection unit and an average value of the position of the object in an image coordinate system recognized from a plurality of captured images of the object captured a plurality of times during the second movement, and estimating the relative position of the camera with respect to the mechanical interface by combining the first position of the object in the robot coordinate system calculated based on the average value of the angles of each arm during the first movement and the average value of the position of the object in the image coordinate system during the first movement, and the second position of the object in the robot coordinate system calculated based on the average value of the angles of each arm during the second movement and the average value of the position of the object in the image coordinate system during the second movement. In this way, even if there is variation in the position to which the robot arm actually moves relative to its target position when controlling the drive of the robot arm, it is possible to measure the angle of each arm at each capturing position and the position of the object in the image coordinate system with high accuracy.

[0057] In addition, in the robot control device disclosed herein, the mechanical interface may be equipped with a workpiece holding unit capable of holding a workpiece supplied to a workpiece supply position, and the target object may be provided at the workpiece supply position.

[0058] In the robot control device disclosed herein, the object may have a plurality of dots arranged at a specified interval, and the camera may capture an image of the plurality of dots, measure the distance between the dots in pixel units in the captured image, and calculate a resolution for converting the position in pixel units to a position in length units. In this way, it is possible to estimate the camera resolution as well as the relative position of the camera with respect to the mechanical interface using a single object, thereby enabling efficient calibration.

[0059] The present disclosure is not limited to the form of a robot control device, but may also be in the form of a robot control method or a robot.

[0060] This specification also discloses the technical idea of ​​changing "a robot control device according to claim 1 or 2" in claim 4 as originally filed to "a robot control device according to any one of claims 1 to 3", the technical idea of ​​changing "a robot control device according to claim 1 or 2" in claim 6 as originally filed to "a robot control device according to any one of claims 1 to 5", the technical idea of ​​changing "a robot control device according to claim 1 or 2" in claim 7 as originally filed to "a robot control device according to any one of claims 1 to 6", and the technical idea of ​​changing "a robot control device according to claim 1 or 2" to "a robot control device according to any one of claims 1 to 7" in claim 8 as originally filed.

[0061] The present disclosure can be used in the manufacturing industry of robots equipped with horizontal articulated robot arms.

[0062] 1 Robot system, 2 Work table, 10 Robot, 11 Base, 20 Robot arm, 21 First arm, 22 Second arm, 23 Shaft, 24 Work holder, 30 First arm drive unit, 32 Motor, 34 Encoder, 40 Second arm drive unit, 42 Motor, 44 Encoder, 50 Shaft drive unit, 52a, 52b Motor, 54a, 54b Encoder, 60 Hand camera, 61 Bracket, 70 Work camera, 80 Work supply unit, 90 Control device, 91 CPU, 92 ROM, 93 RAM, 94 Storage device, 100 Calibration plate, D Dot, Dc Center dot, J1 First joint axis, J2 Second joint axis, J3 Third joint axis, MI Mechanical interface, S Mounting object, W Work.

Claims

1. A robot control device comprising: a robot arm having a first arm that can rotate horizontally, and a second arm that can rotate horizontally relative to the first arm and is provided with a mechanical interface; a camera attached to the second arm; a memory unit that stores a relative position that is the relative position of the camera with respect to the mechanical interface of the second arm; and a detection unit that detects the angle of each arm of the robot arm, wherein the device captures an image of a workpiece with the camera, and recognizes the position of the workpiece in a robot coordinate system based on the angle of each arm detected by the detection unit when the image of the workpiece is captured, the position of the workpiece in an image coordinate system recognized from the captured image of the workpiece, and the relative position of the camera with respect to the mechanical interface stored in the memory unit, a first position of the object in a robot coordinate system calculated based on the angle of each arm detected by the detection unit when the object is imaged at a first imaging position among the plurality of imaging positions and the position of the object in an image coordinate system recognized from the image of the object at the first imaging position; and a second position of the object in the robot coordinate system calculated based on the angle of each arm detected by the detection unit when the object is imaged at a second imaging position among the plurality of imaging positions and the position of the object in an image coordinate system recognized from the image of the object at the second imaging position.

2. A robot control device according to claim 1, wherein the first position of the object in the robot coordinate system based on a point on the rotation axis of the first arm and the second position of the object in the robot coordinate system are the same value, and the relative position of the camera with respect to the mechanical interface is calculated and estimated.

3. A robot control device according to claim 1 or 2, wherein the relative position of the camera with respect to the mechanical interface includes a design value for the position of the camera attached to the second arm and a variation due to variations in the design value; the first position is calculated by converting into a robot coordinate system the sum of the design value and the variation and the position of the object in an image coordinate system recognized from the image of the object at the first imaging position, based on the angle of each arm detected by the detection unit when the object is imaged at the first imaging position; the second position is calculated by converting into a robot coordinate system the sum of the design value and the variation and the position of the object in an image coordinate system recognized from the image of the object at the second imaging position, based on the angle of each arm detected by the detection unit when the object is imaged at the second imaging position; and the robot control device calculates the variation and estimates the relative position of the camera with respect to the mechanical interface by combining the first position and the second position.

4. A robot control device according to claim 1 or 2, wherein the first imaging position and the second imaging position are set so that an image of the object when the object is imaged at the first imaging position and an image of the object when the object is imaged at the second imaging position are positioned at diagonal corners of the field of view of the camera.

5. A robot control device according to claim 4, wherein the camera is replaceable with a lens of a different magnification, and the control device sets the first imaging position and the second imaging position so that they are located at diagonal corners of the camera's field of view based on a setting of the camera's field of view that corresponds to the attached lens.

6. A control device for a robot according to claim 1 or 2, wherein the control device executes a first operation of driving and controlling the robot arm so that the camera is positioned at the first imaging position, and then an image of the object is captured by the camera, and a second operation of driving and controlling the robot arm so that the camera is positioned at the second imaging position, and then an image of the object is captured by the camera, multiple times each; the control device calculates an average value of the angle of each arm detected by the detection unit when the object is captured multiple times in the first operation, and an average value of the position of the object in an image coordinate system recognized from the multiple captured images when the object is captured multiple times in the first operation; and calculating an average value of the angle of each arm in the first movement and an average value of the position of the object in an image coordinate system recognized from a plurality of captured images of the object captured a plurality of times during the second movement, and estimating the relative position of the camera with respect to the mechanical interface by combining the first position of the object in the robot coordinate system calculated based on the average value of the angle of each arm in the first movement and the average value of the position of the object in the image coordinate system during the first movement, and the second position of the object in the robot coordinate system calculated based on the average value of the angle of each arm in the second movement and the average value of the position of the object in the image coordinate system during the second movement.

7. A robot control device according to claim 1 or 2, wherein the mechanical interface is equipped with a workpiece holding unit capable of holding a workpiece supplied to a workpiece supply position, and the target object is provided at the workpiece supply position.

8. A robot control device according to claim 1 or 2, wherein the object has a plurality of dots arranged at a specified interval, the plurality of dots are imaged by the camera, the distance between the dots in the image is measured in pixel units, and a resolution for converting the position in pixel units into a position in length units is calculated.

9. A method for controlling a robot comprising: a robot arm having a first arm capable of horizontal rotation, and a second arm capable of horizontal rotation relative to the first arm and provided with a mechanical interface; a camera attached to the second arm; a memory unit that stores a relative position that is the relative position of the camera with respect to the mechanical interface of the second arm; and a detection unit that detects the angle of each arm of the robot arm, wherein the method captures an image of a workpiece with the camera, and recognizes the position of the workpiece in a robot coordinate system based on the angles of each arm detected by the detection unit when the image of the workpiece is captured, the position of the workpiece in an image coordinate system recognized from the captured image of the workpiece, and the relative position of the camera with respect to the mechanical interface stored in the memory unit, a first position of the object in a robot coordinate system calculated based on the angle of each arm detected by the detection unit when the object is imaged at a first imaging position among the plurality of imaging positions and the position of the object in an image coordinate system recognized from the image of the object at the first imaging position; and a second position of the object in the robot coordinate system calculated based on the angle of each arm detected by the detection unit when the object is imaged at a second imaging position among the plurality of imaging positions and the position of the object in an image coordinate system recognized from the image of the object at the second imaging position.

10. A robot arm having a first arm capable of horizontal rotation and a second arm capable of horizontal rotation relative to the first arm and provided with a mechanical interface; an end effector unit provided on the second arm; a camera attached to the second arm; a memory unit that stores a relative position that is the relative position of the camera with respect to the mechanical interface of the second arm; a detection unit that detects the angle of each arm of the robot arm; and a recognition unit that captures an image of a workpiece with the camera and recognizes the position of the workpiece in a robot coordinate system based on the angle of each arm detected by the detection unit when the workpiece is captured, the position of the workpiece in an image coordinate system recognized from the captured image of the workpiece, and the relative position of the camera with respect to the mechanical interface stored in the memory unit. an estimation unit that drives the robot arm to capture images of the same object with the camera at a plurality of imaging positions, and estimates a relative position of the camera with respect to the mechanical interface by combining: a first position of the object in a robot coordinate system that is calculated based on an angle of each arm detected by the detection unit when the object is captured at a first imaging position of the plurality of imaging positions and a position of the object in an image coordinate system recognized from the captured image of the object at the first imaging position; and a second position of the object in the robot coordinate system that is calculated based on the angle of each arm detected by the detection unit when the object is captured at a second imaging position of the plurality of imaging positions and a position of the object in an image coordinate system recognized from the captured image of the object at the second imaging position.

Images