End effector, robot system, method of controlling end effector, method of controlling robot system, method of manufacturing article, program, and recording medium

Abstract

To capture images of a target object even in a state where a workpiece is supported by work tools.SOLUTION: An end effector (100) is attached to a robot body of a robot system and includes: a body portion (101); a camera (120) which is supported by the body portion and can capture images; and work tools (141, 142) which are movably supported by the body portion, can support a workpiece (W) and perform operations on the workpiece. When images of a target object are captured by the camera, the work tools supporting the workpiece are positioned outside a range of overlap in which the work tools and the workpiece supported by the work tools overlap with the target object in a field of view of the camera.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an end effector, a robot system, a control method for an end effector, a control method for a robot system, a manufacturing method for an article, a program, and a recording medium. [Background technology]

[0002] For example, a robot hand that grasps a workpiece with the fingers and places the workpiece at a desired position has been disclosed (see Patent Document 1). In Patent Document 1, the workpiece is imaged by a camera to obtain geometric information of the workpiece, and the fingers are controlled based on the geometric information to grasp the workpiece. When the workpiece is imaged, the fingers are retracted outside the imaging area. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-120545 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the above-mentioned Patent Document 1, when the workpiece is gripped by the fingers, the installation position where the workpiece is to be installed is blocked by the workpiece and cannot be imaged by the camera. Therefore, the installation position where the workpiece is to be installed cannot be accurately grasped, and the position of the fingers cannot be accurately corrected to the installation position, resulting in a problem that the accuracy of the installation position in the installation operation of the workpiece becomes poor.

[0005] Therefore, an object of the present invention is to provide an end effector, a robot system, a control method for an end effector, a control method for a robot system, a manufacturing method for an article, a program, and a recording medium that are capable of imaging an imaging target while supporting a workpiece with a tool. [Means for solving the problem]

[0006] One aspect of the present invention is an end effector comprising a main body, a camera supported on the main body and capable of taking images, and a tool supported movably on the main body, capable of supporting a workpiece, and capable of performing work on the workpiece, wherein when an image of an object to be imaged is taken with the camera, the tool supporting the workpiece is positioned outside the range in the camera's field of view where at least a portion of the tool or the workpiece supported by the tool overlaps with the object to be imaged.

[0007] One aspect of the present invention is a robot system comprising: an end effector having a camera capable of capturing images and an illumination unit that illuminates light in an optical axis direction; a robot body to which the end effector is attached and capable of moving the position and posture of the end effector; and a control unit that controls the robot body and the end effector, wherein when the control unit performs alignment control to align the end effector with respect to an imaging target from an image captured by the camera, the control unit irradiates an imaging target having a flat portion formed in a planar shape and an inclined portion inclined with respect to the flat portion with the illumination unit so that the optical axis of reflected light reflected from the flat portion is directed toward the camera and the optical axis of reflected light reflected from the inclined portion is directed in a direction different from the camera, and the imaging target is then imaged by the camera.

[0008] One aspect of the present invention is a method for controlling an end effector which includes a main body, a camera supported on the main body and capable of taking images, and a tool supported movably on the main body and performing work on a workpiece, and which is controlled by a control unit, characterized in that when the control unit images an image target with the camera, it positions the tool supporting a workpiece outside the range in the camera's field of view where at least a portion of the tool or a workpiece supported by the tool overlaps with the image target.

[0009] One aspect of the present invention is a control method for a robot system including a main body, an end effector including a camera supported on the main body and capable of taking images, and a tool supported movably on the main body and performing work on a workpiece, a robot body to which the end effector is attached and capable of moving the position and posture of the end effector, and a control unit that controls the robot body and the end effector, the control unit comprising: a retraction step of positioning the tool supporting a workpiece outside a range in the camera's field of view where at least a portion of the tool or a workpiece supported by the tool overlaps with the imaged object when imaging an image of an imaged object with the camera, an imaging step of the control unit imaging the image of the imaged object with the camera while positioning the tool supporting a workpiece outside the range, and an alignment step of the control unit aligning the end effector with respect to the imaged object from the image captured by the camera.

[0010] One aspect of the present invention is a control method for a robot system including an end effector having a camera capable of capturing images and an irradiation unit that irradiates light in an optical axis direction, a robot body to which the end effector is attached and capable of moving the position and posture of the end effector, and a control unit that controls the robot body and the end effector, the control unit comprising: an irradiation step in which the control unit irradiates an imaging target having a flat portion formed in a planar shape and an inclined portion inclined with respect to the flat portion, such that the optical axis of reflected light reflected by the flat portion faces the camera and the optical axis of reflected light reflected by the inclined portion faces in a direction different from that of the camera; an imaging step in which the control unit images the imaging target irradiated with light by the irradiation step, using the camera; and a calculation step in which the control unit calculates a positional relationship between the end effector and the imaging target from the image captured by the camera. Effect of the Invention

[0011] According to the present invention, it is possible to capture an image of an imaging target even in a state where a workpiece is supported by a tool. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating a robot system according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing a control system of the robot system according to the first embodiment. [Diagram 3] FIG. 1 is a perspective view showing a robot hand according to a first embodiment. [Figure 4] FIG. 2 is a bottom view showing the robot hand with a workpiece gripped by the fingers positioned within the field of view of the camera. [Diagram 5] FIG. 2 is a bottom view showing the robot hand in a state where the workpiece gripped by the fingers is outside the field of view of the camera. [Figure 6] FIG. 2 is a side view showing the configuration of the fingers of the robot hand. [Figure 7] FIG. 1 is a block diagram showing image-based visual servo control in a robot device. [Figure 8] 1 is a flowchart showing image-based visual servo control in a robot device. [Figure 9] 5 is a flowchart showing pin assembly control according to the first embodiment. [Figure 10] (a) is a schematic diagram showing an image captured by the camera when the pin is gripped by the fingers. (b) is a schematic diagram showing an image captured by the camera when the robot hand is moved above a hole in a workpiece to be assembled. (c) is a schematic diagram showing an image captured by the camera when the fingers and pin are moved out of the field of view. (d) is a schematic diagram showing an image captured by the camera when the robot hand is aligned with the hole in the workpiece to be assembled. (e) is a schematic diagram showing an image captured by the camera when the fingers and pin are moved above the hole in the workpiece to be assembled. [Figure 11] 10 is a flowchart showing pin assembly control according to the second embodiment. [Figure 12](a) is a schematic diagram showing an image captured by the camera when the pin is gripped by the fingers, (b) is a schematic diagram showing an image captured by the camera when the robot hand is moved above a hole in a workpiece to be assembled with the fingers and pin moved out of the field of view, (c) is a schematic diagram showing an image captured by the camera when the robot hand is aligned with respect to the hole in the workpiece to be assembled, and (d) is a schematic diagram showing an image captured by the camera when the fingers and pin are moved above the hole in the workpiece to be assembled. [Figure 13] FIG. 13 is a schematic diagram showing the structure of a camera and an illumination unit according to a third embodiment. [Figure 14] 13A is a diagram showing an image of a hole in a work to be assembled captured by a general camera, and FIG. 13B is a diagram showing an image of a hole in a work to be assembled captured by a camera according to a third embodiment. [Figure 15] 13A is a diagram showing an image of a pin stand with pins placed in holes captured by a general camera, and FIG. 13B is a diagram showing an image of a pin stand with pins placed in holes captured by a camera according to a third embodiment. [Figure 16] 13 is a flowchart showing posture correction control of a robot arm. [Figure 17] 1A is a schematic diagram showing a robot arm before and after its posture is corrected, and FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] First Embodiment A first embodiment for carrying out the present invention will be described below with reference to FIGS. 1 to 10. FIG.

[0014] [Outline of the robot system] First, a schematic configuration of a robot system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing the robot system according to the first embodiment.

[0015] 1, a robot system 1 installed in, for example, a factory includes a robot device 10 that grips and moves a pin W, which is a workpiece, from a pin placement table 401 on which a metallic pin W, for example, is placed, to a workpiece 402 to be assembled. The robot device 10 includes a base 2, a robot arm 10A as a robot main body of a six-axis articulated robot supported by the base 2, and a robot hand 100 as an end effector attached to the tip of the robot arm 10A. In other words, the robot arm 10A is a so-called manipulator, and is provided at its tip with a flange-shaped flange portion 11 to which the robot hand 100 is attached.

[0016] The pin rest 401 is formed with a plurality of holes 401H into which the pins W are inserted, and the pins W supplied as parts before the work are placed in these holes 401H. The work 402 to be assembled is formed with holes 402H into which the pins W as the work are inserted and fitted for assembly. That is, the robot device 10, which will be described later in detail, executes assembly control to perform the work of assembling the pins W to the work 402 to be assembled, and the completed work 402 to be assembled is manufactured as an article by assembling the pins W to the work 402 to be assembled.

[0017] The robot device 10 also has a robot controller 200 that controls the robot arm 10A, a vision controller 220 that controls a camera 120 (see FIG. 3), which will be described in detail later, and a hand controller 230 that controls the robot hand 100. That is, each of these controllers is electrically connected so as to be able to output commands to the respective devices.

[0018] The robot controller 200 is configured to be connectable to a teaching pendant 300 serving as a teaching device for allowing a user to perform teaching (i.e., instruction) of the robot device 10. The teaching pendant 300 has a display unit 300A serving as a display device for displaying various information. Note that, although the present embodiment describes a configuration in which the teaching pendant 300 serving as a teaching device and the display unit 300A serving as a display device are integrated, they may be configured as separate devices.

[0019] The teaching pendant 300 is configured to be able to drive the robot arm 10A by issuing a command to the robot controller 200, and to control the position and angle of each joint of the robot arm 10A, thereby manipulating the position and posture of the robot arm A. The teaching pendant 300 is also configured to be able to manipulate the position and posture of the robot hand 100, since it can manipulate the position and posture of the robot arm 10A. The position and posture of the robot arm 10A or the position and posture of the robot hand 100 manipulated by the teaching pendant 300 can be stored in a hard disk (HDD) 204 as a storage unit described later. This allows the robot device 10 to store the position and posture of the robot arm 10A or the position and posture of the robot hand 100 in various operations (assembly work and position correction operation), that is, teaching is performed. The robot device 10 performs various operations (various tasks) based on the stored position and posture of the robot arm 10A or the position and posture of the robot hand 100.

[0020] [Control system configuration of robot system] Next, the control system of the robot system 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the control system of the robot system according to the first embodiment.

[0021] 2, the robot controller 200 is configured as a computer. The robot controller 200 has a CPU (Central Processing Unit) 201 as a processor and a control unit. The robot controller 200 also has a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and a HDD (Hard Disk Drive) 204 as examples of a storage unit. Furthermore, the robot controller 200 has a recording disk drive 205 and a plurality of input / output interfaces (I / F) 139-143.

[0022] A ROM 202, a RAM 203, a HDD 204, a recording disk drive 205, and interfaces 206 to 210 are connected to the CPU 201 via a bus 211. Basic programs such as BIOS are stored in the ROM 202. The RAM 203 is a storage device that temporarily stores various data such as the results of calculations performed by the CPU 201.

[0023] The HDD 204 is a storage device that stores the results of arithmetic processing by the CPU 201 and various data acquired from the outside. A program PR for causing the CPU 201 to execute arithmetic processing is recorded in this HDD 204. The CPU 201 executes various controls (control methods) described below and various processes of a manufacturing method for an article based on the program PR recorded (stored) in the HDD 204. The recording disk drive 205 as a recording medium can read out various data, programs, etc. recorded on the recording disk 299. That is, the program PR recorded in the recording disk drive 205 can be read into the HDD 204 and installed.

[0024] A teaching pendant 300 is connected to the interface 206. The CPU 201 acquires input data (input information) from the teaching pendant 300 via the interface 206 and the bus 211. The CPU 201 also transmits image data to a display unit 300A (see FIG. 1) provided on the teaching pendant 300 via the bus 211 and the interface 206, and executes various displays. The interface 207 is configured to be connectable to an external storage device 350, which is a storage unit such as a rewritable nonvolatile memory or an external HDD.

[0025] The interface 208 is connected to the servo control unit 15. The servo control unit 15 is connected to the motor 16, the angle sensor 17, and the torque sensor 18 provided in each joint of the robot arm 10A (see FIG. 1). Note that FIG. 10 shows the configuration of one joint among the multiple joints as a representative. The motor 16 is, for example, a brushless DC motor or an AC motor, and rotates and drives the corresponding joint via a reduction gear (not shown). The angle sensor 17 is, for example, a rotary encoder, and is provided in the motor 16 and configured to be able to detect the rotation angle of the motor 16. The torque sensor 18 is provided in the corresponding joint and configured to be able to detect the torque acting on the corresponding joint.

[0026] That is, the CPU 201 can obtain angle information from the angle sensor 17 and torque information from the torque sensor 18 via the servo control unit 15, the interface 208, and the bus 211. The servo control unit 15 may divide the angle of the motor 16 detected by the angle sensor 17 by a reduction ratio of a reducer (not shown), convert it into angle information of the corresponding joint, and transmit it to the CPU 201. Then, the CPU 201 outputs data of a command value corresponding to each joint to the servo control unit 15 via the bus 211 and the interface 208 at a predetermined time interval (for example, 1 ms). This drives the motor 16 to drive the joint. That is, the CPU 201 (robot controller 200) controls the robot arm 10A to be in a position and posture according to the command value.

[0027] A vision controller 220 is connected to the interface 209. A camera 120 is connected to the vision controller 220, and the vision controller 220 captures images of the connected camera 120 at a predetermined time interval (for example, 30 ms) under the control of the CPU 201. This allows the CPU 201 to obtain visual information, that is, captured image data, from the vision controller 220 at a predetermined time interval (for example, 30 ms). In addition, a light 149 is connected to the vision controller 220, and the illumination light is turned on / off and its intensity is changed under the control of the vision controller 220 based on a command from the CPU 201.

[0028] A hand controller 230 is connected to the interface 210. The robot hand 100 is connected to the hand controller 230. Under the control of the CPU 201, the hand controller 230 drives a first motor 161 and a second motor 151, which will be described in detail later, to open and close the finger portion 140 (a so-called hand).

[0029] [Structure of the robot hand] Next, the detailed configuration of the robot hand 100 will be described with reference to Figs. 3, 4, 5, and 6. Fig. 3 is a perspective view showing the robot hand according to the first embodiment. Fig. 4 is a bottom view showing the robot hand in a state where a workpiece held by the fingers is located within the field of view of the camera. Fig. 5 is a bottom view showing the robot hand in a state where a workpiece held by the fingers is located outside the field of view of the camera. Fig. 6 is a side view showing the configuration of the fingers of the robot hand.

[0030] As shown in FIG. 3, the robot hand 100 detachably attached to the tip of the robot arm 10A (see FIG. 1) is roughly configured to include a base 101 as a main body, a camera 120, and a finger portion 140 as a tool and a hand. The base 101 has an attachment portion 102 attached to the flange portion 11 (see FIG. 1) of the robot arm 10A, and a camera 120 capable of capturing images is fixed and supported on the base 101. The camera 120 is moved so that the position of the robot hand 100 is a position in the sky at a predetermined distance from the workpiece 402 to be assembled by controlling the robot arm 10A, and the position of the camera 120 is controlled so that the focal position P (see FIG. 3) is a hole 402H of the workpiece 402 to be assembled. This enables the camera 120 to capture an image of the hole 402H, which is an image-capturing target, a work target, and a contact position. Similarly, when imaging the pin W or hole 401H placed on the pin stand 401, the position is controlled so that the focal position P (see FIG. 3) becomes the hole 401H or the pin W of the pin stand 401. In this case, the imaging target becomes the hole 401H or the pin W, and the work target or contact position becomes the pin W placed in the hole 401H.

[0031] In addition, the base 101 supports a first finger portion 141 and a second finger portion 142 that constitute a finger portion 140 capable of performing an operation on a pin W, which is a workpiece, in a freely movable manner. The fingertip portion 141b of the first finger portion 141 and the fingertip portion 142b of the second finger portion are capable of supporting the pin W by pinching and gripping the pin W between each other. The fingertip portion 141b of the first finger portion 141 and the fingertip portion 142b of the second finger portion are located between the base 101 and the focal position P (see FIG. 3) of the camera 120 while being within the field of view of the camera 120, as will be described later in detail. This allows the camera 120 to capture an image of an area farther away than the first finger portion 141 and the second finger portion 142 and perform various controls based on the image.

[0032] Next, a structure for driving and moving the finger portion 140 will be described. As described above, the finger portion 140 has a first finger portion 141 and a second finger portion 142 as shown in Figs. 3 and 4, and is configured to be able to grip a pin W, which is a workpiece. The first finger portion 141 is fixed to a first slider 143, and the second finger portion 142 is fixed to a second slider 144. The base portion 101 includes a first guide portion 170 and a second guide portion 180 that movably guide the first slider 143 and the second slider 144, a first drive portion 150 that drives the first slider 143, and a second drive portion 160 that drives the second slider 144.

[0033] In detail, the first guide part 170 as a sliding part has a first linear guide 171, and movable parts 172 and 173 slidably supported thereon. The second guide part 180 as a sliding part has a second linear guide 181, and movable parts 182 and 183 slidably supported thereon. The first slider 143 is fixed to the movable part 173 and the movable part 183, and the second slider 144 is fixed to the movable part 172 and the movable part 182.

[0034] On the other hand, the first driving unit 150 is roughly configured to include a motor 151, a driving pulley 152, a belt 153, a driven pulley 154, a ball screw shaft 155, a ball nut 156, and an angle detection sensor 159. That is, the motor 151 outputs a driving rotation, and the driving pulley 152 fixed to the output shaft of the motor 151 is rotated. A belt 153 is stretched between the driving pulley 152 and the driven pulley 154, and the rotation of the driving pulley 152 is transmitted to the driven pulley 154. The driven pulley 154 is fixed to one end of a ball screw shaft 155, and by rotating the ball screw shaft 155, a ball nut 156 fixed to the first slider 143 is slid in the axial direction via a ball (not shown). Therefore, the first slider 143 linearly slides in the axial direction of the ball screw shaft 155 (in the direction along the first linear guide 171 and the second linear guide 181) while being guided by the first guide portion 170 and the second guide portion 180. Therefore, the first finger portion 141 is driven to move in the axial direction of the ball screw shaft 155 according to the rotation direction of the motor 151. For example, when the motor 151 is rotated forward, the first finger portion 141 moves to one side toward the right in the figure, and when the motor 151 is rotated backward, the first finger portion 141 moves to the other side toward the left in the figure. The rotation angle of the motor 151 is detected by an angle detection sensor 159 constituted by, for example, an encoder, and is output to the robot controller 200 via the hand controller 230. Based on this, the CPU 201 determines the position of the first finger portion 141 by calculating it, and also controls the position of the first finger portion 141 by issuing a command to drive the motor 151 based on the position.

[0035] Similarly, the second driving unit 160 is roughly configured to include a motor 161, a driving pulley 162, a belt 163, a driven pulley 164, a ball screw shaft 165, a ball nut 166, and an angle detection sensor 169. That is, the motor 161 outputs a driving rotation, and the driving pulley 162 fixed to the output shaft of the motor 161 is rotated. A belt 163 is stretched between the driving pulley 162 and the driven pulley 164, and the rotation of the driving pulley 162 is transmitted to the driven pulley 164. The driven pulley 164 is fixed to one end of a ball screw shaft 165, and by rotating the ball screw shaft 165, a ball nut 166 fixed to the second slider 144 is slid in the axial direction via a ball (not shown). Therefore, the second slider 144 linearly slides in the axial direction of the ball screw shaft 165 while being guided by the first guide portion 170 and the second guide portion 180. Therefore, the second finger portion 142 is driven to move in the axial direction of the ball screw shaft 165 according to the rotation direction of the motor 161. For example, when the motor 161 is rotated forward, the second finger portion 142 moves to one side toward the left in the figure, and when the motor 161 is rotated backward, the second finger portion 142 moves to the other side toward the right in the figure. The rotation angle of the motor 161 is detected by an angle detection sensor 169 constituted by, for example, an encoder, and is output to the robot controller 200 via the hand controller 230. Based on this, the CPU 201 determines the position of the second finger portion 142 by calculating it, and controls the position of the second finger portion 142 by issuing a command to drive the motor 161 based on the position.

[0036] With the base 101 configured as described above, the first finger portion 141 and the second finger portion 142 are capable of independently moving on the same first linear motion guide 171 and second linear motion guide 181. This reduces deviation in the relative positions of the first finger portion 141 and the second finger portion 142 other than in the movement direction when the first finger portion 141 and the second finger portion 142 move while gripping the pin W, making it easier to control the movement while gripping the pin W.

[0037] Furthermore, by placing the second linear guide 181 on the opposite side of the first linear guide 171 from the gripping position of the pin W, the moment force generated when placing the pin W8 can be supported by the two linear guides. This makes it possible to improve the durability of the first linear guide 171 and the second linear guide 181.

[0038] When the pin W is to be gripped between the first finger portion 141 and the second finger portion 142, first, the rotation angle of the motor 151 is detected by the angle detection sensor 159, and the rotation angle of the motor 161 is detected by the angle detection sensor 169. Then, based on these values, the CPU 201 controls the motor 151 and the motor 161 so that the first finger portion 141 and the second finger portion 142 are spaced apart from each other by the pin W and are positioned at the pin W. As a result, the pin W is gripped between the first finger portion 141 and the second finger portion 142.

[0039] In addition, the first finger portion 141 and the second finger portion 142 can move in the same direction along the first linear guide 171 and the second linear guide 181 while maintaining the distance between them, that is, they can move while gripping the pin W. That is, the first finger portion 141 and the second finger portion 142 can be moved from the position shown in FIG. 4 to the position shown in FIG. 5. The state shown in FIG. 5 is a state in which the first finger portion 141, the second finger portion 142, and the pin W are in a retracted position outside the field of view of the camera 120 (hereinafter, simply referred to as "outside the field of view"). The movement outside the field of view of the camera 120 will be described later in the pin assembly control. Note that when the first finger portion 141 and the second finger portion 142 are moved while gripping the pin W, the distance between the first finger portion 141 and the second finger portion 142 is maintained. Therefore, based on the values ​​detected by angle detection sensor 159 and angle detection sensor 169, CPU 201 controls first finger portion 141 and second finger portion 142 so as to move their positions in a coordinated manner.

[0040] Here, the shapes of the first finger portion 141 and the second finger portion 142 will be described with reference to Fig. 6. The second finger portion 142 has a second root portion 142a fixed to the second slider 144, a second fingertip portion 142b that contacts and grips the pin W, and a second connecting portion 142c that connects the second root portion 142a and the second fingertip portion 142b. Similarly, the first finger portion 141 has a first root portion 141a fixed to the first slider 143, a first fingertip portion 141b that contacts and grips the pin W, and a first connecting portion 141c that connects the first root portion 141a and the first fingertip portion 141b. In addition, since the shape of the first finger portion 141 and the shape of the second finger portion 142 are substantially the same, the shape of the second finger portion 142 will be described and a description of the first finger portion 141 will be omitted.

[0041] 6 with respect to the imaging direction (i.e., the optical axis direction) of camera 120, and is linearly slid as described above at a position that does not intersect with the imaging direction. Second connecting portion 142c is formed in a bent shape so that second fingertip portion 142b faces the imaging direction, that is, formed so that second fingertip portion 142b can enter inside the field of view of camera 120 (hereinafter simply referred to as "within the field of view"). That is, second fingertip portion 142b moves linearly parallel to second root portion 142a due to the sliding movement of second root portion 142a, and therefore moves from outside the field of view of camera 120 through the field of view to outside the field of view.

[0042] Since the first finger portion 141 and the second finger portion 142 are shaped as described above, the camera 120 can capture an image of the first fingertip portion 141b and the second fingertip portion 142b and the pin W being held. The image captured by the camera 120 does not capture the first connecting portion 141c, the second connecting portion 142c, the first root portion 141a, or the second root portion 142a other than the first fingertip portion 141b and the second fingertip portion 142b. This makes it possible to minimize the area where the first finger portion 141 and the second finger portion 142 overlap with the work position (the hole portion 402H of the workpiece 402 to be assembled, the hole portion 401H of the pin rest 401, the pin W placed on the pin rest 401, etc.) when captured by the camera 120. In addition, such a bent shape makes it possible to achieve a compact structure that can capture images of the first fingertip 141b and the second fingertip 142b without interfering with the camera 120 even when the first finger 141 and the second finger 142 are moved. This makes it possible to reduce the size of the robot hand 100.

[0043] [Visual servo control] Next, the image-based visual servo control for controlling the position and posture of the robot arm 10A will be described with reference to Figs. 7 and 8. Fig. 7 is a block diagram showing the image-based visual servo control in a robot device. Fig. 8 is a flowchart showing the image-based visual servo control in a robot device. Note that the configuration of each part shown in the block diagram of Fig. 7 shows the functions configured by the CPU 201 executing the program PR. Therefore, these functions are actually achieved by the operation of the robot controller 200, the vision controller 220, the camera 120, etc.

[0044] Visual servo control is a method for controlling the position and posture of the robot arm 10A, and is a control method in which a change in the position of a target object is measured as visual information and used as feedback control information. Image-based visual servo is a type of visual servo control that extracts image features contained in an image of a target object on a current image and feeds back a difference between the image features on a target image. In the visual servo control of this embodiment, alignment or correction of the position of the robot hand 100 is performed with respect to the hole 402H of the workpiece 402 to be assembled, the hole 401H of the pin stand 401, the pin W placed on the pin stand 401, etc., which are the image-capture targets. Therefore, visual servo control can also be called alignment control or correction control.

[0045] 7 and 8, when the robot controller 200 starts the image-based visual servo control according to this embodiment, the vision controller 220 first reads the target image stored in the HDD 204 and reads out the target feature (S1). That is, the feature is extracted from the target image and set as the target feature. Next, the imaging unit 501 commands the camera 120 to capture an image of the hole 402H of the work 402 to be assembled or the hole 401H of the pin placement table 401, and acquires the image as a current image (S2). Next, the feature extraction unit 502 extracts the current feature from the acquired current image (S3).

[0046] Next, the joint angle correction amount conversion unit 503 first calculates a difference in feature amount (hereinafter referred to as a "feature amount difference") between the current feature amount acquired in step S3 and the target feature amount of the target image acquired in step S1 (S4). Then, the joint angle correction amount conversion unit 503 converts the feature amount difference into a correction amount (hereinafter referred to as a "joint angle correction amount") for the angle of each joint of the robot arm 10A (S5).

[0047] Next, the PID control unit 504 performs appropriate PID control on each joint angle correction amount, and calculates a control amount for feedback control (hereinafter referred to as a "feedback amount") (S6). Subsequently, the calculated feedback amount is transmitted to the servo control unit 15, and the motor 16 of each axis is driven to operate the robot arm 10A (S7).

[0048] Then, the CPU 201 judges whether the operated robot arm 10A has reached the target position, more specifically, whether the robot arm 10A has reached within the allowable error range of the target position (S8). Here, for example, if the feedback amount calculated in the above step S6 exceeds a predetermined range, it is judged that the position and posture of the robot arm 10A have not yet converged to the vicinity of the target position (No in S8). In this case, the process returns to the above step S2, and the process after the acquisition of the current image is executed again. In this way, by repeating the loop process from step S2 to step S8 at high speed, the robot arm 10A can be converged to the target position and posture. That is, the robot arm 10A can be made to reach the target positional relationship with the hole 402H of the work 402 to be assembled or the hole 401H of the pin placement table 401. Then, when the robot arm 10A reaches the target position (Yes in S8), the image-based visual servo control is terminated.

[0049] [Pin assembly control in the first embodiment] Next, the pin assembly control according to the first embodiment will be described with reference to Figs. 9 and 10. Fig. 9 is a flowchart showing the pin assembly control according to the first embodiment. Fig. 10(a) is a schematic diagram showing an image captured by the camera when the pin is gripped by the fingers. Fig. 10(b) is a schematic diagram showing an image captured by the camera when the robot hand is moved above a hole in a workpiece to be assembled. Fig. 10(c) is a schematic diagram showing an image captured by the camera when the fingers and the pin are moved out of the field of view. Fig. 10(d) is a schematic diagram showing an image captured by the camera when the robot hand is aligned with the hole in the workpiece to be assembled. Fig. 10(e) is a schematic diagram showing an image captured by the camera when the fingers and the pin are moved above a hole in a workpiece to be assembled.

[0050] In the pin assembly control according to the first embodiment, the robot device 10 performs a first operation of grasping (obtaining) the pin W installed in the hole 401H of the pin placement table 401, and a second operation of moving the pin W and inserting it into the hole 402H of the workpiece 402 to be assembled for assembly. By repeating the first operation and the second operation, the multiple pins W are inserted into the respective holes 402H of the workpiece 402 to be assembled and the completed workpiece 402 to be assembled is manufactured as an article.

[0051] 9, first, the CPU 201 drives the robot arm 10A to move the robot hand 100 so that the robot hand 100 is positioned roughly above the pin W placed on the pin stand 401 (S11). In other words, the CPU 201 drives the robot arm 10A so that the finger portion 140 is positioned roughly above the position where the pin W is gripped.

[0052] Next, the pin W on the pin placement table 401 is imaged (image capture step), that is, an image of the gripping position where the pin W is gripped by the finger portion 140 of the robot hand 100 is acquired (S12). Then, the above-mentioned visual servo control is executed, and the robot arm 10A is positioned so that the finger portion 140 is at the gripping position of the pin W (alignment step) (S13). That is, the acquired image of the pin W is set as a current image, and a feature amount difference between the current image and a target image of the pin W is calculated, and the robot arm 10A is controlled so that the feature amount difference converges. In other words, the image captured by the camera 120 is compared with the target image, and the amount of deviation from the optimal position for gripping the pin W is calculated. Then, the calculated amount of deviation is converted into posture data of the robot arm 10A to which the robot hand 100 is attached, and the posture data is used to position the pin W. As a result of the above, the robot arm 10A is positioned, and the finger portion 140 is positioned at the gripping position of the pin W.

[0053] Thereafter, in order to obtain the pin W in the robot hand 100, the first finger portion 141 and the second finger portion 142 of the robot hand 100 are driven by the motors 151 and 161 as described above, and the pin W is gripped by the fingers 140 (S14). In this state, as shown in Fig. 10(a), the pin W gripped by the first fingertip portion 141b of the first finger portion 141 and the second fingertip portion 142b of the second finger portion 142 are within the field of view of the camera 120, and these are in a state in which they are imaged.

[0054] Next, in order to insert and attach the pin W held by the fingers 140 into the hole 402H of the workpiece 402 to be assembled, the robot hand 100 is moved by the robot arm 10A while holding the pin W by the fingers 140 (S15). That is, the robot hand 100 is moved above the hole 402H (insertion position) of the workpiece 402 to be assembled. In this state, as shown in FIG. 10(b), the first fingertip 141b, the second fingertip 142b, and the held pin W are in the field of view of the camera 120, and any of them is imaged overlapping with the hole 402H of the workpiece 402 to be assembled. In other words, the overlapping range is a range in the field of view of the camera 120 where the first fingertip 141b, the second fingertip 142b, and at least a part of the held pin W overlap with the hole 402H of the workpiece 402 to be assembled. In this case, a part of any one of the first fingertip 141b, the second fingertip 142b, and the gripped pin W, or a part of multiple parts of them, are located inside the overlapping range. Therefore, the camera 120 is blocked by any one of the first fingertip 141b, the second fingertip 142b, and the gripped pin W, and is unable to capture an image of the hole 402H of the workpiece 402 to be assembled. In this state, as shown in FIG. 10(b), the visual servo control of the robot arm 10A based on the image of the hole 402H is not completed, so that the pin W gripped by the fingers 140 and the hole 402H are likely to be misaligned.

[0055] Therefore, before the camera 120 captures an image of the hole 402H (insertion position) of the workpiece 402 to be assembled, the first fingertip 141b of the first finger portion 141, the second fingertip 142b of the second finger portion 142, and the pin W are moved so as to be outside the field of view of the camera 120 (S16). That is, the first finger portion 141, the second finger portion 142, and the pin W are retreated so as to be outside the field of view of the camera 120 (retraction process) (see FIG. 5). Specifically, the motors 151 and 161 are driven to slide the first finger portion 141 and the second finger portion 142 while gripping the pin W, and the first finger portion 141, the second finger portion 142, and the pin W are moved to positions where they do not overlap with the hole 402H of the workpiece 402 to be assembled. At this time, the position and posture of the robot arm 10A are not changed, that is, the position and posture of the robot hand 100 remain unchanged, and only the first finger portion 141 and the second finger portion 142 are moved. In this state, as shown in Fig. 10(c), the first finger portion 141, the second finger portion 142, and the pin W are withdrawn from the field of view of the camera 120, and the hole portion 402H of the workpiece 402 to be assembled can be directly imaged.

[0056] In this embodiment, the first finger portion 141, the second finger portion 142, and the pin W are described as moving outside the field of view of the camera 120, but it is sufficient if they do not block the imaging of the hole portion 402H of the workpiece 402 by the camera 120. Therefore, the first finger portion 141, the second finger portion 142, and the pin W may be within the field of view of the camera 120 as long as they are moved outside the above-mentioned overlapping range.

[0057] In this way, when the first finger portion 141, the second finger portion 142, and the pin W are moved so as to be outside the field of view of the camera 120, the camera 120 captures an image of the hole 402H of the workpiece 402 to be assembled, which is the insertion position for inserting the pin W (image capture step) (S17). Then, visual servo control is executed again, and the robot arm 10A is positioned so that the camera 120 is in a correct position relative to the hole 402H of the workpiece 402 to be assembled, which is the insertion position (positioning step) (S18). That is, the acquired image of the hole 402H is taken as a current image, and a feature amount difference between the current image and the target image of the hole 402H is calculated, and the robot arm 10A is controlled so that the feature amount difference converges. In other words, the image captured by the camera 120 is compared with the target image, and the amount of deviation from the optimal position for inserting the pin W into the hole 402H is calculated. The calculated amount of deviation is converted into posture data of the robot arm 10A to which the robot hand 100 is attached, and the posture data is used to position the robot arm 10A relative to the hole 402H. As described above, the robot arm 10A is positioned, and the robot hand 100 is positioned relative to the hole 402H. In this state, as shown in FIG. 10(d), the pin W held by the first fingertip 141b of the first finger 141 and the second fingertip 142b of the second finger 142 are outside the field of view of the camera 120. Therefore, the hole 402H of the workpiece 402 to be assembled is imaged, for example, at approximately the center of the field of view of the camera 120.

[0058] Next, after the camera 120 has completed imaging of the hole 402H (insertion position) of the workpiece 402 to be assembled, the finger 140 is returned to its original position in the robot hand 100 before inserting the pin W into the hole 402H of the workpiece 402 to be assembled (returning step). That is, the first finger 141 and the second finger 142 are moved so that at least a part of the pin W held by the first finger 141 and the second finger 142 is positioned to overlap the hole 402H of the workpiece 402 to be assembled within the field of view of the camera 120 (S19). Specifically, the motor 151 and the motor 161 are driven to slide the first finger 141 and the second finger 142 while holding the pin W, and the pin W moves to a position where it overlaps the hole 402H of the workpiece 402 to be assembled (see FIG. 4). In this case, the position and posture of the robot arm 10A are not changed, that is, the position and posture of the robot hand 100 are kept unchanged, and only the first finger portion 141 and the second finger portion 142 are moved. In this state, as shown in Fig. 10(e), within the field of view of the camera 120, the first fingertip portion 141b of the first finger portion 141, the second fingertip portion 142b of the second finger portion 142, and the pin W return to their original positions before being retracted. In addition, since the position and posture of the robot hand 100 are positioned correctly with respect to the hole portion 402H by visual servo control, the pin W is moved with high precision to a position overlapping approximately directly above the hole portion 402H of the workpiece 402 to be assembled.

[0059] Finally, the robot arm 10A is operated under torque control, and the pin W held by the finger portion 140 of the robot hand 100 is inserted into the hole portion 402H of the workpiece 402 to be assembled (S20). That is, the robot hand 100 is moved by driving the robot arm 10A, and the pin W held by the finger portion 140 of the robot hand 100 is brought into contact with the hole portion 402H of the workpiece 402 to be assembled. Then, the torque sensor 18 of each joint of the robot arm 10A detects the reaction force caused by the pin W contacting the hole portion 402H, and the motor 16 is torque-controlled in a direction in which the reaction force is reduced and a pushing force that moves the pin W downward is generated. In this way, the pin W can be assembled by following the hole portion 402H and the pushing force can be managed by torque control, and the pin W can be inserted into the hole portion 402H without receiving a large resistance. Furthermore, by separating the first finger portion 141 and the second finger portion 142, the finger portion 140 releases the grip of the pin W, whereby the pin W is assembled into the hole portion 402H of the workpiece 402 to be assembled. As described above, the hole portion 402H is a position where the pin W is brought into contact, a position where the pin W is inserted and assembled, and a position where the pin W is installed, and therefore can also be called a contact position, a work position, or an installation position.

[0060] Then, the CPU 201 judges whether or not the assembly of all the pins W to the workpiece 402 has been completed as an operation (S21). For example, the number of pins W to be assembled to the workpiece 402 is stored in advance in the HDD 204 of the robot controller 200, and the judgment is made based on how many times the assembly operation has been completed. If the assembly of all the pins W has not been completed (No in S21), the process returns to step S11 and repeats steps up to S20, that is, the operation of the next pin W is performed.

[0061] Although there is a trade-off between processing performance and calculation time, in the first visual servo control (S13, S18), multiple recognition processes are performed in parallel even for holes with different shapes and diameters. Then, the center positions and diameters of the hole 401H in the pin placement table 401 and the hole 402H in the workpiece 402 are extracted as feature amounts. By doing so, positioning by visual servo control from the second time onwards is possible, so the target image used in the previous visual servo control may be read out and set as the target feature amount.

[0062] The pin assembly control is performed as described above, and when assembly of all the pins W into the workpiece 402 is completed (Yes in S21), the pin assembly control of this embodiment ends.

[0063] [Summary of the first embodiment] As described above, according to the first embodiment, when the camera 120 photographs the workpiece 402 to be assembled and acquires a current image, the finger portions 140 and the pins W move out of the overlapping range that overlaps with the holes 402H of the workpiece 402 to be assembled in the field of view of the camera 120. This allows the contact position or the working position to be imaged without being obstructed by the pins W while the pins W, which are the workpieces, are still being gripped and supported by the fingers 140, that is, the current image can be acquired without being obstructed by the pins W. This allows the robot hand 100 to be accurately aligned with the holes 402H, and improves the accuracy of the assembly position (contact position, installation position) in the assembly operation (contact operation, installation operation) of the pins W.

[0064] Furthermore, before the camera 120 captures an image of the hole 402H, the finger 140 and the pin W are moved from inside the overlapping range in the field of view of the camera 120 to outside the overlapping range. This allows the hole 402H to be captured without the finger 140 and the pin W blocking the hole 402H. Furthermore, when the finger 140 and the pin W are moved, the robot arm 10A is not driven, that is, the base 101 of the robot hand 100 does not change its position and posture, and only the finger 140 moves. This allows the position and posture of the robot hand 100 to be corrected with high accuracy, for example, by visual servo control.

[0065] Furthermore, from a state in which the finger portion 140 and the pin W held by the finger portion 140 are outside the overlapping range, the finger portion 140 and the pin W held by the finger portion 140 are moved so that the pin W overlaps with the hole portion 402H in the field of view of the camera 120 before the assembling operation (task) is performed by the finger portion 140. This allows the pin W to be aligned to a position where it is to be assembled into the hole portion 402H. Similarly, when the finger portion 140 and the pin W held by the robot arm 10A are moved, the robot arm 10A is not driven, that is, the base portion 101 of the robot hand 100 does not change its position and posture, and only the finger portion 140 is moved. This allows the pin W to be aligned with the hole portion 402H with high precision.

[0066] In the robot hand 100, the first root portion 141a of the first finger portion 141 and the second root portion 142a of the second finger portion 142 are driven by the first driving unit 150 and the second driving unit 1500 so as to slide along the first guide portion 170 and the second guide portion 180. This makes it possible to move and retract only the finger portion 140 without changing the position and posture of the base portion 101 of the robot hand 100, and also makes it easy to return the retracted finger portion 140 to align it with the hole portion 402H. In addition, since the first finger portion 141 and the second finger portion 142 can be driven independently, the pin W can be gripped or released.

[0067] <Second embodiment> Next, a second embodiment, which is a partial modification of the first embodiment, will be described with reference to Figs. 11 and 12. Fig. 11 is a flow chart showing pin assembly control according to the second embodiment. Fig. 12(a) is a schematic diagram showing an image captured by the camera when the pin is gripped by the fingers. Fig. 12(b) is a schematic diagram showing an image captured by the camera when the robot hand is moved above a hole in a workpiece to be assembled with the fingers and pin moved out of the field of view. Fig. 12(c) is a schematic diagram showing an image captured by the camera when the robot hand is aligned with respect to the hole in the workpiece to be assembled. Fig. 12(d) is a schematic diagram showing an image captured by the camera when the fingers and pin are moved above a hole in the workpiece to be assembled.

[0068] In the second embodiment, the order of steps S15 and S16 in the pin assembly control is reversed compared to the first embodiment (see Figs. 9 and 11). That is, after the pin W on the pin stand 401 is gripped by the finger portions 140 (S14), the robot hand 100 is moved above the hole portion 402H of the workpiece 402 to be assembled (S15) while (or after) moving the finger portions 140 and the gripped pin W (S16).

[0069] [Pin assembly control in the second embodiment] 11, first, the CPU 201 drives the robot arm 10A to move the robot hand 100 so that the robot hand 100 is roughly positioned above the pin W placed on the pin holder 401 (S11). Next, the pin W on the pin holder 401 is imaged, that is, an image of the gripping position where the pin W is gripped by the finger portion 140 of the robot hand 100 is acquired (imaging step) (S12). Then, the above-mentioned visual servo control is executed, and the robot arm 10A is positioned so that the finger portion 140 is at the gripping position of the pin W (alignment step) (S13). After that, the first finger portion 141 and the second finger portion 142 of the robot hand 100 are driven by the motors 151 and 161, and the pin W is gripped by the finger portion 140 (S14). In this state, as shown in FIG. 12(a), the first fingertip portion 141b of the first finger portion 141, the second fingertip portion 142b of the second finger portion 142, and the pin W being held are within the field of view of the camera 120, and these are in a state to be imaged.

[0070] Here, before the camera 120 captures the image of the hole 402H of the workpiece 402 to be assembled, the first fingertip 141b of the first finger 141, the second fingertip 142b of the second finger 142, and the pin W are moved to be outside the field of view of the camera 120 (evacuation process) (S16). In this movement (evacuation), a predetermined position is registered and stored so that the finger 140 and the pin W do not overlap the hole 402H when the hole 402H of the workpiece 402 to be assembled is captured by the camera 120 in the following step S17, and the finger 140 and the pin W are moved to the predetermined position. This predetermined position may be stored in a storage area such as the HDD 204. In this state, as shown in FIG. 12(b), the first finger 141, the second finger 142, and the pin W are evacuated from the field of view of the camera 120, and the hole 402H of the workpiece 402 to be assembled can be directly captured.

[0071] Next, the robot hand 100 is moved by the robot arm 10A with the pin W being gripped by the finger portions 140 (S15). That is, the robot hand 100 is moved above the hole 402H (insertion position) of the workpiece 402 to be assembled. Note that the movement of the fingers 140 gripping the pin W by the robot hand 100 (S15) and the movement of the robot hand 100 by the robot arm 10A (S16) are executed simultaneously in parallel in this embodiment. However, the movement of the fingers 140 gripping the pin W by the robot hand 100 may be followed by the movement of the robot hand 100 by the robot arm 10A.

[0072] Next, the camera 120 captures an image of the hole 402H of the workpiece 402 to be assembled, which is an insertion position for inserting the pin W (image capture step) (S17). Then, visual servo control is executed again, and the robot arm 10A is positioned so that the camera 120 is correctly positioned with respect to the hole 402H of the workpiece 402 to be assembled, which is the insertion position (positioning step) (S18). In this state, as shown in FIG. 12(c), the pin W held by the first fingertip 141b of the first finger portion 141 and the second fingertip 142b of the second finger portion 142 are outside the field of view of the camera 120. Therefore, the hole 402H of the workpiece 402 to be assembled is captured, for example, at approximately the center of the field of view of the camera 120.

[0073] Next, after the camera 120 has completed imaging of the hole 402H (insertion position) of the workpiece 402 to be assembled, the fingers 140 are returned to their original positions in the robot hand 100 before inserting the pin W into the hole 402H of the workpiece 402 to be assembled (returning step). That is, the first finger 141 and the second finger 142 are moved so that the pin W held by the first finger 141 and the second finger 142 is positioned so as to overlap the hole 402H of the workpiece 402 to be assembled within the field of view of the camera 120 (S19). In this state, as shown in FIG. 12(d), the first fingertip 141b of the first finger 141, the second fingertip 142b of the second finger 142, and the pin W are returned to their original positions before being retracted within the field of view of the camera 120. Furthermore, since the position and posture of the robot hand 100 is correctly positioned with respect to the hole 402H by visual servo control, the pin W is moved with high precision to a position overlapping approximately directly above the hole 402H of the workpiece 402 to be assembled.

[0074] Finally, the robot arm 10A is operated under torque control, and the pin W held by the fingers 140 of the robot hand 100 is inserted into the hole 402H of the workpiece 402 to be assembled (S20). As a result, the pin W is inserted along the hole 402H without receiving significant resistance. In addition, the first finger 141 and the second finger 142 are released from the grip of the pin W by the fingers 140, and the pin W is assembled into the hole 402H of the workpiece 402 to be assembled.

[0075] Then, the CPU 201 determines whether or not the assembly of all the pins W to the workpiece 402 as the operation has been completed (S21), and if not completed (No in S21), returns to step S11 and repeats the process up to step S20, that is, performs the operation of the next pin W. Also, when the assembly of all the pins W to the workpiece 402 as the operation has been completed (Yes in S21), the pin assembly control of this embodiment ends.

[0076] [Summary of the second embodiment] In the second embodiment described above, when the camera 120 photographs the workpiece 402 to be assembled and acquires a current image, the finger portions 140 and the pins W move out of the overlapping range where they overlap with the holes 402H of the workpiece 402 to be assembled and retract in the field of view of the camera 120. This allows the contact position or working position to be imaged without being obstructed by the pins W while the finger portions 140 are still gripping and supporting the pins W, which are the workpieces, and thus allows the current image to be acquired without being obstructed by the pins W.

[0077] In the second embodiment, the robot hand 100 can be moved (retracted) to above the hole 402H of the workpiece 402 to be assembled (S15) while simultaneously moving (retracting) the finger portion 140 and the gripped pin W (S16). This makes it possible to shorten the operation time (work time) compared to the first embodiment in which the finger portion 140 and the pin W are once moved to above the hole 402H of the workpiece 402 to be assembled and then retracted.

[0078] Other configurations, operations, and effects of the second embodiment are similar to those of the first embodiment, and therefore description thereof will be omitted.

[0079] <Third embodiment> Next, a third embodiment, which is a partial modification of the first and second embodiments, will be described with reference to Figs. 13, 14, and 15. Fig. 13 is a schematic diagram showing the structure of a camera and an illumination unit according to the third embodiment. Fig. 14(a) is a diagram showing an image obtained when a hole in a workpiece to be assembled is imaged by a general camera. Fig. 14(b) is a diagram showing an image obtained when a hole in a workpiece to be assembled is imaged by a camera according to the third embodiment. Fig. 15(a) is a diagram showing an image obtained when a pin holder with a pin placed in a hole is imaged by a general camera. Fig. 15(b) is a diagram showing an image obtained when a pin holder with a pin placed in a hole is imaged by a camera according to the third embodiment.

[0080] [Camera and lighting configuration] In the above-described first embodiment, the camera 120 is equipped with a general lens, but in the present third embodiment, the camera 120 is equipped with a telecentric lens 122, which is a telecentric optical system. The camera 120 is also equipped with a coaxial epi-illumination 130 as an illumination unit that is attached to the telecentric lens 122 and irradiates illumination light in the direction of the optical axis AX1 (optical axis direction).

[0081] 13, the camera 120 includes a sensor unit 121 which is an image sensor that scans and acquires an image. The camera 120 also includes a telecentric lens 122 that is connected to the sensor unit 121 and can capture an image from the direction of the optical axis AX1 into the sensor unit 121. The camera 120 also includes a coaxial epi-illumination 130 as a light source that irradiates illumination light coaxially with the optical axis AX1 via a half mirror (not shown) or the like that is provided on the telecentric lens 122.

[0082] The telecentric lens 122 is a lens whose principal ray is parallel to the optical axis AX1, and has an angle of view close to 0 degrees and small distortion, making it possible to accurately capture the dimensions and position of the subject imaged by the sensor unit 121. In addition, the coaxial epi-illumination 130, which irradiates in the same axial direction as the optical axis AX1 of the camera 120, irradiates the irradiated light perpendicularly to, for example, the surface 402s of the workpiece 402 to be assembled. This makes it possible to image the surface 402s with the sensor unit 121 so that it is brighter (whiter) than the hole 402H. Therefore, for example, when the surface of the subject has irregularities, it is possible to easily detect the irregularities of the subject.

[0083] [When imaging the workpiece 402] Here, a case where the workpiece 402 to be assembled is imaged by the camera 120 will be described. As shown in FIG. 13, the workpiece 402 to be assembled has a surface 402s as a flat portion formed in a flat shape as described above, and a hole portion 402H is formed in the surface 402s in a hole-like shape. The hole portion 402H has a hole 402h and a chamfered portion 402m as an inclined portion chamfered at the outer edge of the opening. The chamfered portion 402m is formed as an inclined surface inclined at 45 degrees with respect to the surface 402s, for example. Note that the angle of the chamfered portion 402m is preferably inclined at 30 degrees or more with respect to the surface 402s, but may be any angle.

[0084] When imaging the workpiece 402, the coaxial epi-illumination 130 irradiates the hole 402H of the workpiece 402 with illumination light (illumination step), and the camera 120 images the hole 402H of the workpiece 402 in this state (imaging step).

[0085] Here, as shown in FIG. 14(a), for example, when a hole 402H of a workpiece 402 is imaged by a general camera, the hole 402h is darker than the surface 402s. However, the difference in brightness is small, and the inner peripheral edge 402a of the chamfered portion 402m and the outer peripheral edge 402b of the chamfered portion 402m are like double circles, making it difficult to distinguish their feature amounts. In addition, the difference in brightness between the outer peripheral edge 402b and the surface 402s is also small, making it difficult to distinguish their feature amounts. Therefore, the position of the hole 402H in the image is easily misrecognized, and even if visual servo control is performed in this state as described above, the control amount (correction amount) is not stable, and the positioning of the robot hand 100 and the hole 402H is not highly accurate.

[0086] On the other hand, as shown in FIG. 13, when the hole 402H of the work 402 is imaged by the camera 120 according to the third embodiment, the posture of the robot hand 100 is controlled so that the optical axis AX1 of the camera 120 is perpendicular to the surface 402s of the work 402. Then, illumination light is irradiated from the coaxial epi-illumination 130 toward the optical axis AX1 (irradiation process). Then, the illumination light reflected by the surface 402s is reflected in the direction of the optical axis AX2 parallel to the optical axis AX1, and travels toward the sensor unit 121 via the telecentric lens 122. Moreover, the illumination light irradiated to the hole 402h is not reflected (or is diffusely reflected at the bottom surface and absorbed). Then, the illumination light reflected by the chamfered portion 402m is reflected toward the direction of the optical axis AX3, which is different from the camera 120 with respect to the optical axis AX1.

[0087] Therefore, as shown in FIG. 14(b), when the hole 402H of the workpiece 402 is imaged by the camera 120 according to the third embodiment, the area of ​​the outer surface 402s of the hole 402H (the surface perpendicular to the optical axis AX1 of the camera) is imaged bright (white) (image-capturing process). Conversely, the area inside the hole 402H including the chamfered portion 402m is imaged dark (black). That is, the difference between light and dark is larger than when an image is captured by a general camera with the outer periphery 402b of the chamfered portion 402m as a boundary, and the outer periphery 402b of the chamfered portion 402m can be detected with high accuracy as a feature. Therefore, the position of the hole 402H in the image is recognized accurately, the control amount (correction amount) when visual servo control is performed is stabilized, and the positioning of the robot hand 100 and the hole 402H is highly accurate.

[0088] [When imaging the pin holder 401] Here, a case where the pin stand 401 is imaged by the camera 120 will be described. The hole 401H of the pin stand 401 has the same configuration as the hole 402H of the work 402, and a chamfered portion 401m as an inclined portion is formed on the outer edge of the opening of the hole 401h formed as a hole in the surface 401s as a flat surface portion formed in a flat shape. The pin W is inserted and placed in the hole 401h, which is inside the inner peripheral edge 401a of the chamfered portion 401m.

[0089] When imaging the pin stand 401, the coaxial epi-illumination 130 irradiates the hole 401H of the pin stand 401 with illumination light (illumination step), and the camera 120 images the hole 401H of the pin stand 401 in this state (imaging step).

[0090] Here, depending on the positional relationship between the mounting position of the camera 120 on the robot hand 100 and the positions of the fingers 140 of the robot hand 100, the position where the pin W is grasped may not necessarily be the center of the angle of view of the camera 120. For this reason, as shown in Fig. 15(a), when an image of a hole 401H, which is a pin hole in a pin stand 401, is captured using a general camera, the side surface Ww of the pin W, which is tall, is visible with a normal lens. That is, in this image, the head Ws of the pin W overlaps with the outer circumferential edge 401b of the chamfered portion 401m.

[0091] 15(b), when the hole 401H of the pin stand 401 is imaged by the camera 120 according to the third embodiment, the side Ww of the pin W is not visible in the image captured by the sensor unit 121 via the telecentric lens 122. That is, since the telecentric lens 122 is a lens in which the chief ray is parallel to the optical axis, even if the position of the hole 401H is not at the center of the angle of view of the camera 120, the side Ww of the pin W is not visible, and the outer periphery 402b of the chamfered portion 401m is visible in its entirety. Note that, for example, although the pin W is made of metal, the head Ws of the pin W does not have a polished surface, so that the illumination light from the coaxial epi-illumination 130 is diffused and the head Ws is imaged dark (black) even when imaged.

[0092] In this way, even when the hole 401H of the pin placement table 401 is imaged by the camera 120 according to the third embodiment, the area outside the hole 401H is imaged bright (white). Conversely, the area inside the hole 401H including the chamfered portion 401m is imaged dark (black). That is, the difference between light and dark is larger with the outer periphery 401b of the chamfered portion 401m as the boundary than when an image is taken by a general camera, and the outer periphery 401b of the chamfered portion 401m can be detected with high accuracy as a feature. Therefore, the position of the hole 402H in the image is recognized accurately, the control amount (correction amount) when visual servo control is performed is stabilized, and the positioning of the robot hand 100 and the hole 402H is highly accurate.

[0093] [Common target image] Incidentally, in the case of a general camera, for example, when there is a pin W as shown in FIG. 15(a), the shape acquired as a feature is not circular, so it is necessary to acquire images in both cases where there is a pin W in the hole 401H and where there is not. However, the camera 120 according to the third embodiment uses a telecentric lens 122. Therefore, as shown in FIG. 15(b), whether there is a pin W in the hole 401H or not, the hole 401H (outer periphery 402b) is imaged as a substantially circular shape, that is, there is no effect on image recognition. This makes it possible to use the same target image as a target image for use in performing the visual servo control (see S13 in FIG. 8 and FIG. 9, and S13 in FIG. 11) for gripping the pin W described above, regardless of which of the multiple holes 401H is used.

[0094] Furthermore, the target image used when performing visual servo control (see S18 in FIG. 8 and FIG. 9, and S18 in FIG. 11) to assemble the pin W into the hole 402H of the work 402 to be assembled is also a substantially circular image. Therefore, the same target image can be used in common between the target image used in the visual servo control to grip the pin W and the target image used in the visual servo control to assemble the pin W into the hole 402H of the work 402 to be assembled. In particular, even if the position of the target hole changes or the position or hole diameter moves by several millimeters, the feature amount of the image is common, so that the pin W can be gripped and assembled into a plurality of holes using the same target image (same feature amount). Note that the same target image does not necessarily have to be completely identical, and does not necessarily have to be completely identical in terms of size, etc., and as long as the shape is the same, the target image can be enlarged or reduced for use. In other words, the same target image is a target image with the same shape of the feature part, and can also be said to be the same target image except for the size.

[0095] In the above description, the same target image is used, but in visual servo control, a feature amount is extracted from the target image and set as the target feature amount (see S1 in FIG. 7 and FIG. 8). Then, this target feature amount is stored as data in, for example, the HDD 204 of the robot controller 200 and used. This process of extracting the target feature amount from the target image does not need to be performed every time visual servo control is performed for a plurality of holes 401H. In other words, the target feature amount extracted from the target image in the first visual servo control (see S13 in FIG. 9 and S13 in FIG. 11) and stored as data in the HDD 204 or the like can be repeatedly read and used in the next and subsequent visual servo controls. Furthermore, the target feature amount used when performing visual servo control (see S18 in FIG. 9 and S18 in FIG. 11) to install the pin W into the hole 402H of the workpiece 402 to be installed can also be read and used as the target feature amount set in the first visual servo control.

[0096] [Summary of the third embodiment] As described above, in the third embodiment, when the hole 402H of the work 402 to be assembled and the hole 401H of the pin placement table 401 are image-recognized, the surface 402s and the surface 401s are photographed so that they are bright and the hole 402H and the hole 401H are photographed so that they are dark. That is, the coaxial epi-illumination 130 illuminates the surface 402s or the surface 401s so that the optical axis of the reflected light is directed toward the camera 120. At the same time, the coaxial epi-illumination 130 illuminates the hole 402H or the hole 401H so that the optical axis of the reflected light is directed toward a direction different from the camera 120. In other words, the camera 120 captures the image of the surface 402s or the surface 401s while illuminating it with the coaxial epi-illumination 130, with the optical axis AX1 of the telecentric lens 122 being perpendicular to the surface 402s or the surface 401s. This makes it possible to detect the outer periphery 402b of the chamfered portion 402m or the outer periphery 401b of the chamfered portion 401m with high accuracy, that is, to detect the position of the hole 402H or the hole 401H in the captured (acquired) image with high accuracy. Therefore, the control amount (correction amount) when performing visual servo control is stabilized, and the positioning of the robot hand 100 and the hole 402H or the hole 401H can be performed with high accuracy.

[0097] Further, the positional deviation between the robot hand 100 and the hole 402H of the work 402 to be assembled or the hole 401H of the pin stand 401 is calculated from the current image captured by the camera 120 and the target image, and the position of the robot hand 100 is corrected so as to eliminate the positional deviation. That is, the position of the robot hand 100 is corrected by visual servo control. Since the pin stand 401 has a plurality of holes 401H, the same target image is used in the visual servo control executed for each of the holes 401H of the pin stand 401. Furthermore, the target image used for the holes 401H of the pin stand 401 can be used in the visual servo control executed for each of the holes 402H of the work 402 to be assembled. Since the target image can be shared in this way, it is sufficient to perform a teaching operation for teaching the position and posture of the robot hand 100 using the shared target image. That is, for example, a teaching operation of setting a target image for each hole 402H or each hole 401H and teaching the position and posture of the robot hand 100 for each is not required, and the teaching operation can be simplified. Furthermore, for example, the same target image can be used for each of the holes 402H in the workpiece 402 to be assembled or each of the holes 401H in the pin placement table 401 even if the hole diameters are different. And since the image and feature amount of the hole do not change even if the position of the hole is different, the same target image can be used, and the time for the teaching operation can be significantly reduced.

[0098] Other configurations, operations, and effects of the third embodiment are similar to those of the first and second embodiments, and therefore description thereof will be omitted.

[0099] <Fourth embodiment> Next, a fourth embodiment, which is a partial modification of the third embodiment, will be described with reference to Figs. 16 and 17. Fig. 16 is a flow chart showing posture correction control of a robot arm. Fig. 17(a) is a schematic diagram showing a robot arm before posture correction. Fig. 17(b) is a schematic diagram showing a robot arm after posture correction.

[0100] In the above-mentioned pin assembly control, for example, when the pin stand 401 is imaged by the camera 120, when the workpiece 402 is imaged by the camera 120, etc., it is preferable that the robot hand 100 faces the pin stand 401 and the workpiece 402 with high accuracy. Similarly, in the above-mentioned pin assembly control, for example, when the pin W is gripped by the finger portion 140, when the pin W is inserted into the hole portion 402H by the finger portion 140, etc., it is preferable that the robot hand 100 faces the pin stand 401 and the workpiece 402 with high accuracy. Furthermore, not only in the above-mentioned pin assembly control, but also in a teaching operation for teaching the operation of the robot arm 10A, for example, there are cases where it is preferable that the posture of the robot hand 100 faces the pin stand 401 and the workpiece 402 with high accuracy. Therefore, in the fourth embodiment, a posture correction control for correcting the posture of the robot hand 100 to face the target object will be described. In addition, facing directly refers to a state in which the direction of the central axis of the flange portion 11 of the robot arm 10A (the central axis of the robot hand 100, the optical axis AX1 of the camera 120) is perpendicular to the surface of the target object (the surface 401s of the pin placement table 401 or the surface 401s of the workpiece 402 to be assembled).

[0101] [Robot arm posture correction control] First, when performing posture correction control of the robot arm 10A according to the fourth embodiment, a calibration plate 701 having a surface 701s that reflects illumination light is placed on a stand 700 as shown in Fig. 17(a) and Fig. 17(b). Note that if a workpiece has a surface that can reflect illumination light, such as a workpiece with metallic luster, such a workpiece may be placed without using the calibration plate 701. In particular, the surface 401s of the pin placement table 401 and the surface 402s of the workpiece 702 to be assembled are lusterful and reflect illumination light, and therefore may be used.

[0102] Furthermore, the camera 120 provided in the robot hand 100 according to the fourth embodiment will be described as having the telecentric lens 122 and the coaxial epi-illumination 130 described in the third embodiment. However, a general camera may be used as the camera 120. Also, in Figs. 17(a) and 17(b), only the robot arm 10A is shown as the robot system 1. However, as shown in Figs. 1 and 2, the robot system 1 includes a robot controller 200, a vision controller 220, etc., and is configured to be connectable to a teaching pendant 300.

[0103] As shown in Fig. 16, the CPU 201 of the robot controller 200 first operates the robot arm 10A, and moves the robot hand 100 so that it is positioned above the calibration plate 701 (reflector) as shown in Fig. 17(a) (S31). In this case, the robot hand 100 (flange portion 11, which is the tip of the robot arm 10A) may be tilted due to positioning error or the like, and may not be facing the calibration plate 701 (reflector). Therefore, the following control is performed.

[0104] Next, the CPU 201 acquires an image of the calibration plate 701 at a position above the camera 120 (S32). The CPU 201 then calculates a luminance value from the entire acquired image (S33). By calculating the luminance value, it is possible to know the degree to which the light irradiated from the optical axis AX1 (see FIG. 13) of the camera 120 is specularly reflected and returned, and therefore it is possible to measure the degree to which the robot hand 100 (flange portion 11 of the robot arm 10A) and the calibration plate 701 are directly facing each other.

[0105] Next, the CPU 201 tilts the flange portion 11 of the robot arm 10A (i.e., the robot hand 100) in either the wx direction or the wy direction by, for example, a predetermined amount. Note that, in this case, the horizontal direction of the image is the X axis, and the vertical direction is the Y axis, and the rotational movement around the X axis is the wx direction, and the rotational movement around the Y axis is the wy direction. The rotational direction when tilting may be either clockwise or counterclockwise, that is, either rotational direction may be used.

[0106] Next, the CPU 201 again acquires an image of the calibration plate 701 by the camera 120 with the flange 11 of the robot arm 10A tilted from the state in which the previous image was acquired (S35). Then, the luminance value is calculated from the entire image acquired with the flange 11 of the robot arm 10A tilted (S36). Then, the luminance gradient is calculated from the difference in the luminance values ​​acquired before and after the flange 11 of the robot arm 10A is tilted, and it is determined whether the luminance gradient is a positive number or not (S37). When the luminance gradient is a positive number, the luminance value of the entire image increases, that is, the image becomes brighter. Therefore, if the difference in the luminance values ​​before and after the flange 11 of the robot arm 10A is tilted is a positive number, it is understood that tilting in that direction will result in closer to a straight-on position. Therefore, if the trajectory gradient is a positive number (Yes in S37), the posture of the flange 11 (hand tip) of the robot arm 10A is operated by visual servo control using the luminance gradient as a feature (S39). Thereafter, the processes from step S35 to step S39 are repeated until the luminance gradient is no longer a positive number but becomes a negative number, that is, until the luminance of the acquired image becomes substantially maximum.

[0107] After that, when the trajectory gradient is no longer a positive number (No in S37), the CPU 201 judges whether the processing up to that point is the first loop (S38). For example, in step S34, the flange portion 11 of the robot arm 10A may have been tilted in a direction in which the luminance decreases. Therefore, if the luminance gradient is a negative number but it is the first loop (Yes in S38), a correction is performed to tilt it in the opposite direction by visual servo control (S39). Then, if the processing up to that point is not the first loop (No in S38), it is judged whether corrections in both the wx direction and the wy direction have been completed (S40). If corrections in both the wx direction and the wy direction have not been completed (No in S40), the process returns to step S35 again to perform corrections of the posture directions that have not been completed. Then, if corrections in both the wx direction and the wy direction have been completed (Yes in S40), the posture correction control of the robot arm is terminated. In this way, posture correction is performed until the brightness gradient becomes negative, and the brightness that can be obtained from the image becomes almost maximum, so that the flange portion 11 of the robot arm 10A (i.e., the robot hand 100) faces directly toward the calibration plate 701, as shown in FIG. 17(b).

[0108] In the fourth embodiment, the posture correction control of the robot arm is described as being performed by visual servo control, but is not limited thereto. For example, a UI (user interface) that displays a luminance value may be mounted on the teaching pendant 300, the luminance may be displayed on the display unit 300A, and the worker may manually move the posture of the robot arm 10A while viewing the information, thereby correcting the posture. This posture correction method is particularly effective when teaching the robot arm 10A using the teaching pendant 300. Furthermore, even if the display unit 300A is displayed in this manner, the luminance value may be fed back to the robot controller 200, and the robot controller 200 may execute part or all of the posture correction control process. In addition, the display unit 300A of the teaching pendant 300 is used as an example of a display unit that displays the luminance, but the present invention is not limited thereto, and may be connected to another display unit such as a monitor and displayed thereon.

[0109] In particular, when the posture is corrected when teaching the robot arm 10A, the images and posture information acquired during the correction are stored in the HDD 204 or the like after the posture correction is executed. Then, the images and posture information may be called up before the pin assembly control is executed, and the movement control may be performed so that the robot hand 100 faces the pin placement table 401 or the workpiece 402 to be assembled directly.

[0110] Furthermore, when general lighting and lenses are used for the camera, the accuracy of the posture correction described above decreases, but correction in the depth direction becomes possible. For this reason, a configuration may be used in which correction is performed by combining two types of cameras, a general camera and a camera 120 having a telecentric lens 122.

[0111] [Summary of the fourth embodiment] According to the posture correction control of the robot arm in the fourth embodiment described above, the robot hand 100 and the target work (the pin placement table 401 and the work 402 to be assembled) can be made to face each other. Therefore, it is possible to actually perform precise fitting, for example, of about several micrometers, between the pin W and the finger portion 140 or between the pin W and the hole portion 402H. Furthermore, it is possible to easily perform a teaching work for performing such precise fitting. Furthermore, by performing the posture correction control of the robot arm before or during the execution of the above-mentioned pin assembly control, even if the robot hand 100 is tilted due to a change in the robot operation or environment, it is possible to correct it, and the above-mentioned precise fitting can be reliably performed. Furthermore, by performing the posture correction control of the robot arm, the brightness of the image captured by the pin assembly control is maximized, and the accuracy of image recognition can be improved. Furthermore, by using multiple cameras, not only the accuracy of position recognition in the image but also the accuracy of recognition in the depth direction can be improved.

[0112] <Possibilities for other embodiments> In the first to fourth embodiments described above, the hole 401H of the pin stand 401 and the hole 402H of the workpiece 402, which are the objects to be imaged (working position, contact position), are photographed in a state in which the finger portion 140 and the pin W gripped by the finger portion 140 are retracted. At this time, the finger portion 140 is moved to retract the position of the pin W, and the finger portion 140 is moved to return the position of the pin W. However, the present invention is not limited to this, and the robot arm 10A may be driven to retract or return the pin W together with the robot hand 100. In other words, if the positional relationship between the pin W and the working position (contact position) is known, the pin W can be retracted or returned even if the robot arm 10A is moved by that distance.

[0113] In the first to fourth embodiments, the camera 120 is attached to and supported by the base 101 of the robot hand 100, but the present invention is not limited thereto. For example, the camera 120 may be slid relative to the base 101 of the robot hand 100 without moving the finger 140, and the pin W may be retracted or returned to the camera 120 as a relative positional relationship between the camera 120 and the finger 140. Furthermore, the camera 120 may be attached to the robot arm 10A, or may be installed at a fixed point (for example, a ceiling) of the installation location where the robot system 1 is installed. In particular, when the camera 120 is installed at a fixed point, the pin W may be retracted or returned to the camera 120 by moving the robot arm 10A. In short, it is sufficient that the tool and the workpiece are moved to positions where they do not overlap within the field of view of the camera 120 so that the workpiece does not get in the way when the camera 120 captures the object to be captured (work position, contact position). Then, when the camera is returned to the position to be imaged in order to perform work on the workpiece, not only will the workpiece block the image, but the camera 120 may also move along with the workpiece, and the imaged object may not be within the field of view of the camera 120.

[0114] In the first to fourth embodiments, the tool of the end effector is the finger portion 140 of the robot hand 100, and has been described as gripping the pin W as the workpiece, but the present invention is not limited to this. For example, a screwdriver as a tool that comes into contact with a screw as the workpiece and screws it in is also conceivable. Furthermore, the present invention is not limited to a tool that grips and supports a workpiece, and may be a tool that supports a workpiece by, for example, adsorption or magnetic attachment.

[0115] In the first to fourth embodiments, the robot hand 100 has been described as sliding the first finger portion 141 and the second finger portion 142, but the present invention is not limited to this configuration. For example, the robot hand 100 may have three or more fingers, or the fingers may move on an arc instead of a straight line. In addition, the robot hand 100 may have one guide portion or three or more guide portions so that the first finger portion 141 and the second finger portion 142 slide. However, the present invention is not limited to this, and the robot hand 100 may have one guide portion or three or more guide portions. Of course, the presence of a plurality of guide portions stabilizes the movement of the first finger portion 141 and the second finger portion 142, and the gripping of the workpiece is also stable, so it is preferable to have two or more guide portions. In addition, the robot hand 100 has been described as driving the first finger portion 141 and the second finger portion 142 with the motor 151 and the motor 161, but the present invention is not limited to this, and the robot hand 100 may be driven by, for example, a solenoid or hydraulic pressure.

[0116] In the third embodiment, the camera 120 uses the telecentric lens 122 and the coaxial epi-illumination 130 to irradiate the plane of the image capture target with illumination light perpendicularly and capture an image on the optical axis. However, the present invention is not limited to this, and may be configured such that the camera and illumination are not coaxial. In this case, it is possible to set an angle such that the optical axis of the illumination light is reflected on the plane of the image capture target and directed toward the camera.

[0117] In addition, the present disclosure is not limited to the above-described embodiments, and the embodiments can be modified in many ways within the technical concept of the present disclosure. For example, at least two of the above-described embodiments and modifications may be combined. In addition, the effects described in the present embodiment are merely a list of the most preferable effects resulting from the embodiments of the present disclosure, and the effects of the embodiments of the present disclosure are not limited to those described in the present embodiment.

[0118] In the above-mentioned first to fourth embodiments, the robot body is a vertically articulated robot, but the present invention is not limited to this. The robot body may be, for example, a horizontally articulated robot, a parallel link robot, or an orthogonal robot. The above-mentioned embodiments are applicable to a machine that can automatically perform the actions of expanding and contracting, bending and stretching, moving up and down, moving left and right, or turning, or a combination of these actions, based on information stored in a storage device provided in a control device.

[0119] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0120] <Disclosure of the Present Embodiment> [Configuration 1] A main body portion, a camera supported by the main body and capable of capturing images; a tool that is movably supported on the main body, capable of supporting a workpiece, and capable of performing an operation on the workpiece; When an imaging target is imaged by the camera, the tool supporting the workpiece is positioned outside a range in which at least a part of the tool or the workpiece supported by the tool overlaps with the imaging target in the field of view of the camera. An end effector characterized by: [Configuration 2] From a state in which at least a part of the tool or the supported workpiece is inside the range, the tool supporting the workpiece is moved outside the range before capturing an image of an imaging target by the camera. 2. The end effector of configuration 1. [Configuration 3] the imaging target is a work target on which work is performed using the tool, moving the tool supporting the workpiece from a state in which the tool supporting the workpiece is outside the range so that the workpiece at least partially overlaps the work target in the field of view of the camera before performing a task with the tool; 3. An end effector according to configuration 1 or 2. [Configuration 4] When imaging an imaging target with the camera, the tool and the supported workpiece are positioned outside the field of view of the camera. 4. The end effector according to any one of configurations 1 to 3. [Configuration 5] The imaging target is a contact position where the workpiece supported by the tool comes into contact during the operation. 5. The end effector according to any one of configurations 1 to 4. [Configuration 6] The tool includes a first finger portion and a second finger portion that support a workpiece by gripping the workpiece, 6. The end effector according to any one of configurations 1 to 5, [Configuration 7] The first finger portion has a first root portion that is supported movably relative to the main body, and a first fingertip portion that is disposed at a tip end of the first root portion and grips a workpiece, The second finger portion has a second root portion that is supported movably relative to the main body, and a second fingertip portion that is disposed at a tip end of the second root portion and grips a workpiece. 7. The end effector of configuration 6. [Configuration 8] the first finger portion and the second finger portion are formed such that, when moved relative to the main body, the first fingertip portion and the second fingertip portion move to the inside and outside of the range, and even when the first fingertip portion and the second fingertip portion are inside the range, the first root portion and the second root portion are located outside the range. 8. The end effector of configuration 7. [Configuration 9] The first fingertip portion and the second fingertip portion are positioned between the camera and a focal position of the camera while being positioned inside the range. 9. The end effector according to configuration 7 or 8. [Configuration 10] The main body portion is a slide portion that supports the first root portion and the second root portion so as to be slidable; a first drive unit that drives the first root portion to slide on the slide unit; A second drive unit that can be driven independently of the first drive unit and drives the second root portion to slide on the slide unit. 10. The end effector according to any one of configurations 7 to 9. [Configuration 11] An end effector according to configuration 1; a robot body to which the end effector is attached and capable of changing the position and orientation of the end effector; a control unit that controls the robot body and the end effector, The control unit executes alignment control for aligning the end effector with respect to the imaging target based on the image captured by the camera. A robot system comprising: [Configuration 12] The control unit performs, as the alignment control, a positional deviation between the end effector and the imaging target, based on an image captured by the camera and a target image serving as a target, and executes a correction control to correct the position of the end effector so as to eliminate the positional deviation. 12. The robot system according to claim 11, [Configuration 13] The imaging target has a planar portion formed in a flat shape and an inclined portion inclined with respect to the planar portion, The end effector includes an irradiation unit that irradiates light in an optical axis direction, The control unit illuminates the imaging target with the illumination unit so that an optical axis of the reflected light reflected by the flat portion is directed toward the camera and an optical axis of the reflected light reflected by the inclined portion is directed in a direction different from the camera, and captures an image with the camera. 13. The robot system according to claim 12, [Configuration 14] the imaging target has a hole formed in the planar portion, The inclined portion is a chamfered portion that is chamfered at the outer edge of the opening of the hole. 14. The robot system according to claim 13, [Configuration 15] The camera has a telecentric optical system. The control unit captures an image with the camera in a state where the optical axis of the telecentric optical system is perpendicular to the planar portion. 14. The robot system according to claim 13, [Configuration 16] the illumination unit is a coaxial epi-illumination unit that illuminates light coaxially with an optical axis of the telecentric optical system, the control unit captures an image with the camera while illuminating the planar portion with the coaxial epi-illumination in a state where the optical axis of the telecentric optical system is perpendicular to the planar portion. 16. The robot system according to configuration 15. [Configuration 17] The imaging target is a work position where a work on a workpiece is performed by the tool, When the work positions are multiple locations, the same target image is used in the alignment control executed for each of the multiple work positions. 17. The robot system according to any one of configurations 12 to 16. [Configuration 18] The operations performed at each of the plurality of work positions include a first operation of causing the tool to support a workpiece at at least one of the plurality of work positions, and a second operation of bringing the workpiece supported by the tool into contact with at least one of the plurality of work positions; The same target image is used in the alignment control executed before performing the first operation and the alignment control executed before performing the second operation. 18. The robot system according to configuration 17. [Configuration 19] The control unit is While changing the posture of the end effector, a luminance of light reflected by the planar portion is calculated from an image captured by the camera; executing a posture correction control for correcting the posture of the end effector so as to maximize the calculated brightness; 19. The robot system according to any one of configurations 11 to 18, [Configuration 20] The control unit is A teaching device that operates the posture of the end effector to teach the robot and a display device that displays information can be connected to the teaching device, When the posture of the end effector is changed, a luminance of light reflected by the planar portion is calculated from the image captured by the camera, and the luminance is displayed on the display device. 19. The robot system according to any one of configurations 11 to 18, [Configuration 21] an end effector including a camera capable of capturing an image and an irradiating unit that irradiates light in an optical axis direction; a robot body to which the end effector is attached and capable of changing the position and orientation of the end effector; a control unit that controls the robot body and the end effector, The control unit is When performing alignment control for aligning the end effector with respect to an imaging target based on an image captured by the camera, an imaging target having a flat portion formed in a flat shape and an inclined portion inclined with respect to the flat portion is irradiated by the irradiating unit so that the optical axis of the reflected light reflected by the flat portion is directed toward the camera and the optical axis of the reflected light reflected by the inclined portion is directed in a direction different from the camera, and the imaging target is imaged by the camera; A robot system comprising: [Method 22] A method for controlling an end effector including a main body, a camera supported on the main body and capable of capturing images, and a tool supported movably on the main body and configured to perform an operation on a workpiece, the end effector being controlled by a control unit, the method comprising: When the control unit captures an image of an imaging target with the camera, the control unit positions the tool supporting the workpiece outside a range in which at least a part of the tool or the workpiece supported by the tool overlaps with the imaging target in the field of view of the camera. A method for controlling an end effector comprising the steps of: [Method 23] An end effector including a main body, a camera supported on the main body and capable of capturing images, and a tool supported movably on the main body and performing an operation on a workpiece; a robot body to which the end effector is attached and capable of changing the position and orientation of the end effector; a control unit that controls the robot body and the end effector, When the control unit captures an image of an imaging target by the camera, a retraction process is performed in which the tool supporting the workpiece is positioned outside a range in which at least a part of the tool or the workpiece supported by the tool overlaps with the imaging target in the field of view of the camera; an imaging step in which the control unit captures an image of an imaging target with the camera while the tool supporting a workpiece is positioned outside the range; The control unit includes a positioning step of positioning the end effector with respect to the imaging target based on an image captured by the camera. A method for controlling a robot system comprising: [Method 24] In the retreat step, the control unit moves the tool supporting the workpiece from a state in which at least a part of the tool and the supported workpiece are inside the range to outside the range before the camera captures an image of the imaging target. 24. A method for controlling a robot system according to claim 23. [Method 25] The control unit includes a return process in which, after the imaging process, the tool supporting the workpiece is moved from a state in which the tool supporting the workpiece is outside the range to an inside of the range so that at least a part of the workpiece overlaps the imaging target in the field of view of the camera before the work is performed by the work tool. 25. A method for controlling a robot system according to method 23 or 24. [Method 26] an end effector including a camera capable of capturing an image and an irradiating unit that irradiates light in an optical axis direction; a robot body to which the end effector is attached and capable of changing the position and orientation of the end effector; a control unit that controls the robot body and the end effector, an illumination step in which the control unit illuminates an imaging target having a planar portion formed in a flat shape and an inclined portion inclined with respect to the planar portion, using the illumination unit, such that an optical axis of reflected light reflected by the planar portion is directed toward the camera, and an optical axis of reflected light reflected by the inclined portion is directed in a direction different from that of the camera; an imaging step in which the control unit captures an image of the imaging target illuminated with light in the illumination step by the camera; The control unit includes a calculation step of calculating a positional relationship between the end effector and the imaging target from an image captured by the camera. A method for controlling a robot system comprising: [Method 27] A method for manufacturing an article, comprising the steps of: manufacturing the article using the robot system according to any one of configurations 11 to 21. [Configuration 28] A program for causing a computer to execute the control method for a robot system according to any one of Methods 23 to 25. [Configuration 29] A computer-readable recording medium storing the program according to configuration 28. [Explanation of symbols]

[0121] 1...Robot system / 10A...Robot arm (robot body) / 100...Robot hand (end effector) / 101...Base (main body) / 120...Camera / 122...Telecentric lens (telecentric optical system) / 130...Coaxial incident lighting (illumination section) / 140...Finger section (tool) / 141...First finger section / 141a...First base section / 141b...First fingertip section / 142...Second finger section / 142a...Second base section / 142b...Second finger Tip / 150 ... first drive unit / 160 ... second drive unit / 170 ... first guide unit (slide unit) / 180 ... second guide unit (slide unit) / 201 ... CPU (control unit) / 401H ... hole unit (image target, work target, contact position) / 401m ... chamfered portion (inclined portion) / 401s ... surface (flat portion) / 402H ... hole unit (image target, work target, contact position) / 402m ... chamfered portion (inclined portion) / 402s ... surface (flat portion) / P ... focal position / W ... pin (work)

Claims

1. Base and, The imaging device supported at the base, The tool is movably supported on the base, capable of supporting a workpiece, and capable of performing operations on the workpiece. The base is equipped with a drive mechanism that, when the imaging device images the object to be imaged, positions the tool supporting the workpiece outside the area in the field of view of the imaging device where at least a portion of the tool or the workpiece supported by the tool overlaps with the object to be imaged. An end effector characterized by the following:

2. The drive mechanism moves the tool supporting the workpiece out of the range from a state in which at least a portion of the tool or the supported workpiece is inside the range, before the imaging device images the object to be imaged. The end effector according to feature 1.

3. The object to be imaged is the object to be worked on using the tool, The drive mechanism moves the tool supporting the workpiece from a state where the tool supporting the workpiece is outside the range, to a state where the tool performs work on the work object, such that the workpiece overlaps with at least a portion of the work object in the field of view of the imaging device, before the tool performs work on the work object. The end effector according to feature 1.

4. The drive mechanism, when the imaging device is used to image an object, positions the tool and the supported workpiece outside the field of view of the imaging device. The end effector according to feature 1.

5. The object to be imaged is the contact point where the workpiece supported by the tool is brought into contact during the operation. The end effector according to feature 1.

6. The tool includes a first finger portion and a second finger portion that grip and support the workpiece. The end effector according to feature 1.

7. The first finger portion has a first root portion that is movably supported with respect to the base portion, and a first fingertip portion that is positioned further forward than the first root portion and grips the workpiece. The second finger portion has a second root portion that is movably supported with respect to the base portion, and a second fingertip portion that is positioned further forward than the second root portion and grips the workpiece. The end effector according to feature 6.

8. The first finger portion and the second finger portion are formed such that when moved relative to the base by the drive mechanism, the first fingertip portion and the second fingertip portion move inward and outward from the range, and even when the first fingertip portion and the second fingertip portion are inside the range, the first root portion and the second root portion are located outside the range. The end effector according to feature 7.

9. The first fingertip portion and the second fingertip portion are positioned within the range and between the imaging device and the focal point of the imaging device. The end effector according to feature 7.

10. The aforementioned drive mechanism is A sliding portion that supports the first base portion and the second base portion so as to be slidable, A first drive unit drives the first base portion to slide within the slide portion, The device includes a second drive unit which is drivable independently of the first drive unit and drives the second root portion to slide within the slide portion, The end effector according to feature 7.

11. An end effector comprising a base, an imaging device supported by the base and capable of imaging, and a tool movably supported by the base and capable of performing work on a workpiece, A robot to which the end effector is attached and which can move the position and orientation of the end effector, A robot system comprising the robot and a control unit for controlling the end effector, When the control unit captures an image of the object to be captured using the imaging device, it positions the tool supporting the workpiece outside the area in the field of view of the imaging device where at least a portion of the tool or the workpiece supported by the tool overlaps with the object to be captured. A robotic system characterized by the following features.

12. The control unit performs alignment control to align the end effector with the target being captured based on the image captured by the imaging device. The robot system according to feature 11.

13. The control unit, as part of the alignment control, calculates the positional misalignment between the end effector and the target to be imaged from the image captured by the imaging device and the target image, and performs correction control to correct the position of the end effector so that the positional misalignment is eliminated. The robot system according to claim 12, characterized in that it is the robot system according to claim 12.

14. The object to be imaged has a planar flat portion and an inclined portion that is inclined with respect to the planar portion. The end effector includes an irradiation section that irradiates light in the direction of the optical axis, The control unit illuminates the object to be imaged with the illumination unit such that the optical axis of the reflected light reflected from the planar portion is directed toward the imaging device, and the optical axis of the reflected light reflected from the inclined portion is directed toward a direction different from the imaging device, and then images are taken by the imaging device. The robot system according to claim 13, characterized in that it is the robot system according to claim 13.

15. The object to be imaged has a hole formed in the shape of a hole in the planar portion, The inclined portion is a chamfered portion formed at the outer edge of the opening of the hole. The robot system according to feature 14.

16. The imaging device has a telecentric optical system, The control unit, with the optical axis of the telecentric optical system aligned perpendicular to the plane, captures an image using the imaging device. The robot system according to feature 14.

17. The illumination unit is a coaxial incident illumination that irradiates light coaxially with the optical axis of the telecentric optical system. The control unit aligns the optical axis of the telecentric optical system perpendicular to the plane, and while illuminating the plane with coaxial incident illumination, the imaging device captures an image. The robot system according to feature 16.

18. The object to be imaged is the work position where the tool performs work on the workpiece. If there are multiple work locations, the alignment control performed for each of the multiple work locations uses the same target image. The robot system according to claim 13, characterized in that it is the robot system according to claim 13.

19. Each of the operations performed at the plurality of work positions includes a first operation of causing the tool to support a workpiece located at at least one of the plurality of work positions, and a second operation of bringing the workpiece supported by the tool into contact with at least one of the plurality of work positions. The alignment control performed before the first operation and the alignment control performed before the second operation use the same target image. The robot system according to feature 18.

20. The control unit, While changing the orientation of the end effector, the brightness of the light reflected from the flat surface is calculated from the image captured by the imaging device. The attitude correction control is performed to correct the attitude of the end effector so that the calculated brightness is maximized. The robot system according to feature 14.

21. The control unit, It can be connected to a teaching device that performs teaching by manipulating the posture of the end effector, and a display device that displays information. When the orientation of the end effector is changed, the brightness of the light reflected from the flat surface is calculated from the image captured by the imaging device and displayed on the display device. The robot system according to feature 14.

22. An end effector comprising an imaging device and an illumination unit that irradiates light in the direction of the optical axis, A robot to which the end effector is attached and which can move the position and orientation of the end effector, The system comprises a control unit for controlling the robot and the end effector, The control unit, When performing alignment control to align the end effector with respect to the target object based on the image captured by the imaging device, An imaging target having a planar flat portion and an inclined portion inclined with respect to the planar portion is illuminated by the illumination unit such that the optical axis of the reflected light from the planar portion is directed toward the imaging device, and the optical axis of the reflected light from the inclined portion is directed toward a direction different from the imaging device, and then imaged by the imaging device. A robotic system characterized by the following features.

23. A control method for an end effector comprising a base, an imaging device supported by the base, and a tool movably supported by the base and capable of performing work on a workpiece, When the control unit captures an image of the object to be captured using the imaging device, it positions the tool supporting the workpiece outside the area in the field of view of the imaging device where at least a portion of the tool or the workpiece supported by the tool overlaps with the object to be captured. A method for controlling an end effector characterized by the following:

24. An end effector comprising a base, an imaging device supported by the base, and a tool movably supported by the base and capable of performing work on a workpiece, A robot to which the end effector is attached and which can move the position and orientation of the end effector, A control method for a robot system comprising the robot and a control unit for controlling the end effector, When the control unit captures an image target with the imaging device, it includes the step of positioning the tool supporting the workpiece outside the area in the field of view of the imaging device where at least a portion of the tool or the workpiece supported by the tool overlaps with the image target. A method for controlling a robot system characterized by the following features.

25. In the step of positioning the tool, the control unit moves the tool supporting the workpiece outside the range from a state in which the tool and at least a portion of the supported workpiece are inside the range, before the imaging device captures the target object. The control method for a robot system according to feature 24.

26. The control unit comprises an imaging step of imaging an object to be imaged with the imaging device while the tool supporting the workpiece is positioned outside the range, The control unit includes a return step, after the imaging step, of moving the tool supporting the workpiece inward from a state where the tool supporting the workpiece is outside the range, before performing work with the tool, so that the workpiece overlaps at least a portion of the object being imaged in the field of view of the imaging device. The control method for a robot system according to feature 24.

27. An end effector comprising an imaging device capable of taking images and an illumination unit that irradiates light in the direction of the optical axis, A robot to which the end effector is attached and which can move the position and orientation of the end effector, A control method for a robot system comprising the robot and a control unit for controlling the end effector, The control unit performs an irradiation step in which it irradiates an imaging target having a planar flat portion and an inclined portion inclined with respect to the planar portion with the irradiation unit such that the optical axis of the reflected light reflected from the planar portion is directed toward the imaging device, and the optical axis of the reflected light reflected from the inclined portion is directed toward a direction different from the imaging device. The control unit performs an imaging step in which the imaging target, which has been irradiated with light in the irradiation step, is imaged by the imaging device, The control unit includes a calculation step of calculating the positional relationship between the end effector and the object to be captured from the image captured by the imaging device. A method for controlling a robot system characterized by the following features.

28. A method for manufacturing an article, characterized by manufacturing the article using the robot system described in claim 11.

29. A program for causing a computer to execute the control method for the robot system described in claim 24.

30. A non-temporary recording medium readable by a computer storing the program described in claim 29.

Images