Robot system

Abstract

A robot system having a robot arm that has a movable part, the robot system having a first imaging device and a second imaging device attached to the robot arm, a control unit for controlling the robot system, and a distance information acquisition unit for acquiring information relating to the distance to an object, the control unit being capable of changing a base length which is the distance between the first imaging device and the second imaging device, and the distance information acquisition unit acquiring information that relates to the distance to the object on the basis of the base length.

Description

Technical Field

[0001] The present invention relates to a robot system To the extent .

Background Art

[0002] A robot system including an imaging device is known. For example, Patent Document 1 describes a configuration in which an imaging device is attached to a robot arm.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] One aspect of the robot system of the present invention is a robot system having a robot arm with a movable part, the robot arm having a first imaging device and a second imaging device attached thereto, a control unit for controlling the robot system, and a distance information acquisition unit for acquiring information regarding the distance of an object, wherein the control unit By moving at least one of the first imaging device or the second imaging device can change a baseline length which is the distance between the first imaging device and the second imaging device, and the distance information acquisition unit acquires information regarding the distance of the object based on the baseline length.

Brief Description of the Drawings

[0024]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12A

Figure 12B

Figure 12C

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0025] Hereinafter, a robot system, a robot arm, an end effector, and an adapter according to embodiments of the present invention will be described with reference to the drawings. Note that the scope of the present invention is not limited to the following embodiments and can be arbitrarily changed within the scope of the technical idea of the present invention. In the following drawings, in order to make each configuration easy to understand, the scale, number, etc. in each structure may be different from the scale, number, etc. in the actual structure.

[0026] <First Embodiment> FIG. 1 is a perspective view showing the robot system 10 of this embodiment. FIG. 2 is a block diagram showing a part of the configuration of the robot system 10 of this embodiment. As shown in FIG. 1, the robot system 10 includes a robot 20, an imaging device 30, a control unit 40, and a display unit 50. The robot 20 performs work on an object W on a workbench WB, for example.

[0027] The robot 20 has a robot arm 21, an end effector 22, and an adapter 23. The robot arm 21 has an arm part 24 as a movable part. In the present embodiment, a plurality of arm parts 24 are provided. The robot arm 21 is, for example, a multi-joint arm configured by connecting a plurality of arm parts 24. The arm part 24 includes, for example, five arm parts: a first arm part 24a, a second arm part 24b, a third arm part 24c, a fourth arm part 24d, and a fifth arm part 24e. The first arm part 24a, the second arm part 24b, the third arm part 24c, the fourth arm part 24d, and the fifth arm part 24e are connected in this order from the installation surface of the robot arm 21.

[0028] As shown in FIG. 2, the robot arm 21 has an arm drive unit 25 and an arm position acquisition unit 26. The arm drive unit 25 is, for example, a servo motor. The arm drive unit 25 is provided, for example, for each arm part 24. That is, five arm drive units 25 are provided, for example.

[0029] The arm drive unit 25 provided in the first arm part 24a displaces the first arm part 24a with reference to the installation surface of the robot 20. The arm drive unit 25 provided in the second arm part 24b displaces the second arm part 24b with reference to the first arm part 24a. The arm drive unit 25 provided in the third arm part 24c displaces the third arm part 24c with reference to the second arm part 24b. The arm drive unit 25 provided in the fourth arm part 24d displaces the fourth arm part 24d with reference to the third arm part 24c. The arm drive unit 25 provided in the fifth arm part 24e displaces the fifth arm part 24e with reference to the fourth arm part 24d. Each arm drive unit 25 rotates each arm part 24, for example.

[0030] The arm position acquisition unit 26 includes, for example, a rotary encoder (not shown). The arm position acquisition unit 26 is provided, for example, for each arm unit 24. That is, five arm position acquisition units 26 are provided, for example. The arm position acquisition unit 26 provided in the first arm unit 24a can detect the displacement amount of the first arm unit 24a with respect to the installation surface of the robot 20. The arm position acquisition unit 26 provided in the second arm unit 24b can detect the displacement amount of the second arm unit 24b with respect to the first arm unit 24a. The arm position acquisition unit 26 provided in the third arm unit 24c can detect the displacement amount of the third arm unit 24c with respect to the second arm unit 24b. The arm position acquisition unit 26 provided in the fourth arm unit 24d can detect the displacement amount of the fourth arm unit 24d with respect to the third arm unit 24c. The arm position acquisition unit 26 provided in the fifth arm unit 24e can detect the displacement amount of the fifth arm unit 24e with respect to the fourth arm unit 24d. The displacement amount of each arm unit 24 that can be detected by each arm position acquisition unit 26 includes, for example, the rotation angle of each arm unit 24 detected by a rotary encoder (not shown).

[0031] FIG. 3 is a perspective view showing a part of the robot arm 21, the end effector 22, the adapter 23, and the imaging device 30 of the present embodiment. FIG. 4 is a plan view showing a part of the robot arm 21, the end effector 22, the adapter 23, and the imaging device 30 of the present embodiment. As shown in FIGS. 3 and 4, the end effector 22 is attached to the robot arm 21. In the present embodiment, the end effector 22 is attached to the tip of the fifth arm unit 24e via the adapter 23. The end effector 22 is detachably attached to the robot arm 21, for example. The end effector 22 can be replaced with another end effector.

[0032] Note that, as the end effector 22 attached to the robot arm 21, end effectors having various shapes, structures, and functions can be appropriately adopted according to the work performed by the robot 20. As the end effector 22 attached to the robot arm 21, for example, a robot hand capable of gripping the object W, a processing head for performing laser processing or ultrasonic processing, a camera, an injector for injecting molten metal, resin, or particles for blasting, a manipulator, and an air blow can be mentioned.

[0033] In the present embodiment, the end effector 22 is a multi-fingered robot hand capable of gripping the object W on the workbench WB. As shown in FIG. 3, the end effector 22 includes a base 22a, a plurality of finger portions 22b, an end effector drive unit 28, and an end effector position acquisition unit 29. The base 22a is connected to the fifth arm portion 24e via an adapter 23, for example. The base 22a is, for example, cylindrical with a central axis CL appropriately shown in each figure as the center. The central axis CL appropriately shown in each figure is the central axis of the end effector 22, the adapter 23, and the fifth arm portion 24e.

[0034] In the following description, the direction parallel to the central axis CL is referred to as the "central axis direction" and is appropriately indicated by the Z-axis in each figure. The positive side (+Z side) of the Z-axis in the central axis direction is referred to as the "tip side", and the negative side (-Z side) in the central axis direction is referred to as the "base end side". Also, unless otherwise specified, the radial direction centered on the central axis CL is simply referred to as the "radial direction", and the circumferential direction around the central axis CL is simply referred to as the "circumferential direction".

[0035] In the present embodiment, a guide rail portion 22e is provided on the base 22a. The guide rail portion 22e is, for example, an annular groove surrounding the base 22a along the circumferential direction. The plurality of finger portions 22b protrude from the base 22a toward the tip side (+Z side) in the central axis direction. In the present embodiment, the end effector 22 can grip the object W with the plurality of finger portions 22b. Note that the number of finger portions 22b is not particularly limited.

[0036] The end effector drive unit 28 can drive the end effector 22. The end effector drive unit 28 has a rotational drive unit 22c. The rotational drive unit 22c is provided, for example, inside the base portion 22a. The rotational drive unit 22c is, for example, a servo motor that can rotate around the central axis CL of the base portion 22a. Although illustration is omitted, the end effector drive unit 28 has a finger drive unit that drives the plurality of finger portions 22b. The finger drive unit is provided, for example, for each of the plurality of finger portions 22b. The finger drive unit may be provided one by one for each finger portion 22b, or may be provided in plural for each finger portion 22b. The finger drive unit, for example, displaces the angle of the finger portion 22b with respect to the base portion 22a. The finger drive unit is, for example, a servo motor.

[0037] The end effector position acquisition unit 29 can acquire the relative position of the end effector 22 with respect to the adapter 23. The end effector position acquisition unit 29 has a rotational position acquisition unit 22d. The rotational position acquisition unit 22d is provided, for example, inside the base portion 22a. The rotational position acquisition unit 22d can detect the rotational position of the end effector 22. The rotational position acquisition unit 22d can detect, for example, the rotational angle around the central axis CL of the base portion 22a. The rotational position acquisition unit 22d is, for example, a rotary encoder. The end effector position acquisition unit 29 may include, for example, a sensor that can detect the position and angle of the finger portion 22b with respect to the base portion 22a.

[0038] In this embodiment, a camera unit 60 is attached to the end effector 22. The camera unit 60 is fixed to the base portion 22a. The camera unit 60 includes a support portion 61, a first camera 62, and a second camera 63. The support portion 61 protrudes from the base portion 22a toward the tip side in the central axis direction (+Z side). The end portion on the base end side (-Z side) of the support portion 61 is connected to the outer peripheral surface at the end portion on the tip side of the base portion 22a. The first camera 62 and the second camera 63 are fixed to the end portion on the tip side of the support portion 61. The first camera 62 and the second camera 63 can image a plurality of finger portions 22b and an object W grasped by the finger portions 22b. The first camera 62 and the second camera 63 constitute a stereo camera. In addition, in FIG. 4, the illustration of the camera unit 60 is omitted.

[0039] As shown in FIG. 2, the camera unit 60 includes an image sensor 64, a memory 65, and a digital signal processing unit 66. The image sensor 64 is, for example, a CCD image sensor, a CMOS image sensor, or the like. The image sensor 64 is provided in each of the first camera 62 and the second camera 63. Each image sensor 64 converts an optical signal incident on the camera in which it is provided into an analog electrical signal, and further converts the converted analog electrical signal into a digital image signal and outputs it.

[0040] The digital signal processing unit 66 performs image processing such as digital amplification, color interpolation processing, and white balance processing on the digital image signal output from the image sensor 64. The digital image signal after being processed by the digital signal processing unit 66 may be temporarily stored in the memory 65, or may be output to the control unit 40 without being stored in the memory 65. The digital image signal output from the digital signal processing unit 66 to the control unit 40 is output to a distance information acquisition unit 44 described later.

[0041] The memory 65 can store the digital image signal output from the image sensor 64 and the digital image signal output from the digital signal processing unit 66. The memory 65 is, for example, a volatile memory. Note that the memory 65 may be a non-volatile memory. The digital image signal output from the image sensor 64 is, for example, stored in the memory 65 and then sent from the memory 65 to the digital signal processing unit 66, where image processing is performed.

[0042] Note that the memory 65 and the digital signal processing unit 66 may be provided one by one in the camera unit 60 and used for both the image sensors 64 of the first camera 62 and the image sensors 64 of the second camera 63, or may be provided for each of the image sensors 64 of the first camera 62 and the image sensors 64 of the second camera 63. Also, part or all of the memory 65 and the digital signal processing unit 66 may be provided outside the camera unit 60 such as the control unit 40. Also, the camera unit 60 may have only one camera.

[0043] The adapter 23 is a member for attaching the end effector 22 to the robot arm 21. As shown in FIG. 4, the adapter 23 has a support portion 23a, a pedestal portion 23b, a connection portion 23c, and a pedestal drive portion 23d. The support portion 23a is a portion connected to the robot arm 21. The support portion 23a is, for example, detachably connected to the tip of the fifth arm portion 24e. The support portion 23a has a recessed portion 23g that depresses from the tip side (+Z side) to the base end side (-Z side) in the central axis direction. Note that the support portion 23a may be fixedly attached to the robot arm 21 in a non-detachable manner.

[0044] The pedestal portion 23b is disposed on the distal end side (+Z side) of the support portion 23a. The pedestal portion 23b is the portion to which the end effector 22 is connected. In the present embodiment, the base portion 22a of the end effector 22 is detachably connected to the end portion on the distal end side (+Z side) of the pedestal portion 23b. Note that the end effector 22 may be non-detachably fixed to the pedestal portion 23b. The portion on the proximal end side (-Z side) of the pedestal portion 23b is inserted into the inside of the concave portion 23g, for example. For example, a gap is provided between the support portion 23a and the pedestal portion 23b, and the support portion 23a and the pedestal portion 23b are not in direct contact with each other.

[0045] The connecting portion 23c is provided inside the concave portion 23g. The connecting portion 23c is provided between the support portion 23a and the pedestal portion 23b. The connecting portion 23c connects the support portion 23a and the pedestal portion 23b. That is, in the present embodiment, the support portion 23a and the pedestal portion 23b are indirectly connected to each other via the connecting portion 23c without being in direct contact with each other. The connecting portion 23c supports the mass of the pedestal portion 23b and the mass of the end effector 22.

[0046] The connecting portion 23c has a damper element 23e and a spring element 23f. The spring element 23f may be, for example, an elastic member whose elastic force can be adjusted. In this case, the robot system 10 may have an adjustment portion capable of adjusting the elastic force of the spring element 23f. The spring element 23f may be, for example, an air spring. In this case, the elastic force of the spring element 23f may be adjusted by adjusting the air pressure of the air spring by the adjustment portion. For example, a plurality of connecting portions 23c are provided. The plurality of connecting portions 23c include, for example, a connecting portion 23c having a damper element 23e and a spring element 23f that are displaced by receiving a force in the central axis direction, and a connecting portion 23c having a damper element 23e and a spring element 23f that are displaced by receiving a force in a direction orthogonal to the central axis direction.

[0047] The connecting portion 23c can reduce, for example, the vibration of the end effector 22 and the vibration applied from the outside. The connecting portion 23c suppresses the displacement of the end effector 22 and the displacement of the pedestal portion 23b caused by the self-weight of the end effector 22 and the self-weight of the pedestal portion 23b. The connecting portion 23c can support the end effector 22 in the direction of gravity regardless of the posture of the end effector 22. Note that the connecting portion 23c may have other elements capable of reducing the vibration of the end effector 22 in addition to, for example, the damper element 23e and the spring element 23f. The other element includes, for example, a piezoelectric element (piezo element).

[0048] The pedestal driving portion 23d is provided, for example, between the support portion 23a and the pedestal portion 23b inside the recess 23g. The pedestal driving portion 23d can displace the position of the pedestal portion 23b with respect to the support portion 23a. The pedestal driving portion 23d can move the end effector 22 connected to the pedestal portion 23b by moving the pedestal portion 23b.

[0049] The pedestal driving portion 23d has, for example, a plurality of linear motors 27. The linear motor 27 is, for example, a voice coil motor. The linear motor 27 has a magnetic field generating portion 27a and a magnet portion 27b. Of the magnetic field generating portion 27a and the magnet portion 27b, one is attached to the support portion 23a and the other is attached to the pedestal portion 23b. In FIG. 4, for example, the magnetic field generating portion 27a is attached to the support portion 23a and the magnet portion 27b is attached to the pedestal portion 23b. Note that the magnetic field generating portion 27a may be attached to the pedestal portion 23b and the magnet portion 27b may be attached to the support portion 23a.

[0050] The magnetic field generating unit 27a is, for example, a coil. When an electric current is supplied to the magnetic field generating unit 27a, a magnetic field is generated. Due to the magnetic field generated from the magnetic field generating unit 27a and the magnetic field generated from the magnet unit 27b, a repulsive force or an attractive force is generated between the magnetic field generating unit 27a and the magnet unit 27b. Due to this repulsive force or attractive force, the magnet unit 27b is displaced with respect to the magnetic field generating unit 27a. As a result, the linear motor 27 displaces the pedestal portion 23b to which the magnet unit 27b is attached with respect to the support portion 23a to which the magnetic field generating unit 27a is attached. In this way, the pedestal driving unit 23d can drive the pedestal portion 23b in a non-contact state without directly contacting the support portion 23a and the pedestal portion 23b with each other.

[0051] The plurality of linear motors 27 include, for example, a linear motor 27 capable of displacing the pedestal portion 23b in the central axis direction with respect to the support portion 23a, and a linear motor 27 capable of displacing the pedestal portion 23b in a direction orthogonal to the central axis direction with respect to the support portion 23a.

[0052] Note that the adapter 23 may have any configuration as long as it can attach the end effector 22 to the robot arm 21. As the configuration of the adapter 23, for example, the configuration of the adapter described in International Application PCT / JP2019 / 016043 may be adopted.

[0053] As shown in FIG. 3, in the present embodiment, a plurality of imaging devices 30 are provided. For example, two imaging devices 30, i.e., a first imaging device 31 and a second imaging device 32, are provided. The first imaging device 31 and the second imaging device 32 may be, for example, RGB cameras or infrared cameras. The first imaging device 31 and the second imaging device 32 constitute a stereo camera. In the present embodiment, the first imaging device 31 and the second imaging device 32 are attached to the end effector 22. The first imaging device 31 and the second imaging device 32 are arranged around the end effector 22. The first imaging device 31 and the second imaging device 32 are, for example, located on the outer side in the radial direction of the base portion 22a and are arranged along the circumferential direction.

[0054] In this embodiment, the optical axis AX1 of the first imaging device 31 and the optical axis AX2 of the second imaging device 32 are parallel to each other. The optical axes AX1 and AX2 are parallel to the central axis CL, for example. In this specification, "the optical axes of a plurality of imaging devices are parallel to each other" includes not only the case where the optical axes of the plurality of imaging devices are strictly parallel to each other, but also the case where the optical axes of the plurality of imaging devices are substantially parallel to each other. The case where the optical axes of the plurality of imaging devices are substantially parallel to each other includes, for example, the case where the optical axes of the plurality of imaging devices are inclined to each other within 5°.

[0055] FIG. 5 is a view of a part of the end effector 22, the first imaging device 31, and the second imaging device 32 as seen from the tip side (+Z side) in the central axis direction. FIG. 6 is a view of a part of the end effector 22, the first imaging device 31, and the second imaging device 32 as seen from the tip side (+Z side) in the central axis direction, showing the case where the first imaging device 31 and the second imaging device 32 are located at predetermined initial positions. In FIGS. 5 and 6, illustration of the finger portion 22b of the end effector 22 and the camera unit 60 is omitted.

[0056] As shown in FIGS. 5 and 6, at least one of the first imaging device 31 and the second imaging device 32 is movable relative to the end effector 22. In this embodiment, both the first imaging device 31 and the second imaging device 32 are movable relative to the end effector 22. The relative position between the first imaging device 31 and the second imaging device 32 is variable. In this embodiment, at least one of the first imaging device 31 and the second imaging device 32 is movable in a predetermined circumferential direction around the end effector 22. In this embodiment, the "predetermined circumferential direction" is the circumferential direction around the central axis CL around the base portion 22a.

[0057] In this embodiment, both the first imaging device 31 and the second imaging device 32 are movable in a predetermined circumferential direction around the end effector 22. That is, in this embodiment, one of the first imaging device 31 and the second imaging device 32 is movable in a predetermined circumferential direction around the end effector 22, and the other is also movable in a predetermined circumferential direction around the end effector 22. As shown in FIG. 6, in this embodiment, the first imaging device 31 and the second imaging device 32 can be in contact with each other in the circumferential direction. In this embodiment, when the first imaging device 31 and the second imaging device 32 are each located at the initial positions shown in FIG. 6, they are in a state of being in contact with each other in the circumferential direction.

[0058] As shown in FIG. 3, the first imaging device 31 includes a housing 31a, a first driving unit 31b, a first position acquisition unit 31c, a lens 31e, and an imaging element 31f. The housing 31a has an opening on the tip side (+Z side), for example, and is cylindrical and extends in the central axis direction. The central axis of the housing 31a coincides with the optical axis AX1 of the first imaging device 31, for example. The housing 31a is attached to the base 22a of the end effector 22 via a slider 31d.

[0059] The slider 31d is fixed, for example, to the radially inner portion of the proximal end side (-Z side) of the housing 31a. The slider 31d connects the housing 31a and the base 22a of the end effector 22. That is, in this embodiment, the first imaging device 31 is connected to the end effector 22 via the slider 31d. The slider 31d is connected to the guide rail portion 22e of the end effector 22. The slider 31d is movable in the circumferential direction along the guide rail portion 22e. Thereby, the first imaging device 31 is movable in the circumferential direction along the guide rail portion 22e.

[0060] The lens 31e is fitted into the opening on the tip side (+Z side) of the housing 31a. The lens 31e is, for example, a lens having a circular shape when viewed in the central axis direction. The optical axis AX1 of the first imaging device 31 passes through the center of the lens 31e.

[0061] The imaging element 31f is disposed inside the housing 31a. The imaging element 31f is, for example, a CCD image sensor, a CMOS image sensor, or the like. Light incident into the housing 31a through the lens 31e is incident on the imaging element 31f. The imaging element 31f converts the incident optical signal into an analog electrical signal, and further converts the converted analog electrical signal into a digital image signal and outputs it.

[0062] As shown in FIG. 5, the imaging element 31f has a rectangular shape when viewed in the central axis direction. When viewed in the central axis direction, the long side of the imaging element 31f is orthogonal to the direction passing through the optical axis AX1 of the first imaging device 31 in the radial direction. In the present embodiment, the first imaging device 31 is movable in the circumferential direction so as to maintain a state in which the long side of the imaging element 31f is orthogonal to the radial direction passing through the optical axis AX1 of the first imaging device 31 when viewed in the central axis direction.

[0063] As shown in FIG. 3, the first driving unit 31b is disposed, for example, inside the housing 31a. The first driving unit 31b is, for example, a servo motor. The first driving unit 31b moves the first imaging device 31 in the circumferential direction around the end effector 22. In the present embodiment, the first imaging device 31, together with the slider 31d, moves in the circumferential direction as a whole including the first driving unit 31b.

[0064] The first position acquisition unit 31c is disposed, for example, inside the housing 31a. The first position acquisition unit 31c is, for example, a rotary encoder. The first position acquisition unit 31c can acquire the circumferential position information of the first imaging device 31 by detecting the rotation of the first driving unit 31b. The first position acquisition unit 31c detects the circumferential position of the first imaging device 31 by, for example, setting the rotation speed of the first driving unit 31b to zero when the first imaging device 31 is located at the initial position shown in FIG. 6 and detecting the rotation speed of the first driving unit 31b.

[0065] The second imaging device 32 includes a housing 32a, a second driving unit 32b, a second position acquisition unit 32c, a lens 32e, and an imaging element 32f. The housing 32a has an opening on the tip side (+Z side), for example, and is cylindrical and extends in the central axis direction. The central axis of the housing 32a coincides with the optical axis AX2 of the second imaging device 32, for example. The housing 32a is attached to the base 22a of the end effector 22 via a slider 32d.

[0066] The slider 32d is fixed to, for example, the radially inner portion of the proximal end side (-Z side) of the housing 32a. The slider 32d connects the housing 32a and the base 22a of the end effector 22. That is, in the present embodiment, the second imaging device 32 is connected to the end effector 22 via the slider 32d. The slider 32d is connected to the guide rail portion 22e of the end effector 22. The slider 32d is movable in the circumferential direction along the guide rail portion 22e. Thereby, the second imaging device 32 is movable in the circumferential direction along the guide rail portion 22e.

[0067] As described above, in the present embodiment, the guide rail portion 22e corresponds to the first holding portion that holds the first imaging device 31 and also corresponds to the second holding portion that holds the second imaging device 32. That is, in the present embodiment, the end effector 22 has the guide rail portion 22e as the first holding portion where the first imaging device 31 is held and the second holding portion where the second imaging device 32 is held. In the present embodiment, the first imaging device 31 is movably held by the guide rail portion 22e as the first holding portion, and the second imaging device 32 is movably held by the guide rail portion 22e as the second holding portion.

[0068] The lens 32e is fitted into the opening on the tip side (+Z side) of the housing 32a. The lens 32e is, for example, a lens having a circular shape when viewed in the central axis direction. The optical axis AX2 of the second imaging device 32 passes through the center of the lens 32e.

[0069] The imaging element 32f is disposed inside the housing 32a. The imaging element 32f is, for example, a CCD image sensor, a CMOS image sensor, or the like. Light incident into the housing 32a through the lens 32e is incident on the imaging element 32f. The imaging element 32f converts the incident optical signal into an analog electrical signal and outputs the converted analog electrical signal after converting it into a digital image signal.

[0070] As shown in FIG. 5, the imaging element 32f is rectangular when viewed in the central axis direction. When viewed in the central axis direction, the long side of the imaging element 32f is orthogonal to the direction passing through the optical axis AX2 of the second imaging device 32 in the radial direction. In the present embodiment, the second imaging device 32 is movable in the circumferential direction so as to maintain a state in which the long side of the imaging element 32f is orthogonal to the radial direction passing through the optical axis AX2 of the second imaging device 32 when viewed in the central axis direction. The imaging element 32f has, for example, the same shape and size as the imaging element 31f of the first imaging device 31.

[0071] In the present specification, the "long side of the imaging element" refers to the long side in the rectangular region where light is incident on the imaging element. The imaging elements 31f and 32f shown in each figure show only the main body portion having a rectangular region where light is incident. The imaging elements 31f and 32f may have portions other than the main body portion, such as a frame portion that holds the main body portion where light is incident. In this case, even if the outer shape of the imaging elements 31f and 32f is other than a rectangle when viewed in the directions of the optical axes AX1 and AX2, the long side of the imaging elements 31f and 32f is the long side of the rectangular region where light is incident on the imaging elements 31f and 32f.

[0072] As shown in FIG. 3, the second drive unit 32b is disposed, for example, inside the housing 32a. The second drive unit 32b is, for example, a servo motor. The second drive unit 32b moves the second imaging device 32 in the circumferential direction around the end effector 22. In the present embodiment, the entire second imaging device 32 including the slider 32d and the second drive unit 32b moves in the circumferential direction.

[0073] In the present embodiment, a drive unit 33 for driving the imaging device 30 is configured by a first drive unit 31b and a second drive unit 32b. The drive unit 33 is capable of moving at least one of the first imaging device 31 and the second imaging device 32 with respect to the end effector 22. In the present embodiment, the drive unit 33 is capable of moving both the first imaging device 31 and the second imaging device 32 with respect to the end effector 22 by the respective drive units provided in each imaging device 30.

[0074] The second position acquisition unit 32c is disposed, for example, inside the housing 32a. The second position acquisition unit 32c is, for example, a rotary encoder. The second position acquisition unit 32c can acquire the circumferential position information of the second imaging device 32 by detecting the rotation of the second drive unit 32b. The second position acquisition unit 32c detects the circumferential position of the second imaging device 32, for example, by setting the rotation speed of the second drive unit 32b to zero when the second imaging device 32 is located at the initial position shown in FIG. 6 and then detecting the rotation speed of the second drive unit 32b.

[0075] In the present embodiment, a position acquisition unit 34 for acquiring at least the position information of the first imaging device 31 is configured by a first position acquisition unit 31c and a second position acquisition unit 32c. In the present embodiment, the position acquisition unit 34 can acquire both the position information of the first imaging device 31 and the position information of the second imaging device 32 by the respective position acquisition units provided in each imaging device 30. The position acquisition unit 34 can acquire, for example, the circumferential position of the first imaging device 31 and the circumferential position of the second imaging device 32.

[0076] As shown in Fig. 2, each imaging device 30 includes a memory 35 and a digital signal processing unit 36. The digital signal processing unit 36 performs image processing such as digital amplification, color interpolation processing, and white balance processing on the digital image signals output from the imaging elements of each imaging device 30. The digital image signals after being processed by the digital signal processing unit 36 may be temporarily stored in the memory 35, or may be output to the control unit 40 without being stored in the memory 35. The digital image signals output from the digital signal processing unit 36 to the control unit 40 are output to a distance information acquisition unit 44, which will be described later.

[0077] The memory 35 can store the digital image signals output from the imaging elements of each imaging device 30 and the digital image signals output from the digital signal processing unit 36. The memory 35 is, for example, a volatile memory. Note that the memory 35 may be a non-volatile memory. The digital image signals output from the imaging elements of each imaging device 30 are, for example, stored in the memory 35 and then sent from the memory 35 to the digital signal processing unit 36, where image processing is performed.

[0078] In the above description, the memory 35 and the digital signal processing unit 36 are configured to be provided one by one in each imaging device 30, but the present invention is not limited to this. The memory 35 and the digital signal processing unit 36 may be provided one by one for two imaging devices 30 and used for both the imaging element 31f of the first imaging device 31 and the imaging element 32f of the second imaging device 32. Further, part or all of the memory 35 and the digital signal processing unit 36 may be provided outside the imaging device 30 such as the control unit 40.

[0079] The control unit 40 controls the robot system 10. As shown in FIG. 2, in the present embodiment, the control unit 40 includes an arm control unit 41, an end effector control unit 42, an imaging device control unit 43, and a distance information acquisition unit 44. Each of the arm control unit 41, the end effector control unit 42, the imaging device control unit 43, and the distance information acquisition unit 44 may be realized by dedicated hardware or may be realized by a memory and a microprocessor.

[0080] The arm control unit 41 controls the arm drive unit 25. In the present embodiment, information regarding the position and orientation of the arm unit 24 is input to the arm control unit 41 from the arm position acquisition unit 26, and information regarding the distance to the object W is input from the distance information acquisition unit 44. In the present embodiment, the arm control unit 41 controls the arm drive unit 25 based on the information regarding the position and orientation of the arm unit 24 and the information regarding the distance to the object W. More specifically, for example, the arm control unit 41 calculates target values for the position and orientation of the arm unit 24 based on the information regarding the distance to the object W, and controls the arm drive unit 25 so that the position and orientation of the arm unit 24 become the target values by feedback control using the information from the arm position acquisition unit 26. In this way, the control unit 40 controls at least one of the position and orientation of the robot arm 21 by controlling the arm drive unit 25 by the arm control unit 41. Note that target values for the position and orientation of the arm unit 24 may be input to the arm control unit 41 from the outside.

[0081] The end effector control unit 42 controls the end effector drive unit 28. In the present embodiment, information regarding the position and orientation of the end effector 22 is input to the end effector control unit 42 from the end effector position acquisition unit 29, and information regarding the distance to the target object W is input from the distance information acquisition unit 44. In the present embodiment, the end effector control unit 42 controls the end effector drive unit 28 based on the information regarding the position and orientation of the end effector 22 and the information regarding the distance to the target object W. More specifically, for example, the end effector control unit 42 calculates target values for the position and orientation of the end effector 22 based on the information regarding the distance to the target object W, and controls the end effector drive unit 28 so that the position and orientation of the end effector 22 reach the target values by feedback control using the information from the end effector position acquisition unit 29. In this way, the control unit 40 controls at least one of the position and orientation of the end effector 22 by controlling the end effector drive unit 28 by the end effector control unit 42. Note that target values for the position and orientation of the end effector 22 may be input to the end effector control unit 42 from the outside.

[0082] The imaging device control unit 43 controls the drive unit 33 of the imaging device 30. In the present embodiment, information regarding the position of the imaging device 30 is input to the imaging device control unit 43 from the position acquisition unit 34 of the imaging device 30. More specifically, circumferential position information of the first imaging device 31 is input to the imaging device control unit 43 from the first position acquisition unit 31c, and circumferential position information of the second imaging device 32 is input from the second position acquisition unit 32c. In addition, information regarding the distance to the target object W is input to the imaging device control unit 43 from the distance information acquisition unit 44. The imaging device control unit 43 controls the first drive unit 31b and the second drive unit 32b based on, for example, the position information of the imaging device 30 input from the position acquisition unit 34 and the information regarding the distance to the target object W input from the distance information acquisition unit 44. Thereby, the control unit 40 controls the position of the imaging device 30 by controlling the drive unit 33 by the imaging device control unit 43.

[0083] The imaging device control unit 43 can change the baseline length L, which is the distance between the first imaging device 31 and the second imaging device 32. As shown in FIG. 5, the baseline length L is the distance between the optical axis AX1 of the first imaging device 31 and the optical axis AX2 of the second imaging device 32. For example, by moving the first imaging device 31 and the second imaging device 32 away from each other along the circumferential direction from the position indicated by the two-dot chain line to the position indicated by the solid line in FIG. 5, the baseline length L can be changed from the baseline length L1 to a baseline length L2 that is greater than the baseline length L1. Thereby, the baseline length L between the first imaging device 31 and the second imaging device 32 can be increased. On the other hand, by moving the first imaging device 31 and the second imaging device 32 closer to each other along the circumferential direction, the baseline length L between the first imaging device 31 and the second imaging device 32 can be decreased. Thus, in the present embodiment, the control unit 40 can change the baseline length L, which is the distance between the first imaging device 31 and the second imaging device 32, by controlling the drive unit 33 of the imaging device 30 by the imaging device control unit 43.

[0084] The imaging device control unit 43 calculates, for example, a target value of the baseline length L to be changed based on information regarding the distance of the object W. When the distance from the imaging device 30 to the object W is relatively large, the imaging device control unit 43 makes the baseline length L relatively large. On the other hand, when the distance from the imaging device 30 to the object W is relatively small, the imaging device control unit 43 makes the baseline length L relatively small. Note that a target value of the baseline length L may be input to the imaging device control unit 43 from the outside. A target value of the baseline length L may be input to the imaging device control unit 43 from the distance information acquisition unit 44.

[0085] In the present embodiment, the control unit 40 changes the baseline length L according to the work content of the robot system 10 by the imaging device control unit 43. For example, when searching for the object W on the workbench WB, the control unit 40 makes the baseline length L relatively large. On the other hand, for example, when approaching the end effector 22 to the object W after finding the object W to be worked on, the control unit 40 makes the baseline length L relatively small. At this time, the control unit 40 may decrease the baseline length L as the end effector 22 approaches the object W.

[0086] In this embodiment, after the power supply of the robot system 10 is turned on, the imaging device control unit 43 controls the drive unit 33 to move the imaging device 30 to a predetermined initial position. For example, after the power supply of the robot system 10 is turned on, the imaging device control unit 43 moves the first imaging device 31 to the predetermined initial position shown in FIG. 6, and moves the second imaging device 32 to the predetermined initial position shown in FIG. 6. The movement of the first imaging device 31 and the second imaging device 32 to their respective initial positions is performed, for example, after the power supply of the robot system 10 is turned on and before the first imaging device 31 and the second imaging device 32 are used. The movement of the first imaging device 31 and the second imaging device 32 to their respective initial positions may be performed, for example, immediately after the power supply of the robot system 10 is turned on.

[0087] For example, the imaging device control unit 43 moves the first imaging device 31 and the second imaging device 32 to their respective initial positions suitably and easily by bringing the first imaging device 31 and the second imaging device 32 into contact with each other in the circumferential direction. Note that the robot system 10 may include a sensor capable of detecting that the first imaging device 31 and the second imaging device 32 are in contact with each other in the circumferential direction.

[0088] As described above, in this embodiment, at least the first imaging device 31 moves to a predetermined initial position after the power supply of the robot system 10 is turned on. More specifically, both the first imaging device 31 and the second imaging device 32 move to a predetermined initial position after the power supply of the robot system 10 is turned on. Note that only the first imaging device 31 among the first imaging device 31 and the second imaging device 32 may move to a predetermined initial position after the power supply of the robot system 10 is turned on, or only the second imaging device 32 among the first imaging device 31 and the second imaging device 32 may move to a predetermined initial position after the power supply of the robot system 10 is turned on.

[0089] In this embodiment, when the control unit 40 moves the first imaging device 31 and the second imaging device 32 by the imaging device control unit 43, the member to which the imaging device 30 is attached, that is, the end effector 22 in this embodiment, is set to a stationary state. That is, in this embodiment, the movement of the first imaging device 31 and the movement of the second imaging device 32 are performed in a state where the member to which the first imaging device 31 and the second imaging device 32 are attached is stationary. In this embodiment, the movement of the first imaging device 31 to a predetermined initial position and the movement of the second imaging device 32 to a predetermined initial position are also performed in a state where the member (end effector 22) to which the first imaging device 31 and the second imaging device 32 are attached is stationary.

[0090] When the object W cannot be imaged by at least one of the first imaging device 31 and the second imaging device 32, the control unit 40 may move at least one of the first imaging device 31 and the second imaging device 32 so that both the first imaging device 31 and the second imaging device 32 are at positions where the object W can be imaged. The case where the object W cannot be imaged by the imaging device 30 is, for example, a case where an obstacle is disposed between the imaging device 30 and the object W and the object W does not appear in the imaging device 30. The control unit 40 may move the first imaging device 31 and the second imaging device 32 to positions that do not interfere with the operation of the end effector 22 in accordance with the operation on the object W by the end effector 22.

[0091] The distance information acquisition unit 44 acquires information regarding the distance of the object W. The information regarding the distance of the object W includes, for example, the distance from the imaging device 30 to the object W, the distance from the end effector 22 to the object W, the distance from the robot arm 21 to the object W, the distance between a plurality of objects W, and 3D point cloud data about the object W. Information on the images captured by the image sensors 31f and 32f is input to the distance information acquisition unit 44.

[0092] The distance information acquisition unit 44 receives information regarding the position of the imaging device 30 from the position acquisition unit 34 of the imaging device 30. The distance information acquisition unit 44 acquires the baseline length L based on the position information of the first imaging device 31 acquired by the position acquisition unit 34. In the present embodiment, the distance information acquisition unit 44 acquires the baseline length L based on the position information of the first imaging device 31 acquired by the first position acquisition unit 31c and the position information of the second imaging device 32 acquired by the second position acquisition unit 32c. Specifically, the distance information acquisition unit 44 calculates the distance between the optical axis AX1 of the first imaging device 31 and the optical axis AX2 of the second imaging device 32 from the circumferential direction position of the first imaging device 31 and the circumferential direction position of the second imaging device 32, and acquires the baseline length L. Note that the distance information acquisition unit 44 may acquire the baseline length L from other parts such as the imaging device control unit 43, for example.

[0093] The distance information acquisition unit 44 acquires information regarding the distance of the object W based on the acquired baseline length L, the first image acquired by the first imaging device 31, and the second image acquired by the second imaging device 32. Here, in the present embodiment, when viewed in the central axis direction, the postures of the imaging elements 31f of the first imaging device 31 and the imaging elements 32f of the second imaging device 32 are different from each other. Therefore, the distance information acquisition unit 44 rotates at least one of the first image acquired by the first imaging device 31 and the second image acquired by the second imaging device 32 to align the direction (orientation) of the first image and the direction (orientation) of the second image.

[0094] In this way, in the present embodiment, the distance information acquisition unit 44 adjusts the direction of the acquired image by rotating at least one of the first image acquired by the first imaging device 31 and the second image acquired by the second imaging device 32. The distance information acquisition unit 44 may rotate only the first image acquired by the first imaging device 31 to align the direction of the first image with the direction of the second image, or may rotate only the second image acquired by the second imaging device 32 to align the direction of the second image with the direction of the first image, or may rotate both the first image acquired by the first imaging device 31 and the second image acquired by the second imaging device 32 to align the direction of the first image and the direction of the second image. In the present embodiment, the distance information acquisition unit 44 measures the distance from the imaging device 30 to the object W using the baseline length L and the first image and the second image whose directions are aligned by rotation. The control unit 40 controls at least one of the robot arm 21 and the end effector 22 based on the information regarding the distance of the object W thus acquired.

[0095] The display unit 50 displays information based on the information regarding the distance. The information regarding the distance includes, for example, information regarding the distance of the object W and information regarding the baseline length L which is the distance between the first imaging device 31 and the second imaging device 32. The information based on the information regarding the distance may be the information regarding the distance itself or the information obtained from the information regarding the distance. For example, the current distance to the object W and the current baseline length L are displayed on the display unit 50. For example, the changes in the distance to the object W and the baseline length L may be displayed in a graphed manner on the display unit 50. The display unit 50 may have any structure as long as it can display the information based on the information regarding the distance. The display unit 50 may be provided separately from the robot 20, for example, or may be provided on the robot arm 21. The display unit 50 is controlled by the control unit 40.

[0096] According to the present embodiment, at least one of the first imaging device 31 and the second imaging device 32 attached to the end effector 22 is movable with respect to the end effector 22. Therefore, by moving at least one of the first imaging device 31 and the second imaging device 32, the distance between the first imaging device 31 and the second imaging device 32 can be changed. Thereby, the baseline length L, which is the distance between the first imaging device 31 and the second imaging device 32, can be changed.

[0097] Here, when the baseline length L is relatively large, the resolution for an object W that is relatively far from the imaging device 30 can be made relatively large, and the distance to the object W that is relatively far from the imaging device 30 can be accurately detected. However, in this case, since an object W that is relatively close to the imaging device 30 does not appear in the imaging device 30, the distance to the object W that is relatively close to the imaging device 30 cannot be detected. On the other hand, when the baseline length L is relatively small, an object W that is relatively close to the imaging device 30 can be imaged, but it is difficult to focus on an object W that is relatively far from the imaging device 30, and it is difficult to accurately detect the distance to the object W that is relatively far from the imaging device 30. As described above, depending on the magnitude of the baseline length L, the position of the object W for which the distance can be suitably detected is different. Therefore, for example, in an imaging device in which the magnitude of the baseline length L is fixed, the position of the object W with respect to the imaging device from which information regarding the distance can be suitably acquired is limited. As a result, when only the imaging device is used, the work content of the robot system may be limited.

[0098] In contrast, according to the present embodiment, as described above, the baseline length L between the first imaging device 31 and the second imaging device 32 attached to the end effector 22 can be changed. Therefore, when the distance between the end effector 22 and the object W is relatively large, the baseline length L is made relatively large, and when the distance between the end effector 22 and the object W is relatively small, the baseline length L is made relatively small. Thus, even if the distance between the end effector 22 and the object W changes significantly, the distance to the object W can be accurately measured only by the imaging device 30 attached to the end effector 22. Also, when the distance between the end effector 22 and the object W is relatively large, the baseline length L can be made relatively large, so that the distance to the object W can be detected more accurately by stereo matching. Further, when the distance between the end effector 22 and the object W is relatively small, the baseline length L can be made relatively small, so that the degree of overlap (overlapping portion) between the image captured by the first imaging device 31 and the image captured by the second imaging device 32 can be increased, and stereo matching can be performed within a relatively wide range in the images captured by each imaging device 30 to measure the distance to the object W. As a result, regardless of the distance between the end effector 22 and the object W, the robot system 10 can perform an operation on the object W. Therefore, it is possible to suppress the work content of the robot system 10 from being restricted. Thereby, the workability with respect to the object W can be improved.

[0099] Specifically, for example, operations such as searching for the object W from a relatively far distance on the workbench WB, approaching the end effector 22 to the found object W, grasping the object W with the end effector 22, and moving the grasped object W to another location can be preferably performed while acquiring the distance to the object W using only the first imaging device 31 and the second imaging device 32 attached to the end effector 22.

[0100] Further, for example, when providing equipment capable of measuring the distance to the object W on which the robot system performs work on the ceiling or the like of the place where the robot system is arranged, by making the baseline length L in the equipment different from the baseline length L of the imaging device attached to the end effector, even when the distance between the end effector and the object varies within a certain range, the robot system can perform work while grasping the distance to the object. However, in this case, there is a problem that costs are required to provide the equipment. Also, there is a problem that the robot system can only be used at the location where the equipment is provided.

[0101] On the other hand, according to the present embodiment, since the baseline length L between the first imaging device 31 and the second imaging device 32 attached to the end effector 22 can be changed, even when the distance between the end effector 22 and the object W varies within a certain range, the robot system 10 can perform work on the object W without providing the equipment provided on the ceiling or the like as described above. Thereby, the cost for providing the equipment is unnecessary. Also, the robot system 10 can be used even at a location where the above-described equipment is not provided. Therefore, the degree of freedom of the location where the robot system 10 can be used can be improved.

[0102] Further, when either the first imaging device 31 or the second imaging device 32 is at a position where it cannot image the object W, by moving the first imaging device 31 or the second imaging device 32 with respect to the end effector 22, it is easy to make it possible to image the object W with both the first imaging device 31 and the second imaging device 32 without moving the end effector 22. Thereby, information regarding the distance of the object W can be suitably acquired regardless of the position and posture of the end effector 22.

[0103] Also, at least one of the positions of the first imaging device 31 and the second imaging device 32 can be moved to a suitable position according to the moving paths of the robot arm 21 and the end effector 22 and the surrounding environment where the robot arm 21 and the end effector 22 are arranged. For example, when moving the robot arm 21 and the end effector 22 relative to the object W, the first imaging device 31 and the second imaging device 32 can be moved so that they do not come into contact with other objects or the like. Therefore, the degree of freedom of movement of the robot arm 21 and the end effector 22 can be improved.

[0104] Also, for example, by moving at least one of the first imaging device 31 and the second imaging device 32, the inertia when the robot 20 moves can also be optimized. Specifically, for example, when the end effector 22 is not gripping the object W, by arranging the first imaging device 31 and the second imaging device 32 on opposite sides with the central axis CL in between, it is easier to minimize the overall inertia of the end effector 22, the first imaging device 31, and the second imaging device 32. Thereby, it becomes easier to suitably move the end effector 22 to which the first imaging device 31 and the second imaging device 32 are attached. Also, for example, when the end effector 22 is gripping the object W, at least one of the first imaging device 31 and the second imaging device 32 can be moved to a position that functions as a counterweight with respect to the object W gripped by the first imaging device 31 and the second imaging device 32. Thereby, it is easier to minimize the overall inertia of the end effector 22, the first imaging device 31, the second imaging device 32, and the object W. Therefore, in a state where the end effector 22 is gripping the object W, it becomes easier to suitably move the end effector 22.

[0105] Also, when the distance between the object W and the first imaging device 31 and the second imaging device 32 changes within the range where the object W can be imaged by the first imaging device 31 and the second imaging device 32, the baseline length L may be increased as the first imaging device 31 and the second imaging device 32 approach the object W. Here, the greater the baseline length L, the more the detection accuracy of the distance to the object W can be improved. Therefore, as long as the object W is within the imaging range, by increasing the baseline length L as the object W is approached, the distance to the object W can be obtained more accurately, and it becomes easier to perform precise work on the object W.

[0106] For example, in the case of a robot hand where the end effector 22 grips the object W, and in the case of a tool where the end effector 22 performs work on the object W, etc., as the end effector 22 approaches the object W, the distance to the object W can be accurately obtained, so it becomes easier to perform precise work on the object W by the end effector 22. Also, when the end effector 22 is relatively far from the object W, since the baseline length L is relatively small, the area (overlapping part) where the image captured by the first imaging device 31 and the image captured by the second imaging device 32 overlap becomes larger, and the distance can be measured within a relatively wide range including the object W.

[0107] Also, according to the present embodiment, the optical axis AX1 of the first imaging device 31 and the optical axis AX2 of the second imaging device 32 are parallel to each other. Therefore, based on the first image acquired by the first imaging device 31 and the second image acquired by the second imaging device 32, it is easy to preferably obtain the distance to the object W.

[0108] Also, according to the present embodiment, the control unit 40 that controls the robot system 10 can change the baseline length L, which is the distance between the first imaging device 31 and the second imaging device 32, and the distance information acquisition unit 44 can acquire information regarding the distance of the object W based on the baseline length L. Therefore, the baseline length L can be easily changed according to the work content of the robot system 10. Also, the distance information acquisition unit 44 can easily acquire information regarding the distance of the object W.

[0109] Further, according to the present embodiment, a position acquisition unit 34 for acquiring at least the position information of the first imaging device 31 is provided, and the distance information acquisition unit 44 acquires the baseline length L based on the position information of the first imaging device 31 acquired by the position acquisition unit 34. Therefore, the distance information acquisition unit 44 can preferably acquire the information regarding the distance to the object W based on the acquired baseline length L while preferably acquiring the baseline length L.

[0110] Further, according to the present embodiment, the control unit 40 changes the baseline length L according to the work content of the robot system 10. Therefore, the baseline length L between the first imaging device 31 and the second imaging device 32 can be preferably changed according to the work content of the robot system 10. Thereby, the work can be preferably performed by the robot system 10. Specifically, for example, when performing a work of searching for the object W from a relatively far distance, by making the baseline length L relatively large, the distance to the object W at a relatively far distance can be accurately acquired, and it is easy to search for the object W. Further, for example, when performing a work of gripping the object W by the end effector 22, by making the baseline length L relatively small, the distance to the object W at a relatively small size can be acquired, and it is easy to preferably grip the object W by the end effector 22.

[0111] Further, according to the present embodiment, the distance information acquisition unit 44 adjusts the direction of the acquired image by rotating at least one of the first image acquired by the first imaging device 31 and the second image acquired by the second imaging device 32. Therefore, even if the imaging elements 31f of the first imaging device 31 and the imaging element 32f of the second imaging device 32 are arranged in different postures, the direction of the first image acquired by the first imaging device 31 and the direction of the second image acquired by the second imaging device 32 can be made the same. As a result, regardless of the relative position and relative posture between the first imaging device 31 and the second imaging device 32, the distance information acquisition unit 44 can suitably acquire information regarding the distance of the object W based on the images acquired by the respective imaging devices 30. Therefore, even when at least one of the first imaging device 31 and the second imaging device 32 is moved to set the first imaging device 31 and the second imaging device 32 at arbitrary positions and postures, information regarding the distance of the object W can be suitably acquired.

[0112] Further, according to the present embodiment, the first imaging device 31 and the second imaging device 32 are arranged around the end effector 22. Therefore, it is easy to measure the distance between the end effector 22 and the object W from the images captured by the first imaging device 31 and the second imaging device 32. Further, for example, by arranging the first imaging device 31 and the second imaging device 32 on the outer side in the radial direction of the base portion 22a as in the present embodiment, it is possible to make it difficult for the first imaging device 31 and the second imaging device 32 to interfere with the operation of gripping the object W by the end effector 22.

[0113] Further, according to the present embodiment, at least one of the first imaging device 31 and the second imaging device 32 is movable in a predetermined circumferential direction around the end effector 22. Therefore, by moving at least one of the first imaging device 31 and the second imaging device 32, the circumferential distance between the first imaging device 31 and the second imaging device 32 can be changed, and the baseline length L between the first imaging device 31 and the second imaging device 32 can be easily changed.

[0114] Also, according to the present embodiment, at least the first imaging device 31 moves to a predetermined initial position after the power of the robot system 10 is turned on. Therefore, before using the first imaging device 31, calibration of the first position acquisition unit 31c in the first imaging device 31 can be performed. Thereby, even if the first imaging device 31 is moved, the position of the first imaging device 31 can be accurately detected with reference to the predetermined initial position. In the present embodiment, both the first imaging device 31 and the second imaging device 32 move to a predetermined initial position after the power of the robot system 10 is turned on. Therefore, each position detection unit of each imaging device 30 can be calibrated, and the position of each imaging device 30 can be accurately detected. Thereby, the baseline length L between the first imaging device 31 and the second imaging device 32 can be accurately changed, and information regarding the distance of the object W can be more suitably acquired based on the baseline length L. Further, in the present embodiment, by bringing the first imaging device 31 and the second imaging device 32 into contact with each other in the circumferential direction, each of the first imaging device 31 and the second imaging device 32 can be easily moved to the initial position.

[0115] Also, according to the present embodiment, the movement of the first imaging device 31 to the predetermined initial position is performed in a state where the member to which the first imaging device 31 is attached, that is, the end effector 22 in the present embodiment, is stationary. Therefore, it is easier to move the first imaging device 31 than when the first imaging device 31 is moved to the initial position while the end effector 22 is moving. Further, since the movement of the first imaging device 31 can be suppressed while the end effector 22 is moving, it is possible to suppress the complication of the movement calculation of the end effector 22 and the movement calculation of the robot arm 21. In the present embodiment, the movement of the second imaging device 32 to the predetermined initial position is also performed in a state where the end effector 22 is stationary. Therefore, it is possible to easily move the second imaging device 32 to the initial position. Further, since the movement of the second imaging device 32 can be suppressed while the end effector 22 is moving, it is possible to further suppress the complication of the movement calculation of the end effector 22 and the movement calculation of the robot arm 21.

[0116] In the present embodiment, all of the movement of the first imaging device 31 with respect to the end effector 22 and the movement of the second imaging device 32 with respect to the end effector 22 are performed in a state where the members to which the first imaging device 31 and the second imaging device 32 are attached, that is, the end effector 22 in the present embodiment, is stationary. Therefore, while the end effector 22 is moving, the first imaging device 31 and the second imaging device 32 do not move relative to the end effector 22. As a result, it is possible to further suppress the complication of the movement calculation of the end effector 22 and the movement calculation of the robot arm 21.

[0117] Further, according to the present embodiment, a display unit 50 for displaying information based on information regarding distance is provided. Therefore, an operator or the like of the robot system 10 can easily acquire information regarding distance by looking at the display unit 50.

[0118] In the above description, a method of adjusting the direction of the acquired image by rotating at least one of the first image acquired by the first imaging device 31 and the second image acquired by the second imaging device 32 is adopted, but the method is not limited thereto. In the present embodiment, the control unit 40 may adjust the direction of the acquired image by rotating at least one of the imaging element 31f of the first imaging device 31 and the imaging element 32f of the second imaging device 32. In this case, the control unit 40 rotates at least one of the imaging element 31f of the first imaging device 31 and the imaging element 32f of the second imaging device 32 so that the long sides of the imaging element 31f and the imaging element 32f are parallel, for example. Thereby, the directions of the images acquired by the respective imaging devices 30 can be aligned without performing processing such as rotation on the images after imaging. Therefore, the load of image processing by the control unit 40 can be reduced as compared with the case where processing such as rotation is performed on the acquired images.

[0119] When rotating the imaging element 31f of the first imaging device 31, the control unit 40 may rotate the entire first imaging device 31 together with the imaging element 31f, or may rotate only the imaging element 31f among the first imaging devices 31. When rotating the imaging element 31f of the first imaging device 31, the control unit 40 rotates the imaging element 31f around the optical axis AX1. In this case, the first imaging device 31 is attached to the end effector 22 so as to be rotatable around the optical axis AX1.

[0120] When rotating the imaging element 32f of the second imaging device 32, the control unit 40 may rotate the entire second imaging device 32 together with the imaging element 32f, or may rotate only the imaging element 32f among the second imaging devices 32. When rotating the imaging element 32f of the second imaging device 32, the control unit 40 rotates the imaging element 32f around the optical axis AX2. In this case, the second imaging device 32 is attached to the end effector 22 so as to be rotatable around the optical axis AX2.

[0121] In addition, in the present embodiment, the end effector 22 may have a first holding unit that holds the first imaging device 31 immovably. In this case, the end effector 22 may have a guide rail unit 22e as a second holding unit that holds the second imaging device 32 movably. Further, the end effector 22 may have a second holding unit that holds the second imaging device 32 immovably. In this case, the end effector 22 may have a guide rail unit 22e as a first holding unit that holds the first imaging device 31 movably.

[0122] In addition, in the present embodiment, at least the first imaging device 31 may move to a predetermined end position before the power of the robot system 10 is turned off. In this case, for example, after receiving an instruction to stop the robot system 10, the control unit 40 may bring the first imaging device 31 and the second imaging device 32 into contact with each other in the circumferential direction and move both the first imaging device 31 and the second imaging device 32 to a predetermined end position. The predetermined end position may be the same as the predetermined initial position or may be different from the predetermined initial position.

[0123] For example, when a predetermined end position is the same as a predetermined initial position, the first imaging device 31 and the second imaging device 32 are located at the predetermined initial position when the power of the robot system 10 is turned on again. Therefore, it is not necessary to provide a step of moving the first imaging device 31 and the second imaging device 32 to the predetermined initial position after the power of the robot system 10 is turned on. As a result, even if the first imaging device 31 and the second imaging device 32 are moved immediately after the power of the robot system 10 is turned on, the position acquisition unit 34 can suitably acquire the positions of the respective imaging devices 30.

[0124] The movement of the first imaging device 31 to the predetermined end position is performed, for example, in a state where the member to which the first imaging device 31 is attached is stationary. Therefore, the first imaging device 31 can be easily moved to the end position. Also, while the end effector 22 is moving, the movement of the first imaging device 31 can be suppressed, so that the movement calculation of the end effector 22 and the movement calculation of the robot arm 21 can be further suppressed from becoming complicated.

[0125] The movement of the second imaging device 32 to the predetermined end position is performed, for example, in a state where the member to which the second imaging device 32 is attached is stationary. Therefore, the second imaging device 32 can be easily moved to the end position. Also, while the end effector 22 is moving, the movement of the second imaging device 32 can be suppressed, so that the movement calculation of the end effector 22 and the movement calculation of the robot arm 21 can be further suppressed from becoming complicated.

[0126] Also, with respect to the overlapping portion (the portion that is reflected in both images) of the image captured by the first imaging device 31 and the image captured by the second imaging device 32, the distance to the feature portion such as the object W reflected in the overlapping portion can be measured over the entire overlapping portion. Therefore, the control unit 40 may measure the distance over the entire overlapping portion of the image captured by the first imaging device 31 and the image captured by the second imaging device 32, and create a depth map for the overlapping portion. The depth map is, for example, an image showing distance information by differences in color, shades of color, etc.

[0127] Further, the control unit 40 may measure the distance to the object W using the images captured by the first imaging device 31 and the second imaging device 32 and the image captured by the camera unit 60. For example, when the distance to the object W measured from the images captured by the first imaging device 31 and the second imaging device 32 becomes equal to or less than a predetermined distance, the control unit 40 may turn on the power of the camera unit 60 and measure the distance to the object W using each image captured by the first imaging device 31 and the second imaging device 32 and the image captured by the camera unit 60.

[0128] Further, the control unit 40 may switch between a first imaging mode in which the distance to the object W is measured using the images captured by the first imaging device 31 and the second imaging device 32 and a second imaging mode in which the distance to the object W is measured using the image captured by the camera unit 60 according to the distance to the object W. In this case, for example, when the distance to the object W is greater than a predetermined distance, the control unit 40 may measure the distance to the object W in the first imaging mode described above, and when the distance to the object W is equal to or less than the predetermined distance, the control unit 40 may measure the distance to the object W in the second imaging mode described above.

[0129] <Second Embodiment> FIG. 7 is a perspective view showing a part of the robot system 110 of the present embodiment. In FIG. 7, illustration of the finger portion 22b of the end effector 122 and the camera unit 60 is omitted. Note that, for the configurations similar to those of the above-described embodiment, the description may be omitted by appropriately assigning the same reference numerals.

[0130] As shown in FIG. 7, in the robot system 110 of the present embodiment, the second imaging device 132 is fixed to the end effector 122. That is, in the present embodiment, the second imaging device 132 does not move relative to the end effector 122. Therefore, in the present embodiment, only the first imaging device 31 is movable in a predetermined circumferential direction around the end effector 122. In this way, in the present embodiment, one of the first imaging device 31 and the second imaging device 132 is movable in a predetermined circumferential direction around the end effector 122, and the other is fixed to a predetermined portion of the end effector 122.

[0131] The second imaging device 132 is fixed to the base portion 122a, for example. The base portion 122a has a hole portion 122f that is recessed from the front end side (+Z side) surface of the base portion 122a toward the base end side (-Z side). The hole portion 122f is, for example, a circular hole centered on the central axis CL.

[0132] A portion on the base end side (-Z side) of the second imaging device 132 is fitted into and held in the hole portion 122f. In the present embodiment, the hole portion 122f corresponds to a second holding portion that holds the second imaging device 132. That is, in the present embodiment, the end effector 122 has the hole portion 122f as the second holding portion. In the present embodiment, the second imaging device 132 is held immovably by the hole portion 122f as the second holding portion. A portion on the front end side (+Z side) of the second imaging device 132 protrudes forward from the center of the front end side surface of the base portion 122a. The base portion 122a has the same configuration as the base portion 22a of the first embodiment described above, except that the hole portion 122f is provided.

[0133] The optical axis AX2a of the second imaging device 132 is, for example, parallel to the optical axis AX1 of the first imaging device 31 and coincides with the central axis CL. The second imaging device 132 includes a cylindrical housing 132a, a lens 132e fitted into the opening on the front end side (+Z side) of the housing 132a, and an imaging element 132f disposed within the housing 132a. Each part of the second imaging device 132 can be configured in the same manner as each part of the second imaging device 32 of the first embodiment described above. Different from the second imaging device 32 of the first embodiment, the second imaging device 132 does not have the second driving unit 32b and the second position acquisition unit 32c.

[0134] In this embodiment, the baseline length L between the first imaging device 31 and the second imaging device 132 is fixed. In this embodiment, similar to the first embodiment described above, the distance information acquisition unit 44 acquires information regarding the distance of the object W based on the baseline length L and the two images acquired from the first imaging device 31 and the second imaging device 132.

[0135] Also, in this embodiment, the distance information acquisition unit 44 can also acquire information regarding the distance of the object W based on two images captured by the first imaging device 31. Specifically, for example, when the first imaging device 31 is located at the first position P1 shown by the solid line in FIG. 7, the first imaging device 31 acquires a first image, and when the first imaging device 31 is located at the second position P2 shown by the two-dot chain line in FIG. 7, the first imaging device 31 acquires a second image. In this embodiment, the distance information acquisition unit 44 can also acquire information regarding the distance of the object W based on the first image and the second image obtained in this way. More specifically, the distance information acquisition unit 44 can also acquire information regarding the distance of the object W based on the first image and the second image acquired by the first imaging device 31 and the baseline length La, which is the distance between the first imaging device 31 at the first position P1 and the first imaging device 31 at the second position P2.

[0136] The baseline length La is the distance between the optical axis AX1a of the first imaging device 31 at the first position P1 and the optical axis AX1b of the first imaging device 31 at the second position P2. The baseline length La is determined by the first position P1 and the second position P2. In the present embodiment, the control unit 40 can change the baseline length La by changing the first position P1 and the second position P2 at which the first imaging device 31 acquires an image. The control unit 40 changes the baseline length La, for example, according to the work content of the robot system 110.

[0137] In the present embodiment, the position acquisition unit 134 acquires position information regarding the first position P1 and the second position P2. The position information regarding the first position P1 and the second position P2 includes, for example, information on the first position P1, information on the second position P2, and information indicating the relative positional relationship between the first position P1 and the second position P2. In the present embodiment, the position acquisition unit 134 consists only of the first position acquisition unit 31c of the first imaging device 31. The position acquisition unit 134 acquires the position information of the first position P1 and the second position P2 based on, for example, the rotational position of the drive unit 133. In the present embodiment, the drive unit 133 consists only of the first drive unit 31b of the first imaging device 31. The distance information acquisition unit 44 acquires the baseline length La based on the position information regarding the first position P1 and the second position P2 acquired by the position acquisition unit 134.

[0138] The distance information acquisition unit 44 can select, for example, whether to acquire information regarding the distance of the object W using the images acquired by each of the first imaging device 31 and the second imaging device 132, or to acquire information regarding the distance of the object W using two images acquired by the first imaging device 31 at different first positions P1 and P2.

[0139] When acquiring information regarding the distance of the object W using the images acquired by each of the first imaging device 31 and the second imaging device 132, the distance information acquisition unit 44 adjusts the direction of the acquired images by rotating at least one of the image acquired by the first imaging device 31 and the image acquired by the second imaging device 132, in the same manner as in the first embodiment.

[0140] On the other hand, when acquiring information regarding the distance of the object W using two images acquired at different first position P1 and second position P2 by the first imaging device 31, the distance information acquisition unit 44 adjusts the direction of the acquired images by rotating at least one of the first image acquired at the first position P1 by the first imaging device 31 and the second image acquired at the second position P2 by the first imaging device 31. The distance information acquisition unit 44 may rotate only the first image acquired at the first position P1 to align the direction of the first image with the direction of the second image, or may rotate only the second image acquired at the second position P2 to align the direction of the second image with the direction of the second image, or may rotate both the first image acquired at the first position P1 and the second image acquired at the second position P2 to adjust the direction of the first image and the direction of the second image. Other configurations of the robot system 110 of the present embodiment can be the same as those of the robot system of the above-described embodiment.

[0141] According to the present embodiment, the first imaging device 31 is movable in a predetermined circumferential direction around the end effector 122, and the second imaging device 132 is fixed to a predetermined part of the end effector 122. Therefore, compared with the case where both the first imaging device 31 and the second imaging device 132 are provided movably, the structure of the robot system 110 can be easily simplified.

[0142] Further, according to the present embodiment, the first imaging device 31 captures a first image of the object W at the first position P1 and captures a second image of the object W at a second position P2 different from the first position P1. The distance information acquisition unit 44 acquires information regarding the distance of the object W based on the first image and the second image. Therefore, information regarding the distance of the object W can be acquired using only the images acquired by one first imaging device 31. Thereby, even if the second imaging device 132 is not provided, information regarding the distance of the object W can be acquired by only one first imaging device 31. Also, by changing the relative position between the first position P1 and the second position P2, the baseline length La can be changed.

[0143] Also, according to the present embodiment, the distance information acquisition unit 44 adjusts the direction of the acquired image by rotating at least one of the first image acquired by the first imaging device 31 at the first position P1 and the second image acquired by the first imaging device 31 at the second position P2. Therefore, even if the imaging elements 31f of the first imaging device 31 at the first position P1 and the imaging elements 132f of the second imaging device 132 at the second position P2 are arranged in different postures, the direction of the first image acquired at the first position P1 and the direction of the second image acquired at the second position P2 can be made to coincide. As a result, regardless of the positions of the first position P1 and the second position P2, the distance information acquisition unit 44 can suitably acquire information regarding the distance of the object W based on the image acquired by the first imaging device 31.

[0144] Also, according to the present embodiment, the control unit 40 can change the baseline length La, which is the distance between the first imaging device 31 at the first position P1 and the first imaging device 31 at the second position P2, and the distance information acquisition unit 44 acquires information regarding the distance of the object W based on the baseline length La. Therefore, using only the first imaging device 31, information regarding the distance of the object W can be acquired at different baseline lengths La.

[0145] Also, according to the present embodiment, the position acquisition unit 134 acquires position information regarding the first position P1 and the second position P2, and the distance information acquisition unit 44 acquires the baseline length La based on the position information regarding the first position P1 and the second position P2 acquired by the position acquisition unit 134. Therefore, the distance information acquisition unit 44 can suitably acquire the baseline length La and, based on the acquired baseline length La, suitably acquire information regarding the distance of the object W.

[0146] Further, according to the present embodiment, the control unit 40 changes the baseline length La according to the work content of the robot system 110. Therefore, the baseline length La, which is the distance between the first imaging device 31 at the first position P1 and the first imaging device 31 at the second position P2, can be suitably changed according to the work content of the robot system 110. Thereby, each work can be suitably performed by the robot system 110.

[0147] Further, according to the present embodiment, the distance information acquisition unit 44 adjusts the direction of the acquired image by rotating at least one of the first image acquired by the first imaging device 31 and the second image acquired by the second imaging device 32. Therefore, even if the imaging elements 31f of the first imaging device 31 and the imaging elements 32f of the second imaging device 32 are arranged in different postures, the direction of the first image acquired by the first imaging device 31 and the direction of the second image acquired by the second imaging device 32 can be made to coincide. Thereby, regardless of the relative position and relative posture between the first imaging device 31 and the second imaging device 32, etc., the distance information acquisition unit 44 can suitably acquire information regarding the distance of the object W based on the images acquired by the respective imaging devices 30. Therefore, even when at least one of the first imaging device 31 and the second imaging device 32 is moved to make the first imaging device 31 and the second imaging device 32 in arbitrary positions and postures, information regarding the distance of the object W can be suitably acquired.

[0148] In the present embodiment, the first imaging device 31 may be movable in the radial direction. In this case, by changing the radial position of the first imaging device 31, the baseline length, which is the distance between the first imaging device 31 and the second imaging device 132, can be changed. Also, in the present embodiment, the second imaging device 132 may not be provided. Even in this case, as described above, information regarding the distance of the object W can be acquired using only the first imaging device 31.

[0149] In addition, in the present embodiment, the first imaging device 31 may be movable such that the long sides of the imaging element 31f of the first imaging device 31 at the first position P1 and the long sides of the imaging element 31f of the first imaging device 31 at the second position P2 are parallel to each other. Thereby, the directions of the images captured at the first position P1 and the second position P2 by the first imaging device 31 can be aligned without performing processing such as rotation on the captured images. Therefore, compared with the case where processing such as rotation is performed on the acquired images, the load of image processing by the control unit 40 can be reduced. In this case, the first imaging device 31 is, for example, rotatably attached around the optical axis AX1. For example, the control unit 40 rotates the first imaging device 31 around the optical axis AX1 according to the circumferential position of the first imaging device 31, and adjusts it so that the long side of the imaging element 31f is always in the same direction. Thus, in the present embodiment, the control unit 40 may adjust the direction of the image acquired by the first imaging device 31 by rotating the imaging element 31f of the first imaging device 31.

[0150] Also, when only one first imaging device 31 is attached to a member that can move relative to the first imaging device 31 as in the present embodiment, the first imaging device 31 may be movable relative to the robot arm 21 or may be movable relative to the adapter 23. When the first imaging device 31 is movable relative to the robot arm 21, the first imaging device 31 may be movable in a predetermined circumferential direction around the robot arm 21. When the first imaging device 31 is movable relative to the adapter 23, the first imaging device 31 may be movable in a predetermined circumferential direction around the adapter 23. Even in these cases, the distance information acquisition unit 44 can acquire information regarding the distance of the object W based on the first image and the second image captured at different first positions P1 and P2 by the first imaging device 31.

[0151] <Third Embodiment> FIG. 8 is a view of a part of the robot system 210 of the present embodiment as seen from the tip side (+Z side) in the central axis direction. In FIG. 8, illustration of the finger portion 22b of the end effector 22 and the camera unit 60 is omitted. Note that, for the same configurations as those in the above-described embodiment, the description may be omitted by appropriately assigning the same reference numerals etc.

[0152] As shown in FIG. 8, the robot system 210 of the present embodiment includes three or more imaging devices 230 that image the object W. For example, three imaging devices 230a, 230b, and 230c are provided. In the present embodiment, the three imaging devices 230a, 230b, and 230c are arranged side by side on a predetermined axis VA. The axis VA is a virtual axis extending, for example, in a direction (the left-right direction in FIG. 8) orthogonal to both the central axis direction and the radial direction. The imaging devices 230a, 230b, and 230c are arranged at equal intervals, for example, along the axial direction of the axis VA. In the axial direction of the axis VA, the imaging device 230b is located between the imaging device 230a and the imaging device 230c. The optical axis AX3a of the imaging device 230a, the optical axis AX3b of the imaging device 230b, and the optical axis AX3c of the imaging device 230c extend, for example, in the central axis direction and are parallel to each other.

[0153] The imaging elements 235a of the imaging device 230a, the imaging elements 235b of the imaging device 230b, and the imaging elements 235c of the imaging device 230c are rectangular in shape when viewed in the central axis direction. In the present embodiment, the imaging elements 235a, 235b, and 235c are arranged in the same posture. In the present embodiment, the three imaging devices 230a, 230b, and 230c are arranged such that the long sides of the imaging elements 235a, 235b, and 235c in the three imaging devices 230a, 230b, and 230c are parallel to each other.

[0154] The robot system 210 includes a holding member 230d that holds three imaging devices 230a, 230b, and 230c, and a slider 230e that connects the holding member 230d to the base 22a of the end effector 22. The three imaging devices 230a, 230b, and 230c are fixed to the holding member 230d in a non-relative movable state with respect to each other. The slider 230e is connected to a guide rail portion 22e provided on the base 22a of the end effector 22, similarly to the sliders 31d and 32d of the first embodiment. The slider 230e is movable circumferentially around the base 22a along the guide rail portion 22e.

[0155] Although not shown in the figure, the robot system 210 includes a drive unit that moves the slider 230e circumferentially. When the slider 230e is moved circumferentially along the guide rail portion 22e by the drive unit, the holding member 230d and the three imaging devices 230a, 230b, and 230c held by the holding member 230d move circumferentially. In the present embodiment, the three imaging devices 230a, 230b, and 230c move circumferentially while the long sides of the imaging elements 235a, 235b, and 235c remain parallel to each other.

[0156] In the present embodiment, the control unit 40 acquires information regarding the distance of the object W based on the image information of the object W acquired by two of the three imaging devices 230a, 230b, and 230c. For example, the control unit 40 selects two imaging devices 230 from among the three imaging devices 230a, 230b, and 230c, and acquires information regarding the distance of the object W based on the image information acquired by the selected two imaging devices 230. There are three patterns for the two selected imaging devices 230: the imaging device 230a and the imaging device 230b, the imaging device 230b and the imaging device 230c, and the imaging device 230a and the imaging device 230c.

[0157] The baseline length L3, which is the distance between the imaging device 230a and the imaging device 230b, and the baseline length L4, which is the distance between the imaging device 230a and the imaging device 230c, are different from each other. That is, when the imaging devices 230a and 230b are selected as the two imaging devices 230 and when the imaging devices 230a and 230c are selected as the two imaging devices 230, the baseline lengths are different. Thus, in the present embodiment, when the control unit 40 acquires information regarding the distance of the object W, the control unit 40 can change the baseline length by changing the two selected imaging devices 230. The baseline length L3 is, for example, smaller than the baseline length L4. The baseline length L3 is the distance between the optical axis AX3a of the imaging device 230a and the optical axis AX3b of the imaging device 230b. The baseline length L4 is the distance between the optical axis AX3a of the imaging device 230a and the optical axis AX3c of the imaging device 230c.

[0158] Note that the baseline length, which is the distance between the imaging device 230b and the imaging device 230c, is, for example, the same as the baseline length L3, which is the distance between the imaging device 230a and the imaging device 230b. The baseline length, which is the distance between the imaging device 230b and the imaging device 230c, is the distance between the optical axis AX3b of the imaging device 230b and the optical axis AX3c of the imaging device 230c.

[0159] The control unit 40 changes the two selected imaging devices 230 and changes the baseline length according to, for example, the work content of the robot system 210. The control unit 40 acquires information regarding the distance of the object W based on the images acquired by the two selected imaging devices 230 and the baseline length between the two imaging devices 230 by the distance information acquisition unit 44. The control unit 40 controls at least one of the robot arm 21 and the end effector 22 based on the information regarding the distance of the object W thus acquired.

[0160] In this embodiment, the control unit 40 controls at least one of the robot arm 21 and the end effector 22 connected to the robot arm 21 based on the information of the images acquired by two of the three imaging devices 230a, 230b, and 230c. In this embodiment, the control unit 40 selects two of the three imaging devices 230 and controls at least one of the robot arm 21 and the end effector 22 based on the information of the images acquired by the selected two imaging devices 230.

[0161] Specifically, for example, when the object W is not shown in at least one of the images captured by the two selected imaging devices 230, the control unit 40 moves at least one of the robot arm 21 and the end effector 22 so that the object W can be captured by both of the two selected imaging devices 230.

[0162] Note that, for example, when the object W is not shown in at least one of the images captured by the two selected imaging devices 230, the control unit 40 may move the three imaging devices 230 in the circumferential direction so that the object W can be captured by both of the two selected imaging devices 230.

[0163] In this embodiment, the control unit 40 controls so as to synchronize the imaging by at least two of the three imaging devices 230a, 230b, and 230c. The control unit 40 controls so as to synchronize the imaging by the two selected imaging devices 230 described above, for example.

[0164] According to this embodiment, the control unit 40 acquires information regarding the distance of the object W based on the information of the images of the object W acquired by two of the three imaging devices 230a, 230b, and 230c. Therefore, by changing which two imaging devices 230 are used to acquire the images according to the work content of the robot system 210 and the object W and the like, information regarding the distance of the object W can be suitably acquired. In this embodiment, for example, the baseline length can be changed depending on whether the images acquired by the two imaging devices 230a and 230b are used or the images acquired by the two imaging devices 230a and 230c are used. Therefore, by changing the two imaging devices 230 that are appropriately selected according to the distance to the object W and the like, information regarding the distance of the object W can be suitably acquired.

[0165] Also, according to this embodiment, the control unit 40 selects two imaging devices 230 from among the three imaging devices 230a, 230b, and 230c, and acquires information regarding the distance of the object W based on the information of the images acquired by the selected two imaging devices 230. Therefore, when acquiring information regarding the distance of the object W, imaging may be performed by two of the three imaging devices 230a, 230b, and 230c, and imaging does not have to be performed by the remaining one imaging device 230. Therefore, the load on the control unit 40 when acquiring information regarding the distance of the object W can be reduced.

[0166] Also, according to this embodiment, the control unit 40 controls at least one of the robot arm 21 and the end effector 22 connected to the robot arm 21 based on the information of the images acquired by two of the three imaging devices 230a, 230b, and 230c. Therefore, information such as the position of the object W and the environment in which the robot system 210 is arranged can be acquired from the images acquired by the two imaging devices 230, and the robot arm 21 and the end effector 22 can be suitably moved according to the work content of the robot system 210 and the like.

[0167] Also, according to the present embodiment, the control unit 40 selects two imaging devices 230 from among the three imaging devices 230a, 230b, and 230c, and controls at least one of the robot arm 21 and the end effector 22 connected to the robot arm 21 based on the information of the images acquired by the two selected imaging devices 230. Therefore, when controlling at least one of the robot arm 21 and the end effector 22 based on the information of the images acquired by the two imaging devices 230, it is not necessary to perform imaging with the remaining one imaging device 230. Thereby, the load on the control unit 40 when controlling the robot arm 21 and the end effector 22 can be reduced.

[0168] Also, according to the present embodiment, the control unit 40 controls so as to synchronize the imaging by at least two of the three imaging devices 230a, 230b, and 230c. Therefore, the object W can be preferably imaged at the same timing by at least two imaging devices 230. Thereby, information regarding the distance of the object W can be preferably acquired based on the images acquired by at least two imaging devices 230.

[0169] Also, according to the present embodiment, the three imaging devices 230a, 230b, and 230c are arranged side by side on a predetermined axis VA. Therefore, as in the present embodiment, the imaging devices 230a, 230b, and 230c can be arranged side by side with the postures of the imaging elements 235a, 235b, and 235c aligned. Thereby, even when acquiring information regarding the distance of the object W using the images acquired by any two of the three imaging devices 230a, 230b, and 230c, it is easy to acquire information regarding the distance of the object W without rotating the images acquired by the imaging devices 230 or the like. Therefore, the load on the control unit 40 when acquiring information regarding the distance of the object W can be reduced.

[0170] Further, according to the present embodiment, the optical axes AX3a, AX3b, and AX3c of the three imaging devices 230a, 230b, and 230c are parallel to each other. Therefore, regardless of which two of the three imaging devices 230a, 230b, and 230c are used to acquire an image, information regarding the distance of the object W can be suitably acquired from the two images.

[0171] Further, according to the present embodiment, the three imaging devices 230a, 230b, and 230c are arranged such that the long sides of the image sensors 235a, 235b, and 235c in the three imaging devices 230a, 230b, and 230c are parallel to each other. Therefore, it is easy to acquire information regarding the distance of the object W from the acquired images without rotating the images acquired by the imaging device 230 or the like. Thereby, the load on the control unit 40 when acquiring information regarding the distance of the object W can be reduced.

[0172] In the present embodiment, four or more imaging devices 230 may be arranged side by side on a predetermined axis VA. In the three or more imaging devices 230 arranged side by side on the axis VA, the distances between two adjacent imaging devices 230 may be different from each other.

[0173] <Fourth Embodiment> FIG. 9 is a view of a part of the robot system 310 of the present embodiment as seen from the tip side (+Z side) in the central axis direction. In FIG. 9, illustration of the finger portion 22b of the end effector 22 and the camera unit 60 is omitted. Note that, for the same configurations as those in the above-described embodiments, the description may be omitted by appropriately assigning the same reference numerals.

[0174] As shown in FIG. 9, the robot system 310 of the present embodiment includes a connecting member 336 that connects the first imaging device 331 and the second imaging device 332. The connecting member 336 is, for example, a guide rail. The connecting member 336 has a groove 336a that extends linearly. The groove 336a is, for example, a groove that is recessed from the tip side (+Z side) toward the base end side (-Z side). The groove 336a is open, for example, at both ends in the direction in which the groove 336a extends.

[0175] The robot system 310 includes a first slider 331g that attaches the first imaging device 331 to the connecting member 336, and a second slider 332g that attaches the second imaging device 332 to the connecting member 336. The first slider 331g is fixed to the first imaging device 331. The second slider 332g is fixed to the second imaging device 332. The first slider 331g and the second slider 332g are fitted into the groove 336a so as to be movable in the direction in which the groove 336a extends. Thus, in the present embodiment, the first imaging device 331 and the second imaging device 332 are connected by the connecting member 336 via the first slider 331g and the second slider 332g.

[0176] The first slider 331g and the second slider 332g are restricted from relative movement and relative rotation with respect to the connecting member 336 in directions other than the direction in which the groove 336a extends. The first slider 331g is movable in the circumferential direction and is rotatably attached about the optical axis AX1c of the first imaging device 331. The second slider 332g is movable in the circumferential direction and is rotatably attached about the optical axis AX2c of the second imaging device 332.

[0177] The first imaging device 331 and the second imaging device 332 are movable in the circumferential direction around the base 22a of the end effector 22. In the present embodiment, the first imaging device 331, together with the first slider 331g, is rotatable about the optical axis AX1c of the first imaging device 331. In the present embodiment, the second imaging device 332, together with the second slider 332g, is rotatable about the optical axis AX2c of the second imaging device 332. The long sides of the imaging element 331f of the first imaging device 331 and the long sides of the imaging element 332f of the second imaging device 332 are arranged parallel to each other, for example, and are arranged parallel to the direction in which the groove 336a extends.

[0178] For example, when the first imaging device 331 and the second imaging device 332 move in the circumferential direction from the position indicated by the two-dot chain line to the position indicated by the solid line in FIG. 9, the connecting member 336 that connects the first imaging device 331 and the second imaging device 332 also moves according to the positions of the first imaging device 331 and the second imaging device 332. In FIG. 9, for example, the connecting member 336 moves upward while rotating around the axis in the central axis direction.

[0179] The first imaging device 331 and the second imaging device 332 relatively move with respect to the connecting member 336 in the direction in which the groove 336a extends according to the circumferential position. The relative change in the positions of the first imaging device 331 and the second imaging device 332 in the direction in which the groove 336a extends causes a change in the baseline length, which is the distance between the first imaging device 331 and the second imaging device 332. The baseline length between the first imaging device 331 and the second imaging device 332 is the distance between the optical axis AX1c of the first imaging device 331 and the optical axis AX2c of the second imaging device 332. For example, when the first imaging device 331 and the second imaging device 332 move in the circumferential direction from the position indicated by the two-dot chain line to the position indicated by the solid line in FIG. 9, the first imaging device 331 and the second imaging device 332 move in a direction approaching each other, and the baseline length becomes smaller.

[0180] Here, the first slider 331g and the second slider 332g are restricted from relative movement and relative rotation with respect to the connecting member 336 in a direction other than the direction in which the groove 336a extends. Therefore, the first slider 331g and the second slider 332g rotate around the optical axes AX1c and AX2c of the respective imaging devices so as to maintain the relative posture with the connecting member 336 in accordance with at least one change in the position and posture of the connecting member 336 accompanying the movement of the first imaging device 331 and the second imaging device 332. As a result, the first imaging device 331 to which the first slider 331g is fixed and the second imaging device 332 to which the second slider 332g is fixed also rotate around the respective optical axes AX1c and AX2c so as to maintain the relative posture with the connecting member 336. Therefore, even when the first imaging device 331 and the second imaging device 332 move in the circumferential direction, the relative posture between the imaging element 331f of the first imaging device 331 and the imaging element 332f of the second imaging device 332 can be maintained. That is, regardless of the positions of the first imaging device 331 and the second imaging device 332, the long sides of the imaging element 331f of the first imaging device 331 and the long sides of the imaging element 332f of the second imaging device 332 are kept parallel. Thus, in the present embodiment, the connecting member 336 can hold the first imaging device 331 and the second imaging device 332 while maintaining the relative posture between the first imaging device 331 and the second imaging device 332 in a predetermined posture. Other configurations of the robot system 310 can be the same as those of the robot systems in the above-described respective embodiments.

[0181] According to the present embodiment, as described above, at least one of the first imaging device 331 and the second imaging device 332 moves so that the long sides of the imaging element 331f of the first imaging device 331 and the imaging element 332f of the second imaging device 332 are parallel to each other. Therefore, information regarding the distance of the object W can be suitably acquired based on the image acquired by the first imaging device 331 and the image acquired by the second imaging device 332 without rotating the images acquired by the respective imaging devices.

[0182] Further, according to the present embodiment, a connecting member 336 for connecting the first imaging device 331 and the second imaging device 332 is provided, and the connecting member 336 can hold the first imaging device 331 and the second imaging device 332 while maintaining the relative posture between the first imaging device 331 and the second imaging device 332 in a predetermined posture. Therefore, for example, even if a driving unit for rotating the first imaging device 331 and the second imaging device 332 around the respective optical axes AX1c and AX2c is not provided, the connecting member 336 can easily maintain the relative posture between the imaging element 331f of the first imaging device 331 and the imaging element 332f of the second imaging device 332 in a posture where the long sides are parallel to each other. As a result, even if at least one of the first imaging device 331 and the second imaging device 332 is moved to change the baseline length, information regarding the distance of the object W can be easily and suitably obtained based on the images acquired by the first imaging device 331 and the second imaging device 332.

[0183] In the present embodiment, for example, only the imaging element 331f of the first imaging device 331 may be rotatable around the optical axis AX1c, or only the imaging element 332f of the second imaging device 332 may be rotatable around the optical axis AX2c. In this case, at least one of the imaging element 331f of the first imaging device 331 and the imaging element 332f of the second imaging device 332 may be movable so that the long sides of the imaging element 331f of the first imaging device 331 and the imaging element 332f of the second imaging device 332 are parallel to each other.

[0184] <Fifth Embodiment> FIG. 10 is a view of a part of the robot system 410 of the present embodiment as seen from the tip side (+Z side) in the central axis direction. In FIG. 10, illustration of the finger portion 22b of the end effector 22 and the camera unit 60 is omitted. FIG. 11 is a diagram for explaining a part of the procedure for the robot system 410 of the present embodiment to acquire information regarding the distance of the object W. Note that, for the same configurations as those in the above-described embodiments, the description may be omitted by appropriately assigning the same reference numerals.

[0185] In this embodiment, three or more imaging devices 430 are provided. For example, 24 imaging devices 430 are provided. The plurality of imaging devices 430 are arranged side by side along the circumferential direction around the end effector 22. That is, in this embodiment, three or more imaging devices 430 are arranged side by side on a predetermined circumference. In this embodiment, the predetermined circumference is a circumference centered on the central axis CL. The plurality of imaging devices 430 are arranged at equal intervals, for example, along the circumferential direction over one turn. That is, in this embodiment, three or more imaging devices 430 are arranged at equal intervals on a predetermined circumference. The optical axes AX4 of three or more imaging devices 430 are parallel to each other.

[0186] The long side of the imaging element 435 of each imaging device 430 is orthogonal to the radial direction passing through the optical axis AX4 of each imaging device 430 when viewed in the central axis direction. When the number of imaging devices 430 is N, the N imaging devices 430 are arranged symmetrically N times around the central axis CL. That is, in this embodiment, the 24 imaging devices 430 are arranged symmetrically 24 times around the central axis CL. The imaging device 430 is fixed to the end effector 22, for example. More specifically, the imaging device 430 is fixed to the outer peripheral surface of the base 22a, for example. That is, the end effector 22 has a holding portion that holds three or more imaging devices 430 on the outer peripheral surface of the base 22a, for example.

[0187] In this embodiment, the control unit 40 selects two images from among three or more images acquired by three or more imaging devices 430, and acquires information regarding the distance of the object W based on the information of the two selected images. For example, the control unit 40 selects two images from among 24 images respectively acquired by 24 imaging devices 430 based on the information regarding the occlusion of the object W. The information regarding the occlusion of the object W includes, for example, information on whether the object W is shown in the image, information on the occlusion state of the object W, and information on the ratio of the portion of the object W shown in the image. For example, the control unit 40 selects two images in which the object W is most preferably shown from among the 24 acquired images. The control unit 40 acquires information regarding the distance of the object W based on the two selected images.

[0188] Here, as an example, a case where the control unit 40 selects the image F1 acquired by the imaging device 431 and the image F2 acquired by the imaging device 432 among the plurality of imaging devices 430 will be described. The imaging element 435a of the imaging device 431 and the imaging element 435b of the imaging device 432 are arranged in different postures from each other. In this case, as shown in FIG. 11, the control unit 40 cuts out a part of the two acquired images F1 and F2 along a rectangular frame Fs. The long side of the rectangular frame Fs is parallel to the virtual line IL connecting the optical axis of the imaging device 431 and the optical axis of the imaging device 432. The control unit 40 acquires information regarding the distance of the object W based on a part of the two cut-out images F1 and F2.

[0189] In this embodiment, the control unit 40 selects two images from among three or more images acquired by three or more imaging devices 430, and controls at least one of the robot arm 21 and the end effector 22 connected to the robot arm 21 based on the information of the two selected images. In this embodiment, the control unit 40 controls so as to synchronize the imaging by three or more imaging devices 430. Other configurations of the robot system 410 can be the same as those of the other configurations of the robot systems in the above-described embodiments.

[0190] According to the present embodiment, the control unit 40 selects two images from among three or more images acquired by three or more imaging devices 430, and acquires information regarding the distance of the object W based on the information of the two selected images. Therefore, two images that are most suitable for acquiring information regarding the distance of the object W can be selected from among the three or more images acquired by the three or more imaging devices 430, and information regarding the distance of the object W can be acquired. Thereby, information regarding the distance of the object W can be suitably acquired according to the work content of the robot system 410, the environment in which the robot system 410 is arranged, the position and posture of the robot arm 21, and the like.

[0191] Further, according to the present embodiment, the control unit 40 selects two images from among the three or more images acquired by the plurality of imaging devices 430 based on the information regarding the occlusion of the object W. Therefore, even when at least a part of the object W cannot be imaged by some of the imaging devices 430 due to an occluding object or the like, two images in which the object W is suitably imaged can be preferably selected. Thereby, even when a part of the object W is occluded by an occluding object or the like, it is easy to suitably acquire information regarding the distance of the object W.

[0192] Further, according to the present embodiment, the control unit 40 selects two images from among the three or more images acquired by the three or more imaging devices 430, and controls at least one of the robot arm 21 and the end effector 22 connected to the robot arm 21 based on the information of the two selected images. Therefore, by selecting two images including the optimum information for moving the robot arm 21 and the end effector 22 from among the plurality of images acquired by each imaging device 430, the robot arm 21 and the end effector 22 can be suitably moved.

[0193] Further, according to the present embodiment, three or more imaging devices 430 are arranged side by side on a predetermined circumference. Therefore, a relatively large number of imaging devices 430 can be arranged side by side, for example, around the base 22a of the end effector 22 as in the present embodiment. As a result, compared with the case where the same number of imaging devices 430 are arranged linearly, the imaging devices 430 are less likely to protrude from the robot 20. Therefore, even if a relatively large number of imaging devices 430 are attached to the robot 20, the robot 20 can be easily moved. Further, since a plurality of imaging devices 430 are arranged side by side along the circumference, it is easy to image the object W from various angles by the plurality of imaging devices 430. Therefore, it is easier to more suitably acquire information regarding the distance of the object W using the plurality of imaging devices 430.

[0194] Further, according to the present embodiment, three or more imaging devices 430 are arranged at equal intervals on a predetermined circumference. Therefore, compared with the case where the imaging devices 430 are arranged at unequal intervals, there is less likelihood of bias in the number of imaging devices 430 capable of imaging the object W depending on the position and orientation of the member (for example, the end effector 22) to which the imaging devices 430 are attached. As a result, regardless of the position and orientation of the member to which the imaging devices 430 are attached, it is easy to acquire information regarding the distance of the object W using the imaging devices 430.

[0195] In the present embodiment, the control unit 40 may select two images from among three or more images acquired by the plurality of imaging devices 430 based on at least one of the distance information related to the object W obtained in advance, the information related to the occlusion of the object W, the focal length information of the imaging devices 430, and the information related to the shape change of the images obtained by three or more imaging devices 430.

[0196] The distance information related to the object W required in advance includes, for example, the distance from the robot 20 to the object W when the robot 20 and the object W are arranged at the initial position with the initial posture, the distance to the shield arranged near the object W, and the distance between a plurality of objects W when a plurality of objects W are arranged at the initial position with the initial posture. The distance from the robot 20 to the object W includes, for example, the distance from a certain part of the robot arm 21 to the object W, the distance from a certain part of the end effector 22 to the object W, and the distance from a certain part of the adapter 23 to the object W. The control unit 40 can select two images based on the distance information related to the object W required in advance, so as to select two images with a suitable baseline length according to the position of the object W, and can preferably obtain the information related to the distance of the object W.

[0197] The control unit 40 can select two images based on the focal length information of the imaging device 430, so as to select two images with a suitable baseline length according to the focal length of the imaging device 430. Here, for example, when the zoom ratio of the imaging device 430 is relatively large and two images with a relatively large baseline length are selected, the overlapping part of the two images (the range of the image of the feature part that is reflected overlappingly) becomes small. Therefore, for example, when the zoom ratio of the imaging device 430 is relatively large, by selecting two images with a relatively small baseline length, the overlapping part of the two images can be enlarged. Thereby, based on the two images, the distance to the object W can be obtained more preferably.

[0198] Figures 12A to 12C are diagrams for explaining that the overlapping portion of two images changes according to the baseline length and the zoom ratio. Figure 12A shows an example when the zoom ratio of the imaging device 430 is relatively small and two images F1a and F2a with a relatively large baseline length are selected. Figure 12B shows an example when the zoom ratio of the imaging device 430 is relatively large and two images F1b and F2b with a relatively large baseline length are selected. Figure 12C shows an example when the zoom ratio of the imaging device 430 is relatively large and two images F1c and F2c with a relatively small baseline length are selected. In Figures 12A to 12C, the object W is exemplified as a tree T and a car V.

[0199] In the case shown in Figure 12A, that is, when the zoom ratio of the imaging device 430 is relatively small and two images F1a and F2a with a relatively large baseline length are selected, it is assumed that the entire tree T and the entire car V are reflected in the two images F1a and F2a. In this case, the overlapping portion (the range of the image of the feature portion that is reflected overlapped) OPa of the two images F1a and F2a includes the entire tree T and the entire car V. Therefore, the distance to each object W can be obtained for the entire tree T and the entire car V.

[0200] On the other hand, as shown in Figure 12B, when the zoom ratio is made larger than the case shown in Figure 12A and two images F1b and F2b are selected so that the baseline length is the same as that in the case shown in Figure 12A, while the range reflected in each image F1b and F2b becomes narrower than the range reflected in each image F1a and F2a in Figure 12A, since the baseline length remains relatively large, the deviation in the position where the object W is reflected in each image F1b and F2b remains relatively large. Therefore, as shown in Figure 12B, in each image F1b and F2b, a part of the object W may be cut off, and the overlapping portion of the images F1b and F2b may become smaller.

[0201] In the example of FIG. 12B, most of the left side of the tree T is missing in the image F1b, and the right side of the vehicle V is missing in the image F2b. In this case, the overlapping portion OPb includes only a part of the tree T and a part of the vehicle V. Therefore, even if the entire tree T or the entire vehicle V is reflected in one of the images, the distance to the object W cannot be obtained for the portions not included in the overlapping portion OPb.

[0202] On the other hand, even when the zoom ratio is the same as that shown in FIG. 12B, by selecting two images F1c and F2c whose baseline length is smaller than that shown in FIG. 12B as shown in FIG. 12C, the deviation in the position where the object W is reflected in each of the images F1c and F2c can be reduced. As a result, the range in which the object W is reflected in each of the images F1c and F2c can be increased, and the overlapping portion OPc of each of the images F1c and F2c can be increased. In the example of FIG. 12C, the entire tree T and the entire vehicle V are reflected in each of the images F1c and F2c. That is, the overlapping portion OPc includes the entire tree T and the entire vehicle V as in FIG. 12A. Therefore, the distance to each object W can be obtained for the entire tree T and the entire vehicle V.

[0203] As described above, when the zoom ratio of the imaging device 430 is made relatively large as in FIGS. 12B and 12C, by selecting two images F1c and F2c whose baseline length is relatively small as in FIG. 12C, the overlapping portion OPc of the two images F1c and F2c can be increased. Thereby, based on the two images F1c and F2c, the distance to the object W can be obtained more suitably.

[0204] The information regarding the shape change of the images obtained by three or more imaging devices 430 includes, for example, information about the difference in the appearance of the object W shown in any two images among the three or more imaging devices 430. The difference in the appearance of the object W in the images captured by different imaging devices 430 varies in magnitude depending on the shape of the object W and the shadows caused by the illumination irradiating the object W. The information regarding the shape change of the images obtained by three or more imaging devices 430 includes, for example, the degree of consistency in the appearance of the object W in two images, the shape information of the object W, and the information about the illumination irradiating the object W. The control unit 40 may perform matching for all combinations of selecting two images from the plurality of images acquired by each imaging device 430, and obtain the degree of consistency in the appearance of the object W shown in each image as a parameter. Also, information regarding the shape change of the images may be input to the control unit 40 in advance.

[0205] Here, the greater the baseline length between two imaging devices 430, the more likely the appearance of the object W in the images acquired by the two imaging devices 430 will differ significantly. When the appearance of the object W differs significantly to a certain extent, it becomes difficult to perform matching between the two images. Therefore, in such a case, by reducing the baseline length, it is possible to suppress a significant difference in the appearance of the object W and make it easier to perform matching between the two images. Thus, based on the two images, information regarding the distance of the object W can be suitably obtained.

[0206] Therefore, by the control unit 40 selecting two images based on the information regarding the shape change of the images obtained by three or more imaging devices 430, it is possible to select two images with a suitable baseline length based on the difference in the appearance of the object W for each image. That is, for example, when the object W is an object whose appearance differs greatly for each of the plurality of imaging devices 430, by selecting two images with a smaller baseline length, information regarding the distance of the object W can be suitably obtained based on the two selected images.

[0207] Also, in the present embodiment, the control unit 40 may select two imaging devices 430 from among three or more imaging devices 430 based on at least one of the distance information related to the object W obtained in advance, the information related to the occlusion of the object W, the focal length information of the imaging device 430, and the information related to the shape change of the images obtained by the three imaging devices 430. In this case, imaging is performed only with the two selected imaging devices 430, and the information regarding the distance of the object W can be selected. Therefore, the load on the control unit 40 can be reduced as compared with the case of selecting two images from among the images acquired by performing imaging with all the imaging devices 430.

[0208] Also, in the present embodiment, when information such as the shape of the object W and the difference in the appearance of the luminance of the object W is known in advance, the control unit 40 may determine the imaging device 430 to be selected based on such information. The shape of the object W and the difference in the appearance of the luminance of the object W may be acquired from the distance information related to the object W obtained in advance.

[0209] Also, in the present embodiment, three or more imaging devices 430 may be arranged at non-uniform intervals on a predetermined circumference. Also, three or more imaging devices 430 may be arranged side by side on a predetermined axis, similar to the imaging devices 230a, 230b, 230c of the third embodiment described above. In this case, three or more imaging devices 430 may be arranged such that the long sides of the image sensors 435 in the three or more imaging devices 430 are parallel. Three or more imaging devices 430 may be arranged in a matrix. Also, three or more imaging devices 430 may be a TOF camera (Time Of Flight Camera).

[0210] Further, in the above-described embodiment, the three or more imaging devices 430 are arranged around the end effector 22, but the present invention is not limited to this. The three or more imaging devices 430 may be arranged around any one of the robot arm 21, the end effector 22 connected to the robot arm 21, and the adapter 23 for attaching the end effector 22 to the robot arm 21. Even when the three or more imaging devices 430 are arranged around the robot arm 21 or the adapter 23, the same effects as those obtained when the three or more imaging devices 430 are arranged around the end effector 22 described above can be obtained.

[0211] When the three or more imaging devices 430 are arranged around the robot arm 21, the robot arm 21 may have a holding portion that holds the three or more imaging devices 430 that image the object W. Further, when the three or more imaging devices 430 are arranged around the adapter 23, the adapter 23 may have a holding portion that holds the three or more imaging devices 430 that image the object W. Even in these cases, the control unit 40 may acquire information regarding the distance of the object W based on the information of the images of the object W acquired by two of the three or more imaging devices 430 held by the robot arm 21 or the adapter 23.

[0212] Further, the control unit 40 may perform the operation of acquiring information regarding the distance of the object W based on the images acquired by two of the three or more imaging devices 430 a plurality of times using different combinations of two imaging devices 430. In this case, the control unit 40 can acquire information regarding the distance of the object W more accurately by comparing the information regarding the distance of the object W acquired by the plurality of operations. In particular, when at least a part of the object W is shielded by a shielding object as viewed from at least some of the imaging devices 430, by using the plurality of pieces of information obtained by using a plurality of combinations of two imaging devices 430, the influence of the shielding by the shielding object can be minimized while acquiring information regarding the distance of the object W.

[0213] Further, after the control unit 40 has imaged the object W with all the imaging devices 430, the control unit 40 may move the end effector 22 by moving the robot arm 21 or the like, and image the object W again with all the imaging devices 430 from another location. Thereby, the control unit 40 can acquire images of the object W imaged from multiple angles. At this time, when the number of imaging devices 430 is relatively large, even if the number of times of moving the end effector 22 for imaging is small, a large number of images of the object W imaged from different angles can be acquired. At this time, information such as the position and orientation of the end effector 22 and the position and orientation of the imaging device 430 that captured the image may be associated with each of the acquired images.

[0214] As described above, when imaging the object W multiple times from different positions, the control unit 40 may control the robot system 410 by visual servoing using the acquired images. In this case, the control unit 40 controls, for example, the imaging device 430 to move to a position where the imaging device 430 can image the image of the target object W. Here, when the image of the object W imaged by the imaging device 430 at the initial position is significantly different from the image of the target object W, it is difficult to match the images, and it may be difficult to move the imaging device 430 to a position where the image of the target object W can be imaged.

[0215] On the other hand, by imaging the object W multiple times from different positions and acquiring images of the object W imaged from different angles, it is easier to bring the images imaged from different angles closer to the target image by using these images as intermediate images. That is, it is easier to appropriately move the imaging device 430 to a position where the image of the target object W can be imaged. Further, when distance information from the object W at the time of imaging the intermediate image is associated with the intermediate image, the control unit 40 may arrange the intermediate images in the order in which they were imaged from a position far from the object W, and gradually bring the image imaged by the imaging device 430 closer to the target image through the plurality of intermediate images in that order.

[0216] In addition, in the present embodiment, the robot system 410 may have a plurality of general-purpose cameras that can only capture an image of the object W, in addition to the plurality of imaging devices 430 described above. In this case, the control unit 40 may construct a 3D model of the object W using the plurality of images captured by the plurality of general-purpose cameras. By using the 3D model, for example, the acquisition accuracy of information regarding the distance of the object W using the plurality of imaging devices 430 can be further improved. In this case, as an example, 12 imaging devices 430 may be provided at equal intervals along the circumferential direction, and two general-purpose cameras may be provided side by side in the circumferential direction between adjacent imaging devices 430. In this case, for example, 24 general-purpose cameras may be provided. As the general-purpose camera, for example, a relatively inexpensive camera such as those used in smartphones can be used.

[0217] Further, in the present embodiment, the control unit 40 may perform Simultaneous Localization and Mapping (SLMA). That is, the control unit 40 may simultaneously perform self-position estimation of the robot system 410 and map creation of the environment in which the robot system 410 is arranged. In this case, when a relatively large number of imaging devices 430 are provided, a relatively large amount of 3D point cloud data about the environment can be easily acquired by the plurality of imaging devices 430. Therefore, it is easy to create a map of the environment in which the robot system 410 is arranged.

[0218] <Sixth Embodiment> FIG. 13 is a perspective view showing the robot system 510 of the present embodiment. Note that, for the same configurations as those in the above-described embodiments, the description may be omitted by appropriately assigning the same reference numerals.

[0219] As shown in FIG. 13, in the robot system 510 of the present embodiment, the imaging device 530 includes a first imaging device 531 attached to the robot arm 521 and a second imaging device 32 attached to the end effector 22. The first imaging device 531 is disposed, for example, around the fifth arm portion 524e. The first imaging device 531 is connected to a guide rail portion 521a provided on the fifth arm portion 524e by a structure similar to that in which the first imaging device 31 was connected to the guide rail portion 22e in the first embodiment.

[0220] The guide rail portion 521a is annular and surrounds the fifth arm portion 524e. The first imaging device 531 is movable along the guide rail portion 521a in a predetermined circumferential direction around the robot arm 521. The robot arm 521 has the same configuration as the robot arm 21 of the first embodiment except for the point where the guide rail portion 521a is provided. Other configurations of the robot system 510 of the present embodiment can be the same as those of the robot systems of the above-described embodiments.

[0221] According to the present embodiment, the first imaging device 531 and the second imaging device 32 are attached to different members and are relatively movable with respect to the attached members. Therefore, it is possible to suppress the movement of the first imaging device 531 and the movement of the second imaging device 32 from being inhibited by the other imaging device as compared with the case where the two imaging devices are attached to the same member. Thereby, each of the first imaging device 531 and the second imaging device 32 can be suitably relatively moved with respect to the attached members.

[0222] <Seventh Embodiment> FIG. 14 is a perspective view showing the robot system 610 of the present embodiment. Note that, for configurations similar to those of the above-described embodiments, the description may be omitted by appropriately assigning the same reference numerals.

[0223] As shown in FIG. 14, the robot system 610 of the present embodiment has a projection device 670 that projects optical SL. In the present embodiment, the projection device 670 is disposed around the robot arm 621. The projection device 670 is disposed, for example, around the fifth arm portion 624e. The projection device 670 is connected to the guide rail portion 621a of the fifth arm portion 624e. The guide rail portion 621a has the same configuration as the guide rail portion 521a of the sixth embodiment, except that the projection device 670 is connected instead of the imaging device. In the present embodiment, the projection device 670 is movable along the guide rail portion 621a in a predetermined circumferential direction around the robot arm 621. The projection device 670 projects, for example, a grid-like pattern of optical SL onto the object W. The structure of the projection device 670 is not particularly limited as long as it can project optical SL.

[0224] In the present embodiment, the first imaging device 31 and the second imaging device 32 execute imaging and acquire images in a state where the optical SL is projected by the projection device 670. In the present embodiment, the control unit 40 moves the projection device 670, the first imaging device 31, and the second imaging device 32 so that the optical SL projected by the projection device 670 onto the object W can be imaged by the first imaging device 31 and the second imaging device 32. The robot arm 621 has the same configuration as the robot arm 21 of the first embodiment, except that the guide rail portion 621a is provided. Other configurations of the robot system 610 can be the same as those of the robot systems of the above-described embodiments.

[0225] According to this embodiment, the first imaging device 31 and the second imaging device 32 execute imaging and acquire images in a state where the optical SL is projected by the projection device 670. Therefore, the object W on which the optical SL is projected can be imaged by the first imaging device 31 and the second imaging device 32. Thereby, the first imaging device 31 and the second imaging device 32 can more suitably acquire an image of the object W. Further, by making the optical SL projected from the projection device 670 into light of a pattern such as a lattice pattern, the three-dimensional shape, etc. of the object W can be measured based on the pattern reflected in the images acquired by the first imaging device 31 and the second imaging device 32.

[0226] In this embodiment, only the first imaging device 31 out of the first imaging device 31 and the second imaging device 32 may execute imaging and acquire images in a state where the optical SL is projected by the projection device 670. Also, only the second imaging device 32 out of the first imaging device 31 and the second imaging device 32 may execute imaging and acquire images in a state where the optical SL is projected by the projection device 670. Further, in this embodiment, the robot system 610 may have only the first imaging device 31 out of the first imaging device 31 and the second imaging device 32. In this case, the first imaging device 31 executes imaging and acquires images in a state where the optical SL is projected by the projection device 670, and acquires information regarding the distance of the object W by a method similar to the method of acquiring information regarding the distance of the object W by one first imaging device 31 described in the second embodiment.

[0227] Also, in this embodiment, the projection device 670 may be arranged around the end effector 22, or may be arranged around the adapter 23. Which part of the robot arm 621, the end effector 22, and the adapter 23 the projection device 670 is arranged around can be appropriately determined according to the positions where the first imaging device 31 and the second imaging device 32 are attached, the work content of the robot system 610, etc. By arranging the projection device 670 around these parts, it becomes difficult for the optical SL projected from the projection device 670 to be blocked by the parts of the robot system 610, and it becomes easy to suitably project the optical SL projected from the projection device 670 onto the object W.

[0228] Further, the projection device 670 may be fixed to the attached member so as not to be relatively movable. Also, a plurality of projection devices 670 may be provided. In this case, the plurality of projection devices 670 may be attached to different members.

[0229] <Eighth Embodiment> FIG. 15 is a perspective view showing the robot system 710 of the present embodiment. Note that, for the same configurations as those in the above-described embodiments, the description may be omitted by appropriately assigning the same reference numerals.

[0230] As shown in FIG. 15, in the present embodiment, the adapter 723 has an annular guide rail portion 723h along the circumferential direction. The adapter 723 has the same configuration as the adapter 23 of the first embodiment except for having the guide rail portion 723h. In the present embodiment, the end effector 722 has the same configuration as the end effector 22 of the first embodiment except for not having the guide rail portion 22e and not having the first imaging device 731 and the second imaging device 732 attached thereto.

[0231] In the present embodiment, the first imaging device 731 and the second imaging device 732 are attached to the adapter 723. The first imaging device 731 and the second imaging device 732 are arranged around the adapter 723. The first imaging device 731 is connected to the guide rail portion 723h via a slider 731d. The second imaging device 732 is connected to the guide rail portion 723h via a slider 732d. That is, in the present embodiment, the adapter 723 has the guide rail portion 723h as a first holding portion that holds the first imaging device 731 and a second holding portion that holds the second imaging device 732.

[0232] In the present embodiment, the sliders 731d and 732d extend radially outward from the guide rail portion 723h and protrude radially outward from the end effector 722. As a result, the first imaging device 731 and the second imaging device 732 provided at the respective radially outer ends of the sliders 731d and 732d are located radially outward of the end effector 722.

[0233] In the present embodiment, at least one of the first imaging device 731 and the second imaging device 732 is movable with respect to the adapter 723. At least one of the first imaging device 731 and the second imaging device 732 is movable in a predetermined circumferential direction around the adapter 723. In the present embodiment, both the first imaging device 731 and the second imaging device 732 are movable with respect to the adapter 723 and are movable in a predetermined circumferential direction around the adapter 723. That is, in the present embodiment, the first imaging device 731 and the second imaging device 732 are movably held by the guide rail portion 723h as the first holding portion and the second holding portion. The relative positions of the first imaging device 731 and the second imaging device 732 are variable. The other configurations of the robot system 710 can be the same as the other configurations of the robot systems in the above-described embodiments.

[0234] According to the present embodiment, the first imaging device 731 and the second imaging device 732 attached to the adapter 723 can obtain the same effects as those obtained by the first imaging device 31 and the second imaging device 32 attached to the end effector 22 in the first embodiment.

[0235] Note that in this embodiment, the adapter 723 may have a first holding portion that holds the first imaging device 731 immovably. In this case, the adapter 723 may have a guide rail portion 723h as a second holding portion that holds the second imaging device 732 movably. Further, the adapter 723 may have a second holding portion that holds the second imaging device 732 immovably. In this case, the adapter 723 may have a guide rail portion 723h as a first holding portion that holds the first imaging device 731 movably. Thus, in this embodiment, one of the first imaging device 731 and the second imaging device 732 may be movable in a predetermined circumferential direction around the adapter 723, and the other may be fixed to a predetermined portion of the adapter 723.

[0236] <Ninth Embodiment> FIG. 16 is a perspective view showing the robot system 810 of this embodiment. Note that for configurations similar to those of the above-described embodiments, the description may be omitted by appropriately assigning the same reference numerals.

[0237] As shown in FIG. 16, in this embodiment, the first imaging device 831 and the second imaging device 832 are attached to the robot arm 521. The first imaging device 831 and the second imaging device 832 are disposed around the robot arm 521. More specifically, the first imaging device 831 and the second imaging device 832 are disposed around the fifth arm portion 524e. The first imaging device 831 and the second imaging device 832 are connected to the guide rail portion 521a by a structure similar to the structure in which the first imaging device 31 and the second imaging device 32 are connected in the first embodiment, for example. That is, in this embodiment, the robot arm 521 has a guide rail portion 521a as a first holding portion that holds the first imaging device 831 and a second holding portion that holds the second imaging device 832.

[0238] In the present embodiment, at least one of the first imaging device 831 and the second imaging device 832 is movable with respect to the robot arm 521. At least one of the first imaging device 831 and the second imaging device 832 is movable in a predetermined circumferential direction around the robot arm 521. In the present embodiment, both the first imaging device 831 and the second imaging device 832 are movable with respect to the robot arm 521 and are movable in a predetermined circumferential direction around the robot arm 521. That is, the first imaging device 831 and the second imaging device 832 are movably held by a guide rail portion 521a as a first holding portion and a second holding portion. The relative positions of the first imaging device 831 and the second imaging device 832 are variable. Other configurations of the robot system 810 can be the same as those of the robot systems in the above-described embodiments.

[0239] According to the present embodiment, the first imaging device 831 and the second imaging device 832 attached to the robot arm 521 can obtain the same effects as those obtained by the first imaging device 31 and the second imaging device 32 attached to the end effector 22 in the first embodiment.

[0240] Note that in the present embodiment, the robot arm 521 may have a first holding portion that holds the first imaging device 831 immovably. In this case, the robot arm 521 may have a guide rail portion 521a as a second holding portion that holds the second imaging device 832 movably. Further, the robot arm 521 may have a second holding portion that holds the second imaging device 832 immovably. In this case, the robot arm 521 may have a guide rail portion 521a as a first holding portion that holds the first imaging device 831 movably. Thus, in the present embodiment, one of the first imaging device 831 and the second imaging device 832 may be movable in a predetermined circumferential direction around the robot arm 521, and the other may be fixed to a predetermined portion of the robot arm 521.

[0241] As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and appropriate changes can be made without departing from the spirit of the present invention.

[0242] When the imaging device is movable relative to the member to which it is attached, the calibration of the position of the imaging device may be performed by any method. For example, a panel or the like having a specific mark described at a specific distance from the imaging device may be arranged, and the position of the imaging device may be calibrated by imaging the mark. Further, when the imaging device is movable relative to the member to which it is attached, the imaging device may be movable only between a plurality of predetermined positions. In this case, the positions where the imaging device can move may be determined structurally. In this case, the position of the imaging device can also be grasped structurally.

[0243] When the imaging device is movable relative to the member to which it is attached, the imaging device may move relative to the member in any manner. The imaging device may move linearly relative to the member, or may move in a curved shape other than an arc. When a plurality of imaging devices are movable relative to the member to which they are attached, each imaging device may move along a different movement path. The structure of the drive unit for relatively moving the member to which the imaging device is attached is not particularly limited.

[0244] When a plurality of imaging devices are provided, the plurality of imaging devices may include different types of imaging devices. The plurality of imaging devices may include, for example, an infrared camera and an RGB camera. In this case, the object may be imaged from the same position by each of the infrared camera and the RGB camera to obtain an image.

[0245] The robot system may have an external sensor capable of detecting at least one of the position, orientation, and shape of the robot. The external sensor may be arranged on the ceiling of the place where the robot is arranged, or may be arranged on the floor of the place where the robot is arranged. The external sensor may be, for example, a sensor capable of detecting the position and orientation of a robot arm, a sensor capable of detecting the position and orientation of an end effector, or a sensor capable of detecting the position and orientation of an adapter.

[0246] The external sensor may be, for example, a laser tracker. In this case, the external sensor may detect the position information of each part based on, for example, the ranging result by the Time Of Flight (TOF) method based on the difference between the irradiation timing of irradiating light and the light reception timing of receiving the reflected light. Also, the external sensor may detect the position information of each part by obtaining a geometric positional relationship in the manner of triangulation based on the measurement result of the reflection position of the reflected light generated by irradiating light in a plurality of optical paths. In this case, in order to improve the measurement accuracy of the reflection position of the reflected light, the external sensor may have a variable focus lens (for example, a zoom lens) in the optical system of the light receiving part for receiving the reflected light. Also, for the position detection of each part by the external sensor, a ranging method using an optical communication by an ultrashort time optical pulse may be used.

[0247] The external sensor may be capable of detecting the position and orientation of the imaging device. In this case, when the member to which the imaging device is attached is relatively movable, the control unit may move the imaging device based on the information of the imaging device obtained by the external sensor. Also, in this case, the imaging device may be provided with a marker detectable by the external sensor.

[0248] The external sensor may be an imaging device whose baseline length can be changed. In this case, the control unit may change the baseline length of the external sensor according to, for example, the distance between the object on which the work is performed by the end effector and the external sensor. As an example, when the control unit brings the object grasped by the end effector closer to the external sensor by moving the robot arm, the control unit may reduce the baseline length of the external sensor. The change in the baseline length of the external sensor can be performed by, for example, the same method as the method for changing the baseline length described appropriately in each of the above-described embodiments.

[0249] The applications of the robot system described above are not particularly limited. Each of the configurations and methods described above can be appropriately combined within a range that does not conflict with each other.

Explanation of Signs

[0250] 10, 110, 210, 310, 410, 510, 610, 710, 810... Robot system 20... Robot, 21, 521, 621... Robot arm 22, 122, 722... End effector 22e, 521a, 723h... Guide rail part (first holding part, second holding part) 23, 723... Adapter 24... Arm part (movable part) 30, 230, 230a, 230b, 230c, 430, 431, 432, 530... Imaging device 31, 331, 531, 731, 831... First imaging device 31f, 32f, 132f, 235a, 235b, 235c, 331f, 332f, 435... Image sensor 32, 132, 332, 732, 832... Second imaging device 34, 134... Position acquisition unit 40... Control unit 44... Distance information acquisition unit 50... Display unit 122f... Hole part (second holding part) 336... Connecting member 670... Projection device, AX1, AX1a, AX1b, AX1c, AX2, AX2a, AX2c, AX3a, AX3b, AX3c, AX4... Optical axis L, L1, L2, L3, L4, La... Baseline length P1... First position P2... Second position SL... Light VA... Axis W... Object

Claims

1. A robot system having a robot arm with a movable part, a first imaging device and a second imaging device attached to the robot arm, a control unit for controlling the robot system, having a distance information acquisition unit for acquiring information regarding the distance of an object, the control unit being capable of changing a baseline length, which is the distance between the first imaging device and the second imaging device, by moving at least one of the first imaging device or the second imaging device, the distance information acquisition unit acquiring information regarding the distance of the object based on the baseline length. A robot system.

2. The control unit changes the baseline length according to the work content of the robot system. The robot system according to Claim 1.

3. The work content is work including movement in which at least one of the first imaging device or the second imaging device and the object move apart, and the control unit changes the baseline length so as to increase the baseline length according to the movement of moving apart. The robot system according to Claim 2.

4. The work content is work including movement in which at least one of the first imaging device or the second imaging device and the object approach each other, the control unit changing the baseline length so as to decrease the baseline length according to the approaching movement. The robot system according to Claim 2.

5. The work content is work for searching for the object. The robot system according to any one of Claims 2 to 4.

6. The control unit changes the baseline length based on the distance between the first imaging device and / or the second imaging device and the object. The robot system according to Claim 1.

7. Other objects different from the object are located around the robot arm, When the control unit moves at least one of the first imaging device or the second imaging device with respect to the object by controlling the robot arm, the control unit moves at least one of the first imaging device or the second imaging device with respect to the robot arm so that at least one of the first imaging device or the second imaging device does not contact the other object. The robot system according to claim 1.

8. When at least one of the first imaging device or the second imaging device cannot image the object as the robot arm moves, the control unit moves the first imaging device and / or the second imaging device with respect to the robot arm. The robot system according to claim 1.

9. Further, it has a connecting member that connects the first imaging device and the second imaging device, The robot system according to any one of claims 1 to 8, wherein the connecting member can hold the first imaging device and the second imaging device in a state where the relative posture between the first imaging device and the second imaging device is maintained in a predetermined posture.

10. The distance information acquisition unit adjusts the direction of the acquired image by rotating at least one of the first image acquired by the first imaging device and the second image acquired by the second imaging device. The robot system according to any one of claims 1 to 9.

11. The control unit adjusts the direction of the acquired image by rotating at least one of the imaging elements of the first imaging device and the imaging elements of the second imaging device. The robot system according to any one of claims 1 to 10.

12. The first imaging device and the second imaging device are arranged around the robot arm, and the robot system according to any one of claims 1 to 11.

13. At least one of the first imaging device and the second imaging device is movable in a predetermined circumferential direction around the robot arm, and the robot system according to claim 12.

14. At least one of the first imaging device and the second imaging device is movable so that the long sides of the imaging elements of the first imaging device and the second imaging device are parallel to each other, and the robot system according to any one of claims 8 to 13.

15. Furthermore, it has a position acquisition unit that acquires at least the position information of the first imaging device. The distance information acquisition unit acquires the baseline length based on the position information of the first imaging device acquired by the position acquisition unit, and the robot system according to claim 1.

16. At least the first imaging device moves to a predetermined initial position after the power of the robot system is turned on, and the robot system according to any one of claims 1 to 15.

17. At least the first imaging device moves to a predetermined end position before the power of the robot system is turned off, and the robot system according to any one of claims 1 to 16.

Images