Control system and control method

Abstract

To provide a robot system to acquire information on a distance to an object.SOLUTION: A robot system with a robot arm having a movable part includes: first and second imaging devices mounted on the robot arm; a control part for controlling the robot system; and a distance information acquisition part for acquiring information on a distance to an object. The control part can change a baseline length, which is a distance between the first imaging device and the second imaging device, by moving at least one of the first and second imaging devices. The distance information acquisition part acquires the information on the distance to the object on the basis of the baseline length.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a robot system. [Background technology]

[0002] 2. Description of the Related Art A robot system equipped with an imaging device is known. For example, Patent Document 1 describes a configuration in which an imaging device is attached to a robot arm. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-131685 Summary of the Invention

[0004] One aspect of the robot system of the present invention is a robot system having a robot arm with a movable part, comprising a first imaging device and a second imaging device attached to the robot arm, a control unit that controls the robot system, and a distance information acquisition unit that acquires information about the distance to an object, wherein the control unit is capable of changing a baseline length, which is the distance between the first imaging device and the second imaging device, by moving at least one of the first imaging device or the second imaging device, and the distance information acquisition unit acquires information about the distance to the object based on the baseline length. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a perspective view showing a robot system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing a part of the configuration of the robot system according to the first embodiment. [Figure 3] FIG. 3 is a perspective view showing a part of the robot arm, an end effector, an adapter, and an imaging device according to the first embodiment. [Figure 4]FIG. 4 is a plan view showing a part of the robot arm, an end effector, an adapter, and an imaging device according to the first embodiment. [Figure 5] FIG. 5 is a view of a part of the end effector, the first imaging device, and the second imaging device as seen from the tip side in the central axis direction. [Figure 6] Figure 6 is a view of a part of the end effector, the first imaging device, and the second imaging device viewed from the tip side in the central axis direction, showing the first imaging device and the second imaging device positioned in a predetermined initial position. [Figure 7] FIG. 7 is a perspective view showing a part of the robot system according to the second embodiment. [Figure 8] FIG. 8 is a view of a part of the robot system according to the third embodiment, seen from the tip side in the central axis direction. [Figure 9] FIG. 9 is a view of a part of the robot system according to the fourth embodiment, seen from the tip side in the central axis direction. [Figure 10] FIG. 10 is a view of a part of the robot system according to the fifth embodiment, seen from the tip side in the central axis direction. [Figure 11] FIG. 11 is a diagram for explaining part of the procedure when the robot system according to the fifth embodiment acquires information about the distance to an object. [Figure 12A] FIG. 12A is a diagram showing an example of a case where two images with a relatively large base line length are selected when the zoom magnification of the imaging device in the fifth embodiment is relatively small. [Figure 12B] FIG. 12B is a diagram showing an example of a case where two images with a relatively large base line length are selected when the zoom magnification of the imaging device in the fifth embodiment is relatively large. [Figure 12C] FIG. 12C is a diagram showing an example of a case where two images with a relatively short base line length are selected when the zoom magnification of the imaging device in the fifth embodiment is relatively large. [Figure 13] FIG. 13 is a perspective view showing a robot system according to the sixth embodiment. [Figure 14]FIG. 14 is a perspective view showing a robot system according to the seventh embodiment. [Figure 15] FIG. 15 is a perspective view showing a robot system according to the eighth embodiment. [Figure 16] FIG. 16 is a perspective view showing a robot system according to the ninth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0006] Hereinafter, a robot system, a robot arm, an end effector, and an adapter according to an embodiment of the present invention will be described with reference to the drawings. The scope of the present invention is not limited to the following embodiments, and can be modified as desired within the scope of the technical concept of the present invention. In addition, in the following drawings, the scale and number of each structure may differ from the scale and number of the actual structure in order to make each configuration easier to understand.

[0007] First Embodiment Fig. 1 is a perspective view showing a robot system 10 according to this embodiment. Fig. 2 is a block diagram showing a part of the configuration of the robot system 10 according to this embodiment. 1, the robot system 10 includes a robot 20, an imaging device 30, a control unit 40, and a display unit 50. The robot 20 performs a task on an object W placed on a workbench WB, for example.

[0008] The robot 20 includes a robot arm 21, an end effector 22, and an adapter 23. The robot arm 21 includes an arm unit 24 as a movable unit. In this embodiment, a plurality of arm units 24 are provided. The robot arm 21 is, for example, a multi-joint arm configured by connecting a plurality of arm units 24. The arm units 24 include, for example, five arm units: a first arm unit 24a, a second arm unit 24b, a third arm unit 24c, a fourth arm unit 24d, and a fifth arm unit 24e. The first arm unit 24a, the second arm unit 24b, the third arm unit 24c, the fourth arm unit 24d, and the fifth arm unit 24e are connected in this order from the installation surface of the robot arm 21.

[0009] 2, the robot arm 21 has an arm driver 25 and an arm position acquisition unit 26. The arm driver 25 is, for example, a servo motor. An arm driver 25 is provided for each arm unit 24, for example. That is, for example, five arm drivers 25 are provided.

[0010] The arm driver 25 provided on the first arm unit 24a displaces the first arm unit 24a with reference to the installation surface of the robot 20. The arm driver 25 provided on the second arm unit 24b displaces the second arm unit 24b with reference to the first arm unit 24a. The arm driver 25 provided on the third arm unit 24c displaces the third arm unit 24c with reference to the second arm unit 24b. The arm driver 25 provided on the fourth arm unit 24d displaces the fourth arm unit 24d with reference to the third arm unit 24c. The arm driver 25 provided on the fifth arm unit 24e displaces the fifth arm unit 24e with reference to the fourth arm unit 24d. Each arm driver 25, for example, rotates each arm unit 24.

[0011] The arm position acquisition unit 26 includes, for example, a rotary encoder (not shown). An arm position acquisition unit 26 is provided, for example, for each arm unit 24. That is, for example, five arm position acquisition units 26 are provided. The arm position acquisition unit 26 provided in the first arm unit 24a can detect the amount of displacement of the first arm unit 24a relative to the installation surface of the robot 20. The arm position acquisition unit 26 provided in the second arm unit 24b can detect the amount of displacement of the second arm unit 24b relative to the first arm unit 24a. The arm position acquisition unit 26 provided in the third arm unit 24c can detect the amount of displacement of the third arm unit 24c relative to the second arm unit 24b. The arm position acquisition unit 26 provided in the fourth arm unit 24d can detect the amount of displacement of the fourth arm unit 24d relative to the third arm unit 24c. The arm position acquisition unit 26 provided on the fifth arm unit 24e can detect the amount of displacement of the fifth arm unit 24e relative to the fourth arm unit 24d. The amount of displacement of each arm unit 24 that can be detected by each arm position acquisition unit 26 includes, for example, the rotation angle of each arm unit 24 detected by a rotary encoder (not shown).

[0012] Fig. 3 is a perspective view showing a part of the robot arm 21, the end effector 22, the adapter 23, and the imaging device 30 of this embodiment. Fig. 4 is a plan view showing a part of the robot arm 21, the end effector 22, the adapter 23, and the imaging device 30 of this embodiment. 3 and 4, the end effector 22 is attached to the robot arm 21. In this embodiment, the end effector 22 is attached to the tip of the fifth arm section 24e via an adapter 23. The end effector 22 is detachably attached to the robot arm 21, for example. The end effector 22 is replaceable with another end effector.

[0013] As the end effector 22 attached to the robot arm 21, end effectors having various shapes, structures, and functions can be appropriately adopted depending on the work performed by the robot 20. Examples of the end effector 22 attached to the robot arm 21 include a robot hand capable of grasping the object W, a processing head that performs laser processing or ultrasonic processing, a camera, an injector that injects molten metal / resin or particles for blast processing, a manipulator, and an air blower.

[0014] In this embodiment, the end effector 22 is a multi-fingered robot hand capable of grasping an object W on a workbench WB. As shown in FIG. 3, the end effector 22 has a base 22a, multiple fingers 22b, an end effector driver 28, and an end effector position acquisition unit 29. The base 22a is connected to the fifth arm 24e, for example, via an adapter 23. The base 22a is, for example, cylindrical and centered on a central axis CL, as shown appropriately in each drawing. The central axis CL, as shown appropriately in each drawing, is the central axis of the end effector 22, the adapter 23, and the fifth arm 24e.

[0015] In the following description, the direction parallel to the central axis CL will be referred to as the "central axis direction" and will be indicated as the Z axis in each drawing as appropriate. The positive side of the Z axis (+Z side) in the central axis direction will be referred to as the "tip side," and the negative side (-Z side) in the central axis direction will be referred to as the "base side." Unless otherwise specified, the radial direction centered on the central axis CL will be simply referred to as the "radial direction," and the circumferential direction around the central axis CL will be simply referred to as the "circumferential direction."

[0016] In this embodiment, a guide rail portion 22e is provided on the base portion 22a. The guide rail portion 22e is, for example, an annular groove that surrounds the base portion 22a in the circumferential direction. The multiple finger portions 22b protrude from the base portion 22a toward the tip side (+Z side) in the central axis direction. In this embodiment, the end effector 22 can grasp the object W with the multiple finger portions 22b. Note that the number of finger portions 22b is not particularly limited.

[0017] The end effector driving unit 28 is capable of driving the end effector 22. The end effector driving unit 28 has a rotation driving unit 22c. The rotation driving unit 22c is provided, for example, inside the base 22a. The rotation driving unit 22c is, for example, a servo motor that can rotate the base 22a around the central axis CL. Although not shown, the end effector driving unit 28 has a finger driving unit that drives the multiple fingers 22b. For example, a finger driving unit is provided for each of the multiple fingers 22b. One finger driving unit may be provided for each finger 22b, or multiple finger driving units may be provided for each finger 22b. The finger driving unit displaces, for example, the angle of the finger 22b relative to the base 22a. The finger driving unit is, for example, a servo motor.

[0018] The end effector position acquisition unit 29 can acquire the relative position of the end effector 22 with respect to the adapter 23. The end effector position acquisition unit 29 has a rotational position acquisition unit 22d. The rotational position acquisition unit 22d is provided, for example, inside the base 22a. The rotational position acquisition unit 22d can detect the rotational position of the end effector 22. The rotational position acquisition unit 22d can detect, for example, the rotation angle around the central axis CL of the base 22a. The rotational position acquisition unit 22d is, for example, a rotary encoder. The end effector position acquisition unit 29 may include, for example, a sensor that can detect the position and angle of the finger 22b with respect to the base 22a.

[0019] In this embodiment, a camera unit 60 is attached to the end effector 22. The camera unit 60 is fixed to the base 22a. The camera unit 60 includes a support 61, a first camera 62, and a second camera 63. The support 61 protrudes from the base 22a toward the tip end (+Z side) in the central axis direction. The base end (-Z side) end of the support 61 is connected to the outer circumferential surface at the tip end of the base 22a. The first camera 62 and the second camera 63 are fixed to the tip end of the support 61. The first camera 62 and the second camera 63 are capable of capturing images of the multiple fingers 22b and the object W grasped by the fingers 22b. The first camera 62 and the second camera 63 form a stereo camera. Note that the camera unit 60 is not shown in FIG. 4.

[0020] 2, the camera unit 60 has an image sensor 64, a memory 65, and a digital signal processing unit 66. The image sensor 64 is, for example, a CCD image sensor, a CMOS image sensor, or the like. An image sensor 64 is provided in each of the first camera 62 and the second camera 63. Each image sensor 64 converts an optical signal incident on the camera to an analog electrical signal, and converts the converted analog electrical signal into a digital image signal and outputs it.

[0021] The digital signal processing unit 66 performs image processing such as digital amplification, color interpolation, and white balance processing on the digital image signal output from the imaging element 64. The digital image signal after processing by the digital signal processing unit 66 may be temporarily stored in the memory 65, or may be output to the control unit 40 without being stored in the memory 65. The digital image signal output from the digital signal processing unit 66 to the control unit 40 is output to the distance information acquisition unit 44, which will be described later.

[0022] The memory 65 can store the digital image signal output from the imaging element 64 and the digital image signal output from the digital signal processing unit 66. The memory 65 is, for example, a volatile memory. However, the memory 65 may be a non-volatile memory. The digital image signal output from the imaging element 64 is, for example, stored in the memory 65, and then sent from the memory 65 to the digital signal processing unit 66, where it is subjected to image processing.

[0023] Note that memory 65 and digital signal processing unit 66 may each be provided within camera unit 60 and used for both image sensor 64 of first camera 62 and image sensor 64 of second camera 63, or may be provided separately for image sensor 64 of first camera 62 and image sensor 64 of second camera 63. Also, part or all of memory 65 and digital signal processing unit 66 may be provided outside camera unit 60, such as in control unit 40. Also, camera unit 60 may be configured to have only one camera.

[0024] The adapter 23 is a member for attaching the end effector 22 to the robot arm 21. As shown in FIG. 4, the adapter 23 has a support portion 23a, a base portion 23b, a connection portion 23c, and a base drive portion 23d. The support portion 23a is a portion that is connected to the robot arm 21. The support portion 23a is detachably connected to, for example, the tip portion of the fifth arm portion 24e. The support portion 23a has a recess 23g that is recessed from the tip side (+Z side) to the base side (-Z side) in the central axis direction. Note that the support portion 23a may be fixed to the robot arm 21 in an undetachable manner.

[0025] The pedestal portion 23b is disposed on the tip side (+Z side) of the support portion 23a. The pedestal portion 23b is a portion to which the end effector 22 is connected. In this embodiment, the base portion 22a of the end effector 22 is detachably connected to the end portion on the tip side (+Z side) of the pedestal portion 23b. Note that the end effector 22 may be fixed to the pedestal portion 23b in an undetachable manner. The base end side (-Z side) portion of the pedestal portion 23b is inserted, for example, into the interior of the recess 23g. For example, a gap is provided between the support portion 23a and the pedestal portion 23b, and the support portion 23a and the pedestal portion 23b are not in direct contact with each other.

[0026] The connecting portion 23c is provided inside the recess 23g. The connecting portion 23c is provided between the support portion 23a and the base portion 23b. The connecting portion 23c connects the support portion 23a and the base portion 23b. That is, in this embodiment, the support portion 23a and the base portion 23b are not in direct contact with each other, but are indirectly connected to each other via the connecting portion 23c. The connecting portion 23c supports the mass of the base portion 23b and the mass of the end effector 22.

[0027] The connecting portion 23c has a damper element 23e and a spring element 23f. The spring element 23f may be, for example, an elastic member whose elastic force is adjustable. In this case, the robot system 10 may have an adjustment unit that can adjust the elastic force of the spring element 23f. The spring element 23f may be, for example, an air spring. In this case, the elastic force of the spring element 23f may be adjusted by adjusting the air pressure of the air spring using the adjustment unit. For example, a plurality of connecting portions 23c are provided. The plurality of connecting portions 23c include, for example, connecting portions 23c having damper elements 23e and spring elements 23f that are displaced when subjected to a force in the central axis direction, and connecting portions 23c having damper elements 23e and spring elements 23f that are displaced when subjected to a force in a direction perpendicular to the central axis direction.

[0028] The connecting portion 23c can, for example, reduce vibrations of the end effector 22 and externally applied vibrations. The connecting portion 23c suppresses displacement of the end effector 22 and displacement of the base portion 23b caused by the weight of the end effector 22 and the weight of the base portion 23b. The connecting portion 23c can support the end effector 22 in the direction of gravity regardless of the posture of the end effector 22. Note that the connecting portion 23c may have, for example, other elements capable of reducing vibrations of the end effector 22 in addition to the damper element 23e and the spring element 23f. The other elements include, for example, piezoelectric elements.

[0029] Base driving unit 23d is provided, for example, inside recess 23g, between support unit 23a and base unit 23b. Base driving unit 23d can displace the position of base unit 23b relative to support unit 23a. Base driving unit 23d can move base unit 23b, thereby moving end effector 22 connected to base unit 23b.

[0030] The base drive unit 23d has, for example, a plurality of linear motors 27. The linear motors 27 are, for example, voice coil motors. The linear motors 27 have a magnetic field generator 27a and a magnet unit 27b. One of the magnetic field generator 27a and the magnet unit 27b is attached to the support unit 23a, and the other is attached to the base unit 23b. In FIG. 4, for example, the magnetic field generator 27a is attached to the support unit 23a, and the magnet unit 27b is attached to the base unit 23b. Note that the magnetic field generator 27a may be attached to the base unit 23b, and the magnet unit 27b may be attached to the support unit 23a.

[0031] The magnetic field generating unit 27a is, for example, a coil. A magnetic field is generated by supplying a current to the magnetic field generating unit 27a. A repulsive force or an attractive force is generated between the magnetic field generating unit 27a and the magnet unit 27b due to the magnetic field generated by the magnetic field generating unit 27a and the magnetic field generated by the magnet unit 27b. This repulsive force or attractive force displaces the magnet unit 27b relative to the magnetic field generating unit 27a. As a result, the linear motor 27 displaces the base unit 23b, to which the magnet unit 27b is attached, relative to the support unit 23a, to which the magnetic field generating unit 27a is attached. In this way, the base driving unit 23d can drive the base unit 23b in a non-contact state, without bringing the support unit 23a and the base unit 23b into direct contact with each other.

[0032] The multiple linear motors 27 include, for example, a linear motor 27 that can displace the base portion 23b in the central axis direction relative to the support portion 23a, and a linear motor 27 that can displace the base portion 23b in a direction perpendicular to the central axis direction relative to the support portion 23a.

[0033] The adapter 23 may have any configuration as long as it can attach the end effector 22 to the robot arm 21. As the configuration of the adapter 23, for example, the configuration of the adapter described in International Application No. PCT / JP2019 / 016043 may be adopted.

[0034] As shown in FIG. 3 , a plurality of imaging devices 30 are provided in this embodiment. For example, two imaging devices 30 are provided: a first imaging device 31 and a second imaging device 32. The first imaging device 31 and the second imaging device 32 may be, for example, RGB cameras or infrared cameras. The first imaging device 31 and the second imaging device 32 form a stereo camera. In this embodiment, the first imaging device 31 and the second imaging device 32 are attached to the end effector 22. The first imaging device 31 and the second imaging device 32 are arranged around the end effector 22. The first imaging device 31 and the second imaging device 32 are, for example, located radially outward of the base 22a and arranged along the circumferential direction.

[0035] In this embodiment, the optical axis AX1 of the first imaging device 31 and the optical axis AX2 of the second imaging device 32 are parallel to each other. The optical axes AX1 and AX2 are, for example, parallel to the central axis CL. In this specification, "the optical axes of the multiple imaging devices are parallel to each other" includes not only the case where the optical axes of the multiple imaging devices are strictly parallel to each other, but also the case where the optical axes of the multiple imaging devices are approximately parallel to each other. The case where the optical axes of the multiple imaging devices are approximately parallel to each other includes, for example, the case where the optical axes of the multiple imaging devices are tilted from each other by 5° or less.

[0036] Fig. 5 is a view of a part of the end effector 22, the first imaging device 31, and the second imaging device 32 as viewed from the tip end side (+Z side) in the central axis direction. Fig. 6 is a view of a part of the end effector 22, the first imaging device 31, and the second imaging device 32 as viewed from the tip end side (+Z side) in the central axis direction, illustrating a case where the first imaging device 31 and the second imaging device 32 are positioned at predetermined initial positions. Note that in Figs. 5 and 6, the finger portion 22b of the end effector 22 and the camera unit 60 are not shown.

[0037] 5 and 6, at least one of the first imaging device 31 and the second imaging device 32 is movable relative to the end effector 22. In this embodiment, both the first imaging device 31 and the second imaging device 32 are movable relative to the end effector 22. The relative positions of the first imaging device 31 and the second imaging device 32 are variable. In this embodiment, at least one of the first imaging device 31 and the second imaging device 32 is movable in a predetermined circumferential direction around the end effector 22. In this embodiment, the "predetermined circumferential direction" is the circumferential direction around the central axis CL around the base 22a.

[0038] In this embodiment, both the first imaging device 31 and the second imaging device 32 are movable in a predetermined circumferential direction around the end effector 22. That is, in this embodiment, one of the first imaging device 31 and the second imaging device 32 is movable in a predetermined circumferential direction around the end effector 22, and the other is also movable in the predetermined circumferential direction around the end effector 22. As shown in FIG. 6 , in this embodiment, the first imaging device 31 and the second imaging device 32 can come into contact with each other in the circumferential direction. In this embodiment, the first imaging device 31 and the second imaging device 32 are in contact with each other in the circumferential direction when they are located at the initial positions shown in FIG. 6 .

[0039] 3, the first imaging device 31 includes a housing 31a, a first driving unit 31b, a first position acquisition unit 31c, a lens 31e, and an imaging element 31f. The housing 31a has, for example, an opening on the tip side (+Z side) and is cylindrical extending in the direction of its central axis. The central axis of the housing 31a coincides with, for example, the optical axis AX1 of the first imaging device 31. The housing 31a is attached to the base 22a of the end effector 22 via a slider 31d.

[0040] The slider 31d is fixed, for example, to a radially inner portion of the base end side (-Z side) of the housing 31a. The slider 31d connects the housing 31a and the base portion 22a of the end effector 22. That is, in this embodiment, the first imaging device 31 is connected to the end effector 22 via the slider 31d. The slider 31d is coupled to the guide rail portion 22e of the end effector 22. The slider 31d is movable in the circumferential direction along the guide rail portion 22e. This allows the first imaging device 31 to move in the circumferential direction along the guide rail portion 22e.

[0041] The lens 31e is fitted into an opening on the tip side (+Z side) of the housing 31a. The lens 31e is, for example, a circular lens when viewed in the direction of the central axis. The optical axis AX1 of the first imaging device 31 passes through the center of the lens 31e.

[0042] The imaging element 31f is disposed inside the housing 31a. The imaging element 31f is, for example, a CCD image sensor, a CMOS image sensor, or the like. Light that enters the housing 31a via the lens 31e is incident on the imaging element 31f. The imaging element 31f converts the incident optical signal into an analog electrical signal, and then converts the converted analog electrical signal into a digital image signal and outputs it.

[0043] 5, the imaging element 31f has a rectangular shape when viewed in the central axis direction. When viewed in the central axis direction, the long side of the imaging element 31f is perpendicular to the radial direction passing through the optical axis AX1 of the first imaging device 31. In this embodiment, the first imaging device 31 is movable in the circumferential direction so as to maintain a state in which the long side of the imaging element 31f is perpendicular to the radial direction passing through the optical axis AX1 of the first imaging device 31 when viewed in the central axis direction.

[0044] 3, the first driving unit 31b is disposed inside the housing 31a, for example. The first driving unit 31b is, for example, a servo motor. The first driving unit 31b moves the first imaging device 31 in the circumferential direction around the end effector 22. In this embodiment, the first imaging device 31, including the first driving unit 31b, moves in the circumferential direction together with the slider 31d.

[0045] The first position acquisition unit 31c is disposed inside the housing 31a, for example. The first position acquisition unit 31c is, for example, a rotary encoder. The first position acquisition unit 31c can acquire circumferential position information of the first imaging device 31 by detecting the rotation of the first driving unit 31b. For example, the first position acquisition unit 31c detects the circumferential position of the first imaging device 31 by detecting the number of rotations of the first driving unit 31b, assuming that the number of rotations of the first driving unit 31b is zero when the first imaging device 31 is located at the initial position shown in FIG. 6.

[0046] The second imaging device 32 includes a housing 32a, a second driving unit 32b, a second position acquisition unit 32c, a lens 32e, and an imaging element 32f. The housing 32a has, for example, an opening on the tip side (+Z side) and is cylindrical extending in the direction of its central axis. The central axis of the housing 32a coincides with, for example, the optical axis AX2 of the second imaging device 32. The housing 32a is attached to the base 22a of the end effector 22 via a slider 32d.

[0047] The slider 32d is fixed, for example, to a radially inner portion of the base end side (-Z side) of the housing 32a. The slider 32d connects the housing 32a and the base portion 22a of the end effector 22. That is, in this embodiment, the second imaging device 32 is connected to the end effector 22 via the slider 32d. The slider 32d is coupled to the guide rail portion 22e of the end effector 22. The slider 32d is movable in the circumferential direction along the guide rail portion 22e. This allows the second imaging device 32 to move in the circumferential direction along the guide rail portion 22e.

[0048] As described above, in this embodiment, the guide rail portion 22e corresponds to a first holding portion that holds the first imaging device 31 and corresponds to a second holding portion that holds the second imaging device 32. That is, in this embodiment, the end effector 22 has the guide rail portion 22e as a first holding portion that holds the first imaging device 31 and a second holding portion that holds the second imaging device 32. In this embodiment, the first imaging device 31 is movably held by the guide rail portion 22e as a first holding portion, and the second imaging device 32 is movably held by the guide rail portion 22e as a second holding portion.

[0049] The lens 32e is fitted into an opening on the tip side (+Z side) of the housing 32a. The lens 32e is, for example, a circular lens when viewed in the direction of the central axis. The optical axis AX2 of the second imaging device 32 passes through the center of the lens 32e.

[0050] The imaging element 32f is disposed inside the housing 32a. The imaging element 32f is, for example, a CCD image sensor, a CMOS image sensor, or the like. Light that enters the housing 32a via the lens 32e is incident on the imaging element 32f. The imaging element 32f converts the incident optical signal into an analog electrical signal, and then converts the converted analog electrical signal into a digital image signal and outputs it.

[0051] 5, the imaging element 32f has a rectangular shape when viewed in the central axis direction. When viewed in the central axis direction, the long side of the imaging element 32f is perpendicular to the radial direction passing through the optical axis AX2 of the second imaging device 32. In this embodiment, the second imaging device 32 is movable in the circumferential direction so that the long side of the imaging element 32f remains perpendicular to the radial direction passing through the optical axis AX2 of the second imaging device 32 when viewed in the central axis direction. The imaging element 32f has, for example, the same shape and size as the imaging element 31f of the first imaging device 31.

[0052] In this specification, the term "long side of the imaging element" refers to the long side of the rectangular region of the imaging element where light is incident. The imaging elements 31f and 32f shown in each figure only show the main body portion having the rectangular region where light is incident. The imaging elements 31f and 32f may have parts other than the main body portion, such as a frame portion that holds the main body portion where light is incident. In this case, even if the external shape of the imaging elements 31f and 32f is a shape other than a rectangle when viewed in the direction of the optical axes AX1 and AX2, the long side of the imaging elements 31f and 32f refers to the long side of the rectangular region of the imaging elements 31f and 32f where light is incident.

[0053] 3, the second driving unit 32b is disposed, for example, inside the housing 32a. The second driving unit 32b is, for example, a servo motor. The second driving unit 32b moves the second imaging device 32 in the circumferential direction around the end effector 22. In this embodiment, the entire second imaging device 32, including the second driving unit 32b, moves in the circumferential direction together with the slider 32d.

[0054] In this embodiment, the first drive unit 31b and the second drive unit 32b constitute a drive unit 33 that drives the imaging device 30. The drive unit 33 can move at least one of the first imaging device 31 and the second imaging device 32 relative to the end effector 22. In this embodiment, the drive unit 33 can move both the first imaging device 31 and the second imaging device 32 relative to the end effector 22 by using the drive units provided in each imaging device 30.

[0055] The second position acquisition unit 32c is disposed, for example, inside the housing 32a. The second position acquisition unit 32c is, for example, a rotary encoder. The second position acquisition unit 32c can acquire circumferential position information of the second imaging device 32 by detecting the rotation of the second drive unit 32b. For example, the second position acquisition unit 32c detects the circumferential position of the second imaging device 32 by detecting the number of rotations of the second drive unit 32b, assuming that the number of rotations of the second drive unit 32b is zero when the second imaging device 32 is located at the initial position shown in FIG. 6.

[0056] In this embodiment, the first position acquisition unit 31c and the second position acquisition unit 32c constitute a position acquisition unit 34 that acquires position information of at least the first imaging device 31. In this embodiment, the position acquisition unit 34 can acquire both the position information of the first imaging device 31 and the position information of the second imaging device 32 using the position acquisition units provided in each imaging device 30. The position acquisition unit 34 can acquire, for example, the circumferential position of the first imaging device 31 and the circumferential position of the second imaging device 32.

[0057] 2, each imaging device 30 has a memory 35 and a digital signal processing unit 36. The digital signal processing unit 36 ​​performs image processing such as digital amplification, color interpolation, and white balance processing on the digital image signal output from the imaging element of each imaging device 30. The digital image signal after processing by the digital signal processing unit 36 ​​may be temporarily stored in the memory 35, or may be output to the control unit 40 without being stored in the memory 35. The digital image signal output from the digital signal processing unit 36 ​​to the control unit 40 is output to a distance information acquisition unit 44, which will be described later.

[0058] The memory 35 can store digital image signals output from the imaging elements of each imaging device 30 and digital image signals output from the digital signal processing unit 36. The memory 35 is, for example, a volatile memory. However, the memory 35 may also be a non-volatile memory. The digital image signals output from the imaging elements of each imaging device 30 are, for example, stored in the memory 35, and then sent from the memory 35 to the digital signal processing unit 36, where they are subjected to image processing.

[0059] In the above description, one memory 35 and one digital signal processor 36 are provided in each imaging device 30, but this is not limiting. One memory 35 and one digital signal processor 36 may be provided for each of the two imaging devices 30, and may be used for both the image sensor 31f of the first imaging device 31 and the image sensor 32f of the second imaging device 32. Also, part or all of the memory 35 and the digital signal processor 36 may be provided outside the imaging device 30, such as in the control unit 40.

[0060] The control unit 40 controls the robot system 10. As shown in Fig. 2, in this embodiment, the control unit 40 has an arm control unit 41, an end effector control unit 42, an imaging device control unit 43, and a distance information acquisition unit 44. Each of the arm control unit 41, the end effector control unit 42, the imaging device control unit 43, and the distance information acquisition unit 44 may be realized by dedicated hardware or by a memory and a microprocessor.

[0061] The arm control unit 41 controls the arm driver 25. In this embodiment, information about the position and posture of the arm unit 24 is input to the arm control unit 41 from the arm position acquisition unit 26, and information about the distance to the target object W is input to the arm control unit 41 from the distance information acquisition unit 44. In this embodiment, the arm control unit 41 controls the arm driver 25 based on the information about the position and posture of the arm unit 24 and the information about the distance to the target object W. More specifically, the arm control unit 41 calculates target values ​​for the position and posture of the arm unit 24 based on the information about the distance to the target object W, and controls the arm driver 25 by feedback control using the information from the arm position acquisition unit 26 so that the position and posture of the arm unit 24 become the target values. In this way, the control unit 40 controls at least one of the position and posture of the robot arm 21 by controlling the arm driver 25 with the arm control unit 41. Note that target values ​​for the position and posture of the arm unit 24 may be input to the arm control unit 41 from an external source.

[0062] The end effector control unit 42 controls the end effector driving unit 28. In this embodiment, information about the position and orientation of the end effector 22 is input to the end effector control unit 42 from the end effector position acquisition unit 29, and information about the distance to the object W is input to the end effector control unit 42 from the distance information acquisition unit 44. In this embodiment, the end effector control unit 42 controls the end effector driving unit 28 based on the information about the position and orientation of the end effector 22 and the information about the distance to the object W. More specifically, the end effector control unit 42 calculates target values ​​for the position and orientation of the end effector 22 based on the information about the distance to the object W, and controls the end effector driving unit 28 by feedback control using the information from the end effector position acquisition unit 29 so that the position and orientation of the end effector 22 become the target values. In this way, the control unit 40 controls at least one of the position and orientation of the end effector 22 by controlling the end effector driving unit 28 with the end effector control unit 42. Note that target values ​​for the position and orientation of the end effector 22 may be input to the end effector control unit 42 from outside.

[0063] The imaging device control unit 43 controls the drive unit 33 of the imaging device 30. In this embodiment, information about the position of the imaging device 30 is input to the imaging device control unit 43 from the position acquisition unit 34 of the imaging device 30. More specifically, circumferential position information of the first imaging device 31 is input to the imaging device control unit 43 from the first position acquisition unit 31c, and circumferential position information of the second imaging device 32 is input to the imaging device control unit 43 from the second position acquisition unit 32c. In addition, information about the distance to the object W is input to the imaging device control unit 43 from the distance information acquisition unit 44. The imaging device control unit 43 controls the first drive unit 31b and the second drive unit 32b, for example, based on the position information of the imaging device 30 input from the position acquisition unit 34 and the information about the distance to the object W input from the distance information acquisition unit 44. As a result, the control unit 40 controls the drive unit 33 using the imaging device control unit 43, thereby controlling the position of the imaging device 30.

[0064] The imaging device control unit 43 can change the base length L, which is the distance between the first imaging device 31 and the second imaging device 32. As shown in FIG. 5 , the base length L is the distance between the optical axis AX1 of the first imaging device 31 and the optical axis AX2 of the second imaging device 32. For example, by moving the first imaging device 31 and the second imaging device 32 away from each other in the circumferential direction from the position indicated by the two-dot chain line to the position indicated by the solid line in FIG. 5 , the base length L can be changed from the base length L1 to the base length L2, which is longer than the base length L1. This increases the base length L between the first imaging device 31 and the second imaging device 32. On the other hand, by moving the first imaging device 31 and the second imaging device 32 closer to each other in the circumferential direction, the base length L between the first imaging device 31 and the second imaging device 32 can be reduced. In this way, in this embodiment, the control unit 40 can change the baseline length L, which is the distance between the first imaging device 31 and the second imaging device 32, by controlling the drive unit 33 of the imaging device 30 using the imaging device control unit 43.

[0065] The imaging device control unit 43 calculates a target value of the baseline length L to be changed based on, for example, information about the distance to the object W. When the distance from the imaging device 30 to the object W is relatively long, the imaging device control unit 43 sets the baseline length L to a relatively long value. On the other hand, when the distance from the imaging device 30 to the object W is relatively short, the imaging device control unit 43 sets the baseline length L to a relatively short value. Note that the target value of the baseline length L may be input to the imaging device control unit 43 from an external source. The target value of the baseline length L may be input to the imaging device control unit 43 from the distance information acquisition unit 44.

[0066] In this embodiment, the control unit 40 changes the baseline length L using the imaging device control unit 43 depending on the work content of the robot system 10. For example, when searching for an object W on the work table WB, the control unit 40 sets the baseline length L to be relatively large. On the other hand, for example, when the control unit 40 finds the object W to be worked on and then moves the end effector 22 closer to the object W, the control unit 40 sets the baseline length L to be relatively small. At this time, the control unit 40 may gradually decrease the baseline length L as the end effector 22 moves closer to the object W.

[0067] In this embodiment, after the robot system 10 is powered on, the imaging device control unit 43 controls the drive unit 33 to move the imaging device 30 to a predetermined initial position. For example, after the robot system 10 is powered on, the imaging device control unit 43 moves the first imaging device 31 to the predetermined initial position shown in FIG. 6 and moves the second imaging device 32 to the predetermined initial position shown in FIG. 6. The first imaging device 31 and the second imaging device 32 are moved to their respective initial positions after the robot system 10 is powered on and before the first imaging device 31 and the second imaging device 32 are used. The first imaging device 31 and the second imaging device 32 may be moved to their respective initial positions, for example, immediately after the robot system 10 is powered on.

[0068] The imaging device control unit 43, for example, brings the first imaging device 31 and the second imaging device 32 into contact with each other in the circumferential direction, thereby suitably and easily moving each of the first imaging device 31 and the second imaging device 32 to their initial positions. Note that the robot system 10 may have a sensor capable of detecting that the first imaging device 31 and the second imaging device 32 have come into contact with each other in the circumferential direction.

[0069] As described above, in this embodiment, at least the first imaging device 31 moves to a predetermined initial position after the robot system 10 is powered on. More specifically, both the first imaging device 31 and the second imaging device 32 move to their predetermined initial positions after the robot system 10 is powered on. Note that, of the first imaging device 31 and the second imaging device 32, only the first imaging device 31 may move to its predetermined initial position after the robot system 10 is powered on, or, of the first imaging device 31 and the second imaging device 32, only the second imaging device 32 may move to its predetermined initial position after the robot system 10 is powered on.

[0070] In this embodiment, when the control unit 40 moves the first imaging device 31 and the second imaging device 32 using the imaging device control unit 43, the control unit 40 keeps the member to which the imaging device 30 is attached, that is, the end effector 22 in this embodiment, stationary. That is, in this embodiment, the movement of the first imaging device 31 and the movement of the second imaging device 32 are performed while the members to which the first imaging device 31 and the second imaging device 32 are attached are stationary. In this embodiment, the movement of the first imaging device 31 to a predetermined initial position and the movement of the second imaging device 32 to a predetermined initial position are also performed while the members to which the first imaging device 31 and the second imaging device 32 are attached (end effector 22) are stationary.

[0071] When at least one of the first imaging device 31 and the second imaging device 32 cannot capture an image of the object W, the control unit 40 may move at least one of the first imaging device 31 and the second imaging device 32 to positions where both the first imaging device 31 and the second imaging device 32 can capture an image of the object W. A case where the imaging device 30 cannot capture an image of the object W is, for example, when an obstacle is placed between the imaging device 30 and the object W and the object W is not captured by the imaging device 30. The control unit 40 may move the first imaging device 31 and the second imaging device 32 to positions where they do not interfere with the operation of the end effector 22, depending on the operation on the object W by the end effector 22.

[0072] The distance information acquisition unit 44 acquires information related to the distance to the object W. The information related to the distance to the object W includes, for example, the distance from the imaging device 30 to the object W, the distance from the end effector 22 to the object W, the distance from the robot arm 21 to the object W, the distances between a plurality of objects W, and 3D point cloud data related to the object W. Information on images captured by the imaging elements 31f, 32f is input to the distance information acquisition unit 44.

[0073] The distance information acquisition unit 44 receives information about the position of the imaging device 30 from the position acquisition unit 34 of the imaging device 30. The distance information acquisition unit 44 acquires the base length L based on the position information of the first imaging device 31 acquired by the position acquisition unit 34. In this embodiment, the distance information acquisition unit 44 acquires the base length L based on the position information of the first imaging device 31 acquired by the first position acquisition unit 31c and the position information of the second imaging device 32 acquired by the second position acquisition unit 32c. Specifically, the distance information acquisition unit 44 calculates the distance between the optical axis AX1 of the first imaging device 31 and the optical axis AX2 of the second imaging device 32 from the circumferential positions of the first imaging device 31 and the second imaging device 32, thereby acquiring the base length L. Note that the distance information acquisition unit 44 may acquire the base length L from another component, such as the imaging device control unit 43.

[0074] The distance information acquisition unit 44 acquires information about the distance to the object W based on the acquired base line length L, the first image acquired by the first imaging device 31, and the second image acquired by the second imaging device 32. In this embodiment, the orientation of the imaging element 31f of the first imaging device 31 and the orientation of the imaging element 32f of the second imaging device 32 are different from each other when viewed in the central axis direction. Therefore, the distance information acquisition unit 44 rotates at least one of the first image acquired by the first imaging device 31 and the second image acquired by the second imaging device 32 to align the direction (orientation) of the first image with the direction (orientation) of the second image.

[0075] In this manner, in the present embodiment, the distance information acquisition unit 44 adjusts the orientation of the acquired image by rotating at least one of the first image acquired by the first imaging device 31 and the second image acquired by the second imaging device 32. The distance information acquisition unit 44 may rotate only the first image acquired by the first imaging device 31 to align the orientation of the first image with the orientation of the second image, or may rotate only the second image acquired by the second imaging device 32 to align the orientation of the second image with the orientation of the first image, or may rotate both the first image acquired by the first imaging device 31 and the second image acquired by the second imaging device 32 to align the orientation of the first image with the orientation of the second image. In this embodiment, the distance information acquisition unit 44 measures the distance from the imaging device 30 to the object W using the baseline length L and the first and second images whose orientations have been adjusted by rotating them. The control unit 40 controls at least one of the robot arm 21 and the end effector 22 based on the information regarding the distance to the object W acquired in this manner.

[0076] The display unit 50 displays information based on the information about distance. The information about distance includes, for example, information about the distance to the object W and information about the base line length L, which is the distance between the first image capturing device 31 and the second image capturing device 32. The information based on the information about distance may be the information about distance itself, or may be information obtained from the information about distance. The display unit 50 displays, for example, the current distance to the object W and the current base line length L. For example, the display unit 50 may display a graph of the distance to the object W and changes in the base line length L. The display unit 50 may have any structure as long as it can display information based on the information about distance. The display unit 50 may be provided separately from the robot 20, or may be provided on the robot arm 21. The display unit 50 is controlled by the control unit 40.

[0077] According to this embodiment, at least one of the first imaging device 31 and the second imaging device 32 attached to the end effector 22 is movable relative to the end effector 22. Therefore, by moving at least one of the first imaging device 31 and the second imaging device 32, the distance between the first imaging device 31 and the second imaging device 32 can be changed. This makes it possible to change the base length L, which is the distance between the first imaging device 31 and the second imaging device 32.

[0078] Here, when the base line length L is relatively large, the resolution for an object W that is relatively far from the imaging device 30 can be relatively high, and the distance from the imaging device 30 to the object W that is relatively far can be accurately detected. However, in this case, the object W that is relatively close to the imaging device 30 will not be captured by the imaging device 30, and the distance to the object W that is relatively close to the imaging device 30 will not be detected. On the other hand, when the base line length L is relatively small, the imaging device 30 can capture an image of an object W that is relatively close to the imaging device 30, but it is difficult to focus on an object W that is relatively far from the imaging device 30, and it is difficult to accurately detect the distance to the object W that is relatively far from the imaging device 30. As such, the position of the object W at which the distance can be suitably detected varies depending on the base line length L. Therefore, for example, with an imaging device with a fixed base line length L, the position of the object W relative to the imaging device from which distance information can be suitably acquired is limited. As a result, when only this imaging device is used, the work content of the robot system may be limited.

[0079] In contrast to this, according to the present embodiment, the base line length L between the first imaging device 31 and the second imaging device 32 attached to the end effector 22 can be changed as described above. Therefore, by making the base line length L relatively large when the distance between the end effector 22 and the object W is relatively large and making the base line length L relatively small when the distance between the end effector 22 and the object W is relatively small, the distance to the object W can be accurately measured using only the imaging device 30 attached to the end effector 22, even if the distance between the end effector 22 and the object W changes significantly to some extent. Furthermore, by making the base line length L relatively large when the distance between the end effector 22 and the object W is relatively large, the distance to the object W can be detected more accurately by stereo matching. Furthermore, by making the base line length L relatively small when the distance between the end effector 22 and the object W is relatively small, the degree of overlap (overlapping portion) between the image captured by the first imaging device 31 and the image captured by the second imaging device 32 can be increased, and stereo matching can be performed within a relatively wide range within the images captured by each imaging device 30 to measure the distance to the object W. As a result, the robot system 10 can perform work on the object W regardless of the distance between the end effector 22 and the object W. Therefore, restrictions on the work content of the robot system 10 can be suppressed. This improves the workability on the object W.

[0080] Specifically, for example, tasks such as searching the workbench WB from a relatively long distance to find the object W, bringing the end effector 22 closer to the found object W, grabbing the object W with the end effector 22, and moving the grabbed object W to another location can be performed while preferably obtaining the distance to the object W using only the first imaging device 31 and the second imaging device 32 attached to the end effector 22.

[0081] Furthermore, for example, if a device capable of measuring the distance to an object W on which the robot system is to work is installed on the ceiling or the like of the location where the robot system is installed, the baseline length L of the device can be made different from the baseline length L of the imaging device attached to the end effector, so that the robot system can perform work while grasping the distance to the object even when the distance between the end effector and the object varies over a fairly wide range. However, in this case, there is a problem that the cost of installing the device is required. Also, there is a problem that the robot system can only be used in the location where the device is installed.

[0082] In contrast, according to the present embodiment, the base length L between the first imaging device 31 and the second imaging device 32 attached to the end effector 22 can be changed. Therefore, even if the distance between the end effector 22 and the object W varies over a fairly wide range, the robot system 10 can perform work on the object W without installing the above-mentioned equipment on the ceiling or the like. This eliminates the cost of installing such equipment. Furthermore, the robot system 10 can be used even in places where such equipment is not installed. This improves the degree of freedom in the places where the robot system 10 can be used.

[0083] Furthermore, when either the first imaging device 31 or the second imaging device 32 is in a position where it cannot image the object W, it is easy to image the object W by both the first imaging device 31 and the second imaging device 32 without moving the end effector 22 by moving the first imaging device 31 or the second imaging device 32 relative to the end effector 22. This makes it possible to preferably acquire information about the distance to the object W regardless of the position, posture, etc. of the end effector 22.

[0084] Furthermore, at least one of the first imaging device 31 and the second imaging device 32 can be moved to a suitable position in accordance with the path of movement of the robot arm 21 and the end effector 22 and the surrounding environment in which the robot arm 21 and the end effector 22 are placed. For example, when the robot arm 21 and the end effector 22 are moved relative to the object W, the first imaging device 31 and the second imaging device 32 can be moved so that they do not come into contact with other objects, etc. This improves the degree of freedom of movement of the robot arm 21 and the end effector 22.

[0085] Furthermore, for example, by moving at least one of the first imaging device 31 and the second imaging device 32, the inertia of the robot 20 when it moves can be optimized. Specifically, for example, when the end effector 22 is not gripping the object W, the total inertia of the end effector 22, the first imaging device 31, and the second imaging device 32 can be minimized by arranging the first imaging device 31 and the second imaging device 32 on opposite sides of the central axis CL. This makes it easier to move the end effector 22 to which the first imaging device 31 and the second imaging device 32 are attached. Also, for example, when the end effector 22 is gripping the object W, at least one of the first imaging device 31 and the second imaging device 32 can be moved to a position where the first imaging device 31 and the second imaging device 32 function as a counterweight for the gripped object W. This makes it easier to minimize the total inertia of the end effector 22, the first imaging device 31, the second imaging device 32, and the object W. Therefore, in a state in which the end effector 22 is gripping the object W, the end effector 22 can be easily moved in a suitable manner.

[0086] Furthermore, if the distance between the object W and the first and second imaging devices 31 and 32 changes within a range in which the object W can be imaged by the first and second imaging devices 31 and 32, the base line length L may be increased as the first and second imaging devices 31 and 32 get closer to the object W. Here, the greater the base line length L, the more accurate the detection of the distance to the object W can be. Therefore, as long as the object W is within a range in which it can be imaged, by increasing the base line length L as the object gets closer to the object W, the distance to the object W can be obtained with greater accuracy, making it easier to perform precision work on the object W.

[0087] For example, when the end effector 22 is a robot hand that grasps the object W, or when the end effector 22 is a tool that performs work on the object W, the closer the end effector 22 is to the object W, the more accurately the distance to the object W can be obtained, making it easier to perform precise work on the object W using the end effector 22. Furthermore, when the end effector 22 is relatively far from the object W, the base line length L is relatively small, so the overlapping area (overlapping portion) between the image captured by the first imaging device 31 and the image captured by the second imaging device 32 becomes large, and the distance can be measured within a relatively wide range that includes the object W.

[0088] Furthermore, according to this embodiment, the optical axis AX1 of the first imaging device 31 and the optical axis AX2 of the second imaging device 32 are parallel to each other. Therefore, it is easy to preferably obtain the distance to the object W based on the first image obtained by the first imaging device 31 and the second image obtained by the second imaging device 32.

[0089] Furthermore, according to this embodiment, the control unit 40 that controls the robot system 10 can change the base line length L, which is the distance between the first imaging device 31 and the second imaging device 32, and the distance information acquisition unit 44 can acquire information about the distance to the object W based on the base line length L. Therefore, the base line length L can be easily changed depending on the work content of the robot system 10. Furthermore, the distance information acquisition unit 44 can easily acquire information about the distance to the object W.

[0090] Furthermore, according to this embodiment, a position acquisition unit 34 is provided that acquires at least the position information of the first imaging device 31, and the distance information acquisition unit 44 acquires the base line length L based on the position information of the first imaging device 31 acquired by the position acquisition unit 34. Therefore, the distance information acquisition unit 44 can preferably acquire the base line length L, and can also preferably acquire information about the distance to the target W based on the acquired base line length L.

[0091] Furthermore, according to the present embodiment, the control unit 40 changes the base line length L in accordance with the work content of the robot system 10. Therefore, the base line length L between the first imaging device 31 and the second imaging device 32 can be suitably changed in accordance with the work content of the robot system 10. This allows the robot system 10 to suitably perform the work. Specifically, for example, when performing work to search for an object W from a relatively long distance, the base line length L can be made relatively large to accurately obtain the distance to the object W, which is relatively far away, making it easier to search for the object W. Furthermore, for example, when performing work to grasp the object W with the end effector 22, the base line length L can be made relatively small to obtain the distance to the object W, which is relatively small, making it easier to suitably grasp the object W with the end effector 22.

[0092] Furthermore, according to this embodiment, the distance information acquisition unit 44 adjusts the orientation of the acquired image by rotating at least one of the first image acquired by the first imaging device 31 and the second image acquired by the second imaging device 32. Therefore, even if the imaging element 31f of the first imaging device 31 and the imaging element 32f of the second imaging device 32 are disposed in different orientations, the orientation of the first image acquired by the first imaging device 31 and the orientation of the second image acquired by the second imaging device 32 can be aligned. This allows the distance information acquisition unit 44 to suitably acquire information about the distance to the object W based on the images acquired by each imaging device 30, regardless of the relative position and relative orientation between the first imaging device 31 and the second imaging device 32. Therefore, even if at least one of the first imaging device 31 and the second imaging device 32 is moved to place the first imaging device 31 and the second imaging device 32 in any position and orientation, information about the distance to the object W can be suitably acquired.

[0093] Furthermore, according to this embodiment, the first imaging device 31 and the second imaging device 32 are disposed around the end effector 22. Therefore, it is easy to measure the distance between the end effector 22 and the object W from images captured by the first imaging device 31 and the second imaging device 32. Furthermore, for example, by disposing the first imaging device 31 and the second imaging device 32 radially outward of the base 22a as in this embodiment, the first imaging device 31 and the second imaging device 32 are less likely to interfere with the operation of gripping the object W with the end effector 22.

[0094] Furthermore, according to this embodiment, at least one of the first imaging device 31 and the second imaging device 32 is movable in a predetermined circumferential direction around the end effector 22. Therefore, by moving at least one of the first imaging device 31 and the second imaging device 32, the circumferential distance between the first imaging device 31 and the second imaging device 32 can be changed, and the base length L between the first imaging device 31 and the second imaging device 32 can be easily changed.

[0095] Furthermore, according to this embodiment, at least the first imaging device 31 moves to a predetermined initial position after the robot system 10 is powered on. Therefore, before using the first imaging device 31, the first position acquisition unit 31c of the first imaging device 31 can be calibrated. As a result, even if the first imaging device 31 is moved, the position of the first imaging device 31 can be accurately detected based on the predetermined initial position. In this embodiment, both the first imaging device 31 and the second imaging device 32 move to their predetermined initial positions after the robot system 10 is powered on. Therefore, the position detection units of each imaging device 30 can be calibrated, and the positions of each imaging device 30 can be accurately detected. This allows the base length L between the first imaging device 31 and the second imaging device 32 to be accurately changed, and information about the distance to the target W can be more appropriately acquired based on the base length L. Furthermore, in this embodiment, by bringing the first imaging device 31 and the second imaging device 32 into contact with each other in the circumferential direction, the first imaging device 31 and the second imaging device 32 can be easily moved to their initial positions.

[0096] Furthermore, according to this embodiment, the movement of the first imaging device 31 to a predetermined initial position is performed while the member to which the first imaging device 31 is attached, i.e., in this embodiment, the end effector 22, is stationary. Therefore, it is easier to move the first imaging device 31 than when the first imaging device 31 is moved to the initial position while the end effector 22 is moving. Furthermore, since the first imaging device 31 can be prevented from moving while the end effector 22 is moving, it is possible to prevent the movement calculations of the end effector 22 and the robot arm 21 from becoming complicated. In this embodiment, the movement of the second imaging device 32 to a predetermined initial position is also performed while the end effector 22 is stationary. Therefore, it is easier to move the second imaging device 32 to the initial position. Furthermore, since the second imaging device 32 can be prevented from moving while the end effector 22 is moving, it is possible to further prevent the movement calculations of the end effector 22 and the robot arm 21 from becoming complicated.

[0097] In this embodiment, the movement of the first imaging device 31 relative to the end effector 22 and the movement of the second imaging device 32 relative to the end effector 22 are all performed while the members to which the first imaging device 31 and the second imaging device 32 are attached, i.e., in this embodiment, the end effector 22, are stationary. Therefore, while the end effector 22 is moving, the first imaging device 31 and the second imaging device 32 do not move relative to the end effector 22. This makes it possible to further prevent the movement calculations of the end effector 22 and the robot arm 21 from becoming complicated.

[0098] Furthermore, according to this embodiment, a display unit 50 is provided that displays information based on information related to distance. Therefore, an operator of the robot system 10 can easily obtain information related to distance by looking at the display unit 50.

[0099] In the above description, the orientation of the captured image is adjusted by rotating at least one of the first image captured by the first imaging device 31 and the second image captured by the second imaging device 32. However, this is not limiting. In this embodiment, the control unit 40 may adjust the orientation of the captured image by rotating at least one of the imaging element 31f of the first imaging device 31 and the imaging element 32f of the second imaging device 32. In this case, the control unit 40 rotates at least one of the imaging element 31f of the first imaging device 31 and the imaging element 32f of the second imaging device 32 so that the long sides of the imaging element 31f and the imaging element 32f are parallel to each other. This allows the orientation of the images captured by each imaging device 30 to be aligned without performing processing such as rotation on the captured images. Therefore, the load of image processing by the control unit 40 can be reduced compared to when processing such as rotation is performed on the captured images.

[0100] When rotating the imaging element 31f of the first imaging device 31, the control unit 40 may rotate the entire first imaging device 31 together with the imaging element 31f, or may rotate only the imaging element 31f of the first imaging device 31. When rotating the imaging element 31f of the first imaging device 31, the control unit 40 rotates the imaging element 31f about the optical axis AX1. In this case, the first imaging device 31 is attached to the end effector 22 so as to be rotatable about the optical axis AX1.

[0101] When rotating the imaging element 32f of the second imaging device 32, the control unit 40 may rotate the entire second imaging device 32 including the imaging element 32f, or may rotate only the imaging element 32f of the second imaging device 32. When rotating the imaging element 32f of the second imaging device 32, the control unit 40 rotates the imaging element 32f about the optical axis AX2. In this case, the second imaging device 32 is attached to the end effector 22 so as to be rotatable about the optical axis AX2.

[0102] In this embodiment, the end effector 22 may have a first holding portion that immobilizes the first imaging device 31. In this case, the end effector 22 may have a guide rail portion 22e as a second holding portion that movably holds the second imaging device 32. In addition, the end effector 22 may have a second holding portion that immobilizes the second imaging device 32. In this case, the end effector 22 may have a guide rail portion 22e as a first holding portion that movably holds the first imaging device 31.

[0103] In this embodiment, at least the first imaging device 31 may be moved to a predetermined end position before the robot system 10 is powered off. In this case, the control unit 40 may, for example, after receiving a command to stop the robot system 10, bring the first imaging device 31 and the second imaging device 32 into contact with each other in the circumferential direction and move both the first imaging device 31 and the second imaging device 32 to the predetermined end position. The predetermined end position may be the same as or different from the predetermined initial position.

[0104] For example, if the predetermined end position is the same as the predetermined initial position, the first imaging device 31 and the second imaging device 32 will be located at the predetermined initial position when the robot system 10 is powered on again. Therefore, there is no need to provide a step of moving the first imaging device 31 and the second imaging device 32 to the predetermined initial position after the robot system 10 is powered on. As a result, even if the first imaging device 31 and the second imaging device 32 are moved immediately after the robot system 10 is powered on, the position acquisition unit 34 can preferably acquire the positions of the imaging devices 30.

[0105] The movement of the first imaging device 31 to the predetermined end position is performed, for example, while the member to which the first imaging device 31 is attached is stationary. This makes it easier to move the first imaging device 31 to the end position. In addition, since the first imaging device 31 can be prevented from moving while the end effector 22 is moving, it is possible to further prevent the movement calculations of the end effector 22 and the robot arm 21 from becoming complicated.

[0106] The movement of the second imaging device 32 to the predetermined end position is performed, for example, while the member to which the second imaging device 32 is attached is stationary. This makes it easier to move the second imaging device 32 to the end position. In addition, since the second imaging device 32 can be prevented from moving while the end effector 22 is moving, it is possible to further prevent the movement calculations of the end effector 22 and the robot arm 21 from becoming complicated.

[0107] Furthermore, for an overlapping portion (a portion that appears in both images) between the image captured by the first imaging device 31 and the image captured by the second imaging device 32, the distance to a characteristic portion such as the object W reflected in the overlapping portion can be measured throughout the overlapping portion. Therefore, the control unit 40 may measure the distance throughout the overlapping portion between the image captured by the first imaging device 31 and the image captured by the second imaging device 32 and create a depth map for the overlapping portion. The depth map is an image that indicates distance information using, for example, differences in color or shades of color.

[0108] Furthermore, the control unit 40 may measure the distance to the object W using the images captured by the first imaging device 31 and the second imaging device 32 and the image captured by the camera unit 60. For example, when the distance to the object W measured from the images captured by the first imaging device 31 and the second imaging device 32 becomes equal to or shorter than a predetermined distance, the control unit 40 may turn on the power of the camera unit 60 and measure the distance to the object W using the images captured by the first imaging device 31 and the second imaging device 32 and the image captured by the camera unit 60.

[0109] Furthermore, the control unit 40 may switch between a first imaging mode in which the distance to the object W is measured using images captured by the first imaging device 31 and the second imaging device 32, and a second imaging mode in which the distance to the object W is measured using images captured by the camera unit 60, depending on the distance to the object W. In this case, the control unit 40 may, for example, measure the distance to the object W in the first imaging mode described above when the distance to the object W is greater than a predetermined distance, and may measure the distance to the object W in the second imaging mode described above when the distance to the object W is equal to or less than the predetermined distance.

[0110] Second Embodiment Fig. 7 is a perspective view showing a part of the robot system 110 of this embodiment. In Fig. 7, the finger portion 22b of the end effector 122 and the camera unit 60 are not shown. Note that the same components as those in the above-described embodiment may be denoted by the same reference numerals as appropriate and the description thereof may be omitted.

[0111] 7, in the robot system 110 of this embodiment, the second imaging device 132 is fixed to the end effector 122. That is, in this embodiment, the second imaging device 132 does not move relative to the end effector 122. Therefore, in this embodiment, only the first imaging device 31 is movable in a predetermined circumferential direction around the end effector 122. In this way, in this embodiment, one of the first imaging device 31 and the second imaging device 132 is movable in a predetermined circumferential direction around the end effector 122, and the other is fixed to a predetermined portion of the end effector 122.

[0112] The second imaging device 132 is fixed to, for example, the base 122a. The base 122a has a hole 122f recessed from the surface on the tip side (+Z side) of the base 122a toward the base side (-Z side). The hole 122f is, for example, a circular hole centered on the central axis CL.

[0113] A proximal end (-Z side) portion of the second imaging device 132 is fitted into and held in the hole 122f. In this embodiment, the hole 122f corresponds to a second holding portion that holds the second imaging device 132. That is, in this embodiment, the end effector 122 has the hole 122f as the second holding portion. In this embodiment, the second imaging device 132 is held immovably by the hole 122f that serves as the second holding portion. A distal end (+Z side) portion of the second imaging device 132 protrudes toward the distal end from the center of the distal end surface of the base 122a. The base 122a has the same configuration as the base 22a of the first embodiment described above, except that the hole 122f is provided.

[0114] The optical axis AX2a of the second imaging device 132 is, for example, parallel to the optical axis AX1 of the first imaging device 31 and coincides with the central axis CL. The second imaging device 132 has a cylindrical housing 132a, a lens 132e fitted in an opening on the tip side (+Z side) of the housing 132a, and an imaging element 132f arranged inside the housing 132a. Each component of the second imaging device 132 can be similar to each component of the second imaging device 32 of the first embodiment described above. Unlike the second imaging device 32 of the first embodiment, the second imaging device 132 does not have a second driving unit 32b or a second position acquisition unit 32c.

[0115] In this embodiment, the base length L between the first imaging device 31 and the second imaging device 132 is fixed. In this embodiment, the distance information acquisition unit 44 acquires information about the distance to the object W based on the base length L and the two images acquired from the first imaging device 31 and the second imaging device 132, similar to the first embodiment described above.

[0116] In this embodiment, the distance information acquisition unit 44 can also acquire information about the distance to the object W based on two images captured by the first imaging device 31. Specifically, for example, the first imaging device 31 acquires a first image when the first imaging device 31 is located at a first position P1 indicated by a solid line in FIG. 7, and the first imaging device 31 acquires a second image when the first imaging device 31 is located at a second position P2 indicated by a two-dot chain line in FIG. 7. In this embodiment, the distance information acquisition unit 44 can also acquire information about the distance to the object W based on the first and second images acquired in this manner. More specifically, the distance information acquisition unit 44 can also acquire information about the distance to the object W based on the first and second images acquired by the first imaging device 31 and the base length La, which is the distance between the first imaging device 31 at the first position P1 and the first imaging device 31 at the second position P2.

[0117] The base length La is the distance between the optical axis AX1a of the first image capturing device 31 at the first position P1 and the optical axis AX1b of the first image capturing device 31 at the second position P2. The base length La is determined by the first position P1 and the second position P2. In this embodiment, the control unit 40 can change the base length La by changing the first position P1 and the second position P2 at which the first image capturing device 31 captures an image. The control unit 40 changes the base length La depending on, for example, the work content of the robot system 110.

[0118] In this embodiment, the position acquisition unit 134 acquires position information regarding the first position P1 and the second position P2. The position information regarding the first position P1 and the second position P2 includes, for example, information regarding the first position P1, information regarding the second position P2, and information indicating the relative positional relationship between the first position P1 and the second position P2. In this embodiment, the position acquisition unit 134 is composed only of the first position acquisition unit 31c of the first imaging device 31. The position acquisition unit 134 acquires the position information regarding the first position P1 and the second position P2 based on, for example, the rotational position of the drive unit 133. In this embodiment, the drive unit 133 is composed only of the first drive unit 31b of the first imaging device 31. The distance information acquisition unit 44 acquires the baseline length La based on the position information regarding the first position P1 and the second position P2 acquired by the position acquisition unit 134.

[0119] The distance information acquisition unit 44 can select, for example, depending on the work content of the robot system 110, whether to acquire information regarding the distance to the object W using images acquired by each of the first imaging device 31 and the second imaging device 132, or to acquire information regarding the distance to the object W using two images acquired by the first imaging device 31 at different first and second positions P1 and P2.

[0120] When obtaining information regarding the distance to the object W using images obtained by the first imaging device 31 and the second imaging device 132, the distance information obtaining unit 44 adjusts the orientation of the obtained images by rotating at least one of the images obtained by the first imaging device 31 and the second imaging device 132, as in the first embodiment.

[0121] On the other hand, when information about the distance to the target object W is acquired using two images acquired by the first imaging device 31 at different positions, a first position P1 and a second position P2, the distance information acquisition unit 44 adjusts the orientation of the acquired images by rotating at least one of the first image acquired by the first imaging device 31 at the first position P1 and the second image acquired by the first imaging device 31 at the second position P2. The distance information acquisition unit 44 may rotate only the first image acquired at the first position P1 to align the orientation of the first image with the orientation of the second image, or may rotate only the second image acquired at the second position P2 to align the orientation of the second image with the orientation of the second image, or may rotate both the first image acquired at the first position P1 and the second image acquired at the second position P2 to adjust the orientation of the first image and the orientation of the second image. Other configurations of the robot system 110 of this embodiment may be similar to other configurations of the robot systems of the above-described embodiments.

[0122] According to this embodiment, the first imaging device 31 is movable in a predetermined circumferential direction around the end effector 122, and the second imaging device 132 is fixed to a predetermined portion of the end effector 122. Therefore, it is easier to simplify the structure of the robot system 110 compared to when both the first imaging device 31 and the second imaging device 132 are provided to be movable.

[0123] Furthermore, according to this embodiment, the first imaging device 31 captures a first image of the object W at a first position P1, and captures a second image of the object W at a second position P2 different from the first position P1. The distance information acquisition unit 44 acquires information about the distance to the object W based on the first image and the second image. Therefore, information about the distance to the object W can be acquired using only the image acquired by the single first imaging device 31. As a result, even if the second imaging device 132 is not provided, information about the distance to the object W can be acquired using only the single first imaging device 31. Furthermore, the base length La can be changed by changing the relative position between the first position P1 and the second position P2.

[0124] Furthermore, according to this embodiment, the distance information acquisition unit 44 adjusts the orientation of the acquired image by rotating at least one of the first image acquired by the first imaging device 31 at the first position P1 and the second image acquired by the first imaging device 31 at the second position P2. Therefore, even if the imaging element 31f of the first imaging device 31 at the first position P1 and the imaging element 132f of the second imaging device 132 at the second position P2 are disposed in different orientations, the orientation of the first image acquired at the first position P1 can be aligned with the orientation of the second image acquired at the second position P2. As a result, regardless of the positions of the first position P1 and the second position P2, the distance information acquisition unit 44 can suitably acquire information about the distance to the object W based on the image acquired by the first imaging device 31.

[0125] Furthermore, according to this embodiment, the control unit 40 can change the base length La, which is the distance between the first image capturing device 31 at the first position P1 and the first image capturing device 31 at the second position P2, and the distance information acquisition unit 44 acquires information about the distance to the object W based on the base length La. Therefore, information about the distance to the object W at different base lengths La can be acquired using only the first image capturing device 31.

[0126] Furthermore, according to this embodiment, the position acquisition unit 134 acquires position information regarding the first position P1 and the second position P2, and the distance information acquisition unit 44 acquires the base line length La based on the position information regarding the first position P1 and the second position P2 acquired by the position acquisition unit 134. Therefore, the distance information acquisition unit 44 can preferably acquire the base line length La, and can also preferably acquire information regarding the distance to the target W based on the acquired base line length La.

[0127] Furthermore, according to this embodiment, the control unit 40 changes the base line length La in accordance with the work content of the robot system 110. Therefore, the base line length La, which is the distance between the first image capturing device 31 at the first position P1 and the first image capturing device 31 at the second position P2, can be suitably changed in accordance with the work content of the robot system 110. This allows the robot system 110 to suitably perform each work.

[0128] Furthermore, according to this embodiment, the distance information acquisition unit 44 adjusts the orientation of the acquired image by rotating at least one of the first image acquired by the first imaging device 31 and the second image acquired by the second imaging device 32. Therefore, even if the imaging element 31f of the first imaging device 31 and the imaging element 32f of the second imaging device 32 are disposed in different orientations, the orientation of the first image acquired by the first imaging device 31 and the orientation of the second image acquired by the second imaging device 32 can be aligned. This allows the distance information acquisition unit 44 to suitably acquire information about the distance to the object W based on the images acquired by each imaging device 30, regardless of the relative position and relative orientation between the first imaging device 31 and the second imaging device 32. Therefore, even if at least one of the first imaging device 31 and the second imaging device 32 is moved to place the first imaging device 31 and the second imaging device 32 in any position and orientation, information about the distance to the object W can be suitably acquired.

[0129] In this embodiment, the first imaging device 31 may be movable in the radial direction. In this case, by changing the radial position of the first imaging device 31, it is possible to change the base length, which is the distance between the first imaging device 31 and the second imaging device 132. In this embodiment, the second imaging device 132 does not have to be provided. Even in this case, as described above, it is possible to obtain information about the distance to the object W using only the first imaging device 31.

[0130] Furthermore, in this embodiment, the first imaging device 31 may be movable so that the long sides of the imaging element 31f of the first imaging device 31 at the first position P1 and the long sides of the imaging element 31f of the first imaging device 31 at the second position P2 are parallel to each other. This allows the orientation of the images captured by the first imaging device 31 at the first position P1 and the second position P2 to be aligned without performing processing such as rotation on the captured images. This reduces the image processing load on the control unit 40 compared to when performing processing such as rotation on the captured images. In this case, the first imaging device 31 is mounted rotatably around the optical axis AX1, for example. For example, the control unit 40 rotates the first imaging device 31 around the optical axis AX1 depending on the circumferential position of the first imaging device 31, adjusting the long sides of the imaging element 31f to always be oriented in the same direction. In this way, in this embodiment, the control unit 40 may adjust the orientation of the image captured by the first imaging device 31 by rotating the imaging element 31f of the first imaging device 31.

[0131] Furthermore, in a case where only one first imaging device 31 is movable relative to the member to which it is attached, as in the present embodiment, the first imaging device 31 may be movable relative to the robot arm 21 or the adapter 23. When the first imaging device 31 is movable relative to the robot arm 21, the first imaging device 31 may be movable in a predetermined circumferential direction around the robot arm 21. When the first imaging device 31 is movable relative to the adapter 23, the first imaging device 31 may be movable in a predetermined circumferential direction around the adapter 23. Even in these cases, the distance information acquisition unit 44 can acquire information about the distance to the target W based on the first image and the second image captured by the first imaging device 31 at different first position P1 and second position P2, respectively.

[0132] <Third embodiment> 8 is a view of a part of the robot system 210 of this embodiment as seen from the tip side (+Z side) in the central axis direction. The finger portion 22b of the end effector 22 and the camera unit 60 are not shown in FIG. Note that the same components as those in the above-described embodiments may be denoted by the same reference numerals as appropriate and the description thereof may be omitted.

[0133] As shown in FIG. 8, the robot system 210 of this embodiment has three or more imaging devices 230 that capture images of the target object W. For example, three imaging devices 230 are provided: imaging device 230a, imaging device 230b, and imaging device 230c. In this embodiment, the three imaging devices 230a, 230b, and 230c are arranged side by side on a predetermined axis VA. The axis VA is, for example, a virtual axis extending in a direction perpendicular to both the central axis direction and the radial direction (the left-right direction in FIG. 8). The imaging devices 230a, 230b, and 230c are arranged at equal intervals along the axis VA. In the axis VA direction, the imaging device 230b is located between the imaging devices 230a and 230c. The optical axis AX3a of the imaging device 230a, the optical axis AX3b of the imaging device 230b, and the optical axis AX3c of the imaging device 230c extend, for example, in the central axis direction and are parallel to each other.

[0134] The imaging element 235a of the imaging device 230a, the imaging element 235b of the imaging device 230b, and the imaging element 235c of the imaging device 230c are rectangular when viewed in the central axis direction. In this embodiment, the imaging elements 235a, 235b, and 235c are arranged in the same orientation. In this embodiment, the three imaging devices 230a, 230b, and 230c are arranged so that the long sides of the imaging elements 235a, 235b, and 235c of the three imaging devices 230a, 230b, and 230c are parallel to each other.

[0135] The robot system 210 includes a holding member 230d that holds three imaging devices 230a, 230b, and 230c, and a slider 230e that connects the holding member 230d to the base 22a of the end effector 22. The three imaging devices 230a, 230b, and 230c are fixed to the holding member 230d so as to be immovable relative to one another. Similar to the sliders 31d and 32d of the first embodiment, the slider 230e is connected to a guide rail portion 22e provided on the base 22a of the end effector 22. The slider 230e is movable circumferentially around the base 22a along the guide rail portion 22e.

[0136] Although not shown, the robot system 210 has a drive unit that moves the slider 230e in the circumferential direction. When the drive unit moves the slider 230e in the circumferential direction along the guide rail portion 22e, the holding member 230d and the three imaging devices 230a, 230b, and 230c held by the holding member 230d move in the circumferential direction. In this embodiment, the three imaging devices 230a, 230b, and 230c move in the circumferential direction while the long sides of the imaging elements 235a, 235b, and 235c remain parallel to each other.

[0137] In this embodiment, the control unit 40 acquires information about the distance to the object W based on information about images of the object W acquired by two of the three imaging devices 230a, 230b, and 230c. The control unit 40 selects, for example, two of the three imaging devices 230a, 230b, and 230c, and acquires information about the distance to the object W based on information about images acquired by the selected two imaging devices 230. There are three patterns for the two imaging devices 230 to be selected: imaging device 230a and imaging device 230b, imaging device 230b and imaging device 230c, and imaging device 230a and imaging device 230c.

[0138] Baseline length L3, which is the distance between image capturing device 230a and image capturing device 230b, and baseline length L4, which is the distance between image capturing device 230a and image capturing device 230c, are different from each other. That is, the baseline length differs between when image capturing device 230a and image capturing device 230b are selected as the two image capturing devices 230 and when image capturing device 230a and image capturing device 230c are selected as the two image capturing devices 230. As a result, in this embodiment, the control unit 40 can change the baseline length by changing the two image capturing devices 230 selected when acquiring information about the distance to the target W. Baseline length L3 is, for example, shorter than baseline length L4. Baseline length L3 is the distance between the optical axis AX3a of image capturing device 230a and the optical axis AX3b of image capturing device 230b. Baseline length L4 is the distance between the optical axis AX3a of image capturing device 230a and the optical axis AX3c of image capturing device 230c.

[0139] The base length, which is the distance between imaging device 230b and imaging device 230c, is the same as, for example, base length L3, which is the distance between imaging device 230a and imaging device 230b. The base length, which is the distance between imaging device 230b and imaging device 230c, is the distance between optical axis AX3b of imaging device 230b and optical axis AX3c of imaging device 230c.

[0140] The control unit 40 changes the two selected imaging devices 230 and changes the baseline length, for example, depending on the work content of the robot system 210. The control unit 40 acquires information about the distance to the object W using the distance information acquisition unit 44, based on the images acquired by the two selected imaging devices 230 and the baseline length between the two imaging devices 230. The control unit 40 controls at least one of the robot arm 21 and the end effector 22, based on the information about the distance to the object W acquired in this manner.

[0141] In this embodiment, the control unit 40 controls at least one of the robot arm 21 and the end effector 22 connected to the robot arm 21 based on information about images acquired by two of the three imaging devices 230a, 230b, and 230c. In this embodiment, the control unit 40 selects two of the three imaging devices 230a, 230b, and 230c, and controls at least one of the robot arm 21 and the end effector 22 based on information about images acquired by the selected two imaging devices 230.

[0142] Specifically, for example, when the object W is not captured in at least one of the images captured by the two selected imaging devices 230, the control unit 40 moves at least one of the robot arm 21 and the end effector 22 so that the object W can be captured by both of the two selected imaging devices 230.

[0143] In addition, for example, if the object W is not captured in at least one of the images captured by the two selected imaging devices 230, the control unit 40 may move the three imaging devices 230 in a circumferential direction so that the object W can be captured by both of the two selected imaging devices 230.

[0144] In this embodiment, the control unit 40 controls so as to synchronize the imaging by at least two of the three imaging devices 230a, 230b, and 230c. The control unit 40 controls so as to synchronize the imaging by the two selected imaging devices 230, for example.

[0145] According to this embodiment, the control unit 40 acquires information about the distance to the object W based on information about images of the object W acquired by two of the three imaging devices 230a, 230b, and 230c. Therefore, by changing which of the two imaging devices 230 to use images acquired by, depending on the work content of the robot system 210, the object W, and the like, information about the distance to the object W can be suitably acquired. In this embodiment, for example, the baseline length can be changed depending on whether to use images acquired by the two imaging devices 230a and 230b or images acquired by the two imaging devices 230a and 230c. Therefore, by appropriately selecting which of the two imaging devices 230 to use depending on the distance to the object W and the like, information about the distance to the object W can be suitably acquired.

[0146] Furthermore, according to this embodiment, the control unit 40 selects two of the three imaging devices 230a, 230b, and 230c, and acquires information about the distance to the object W based on information about images acquired by the two selected imaging devices 230. Therefore, when acquiring information about the distance to the object W, it is sufficient to perform imaging using two of the three imaging devices 230a, 230b, and 230c, and the remaining imaging device 230 does not need to perform imaging. Therefore, the load on the control unit 40 when acquiring information about the distance to the object W can be reduced.

[0147] Furthermore, according to this embodiment, the control unit 40 controls at least one of the robot arm 21 and the end effector 22 connected to the robot arm 21 based on information about images acquired by two of the three imaging devices 230a, 230b, and 230c. Therefore, information such as the position of the target object W and the environment in which the robot system 210 is placed can be acquired from the images acquired by the two imaging devices 230, and the robot arm 21 and the end effector 22 can be suitably moved according to the work content of the robot system 210, etc.

[0148] Furthermore, according to this embodiment, the control unit 40 selects two imaging devices 230 from the three imaging devices 230a, 230b, and 230c, and controls at least one of the robot arm 21 and the end effector 22 connected to the robot arm 21 based on information about images acquired by the selected two imaging devices 230. Therefore, when controlling at least one of the robot arm 21 and the end effector 22 based on information about images acquired by the two imaging devices 230, it is not necessary to perform imaging using the remaining imaging device 230. This reduces the load on the control unit 40 when controlling the robot arm 21 and the end effector 22.

[0149] Furthermore, according to this embodiment, the control unit 40 controls the imaging by at least two of the three imaging devices 230a, 230b, and 230c to be synchronized. Therefore, the object W can be suitably imaged at the same time by the at least two imaging devices 230. This makes it possible to suitably acquire information about the distance to the object W based on the images acquired by the at least two imaging devices 230.

[0150] Furthermore, according to this embodiment, the three imaging devices 230a, 230b, and 230c are arranged side by side on a predetermined axis VA. Therefore, as in this embodiment, the imaging devices 230a, 230b, and 230c can be arranged side by side with the orientations of the image sensors 235a, 235b, and 235c aligned. As a result, even when information about the distance to the object W is acquired using images acquired by any two of the three imaging devices 230a, 230b, and 230c, the information about the distance to the object W can be easily acquired without rotating the images acquired by the imaging devices 230. Therefore, the load on the control unit 40 when acquiring information about the distance to the object W can be reduced.

[0151] Furthermore, according to this embodiment, the optical axes AX3a, AX3b, and AX3c of the three imaging devices 230a, 230b, and 230c are parallel to one another. Therefore, when using images captured by any two of the three imaging devices 230a, 230b, and 230c, information about the distance to the object W can be suitably acquired from the two images.

[0152] Furthermore, according to this embodiment, the three imaging devices 230a, 230b, and 230c are arranged so that the long sides of the imaging elements 235a, 235b, and 235c in the three imaging devices 230a, 230b, and 230c are parallel to each other. Therefore, it is easy to obtain information about the distance to the object W from the image obtained by the imaging device 230 without rotating the image. This reduces the load on the control unit 40 when obtaining information about the distance to the object W.

[0153] In this embodiment, four or more image capturing devices 230 may be arranged side by side on a predetermined axis VA. In the three or more image capturing devices 230 arranged side by side on the axis VA, the distance between two adjacent image capturing devices 230 may be different from each other.

[0154] <Fourth embodiment> 9 is a view of a part of the robot system 310 of this embodiment as seen from the tip side (+Z side) in the central axis direction. The finger portion 22b of the end effector 22 and the camera unit 60 are not shown in FIG. Note that the same components as those in the above-described embodiments may be denoted by the same reference numerals as appropriate and the description thereof may be omitted.

[0155] 9, the robot system 310 of this embodiment has a connecting member 336 that connects the first imaging device 331 and the second imaging device 332. The connecting member 336 is, for example, a guide rail. The connecting member 336 has a groove 336a that extends linearly. The groove 336a is, for example, a groove that is recessed from the tip end side (+Z side) to the base end side (-Z side). The groove 336a is, for example, open at both ends in the direction in which the groove 336a extends.

[0156] The robot system 310 has a first slider 331g that attaches the first imaging device 331 to a connecting member 336, and a second slider 332g that attaches the second imaging device 332 to the connecting member 336. The first slider 331g is fixed to the first imaging device 331. The second slider 332g is fixed to the second imaging device 332. The first slider 331g and the second slider 332g are fitted into the groove 336a so as to be movable in, for example, the direction in which the groove 336a extends. In this way, in this embodiment, the first imaging device 331 and the second imaging device 332 are connected by the connecting member 336 via the first slider 331g and the second slider 332g.

[0157] The first slider 331g and the second slider 332g are restricted from moving and rotating relative to the connecting member 336 in directions other than the direction in which the groove 336a extends. The first slider 331g is attached so as to be movable in the circumferential direction and rotatable about the optical axis AX1c of the first imaging device 331. The second slider 332g is attached so as to be movable in the circumferential direction and rotatable about the optical axis AX2c of the second imaging device 332.

[0158] The first imaging device 331 and the second imaging device 332 are movable in the circumferential direction around the base 22a of the end effector 22. In this embodiment, the first imaging device 331 is rotatable together with the first slider 331g about the optical axis AX1c of the first imaging device 331. In this embodiment, the second imaging device 332 is rotatable together with the second slider 332g about the optical axis AX2c of the second imaging device 332. The long sides of the imaging element 331f of the first imaging device 331 and the long sides of the imaging element 332f of the second imaging device 332 are, for example, arranged parallel to each other and to the extending direction of the groove 336a.

[0159] 9, when the first imaging device 331 and the second imaging device 332 move in the circumferential direction from the position indicated by the two-dot chain line to the position indicated by the solid line, the connecting member 336 connecting the first imaging device 331 and the second imaging device 332 also moves according to the positions of the first imaging device 331 and the second imaging device 332. In FIG. 9, for example, the connecting member 336 moves upward while rotating around an axis in the central axis direction.

[0160] The first imaging device 331 and the second imaging device 332 move relative to the connecting member 336 in the direction in which the groove 336a extends, depending on their circumferential positions. The relative change in the positions of the first imaging device 331 and the second imaging device 332 in the direction in which the groove 336a extends changes the base length, which is the distance between the first imaging device 331 and the second imaging device 332. The base length between the first imaging device 331 and the second imaging device 332 is the distance between the optical axis AX1c of the first imaging device 331 and the optical axis AX2c of the second imaging device 332. For example, when the first imaging device 331 and the second imaging device 332 move circumferentially from the position indicated by the two-dot chain line to the position indicated by the solid line in FIG. 9, the first imaging device 331 and the second imaging device 332 move closer to each other, and the base length decreases.

[0161] Here, the first slider 331g and the second slider 332g are restricted from moving and rotating relative to the connecting member 336 in directions other than the direction in which the groove 336a extends. Therefore, the first slider 331g and the second slider 332g rotate about the optical axes AX1c, AX2c of the imaging devices so as to maintain their relative orientation with the connecting member 336 in response to changes in at least one of the position and orientation of the connecting member 336 accompanying movements of the first imaging device 331 and the second imaging device 332. As a result, the first imaging device 331 to which the first slider 331g is fixed and the second imaging device 332 to which the second slider 332g is fixed also rotate about the optical axes AX1c, AX2c so as to maintain their relative orientation with the connecting member 336. Therefore, even when the first imaging device 331 and the second imaging device 332 move in the circumferential direction, the relative orientation between the imaging element 331f of the first imaging device 331 and the imaging element 332f of the second imaging device 332 can be maintained. In other words, regardless of the positions of the first imaging device 331 and the second imaging device 332, the long side of the imaging element 331f of the first imaging device 331 and the long side of the imaging element 332f of the second imaging device 332 are kept parallel to each other. In this way, in this embodiment, the connecting member 336 can hold the first imaging device 331 and the second imaging device 332 in a state where the relative orientation between the first imaging device 331 and the second imaging device 332 is maintained at a predetermined orientation. Other configurations of the robot system 310 can be similar to other configurations of the robot systems of the above-mentioned embodiments.

[0162] According to this embodiment, as described above, at least one of the first imaging device 331 and the second imaging device 332 is movable so that the long sides of the imaging element 331f of the first imaging device 331 and the imaging element 332f of the second imaging device 332 are parallel to each other. Therefore, information about the distance to the object W can be suitably acquired based on the image acquired by the first imaging device 331 and the image acquired by the second imaging device 332 without rotating the images acquired by each imaging device.

[0163] Furthermore, according to this embodiment, a connecting member 336 is provided that connects the first imaging device 331 and the second imaging device 332. The connecting member 336 can hold the first imaging device 331 and the second imaging device 332 in a state where the relative orientation between the first imaging device 331 and the second imaging device 332 is maintained in a predetermined orientation. Therefore, even without providing a drive unit that rotates the first imaging device 331 and the second imaging device 332 about the optical axes AX1c and AX2c, for example, the connecting member 336 can easily maintain the relative orientation between the imaging element 331f of the first imaging device 331 and the imaging element 332f of the second imaging device 332 in an orientation where the long sides are parallel to each other. As a result, even if at least one of the first imaging device 331 and the second imaging device 332 is moved to change the baseline length, information about the distance to the object W can be easily and conveniently acquired based on the images acquired by the first imaging device 331 and the second imaging device 332.

[0164] In this embodiment, for example, only the imaging element 331f of the first imaging device 331 may be rotatable about the optical axis AX1c, or only the imaging element 332f of the second imaging device 332 may be rotatable about the optical axis AX2c. In this case, at least one of the imaging element 331f of the first imaging device 331 and the imaging element 332f of the second imaging device 332 may be movable so that the long sides of the imaging element 331f of the first imaging device 331 and the imaging element 332f of the second imaging device 332 are parallel to each other.

[0165] Fifth Embodiment Fig. 10 is a view of a part of the robot system 410 of this embodiment as seen from the tip side (+Z side) in the central axis direction. In Fig. 10, the finger portion 22b of the end effector 22 and the camera unit 60 are not shown. Fig. 11 is a diagram for explaining part of the procedure when the robot system 410 of this embodiment acquires information about the distance to the target object W. Note that, for configurations similar to those of the above-described embodiments, the description may be omitted by appropriately assigning the same reference numerals, etc.

[0166] In this embodiment, three or more imaging devices 430 are provided. For example, 24 imaging devices 430 are provided. The multiple imaging devices 430 are arranged side by side along the circumferential direction around the end effector 22. That is, in this embodiment, the three or more imaging devices 430 are arranged side by side on a predetermined circumference. In this embodiment, the predetermined circumference is a circumference centered on the central axis CL. The multiple imaging devices 430 are arranged, for example, at equal intervals around one circumference along the circumferential direction. That is, in this embodiment, the three or more imaging devices 430 are arranged at equal intervals on the predetermined circumference. The optical axes AX4 of the three or more imaging devices 430 are parallel to each other.

[0167] The long side of the imaging element 435 of each imaging device 430, when viewed in the central axis direction, is perpendicular to the radial direction passing through the optical axis AX4 of each imaging device 430. When the number of imaging devices 430 is N, the N imaging devices 430 are arranged with N-fold symmetry around the central axis CL. That is, in this embodiment, the 24 imaging devices 430 are arranged with 24-fold symmetry around the central axis CL. The imaging devices 430 are fixed to, for example, the end effector 22. More specifically, the imaging devices 430 are fixed to, for example, the outer peripheral surface of the base 22a. That is, the end effector 22 has, for example, a holder that holds three or more imaging devices 430 on the outer peripheral surface of the base 22a.

[0168] In this embodiment, the control unit 40 selects two images from three or more images acquired by three or more imaging devices 430, and acquires information about the distance to the object W based on information about the two selected images. The control unit 40 selects two images from, for example, 24 images acquired by the 24 imaging devices 430, based on information about occlusion of the object W. The information about occlusion of the object W includes, for example, information about whether the object W is captured in the image, information about the occlusion state of the object W, and information about the proportion of the object W that is captured in the image. The control unit 40 selects, for example, two images from the 24 acquired images that best capture the object W. The control unit 40 acquires information about the distance to the object W based on the two selected images.

[0169] Here, as an example, a case will be described in which the control unit 40 selects an image F1 acquired by the imaging device 431 and an image F2 acquired by the imaging device 432 among the multiple imaging devices 430. The imaging element 435a of the imaging device 431 and the imaging element 435b of the imaging device 432 are disposed in different orientations. In this case, the control unit 40 cuts out portions of the two acquired images F1 and F2 along a rectangular frame Fs, as shown in FIG. 11. The long side of the rectangular frame Fs is parallel to an imaginary line IL connecting the optical axis of the imaging device 431 and the optical axis of the imaging device 432. The control unit 40 acquires information about the distance to the target W based on the cut-out portions of the two images F1 and F2.

[0170] In this embodiment, the control unit 40 selects two images from three or more images acquired by three or more imaging devices 430, and controls at least one of the robot arm 21 and the end effector 22 connected to the robot arm 21 based on information on the selected two images. In this embodiment, the control unit 40 controls the imaging by the three or more imaging devices 430 so as to synchronize them. Other configurations of the robot system 410 can be similar to other configurations of the robot systems in the above-described embodiments.

[0171] According to this embodiment, the control unit 40 selects two images from three or more images acquired by three or more imaging devices 430, and acquires information about the distance to the object W based on the information of the two selected images. Therefore, it is possible to acquire information about the distance to the object W by selecting two images that are most suitable for acquiring information about the distance to the object W from three or more images acquired by three or more imaging devices 430. This makes it possible to suitably acquire information about the distance to the object W depending on the work content of the robot system 410, the environment in which the robot system 410 is placed, the position and posture of the robot arm 21, etc.

[0172] Furthermore, according to this embodiment, the control unit 40 selects two images from three or more images acquired by the multiple image capturing devices 430 based on information related to occlusion of the object W. Therefore, even if at least a portion of the object W cannot be captured by some of the image capturing devices 430 due to an obstruction or the like, it is possible to select two images in which the object W is preferably captured. This makes it easy to preferably acquire information related to the distance to the object W, even if a portion of the object W is obstructed by an obstruction or the like.

[0173] Furthermore, according to this embodiment, the control unit 40 selects two images from three or more images acquired by three or more imaging devices 430, and controls at least one of the robot arm 21 and the end effector 22 connected to the robot arm 21 based on the information of the two selected images. Therefore, by selecting two images containing optimal information for moving the robot arm 21 and the end effector 22 from the multiple images acquired by each imaging device 430, the robot arm 21 and the end effector 22 can be moved appropriately.

[0174] Furthermore, according to this embodiment, three or more imaging devices 430 are arranged side by side on a predetermined circumference. Therefore, a relatively large number of imaging devices 430 can be arranged side by side, for example, around the base 22a of the end effector 22 as in this embodiment. This makes it possible to reduce the protrusion of the imaging devices 430 from the robot 20 compared to when the same number of imaging devices 430 are arranged in a line. Therefore, even if a relatively large number of imaging devices 430 are attached to the robot 20, the robot 20 can be easily moved. Furthermore, by arranging multiple imaging devices 430 side by side along a circumference, it becomes easier to capture images of the object W from various angles using the multiple imaging devices 430. Therefore, it is easier to obtain information about the distance to the object W more preferably using the multiple imaging devices 430.

[0175] Furthermore, according to this embodiment, three or more imaging devices 430 are arranged at equal intervals on a predetermined circumference. Therefore, compared to when the imaging devices 430 are arranged at non-equidistant intervals, a deviation in the number of imaging devices 430 that can image the target W is less likely to occur depending on the position and orientation of the member (e.g., the end effector 22) to which the imaging devices 430 are attached. This makes it easier to obtain information about the distance to the target W using the imaging devices 430, regardless of the position and orientation of the member to which the imaging devices 430 are attached.

[0176] In this embodiment, the control unit 40 may select two images from three or more images acquired by the multiple imaging devices 430 based on at least one of the distance information related to the object W obtained in advance, information regarding occlusion of the object W, focal length information of the imaging device 430, and information regarding shape changes of images obtained by the three or more imaging devices 430.

[0177] The distance information related to the object W obtained in advance includes, for example, the distance from the robot 20 to the object W when the robot 20 and the object W are placed at their initial positions and in their initial postures, the distance to an obstruction placed near the object W, and the distances between multiple objects W when multiple objects W are placed at their initial positions and in their initial postures. The distance from the robot 20 to the object W includes, for example, the distance from a certain portion of the robot arm 21 to the object W, the distance from a certain portion of the end effector 22 to the object W, and the distance from a certain portion of the adapter 23 to the object W. By selecting two images based on the distance information related to the object W obtained in advance, the control unit 40 can select two images with suitable baseline lengths depending on the position of the object W, and can suitably acquire information about the distance to the object W.

[0178] The control unit 40 selects two images based on the focal length information of the imaging device 430, thereby selecting two images with an appropriate baseline length according to the focal length of the imaging device 430. Here, for example, if two images with a relatively large baseline length are selected when the zoom magnification of the imaging device 430 is relatively large, the overlapping portion of the two images (the range of images of overlapping characteristic portions) will be small. Therefore, for example, if the zoom magnification of the imaging device 430 is relatively large, the overlapping portion of the two images can be increased by selecting two images with a relatively small baseline length. This makes it possible to more appropriately acquire the distance to the object W based on the two images.

[0179] 12A to 12C are diagrams for explaining how the overlapping portion of two images changes depending on the baseline length and zoom magnification. FIG. 12A is a diagram showing an example of a case where two images F1a and F2a, which have a relatively large baseline length, are selected when the zoom magnification of the imaging device 430 is relatively small. FIG. 12B is a diagram showing an example of a case where two images F1b and F2b, which have a relatively large baseline length, are selected when the zoom magnification of the imaging device 430 is relatively large. FIG. 12C is a diagram showing an example of a case where two images F1c and F2c, which have a relatively small baseline length, are selected when the zoom magnification of the imaging device 430 is relatively large. In FIGS. 12A to 12C, a tree T and a car V are shown as the object W.

[0180] 12A, that is, when the zoom magnification of the imaging device 430 is relatively small and two images F1a and F2a with a relatively large baseline length are selected, the entire tree T and the entire car V are reflected in the two images F1a and F2a. In this case, the overlapping portion OPa of the two images F1a and F2a (the range of images of the overlapping characteristic portions) includes the entire tree T and the entire car V. Therefore, the distance to each object W can be obtained for the entire tree T and the entire car V.

[0181] On the other hand, as shown in Fig. 12B, if the zoom magnification is made larger than that shown in Fig. 12A and two images F1b and F2b are selected so that the baseline length is the same as that shown in Fig. 12A, the range reflected in each of the images F1b and F2b will be narrower than the range reflected in each of the images F1a and F2a in Fig. 12A, but the deviation in the position where the object W is reflected in each of the images F1b and F2b will remain relatively large because the baseline length remains relatively large. Therefore, as shown in Fig. 12B, part of the object W will be cut off in each of the images F1b and F2b, and the overlapping portion of the images F1b and F2b may become small.

[0182] In the example of FIG. 12B, most of the tree T, including its left side, is cut off in image F1b, and the right side of the car V is cut off in image F2b. In this case, the overlapping portion OPb includes only a part of the tree T and a part of the car V. Therefore, even if the entire tree T or the entire car V is reflected in one of the images, the distance to the object W cannot be obtained for the part not included in the overlapping portion OPb.

[0183] In contrast, even if the zoom magnification is the same as in the case shown in FIG. 12B, by selecting two images F1c and F2c in which the baseline length is smaller than that shown in FIG. 12B, as shown in FIG. 12C, the deviation in the position in which the object W is reflected in each of the images F1c and F2c can be reduced. This increases the range in which the object W is reflected in each of the images F1c and F2c, and increases the overlapping portion OPc between the images F1c and F2c. In the example of FIG. 12C, the entire tree T and the entire car V are reflected in each of the images F1c and F2c. In other words, the overlapping portion OPc includes the entire tree T and the entire car V, as in FIG. 12A. Therefore, the distance to each object W can be obtained for the entire tree T and the entire car V.

[0184] 12B and 12C, the overlapping portion OPc between the two images F1c and F2c can be increased by selecting two images F1c and F2c with a relatively short base line length, as shown in Fig. 12C. This allows the distance to the object W to be more appropriately acquired based on the two images F1c and F2c.

[0185] The information regarding the shape change of the images acquired by the three or more imaging devices 430 includes, for example, information regarding the difference in appearance of the object W in any two images among the three or more imaging devices 430. The difference in appearance of the object W in the images acquired by the different imaging devices 430 varies in magnitude depending on the shape of the object W and the shadows caused by the illumination irradiating the object W. The information regarding the shape change of the images acquired by the three or more imaging devices 430 includes, for example, the degree of similarity in the appearance of the object W in the two images, shape information of the object W, and information about the illumination irradiating the object W. The control unit 40 may, for example, perform matching for all combinations of two images selected from the multiple images acquired by each imaging device 430, and acquire the degree of similarity in the appearance of the object W in each image as a parameter. Furthermore, information regarding the shape change of the images may be input to the control unit 40 in advance.

[0186] Here, the longer the base line length between the two image capturing devices 430, the more likely it is that the appearance of the object W will differ significantly between the images captured by the two image capturing devices 430. If the appearance of the object W differs significantly to a certain extent, it becomes difficult to match the two images. Therefore, in such a case, by shortening the base line length, it is possible to prevent the appearance of the object W from differing significantly, making it easier to match the two images. Therefore, information regarding the distance to the object W can be suitably acquired based on the two images.

[0187] Therefore, the control unit 40 can select two images with an appropriate baseline length based on the difference in how the object W appears in each image by selecting two images based on information about changes in shape of the images obtained by three or more image capturing devices 430. That is, for example, if the object W is an object that appears significantly different in each of the multiple image capturing devices 430, by selecting two images with a short baseline length, information about the distance to the object W can be appropriately acquired based on the two selected images.

[0188] Furthermore, in this embodiment, the control unit 40 may select two of the three or more imaging devices 430 based on at least one of distance information related to the object W obtained in advance, information about occlusion of the object W, focal length information of the imaging devices 430, and information about changes in shape of images obtained by the three imaging devices 430. In this case, image capturing can be performed using only the two selected imaging devices 430, and information about the distance to the object W can be selected. Therefore, the load on the control unit 40 can be reduced compared to when two images are selected from images obtained by performing image capturing using all of the imaging devices 430.

[0189] Furthermore, in this embodiment, when information such as the shape of the object W and the difference in appearance of the luminance of the object W is known in advance, the control unit 40 may determine the image capture device 430 to select based on that information. The shape of the object W and the difference in appearance of the luminance of the object W may be acquired from distance information related to the object W that has been obtained in advance.

[0190] Furthermore, in this embodiment, the three or more imaging devices 430 may be arranged at unequal intervals on a predetermined circumference. Furthermore, the three or more imaging devices 430 may be arranged side by side on a predetermined axis, similar to the imaging devices 230a, 230b, and 230c of the third embodiment described above. In this case, the three or more imaging devices 430 may be arranged such that the long sides of the image sensors 435 in the three or more imaging devices 430 are parallel to each other. The three or more imaging devices 430 may be arranged in a matrix. Furthermore, the three or more imaging devices 430 may be TOF cameras (Time Of Flight Cameras).

[0191] Furthermore, in the above-described embodiment, the three or more imaging devices 430 are configured to be arranged around the end effector 22, but this is not limiting. The three or more imaging devices 430 may be arranged around any of the robot arm 21, the end effector 22 connected to the robot arm 21, and the adapter 23 for attaching the end effector 22 to the robot arm 21. Even when three or more imaging devices 430 are arranged around the robot arm 21 or the adapter 23, the same effects as those obtained when three or more imaging devices 430 are arranged around the end effector 22 described above can be obtained.

[0192] When three or more imaging devices 430 are arranged around the robot arm 21, the robot arm 21 may have a holding unit that holds the three or more imaging devices 430 that capture images of the object W. When three or more imaging devices 430 are arranged around the adapter 23, the adapter 23 may have a holding unit that holds the three or more imaging devices 430 that capture images of the object W. In these cases, the control unit 40 may also acquire information about the distance to the object W based on information about images of the object W acquired by two of the three or more imaging devices 430 held by the robot arm 21 or the adapter 23.

[0193] Furthermore, the control unit 40 may perform the operation of acquiring information about the distance to the object W based on images acquired by two of the three or more image capturing devices 430 multiple times using different combinations of two image capturing devices 430. In this case, the control unit 40 can acquire information about the distance to the object W with greater accuracy by comparing the information about the distance to the object W acquired by the multiple operations. In particular, when at least a portion of the object W is obstructed by an obstruction as viewed from at least some of the image capturing devices 430, the control unit 40 can acquire information about the distance to the object W while minimizing the influence of obstruction by the obstruction by using multiple pieces of information acquired using multiple combinations of two image capturing devices 430.

[0194] Furthermore, after capturing images of the object W with all of the imaging devices 430, the control unit 40 may move the end effector 22 by, for example, moving the robot arm 21, and then capture images of the object W again from another location with all of the imaging devices 430. This allows the control unit 40 to acquire images of the object W captured from multiple angles. In this case, if the number of imaging devices 430 is relatively large, many images of the object W captured from different angles can be acquired even if the number of times the end effector 22 is moved to capture images is small. Note that each acquired image may be associated with information such as the position and orientation of the end effector 22 when the image was captured and the position and orientation of the imaging device 430 that captured the image.

[0195] As described above, when the object W is imaged multiple times from different positions, the control unit 40 may use the acquired images to control the robot system 410 by visual servoing. In this case, the control unit 40 controls, for example, the imaging device 430 to move to a position where the image of the target object W can be captured by the imaging device 430. Here, if the image of the object W captured by the imaging device 430 at the initial position is significantly different from the image of the target object W, it may be difficult to establish a correspondence between the images, and it may be difficult to move the imaging device 430 to a position where the image of the target object W can be captured.

[0196] In contrast, by capturing images of the object W multiple times from different positions and acquiring images of the object W captured from different angles, the images captured by the imaging device 430 can be easily made closer to the target image by using the images captured from the different angles as intermediate images. In other words, it is easy to suitably move the imaging device 430 to a position where it can capture an image of the target object W. Furthermore, if the intermediate images are associated with information about the distance from the object W when the intermediate images were captured, the control unit 40 may arrange the intermediate images in the order in which they were captured at positions farthest from the object W, and move the image captured by the imaging device 430 closer to the target image by going through the multiple intermediate images in that order.

[0197] Furthermore, in this embodiment, the robot system 410 may include, for example, a plurality of general-purpose cameras that can simply capture images of the object W, in addition to the plurality of imaging devices 430 described above. In this case, the control unit 40 may construct a 3D model of the object W using a plurality of images captured by the plurality of general-purpose cameras. By using the 3D model, for example, the accuracy of acquiring information regarding the distance to the object W using the plurality of imaging devices 430 can be further improved. In this case, as an example, 12 imaging devices 430 may be provided at equal intervals along the circumferential direction, and two general-purpose cameras may be provided between adjacent imaging devices 430 in the circumferential direction. In this case, for example, 24 general-purpose cameras may be provided. For example, a relatively inexpensive camera such as that used in a smartphone may be used as the general-purpose camera.

[0198] Furthermore, in this embodiment, the control unit 40 may perform simultaneous localization and mapping (SLMA). That is, the control unit 40 may simultaneously estimate the self-position of the robot system 410 and create a map of the environment in which the robot system 410 is placed. In this case, if a relatively large number of image capturing devices 430 are provided, a relatively large amount of 3D point cloud data about the environment can be easily acquired by the multiple image capturing devices 430. Therefore, it becomes easy to create a map of the environment in which the robot system 410 is placed.

[0199] Sixth Embodiment 13 is a perspective view showing a robot system 510 of this embodiment. Note that the same components as those in the above-described embodiments may be appropriately denoted by the same reference numerals and the description thereof may be omitted.

[0200] 13, in a robot system 510 of this embodiment, an imaging device 530 includes a first imaging device 531 attached to a robot arm 521 and a second imaging device 32 attached to an end effector 22. The first imaging device 531 is disposed, for example, around a fifth arm unit 524e. The first imaging device 531 is connected to a guide rail unit 521a provided on the fifth arm unit 524e, for example, in a structure similar to that in the first embodiment in which the first imaging device 31 is connected to the guide rail unit 22e.

[0201] The guide rail portion 521a is annular and surrounds the fifth arm portion 524e. The first imaging device 531 is movable in a predetermined circumferential direction around the robot arm 521 along the guide rail portion 521a. The robot arm 521 has the same configuration as the robot arm 21 of the first embodiment, except that the guide rail portion 521a is provided. The other configurations of the robot system 510 of this embodiment can be the same as the other configurations of the robot systems of the above-mentioned embodiments.

[0202] According to this embodiment, the first imaging device 531 and the second imaging device 32 are attached to different members and are movable relative to the members to which they are attached. Therefore, compared to when the two imaging devices are attached to the same member, it is possible to prevent the movement of the first imaging device 531 and the movement of the second imaging device 32 from being hindered by the other imaging device. This allows the first imaging device 531 and the second imaging device 32 to be moved appropriately relative to the members to which they are attached.

[0203] Seventh Embodiment 14 is a perspective view showing a robot system 610 of this embodiment. Note that the same components as those in the above-described embodiments may be denoted by the same reference numerals as appropriate, and the description thereof may be omitted.

[0204] As shown in FIG. 14 , the robot system 610 of this embodiment includes a projection device 670 that projects light SL. In this embodiment, the projection device 670 is disposed around the robot arm 621. For example, the projection device 670 is disposed around the fifth arm unit 624e. The projection device 670 is connected to a guide rail unit 621a of the fifth arm unit 624e. The guide rail unit 621a has a configuration similar to that of the guide rail unit 521a of the sixth embodiment, except that the projection device 670 is connected to the guide rail unit 621a instead of the imaging device. In this embodiment, the projection device 670 is movable in a predetermined circumferential direction around the robot arm 621 along the guide rail unit 621a. The projection device 670 projects light SL in a grid pattern onto the target W, for example. The structure of the projection device 670 is not particularly limited as long as it is capable of projecting light SL.

[0205] In this embodiment, the first imaging device 31 and the second imaging device 32 perform imaging and acquire images while light SL is projected by the projection device 670. In this embodiment, the control unit 40 moves the projection device 670, the first imaging device 31, and the second imaging device 32 so that the light SL projected by the projection device 670 onto the object W can be captured by the first imaging device 31 and the second imaging device 32. The robot arm 621 has the same configuration as the robot arm 21 of the first embodiment, except that a guide rail portion 621a is provided. The other configurations of the robot system 610 can be the same as the other configurations of the robot systems of the above-mentioned embodiments.

[0206] According to this embodiment, the first imaging device 31 and the second imaging device 32 perform imaging and acquire images while light SL is projected by the projection device 670. Therefore, the first imaging device 31 and the second imaging device 32 can capture images of the object W onto which light SL is projected. This allows the first imaging device 31 and the second imaging device 32 to more suitably acquire images of the object W. Furthermore, by making the light SL projected from the projection device 670 light of a grid-like pattern or the like, it is also possible to measure the three-dimensional shape of the object W, etc., based on the pattern reflected in the images acquired by the first imaging device 31 and the second imaging device 32.

[0207] In this embodiment, only the first imaging device 31 of the first imaging device 31 and the second imaging device 32 may perform imaging and acquire an image with light SL projected by the projection device 670. Alternatively, only the second imaging device 32 of the first imaging device 31 and the second imaging device 32 may perform imaging and acquire an image with light SL projected by the projection device 670. In this embodiment, the robot system 610 may include only the first imaging device 31 of the first imaging device 31 and the second imaging device 32. In this case, the first imaging device 31 may perform imaging and acquire an image with light SL projected by the projection device 670, and may acquire information about the distance to the object W by a method similar to the method of acquiring information about the distance to the object W by one first imaging device 31 described in the second embodiment.

[0208] Furthermore, in this embodiment, the projection device 670 may be disposed around the end effector 22 or around the adapter 23. Whether the projection device 670 is disposed around any of the robot arm 621, the end effector 22, or the adapter 23 is determined appropriately depending on the positions at which the first imaging device 31 and the second imaging device 32 are attached, the work content of the robot system 610, etc. By disposing the projection device 670 around these parts, the light SL projected from the projection device 670 is less likely to be blocked by parts of the robot system 610, and the light SL projected from the projection device 670 can be more easily projected onto the target W in an appropriate manner.

[0209] Furthermore, the projection device 670 may be fixed so as not to be able to move relative to the member to which it is attached. Furthermore, a plurality of projection devices 670 may be provided. In this case, the plurality of projection devices 670 may be attached to different members.

[0210] Eighth Embodiment 15 is a perspective view showing a robot system 710 of this embodiment. Note that the same components as those in the above-described embodiments may be appropriately denoted by the same reference numerals and descriptions thereof may be omitted.

[0211] 15, the adapter 723 in this embodiment has a circular guide rail portion 723h along the circumferential direction. The adapter 723 has the same configuration as the adapter 23 in the first embodiment, except for having the guide rail portion 723h. The end effector 722 in this embodiment has the same configuration as the end effector 22 in the first embodiment, except for not having the guide rail portion 22e and not having the first imaging device 731 and the second imaging device 732 attached thereto.

[0212] In this embodiment, the first imaging device 731 and the second imaging device 732 are attached to the adapter 723. The first imaging device 731 and the second imaging device 732 are arranged around the adapter 723. The first imaging device 731 is connected to the guide rail portion 723h via a slider 731d. The second imaging device 732 is connected to the guide rail portion 723h via a slider 732d. In other words, in this embodiment, the adapter 723 has the guide rail portion 723h as a first holding portion that holds the first imaging device 731 and a second holding portion that holds the second imaging device 732.

[0213] In this embodiment, the sliders 731d and 732d extend radially outward from the guide rail portion 723h and protrude radially outward beyond the end effector 722. As a result, the first imaging device 731 and the second imaging device 732 provided at the radially outer ends of the sliders 731d and 732d, respectively, are positioned radially outward beyond the end effector 722.

[0214] In this embodiment, at least one of the first imaging device 731 and the second imaging device 732 is movable relative to the adapter 723. At least one of the first imaging device 731 and the second imaging device 732 is movable in a predetermined circumferential direction around the adapter 723. In this embodiment, both the first imaging device 731 and the second imaging device 732 are movable relative to the adapter 723 and are movable in a predetermined circumferential direction around the adapter 723. That is, in this embodiment, the first imaging device 731 and the second imaging device 732 are movably held by a guide rail portion 723h serving as a first holding portion and a second holding portion. The relative positions of the first imaging device 731 and the second imaging device 732 are variable. Other configurations of the robot system 710 can be similar to other configurations of the robot systems of the above-described embodiments.

[0215] According to this embodiment, the first imaging device 731 and the second imaging device 732 attached to the adapter 723 can achieve the same effects as those achieved by the first imaging device 31 and the second imaging device 32 attached to the end effector 22 in the first embodiment.

[0216] In this embodiment, the adapter 723 may have a first holding portion that immobilizes the first imaging device 731. In this case, the adapter 723 may have a guide rail portion 723h as a second holding portion that movably holds the second imaging device 732. The adapter 723 may also have a second holding portion that immobilizes the second imaging device 732. In this case, the adapter 723 may have a guide rail portion 723h as a first holding portion that movably holds the first imaging device 731. In this way, in this embodiment, one of the first imaging device 731 and the second imaging device 732 may be movable in a predetermined circumferential direction around the adapter 723, and the other may be fixed to a predetermined portion of the adapter 723.

[0217] Ninth Embodiment 16 is a perspective view showing a robot system 810 of this embodiment. Note that the same components as those in the above-described embodiments may be appropriately denoted by the same reference numerals and descriptions thereof may be omitted.

[0218] 16, in this embodiment, the first imaging device 831 and the second imaging device 832 are attached to the robot arm 521. The first imaging device 831 and the second imaging device 832 are arranged around the robot arm 521. More specifically, the first imaging device 831 and the second imaging device 832 are arranged around the fifth arm portion 524e. The first imaging device 831 and the second imaging device 832 are connected to the guide rail portion 521a by, for example, a structure similar to the structure in which the first imaging device 31 and the second imaging device 32 are connected in the first embodiment. That is, in this embodiment, the robot arm 521 has the guide rail portion 521a as a first holding portion that holds the first imaging device 831 and a second holding portion that holds the second imaging device 832.

[0219] In this embodiment, at least one of the first imaging device 831 and the second imaging device 832 is movable relative to the robot arm 521. At least one of the first imaging device 831 and the second imaging device 832 is movable in a predetermined circumferential direction around the robot arm 521. In this embodiment, both the first imaging device 831 and the second imaging device 832 are movable relative to the robot arm 521 and are movable in a predetermined circumferential direction around the robot arm 521. In other words, the first imaging device 831 and the second imaging device 832 are movably held by the guide rail portion 521a, which serves as the first holding portion and the second holding portion. The relative positions of the first imaging device 831 and the second imaging device 832 are variable. Other configurations of the robot system 810 can be similar to other configurations of the robot systems of the above-described embodiments.

[0220] According to this embodiment, the first imaging device 831 and the second imaging device 832 attached to the robot arm 521 can obtain the same effects as those obtained by the first imaging device 31 and the second imaging device 32 attached to the end effector 22 in the first embodiment.

[0221] In this embodiment, the robot arm 521 may have a first holding unit that immobilizes the first imaging device 831. In this case, the robot arm 521 may have a guide rail unit 521a as a second holding unit that movably holds the second imaging device 832. The robot arm 521 may also have a second holding unit that immobilizes the second imaging device 832. In this case, the robot arm 521 may have a guide rail unit 521a as a first holding unit that movably holds the first imaging device 831. In this way, in this embodiment, one of the first imaging device 831 and the second imaging device 832 may be movable in a predetermined circumferential direction around the robot arm 521, and the other may be fixed to a predetermined portion of the robot arm 521.

[0222] The above describes an embodiment of the present invention in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and appropriate modifications can be made within the scope that does not deviate from the spirit of the present invention.

[0223] When the imaging device is movable relative to the member to which it is attached, any method may be used to calibrate the position of the imaging device. For example, the position of the imaging device may be calibrated by placing a panel or the like with a specific mark at a specific distance from the imaging device and capturing an image of the mark. Furthermore, when the imaging device is movable relative to the member to which it is attached, the imaging device may be movable only between a plurality of predetermined positions. In this case, the positions to which the imaging device can be moved may be structurally determined. In this case, the position of the imaging device can also be determined structurally.

[0224] When an imaging device is movable relative to a member to which it is attached, the imaging device may move in any manner relative to the member. The imaging device may move linearly relative to the member, or may move in a curved line other than a circular arc. When multiple imaging devices are movable relative to the member to which they are attached, each imaging device may move along a different movement path. The structure of the drive unit that moves the imaging device relative to the member to which it is attached is not particularly limited.

[0225] When multiple imaging devices are provided, the multiple imaging devices may include imaging devices of different types. For example, the multiple imaging devices may include an infrared camera and an RGB camera. In this case, the infrared camera and the RGB camera may each capture an image of the object from the same position to obtain an image.

[0226] The robot system may have an external sensor capable of detecting at least one of the position, posture, shape, etc. of the robot. The external sensor may be disposed on the ceiling of the location where the robot is disposed, or on the floor of the location where the robot is disposed. The external sensor may be, for example, a sensor capable of detecting the position and posture of the robot arm, a sensor capable of detecting the position and posture of the end effector, or a sensor capable of detecting the position and posture of the adapter.

[0227] The external sensor may be, for example, a laser tracker. In this case, the external sensor may detect the position information of each part based on distance measurement results using a time-of-flight (TOF) method, which is based on the difference between the timing of emitting light and the timing of receiving reflected light. Alternatively, the external sensor may detect the position information of each part by determining the geometric positional relationship using triangulation based on the measurement results of the reflection positions of reflected light generated by emitting light along multiple optical paths. In this case, to improve the measurement accuracy of the reflection positions of reflected light, the external sensor may include a variable-focus lens (e.g., a zoom lens) in the optical system of the light-receiving unit that receives the reflected light. Alternatively, the external sensor may use a distance measurement method that uses an optical comb generated by extremely short-duration optical pulses to detect the position of each part.

[0228] The external sensor may be capable of detecting the position and orientation of the imaging device. In this case, if the imaging device is movable relative to the member to which it is attached, the control unit may move the imaging device based on information about the imaging device obtained by the external sensor. In this case, the imaging device may be provided with a marker that can be detected by the external sensor.

[0229] The external sensor may be an imaging device with a variable baseline length. In this case, the control unit may change the baseline length of the external sensor, for example, depending on the distance between the external sensor and an object on which work is performed by the end effector. As an example, the control unit may reduce the baseline length of the external sensor when the object grasped by the end effector is brought closer to the external sensor by moving the robot arm. The baseline length of the external sensor may be changed, for example, using the same method as the baseline length changing method appropriately described in each of the above-mentioned embodiments.

[0230] The applications of the robot system described above are not particularly limited, and the configurations and methods described above can be combined as appropriate within the scope of not contradicting each other. [Explanation of symbols]

[0231] 10,110,210,310,410,510,610,710,810...Robot systems 20...Robot, 21,521,621...Robot arm 22,122,722...End effector 22e, 521a, 723h... Guide rail section (first holding section, second holding section) 23,723...Adapter 24...Arm part (movable part) 30, 230, 230a, 230b, 230c, 430, 431, 432, 530...imaging device 31,331,531,731,831...First imaging device 31f, 32f, 132f, 235a, 235b, 235c, 331f, 332f, 435...imaging element 32,132,332,732,832...Second imaging device 34,134...Position acquisition part 40...Control unit 44…Distance information acquisition unit 50...Display section 122f...Hole part (second holding part) 336...Connecting member 670…projection device, AX1, AX1a, AX1b, AX1c, AX2, AX2a, AX2c, AX3a, AX3b, AX3c, AX4…optical axis L, L1, L2, L3, L4, La…baseline length P1…1st position P2…2nd position SL…Light VA…axis W...object

Claims

1. A control system used to control a robot, comprising: an imaging device attached to the robot; a control device for controlling the robot; Equipped with The control device generating a first control signal for controlling the robot to move the imaging device to a second position different from the first position based on a first imaging result acquired by the imaging device disposed at a first position; generating a second control signal different from the first control signal for the robot to perform a process on the target object based on a second image capture result of the target object acquired by the imaging device disposed at the second position; Control system.

2. The control device determines the second position based on the first imaging result. The control system of claim 1 .

3. The control device acquiring information about the distance to the target object based on the second imaging result; generating the second control signal based on information about the distance; 3. The control system according to claim 1 or 2.

4. The control device generating the first control signal when the image data of the target object differs from the first imaging result; A control system according to any one of claims 1 to 3.

5. The control device generating the first control signal when the first imaging result does not capture the target object; A control system according to any one of claims 1 to 4.

6. The second position is a position where the second imaging result, which is an imaging result that approaches target image data compared to the first imaging result, can be captured. A control system according to any one of claims 1 to 5.

7. The second position is a position closer to the target object compared to the first position. A control system according to any one of claims 1 to 5.

8. The angle between the imaging device and the target object at the second position is different from the angle between the imaging device and the target object at the first position. A control system according to any one of claims 1 to 7.

9. The control device generating a fourth control signal for controlling the robot to move the imaging device to a third position different from the second position; generating the second control signal based on the second imaging result and a third imaging result acquired by the imaging device disposed at the third position; A control system according to any one of claims 1 to 8.

10. The imaging device includes a first imaging device and a second imaging device different from the first imaging device. A control system according to any one of claims 1 to 9.

11. The first imaging result is obtained by the first imaging device, The second imaging result is acquired by the first imaging device and the second imaging device. The control system of claim 10.

12. The first imaging result is obtained by the first imaging device and the second imaging device; The second imaging result is acquired by the first imaging device and the second imaging device. The control system of claim 10.

13. At least one of the first imaging device and the second imaging device is movable relative to the robot; the control device generates a control signal for moving at least one of the first imaging device and the second imaging device. A control system according to any one of claims 10 to 12.

14. The process includes the robot approaching the target object. A control system according to any one of claims 1 to 13.

15. The process includes the robot grasping the target object. A control system according to any one of claims 1 to 14.

16. The robot comprises a robot arm, the imaging device is disposed on the robot arm; A control system according to any one of claims 1 to 15.

17. The imaging device includes a first imaging device, a second imaging device, and a third imaging device; the control device generates the second control signal based on the second imaging results acquired using at least two of the first imaging device, the second imaging device, and the third imaging device; A control system according to any one of claims 1 to 15.

18. The imaging device is a first imaging device and further includes a fourth imaging device different from the first imaging device; the control device selects an imaging device to be used to acquire the second imaging result from the first imaging device or the fourth imaging device according to a distance to the target object; A control system according to any one of claims 1 to 17.

19. A control method for a control system used to control a robot, comprising: The control system includes: an imaging device attached to the robot; a control device for controlling the robot; Equipped with The control method includes: generating a first control signal for controlling the robot to move the imaging device to a second position different from the first position based on a first imaging result acquired by the imaging device disposed at a first position; generating a second control signal different from the first control signal for the robot to perform a process on the target object based on a second image capture result of the target object acquired by the imaging device disposed at the second position; Control method.

Images