Control device, control system, robot system, control method, and computer program

Abstract

This control device generates control information for controlling at least one of: a robot that comprises a holding device for holding a target object and moves the holding device; and the holding device. The control device generates the control information before the target object held by the holding device is passed to a change device capable of changing at least one of the position and the posture of the target object, on the basis of an imaging result by the imaging system of at least one of: at least a part of the target object; and at least a part of the change device.

Description

Control device, control system, robot system, control method, and computer program

[0001] The present invention relates to the technical fields of a control device, a control system, a robot system, a control method, and a computer program that can generate a control signal for controlling a robot, for example.

[0002] An example of a control device for controlling a robot provided with a holding device capable of holding an object is described in Patent Document 1. Such a control device is required to appropriately control the robot.

[0003] US Patent Application Publication No. 2013 / 0230235

[0004] According to a first aspect, there is provided a control device that is provided with a holding device for holding a target object and that generates control information for controlling at least one of the holding device and a robot that moves the holding device, the control device comprising: an arithmetic device that generates the control information; and a communication device that outputs the control information generated by the arithmetic device, wherein the arithmetic device generates the control information based on an imaging result of at least one of at least a portion of the target object and at least a portion of the modification device by an imaging system before the target object held by the holding device is handed over to a modification device that can change at least one of the position and posture of the target object.

[0005] According to a second aspect, there is provided a control device that is provided with a holding device for holding a target object and that generates control information for controlling at least one of the holding device and a robot that moves the holding device, the control device comprising: an arithmetic device that generates the control information; and a communication device that outputs the control information generated by the arithmetic device, wherein the arithmetic device generates the control information based on an imaging result obtained by an imaging system of at least one of at least a portion of the target object whose position and / or posture has been changed by a modification device that changes at least one of the position and / or posture of the target object and at least a portion of the modification device.

[0006] According to a third aspect, there is provided a control system including the control device provided by the first or second aspect and an imaging system.

[0007] According to a fourth aspect, there is provided a robot system including the control device provided by the first or second aspect, an imaging system, and the robot.

[0008] According to a fifth aspect, there is provided a control method in which a holding device is provided to hold a target object, and which generates control information for controlling at least one of the holding device and a robot that moves the holding device, the control method including generating the control information based on an imaging result of at least one of at least a portion of the target object and at least a portion of the modification device by an imaging system before the target object held by the holding device is handed over to a modification device that can change at least one of the position and posture of the target object.

[0009] According to a sixth aspect, there is provided a control method in which a holding device is provided for holding a target object, and which generates control information for controlling at least one of the holding device and a robot that moves the holding device, the control method including: after the holding device holds the target object, transferring the target object held by the holding device to a change device that is capable of changing at least one of the positions or postures of the target object; and, after the change device changes at least one of the positions or postures of the target object transferred from the holding device, generating the control information based on an imaging result obtained by an imaging system imaging at least a portion of the change device and at least one of at least a portion of the target object held by the change device.

[0010] According to a seventh aspect, there is provided a computer program for causing a computer to execute the control method provided by the fifth or sixth aspect.

[0011] According to an eighth aspect, there is provided a robot system including a holding device for holding a target object and an imaging system, a robot for moving the holding device and the imaging system, and a change device capable of changing at least one of the position or posture of the target object received from the holding device.

[0012] According to a ninth aspect, there is provided a control device that is provided with a holding device for holding a target object, and that generates control information for controlling at least one of the holding device and a robot that moves the holding device, the control device comprising an arithmetic device that generates the control information, and a communication device that outputs the control information generated by the arithmetic device, and the arithmetic device performs an operation to hold the target object by the holding device, and then generates the control information based on the imaging results of an imaging system of an area where the target object is expected to be present when the target object is held by the holding device.

[0013] According to a tenth aspect, there is provided a control device that is provided with a holding device for holding a target object, and that generates control information for controlling at least one of the holding device and a robot that moves the holding device, the control device comprising an arithmetic device that generates the control information, and a communication device that outputs the control information generated by the arithmetic device, and the arithmetic device performs an operation to hold the target object by the holding device, and then determines whether the holding device is holding the target object based on the imaging results by an imaging system.

[0014] FIG. 1 is a block diagram showing the configuration of a robot system according to this embodiment. FIG. 2 is a side view showing the appearance of a robot according to this embodiment. FIG. 3 is a side view conceptually showing the appearance of a position and orientation changing device. FIG. 4 is a block diagram showing the configuration of a control device according to this embodiment. FIG. 5 is a block diagram showing the configuration of a robot control device according to this embodiment. FIGS. 6A and 6B are side views showing the positional relationship between a robot and a workpiece at a certain point in time during a holding process for holding a workpiece placed on a mounting device, and FIGS. 6C to 6E are side views showing the positional relationship between a robot and a workpiece at a certain point in time during a release process for placing a workpiece on the mounting device. FIG. 7 is a flowchart showing the overall flow of robot control processing. FIG. 8 is a flowchart showing the flow of holding control processing. FIG. 9 is a flowchart showing the flow of workpiece position and orientation changing processing. FIGS. 10A and 10B are cross-sectional views showing a workpiece held by an end effector. FIGS. 11A to 11E are side views showing a position and orientation changing device for performing workpiece position and orientation changing processing. FIG. 12 is a flowchart showing the flow of release control processing. Fig. 13 is a flowchart showing the flow of workpiece position and orientation change processing, including workpiece holding determination processing. Fig. 14 is a cross-sectional view showing a workpiece holding area. Fig. 15 is a flowchart showing the flow of workpiece position and orientation change processing, including retry processing. Figs. 16A to 16D are cross-sectional views showing a workpiece held by an end effector together with the end effector. Figs. 17A to 17D are cross-sectional views showing a workpiece held by an end effector together with the end effector. Fig. 18 is a flowchart showing the flow of workpiece position and orientation change processing, including device position and orientation calculation processing.Fig. 19A is a cross-sectional view showing a first target position relationship and a first target posture relationship between a workpiece and a holding member, Fig. 19B is a cross-sectional view conceptually showing an end effector moving toward a holding member when the holding member is located at a known position and / or has a known posture, and Fig. 19C is a cross-sectional view conceptually showing an end effector moving toward a holding member when the holding member is located at a position different from the known position and / or has a posture different from the known posture. Fig. 20 is a flowchart showing the flow of workpiece position and posture change processing including workpiece position and posture calculation processing. FIG. 21A is a cross-sectional view showing a second target positional relationship and a second target posture relationship between a workpiece held by a holding member and an end effector. FIG. 21B is a cross-sectional view conceptually showing an end effector moving toward a workpiece held by a holding member when the workpiece held by the holding member is located at a known position and / or has a known posture. FIG. 21C is a cross-sectional view conceptually showing an end effector moving toward a workpiece held by a holding member when the workpiece held by the holding member is located at a position different from the known position and / or has a posture different from the known posture. FIG. 22 is a flowchart showing the flow of a release control process including a jig position and posture calculation process. FIG. 23A is a cross-sectional view showing a jig located at a position known to the control device and / or having a posture known to the control device. FIG. 23B is a cross-sectional view showing a jig located at a position different from the known position and / or having a posture different from the known posture. FIG. 24 is a block diagram showing the configuration of a robot system equipped with a measurement system. FIG. 25 is a side view showing the appearance of a modified robot.

[0015] Next, embodiments of a control device, a control system, a robot system, a control method, and a computer program will be described with reference to the drawings. Hereinafter, the embodiments of a control device, a control system, a robot system, a control method, and a computer program will be described using a robot system SYS.

[0016] (1) Configuration of the Robot System SYS First, the configuration of the robot system SYS will be described.

[0017] (1-1) Overall Configuration of Robot System SYS First, the overall configuration of the robot system SYS will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the overall configuration of the robot system SYS.

[0018] As shown in FIG. 1, the robot system SYS includes a robot 1, an imaging system 2, a control device 3, an end effector 4, and a position / posture changing device 5.

[0019] The robot 1 is a device capable of performing a predetermined process on a target object OBJ. An example of the robot 1 is shown in FIG. 2. FIG. 2 is a side view showing the appearance of the robot 1. As shown in FIG. 2, the robot 1 includes, for example, a base 11, a robot arm 12, and a robot control device 13.

[0020] The base 11 is a component that forms the base of the robot 1. The base 11 is placed on a support surface S such as a floor surface. The base 11 may be fixed to the support surface S. Alternatively, the base 11 may be movable relative to the support surface S. Note that FIG. 2 shows an example in which the base 11 is fixed to the support surface S.

[0021] The robot arm 12 is attached to the base 11. The robot arm 12 is a device in which a plurality of links 121 are connected via joints 122. An actuator is built into the joint 122. The link 121 may be rotatable around an axis defined by the joint 122 by the actuator built into the joint 122. At least one link 121 may be extendable and contractible along the direction in which the link 121 extends. A device including the device in which a plurality of links 121 are connected via joints 122 and the base 11 may be referred to as the robot arm 12.

[0022] An end effector 4 is attached to (or provided on) the robot arm 12. That is, the end effector 4 is attached to the robot 1. In the example shown in FIG. 2 , the end effector 4 is attached to the tip of the robot arm 12. The end effector 4 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the end effector 4. That is, the robot 1 moves the end effector 4.

[0023] The end effector 4 is a device that performs a predetermined process (in other words, a predetermined operation) on the target object OBJ. The end effector 4 that performs a predetermined process on the target object OBJ may also be called a processing device.

[0024] For example, the end effector 4 may perform a holding process to hold the target object OBJ as an example of the predetermined process. In this case, the end effector 4 may be considered to be performing the holding process on the target object OBJ that the end effector 4 is to hold. An end effector 4 capable of performing the holding process may be referred to as a holding device. Note that holding the target object OBJ may be considered to be equivalent to picking up the target object OBJ. The holding process of holding the target object OBJ may be considered to be equivalent to a pick-up process of picking up the target object OBJ.

[0025] For example, holding the target object OBJ may include grasping the target object OBJ. For example, holding the target object OBJ may include gripping the target object OBJ using a hand gripper, which is an example of the end effector 4. Holding the target object OBJ may include attracting the target object OBJ. For example, holding the target object OBJ may include attracting (vacuum suction) the target object OBJ using a vacuum gripper, which is an example of the end effector 4. For example, holding the target object OBJ may include attracting (vacuum suction) the target object OBJ using a magnetic gripper (e.g., a magnetically attractive gripper), which is an example of the end effector 4.

[0026] For example, as an example of the predetermined process, the end effector 4 may perform a release process (in other words, a release operation) to release (i.e., let go of) the target object OBJ that it is holding. In this case, the end effector 4 may be considered to be performing the release process on the target object OBJ that it is holding. An end effector 4 that is capable of performing the release process may be called a release device. The release process may also be called a placement process.

[0027] A hand gripper is an example of an end effector 4 capable of holding and releasing an object OBJ. A hand gripper is an end effector 4 that can hold (e.g., grasp) a target object OBJ by physically pinching the target object OBJ using multiple (e.g., two, three, or four) finger or claw members. Another example of an end effector 4 capable of holding and releasing an object OBJ is a vacuum gripper (e.g., a vacuum suction gripper). A vacuum gripper is an end effector 4 that can hold (e.g., attract) a target object OBJ by vacuum suction. Another example of an end effector 4 capable of holding and releasing an object OBJ is a magnetic gripper (e.g., a magnetic suction gripper). FIG. 2 illustrates an example in which the end effector 4 is a hand gripper. However, the end effector 4 capable of holding and releasing an object OBJ is not limited to the above example, and may be any other existing end effector capable of holding and releasing an object OBJ. For example, the end effector 4 may be a Bernoulli chuck capable of holding the target object OBJ in a non-contact manner. Note that the end effector 4 capable of performing at least one of the holding process and the release process is not limited to the end effector described above, and may be another existing end effector.

[0028] The robot 1 may perform, as an example of a predetermined process, a placement process (in other words, a placement operation) for placing a target object OBJ at a desired position using the end effector 4 capable of performing a holding process and a release process. For example, the robot 1 may use the end effector 4 to hold a first target object OBJ that is a first example of the target object OBJ, and then perform a placement process for placing the first target object OBJ held by the end effector 4 at a desired position of a second target object OBJ that is a second example of the target object OBJ and different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process for the second target object OBJ on which the end effector 4 is to place the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process for the first target object OBJ that the end effector 4 is to release.

[0029] An example of the second target object OBJ is a jig that supports (and, in some cases, holds) the first target object OBJ. In this case, the robot 1 may perform a placement process to place (in other words, set) the first target object OBJ held by the end effector 4 on the jig. An example of the second target object OBJ is a processing device (e.g., a stage of the processing device) that processes the first target object OBJ. In this case, the robot 1 may perform a placement process to place (in other words, set) the first target object OBJ held by the end effector 4 on the processing device (e.g., a stage of the processing device). An example of the second target object OBJ is a measurement device (e.g., a stage of the measurement device) that measures the first target object OBJ. In this case, the robot 1 may perform a placement process to place (in other words, set) the first target object OBJ held by the end effector 4 on the measurement device (e.g., a stage of the measurement device). An example of the second target object OBJ is the position and orientation changing device 5. In this case, the robot 1 may perform a placement process to place (in other words, transfer) the first target object OBJ held by the end effector 4 to the position and orientation changing device 5.

[0030] The robot 1 may use the end effector 4 capable of holding and releasing operations to perform a fitting process (i.e., a fitting operation) for fitting a first target object OBJ into a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (i.e., a placement operation). The fitting process may include, for example, a process for fitting (i.e., inserting) the first target object OBJ (e.g., a convex portion of the first target object OBJ) into a concave portion (e.g., a hole) formed in the second target object OBJ. The fitting process may include, for example, a process for fitting the first target object OBJ (e.g., a concave portion of the first target object OBJ) into a convex portion (e.g., a rod) formed in the second target object OBJ. In this case, the second target object OBJ may be an object (workpiece) into which the first target object OBJ is to be fitted.

[0031] The robot 1 may use the end effector 4 capable of performing a holding process and a release process to perform a placing process (in other words, a placing operation) for placing a first target object OBJ on a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placing operation). In this case, the second target object OBJ may be an object (workpiece) onto which the first target object OBJ is to be placed. The robot 1 may use the end effector 4 capable of performing a holding process and a release process to perform a pasting process (in other words, a pasting operation) for pasting the first target object OBJ on a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placing operation). In this case, the second target object OBJ may be an object (workpiece) onto which the first target object OBJ is to be pasted. The robot 1 may use the end effector 4 capable of holding and releasing processes to perform a bonding process (in other words, a bonding operation) for bonding a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation). In this case, the second target object OBJ may be an object (workpiece) to which the first target object OBJ is to be bonded. Alternatively, another end effector capable of dispensing adhesive for the bonding process may be provided on the robot 1 or another robot. The robot 1 may use the end effector 4 capable of holding and releasing processes to perform a welding process (in other words, a welding operation) for welding the first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation). In this case, the second target object OBJ may be an object (workpiece) to which the first target object OBJ is to be welded. Furthermore, another end effector (for example, a processing device for welding with an energy beam) for welding the first target object OBJ and the second target object OBJ may be provided on the robot 1 or another robot.The robot 1 may use the end effector 4 capable of holding and releasing operations to perform a screw tightening operation (i.e., a screw tightening operation) for tightening a first target object OBJ capable of functioning as a screw into a screw hole formed in a second target object OBJ different from the first target object OBJ, as a specific example of a placement operation (i.e., a placement operation). In this case, the first target object OBJ may be a screw member capable of functioning as a screw, such as a bolt or a nut, and the second target object OBJ may be an object (workpiece) to which the first target object OBJ is to be tightened. Furthermore, the end effector 4 may be a tool such as a screwdriver capable of tightening screws. At least one of the adhering operation, the bonding operation, the welding operation, and the screw tightening operation may be referred to as a processing operation.

[0032] The robot 1 may perform a movement process (in other words, a movement operation) for moving a target object OBJ, as an example of a predetermined process, using the end effector 4 capable of performing a holding process and a release process. For example, the robot 1 may hold a first target object OBJ, which is a first example of the target object OBJ, using the end effector 4, and then move the end effector 4 to perform a movement process for moving the first target object OBJ held by the end effector 4. In this case, the end effector 4 may be considered to be performing a movement process on the first target object OBJ that is to be moved by the end effector 4.

[0033] The robot 1 may use an end effector 4 capable of holding and releasing processes to perform a process for discarding the target object OBJ (in other words, a discarding operation) as a specific example of a placement process (in other words, a placement operation).

[0034] The end effector 4 may perform a predetermined process on each of the multiple target objects OBJ. That is, the end effector 4 may perform the predetermined process on the multiple target objects OBJ in sequence. In this case, the robot 1 may move the end effector 4 to a first position where the end effector 4 can perform the predetermined process on one target object OBJ, and after the end effector 4 has moved to the first position, the end effector 4 may perform the predetermined process on the one target object OBJ. Thereafter, the robot 1 may move the end effector 4 to a second position where the end effector 4 can perform the predetermined process on another target object OBJ different from the one target object OBJ, and after the end effector 4 has moved to the second position, the end effector 4 may perform the predetermined process on the other target object OBJ.

[0035] For example, the end effector 4 may perform a predetermined process on each of multiple portions of a single target object OBJ. That is, the end effector 4 may sequentially perform a predetermined process on multiple portions of a single target object OBJ. In this case, the robot 1 may move the end effector 4 to a third position where the end effector 4 can perform a predetermined process on a first portion of the target object OBJ, and after the end effector 4 has moved to the third position, the end effector 4 may perform the predetermined process on the first portion of the target object OBJ. Thereafter, the robot 1 may move the end effector 4 to a fourth position where the end effector 4 can perform a predetermined process on a second portion of the target object OBJ, and after the end effector 4 has moved to the fourth position, the end effector 4 may perform the predetermined process on the second portion of the target object OBJ.

[0036] As shown in FIG. 2 , the target object OBJ on which the end effector 4 performs a predetermined process may include a workpiece W. The workpiece W may include, for example, a part or member used to manufacture a desired product. The workpiece W may include, for example, a part or member to be processed to manufacture the desired product. The workpiece W may include, for example, a part or member to be transported to manufacture the desired product. The workpiece W may include, for example, a part or member that is moving due to transportation to manufacture the desired product. The workpiece W may include, for example, a part or member that is moving to manufacture the desired product.

[0037] As shown in FIG. 2 , the target object OBJ on which the end effector 4 performs a predetermined process may include a placement device T on which a workpiece W is placed. An example of the placement device T is a container (storage box) CB. The container CB may have a bottom wall BS and a side wall SS protruding upward from the bottom wall BS. The placement device T may be a placement device T in which the workpiece W is placed on the bottom wall BS in a storage space SP enclosed by the bottom wall BS and the side wall SS. The container CB may also have a bottom wall BS and a side wall SS protruding upward from the bottom wall BS. The placement device T may be a placement device T capable of accommodating the workpiece W in the storage space SP enclosed by the bottom wall BS and the side wall SS. However, the container CB does not necessarily have a side wall SS. A container CB without a side wall SS may be referred to as a pallet. Furthermore, the placement device T is not limited to a container CB or a pallet, but may also be an existing object on which the workpiece W can be placed. The mounting device T may also be referred to as a mounting member.

[0038] The placement device T may be disposed on a support surface S. The placement device T may be fixed to the support surface S. Alternatively, at least a portion of the placement device T may be movable relative to the support surface S. A first example in which at least a portion of the placement device T is movable relative to the support surface S is when the placement device T is supported by a transport device that can move (in other words, transport) the placement device T. In this case, the transport device may be, for example, a belt conveyor. A second example in which at least a portion of the placement device T is movable relative to the support surface S is when the placement device T is supported by a movable device (movable placement device) that can move the placement device T. An example of the movable placement device is at least one of an automatic guided vehicle (AGV), an autonomous mobile robot, an unmanned aerial vehicle (e.g., a drone), and a submarine. A third example in which at least a part of the mounting device T is movable relative to the support surface S is an example in which the mounting device T functions as a movable mounting device. That is, a third example in which at least a part of the mounting device T is movable relative to the support surface S is an example in which a movable mounting device is used as the mounting device T. Note that FIG. 2 shows an example in which the mounting device T is self-propelled on the support surface S.

[0039] The robot control device 13 controls the operation of the robot 1 .

[0040] Specifically, the robot control device 13 may control the movement of the robot arm 12. For example, the robot control device 13 may control the movement of the robot arm 12 so that a desired link 121 rotates around an axis defined by a desired joint 122. For example, the robot control device 13 may control the movement of the robot arm 12 so that the end effector 4 attached to the robot arm 12 is positioned at a desired position. For example, the robot control device 13 may control the movement of the robot arm 12 so that the end effector 4 attached to the robot arm 12 moves to a desired position.

[0041] In addition to controlling the operation of the robot 1, the robot control device 13 may also control the operation of the end effector 4 attached to the robot 1 (processing performed by the end effector 4). For example, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 holds the target object OBJ at a desired timing. That is, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 performs a holding process at a desired timing. For example, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 releases the held target object OBJ at a desired timing. That is, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 performs a release process at a desired timing. In order for the end effector 4 to hold or release the target object OBJ, if the end effector 4 is a hand gripper, the robot control device 13 may control the timing of opening and closing the hand gripper. If the end effector 4 is a vacuum gripper, the robot control device 13 may control the timing of turning on / off the vacuum device of the vacuum gripper (or the vacuum suction force).If the end effector 4 is a magnetic gripper, the robot control device 13 may control the timing of turning on / off the magnetic suction device of the magnetic gripper (or the magnetic force).

[0042] Note that FIG. 2 shows an example in which the robot 1 is a robot arm 12 (i.e., a vertical articulated robot). However, the robot 1 may be a robot different from a vertical articulated robot. For example, the robot 1 may be a SCARA robot (i.e., a horizontal articulated robot). For example, the robot 1 may be a parallel link robot. For example, the robot 1 may be a dual-arm robot having two robot arms 12. For example, the robot 1 may be a Cartesian coordinate robot. For example, the robot 1 may be a cylindrical coordinate robot.

[0043] The robot 1 may be installed on a movable device (robot movable device) that can move the robot 1. Examples of the robot movable device include at least one of an automatic guided vehicle (AGV), an autonomous mobile robot, an unmanned aerial vehicle (e.g., a drone), and a submarine. When the robot 1 is installed on a robot movable device different from the robot 1, a device including the robot 1 and the robot movable device on which the robot 1 is installed may be referred to as a movable device (robot movable device). When the robot 1 is installed on a robot movable device, the robot control device 13 may control the operation of the robot movable device on which the robot 1 is installed in addition to controlling the operation of the robot 1. Note that since the robot 1 moves the robot arm 12, the robot 1 itself may be considered to be a movable device.

[0044] 1, the imaging system 2 captures an image of a target object OBJ. To capture an image of the target object OBJ, the imaging system 2 includes an imaging device 21 and an illumination device 23.

[0045] The imaging device 21 is a camera capable of capturing an image of the target object OBJ. For example, the imaging device 21 may capture an image of the target object OBJ under the control of the control device 3. The imaging device 21 generates image data IMG by capturing an image of the target object OBJ. In other words, the imaging device 21 generates image data IMG that is the result of capturing the image of the target object OBJ. The image data IMG generated by the imaging device 21 is output from the imaging device 21 to the control device 3. As a result, the control device 3 acquires the image data IMG acquired by the imaging device 21 capturing an image of the target object OBJ.

[0046] The imaging system 2 may include an imaging device 21 including a monocular camera. In this case, the imaging device 21 may generate image data IMG including one image data item generated by the monocular camera. Alternatively, the imaging system 2 may include an imaging device 21 including a stereo camera including two monocular cameras. In this case, the imaging device 21 may generate image data IMG including two image data items generated by the two monocular cameras.

[0047] Alternatively, the imaging system 2 may include an imaging device 21 including a first monocular camera and a stereo camera including two second monocular cameras different from the first monocular camera. In this case, the imaging device 21 may generate image data IMG including one image data generated by the first monocular camera and image data IMG including two image data generated by the two second monocular cameras. In this case, the imaging system 2 can also be said to be a system including a first imaging device including the first monocular camera and a second imaging device including a stereo camera.

[0048] The imaging system 2 may include an imaging device 21 that includes a camera other than a monocular camera and a stereo camera. For example, the imaging system 2 may include an imaging device 21 that includes three or more monocular cameras. For example, the imaging system 2 may include an imaging device 21 that includes at least one of a light field camera, a plenoptic camera, and a multispectral camera.

[0049] The imaging device 21 may capture an image of the entire target object OBJ. Alternatively, the imaging device 21 may capture an image of a portion of the target object OBJ. In other words, the imaging device 21 may capture an image of a portion of the target object OBJ while not capturing an image of another portion of the target object OBJ.

[0050] The imaging device 21 may capture an image of a single target object OBJ. In other words, a single target object OBJ may appear in the image represented by the image data IMG. Alternatively, the imaging device 21 may capture an image of multiple target objects OBJ. In other words, a plurality of target objects OBJ may appear in the image represented by the image data IMG. In this case, as will be described in detail later, the control device 3 may determine (in other words, select) one of the multiple target objects OBJ captured by the imaging device 21 as the target object OBJ on which the end effector 4 will actually perform a predetermined process. Note that one of the multiple target objects OBJ captured by the imaging device 21 on which the end effector 4 will actually perform a predetermined process may be referred to as a process execution object.

[0051] When the imaging device 21 captures images of multiple target objects OBJ, the multiple target objects OBJ captured by the imaging device 21 may be arranged such that at least two of the multiple target objects OBJ at least partially overlap. As an example, when the target objects OBJ are workpieces W corresponding to components used to manufacture a desired product, the multiple workpieces W (i.e., multiple components) may be arranged such that at least two of the multiple workpieces W at least partially overlap. In this case, the end effector 4 may perform the above-described holding process of holding at least one workpiece W among the multiple workpieces W that are randomly arranged. In other words, the robot 1 may perform bulk picking, in which workpieces W are picked one by one from the multiple workpieces W that are randomly arranged. Note that the multiple workpieces W that are randomly arranged may also be referred to as multiple workpieces W that are irregularly arranged, multiple workpieces W that are casually arranged, or multiple workpieces W that are randomly arranged.

[0052] In this embodiment, the state of "two objects overlapping" may include a state of "two objects overlapping while being in contact with each other." The state of "two objects overlapping" may include a state of "two objects overlapping without being in contact with each other." The state of "two objects overlapping" may include a state of "two objects entangled." The state of "two objects overlapping" may include a state in which "one of the two objects at least partially covers the other of the two objects." The state of "two objects overlapping" may include a state in which "one of the two objects is located on at least a portion of the other of the two objects."

[0053] Alternatively, the multiple target objects OBJ captured by the imaging device 21 may be arranged regularly. As an example, if the target objects OBJ are workpieces W corresponding to parts used to manufacture a desired product, the multiple workpieces W (i.e., multiple parts) may be arranged in a matrix. In this case, the end effector 4 may perform the above-mentioned holding process of holding at least one workpiece W among the multiple regularly arranged workpieces W. In other words, the robot 1 may pick up a workpiece W one by one from the multiple regularly arranged workpieces W. Note that the multiple regularly arranged workpieces W can be rephrased as multiple workpieces W that are orderly arranged, or multiple workpieces W that are arranged according to a certain arrangement rule.

[0054] Similarly to the holding process, the placement process for placing the target object OBJ held by the end effector 4 may include a placement process for placing the target object OBJ randomly (in other words, haphazardly), or a placement process for placing the target object OBJ regularly (in other words, orderly). For example, the robot 1 may use the end effector 4 to hold one of a plurality of workpieces W that are regularly or randomly placed on the first placement device T, and then place the one workpiece W held by the end effector 4 regularly or randomly within the second placement device T.

[0055] The illumination device 23 is a device capable of irradiating illumination light onto the target object OBJ (for example, at least one target object OBJ when multiple target objects OBJ exist). For example, the illumination device 23 may irradiate the target object OBJ with illumination light under the control of the control device 3. In particular, the illumination device 23 is a device capable of irradiating the target object OBJ with illumination light, thereby illuminating the target object OBJ with illumination light. In this case, the imaging device 21 may capture an image of the target object OBJ illuminated with illumination light. However, the illumination device 23 does not need to irradiate the target object OBJ with illumination light. In this case, the imaging system 2 (robot system SYS) does not need to be equipped with the illumination device 23.

[0056] When the imaging device 21 includes a stereo camera, the illumination device 23 may be a device capable of projecting a desired projection pattern onto the target object OBJ by irradiating the target object OBJ with illumination light. The desired projection pattern may include, for example, a random pattern. The random pattern may include a random dot pattern. The desired projection pattern may include, for example, a one-dimensional or two-dimensional grid pattern. The desired projection pattern may include, for example, a line pattern. The desired projection pattern may include, for example, a stripe pattern. The desired projection pattern may include other projection patterns. However, even when the imaging device 21 does not include a stereo camera (i.e., the imaging device 21 includes a monocular camera), the illumination device 23 may project the desired projection pattern onto the target object OBJ by irradiating the target object OBJ with illumination light. In this case, the illumination device 23 may irradiate the target object OBJ with illumination light having a uniform intensity distribution (e.g., a uniform intensity distribution). The desired projection pattern can also be referred to as light having a desired intensity distribution.

[0057] The imaging system 2 is attached to the robot arm 12, similar to the end effector 4. That is, the imaging device 21 and the lighting device 23 are attached to the robot arm 12. For example, as shown in FIG. 2 , the imaging device 21 and the lighting device 23 may be attached to the tip of the robot arm 12, similar to the end effector 4. In this case, the imaging device 21 and the lighting device 23 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the imaging device 21 and the lighting device 23.

[0058] The imaging device 21 may capture an image of the target object OBJ (e.g., at least one target object OBJ when multiple target objects OBJ are present) during a period in which the imaging device 21 and the target object OBJ are displaced relative to each other. Note that the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may refer to a state in which the relative positional relationship between the imaging device 21 and the target object OBJ is changing. The state in which the imaging device 21 and the target object OBJ are displaced relative to each other may refer to a state in which the imaging device 21 and the target object OBJ are moving relative to each other. For example, the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the target object OBJ is moving relative to the imaging device 21. For example, the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the imaging device 21 is moving relative to the target object OBJ. In this case, the imaging device 21 does not need to be stationary to capture an image of the target object OBJ, and the robot system SYS can efficiently perform predetermined processing on the target object OBJ using the end effector 4.

[0059] Alternatively, the imaging device 21 may capture an image of the target object OBJ during a period in which there is no relative displacement between the imaging device 21 and the target object OBJ. Note that the state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the relative positional relationship between the imaging device 21 and the target object OBJ is not changing. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which there is no relative movement between the imaging device 21 and the target object OBJ. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the imaging device 21 and the target object OBJ are stationary. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the imaging device 21 and the target object OBJ are moving at the same moving speed in the same moving direction.

[0060] The control device 3 performs robot control processing. The robot control processing includes processing for generating a robot control signal for controlling the robot 1. Specifically, the control device 3 generates the robot control signal based on image data IMG output from the imaging system 2 (e.g., the imaging device 21). In this embodiment, the control device 3 calculates at least one of the position and orientation of the target object OBJ in the global coordinate system of the robot system SYS based on the image data IMG, and generates the robot control signal based on the calculated at least one of the position and orientation of the target object OBJ.

[0061] The global coordinate system is a coordinate system that serves as the reference for the robot system SYS. For example, the global coordinate system may be a coordinate system that serves as the reference for the robot 1. The global coordinate system can also be said to be a coordinate system used to control the robot 1. For example, a world coordinate system that is defined based on the support surface S on which the robot 1 is placed may be used as the global coordinate system. In other words, a world coordinate system that is fixed with respect to the support surface S on which the robot 1 is placed may be used as the global coordinate system. In the following description, unless otherwise specified, the X-axis, Y-axis, and Z-axis may refer to the X-axis, Y-axis, and Z-axis in the global coordinate system, respectively.

[0062] However, the control device 3 may calculate at least one of the position and orientation of the target object OBJ in a coordinate system different from the global coordinate system based on the image data IMG. The coordinate system different from the global coordinate system may include at least one of the robot coordinate system and the imaging coordinate system. The robot coordinate system may be a coordinate system defined based on the robot 1. That is, the robot coordinate system may be a coordinate system fixed with respect to the robot 1 (for example, fixed with respect to the base 11 of the robot 1). The imaging coordinate system may be a coordinate system defined based on the imaging device 21. That is, the imaging coordinate system may be a coordinate system fixed with respect to the imaging device 21. An example of the imaging coordinate system is a coordinate system defined based on the optical axis AX21 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 21. An example of the imaging coordinate system is a coordinate system in which one of the three coordinate axes constituting the imaging coordinate system is an axis along the optical axis AX21 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 21.

[0063] The control device 3 may perform end effector control processing in addition to or instead of performing robot control processing. The end effector control processing may include processing for generating an end effector control signal for controlling the end effector 4. For example, the control device 3 may generate the end effector control signal based on image data IMG output from the imaging system 2 (e.g., the imaging device 21). Specifically, the control device 3 may generate the end effector control signal based on at least one of the position and orientation of the target object OBJ calculated from the image data IMG.

[0064] The end effector control processing may or may not be included in the robot control processing. In other words, the end effector control signal generated by the control device 3 may or may not be included in the robot control signal. In the following description, for convenience of explanation, an example will be described in which the end effector control processing is included in the robot control processing (i.e., the end effector control signal is included in the robot control signal). Therefore, in the following description, the robot control processing may refer to processing for generating at least one of a robot control signal and an end effector control signal. In addition, in the following description, the robot control signal may refer to at least one of a signal for controlling the robot 1 and a signal for controlling the end effector 4. The robot control signal may simply be referred to as a control signal. The robot control signal may also be referred to as robot control information or control information.

[0065] As described above, when the robot 1 is installed on a robotic movable device (e.g., at least one of an automatic guided vehicle, an autonomously traveling transport robot, an unmanned aerial vehicle, and a submarine), the control device 3 may perform a movable device control process in addition to or instead of performing a robot control process. The movable device control process may include a process of generating a movable device control signal for controlling the robotic movable device. For example, the control device 3 may generate the movable device control signal based on image data IMG output from the imaging system 2 (e.g., the imaging device 21). Specifically, the control device 3 may generate the movable device control signal based on at least one of the position and orientation of the target object OBJ calculated from the image data IMG.

[0066] The movable device control process may or may not be included in the robot control process. In other words, the movable device control signal generated by the control device 3 may or may not be included in the robot control signal. In the following description, for convenience of explanation, an example will be described in which the movable device control process is included in the robot control process (i.e., the movable device control signal is included in the robot control signal). Therefore, in the following description, the robot control process may mean a process for generating at least one of a robot control signal, an end effector control signal, and a movable device control signal. In addition, in the following description, the robot control signal may mean at least one of a signal for controlling the robot 1, a signal for controlling the end effector 4, and a signal for controlling the robot movable device.

[0067] In this way, the control device 3 and the imaging system 2 are used to control the robot 1. Therefore, a system including the control device 3 and the imaging system 2 may be referred to as a robot control system or a control system.

[0068] The robot control signal generated by the control device 3 is output to the robot control device 13 of the robot 1. The robot control device 13 controls the operation of the robot 1 based on the robot control signal generated by the control device 3. For this reason, the robot control signal may include a signal for controlling the operation of the robot 1.

[0069] As described above, when the robot control signal includes a signal for controlling the robot arm 12, the robot control device 13 may control the robot arm 12 based on the robot control signal. For example, the robot control device 13 may control the operation of the actuator built into the joint 122 based on the robot control signal, thereby controlling the operation of the robot arm 12.

[0070] For example, as described above, the robot arm 12 moves the end effector 4. In this case, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at a desired position. The robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves to a desired position. The robot control signal may include a signal for controlling the robot arm 12 so that the positional relationship between the end effector 4 and the target object OBJ is a desired positional relationship. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 is located at a desired position. The robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves to a desired position. The robot control device 13 may control the robot arm 12 based on the robot control signal so that the positional relationship between the end effector 4 and the target object OBJ is a desired positional relationship.

[0071] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves toward (i.e., approaches) a first desired position where the end effector 4 can hold the target object OBJ. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at the first desired position. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves toward (i.e., approaches) the first desired position. That is, the robot control signal may control the robot arm 12 so that the end effector 4 is located at the first desired position. Note that, because the end effector 4 located at the first desired position holds the target object OBJ, controlling the robot arm 12 so that the end effector 4 approaches the first desired position may be considered equivalent to controlling the robot arm 12 so that the end effector 4 approaches the target object OBJ that is to be held at the first desired position.

[0072] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the robot arm 12 so that the posture of the end effector 4 becomes a first desired posture that enables the end effector 4 to hold the target object OBJ. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the posture of the end effector 4 becomes the first desired posture.

[0073] As another example, when performing a release process to release the target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves toward (i.e., approaches) a second desired position where the target object OBJ held by the end effector 4 should be released. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at the second desired position. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves toward (i.e., approaches) the second desired position. That is, the robot control signal may control the robot arm 12 so that the end effector 4 is located at the second desired position. Furthermore, since the end effector 4 located at the second desired position releases the first target object OBJ that it is holding to the second target object OBJ, controlling the robot arm 12 so that the end effector 4 approaches the second desired position may be considered equivalent to controlling the robot arm 12 so that at least one of the end effector 4 and the first target object OBJ approaches the second target object OBJ from which the first target object OBJ is released at the second desired position.

[0074] As another example, when performing a release process to release the target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the robot arm 12 so that the posture of the end effector 4 becomes a second desired posture that allows the end effector 4 to release the target object OBJ. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the posture of the end effector 4 becomes the second desired posture.

[0075] As described above, when the robot control signal includes a signal for controlling the end effector 4, the robot control device 13 may control the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the end effector 4 by controlling the operation of an actuator that moves a hand gripper that constitutes the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the end effector 4 by controlling the operation of a vacuum device of a vacuum gripper that constitutes the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the end effector 4 by controlling the operation of a magnetic attraction device of a magnetic gripper that constitutes the end effector 4 based on the robot control signal.

[0076] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the end effector 4 so that the end effector 4 located at the above-mentioned first desired position and / or in the above-mentioned first desired posture holds the target object OBJ. In this case, the robot control device 13 may control the end effector 4 based on the robot control signal so that the end effector 4 located at the first desired position and / or in the first desired posture holds the target object OBJ.

[0077] As another example, when performing a release process to release the target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the end effector 4 to release the target object OBJ held by the end effector 4 located at the above-mentioned second desired position and / or in the above-mentioned second desired orientation. In this case, the robot control device 13 may control the end effector 4 based on the robot control signal to release the target object OBJ held by the end effector 4 located at the second desired position and / or in the second desired orientation.

[0078] As described above, when the robot control signal includes a signal for controlling a robotic movable device (e.g., at least one of an automatic guided vehicle, an autonomously traveling transport robot, an unmanned aerial vehicle, and a submarine) on which the robot 1 is installed, the robot controller 13 may control the robotic movable device based on the robot control signal. For example, the robot controller 13 may control a power source (e.g., a motor or an engine) of the robotic movable device based on the robot control signal so that the robot 1 moves to a target position indicated directly or indirectly by the robot control signal.

[0079] The above description has been given of an example in which the robot 1 having the robot arm 12 is installed on a robot movable apparatus. In this case, the end effector 4 attached to the robot arm 12 may be considered to be installed on the robot movable apparatus via the robot 1 (e.g., via the robot arm 12). On the other hand, the end effector 4 may be installed on the robot movable apparatus without the robot 1 (e.g., without the robot arm 12). In this case, the robot 1 (e.g., the robot arm 12) may not be installed on the robot movable apparatus. In this case, the control device 3 may generate at least one of a movable apparatus control signal for controlling the robot movable apparatus on which the end effector 4 is installed and an end effector control signal for controlling the end effector 4 installed on the robot movable apparatus, without generating a robot control signal for controlling the robot 1.

[0080] In addition, when the robot 1 (e.g., the robot arm 12) is not installed on the robot movable apparatus, the imaging system 2 may be installed on the robot movable apparatus. In this case, the control device 3 may generate at least one of a movable apparatus control signal for controlling the robot movable apparatus on which the end effector 4 is installed and an end effector control signal for controlling the end effector 4 installed on the robot movable apparatus, based on the image data IMG generated by the imaging system 2 installed on the robot movable apparatus. In addition, when the end effector 4 (and, in some cases, the imaging system 2) is installed on the robot movable apparatus, the control device 3 may also be disposed inside the robot movable apparatus.

[0081] The robot control signal may include a signal that can be used as is by the robot control device 13 to control the operation of the robot 1. The robot control signal may include a signal that can be used as is as a robot drive signal that the robot control device 13 uses to control the operation of the robot 1. In this case, the robot control device 13 may use the robot control signal as is to control the operation of the robot 1. For example, the control device 3 may generate a drive signal for an actuator built into the joint 122 of the robot arm 12 as the robot control signal, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the actuator built into the joint 122 of the robot arm 12.

[0082] The robot control signal may include a signal that can be used directly by the robot control device 13 to control the operation of the end effector 4. The robot control signal may include a signal that can be used directly as an end effector drive signal used by the robot control device 13 to control the operation of the end effector 4. In this case, the robot control device 13 may use the robot control signal directly to control the operation of the end effector 4. For example, the control device 3 may generate, as the robot control signal, a drive signal (end effector drive signal) for an actuator that moves a hand gripper that constitutes the end effector 4, and the robot control device 13 may use the robot control signal generated by the control device 3 directly to control the actuator of the end effector 4. For example, the control device 3 may generate, as the robot control signal, a drive signal (end effector drive signal) for driving a vacuum device of a vacuum gripper that constitutes the end effector 4, and the robot control device 13 may use the robot control signal generated by the control device 3 directly to control the vacuum device of the end effector 4. For example, the control device 3 may generate a drive signal (end effector drive signal) for driving the magnetic adsorption device of the magnetic gripper that constitutes the end effector 4 as a robot control signal, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the magnetic adsorption device of the end effector 4.

[0083] The robot control signal may include a signal that can be used directly by the robot controller 13 to control the operation of a robot movable device on which the robot 1 is installed. The robot control signal may include a signal that can be used directly as a movable device drive signal that the robot controller 13 uses to control the operation of the robot movable device. In this case, the robot controller 13 may use the robot control signal directly to control the operation of the robot movable device. For example, the controller 3 may generate a power source drive signal (movable device drive signal) for driving a power source (e.g., a motor or engine) of the robot movable device as the robot control signal, and the robot controller 13 may use the robot control signal generated by the controller 3 directly to control the power source of the robot movable device.

[0084] As described above, if the robot control signal includes a signal that the robot control device 13 can use to control the operation of at least one of the robot 1, the end effector 4, and the robot movable device, the robot 1 does not need to be equipped with the robot control device 13. In this case, the control device 3 may use the robot control signal to control an actuator built into the joint 122 of the robot arm 12. For example, the control device 3 may use the robot control signal (end effector drive signal) to control an actuator that moves a hand gripper that constitutes the end effector 4. For example, the control device 3 may use the robot control signal (end effector drive signal) to control a vacuum device of a vacuum gripper that constitutes the end effector 4. For example, the control device 3 may use the robot control signal (end effector drive signal) to control a magnetic attraction device of a magnetic gripper that constitutes the end effector 4. For example, the control device 3 may use the robot control signal (end effector drive signal) to control a robot movable device installed on the robot 1.

[0085] Alternatively, the robot control signal may include a signal that can be used by the robot control device 13 to generate a robot drive signal for controlling the operation of the robot 1. In this case, the robot control device 13 may generate a robot drive signal for controlling the operation of the robot 1 based on the robot control signal, and control the operation of the robot 1 based on the generated robot drive signal. For example, the robot control device 13 may generate a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 based on the robot control signal, and control the actuator built into the joint 122 of the robot arm 12 based on the generated robot drive signal.

[0086] The robot control signal may include a signal that the robot control device 13 can use to generate an end effector drive signal for controlling the operation of the end effector 4. In this case, the robot control device 13 may generate an end effector drive signal for controlling the operation of the end effector 4 based on the robot control signal and control the operation of the end effector 4 based on the generated end effector drive signal. For example, if the end effector 4 is a hand gripper, the robot control device 13 may generate an end effector drive signal for driving an actuator of the hand gripper based on the robot control signal and control the actuator of the hand gripper based on the generated end effector drive signal. For example, if the end effector 4 is a magnetic gripper, the robot control device 13 may generate an end effector drive signal for driving a magnetic attraction device of the magnetic gripper based on the robot control signal and control the magnetic attraction device based on the generated end effector drive signal.

[0087] Note that, if the robot system SYS includes a control device for controlling the end effector 4 in addition to the robot control device 13, the robot control signal may include a signal that can be used by the control device for controlling the end effector 4 to generate an end effector drive signal for controlling the operation of the end effector 4. In this case, the control device for controlling the end effector 4 may generate the end effector drive signal for controlling the operation of the end effector 4 based on the robot control signal, and control the operation of the end effector 4 based on the generated end effector drive signal.

[0088] The robot control signal may include a signal usable by the robot controller 13 to generate a movable unit drive signal for controlling the operation of a robot movable unit on which the robot 1 is installed. In this case, the robot controller 13 may generate a movable unit drive signal for controlling the operation of the robot movable unit based on the robot control signal, and control the operation of the robot movable unit based on the generated movable unit drive signal. For example, the robot controller 13 may generate a power source drive signal (movable unit drive signal) for driving a power source (e.g., a motor or an engine) of the robot movable unit based on the robot control signal, and control the power source of the robot movable unit based on the generated movable unit drive signal.

[0089] In addition, if the robot system SYS includes a control device for controlling the robot movable device in addition to the robot control device 13, the robot control signal may include a signal usable by the control device for controlling the robot movable device to generate a movable device drive signal for controlling the operation of the robot movable device. In this case, the control device for controlling the robot movable device may generate a movable device drive signal for controlling the operation of the robot movable device based on the robot control signal, and control the operation of the robot movable device based on the generated movable device drive signal.

[0090] The signals available to the robot controller 13 for generating the robot drive signals may include signals representing at least one of the position and orientation of the target object OBJ in a global coordinate system. The signals available to the robot controller 13 for generating the robot drive signals may include signals representing a desired positional relationship between the robot 1 and the target object OBJ in a global coordinate system.

[0091] The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value (target position) of the position of the end effector 4 in the global coordinate system. An example of a target position is a processing position where the end effector 4 should process the target object OBJ. For example, the target position may include a position where the end effector 4 should hold the target object OBJ. For example, the target position may include a position where the end effector 4 should release the target object OBJ. The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value of the position of the tip of the robot arm 12 (e.g., a tool center point) in the global coordinate system. The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value of the position of the imaging system 2 in the global coordinate system.

[0092] The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value (target posture) of the posture of the end effector 4 in the global coordinate system. An example of a target posture is a posture (processing posture) that the end effector 4 should take when processing the target object OBJ. For example, the target posture may include a posture that the end effector 4 should take when holding the target object OBJ. For example, the target posture may include a posture that the end effector 4 should take when releasing the target object OBJ. The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value of the posture of the tip of the robot arm 12 (e.g., a tool center point) in the global coordinate system. The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value of the posture of the imaging system 2 in the global coordinate system.

[0093] The signal that the robot control device 13 can use to generate the robot drive signal may be a signal that indicates the amount and direction of movement from the current position of the end effector 4 to the target position of the end effector 4 .

[0094] Next, the position and orientation changing device 5 is a device that can receive a target object OBJ held by the robot 1 using the end effector 4 from the robot 1 (i.e., from the end effector 4). In this case, the robot 1 may perform a process of passing the target object OBJ held by the end effector 4 to the position and orientation changing device 5 (in other words, placing or releasing the target object OBJ on the position and orientation changing device 5) as an example of a predetermined process (e.g., a release process). In this case, the end effector 4 may be considered to be performing a release process on the position and orientation changing device 5 to which the end effector 4 passes the target object OBJ.

[0095] The position and orientation changing device 5 may be able to receive, from the robot 1, a target object OBJ held by the robot 1 using the end effector 4 under the control of the control device 3. In this case, the control device 3 may generate a control signal for controlling the position and orientation changing device 5 to receive the target object OBJ from the robot 1, and output the generated control signal to the position and orientation changing device 5. The position and orientation changing device 5 may receive the target object OBJ from the robot 1 based on the control signal generated by the control device 3. In the following description, for convenience of explanation, the control signal for controlling the position and orientation changing device 5 will be referred to as a position and orientation control signal. The position and orientation control signal may also be referred to as position and orientation control information or control information.

[0096] Furthermore, the position and orientation changing device 5 is a device that can change at least one of the position and orientation of the target object OBJ received from the robot 1 (i.e., the end effector 4). For example, the position and orientation changing device 5 may be a device that can change the position of the target object OBJ along at least one of the X-axis, Y-axis, and Z-axis. For example, the position and orientation changing device 5 may be a device that can change the orientation of the target object OBJ around at least one of the X-axis, Y-axis, and Z-axis.

[0097] The position and orientation modification device 5 may be capable of modifying at least one of the position and orientation of the target object OBJ under the control of the control device 3. In this case, the control device 3 may generate a position and orientation control signal for controlling the position and orientation modification device 5 to modify at least one of the position and orientation of the target object OBJ, and output the generated position and orientation control signal to the position and orientation modification device 5. The position and orientation modification device 5 may modify at least one of the position and orientation of the target object OBJ based on the position and orientation control signal generated by the control device 3.

[0098] The position and orientation changing device 5 may also be referred to as a position changing device, orientation changing device, changing device, inverting machine, inverting mechanism, or tool mover. As an example, a position and orientation changing device 5 capable of changing the position of the target object OBJ may also be referred to as a position changing device or changing device. For example, a position and orientation changing device 5 capable of changing the orientation of the target object OBJ may also be referred to as an orientation changing device or changing device. A position and orientation changing device 5 capable of changing the orientation of the target object OBJ so that the target object OBJ is inverted may also be referred to as an orientation changing device, changing device, inverting machine, inverting mechanism, or tool mover.

[0099] Furthermore, the position and orientation changing device 5 is a device that can deliver the target object OBJ, the position and / or orientation of which has been changed, to the robot 1 (i.e., to the end effector 4). In this case, the robot 1 may perform a process of receiving (in other words, holding) the target object OBJ held by the position and orientation changing device 5 as an example of a predetermined process (e.g., holding process). In this case, the end effector 4 may be considered to be performing a holding process on the position and orientation changing device 5 that holds the target object OBJ that the end effector 4 should hold.

[0100] The position and orientation change device 5 may be able to pass the target object OBJ to the robot 1 under the control of the control device 3. In this case, the control device 3 may generate a position and orientation control signal for controlling the position and orientation change device 5 to pass the target object OBJ to the robot 1, and output the generated position and orientation control signal to the position and orientation change device 5. The position and orientation change device 5 may pass the target object OBJ to the robot 1 based on the position and orientation control signal generated by the control device 3.

[0101] Note that the action of passing the target object OBJ to the robot 1 may include an action of actively passing the target object OBJ to the robot 1. The action of actively passing the target object OBJ to the robot 1 may include an action of passing the target object OBJ to the robot 1 by moving the target object OBJ. In other words, the action of actively passing the target object OBJ to the robot 1 may include an action of moving the target object OBJ in order to pass it to the robot 1. In this case, the position and posture changing device 5 may pass the target object OBJ to the robot 1 by moving the target object OBJ toward the robot 1.

[0102] Note that the action of passing the target object OBJ to the robot 1 may include an action of passively passing the target object OBJ to the robot 1. The action of passively passing the target object OBJ to the robot 1 may include an action of passing the target object OBJ to the robot 1 without moving the target object OBJ. In other words, the action of passively passing the target object OBJ to the robot 1 may include an action that does not require moving the target object OBJ to pass it to the robot 1. In this case, the robot 1 may move to receive the target object OBJ, and the position and posture changing device 5 may release (in other words, release) the target object OBJ, thereby passing the target object OBJ to the robot 1 that has come to receive it.

[0103] An example of such a position and orientation changing device 5 is shown in Fig. 3. Fig. 3 is a side view conceptually showing the appearance of the position and orientation changing device 5. Note that the configuration shown in Fig. 3 is merely an example of the position and orientation changing device 5, and the configuration of the position and orientation changing device 5 is not limited to the configuration shown in Fig. 3.

[0104] 3 , the position and orientation changing device 5 may include a base 51, an actuator 52, and a holding member 53. The base 51 is a member that serves as the base of the position and orientation changing device 5. An actuator 52 such as a motor may be attached to the base 51. A holding member 53 may be attached to the actuator 52. However, the holding member 53 may be attached to the base 51.

[0105] The holding member 53 is a member for holding the target object OBJ. In other words, the holding member 53 is a member that can hold the target object OBJ. Furthermore, the holding member 53 is a member that can release (in other words, can let go of) the held target object OBJ. Examples of the holding member 53 include at least one of a hand gripper, a vacuum gripper (e.g., a vacuum suction-type gripper), and a magnetic gripper (e.g., a magnetic suction-type gripper). Note that the hand gripper, vacuum gripper, and magnetic gripper have already been described when describing the end effector 4, and therefore detailed description thereof will be omitted.

[0106] The actuator 52 is capable of changing at least one of the position and the orientation of the holding member 53. For example, the actuator 52 may be capable of changing the position of the holding member 53 along at least one of the X-axis, the Y-axis, and the Z-axis. As an example, the actuator 52 may be capable of changing the position of the holding member 53 along at least one of the X-axis, the Y-axis, and the Z-axis by moving (i.e., linearly moving) the holding member 53 along at least one of the X-axis, the Y-axis, and the Z-axis. For example, the actuator 52 may be capable of changing the orientation of the holding member 53 around at least one of the X-axis, the Y-axis, and the Z-axis by rotating (i.e., rotationally moving) the holding member 53 around at least one of a rotation axis along the X-axis, a rotation axis along the Y-axis, and a rotation axis along the Z-axis.

[0107] The actuator 52 may change at least one of the position and orientation of the holding member 53 during at least a portion of the time period during which the holding member 53 holds the target object OBJ. That is, the actuator 52 may change at least one of the position and orientation of the holding member 53 holding the target object OBJ. As a result, as the position of the holding member 53 changes, the position of the target object OBJ held by the holding member 53 also changes. Similarly, as the orientation of the holding member 53 changes, the orientation of the target object OBJ held by the holding member 53 also changes. In this way, the position and orientation changing device 5 may change at least one of the position and orientation of the holding member 53 holding the target object OBJ, thereby changing at least one of the position and orientation of the holding member 53.

[0108] 3 is merely one example of the position / posture changing device 5, and the configuration of the position / posture changing device 5 is not limited to the configuration shown in Fig. 3. For example, a vertical articulated robot may be used as the position / posture changing device 5. In other words, a robot different from the robot 1 may be used as the position / posture changing device 5.

[0109] In the above description, the control device 3 controls the position and orientation changing device 5. However, a control device different from the control device 3 may control the position and orientation changing device 5. For example, the robot control device 13 included in the robot 1 may control the position and orientation changing device 5. For example, if the position and orientation changing device 5 includes a control device, the robot control device 13 included in the position and orientation changing device 5 may control the position and orientation changing device 5 itself. However, for convenience of explanation, the following description will be given of an example in which the control device 3 controls the position and orientation changing device 5.

[0110] (1-2) Configuration of the Control Device 3 Next, the configuration of the control device 3 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the control device 3.

[0111] 4, the control device 3 includes a calculation device 31, a storage device 32, and a communication device 33. The control device 3 may further include an input device 34 and an output device 35. However, the control device 3 does not necessarily have to include at least one of the input device 34 and the output device 35. The calculation device 31, the storage device 32, the communication device 33, the input device 34, and the output device 35 may be connected via a data bus 36.

[0112] The arithmetic device 31 is hardware that includes at least a circuit (for example, at least one of an electronic circuit and an electric circuit). For this reason, the arithmetic device 31 may be referred to as a group of circuits.

[0113] The arithmetic device 31 includes at least one processor (i.e., one processor or multiple processors) as hardware. The processor may include, for example, a processor conforming to a von Neumann computer architecture. The processor conforming to the von Neumann computer architecture may include at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor may include, for example, a processor conforming to a non-von Neumann computer architecture. The processor conforming to the non-von Neumann computer architecture may include at least one of an FPGA (Field Programmable Gate Array) and an ASIC (Application Specific Circuit). The processor may be realized by a group of circuits (e.g., at least one of an electronic circuit and an electric circuit).

[0114] The arithmetic device 31 reads a computer program 321 including at least one of computer program code and computer program instructions. For example, the arithmetic device 31 may read the computer program 321 stored in the storage device 32. For example, the arithmetic device 31 may read the computer program 321 stored in a computer-readable, non-transitory storage medium using a storage medium reading device (not shown) included in the control device 3. The computer program 321 read from the storage medium may be stored in the storage device 32. The arithmetic device 31 may acquire (i.e., download or read) the computer program 321 from a device (not shown) located outside the control device 3 via the communication device 33 (or another communication device). The downloaded computer program 321 may be stored in the storage device 32.

[0115] The arithmetic device 31 executes the loaded computer program 321. As a result, logical functional blocks for executing the processing to be performed by the control device 3 (for example, the robot control processing described above) are realized within the arithmetic device 31. In other words, the arithmetic device 31, together with the storage device 32 or the like in which the computer program 321 is recorded (in other words, together with the storage device 32 and the computer program 321 recorded in the storage device 32 or the like), can function as a controller or computer for realizing the logical functional blocks for executing the processing to be performed by the control device 3. In other words, the at least one processor included in the arithmetic device 31, the memory (recording medium) included in the storage device 32 or the like, and the computer program 321 are configured so that the control device 3 performs the processing to be performed by the control device 3 (for example, the robot control processing described above).

[0116] The arithmetic device 31 may include a single processor. In this case, the arithmetic device 31 may use the single processor to perform the processing to be performed by the control device 3 (e.g., the robot control processing described above). For example, if the arithmetic device 31 performs a first operation (e.g., a first process that is part of the robot control processing) and a second operation (e.g., a second process that is another part of the robot control processing), the arithmetic device 31 may use a single processor to perform both the first and second operations. Alternatively, the arithmetic device 31 may include multiple processors. In this case, the arithmetic device 31 may use any one of the multiple processors to perform the processing to be performed by the control device 3 (e.g., the robot control processing described above). For example, if the arithmetic device 31 includes first and second processors and performs the first and second operations, the arithmetic device 31 may use any one of the first and second processors to perform each of the first and second operations. For example, the computing device 31 may perform a first operation using a first processor, may perform a second operation using the first processor, may perform the first operation using a second processor, or may perform the second operation using the second processor.

[0117] A computational model that can be constructed by machine learning may be implemented in the computational device 31 by the computational device 31 executing the computer program 321. An example of a computational model that can be constructed by machine learning is a computational model including a neural network (so-called artificial intelligence (AI)). In this case, learning of the computational model may include learning of parameters of the neural network (e.g., at least one of a weight and a bias). The computational device 31 may execute a robot control process using the computational model. That is, the operation of executing the robot control process may include an operation of executing the robot control process using the computational model. For example, the computational device 31 may execute at least a part of the robot control process shown in FIG. 7 (described later) using the computational model. As will be described in detail later, the robot control process shown in FIG. 7 includes a holding control process shown in FIG. 8 (described later), a workpiece position / posture change process shown in FIG. 9 (described later), and a release control process shown in FIG. 12 (described later). In this case, the calculation device 31 may use the calculation model to execute at least a part of the hold control process shown in FIG. 8 (described later). The calculation device 31 may use the calculation model to execute at least a part of the workpiece position and orientation change process shown in FIG. 9 (described later). The calculation device 31 may use the calculation model to execute at least a part of the release control process shown in FIG. 12 (described later). Furthermore, as will be described in detail later, the control device 3 may execute at least one of the workpiece position and orientation change process shown in FIG. 13 (described later), the workpiece position and orientation change process shown in FIG. 15 (described later), the workpiece position and orientation change process shown in FIG. 18 (described later), and the workpiece position and orientation change process shown in FIG. 20 (described later), instead of the workpiece position and orientation change process shown in FIG. 9. In this case, the calculation device 31 may use the calculation model to execute at least a part of the workpiece position and orientation change process shown in FIG. 13. The calculation device 31 may use the calculation model to execute at least a part of the workpiece position and orientation change process shown in FIG. 15. The calculation device 31 may use the calculation model to execute at least a part of the workpiece position and orientation change process shown in FIG. 18 (described later). The calculation device 31 may use the calculation model to perform at least a part of the workpiece position and orientation change processing shown in FIG. 20 .Furthermore, as will be described in detail later, the control device 3 may perform a release control process shown in Fig. 22, which will be described later, instead of the release control process shown in Fig. 12. In this case, the calculation device 31 may perform at least a part of the release control process shown in Fig. 22 using a calculation model.

[0118] Note that a computational model already constructed by offline machine learning using teacher data may be implemented in the computational device 31. Furthermore, the computational model implemented in the computational device 31 may be updated by online machine learning on the computational device 31. Alternatively, the computational device 31 may execute the robot control process using a computational model implemented in a device external to the computational device 31 (i.e., a device provided outside the control device 3) in addition to or instead of the computational model implemented in the computational device 31.

[0119] The recording medium for recording the computer program 321 executed by the arithmetic device 31 may be at least one of the following: a CD-ROM, CD-R, CD-RW, a flexible disk, an MO, a DVD-ROM, a DVD-RAM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, or an optical disk such as Blu-ray (registered trademark), a magnetic medium such as a magnetic tape, a magneto-optical disk, a semiconductor memory such as a USB memory, or any other medium capable of storing a program. The recording medium may include a device capable of recording the computer program 321 (for example, a general-purpose device or a dedicated device in which the computer program 321 is implemented in a state in which it can be executed in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program 321 may be realized by a logical processing block realized within the arithmetic device 31 (i.e., processor) when the arithmetic device 31 executes the computer program 321, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) provided in the arithmetic device 31, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.

[0120] Fig. 4 shows an example of logical functional blocks realized in the arithmetic device 31 for executing robot control processing. As shown in Fig. 4, a position and orientation calculation unit 311 and a signal generation unit 312 are realized in the arithmetic device 31. Note that the processing performed by the position and orientation calculation unit 311 and the signal generation unit 312 will be described in detail later with reference to Fig. 7 etc., and therefore will not be described here.

[0121] The storage device 32 includes at least one memory capable of storing desired data. In other words, the storage device 32 includes at least one memory containing desired data. The memory may be realized by a group of circuits (e.g., at least one of electronic circuits and electric circuits). For example, the storage device 32 may store a computer program 321 executed by the arithmetic device 31. In this case, the storage device 32 (memory) may be used as the above-mentioned recording medium for recording the computer program 321 executed by the arithmetic device 31. The storage device 32 may temporarily store data used by the arithmetic device 31 when the arithmetic device 31 is executing the computer program 321. The storage device 32 may also store data to be stored long-term by the control device 3. The storage device 32 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. That is, the storage device 32 may include a non-transitory recording medium.

[0122] The communication device 33 is capable of communicating with both the robot 1 and the imaging system 2 via a communication network (not shown). Alternatively, the communication device 33 may be capable of communicating with another device different from the robot 1 and the imaging system 2, in addition to or instead of at least one of the robot 1 and the imaging system 2, via a communication network (not shown). In this embodiment, the communication device 33 may receive (i.e., acquire) image data IMG from the imaging system 2. Furthermore, the communication device 33 may transmit (i.e., output) a robot control signal to the robot 1. Note that the communication device 33 that outputs the robot control signal to the robot 1 may be referred to as an output unit or an output device.

[0123] The input device 34 is a device that accepts information input to the control device 3 from outside the control device 3. For example, the input device 34 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by a user of the control device 3. For example, the input device 34 may include a recording medium reading device that can read information recorded as data on a recording medium that can be externally attached to the control device 3.

[0124] It should be noted that information can be input as data to the control device 3 from a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an input device that accepts information input to the control device 3 from outside the control device 3.

[0125] The output device 35 is a device that outputs information to the outside of the control device 3. For example, the output device 35 may output information as an image. That is, the output device 35 may include a display device (so-called display) 37 that can display an image. In this case, the calculation device 31 may generate a display signal for displaying the image on the display device 37. Specifically, the calculation device 31 may generate, as the display signal, a display control signal for controlling the display device 37 to display the image. The calculation device 31 may output the generated display signal (display control signal) to the display device 37 via the data bus 36. The display device 37 may display an image based on the display signal (display control signal) generated by the calculation device 31.

[0126] The output device 35 may include an output device different from the display device 37. For example, the output device 35 may output information as sound. That is, the output device 35 may include an audio device (a so-called speaker) capable of outputting sound. For example, the output device 35 may output information on paper. That is, the output device 35 may include a printing device (a so-called printer) capable of printing desired information on paper. For example, the output device 35 may output information as data to a recording medium that can be externally attached to the control device 3.

[0127] The control device 3 can output information as data to a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an output device that outputs information to a device external to the control device 3.

[0128] The robot control device 13 provided in the robot 1 described above may also have the same configuration as the control device 3. That is, as shown in Fig. 5, which is a block diagram showing the configuration of the robot control device 13, the robot control device 13 includes an arithmetic device 131, a storage device 132, and a communication device 133. The robot control device 13 may further include an input device 134 and an output device 135. However, the robot control device 13 does not have to include at least one of the input device 134 and the output device 135. The arithmetic device 131, the storage device 132, the communication device 133, the input device 134, and the output device 135 may be connected via a data bus 136.

[0129] The characteristics of the arithmetic unit 131, the storage device 132, the communication device 133, the input device 134, and the output device 135 may be the same as the characteristics of the arithmetic unit 31, the storage device 32, the communication device 33, the input device 34, and the output device 35, respectively. The above description of the control device 3 can be used as a description of the robot control device 13 by replacing the terms control device 3, arithmetic unit 31, storage device 32, communication device 33, input device 34, and output device 35 with the terms robot control device 13, arithmetic unit 131, storage device 132, communication device 133, input device 134, and output device 135, respectively. Therefore, to avoid redundant description, the description of the robot control device 13 will be omitted.

[0130] (2) Robot Control Processing Next, the robot control processing performed by the control device 3 will be described. For convenience of explanation, the following description will discuss robot control processing for controlling at least one of the robot 1, the end effector 4, and the robot movable device to perform the processing shown in FIGS. 6A to 6C . That is, for convenience of explanation, the following description will discuss an example in which the control device 3 controls at least one of the robot 1, the end effector 4, and the robot movable device to perform the processing shown in FIGS. 6A to 6C . Specifically, as shown in FIGS. 6A to 6C , the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 performs a holding process to hold a workpiece W (i.e., a first target object OBJ#1, which is an example of a target object OBJ) placed on the placement device T. In this case, as shown in FIG. 6A , the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 approaches the workpiece W. That is, the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 moves toward the workpiece W. Thereafter, as shown in Fig. 6B, the control device 3 may control at least the end effector 4 so that the end effector 4 holds the workpiece W. Thereafter, as shown in Fig. 6C, the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holding the workpiece W moves away from (in other words, moves away from) the placement device T.

[0131] 6C to 6E , after the end effector 4 holds the workpiece W, the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holding the workpiece W performs a release process to release the workpiece W held by the end effector 4. For convenience of explanation, the following description will be given of an example in which the control device 3 controls at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holding the workpiece W performs a release process to release the workpiece W held by the end effector 4 to a jig J, which is an example of a placement device T (i.e., a second target object OBJ#2, which is an example of a target object OBJ). However, the control device 3 may also control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holding the workpiece W performs a release process to release the workpiece W held by the end effector 4 to an arbitrary second target object OBJ#2 other than the jig J (e.g., at least one of the stage of the processing device and the stage of the measuring device described above). In the example shown in FIGS. 6C to 6E , a through hole is formed in the workpiece W, and the jig J has a rod-shaped support rod JB. The control device 3 controls at least one of the robot 1, the end effector 4, and the robot movable device so that the support rod JB of the jig J is inserted into the through hole of the workpiece W to be released by the end effector 4. In this case, as shown in FIG. 6C , the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holding the workpiece W approaches the jig J. In other words, the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holding the workpiece W moves toward the jig J. Thereafter, as shown in FIG. 6D , the control device 3 may control at least the end effector 4 so that the end effector 4 releases the workpiece W to the jig J. Thereafter, as shown in FIG. 6E , the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4, having released the workpiece W, moves away from the jig J.

[0132] However, after the end effector 4 holds the workpiece W, another end effector provided on a robot arm other than the robot arm 12 may hold the workpiece W held by the end effector 4. In this case, the control device 3 may control the end effector 4 so that the end effector 4 releases (i.e., releases) the workpiece W without moving the end effector 4 holding the workpiece W. Thereafter, the other end effector holding the workpiece W may release the workpiece W to the jig J.

[0133] (2-1) Overall Flow of Robot Control Processing First, the overall flow of the robot control processing performed by the control device 3 will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the overall flow of the robot control processing.

[0134] (2-1-1) Holding Control Processing As shown in Fig. 7, the control device 3 (particularly, the arithmetic device 31) controls at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 performs a holding process to hold the workpiece W (step S1). In other words, the control device 3 performs a process (holding control process) to control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holds the workpiece W (step S1). The flow of the holding control process will be described below with reference to Fig. 8. Fig. 8 is a flowchart showing the flow of the holding control process.

[0135] As shown in FIG. 8 , the position and orientation calculation unit 311 included in the control device 3 acquires image data IMG from the imaging system 2 using the communication device 33 (step S11). Specifically, the imaging system 2 captures an image of the workpiece W placed on the mounting device T at a predetermined imaging rate (specifically, captures an image of at least a portion of the workpiece W; the same applies below). For example, the imaging system 2 may capture images of the workpiece W at an imaging rate of tens to hundreds (e.g., 500) times per second. As a result, the imaging system 2 generates image data IMG at a period corresponding to the predetermined imaging rate. For example, the imaging system 2 may generate tens to hundreds (e.g., 500) pieces of image data IMG per second. The position and orientation calculation unit 311 acquires the image data IMG each time the imaging system 2 generates image data IMG. In other words, the position and orientation calculation unit 311 may acquire tens to hundreds (e.g., 500) pieces of image data IMG per second.

[0136] However, the imaging system 2 does not have to periodically capture images of the workpiece W at a desired imaging rate. For example, the imaging system 2 may capture an image of the workpiece W when receiving a control signal from the control device 3 (or another control device different from the control device 3) for controlling the imaging system 2 to capture an image of the workpiece W.

[0137] In addition to the workpiece W on which the end effector 4 performs a predetermined process (in this case, holds it), the imaging device 21 may also capture an image of another object different from the workpiece W (specifically, it may capture an image of at least a portion of the other object; the same applies below). Note that, because the other object is not a target on which the robot 1 performs a predetermined process, the other object different from the workpiece W will be referred to as a non-target object in the following description. An example of a non-target object is at least a portion of the end effector 4. An example of a non-target object is at least a portion of the robot arm 12. An example of a non-target object is at least a portion of a peripheral object, which is an object located around the robot 1. An example of a peripheral object is the placement device T. For example, if both the workpiece W and the non-target object are included in the imaging range (field of view) of the imaging device 21, the imaging device 21 may capture an image of both the workpiece W and the non-target object. As a result, the imaging device 21 may generate image data IMG that shows an image in which both the workpiece W and the non-target object are captured. However, the imaging device 21 may image the workpiece W but not the non-target object. In other words, the imaging device 21 may generate image data IMG showing an image in which the workpiece W is captured but the non-target object is not captured. In either case, the imaging device 21 generates image data IMG showing an image in which at least the workpiece W is captured. In other words, the imaging device 21 generates image data IMG that includes at least image data of the workpiece W.

[0138] When multiple workpieces W are placed on the mounting device T, the imaging system 2 may capture images of at least some of the multiple workpieces W placed on the mounting device T. In other words, the imaging system 2 may capture images of a group of target objects including at least some of the multiple workpieces W placed on the mounting device T. The group of target objects captured by the imaging system 2 may include multiple workpieces W included in the imaging field of view of the imaging system 2 out of the multiple workpieces W placed on the mounting device T (or, in some cases, one workpiece W included in the imaging field of view of the imaging system 2).

[0139] Each time the position and orientation calculation unit 311 acquires image data IMG in step S11, the position and orientation calculation unit 311 calculates at least one of the position and orientation of the workpiece W based on the image data IMG acquired in step S11 (step S12). As a result, the position and orientation calculation unit 311 generates position and orientation data POI that indicates at least one of the position and orientation of the workpiece W.

[0140] In the following description, as described above, an example will be described in which the position and orientation calculation unit 311 calculates at least one of the position and orientation of the workpiece W in the global coordinate system. That is, in the following description, an example will be described in which the position and orientation calculation unit 311 generates position and orientation data POI that indicates at least one of the position and orientation of the workpiece W in the global coordinate system.

[0141] The position and orientation calculation unit 311 may perform a matching process using the image data IMG acquired in step S11 to calculate at least one of the position and orientation of the workpiece W. Specifically, the position and orientation calculation unit 311 may perform a matching process using the image data IMG and a model (template model) that represents at least a part of the workpiece W (for example, that represents the shape of at least a part of the workpiece W) to calculate at least one of the position and orientation of the workpiece W.

[0142] As a first example, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W by performing a contour matching process (in other words, an edge matching process), which is an example of a matching process. Specifically, to perform the contour matching process, the position and orientation calculation unit 311 may use image data IMG generated by the imaging device 21 including a monocular camera. Note that the image data IMG generated by the imaging device 21 including a monocular camera is image data IMG including one image generated by the monocular camera. In the following description, the image data IMG generated by the imaging device 21 including a monocular camera will be referred to as 2D image data IMG_2D as necessary. In this case, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W by performing a contour matching process using the 2D image data IMG_2D (i.e., the image indicated by the 2D image data IMG_2D) and a contour model (in other words, an edge model), which is an example of a template model. The contour model may represent at least a portion of the contour of the workpiece W. In other words, the contour model does not have to represent a portion of the workpiece W that is surrounded by the contour. The contour of the workpiece W may also be referred to as the edge of the workpiece W. In this case, the position and orientation calculation unit 311 may perform, as the contour matching process, an object detection process that detects the workpiece W (in other words, the contour of the workpiece W) by detecting a contour (edge) that corresponds to the contour model within the image indicated by the 2D image data IMG_2D.

[0143] In this case, the position and orientation calculation unit 311 may translate, enlarge, reduce, and / or rotate the contour model within the imaging coordinate system based on the imaging device 21 so that the contour model approaches (e.g., coincides with) the contour of the workpiece W captured in the image represented by the 2D image data IMG_2D. As a result, the position and orientation calculation unit 311 can identify the positional relationship between the coordinate system of the contour model and the imaging coordinate system. The position and orientation calculation unit 311 may then calculate at least one of the position and orientation of the workpiece W in the imaging coordinate system from at least one of the position and orientation of the workpiece W in the coordinate system of the contour model based on the positional relationship between the coordinate system of the contour model and the imaging coordinate system. The position and orientation calculation unit 311 may then convert at least one of the position and orientation of the workpiece W in the imaging coordinate system into at least one of the position and orientation of the workpiece W in the global coordinate system (or the robot coordinate system) based on a transformation matrix for converting three-dimensional coordinates in the imaging coordinate system into three-dimensional coordinates in the global coordinate system (or the robot coordinate system).

[0144] As a second example, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W by performing a 2D matching process, which is an example of a matching process. Specifically, the position and orientation calculation unit 311 may use the 2D image data IMG_2D to perform the 2D matching process. In this case, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W by performing a 2D matching process using the 2D image data IMG_2D (i.e., the image indicated by the 2D image data IMG_2D) and a two-dimensional model, which is an example of a template model. The two-dimensional model may represent the two-dimensional shape of at least a portion of the workpiece W using both at least a portion of the outline of the workpiece W and at least a portion of a portion of the workpiece W surrounded by the outline. In this case, the position and orientation calculation unit 311 may perform, as the 2D matching process, an object detection process to detect the workpiece W indicated by the two-dimensional model (in other words, the image portion corresponding to the two-dimensional model) within the image indicated by the 2D image data IMG_2D. The 2D matching process (in this case, the object detection process) itself may be the same as an existing 2D matching process. For example, the position and orientation calculation unit 311 may perform the 2D matching process using a well-known method such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed-Up Robust Feature).

[0145] In this case, the position and orientation calculation unit 311 may translate, enlarge, reduce, and / or rotate the two-dimensional model within the imaging coordinate system based on the imaging device 21 so that characteristic locations of the two-dimensional model (e.g., at least one of the feature points and edges) approach (e.g., coincide with) characteristic locations of the workpiece W captured in the image represented by the 2D image data IMG_2D. As a result, the position and orientation calculation unit 311 can identify the positional relationship between the coordinate system of the two-dimensional model and the imaging coordinate system. Thereafter, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W in the imaging coordinate system from at least one of the position and orientation of the workpiece W in the coordinate system of the two-dimensional model based on the positional relationship between the coordinate system of the two-dimensional model and the imaging coordinate system. Then, the position and orientation calculation unit 311 may convert at least one of the position and orientation of the workpiece W in the imaging coordinate system to at least one of the position and orientation of the workpiece W in the global coordinate system (or the robot coordinate system) based on a transformation matrix for converting three-dimensional coordinates in the imaging coordinate system into three-dimensional coordinates in the global coordinate system (or the robot coordinate system).

[0146] As a third example, if the imaging device 21 includes a stereo camera, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W by performing a 3D matching process, which is an example of a matching process. Specifically, the position and orientation calculation unit 311 may use image data IMG generated by the imaging device 21 including a stereo camera to perform the 3D matching process. Note that the image data IMG generated by the imaging device 21 including a stereo camera is image data IMG including two images generated by the stereo camera. In the following description, the image data IMG generated by the imaging device 21 including a stereo camera will be referred to as 3D image data IMG_3D as necessary. In this case, the position and orientation calculation unit 311 may generate three-dimensional position data indicating the three-dimensional position of at least a portion of the workpiece W based on the 3D image data IMG_3D (i.e., the two images indicated by the 3D image data IMG_3D). In this case, the position and orientation calculation unit 311 may calculate the parallax based on the 3D image data IMG_3D (i.e., the two image data indicated by the 3D image data IMG_3D) and generate three-dimensional position data using the calculated parallax using a well-known method based on the principle of triangulation. The three-dimensional position data may be data indicating the three-dimensional position of each of multiple points on the workpiece W. The three-dimensional position data may be data indicating the three-dimensional position of each of multiple points on the surface of the workpiece W. The three-dimensional position data may be data indicating the three-dimensional position of each of multiple points corresponding to multiple locations on the surface of the workpiece W. In the following description, an example will be described in which point cloud data indicating a point cloud is used as the three-dimensional position data. However, three-dimensional position data other than point cloud data indicating a point cloud (e.g., depth image data indicating a depth image) may also be used. Thereafter, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W by performing a 3D matching process using the three-dimensional position data (e.g., point cloud data) and a three-dimensional model, which is an example of a template model. The three-dimensional model may represent the three-dimensional shape of at least a portion of the workpiece W using both at least a portion of the outline of the workpiece W and at least a portion of the area of ​​the workpiece W surrounded by the outline.In this case, the position and orientation calculation unit 311 may perform, as the 3D matching process, an object detection process that detects a workpiece W indicated by the 3D model (in other words, a set of points corresponding to the 3D model) within a point cloud indicated by the 3D position data. Note that the 3D matching process (in this case, the object detection process) itself may be the same as an existing 3D matching process. For example, the position and orientation calculation unit 311 may perform the 3D matching process using a well-known method including at least one of RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), ICP (Iterative Closest Point), and DSO (Direct Sparse Odometry).

[0147] Note that even if the imaging device 21 is an imaging device 21 that includes a monocular camera (in other words, an imaging device 21 that does not include a stereo camera), when the imaging device 21 captures an image of a workpiece W onto which a projection pattern is projected by the lighting device 23, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W by performing 3D matching processing. This is because the position and orientation calculation unit 311 can calculate a parallax that can be used to generate three-dimensional position data based on the projection pattern (specifically, the projection pattern deformed according to the shape of the workpiece W) that appears in the image indicated by the 2D image data IMG_2D.

[0148] When 3D matching processing is performed, the position and orientation calculation unit 311 may translate, enlarge, reduce, and / or rotate the three-dimensional model within the imaging coordinate system so that characteristic locations of the three-dimensional model (e.g., at least one of a characteristic point and an edge) approach (e.g., coincide with) characteristic locations of the workpiece W whose three-dimensional position data indicates the three-dimensional position (e.g., a point cloud corresponding to the workpiece W indicated by the three-dimensional position data). As a result, the position and orientation calculation unit 311 can identify the positional relationship between the coordinate system of the three-dimensional model and the imaging coordinate system. Thereafter, even when 3D matching processing is performed, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W in the global coordinate system (or the robot coordinate system) by performing processing similar to that when 2D matching processing is performed.

[0149] The position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W by performing at least two of a contour matching process, a 2D matching process, and a 3D matching process. For example, the position and orientation calculation unit 311 may calculate at least a portion of the position and orientation of the workpiece W by performing at least one of a contour matching process, a 2D matching process, and a 3D matching process, and calculate at least another portion of the position and orientation of the workpiece W by performing at least another of the contour matching process, the 2D matching process, and the 3D matching process. For example, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W by merging at least two of the calculation results of at least one of the position and orientation of the workpiece W by the contour matching process, the calculation results of at least one of the position and orientation of the workpiece W by the 2D matching process, and the calculation results of at least one of the position and orientation of the workpiece W by the 3D matching process. For example, the position and orientation calculation unit 311 may calculate an average value of at least two of the calculation results of at least one of the position and orientation of the workpiece W by the contour matching process, the calculation results of at least one of the position and orientation of the workpiece W by the 2D matching process, and the calculation results of at least one of the position and orientation of the workpiece W by the 3D matching process, and use the calculated average value as at least one of the positions and orientations of the workpiece W. For example, the position and orientation calculation unit 311 may calculate a median value of at least two of the calculation results of at least one of the position and orientation of the workpiece W by the contour matching process, the calculation results of at least one of the position and orientation of the workpiece W by the 2D matching process, and the calculation results of at least one of the position and orientation of the workpiece W by the 3D matching process, and use the calculated median value as at least one of the positions and orientations of the workpiece W.

[0150] The position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W based on the image data IMG using an existing method other than matching processing. For example, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W based on the image data IMG using an inference model generated by machine learning. The inference model may be generated by machine learning so that, when the image data IMG is input, at least one of the position and orientation of the workpiece W reflected in the image indicated by the image data IMG is output.

[0151] By performing the matching process, the position and orientation calculation unit 311 may calculate, as the position of the workpiece W in the global coordinate system (or the robot coordinate system), at least one of the following: a position Tx of the workpiece W in an X-axis direction parallel to the X-axis of the global coordinate system (or the robot coordinate system), a position Ty of the workpiece W in a Y-axis direction parallel to the Y-axis of the global coordinate system (or the robot coordinate system), and a position Tz of the workpiece W in a Z-axis direction parallel to the Z-axis of the global coordinate system (or the robot coordinate system). By performing the matching process, the position and orientation calculation unit 311 may calculate, as the orientation of the workpiece W in the global coordinate system (or the robot coordinate system), at least one of the rotation amount Rx of the workpiece W about the X-axis of the global coordinate system, the rotation amount Ry of the workpiece W about the Y-axis of the global coordinate system (or the robot coordinate system), and the rotation amount Rz of the workpiece W about the Z-axis of the global coordinate system (or the robot coordinate system). This is because the amount of rotation Rx of the workpiece W around the X-axis, the amount of rotation Ry of the workpiece W around the Y-axis, and the amount of rotation Rz of the workpiece W around the Z-axis are equivalent to parameters representing the orientation of the workpiece W around the X-axis, the parameters representing the orientation of the workpiece W around the Y-axis, and the parameters representing the orientation of the workpiece W around the Z-axis, respectively. For this reason, in the following description, the amount of rotation Rx of the workpiece W around the X-axis, the amount of rotation Ry of the workpiece W around the Y-axis, and the amount of rotation Rz of the workpiece W around the Z-axis will be referred to as the orientation Rx of the workpiece W around the X-axis, the orientation Ry of the workpiece W around the Y-axis, and the orientation Rz of the workpiece W around the Z-axis, respectively.

[0152] The orientation Rx of the workpiece W around the X-axis, the orientation Ry of the workpiece W around the Y-axis, and the orientation Rz of the workpiece W around the Z-axis may be considered to indicate the position of the workpiece W in the rotational direction around the X-axis, the position of the workpiece W in the rotational direction around the Y-axis, and the position of the workpiece W in the rotational direction around the Z-axis, respectively. In other words, the orientation Rx of the workpiece W around the X-axis, the orientation Ry of the workpiece W around the Y-axis, and the orientation Rz of the workpiece W around the Z-axis may all be considered to be parameters that represent the position of the workpiece W.

[0153] In step S12, the position and orientation calculation unit 311 may calculate all of the position Tx, position Ty, position Tz, orientation Rx, orientation Ry, and orientation Rz. That is, the position and orientation calculation unit 311 may generate position and orientation data POI indicating all of the position Tx, position Ty, position Tz, orientation Rx, orientation Ry, and orientation Rz. Alternatively, the position and orientation calculation unit 311 may calculate some of the position Tx, position Ty, position Tz, orientation Rx, orientation Ry, and orientation Rz. That is, the position and orientation calculation unit 311 may generate position and orientation data POI indicating some of the position Tx, position Ty, position Tz, orientation Rx, orientation Ry, and orientation Rz. As an example, the position and orientation calculation unit 311 may calculate the position Tx, position Ty, and orientation Rz, but not calculate the position Tz, orientation Rx, and orientation Ry. In other words, the position and orientation calculation unit 311 may generate position and orientation data POI that indicates the position Tx, the position Ty, and the orientation Rz, but that does not necessarily indicate the position Tz, the orientation Rx, and the orientation Ry.

[0154] When performing the matching process, the position and orientation calculation unit 311 may calculate a matching similarity, which is the similarity between the template model and the detected workpiece W. In this case, in step S12 of FIG. 8 , if a workpiece W whose calculated matching similarity exceeds a predetermined matching determination threshold used in the matching process is detected by the matching process, the position and orientation calculation unit 311 may select (in other words, determine) the workpiece W as a processing execution object on which the end effector 4 should actually perform a predetermined processing. On the other hand, if a workpiece W whose calculated matching similarity is below the matching determination threshold is detected by the matching process, the position and orientation calculation unit 311 may not select the workpiece W as a processing execution object on which the end effector 4 should actually perform a predetermined processing.

[0155] The matching determination threshold is a threshold used to detect the workpiece W from the image represented by the image data IMG through the matching process. Specifically, the matching determination threshold is a threshold used to distinguish, from the matching similarity of the object detected through the matching process, a state in which the object detected through the matching process is highly likely to be the workpiece W and a state in which the object detected through the matching process is unlikely to be the workpiece W. The matching determination threshold can also be said to be a threshold used to distinguish, from the matching similarity, a state in which the accuracy of at least one of the position and orientation of the workpiece W calculated through the matching process is high and a state in which the accuracy of at least one of the position and orientation of the workpiece W calculated through the matching process is low.

[0156] As described above, the imaging system 2 may capture images of multiple workpieces W placed on the mounting device T. In this case, multiple workpieces W may appear in the image represented by the image data IMG generated by the imaging system 2. In this case, the position and orientation calculation unit 311 may select (in other words, determine) one of the multiple workpieces W appearing in the image represented by the image data IMG as the processing object to be actually held by the end effector 4. For example, the position and orientation calculation unit 311 may select one of the multiple workpieces W appearing in the image represented by the image data IMG as the processing object based on the matching similarity. As an example, the position and orientation calculation unit 311 may select one of the multiple workpieces W corresponding to a matching similarity that exceeds the matching determination threshold as the processing object. As another example, the position and orientation calculation unit 311 may select one of the multiple workpieces W corresponding to the highest matching similarity that exceeds the matching determination threshold as the processing object. As another example, the position and orientation calculation unit 311 may select, as the processing object, one workpiece W among the multiple workpieces W that exceeds the matching judgment threshold and corresponds to the Nth highest matching similarity (where N is a constant indicating an integer greater than or equal to 2). As another example, the position and orientation calculation unit 311 may select, as the processing object, one workpiece W among the multiple workpieces W that corresponds to a matching similarity that exceeds the matching judgment threshold and is closest to the end effector 4. As another example, when multiple workpieces W are placed on the placement device T (e.g., when multiple objects are stacked in bulk), the position and orientation calculation unit 311 may select, as the processing object, one workpiece W among the multiple workpieces W that corresponds to a matching similarity that exceeds the matching judgment threshold and has the largest Z coordinate along the Z axis (located at the highest position). As another example, the position and orientation calculation unit 311 may select, as the processing object, one workpiece W among the multiple workpieces W that corresponds to a matching similarity that exceeds the matching judgment threshold and on which the end effector 4 can perform a predetermined process.

[0157] Thereafter, the signal generation unit 312 included in the control device 3 generates a robot control signal for controlling at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holds the workpiece W selected as the processing execution object, based on the position and orientation data POI generated in step S12 (step S13). Thereafter, the signal generation unit 312 outputs the robot control signal generated in step S13 to the robot control device 13 using the communication device 33. As a result, the robot control device 13 controls at least one of the robot 1, the end effector 4, and the robot movable device based on the robot control signal.

[0158] For example, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the robot movable device so that the end effector 4 moves toward a position where the end effector 4 can hold the workpiece W located at the position and / or having the posture calculated in step S12. For example, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the robot movable device so that the end effector 4 takes a posture where the end effector 4 can hold the workpiece W located at the position and / or having the posture calculated in step S12.

[0159] The holding position and posture, which indicates at least one of the position and posture for the end effector 4 to hold the workpiece W, may be registered in advance. Note that the holding position and posture may indicate at least one of the position and posture of the end effector 4 at the time when the end effector 4 holds the workpiece W. The holding position and posture may indicate at least one of the position and posture of the end effector 4 with respect to the workpiece W at the time when the end effector 4 holds the workpiece W. The holding position and posture may indicate at least one of the position and posture of the workpiece W with respect to the end effector 4 at the time when the end effector 4 holds the workpiece W. The holding position and posture may indicate at least one of the positional relationship and posture relationship between the end effector 4 and the workpiece at the time when the end effector 4 holds the workpiece W.

[0160] A plurality of different holding positions and postures may be registered in advance. In this case, the signal generating unit 312 may select one of the plurality of holding positions and postures, and generate a robot control signal for moving the end effector 4 so that the end effector 4 can hold the workpiece W at the selected holding position and posture. As a result, when at least one of the positions and postures of the workpiece W placed on the placement device T varies, the control device 3 can generate a robot control signal so that the end effector 4 can hold the workpiece W by selecting one holding position and posture corresponding to at least one of the positions and postures of the workpiece W to be held by the end effector 4.

[0161] As a result, as shown in FIG. 6A above, the end effector 4 approaches the workpiece W in the global coordinate system (or the robot coordinate system). Thereafter, the signal generating unit 312 may generate a robot control signal for controlling at least the end effector 4 so that the end effector 4 approaching the workpiece W holds the workpiece W. As a result, as shown in FIG. 6B above, the end effector 4 holds the workpiece W. Furthermore, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the robot movable device so that the end effector 4 holding the workpiece W moves away from (in other words, moves away from) the placement device T. As a result, as shown in FIG. 6C above, the end effector 4 holding the workpiece W moves away from (in other words, moves away from) the placement device T.

[0162] During a period in which at least one of the robot 1, the end effector 4, and the robot movable device is controlled based on the robot control signal (e.g., a period in which the end effector 4 is approaching the workpiece W), the imaging system 2 captures an image of the workpiece W (particularly, the processing execution object), and the position and orientation calculation unit 311 reacquires image data IMG generated by the imaging system 2 re-capturing the image of the workpiece W (particularly, the processing execution object) (step S11 in FIG. 8 ). The position and orientation calculation unit 311 may regenerate position and orientation data POI based on the re-acquired image data IMG (step S12 in FIG. 8 ). In other words, the position and orientation calculation unit 311 may update at least one of the position and orientation of the workpiece W (particularly, the processing execution object). In other words, the position and orientation calculation unit 311 may update the position and orientation data POI. Thereafter, the signal generation unit 312 may regenerate a robot control signal based on the updated position and orientation data POI (step S13 in FIG. 8 ). That is, during a period in which at least one of the robot 1, the end effector 4, and the robot movable device is controlled based on the robot control signal (e.g., during a period in which the end effector 4 is approaching the workpiece W), the signal generator 312 may repeatedly generate the robot control signal (i.e., may repeatedly generate the movement path of the end effector 4). In other words, during a period in which at least one of the robot 1, the end effector 4, and the robot movable device is controlled based on the robot control signal (e.g., during a period in which the end effector 4 is approaching the workpiece W), the signal generator 312 may update the robot control signal (i.e., may update the movement path of the end effector 4). That is, during a period in which at least one of the robot 1, the end effector 4, and the robot movable device is controlled based on the robot control signal (e.g., during a period in which the end effector 4 is approaching the workpiece W), the control device 3 may repeat the processes of steps S11 to S13 in FIG. 8 .

[0163] In particular, when the workpiece W moves during a period in which at least one of the robot 1, the end effector 4, and the robot movable device is controlled based on the robot control signal (e.g., a period in which the end effector 4 is approaching the workpiece W), the control device 3 may repeat the processes of steps S11 to S13 in FIG. 8 . In this case, the imaging system 2 may repeatedly capture images of the moving workpiece W. As a result, when the workpiece W moves during a period in which the end effector 4 is moving, the position and orientation calculation unit 311 can appropriately update the position and orientation data POI of the workpiece W so that the movement of the workpiece W is reflected in the position and orientation data POI. As a result, the signal generation unit 312 can generate a robot control signal based on the updated position and orientation data POI. Therefore, the signal generation unit 312 can control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holds the moving workpiece W.

[0164] (2-1-2) Workpiece Position and Orientation Change Processing Referring back to Fig. 7, after the holding control processing has been performed, the control device 3 (particularly the arithmetic device 31) performs a workpiece position and orientation change processing to change at least one of the position and orientation of the workpiece W held by the end effector 4 (step S2). The flow of the workpiece position and orientation change processing will be described below with reference to Fig. 9. Fig. 9 is a flowchart showing the flow of the workpiece position and orientation change processing.

[0165] 9 , after the end effector 4 holds the workpiece W, the imaging device 21 captures an image of at least a portion of the workpiece W held by the end effector 4 (step S21). As a result, the position and orientation calculation unit 311 uses the communication device 33 to acquire image data IMG generated by the imaging device 21 capturing an image of at least a portion of the workpiece W held by the end effector 4 (step S21).

[0166] Thereafter, the position and orientation calculation unit 311 calculates at least one of the position and orientation of the workpiece W held by the end effector 4 based on the image data IMG acquired in step S21 (step S22). Note that the process of calculating at least one of the position and orientation of the workpiece W based on the image data IMG in the workpiece position and orientation change process may be the same as the process of calculating at least one of the position and orientation of the workpiece W based on the image data IMG in the above-mentioned hold control process (step S12 in FIG. 8). Therefore, a detailed description of step S22 will be omitted.

[0167] Then, the signal generating unit 312 determines whether or not at least one of the position and posture of the workpiece W held by the end effector 4 should be changed based on the calculation results of at least one of the position and posture of the workpiece W in step S22 (step S23).

[0168] For example, if the actual position of the workpiece W calculated in step S22 differs from the desired target workpiece position where the workpiece W held by the end effector 4 should be located, the signal generating unit 312 may determine that at least one of the position and the orientation of the workpiece W held by the end effector 4 should be changed. On the other hand, if the actual position of the workpiece W calculated in step S22 is the same as the desired target workpiece position where the workpiece W held by the end effector 4 should be located, the signal generating unit 312 may determine that at least one of the position and the orientation of the workpiece W held by the end effector 4 does not need to be changed.

[0169] A state in which the actual position of the workpiece W differs from the desired target workpiece position may mean a state in which the actual position (relative position) of the workpiece W held by the end effector 4 relative to the end effector 4 holding the workpiece W differs from the desired target workpiece position (target value of the relative position) at which the workpiece W held by the end effector 4 should be located relative to the end effector 4 holding the workpiece W. In other words, a state in which the actual position of the workpiece W differs from the desired target workpiece position may mean a state in which the actual positional relationship between the end effector 4 holding the workpiece W and the workpiece W held by the end effector 4 differs from the target value of the positional relationship between the end effector 4 holding the workpiece W and the workpiece W held by the end effector 4 (i.e., the positional relationship corresponding to the desired target workpiece position). On the other hand, the state in which the actual position of the workpiece W is the same as the desired target workpiece position may also mean a state in which the actual position (relative position) of the workpiece W held by the end effector 4 relative to the end effector 4 holding the workpiece W is the same as the desired target workpiece position (target value of the relative position) at which the workpiece W held by the end effector 4 should be located relative to the end effector 4 holding the workpiece W. In other words, the state in which the actual position of the workpiece W is the same as the desired target workpiece position may also mean a state in which the actual positional relationship between the end effector 4 holding the workpiece W and the workpiece W held by the end effector 4 is the same as the target positional relationship between the end effector 4 holding the workpiece W and the workpiece W held by the end effector 4 (i.e., the positional relationship corresponding to the desired target workpiece position).

[0170] The desired target work position may include a position of the workpiece W that satisfies the condition that the end effector 4 can properly release the workpiece W held by it to the jig J. Specifically, the desired target work position may include a position of the workpiece W that satisfies the condition that the end effector 4 can properly release the workpiece W held by it to the jig J when the workpiece W held by it is located at the desired target work position. The desired target work position may include a position of the workpiece W that satisfies the condition that the end effector 4 cannot properly release the workpiece W held by it to the jig J when the workpiece W held by the end effector 4 is located at a position different from the desired target work position.

[0171] The desired target workpiece position may include the position of the workpiece W that is determined based on a holding position and posture that indicates at least one of the position and posture for the end effector 4 to hold the workpiece W. For example, as described above, the control device 3 controls the end effector 4 to hold the workpiece W at a holding position and posture that is registered in advance. Therefore, the desired target workpiece position may include the position of the workpiece W when the end effector 4 holds the workpiece W at the holding position and posture that is registered in advance.

[0172] In addition, when a plurality of holding positions and postures are registered as described above, a plurality of target workpiece positions corresponding to the plurality of holding positions and postures may be used. In this case, the signal generating unit 312 may determine that at least one of the position and posture of the workpiece W held by the end effector 4 should be changed when the actual position of the workpiece W calculated in step S22 differs from one target workpiece position corresponding to one holding position and posture selected as at least one of the position and posture for the end effector 4 to hold the workpiece W in the holding control process.

[0173] For example, if the actual posture of the workpiece W calculated in step S22 differs from the desired target workpiece posture that the workpiece W held by the end effector 4 should assume, the signal generating unit 312 may determine that at least one of the position and posture of the workpiece W held by the end effector 4 should be changed. On the other hand, if the actual posture of the workpiece W calculated in step S22 is the same as the desired target workpiece posture that the workpiece W held by the end effector 4 should assume, the signal generating unit 312 may determine that at least one of the position and posture of the workpiece W held by the end effector 4 does not need to be changed.

[0174] A state in which the actual posture of the workpiece W differs from the desired target workpiece posture may mean a state in which the actual posture (relative posture) of the workpiece W held by the end effector 4 relative to the end effector 4 holding the workpiece W differs from the desired target workpiece posture (target value of the relative posture) that the workpiece W held by the end effector 4 should assume relative to the end effector 4 holding the workpiece W. In other words, a state in which the actual posture of the workpiece W differs from the desired target workpiece posture may mean a state in which the actual posture relationship between the end effector 4 holding the workpiece W and the workpiece W held by the end effector 4 differs from the target value of the posture relationship between the end effector 4 holding the workpiece W and the workpiece W held by the end effector 4 (i.e., the posture relationship corresponding to the desired target workpiece posture). On the other hand, the state in which the actual posture of the workpiece W is the same as the desired target workpiece posture may also mean a state in which the actual posture (relative posture) of the workpiece W held by the end effector 4 relative to the end effector 4 holding the workpiece W is the same as the desired target workpiece posture (target value of the relative posture) that the workpiece W held by the end effector 4 should assume relative to the end effector 4 holding the workpiece W. In other words, the state in which the actual posture of the workpiece W is the same as the desired target workpiece posture may also mean a state in which the actual posture relationship between the end effector 4 holding the workpiece W and the workpiece W held by the end effector 4 is the same as the target value of the posture relationship between the end effector 4 holding the workpiece W and the workpiece W held by the end effector 4 (i.e., the posture relationship corresponding to the desired target workpiece posture).

[0175] The desired target workpiece orientation may include an orientation of the workpiece W that satisfies the condition that the end effector 4 can appropriately release the workpiece W held by it to the jig J. Specifically, the desired target workpiece orientation may include an orientation of the workpiece W that satisfies the condition that the end effector 4 can appropriately release the workpiece W held by it to the jig J when the workpiece W held by it assumes the desired target workpiece orientation. The desired target workpiece orientation may include an orientation of the workpiece W that satisfies the condition that the end effector 4 cannot appropriately release the workpiece W held by it to the jig J when the workpiece W held by it assumes an orientation different from the desired target workpiece orientation.

[0176] The desired target workpiece orientation may include an orientation of the workpiece W that is determined based on a holding position and orientation that indicates at least one of the position and orientation for the end effector 4 to hold the workpiece W. For example, as described above, the control device 3 controls the end effector 4 to hold the workpiece W at a holding position and orientation that is registered in advance. Therefore, the desired target workpiece orientation may include the orientation of the workpiece W when the end effector 4 holds the workpiece W at the holding position and orientation that is registered in advance.

[0177] In addition, when a plurality of holding positions and postures are registered as described above, a plurality of target workpiece postures corresponding to the plurality of holding positions and postures may be used. In this case, the signal generating unit 312 may determine that at least one of the position and posture of the workpiece W held by the end effector 4 should be changed when the actual posture of the workpiece W calculated in step S22 differs from one target workpiece posture corresponding to one holding position and posture selected as at least one of the position and posture for the end effector 4 to hold the workpiece W in the holding control process.

[0178] As an example, Fig. 10A shows the ideal positional relationship and ideal posture relationship between the end effector 4, the workpiece W, and the jig J when the end effector 4 releases the workpiece W held by the end effector 4 to the jig J. In other words, Fig. 10A shows the positional relationship and posture relationship between the end effector 4, the workpiece W, and the jig J under conditions in which the end effector 4 can appropriately release the workpiece W held by the end effector 4 to the jig J. In other words, Fig. 10A shows the desired target workpiece position at which the workpiece W held by the end effector 4 should be located, and the desired target workpiece posture that the workpiece W held by the end effector 4 should take.

[0179] 10A , if the actual position of the workpiece W calculated in step S22 is the same as the position of the workpiece W shown in Fig. 10A and the actual orientation of the workpiece W calculated in step S22 is the same as the orientation of the workpiece W shown in Fig. 10A , it may be determined that there is no need to change at least one of the position and orientation of the workpiece W held by the end effector 4. In the example shown in Fig. 10A , at least one of the target workpiece position and the target workpiece orientation satisfies the conditions that (i) the end effector 4 holds a desired portion of the workpiece W so that the end effector 4 does not get in the way when the support rod JB of the jig J is inserted into the through-hole formed in the workpiece W, and (ii) the end effector 4 holds the workpiece W in a desired orientation so that the depression in the workpiece W faces upward (in other words, toward the end effector 4).

[0180] On the other hand, as shown in FIG. 10B , if the actual position of the workpiece W calculated in step S22 differs from the position of the workpiece W shown in FIG. 10A and / or the actual orientation of the workpiece W calculated in step S22 differs from the orientation of the workpiece W shown in FIG. 10A , it may be determined that at least one of the position and orientation of the workpiece W held by the end effector 4 should be changed. In the example shown in FIG. 10B , the end effector 4 holds the workpiece W so as to block at least a portion of the through-hole of the workpiece W. Therefore, when the support rod JB of the jig J is inserted into the through-hole formed in the workpiece W, the end effector 4 gets in the way. Therefore, in this case, the signal generating unit 312 may determine that the actual position of the workpiece W (e.g., its position relative to the end effector 4) differs from the target workpiece position. Also, in the example shown in FIG. 10B , the depression in the workpiece W faces downward (i.e., away from the end effector 4). Therefore, in this case, the signal generating unit 312 may determine that the actual posture of the workpiece W (for example, the posture relative to the end effector 4) is different from the target workpiece posture.

[0181] 9 , if it is determined in step S23 that at least one of the position and posture of the workpiece W held by the end effector 4 does not need to be changed (step S23: No), the control device 3 may terminate the workpiece position and posture change process shown in FIG. 9 . In other words, the control device 3 does not need to control at least one of the robot 1, the robot movable device, the end effector 4, and the position and posture change device 5 so as to transfer the workpiece W held by the end effector 4 to the position and posture change device 5. In this case, the control device 3 may perform a release control process, which will be described later, without the workpiece W held by the end effector 4 passing through the position and posture change device 5. In other words, the control device 3 may generate a robot control signal to control at least one of the robot 1, the robot movable device, and the end effector 4 so as to perform a release process to release the workpiece W held by the end effector 4 without the workpiece W passing through the position and posture change device 5.

[0182] The control device 3 may generate a robot control signal to control at least one of the robot 1, the robot movable device, and the end effector 4 so that a predetermined process other than a release process is performed on the workpiece W held by the end effector 4 without the workpiece W being moved by the position and posture changing device 5. For example, the control device 3 may generate a robot control signal to control at least one of the robot 1, the robot movable device, and the end effector 4 so that the workpiece W held by the end effector 4 is moved by the workpiece W being moved by the end effector 4 without the workpiece W being moved by the position and posture changing device 5.

[0183] On the other hand, if it is determined in step S23 that at least one of the position and the orientation of the workpiece W held by the end effector 4 should be changed (step S23: Yes), the control device 3 changes at least one of the position and the orientation of the workpiece W held by the end effector 4 (step S24). For example, the control device 3 changes at least one of the position and the orientation of the workpiece W held by the end effector 4 based on the calculation results of at least one of the position and the orientation of the workpiece W in step S22 (step S24).

[0184] In order to change at least one of the position and posture of the workpiece W held by the end effector 4, the control device 3 first controls at least one of the robot 1, the end effector 4, the robot movable device, and the position and posture change device 5 so that the end effector 4 passes the workpiece W held by the end effector 4 to the position and posture change device 5 (e.g., the holding member 53) (step S241).

[0185] 11A , the signal generator 312 may generate a robot control signal that controls at least one of the robot 1 and the robot movable device so that the end effector 4 holding the workpiece W moves toward the position and orientation changing device 5 (e.g., the holding member 53). For example, the signal generator 312 may generate a robot control signal that controls at least one of the robot 1 and the robot movable device so that the end effector 4 holding the workpiece W moves toward the position and orientation changing device 5 (e.g., the holding member 53). In particular, the signal generator 312 may generate a robot control signal that controls at least one of the robot 1 and the robot movable device so that the end effector 4 holding the workpiece W moves toward a transfer position where the end effector 4 should be located to transfer the workpiece W held by the end effector 4 to the position and orientation changing device 5 (e.g., the holding member 53). That is, the signal generating unit 312 may generate a robot control signal that controls at least one of the robot 1 and the robot movable device so that the end effector 4 holding the workpiece W moves toward a position for transferring the workpiece W held by the end effector 4 to the position and posture changing device 5 (e.g., the holding member 53). In other words, the signal generating unit 312 may generate a robot control signal that controls at least one of the robot 1 and the robot movable device so that the workpiece W held by the end effector 4 moves toward the position and posture changing device 5 (e.g., the holding member 53). In particular, the signal generating unit 312 may generate a robot control signal that controls at least one of the robot 1 and the robot movable device so that the workpiece W held by the end effector 4 moves toward a first receiving position where the position and posture changing device 5 (e.g., the holding member 53) receives the workpiece W held by the end effector 4. In other words, the signal generating unit 312 may generate a robot control signal that controls at least one of the robot 1 and the robot movable device so that the workpiece W held by the end effector 4 moves toward a position where the position / posture changing device 5 (e.g., the holding member 53) can receive the workpiece W held by the end effector 4.

[0186] The signal generating unit 312 may generate a robot control signal to move the end effector 4 holding the workpiece W so that, when the end effector 4 holding the workpiece W is located at the delivery position, the positional relationship between the workpiece W held by the end effector 4 and the holding member 53 that receives the workpiece W from the end effector 4 matches a target value (desired first target positional relationship) for the positional relationship between the workpiece W and the holding member 53 when the holding member 53 receives the workpiece W from the end effector 4. Furthermore, the signal generating unit 312 may generate a robot control signal to move the end effector 4 holding the workpiece W so that, when the end effector 4 holding the workpiece W is located at the delivery position, the posture relationship between the workpiece W held by the end effector 4 and the holding member 53 that receives the workpiece W from the end effector 4 matches a target value (desired first target posture relationship) for the posture relationship between the workpiece W and the holding member 53 when the holding member 53 receives the workpiece W from the end effector 4. In this case, the signal generating unit 312 may generate a robot control signal for moving the end effector 4 holding the workpiece W based on at least one of the calculation results of the position and posture of the workpiece W held by the end effector 4 in step S22 so that the positional relationship between the workpiece W located at the position calculated in step S22 and the holding member 53 matches the first target positional relationship, and / or so that the posture relationship between the workpiece W assuming the posture calculated in step S22 and the holding member 53 matches the first target posture relationship.

[0187] Note that information regarding at least one of the position and orientation of the position and orientation changing device 5 itself may be known to the control device 3. In other words, the position and orientation changing device 5 may be disposed at a known position or in a known orientation. Furthermore, because the holding member 53 included in the position and orientation changing device 5 operates under the control of the control device 3, information regarding at least one of the position and orientation of the holding member 53 is also known to the control device 3. In this case, the signal generation unit 312 may generate a robot control signal for moving the end effector 4 holding the workpiece W based on the calculation result of at least one of the position and orientation of the workpiece W held by the end effector 4 in step S22 and known information regarding at least one of the positions and orientations of the position and orientation changing device 5 and the holding member 53, so that the positional relationship between the workpiece W located at the position calculated in step S22 and the holding member 53 located at the known position coincides with the first target positional relationship and / or the orientation relationship between the workpiece W assuming the orientation calculated in step S22 and the holding member 53 assuming the known orientation coincides with the first target orientation relationship. However, as will be explained later in the device position and orientation calculation process shown in Figure 18, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of at least one of the position and orientation of the position and orientation change device 5 and the holding member 53 by performing an operation similar to the process of calculating at least one of the position and orientation of the workpiece W.

[0188] 11B , the signal generation unit 312 may generate a robot control signal that controls at least the end effector 4 so that the end effector 4 located at the delivery position passes the workpiece W held by the end effector 4 (i.e., the workpiece W located at the first receiving position) to the position and posture change device 5. Furthermore, as shown in FIG. 11B , the signal generation unit 312 may generate a position and posture control signal that controls the position and posture change device 5 so that the end effector 4 receives the workpiece W held by the end effector 4 from the end effector 4. As a result, the end effector 4 releases the workpiece W, and the position and posture change device 5 newly holds the workpiece W.

[0189] Referring again to FIG. 9, the control device 3 then controls the position and posture change device 5 so that the position and posture change device 5 changes at least one of the position and posture of the workpiece W held by the position and posture change device 5 (step S242).

[0190] 11C , the signal generating unit 312 may generate a position and orientation control signal for controlling the position and orientation changing device 5 so as to change at least one of the position and orientation of the workpiece W held by the position and orientation changing device 5. As a result, at least one of the position and orientation of the workpiece W held by the position and orientation changing device 5 is changed. Note that Fig. 11C shows an example in which the position and orientation changing device 5 changes the orientation of the workpiece W held by the position and orientation changing device 5 by rotating the workpiece W held by the position and orientation changing device 5 (specifically, by rotating the holding member 53 that holds the workpiece W).

[0191] The signal generating unit 312 may generate a position and orientation control signal for changing at least one of the position and orientation of the workpiece W held by the position and orientation changing device 5 so that, when the end effector 4 again holds the workpiece W whose position and / or orientation has been changed by the position and orientation changing device 5, the actual position of the workpiece W again held by the end effector 4 becomes the same as the desired target workpiece position and the actual orientation of the workpiece W again held by the end effector 4 becomes the same as the desired target workpiece orientation. In other words, the signal generating unit 312 may generate a position and orientation control signal for changing at least one of the position and orientation of the workpiece W held by the position and orientation changing device 5 so that, when the end effector 4 again holds the workpiece W whose position and / or orientation has been changed by the position and orientation changing device 5, the actual position of the workpiece W becomes the same as the desired target workpiece position and the actual orientation of the workpiece W becomes the same as the desired target workpiece orientation, allowing the end effector 4 to receive the workpiece W from the position and orientation changing device 5.

[0192] 9 again, thereafter, the control device 3 controls at least one of the robot 1, the end effector 4, the robot movable device, and the position and orientation change device 5 so that the end effector 4 receives the workpiece W held by the position and orientation change device from the position and orientation change device 5 (e.g., the holding member 53) (step S243). In other words, the control device 3 controls at least one of the robot 1, the end effector 4, the robot movable device, and the position and orientation change device 5 so that the position and orientation change device 5 passes the workpiece W held by the position and orientation change device to the end effector 4 (step S243).

[0193] 11D , the signal generating unit 312 may generate a robot control signal that controls at least one of the robot 1 and the robot movable device so that the end effector 4, which has handed over the workpiece W to the position and orientation change device 5, moves toward a second receiving position where the end effector 4 should be located in order to receive the workpiece W held by the position and orientation change device 5 from the position and orientation change device 5. The signal generating unit 312 may generate a robot control signal that controls at least one of the robot 1 and the robot movable device so that the end effector 4, which has handed over the workpiece W to the position and orientation change device 5, moves toward a position where the end effector 4 will receive the workpiece W held by the position and orientation change device 5 from the position and orientation change device 5.

[0194] The signal generating unit 312 may generate a robot control signal to move the end effector 4 so that, when the end effector 4 is located at the second receiving position, the positional relationship between the workpiece W held by the holding member 53 and the end effector 4 receiving the workpiece W from the holding member 53 matches a target value (desired second target positional relationship) for the positional relationship between the workpiece W and the end effector 4 when the end effector 4 receives the workpiece W from the holding member 53. Furthermore, the signal generating unit 312 may generate a robot control signal to move the end effector 4 so that, when the end effector 4 is located at the second receiving position, the posture relationship between the workpiece W held by the holding member 53 and the end effector 4 receiving the workpiece W from the holding member 53 matches a target value (desired second target posture relationship) for the posture relationship between the workpiece W and the end effector 4 when the end effector 4 receives the workpiece W from the holding member 53.

[0195] Note that, because the end effector 4 moves under the control of the control device 3, information regarding at least one of the position and orientation of the end effector 4 is known to the control device 3. Furthermore, as described above, because the holding member 53 included in the position and orientation changing device 5 operates under the control of the control device 3, information regarding at least one of the position and orientation of the holding member 53 is also known to the control device 3. Therefore, information regarding at least one of the position and orientation of the workpiece W held by the holding member 53 is also known to the control device 3. In this case, the signal generating unit 312 may generate a robot control signal for moving the end effector 4 based on the known information regarding at least one of the position and orientation of the workpiece W held by the holding member 53 and the known information regarding at least one of the position and orientation of the end effector 4 so that the positional relationship between the workpiece W located at a known position and the end effector 4 located at a known position coincides with the second target positional relationship and / or so that the orientation relationship between the workpiece W assuming a known orientation and the end effector 4 assuming a known orientation coincides with the second target orientation relationship. 15, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the end effector 4 by performing an operation similar to the process of calculating at least one of the position and orientation of the workpiece W. As will be described later in the workpiece position and orientation calculation process shown in FIG. 20, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W held by the holding member 53 by performing an operation similar to the process of calculating at least one of the position and orientation of the workpiece W.

[0196] 11E, the signal generation unit 312 may generate a robot control signal that controls at least the end effector 4 so that the end effector 4 located at the second receiving position receives the workpiece W held by the position and orientation change device 5 from the position and orientation change device 5. Furthermore, as shown in Fig. 11E, the signal generation unit 312 may generate a position and orientation control signal that controls the position and orientation change device 5 so that the position and orientation change device 5 passes the workpiece W held by it to the end effector 4. As a result, the position and orientation change device 5 releases the workpiece W, and the end effector 4 newly holds the workpiece W.

[0197] 11A and 11E , at least one of the position and orientation of the workpiece W held by the end effector 4 is changed. In other words, at least one of the position and orientation of the workpiece W held by the end effector 4 after the position and orientation changing device 5 changes at least one of the position and orientation of the workpiece W (see FIG. 11E ) is changed relative to at least one of the position and orientation of the workpiece W held by the end effector 4 before the position and orientation changing device 5 changed at least one of the position and orientation of the workpiece W (see FIG. 11A ). In this manner, in this embodiment, the process of changing at least one of the position and orientation of the workpiece W held by the end effector 4 may mean a process of changing at least one of the position and orientation of the workpiece W so that at least one of the position and orientation of the workpiece W held by the end effector 4 after the position and orientation changing device 5 changes at least one of the position and orientation of the workpiece W is different from at least one of the position and orientation of the workpiece W held by the end effector 4 before the position and orientation changing device 5 changed at least one of the position and orientation of the workpiece W.

[0198] As described above, in this embodiment, the robot system SYS can use the position and posture changing device 5 to change at least one of the position and posture of the workpiece W held by the end effector 4. Therefore, even if the actual position of the workpiece W held by the end effector 4 differs from the desired target workpiece position, the position and posture changing device 5 can change at least one of the position and posture of the workpiece W so that the actual position of the workpiece W held by the end effector 4 becomes the same as the desired target workpiece position. Similarly, even if the actual posture of the workpiece W held by the end effector 4 differs from the desired target workpiece posture, the position and posture changing device 5 can change at least one of the position and posture of the workpiece W so that the actual posture of the workpiece W held by the end effector 4 becomes the same as the desired target workpiece posture.

[0199] Furthermore, in this embodiment, before the workpiece W held by the end effector 4 is transferred to the position and orientation changing device 5, the imaging system 2 captures an image of the workpiece W held by the end effector 4 to generate image data IMG, and at least one of a robot control signal and a position and orientation control signal is generated based on the image data IMG. Specifically, based on the image data IMG, it is determined whether or not to change at least one of the position and orientation of the workpiece W held by the end effector 4, and at least one of a robot control signal and a position and orientation control signal is generated based on the determination result of whether or not to change at least one of the position and orientation of the workpiece W held by the end effector 4. Therefore, compared to a case where the workpiece W held by the end effector 4 is unconditionally transferred to the position and orientation changing device 5 regardless of at least one of the position and orientation of the workpiece W held by the end effector 4, the workpiece W held by the end effector 4 is not transferred to the position and orientation changing device 5 even when at least one of the position and orientation of the workpiece W held by the end effector 4 does not need to be changed. This reduces the time required to perform the workpiece position and orientation changing process. That is, the throughput of the workpiece position / orientation change process (that is, the throughput of the robot control process including the workpiece position / orientation change process) is improved.

[0200] Furthermore, in this embodiment, because the imaging system 2 is attached to the movable robot 1 (particularly the robot arm 12), the same imaging system 2 can image the workpiece W placed on the mounting device T to generate image data IMG for performing the holding control processing, and can also image the workpiece W held by the end effector 4 to generate image data IMG for performing the workpiece position and orientation change processing. If the imaging system 2 were attached to a fixed support device (e.g., a structure such as a tower) different from the robot 1, the imaging system 2 would not be movable, and the robot system SYS would likely have to include a first imaging system 2 for imaging the workpiece W placed on the mounting device T to generate image data IMG for performing the holding control processing, and a second imaging system 2 for imaging the workpiece W held by the end effector 4 to generate image data IMG for performing the workpiece position and orientation change processing. Therefore, in this embodiment, the number of imaging systems 2 required is reduced, thereby reducing the cost of the robot system SYS.

[0201] 9 , the position and orientation calculation unit 311 calculates at least one of the position and orientation of the workpiece W held by the end effector 4 based on the image data IMG generated by the imaging system 2 capturing an image of the workpiece W held by the end effector 4 after the end effector 4 holds the workpiece W. However, in step S22 of FIG. 9 , the position and orientation calculation unit 311 may estimate at least one of the position and orientation of the workpiece W held by the end effector 4 based on the image data IMG generated by the imaging system 2 capturing an image of the workpiece W to be held by the end effector 4 (i.e., the workpiece W selected as the processing execution object) before the end effector 4 holds the workpiece W. Specifically, it is highly likely that at least one of the position and orientation of the workpiece W held by the end effector 4 depends on at least one of the position and orientation of the workpiece W before the end effector 4 holds the workpiece W. For example, when the end effector 4 holds a workpiece W placed on the mounting device T in a first orientation, the end effector 4 may be able to hold the workpiece W in a target workpiece orientation. However, when the end effector 4 holds a workpiece W placed on the mounting device T in a second orientation that is significantly different from the first orientation, the end effector 4 may only be able to hold the workpiece W in an orientation different from the target workpiece orientation. Therefore, if at least one of the position and orientation of the workpiece W before the end effector 4 holds the workpiece W is known, it may be possible to estimate at least one of the position and orientation of the workpiece W held by the end effector 4. Therefore, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W placed on the mounting device T based on image data IMG generated by the imaging system 2 capturing an image of the workpiece W placed on the mounting device T before the end effector 4 holds the workpiece W (i.e., the workpiece W selected as the processing execution object).The image data IMG used to estimate at least one of the position and orientation of the workpiece W held by the end effector 4 may be the same as the image data IMG used for the above-described hold control process (i.e., the image data IMG acquired in step S11 of FIG. 8 ), or may be image data IMG acquired separately from the image data IMG used for the above-described hold control process. Thereafter, based on the calculation results of at least one of the position and orientation of the workpiece W placed on the mounting device T, at least one of the position and orientation of the workpiece W held by the end effector 4 when this workpiece W is held by the end effector 4 may be estimated. Thereafter, in step S23 of FIG. 9 , the signal generation unit 312 may determine whether or not at least one of the position and orientation of the workpiece W held by the end effector 4 should be changed, based on the estimation results of at least one of the position and orientation of the workpiece W in step S22.

[0202] Furthermore, the end effector 4 that has handed over the workpiece W to the position and orientation change device 5 may receive the workpiece W held by the position and orientation change device 5 from the position and orientation change device 5. In other words, the end effector 4 that hands over the workpiece W to the position and orientation change device 5 and the end effector 4 that receives the workpiece W held by the position and orientation change device 5 from the position and orientation change device 5 may be the same. In this case, as described above, even if the actual position of the workpiece W held by the end effector 4 differs from the desired target workpiece position before the position and orientation change device 5 changes at least one of the position and orientation of the workpiece W, the end effector 4 can re-hold the workpiece W via the position and orientation change device 5 so that the actual position of the workpiece W becomes the same as the desired target workpiece position. Similarly, even if the actual orientation of the workpiece W held by the end effector 4 differs from the desired target workpiece orientation before the position and orientation change device 5 changes at least one of the position and orientation of the workpiece W, the end effector 4 can re-hold the workpiece W via the position and orientation change device 5 so that the actual orientation of the workpiece W becomes the same as the desired target workpiece orientation.

[0203] Alternatively, an end effector 4 different from the end effector 4 that handed over the workpiece W to the position and orientation changing device 5 may receive the workpiece W held by the position and orientation changing device 5 from the position and orientation changing device 5. In other words, the end effector 4 that hands over the workpiece W to the position and orientation changing device 5 may be different from the end effector 4 that receives the workpiece W held by the position and orientation changing device 5 from the position and orientation changing device 5. As an example, a first end effector 4 attached to the first robot 1 may hand over the workpiece W held by the first end effector 4 to the position and orientation changing device 5. Thereafter, a second end effector 4 different from the first end effector 4 attached to a second robot different from the first robot 1 (or attached to the first robot 1) may receive the workpiece W held by the position and orientation changing device 5 from the position and orientation changing device 5. In this case, even if the actual position of the workpiece W held by the first end effector 4 differs from the desired target workpiece position before the position and orientation changing device 5 changes at least one of the position and orientation of the workpiece W, the second end effector 4 can hold the workpiece W via the position and orientation changing device 5 so that the actual position of the workpiece W becomes the same as the desired target workpiece position. Similarly, even if the actual orientation of the workpiece W held by the first end effector 4 differs from the desired target workpiece orientation before the position and orientation changing device 5 changes at least one of the position and orientation of the workpiece W, the second end effector 4 can re-hold the workpiece W via the position and orientation changing device 5 so that the actual orientation of the workpiece W becomes the same as the desired target workpiece orientation.

[0204] Alternatively, if the end effector 4 that passes the workpiece W to the position and orientation changing device 5 is different from the end effector 4 that receives the workpiece W held by the position and orientation changing device 5 from the position and orientation changing device 5, the position and orientation changing device 5 may change at least one of the position and orientation of the workpiece W held by the position and orientation changing device 5 so as to pass the workpiece W received from the first end effector 4 to the second end effector 4. For example, the position and orientation changing device 5 may change the position of the workpiece W held by the position and orientation changing device 5 so as to move the workpiece W received from the first end effector 4 toward the second end effector 4. In this case, the position and orientation changing device 5 may be considered to function as a transport device that transports the workpiece W from the first end effector 4 to the second end effector 4.

[0205] (2-1-3) Release Control Processing Referring again to FIG. 7, after the workpiece position and orientation change processing has been performed, the control device 3 (particularly the arithmetic device 31) controls at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 performs a release processing to release the workpiece W held by the end effector 4 (step S3). In other words, the control device 3 performs a process (release control processing) to control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 releases the workpiece W held by the end effector 4 (step S3). The flow of the release control processing will be explained below with reference to FIG. 12. FIG. 12 is a flowchart showing the flow of the release control processing.

[0206] 12 , after the end effector 4 holds the workpiece W, the position and orientation calculation unit 311 included in the control device 3 acquires image data IMG from the imaging system 2 using the communication device 33 (step S31). The processing of step S31 in FIG. 12 may differ from the processing of step S11 in FIG. 8 in that the imaging system 2 captures an image of the workpiece W held by the end effector 4. Other features of the processing of step S31 in FIG. 12 may be the same as other features of the processing of step S11 in FIG. 8. Therefore, in order to avoid redundant description, a detailed description of step S31 in FIG. 12 will be omitted.

[0207] Each time the position and orientation calculation unit 311 acquires image data IMG in step S31, the position and orientation calculation unit 311 calculates at least one of the position and orientation of the workpiece W held by the end effector 4 based on the image data IMG acquired in step S31 (step S32). As a result, the position and orientation calculation unit 311 generates position and orientation data POI that indicates at least one of the position and orientation of the workpiece W held by the end effector 4. Note that the process of calculating at least one of the position and orientation of the workpiece W in step S32 in Fig. 12 may be the same as the process of calculating at least one of the position and orientation of the workpiece W in step S12 in Fig. 8. Therefore, in order to avoid redundant explanation, a detailed description of step S32 in Fig. 12 will be omitted.

[0208] However, because the end effector 4 holds the workpiece W under the control of the control device 3, there is a possibility that at least one of the position and orientation of the workpiece W held by the end effector 4 is already known to the control device 3. Therefore, if at least one of the position and orientation of the workpiece W held by the end effector 4 is already known to the control device 3, the control device 3 does not need to perform the processes from step S31 to step S32 described above.

[0209] Thereafter, the signal generation unit 312 generates a robot control signal for controlling at least one of the robot 1, the end effector 4, and the robot movable device, based on the position and orientation data POI generated in step S32, so as to release the workpiece W held by the end effector 4 to the jig J (step S33). Thereafter, the signal generation unit 312 outputs the robot control signal generated in step S33 to the robot control device 13, using the communication device 33. As a result, the robot control device 13 controls at least one of the robot 1, the end effector 4, and the robot movable device, based on the robot control signal.

[0210] For example, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the robot movable device so that the end effector 4 moves toward a position where the end effector 4 can release the workpiece W, which is located at the position calculated in step S32 and / or has the posture calculated in step S32, to the jig J. For example, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the robot movable device so that the end effector 4 takes a posture where the end effector 4 can release the workpiece W, which is located at the position calculated in step S32 and / or has the posture calculated in step S32, to the jig J.

[0211] Note that information regarding at least one of the position and orientation of the jig J may be information known to the control device 3. In other words, the jig J may be placed at a known position or in a known orientation. However, as will be described later in the jig position and orientation calculation process shown in FIG. 22 , the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the jig J by performing an operation similar to the process of calculating at least one of the position and orientation of the workpiece W.

[0212] As a result, as shown in FIG. 6C above, the end effector 4 holding the workpiece W approaches the jig J in the global coordinate system (or the robot coordinate system). Thereafter, the signal generator 312 may generate a robot control signal for controlling at least the end effector 4 so that the end effector 4 approaching the jig J releases the workpiece W. As a result, as shown in FIG. 6D above, the end effector 4 releases the workpiece W to the jig J. Furthermore, the signal generator 312 may generate a robot control signal for controlling at least one of the robot 1 and the robot movable device so that the end effector 4 having released the workpiece W moves away from (in other words, moves away from) the jig J. As a result, as shown in FIG. 6E above, the end effector 4 holding the workpiece W moves away from (in other words, moves away from) the jig J.

[0213] Here, even though at least one of the position and orientation of the workpiece W held by the end effector 4 has been changed by the position and orientation changing device 5, there is a possibility that the position of the workpiece W re-held by the end effector 4 may differ from the desired target workpiece position. In this case, since at least one of the position and orientation of the workpiece W held by the end effector 4 is calculated in step S32, the signal generating unit 312 may generate a robot control signal taking into account the deviation between the actual position of the workpiece W and the desired target workpiece position. For example, even if the actual position of the workpiece W differs from the desired target workpiece position, the signal generating unit 312 may generate a robot control signal so that the end effector 4 can accurately release the workpiece W to the jig J in the same way as when the actual position of the workpiece W is the same as the desired target workpiece position. As an example, when the actual position of the workpiece W differs from the desired target workpiece position, the signal generating unit 312 may generate a robot control signal by correcting the movement amount of the end effector 4 when the actual position of the workpiece W and the desired target workpiece position are the same, so as to offset the deviation between the actual position of the workpiece W and the desired target workpiece position. Therefore, even if at least one of the position and orientation of the workpiece W held by the end effector 4 has been changed by the position and orientation changing device 5, and the position of the workpiece W re-held by the end effector 4 differs from the desired target workpiece position, the end effector 4 can accurately release the workpiece W held by the end effector 4 to the jig J. However, if at least one of the position and orientation of the workpiece W held by the end effector 4 has been changed by the position and orientation changing device 5, and the position of the workpiece W re-held by the end effector 4 differs from the desired target workpiece position, the control device 3 may perform the above-described workpiece position and orientation changing process again.

[0214] Similarly, even though at least one of the position and orientation of the workpiece W held by the end effector 4 has been changed by the position and orientation changing device 5, the orientation of the workpiece W re-held by the end effector 4 may differ from the desired target workpiece orientation. In this case, because at least one of the position and orientation of the workpiece W held by the end effector 4 is calculated in step S32, the signal generating unit 312 may generate a robot control signal after taking into consideration the deviation between the actual orientation of the workpiece W and the desired target workpiece orientation. For example, even when the actual orientation of the workpiece W differs from the desired target workpiece orientation, the signal generating unit 312 may generate a robot control signal so that the end effector 4 can accurately release the workpiece W to the jig J in the same way as when the actual orientation of the workpiece W is the same as the desired target workpiece orientation. As an example, when the actual orientation of the workpiece W differs from the desired target workpiece orientation, the signal generating unit 312 may generate a robot control signal by correcting the movement amount of the end effector 4 when the actual orientation of the workpiece W and the desired target workpiece orientation are the same, so as to offset the deviation between the actual orientation of the workpiece W and the desired target workpiece orientation. Therefore, even if at least one of the position and orientation of the workpiece W held by the end effector 4 has been changed by the position and orientation changing device 5, and the orientation of the workpiece W re-held by the end effector 4 differs from the desired target workpiece orientation, the end effector 4 can accurately release the workpiece W held by the end effector 4 to the jig J. However, if at least one of the position and orientation of the workpiece W held by the end effector 4 has been changed by the position and orientation changing device 5, and the orientation of the workpiece W re-held by the end effector 4 differs from the desired target workpiece orientation, the control device 3 may perform the above-described workpiece position and orientation changing process again.

[0215] During a period in which at least one of the robot 1, the end effector 4, and the robot movable device is controlled based on the robot control signal (e.g., during a period in which the end effector 4 holding the workpiece W is approaching the jig J), the imaging system 2 may again capture an image of the workpiece W held by the end effector 4, and the position and orientation calculation unit 311 may again acquire image data IMG generated by the imaging system 2 again capturing an image of the workpiece W held by the end effector 4 (step S31 in FIG. 12 ). The position and orientation calculation unit 311 may again generate position and orientation data POI based on the reacquired image data IMG (step S32 in FIG. 12 ). In other words, the position and orientation calculation unit 311 may update at least one of the position and orientation of the workpiece W held by the end effector 4. In other words, the position and orientation calculation unit 311 may update the position and orientation data POI. Thereafter, the signal generation unit 312 may again generate a robot control signal based on the updated position and orientation data POI (step S33 in FIG. 12 ). That is, during a period in which at least one of the robot 1, the end effector 4, and the robot movable device is controlled based on the robot control signal (for example, a period in which the end effector 4 holding the workpiece W is approaching the jig J), the signal generator 312 may repeatedly generate the robot control signal (that is, may repeatedly generate the movement path of the end effector 4). In other words, during a period in which at least one of the robot 1, the end effector 4, and the robot movable device is controlled based on the robot control signal (for example, a period in which the end effector 4 holding the workpiece W is approaching the jig J), the signal generator 312 may update the robot control signal (that is, may update the movement path of the end effector 4). That is, during a period in which at least one of the robot 1, the end effector 4, and the robot movable device is controlled based on the robot control signal (for example, a period in which the end effector 4 holding the workpiece W is approaching the jig J), the control device 3 may repeat the processes of steps S31 to S33 in FIG. 12 .As a result, when the workpiece W held by the end effector 4 moves in accordance with the movement of the end effector 4, the position and orientation calculation unit 311 can appropriately update the position and orientation data POI of the workpiece W so that the movement of the workpiece W held by the end effector 4 is reflected in the position and orientation data POI. As a result, the signal generation unit 312 can generate a robot control signal based on the updated position and orientation data POI. Therefore, the signal generation unit 312 can control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 releases the workpiece W that moves in accordance with the movement of the end effector 4.

[0216] 7 , the control device 3 may thereafter repeat the holding control process (step S1), the workpiece position / posture change process (step S2), and the release control process (step S3) (step S4) until it is determined that the robot control process is to be ended. For example, the control device 3 may repeat the process of controlling at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holds one workpiece W among multiple workpieces W (e.g., multiple workpieces W placed on the placement device T) and releases the held workpiece W. In this case, each time the end effector 4 holds one workpiece W, the control device 3 may control the position / posture change device 5 to change at least one of the position and posture of the workpiece W held by the end effector 4, as necessary.

[0217] As an example, the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holds a first workpiece W among the multiple workpieces W. Thereafter, as necessary, the control device 3 may control the position and orientation changing device 5 so that the end effector 4 changes at least one of the position and orientation of the first workpiece W held by the end effector 4. Thereafter, the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 releases the first workpiece W held by the end effector 4. Thereafter, the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holds a second workpiece W different from the first workpiece W among the multiple workpieces W. Thereafter, as necessary, the control device 3 may control the position and orientation changing device 5 so that the end effector 4 changes at least one of the position and orientation of the second workpiece W held by the end effector 4. Thereafter, the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 releases the second workpiece W held by the end effector 4.

[0218] However, the control device 3 does not have to repeat the holding control process (step S1), the workpiece position / posture change process (step S2), and the release control process (step S3). For example, the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holds a single workpiece W (e.g., a single workpiece W placed on the placement device T). Thereafter, the control device 3 may control the position / posture change device 5 so that the end effector 4 changes at least one of the position and posture of the workpiece W held by the end effector 4. Thereafter, the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 releases the workpiece W held by the end effector 4. Thereafter, the control device 3 may end the robot control process.

[0219] (2-2) Workpiece Position and Orientation Change Processing Including Workpiece Holding Determination Processing In the robot control processing described above, the workpiece position and orientation change processing is performed after the holding control processing. That is, after the holding control processing is performed, the control device 3 calculates at least one of the position and orientation of the workpiece W held by the end effector 4, determines whether to change at least one of the position and orientation of the workpiece W held by the end effector 4, controls at least one of the robot 1, the robot movable device, the end effector 4, and the position and orientation change device 5 so as to transfer the workpiece W held by the end effector 4 from the end effector 4 to the position and orientation change device 5, controls the position and orientation change device 5 so as to change at least one of the position and orientation of the workpiece W held by the position and orientation change device 5, and controls at least one of the robot 1, the robot movable device, the end effector 4, and the position and orientation change device 5 so as to transfer the workpiece W held by the position and orientation change device 5 from the position and orientation change device 5 to the end effector 4.

[0220] However, in some cases, even when the holding control process is performed, there is a possibility that the end effector 4 is not actually holding the workpiece W. In other words, even when the holding control process is performed, there is a possibility that the end effector 4 fails to hold the workpiece W. In this way, when the end effector 4 is not holding the workpiece W despite the holding control process being performed, there is little need to perform the workpiece position and posture change process in the first place.

[0221] Therefore, the control device 3 may perform workpiece position and posture change processing including workpiece holding determination processing as an example of workpiece position and posture change processing. The workpiece holding determination processing may include processing for determining whether or not the end effector 4 is actually holding the workpiece W after the holding control processing in step S1 of FIG. 6 has been performed. Hereinafter, the workpiece position and posture change processing including the workpiece holding determination processing will be described with reference to FIG. 13. FIG. 13 is a flowchart showing the flow of the workpiece position and posture change processing including the workpiece holding determination processing. In the following description, processing that has already been described will be assigned the same step numbers, and detailed description thereof will be omitted.

[0222] 13 , even when a workpiece position / posture change process including a workpiece holding determination process is performed, the image capturing device 21 captures an image of at least a part of the workpiece W held by the end effector 4 after the holding control process is performed (step S21), just as when a workpiece position / posture change process not including a workpiece holding determination process is performed. As a result, the position / posture calculation unit 311 acquires image data IMG (step S21).

[0223] However, as described above, even if the holding control process has been performed, there is a possibility that the end effector 4 is not actually holding the workpiece W. Therefore, at the time when the imaging device 21 images at least a portion of the workpiece W held by the end effector 4 in step S21, there is a possibility that the end effector 4 is not holding the workpiece W. Therefore, in step S21, the imaging device 21 may at least image an area where the workpiece W is expected to exist if the end effector 4 holds the workpiece W (hereinafter referred to as the workpiece holding area WHA). When the end effector 4 is actually holding the workpiece W, the imaging device 21 that images the workpiece holding area WHA may image the workpiece holding area WHA where at least a portion of the workpiece W exists. In other words, the imaging device 21 may image at least a portion of the workpiece W held by the end effector 4 (i.e., at least a portion of the workpiece W included in the workpiece holding area WHA). On the other hand, when the end effector 4 is not actually holding a workpiece W, the imaging device 21 that images the workpiece holding area WHA may image the workpiece holding area WHA in which no workpiece W is present. In other words, the imaging device 21 may image the workpiece holding area WHA without imaging the workpiece W.

[0224] In addition to or instead of the workpiece W held by the end effector 4, an object other than the workpiece W held by the end effector 4 may be present in the workpiece holding area WHA. In this case, in step S21, the imaging device 21 may image the object other than the workpiece W held by the end effector 4 by imaging the workpiece holding area WHA. For example, as shown in FIG. 14 , the workpiece holding area WHA may include at least a portion of the end effector 4. In this case, in step S21, the imaging device 21 may image at least a portion of the end effector 4 by imaging the workpiece holding area WHA.

[0225] An example of the workpiece holding area WHA is shown in Fig. 14. As shown in Fig. 14, the workpiece holding area WHA may include an area of ​​the end effector 4 where a holding member 41 that actually holds the workpiece W is located. In the example shown in Fig. 14, the end effector 4 is a magnetic gripper, and the holding member 41 is a member that includes a magnetic source that is arranged at the tip of the end effector 4 so as to actually apply a magnetic force to the workpiece W.

[0226] The workpiece holding area WHA may include an area where the reference point of the end effector 4 is located. An example of the reference point of the end effector 4 is the tool center point (TCP) of the end effector 4. The area where the holding member 41 of the end effector 4 is located and the area where the tool center point (i.e., the reference point) of the end effector 4 are located may at least partially overlap. For example, if the tool center point is set in the holding member 41, the area where the holding member 41 of the end effector 4 is located and the area where the tool center point (i.e., the reference point) of the end effector 4 are located may at least partially overlap. The reference point of the end effector 4 may be considered as the reference point of the robot 1. The reference point of the end effector 4 (e.g., the tool center point) may be considered as the reference point of the robot 1 (e.g., the tool center point).

[0227] Because both the end effector 4 and the imaging device 21 are attached to the robot arm 12, the positional relationship between the end effector 4 and the imaging device 21 is fixed. In this case, the imaging device 21 may be aligned with the end effector 4 (in other words, with respect to the robot arm 12 to which the end effector 4 is attached) so that the imaging range of the imaging device 21 includes the workpiece holding area WHA. If the end effector 4 attached to the robot arm 12 is replaceable, the imaging device 21 may be aligned with the robot arm 12 to which each of the multiple end effectors 4 is attached so that the imaging range of the imaging device 21 includes multiple workpiece holding areas WHA corresponding to the multiple end effectors 4, respectively.

[0228] 13 , the signal generation unit 312 determines whether the end effector 4 is holding the workpiece W based on the image data IMG acquired in step S21 (step S41). As an example, the position and orientation calculation unit 311 may perform the above-described matching process based on the image data IMG acquired in step S21, and the signal generation unit 312 may determine whether the end effector 4 is holding the workpiece W based on the results of the matching process. For example, if the matching process does not detect an object (i.e., the workpiece W) corresponding to a matching similarity exceeding the matching judgment threshold within the workpiece holding area WHA, the signal generation unit 312 may determine that the end effector 4 is not holding the workpiece W. For example, if the matching process detects an object (i.e., the workpiece W) corresponding to a matching similarity exceeding the matching judgment threshold within the workpiece holding area WHA, the signal generation unit 312 may determine that the end effector 4 is holding the workpiece W. For example, if the matching process does not calculate at least one of the position and orientation of the workpiece W, the signal generation unit 312 may determine that the end effector 4 is not holding the workpiece W. For example, when at least one of the position and posture of the workpiece W present within the workpiece holding area WHA is calculated by the matching process, the signal generating unit 312 may determine that the end effector 4 is holding the workpiece W.

[0229] In this way, the control device 3 determines whether or not the end effector 4 is holding the workpiece W based on the image data IMG acquired in step S21. Therefore, the workpiece holding determination process for determining whether or not the end effector 4 is holding the workpiece W may be a process including the processes of step S21 and step S41 in FIG.

[0230] If the result of the determination in step S41 indicates that the end effector 4 is holding a workpiece W (step S41: Yes), the control device 3 may continue the workpiece position and orientation change process. That is, the control device 3 may perform the processes from step S22 to step S24. Specifically, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W held by the end effector 4 based on the image data IMG acquired in step S21 (step S22). Thereafter, the signal generation unit 312 may determine whether or not at least one of the position and orientation of the workpiece W held by the end effector 4 should be changed based on the calculation result of at least one of the position and orientation of the workpiece W in step S22 (step S23). Thereafter, if it is determined that at least one of the position and orientation of the workpiece W held by the end effector 4 should be changed (step S23: Yes), the control device 3 may change at least one of the position and orientation of the workpiece W held by the end effector 4 (step S24).

[0231] In the above description, in step S23, the signal generating unit 312 determines whether or not at least one of the position and orientation of the workpiece W held by the end effector 4 should be changed based on the image data IMG acquired in step S21 (specifically, based on at least one of the position and orientation of the workpiece W held by the end effector 4 calculated based on the image data IMG acquired in step S21). In other words, the image data IMG used in step S23 to determine whether or not at least one of the position and orientation of the workpiece W held by the end effector 4 should be changed is the same as the image data IMG used in step S41 to determine whether or not the end effector 4 is holding a workpiece W. However, the image data IMG used in step S23 to determine whether or not at least one of the position and orientation of the workpiece W held by the end effector 4 should be changed may be different from the image data IMG used in step S41 to determine whether or not the end effector 4 is holding a workpiece W. For example, in step S23, the signal generating unit 312 may determine whether or not to change at least one of the position and the posture of the workpiece W held by the end effector 4, based on the image data IMG generated by the imaging system 2 capturing an image of the workpiece W at a first timing. For example, in step S41, the signal generating unit 312 may determine whether or not the end effector 4 is holding the workpiece W, based on the image data IMG generated by the imaging system 2 capturing an image of the workpiece holding area WHA at a second timing different from the first timing.As an example, in step S23, the signal generating unit 312 determines whether or not at least one of the position and posture of the workpiece W held by the end effector 4 should be changed based on image data IMG generated by the imaging system 2 capturing an image of the workpiece W held by the end effector 4 at a first timing after the holding control process has been performed, and in step S41, determines whether or not the end effector 4 is holding the workpiece W based on image data IMG generated by the imaging system 2 capturing an image of the workpiece holding area WHA at a second timing after the holding control process has been performed. As another example, in step S23, the signal generating unit 312 may determine whether or not at least one of the position and posture of the workpiece W held by the end effector 4 should be changed based on image data IMG generated by the imaging system 2 capturing an image of the workpiece W that the end effector 4 is about to hold at a first timing before the holding control process is performed, and in step S41, determine whether or not the end effector 4 is holding the workpiece W based on image data IMG generated by the imaging system 2 capturing an image of the workpiece holding area WHA at a second timing after the holding control process is performed.

[0232] On the other hand, if it is determined in step S41 that the end effector 4 is not holding the workpiece W (step S41: No), the control device 3 may end the workpiece position / posture change process and start the hold control process (step S1 in FIG. 7). In other words, the control device 3 may generate a robot control signal so that the end effector 4 holds the workpiece W.

[0233] In this way, when the workpiece position and orientation change processing including the workpiece holding determination processing is performed, the control device 3 generates at least one of a robot control signal and a position and orientation control signal based on the image data IMG generated by the imaging system 2 capturing an image of the workpiece holding area WHA. Specifically, the control device 3 determines whether the end effector 4 is holding a workpiece W based on the image data IMG generated by the imaging system 2 capturing an image of the workpiece holding area WHA. Furthermore, the control device 3 generates at least one of a robot control signal and a position and orientation control signal based on the determination result of whether the end effector 4 is holding a workpiece W. Specifically, when the end effector 4 is holding a workpiece W, the control device 3 generates at least one of a robot control signal and a position and orientation control signal to perform the workpiece position and orientation change processing. On the other hand, when the end effector 4 is not holding a workpiece W, the control device 3 generates a robot control signal to terminate the workpiece position and orientation change processing and then perform the holding control processing. As a result, the control device 3 can terminate the workpiece position and orientation change processing in a situation where there is little need to perform the workpiece position and orientation change processing because the end effector 4 is not holding a workpiece W despite the holding control processing being performed. Therefore, compared to a case where the process of transferring the workpiece W from the end effector 4 to the position / posture changing device 5 is always performed even when the end effector 4 is not holding the workpiece W without determining whether the end effector 4 actually holds the workpiece W, the control device 3 does not need to perform unnecessary processes. This reduces the time required to perform the workpiece position / posture changing process. In other words, the throughput of the workpiece position / posture changing process (i.e., the throughput of the robot control process including the workpiece position / posture changing process) is improved.

[0234] Furthermore, in this embodiment, because the imaging system 2 is attached to the movable robot 1 (particularly the robot arm 12), the same imaging system 2 can image the workpiece W placed on the mounting device T to generate image data IMG for performing the holding control process, and can also image the workpiece holding area WHA to generate image data IMG for performing the workpiece holding determination process. In other words, the same imaging system 2 can simultaneously or separately generate the image data IMG for performing the holding control process and the image data IMG for performing the workpiece holding determination process. If the imaging system 2 were attached to a fixed support device different from the robot 1, because the imaging system 2 is not movable, the robot system SYS may need to separately include a first imaging system 2 for imaging the workpiece W placed on the mounting device T to generate image data IMG for performing the holding control process, and a second imaging system 2 for imaging the workpiece holding area WHA to generate image data IMG for performing the workpiece holding determination process. Therefore, in this embodiment, the number of required imaging systems 2 is reduced, and the cost of the robot system SYS can be reduced.

[0235] In addition, in step S21 of Figure 13, the imaging system 2 generates one piece of image data IMG by imaging the work holding area WHA once, and in step S41 of Figure 13, the control device 3 may determine whether the end effector 4 is holding the work W based on the one piece of image data IMG generated by imaging the work holding area WHA once.

[0236] Alternatively, in step S21 of Fig. 13 , the imaging system 2 may generate multiple pieces of image data IMG by imaging the workpiece holding area WHA multiple times, and in step S41 of Fig. 13 , the control device 3 may determine whether the end effector 4 is holding a workpiece W based on the multiple pieces of image data IMG generated by imaging the workpiece holding area WHA multiple times. In this case, the imaging system 2 may generate the multiple pieces of image data IMG by imaging the workpiece holding area WHA periodically or at random timing, and in step S41 of Fig. 13 , the control device 3 may continuously determine whether the end effector 4 is holding a workpiece W based on the multiple pieces of image data IMG generated by imaging the workpiece holding area WHA multiple times.

[0237] Furthermore, not only when the holding control process is performed so that the end effector 4 holds the workpiece W placed on the mounting device T, but also when the workpiece position and orientation change process is performed so that the end effector 4 holds (i.e., receives) the workpiece W held by the position and orientation change device 5, there is a possibility that the end effector 4 will fail to hold the workpiece W. For this reason, in the workpiece position and orientation change process, after the process for the end effector 4 to hold (i.e., receive) the workpiece W held by the position and orientation change device 5 (specifically, step S24 in FIG. 9 ) is performed, the control device 3 may perform a workpiece holding determination process. In other words, after the process for the end effector 4 to hold (i.e., receive) the workpiece W held by the position and orientation change device 5 (specifically, step S24 in FIG. 9 ) is performed, the imaging system 2 may capture an image of the workpiece holding area WHA, and the signal generation unit 312 may determine whether the end effector 4 is holding the workpiece W. If it is determined that the end effector 4 is holding the workpiece W, the control device 3 may terminate the workpiece position and orientation change process and then perform the release control process. On the other hand, if it is determined that the end effector 4 is not holding the workpiece W, there is a high possibility that the position and orientation change device 5 is still holding the workpiece W, and therefore the control device 3 may again perform the process (specifically, step S24 in FIG. 9 ) for the end effector 4 to receive (i.e., hold) the workpiece W held by the position and orientation change device 5.

[0238] (2-3) Workpiece Position and Posture Change Processing Including Retry Processing In the robot control processing described above, after the holding control processing is performed, a workpiece position and posture change processing is performed, which includes processing for transferring the workpiece W held by the end effector 4 from the end effector 4 to the position and posture change device 5. However, in some cases, the workpiece W held by the end effector 4 may be unstable. Note that a state in which the workpiece W held by the end effector 4 is unstable may include a state in which the workpiece W held by the end effector 4 may fall from the end effector 4. A state in which the workpiece W held by the end effector 4 is unstable may include a state in which the workpiece W held by the end effector 4 may move unintentionally relative to the end effector 4. A state in which the workpiece W held by the end effector 4 is unstable may include a state in which the workpiece W held by the end effector 4 may be unintentionally displaced relative to the end effector 4. A state in which the workpiece W held by the end effector 4 is unstable may include a state in which the workpiece W held by the end effector 4 may unintentionally sway relative to the end effector 4.

[0239] For example, even if the end effector 4 holds the workpiece W through the holding control process, the portion of the workpiece W held by the end effector 4 may be different from the portion of the workpiece W that the end effector 4 is supposed to hold, which may cause the workpiece W held by the end effector 4 to become unstable. As an example, even if the end effector 4 holds the workpiece W through the holding control process, if the end effector 4 holds the workpiece W in a state where the end effector 4 is slightly in contact with an edge portion of the workpiece W, the workpiece W held by the end effector 4 may become unstable. For example, even if the end effector 4 holds the workpiece W through the holding control process, the posture of the workpiece W held by the end effector 4 may be different from the posture of the workpiece W that the end effector 4 is supposed to hold, which may cause the workpiece W held by the end effector 4 to become unstable. As an example, if the end effector 4 holds the workpiece W through the holding control process, but an inertial force acts on the workpiece W in an unintended direction as the end effector 4 moves due to the posture of the workpiece W held by the end effector 4, the workpiece W held by the end effector 4 may become unstable.

[0240] If the control device 3 performs processing to transfer the workpiece W held by the end effector 4 from the end effector 4 to the position and orientation changing device 5 under such circumstances where the workpiece W held by the end effector 4 is unstable, there is a possibility that the workpiece W held by the end effector 4 will fall from the end effector 4 before it is transferred from the end effector 4 to the position and orientation changing device 5. Furthermore, if the fallen workpiece W collides with the position and orientation changing device 5, this may lead to at least one of damage to the workpiece W, distortion of the workpiece W, failure of the position and orientation changing device 5, and damage to the position and orientation changing device 5.

[0241] Furthermore, the workpiece W held by the end effector 4 may move unintentionally before being transferred from the end effector 4 to the position and orientation changing device 5, resulting in the workpiece W being transferred from the end effector 4 to the position and orientation changing device 5 from an unintended position. Alternatively, the workpiece W held by the end effector 4 may move unintentionally before being transferred from the end effector 4 to the position and orientation changing device 5, resulting in the workpiece W being transferred from the end effector 4 to the position and orientation changing device 5 in an unintended orientation. As a result, the position and orientation changing device 5 may not be able to stably hold a workpiece that has been transferred to the position and orientation changing device 5 from an unintended position and / or in an unintended orientation. Furthermore, the workpiece W that has been transferred to the position and orientation changing device 5 from an unintended position and / or in an unintended orientation may collide with the position and orientation changing device 5. If the workpiece W collides with the position and orientation changing device 5, this may result in at least one of damage to the workpiece W, distortion of the workpiece W, malfunction of the position and orientation changing device 5, and damage to the position and orientation changing device 5. Furthermore, the impact of the collision may cause the workpiece W that the position and orientation changing device 5 is trying to hold to fall from the position and orientation changing device 5.

[0242] Therefore, the control device 3 may perform a workpiece position / orientation change process including a retry process as an example of the workpiece position / orientation change process. The retry process may include a process in which, if the workpiece W held by the end effector 4 is unstable after the holding control process in step S1 of FIG. 6 is performed, the end effector 4 temporarily releases the workpiece W held by the end effector 4, and then controls at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holds the workpiece W. Hereinafter, the workpiece position / orientation change process including the retry process will be described with reference to FIG. 15. FIG. 15 is a flowchart showing the flow of the workpiece position / orientation change process including the retry process.

[0243] 15 , even when a workpiece position / orientation change process including a retry process is performed, the image capturing device 21 captures an image of at least a part of the workpiece W held by the end effector 4 after the holding control process is performed (step S21), just as when a workpiece position / orientation change process not including a retry process is performed. As a result, the position / orientation calculation unit 311 acquires image data IMG (step S21).

[0244] Thereafter, the signal generating unit 312 determines whether or not the workpiece W held by the end effector 4 is unstable based on the image data IMG acquired in step S21 (step S51). As an example, the position and orientation calculating unit 311 may calculate at least one of the position and orientation of the workpiece W held by the end effector 4 by performing the above-described matching process based on the image data IMG acquired in step S21, and the signal generating unit 312 may determine whether or not the workpiece W held by the end effector 4 is unstable based on the calculation result of at least one of the position and orientation of the workpiece W held by the end effector 4.

[0245] As a first example of determining whether the workpiece W held by the end effector 4 is unstable, the signal generation unit 312 may calculate the amount of change in the position of the workpiece W held by the end effector 4 by repeating the process of calculating the position of the workpiece W held by the end effector 4. That is, the signal generation unit 312 may generate information regarding the amount of change in the position of the workpiece W held by the end effector 4. Specifically, in this case, in step S21, the imaging device 21 may repeat the process of capturing an image of at least a portion of the workpiece W held by the end effector 4. The imaging device 21 may capture an image of at least a portion of the workpiece W held by the end effector 4 multiple times. As a result, the imaging device 21 may generate multiple image data IMG. Thereafter, the position and orientation calculation unit 311 may repeat the process of calculating the position of the workpiece W held by the end effector 4 based on the multiple image data IMG the number of times equal to the number of image data IMG. Thereafter, the signal generating unit 312 may generate information regarding the amount of change in the position of the workpiece W held by the end effector 4 based on the calculation results of the position of the workpiece W based on the multiple image data IMG.

[0246] The amount of change in the position of the workpiece W held by the end effector 4 may refer to the amount of change in the position (i.e., the relative position) of the workpiece W held by the end effector 4 with respect to the end effector 4. In this case, in step S21, the imaging device 21 may capture images of at least a portion of the end effector 4 holding the workpiece W multiple times, in addition to at least a portion of the workpiece W held by the end effector 4. Thereafter, in step S22, the position and orientation calculation unit 311 may calculate the position of the end effector 4 in addition to the position of the workpiece W. Specifically, the position and orientation calculation unit 311 may calculate the position of the end effector 4 by performing processing similar to the processing for calculating the position of the workpiece W in step S12 of FIG. 8. In other words, the above-mentioned description of the processing for calculating the position of the workpiece W in step S12 of FIG. 8 can be used as a description of the processing for calculating the position of the end effector 4 by replacing the term "workpiece W" with the term "end effector 4." Therefore, to avoid redundant explanation, detailed explanation of the process of calculating the position of the end effector 4 in step S51 of Fig. 15 will be omitted. Thereafter, the signal generating unit 312 may calculate the amount of change in the position of the workpiece W held by the end effector 4 relative to the end effector 4 based on the calculation results of the positions of the workpiece W and the end effector 4 based on the multiple image data IMG.

[0247] Thereafter, the signal generating unit 312 may determine whether the workpiece W held by the end effector 4 is unstable based on the amount of change in the position of the workpiece W held by the end effector 4. For example, FIGS. 16A to 16D are cross-sectional views showing the workpiece W held by the end effector 4 together with the end effector 4 at time t11 and time t12, which is later than time t11. For example, as shown in FIG. 16A , the signal generating unit 312 may determine that the workpiece W held by the end effector 4 is not unstable (i.e., stable) if the position of the workpiece W held by the end effector 4 does not change between time t11 and time t12. On the other hand, as shown in FIG. 16B , for example, the signal generating unit 312 may determine that the workpiece W held by the end effector 4 is unstable if the position of the workpiece W held by the end effector 4 changes between time t11 and time t12. 16C , the signal generating unit 312 may determine that the workpiece W held by the end effector 4 is not unstable (i.e., stable) if the amount of change in the position of the workpiece W held by the end effector 4 between time t11 and time t12 is below a predetermined allowable position threshold. On the other hand, for example, as shown in FIG. 16D , the signal generating unit 312 may determine that the workpiece W held by the end effector 4 is unstable if the amount of change in the position of the workpiece W held by the end effector 4 between time t11 and time t12 exceeds a predetermined allowable position threshold.

[0248] The allowable position threshold may be set to a desired value that enables a distinction between a state in which the workpiece W held by the end effector 4 is unstable and a state in which the workpiece W held by the end effector 4 is not unstable, based on the amount of change in the position of the workpiece W held by the end effector 4. The allowable position threshold may be set in advance, or may be set appropriately at the timing when the workpiece position and orientation change process is performed. The allowable position threshold may be set by a user of the robot system SYS, or may be set by the control device 3.

[0249] As a second example of determining whether the workpiece W held by the end effector 4 is unstable, the signal generation unit 312 may calculate the amount of change in the attitude of the workpiece W held by the end effector 4 by repeating the process of calculating the attitude of the workpiece W held by the end effector 4. That is, the signal generation unit 312 may generate information regarding the amount of change in the attitude of the workpiece W held by the end effector 4. Specifically, in this case, in step S21, the imaging device 21 may repeat the process of capturing an image of at least a portion of the workpiece W held by the end effector 4. The imaging device 21 may capture an image of at least a portion of the workpiece W held by the end effector 4 multiple times. As a result, the imaging device 21 may generate multiple image data IMG. Thereafter, the position and orientation calculation unit 311 may repeat the process of calculating the attitude of the workpiece W held by the end effector 4 based on the multiple image data IMG the number of times equal to the number of image data IMG. Thereafter, the signal generating unit 312 may generate information regarding the amount of change in the posture of the workpiece W held by the end effector 4 based on the calculation results of the posture of the workpiece W based on the multiple image data IMG.

[0250] The amount of change in the orientation of the workpiece W held by the end effector 4 may refer to the amount of change in the orientation of the workpiece W held by the end effector 4 with respect to the end effector 4 (i.e., the relative orientation). In this case, in step S21, the imaging device 21 may capture an image of at least a portion of the end effector 4 holding the workpiece W multiple times, in addition to capturing an image of at least a portion of the workpiece W held by the end effector 4. Thereafter, in step S22, the position and orientation calculation unit 311 may calculate the orientation of the end effector 4 in addition to the orientation of the workpiece W. Specifically, the position and orientation calculation unit 311 may calculate the orientation of the end effector 4 by performing processing similar to the processing for calculating the orientation of the workpiece W in step S12 of FIG. 8. In other words, the above-described description of the processing for calculating the orientation of the workpiece W in step S12 of FIG. 8 can be used as a description of the processing for calculating the orientation of the end effector 4 by replacing the term "workpiece W" with the term "end effector 4." 15. Therefore, to avoid redundant explanation, detailed explanation of the process of calculating the attitude of the end effector 4 in step S51 will be omitted. Thereafter, the signal generating unit 312 may calculate the amount of change in the attitude of the workpiece W held by the end effector 4 relative to the end effector 4 based on the calculation results of the attitudes of the workpiece W and the end effector 4 based on the multiple image data IMG.

[0251] Thereafter, the signal generating unit 312 may determine whether the workpiece W held by the end effector 4 is unstable based on the amount of change in the posture of the workpiece W held by the end effector 4. For example, FIGS. 17A to 17D are cross-sectional views showing the workpiece W held by the end effector 4 together with the end effector 4 at time t11 and time t12, which is later than time t11. For example, as shown in FIG. 17A , the signal generating unit 312 may determine that the workpiece W held by the end effector 4 is not unstable (i.e., stable) if the posture of the workpiece W held by the end effector 4 does not change between time t11 and time t12. On the other hand, as shown in FIG. 17B , for example, the signal generating unit 312 may determine that the workpiece W held by the end effector 4 is unstable if the posture of the workpiece W held by the end effector 4 changes between time t11 and time t12. 17C , the signal generating unit 312 may determine that the workpiece W held by the end effector 4 is not unstable (i.e., stable) when the amount of change in the posture of the workpiece W held by the end effector 4 between time t11 and time t12 is below a predetermined allowable posture threshold. On the other hand, as shown in FIG. 17D , the signal generating unit 312 may determine that the workpiece W held by the end effector 4 is unstable when the amount of change in the posture of the workpiece W held by the end effector 4 between time t11 and time t12 exceeds a predetermined allowable posture threshold.

[0252] The allowable posture threshold may be set to a desired value that enables a distinction between a state in which the workpiece W held by the end effector 4 is unstable and a state in which the workpiece W held by the end effector 4 is not unstable, based on the amount of change in the posture of the workpiece W held by the end effector 4. The allowable posture threshold may be set in advance, or may be set appropriately at the timing when the workpiece position / posture change process is performed. The allowable posture threshold may be set by a user of the robot system SYS, or may be set by the control device 3.

[0253] As a third example of determining whether the workpiece W held by the end effector 4 is unstable, the signal generator 312 may calculate the difference between the calculated position of the workpiece W held by the end effector 4 (i.e., the actual position) and the target workpiece position described above. That is, the signal generator 312 may generate information regarding the difference between the actual position of the workpiece W held by the end effector 4 and the target workpiece position. The smaller the difference between the actual position of the workpiece W held by the end effector 4 and the target workpiece position, the higher the likelihood that the end effector 4 is holding the workpiece W at the intended position. Therefore, the smaller the difference between the actual position of the workpiece W held by the end effector 4 and the target workpiece position, the lower the likelihood that the workpiece W held by the end effector is unstable. On the other hand, the larger the difference between the actual position of the workpiece W held by the end effector 4 and the target workpiece position, the higher the likelihood that the end effector 4 is holding the workpiece W at an unintended position. Therefore, the greater the difference between the actual position of the workpiece W held by the end effector 4 and the target workpiece position, the higher the possibility that the workpiece W held by the end effector 4 is unstable. Therefore, the signal generating unit 312 may determine whether the workpiece W held by the end effector 4 is unstable based on the difference between the actual position of the workpiece W held by the end effector 4 and the target workpiece position. For example, the signal generating unit 312 may determine that the workpiece W held by the end effector 4 is unstable when the difference between the actual position of the workpiece W held by the end effector 4 and the target workpiece position exceeds a predetermined position difference threshold. On the other hand, for example, the signal generating unit 312 may determine that the workpiece W held by the end effector 4 is not unstable when the difference between the actual position of the workpiece W held by the end effector 4 and the target workpiece position is below a predetermined position difference threshold.

[0254] The position difference threshold may be set to a desired value that enables a distinction between a state in which the workpiece W held by the end effector 4 is unstable and a state in which the workpiece W held by the end effector 4 is not unstable, based on the difference between the actual position of the workpiece W held by the end effector 4 and the target workpiece position. The position difference threshold may be set in advance, or may be set appropriately at the timing when the workpiece position and orientation change process is performed. The position difference threshold may be set by a user of the robot system SYS, or may be set by the control device 3.

[0255] As a fourth example of determining whether the workpiece W held by the end effector 4 is unstable, the signal generator 312 may calculate the difference between the calculated result of the posture of the workpiece W held by the end effector 4 (i.e., the actual posture) and the above-described target workpiece posture. That is, the signal generator 312 may generate information regarding the difference between the actual posture of the workpiece W held by the end effector 4 and the target workpiece posture. The smaller the difference between the actual posture of the workpiece W held by the end effector 4 and the target workpiece posture, the higher the likelihood that the end effector 4 is holding the workpiece W in the intended posture. Therefore, the smaller the difference between the actual posture of the workpiece W held by the end effector 4 and the target workpiece posture, the lower the likelihood that the workpiece W held by the end effector is unstable. On the other hand, the larger the difference between the actual posture of the workpiece W held by the end effector 4 and the target workpiece posture, the higher the likelihood that the end effector 4 is holding the workpiece W in an unintended posture. Therefore, the greater the difference between the actual posture of the workpiece W held by the end effector 4 and the target workpiece posture, the higher the likelihood that the workpiece W held by the end effector 4 is unstable. Therefore, the signal generating unit 312 may determine whether the workpiece W held by the end effector 4 is unstable based on the difference between the actual posture of the workpiece W held by the end effector 4 and the target workpiece posture. For example, the signal generating unit 312 may determine that the workpiece W held by the end effector 4 is unstable when the difference between the actual posture of the workpiece W held by the end effector 4 and the target workpiece posture exceeds a predetermined posture difference threshold. On the other hand, for example, the signal generating unit 312 may determine that the workpiece W held by the end effector 4 is not unstable when the difference between the actual posture of the workpiece W held by the end effector 4 and the target workpiece posture is below a predetermined posture difference threshold.

[0256] The posture difference threshold may be set to a desired value that enables a distinction between a state in which the workpiece W held by the end effector 4 is unstable and a state in which the workpiece W held by the end effector 4 is not unstable, based on the difference between the actual posture of the workpiece W held by the end effector 4 and the target workpiece posture. The posture difference threshold may be set in advance, or may be set appropriately at the timing when the workpiece position / posture change process is performed. The posture difference threshold may be set by a user of the robot system SYS, or may be set by the control device 3.

[0257] As described in the first and second examples above, when the signal generating unit 312 determines whether the workpiece W held by the end effector 4 is unstable based on the amount of change in at least one of the position and posture of the workpiece W, the end effector 4 holding the workpiece W may move during at least a portion of the period during which the imaging device 21 captures images of the workpiece W held by the end effector 4 multiple times. In other words, the imaging device 21 may capture images of the workpiece W held by the end effector 4 during at least a portion of the period during which the end effector 4 holding the workpiece W is moving. In this case, if the workpiece W held by the end effector 4 is unstable, there is a possibility that at least one of the position and posture of the workpiece W held by the end effector 4 will change as the end effector 4 moves. On the other hand, if the workpiece W held by the end effector 4 is not unstable, there is a low possibility that at least one of the position and posture of the workpiece W held by the end effector 4 will change as the end effector 4 moves. Therefore, the amount of change in at least one of the position and posture of the workpiece W calculated based on the plurality of image data IMG, including image data IMG generated by the imaging device 21 capturing an image of the workpiece W held by the end effector 4 during at least a portion of the period during which the end effector 4 holding the workpiece W is moving, is likely to more accurately indicate whether the workpiece W held by the end effector 4 is unstable. As a result, the signal generating unit 312 can more accurately determine whether the workpiece W held by the end effector 4 is unstable based on the amount of change in at least one of the position and posture of the workpiece W.

[0258] The imaging device 21 may image the workpiece W before the end effector 4 holding the workpiece W moves, and may also image the workpiece W after the end effector 4 holding the workpiece W has moved. In this case, too, the amount of change in at least one of the position and posture of the workpiece W calculated based on a plurality of image data IMG including image data IMG generated by the imaging device 21 capturing an image of the workpiece W held by the end effector 4 before the end effector 4 holding the workpiece W moves, and image data IMG generated by the imaging device 21 capturing an image of the workpiece W held by the end effector 4 after the end effector 4 holding the workpiece W has moved, is more likely to more accurately indicate whether the workpiece W held by the end effector 4 is unstable. As a result, the signal generation unit 312 can more accurately determine whether the workpiece W held by the end effector 4 is unstable based on the amount of change in at least one of the position and posture of the workpiece W.

[0259] As an example, as described above, when the workpiece position and orientation change process is performed, the end effector 4 holding the workpiece W moves toward a transfer position where the end effector 4 should be located in order to transfer the workpiece W held by the end effector 4 to the position and orientation change device 5. That is, the end effector 4 holding the workpiece W moves toward a position where the end effector 4 will transfer the workpiece W held by the end effector 4 to the position and orientation change device 5. In other words, the workpiece W held by the end effector 4 moves toward a first receiving position where the position and orientation change device 5 will receive the workpiece W held by the end effector 4. In this case, for example, the imaging device 21 may capture an image of the workpiece W held by the end effector 4 during at least a portion of the period during which the end effector 4 holding the workpiece W is moving toward the transfer position (that is, the period during which the workpiece W held by the end effector 4 is moving toward the first receiving position). For example, the imaging device 21 may image the workpiece W held by the end effector 4 before the end effector 4 holding the workpiece W moves toward the transfer position, and may image the workpiece W held by the end effector 4 after the end effector 4 holding the workpiece W moves toward the transfer position. In other words, the imaging device 21 may image the workpiece W held by the end effector 4 before the end effector 4 holding the workpiece W moves toward the transfer position, and may image the workpiece W held by the end effector 4 while the end effector 4 holding the workpiece W is located at the transfer position. In this case, the robot system SYS can effectively utilize at least a portion of the period during which the end effector 4 holding the workpiece W moves toward the transfer position as at least a portion of the period during which the imaging device 21 images the workpiece W held by the end effector 4. As a result, the time required to perform the work position and posture change processing can be shortened compared to when a dedicated period is secured for the imaging device 21 to capture an image of the work W held by the end effector 4, separate from the period during which the end effector 4 holding the work W moves toward the delivery position.

[0260] As another example, as described above, the end effector 4 holding the workpiece W moves away (retracts) from the mounting device T after holding the workpiece W (see FIG. 6C ). In this case, for example, the imaging device 21 may capture an image of the workpiece W held by the end effector 4 during at least a portion of the period during which the end effector 4 holding the workpiece W moves away from the mounting device T. In this case, the robot system SYS can effectively utilize at least a portion of the period during which the end effector 4 holding the workpiece W moves away from the mounting device T as at least a portion of the period during which the imaging device 21 captures an image of the workpiece W held by the end effector 4. As a result, the time required to perform the workpiece position / posture change processing can be shortened compared to when a dedicated period for the imaging device 21 to capture an image of the workpiece W held by the end effector 4 is secured separately from the period during which the end effector 4 holding the workpiece W moves away from the mounting device T.

[0261] As another example, the end effector 4 holding the workpiece W may move during any period different from the period during which the end effector 4 moves toward the delivery position and the period during which the end effector 4 moves away from the mounting device T. In this case, the imaging device 21 may capture an image of the workpiece W held by the end effector 4 during at least a part of the period during which the end effector 4 holding the workpiece W moves.

[0262] When the end effector 4 holding the workpiece W moves for any period of time, the end effector 4 holding the workpiece 4 may move in order to move the workpiece W, which may be held in an unstable state by the end effector 4. In other words, the end effector 4 holding the workpiece 4 may move for a purpose other than transporting the workpiece W held by the end effector 4. As an example, the end effector 4 holding the workpiece 4 may move so as to intentionally vibrate back and forth, left and right, and / or up and down. In this case, too, the imaging device 21 may capture an image of the workpiece W held by the end effector 4 for at least a portion of the period during which the end effector 4 holding the workpiece W moves.

[0263] 15 , if the result of the determination in step S51 indicates that the workpiece W held by the end effector 4 is not unstable (step S51: No), the control device 3 may determine that a retry process is unnecessary. In this case, determining whether the workpiece W held by the end effector 4 is unstable may be considered equivalent to determining whether a retry process is unnecessary. In this case, the control device 3 may continue the workpiece position and orientation change process. That is, the control device 3 may perform the processes from step S22 to step S24. Specifically, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the workpiece W held by the end effector 4 based on the image data IMG acquired in step S21 (step S22). Thereafter, the signal generation unit 312 may determine whether at least one of the position and orientation of the workpiece W held by the end effector 4 should be changed based on the calculation result of at least one of the position and orientation of the workpiece W in step S22 (step S23). Thereafter, if it is determined that at least one of the position and posture of the workpiece W held by the end effector 4 should be changed (step S23: Yes), the control device 3 may change at least one of the position and posture of the workpiece W held by the end effector 4 (step S24).

[0264] In the above description, in step S23, the signal generating unit 312 determines whether or not at least one of the position and orientation of the workpiece W held by the end effector 4 should be changed based on the image data IMG acquired in step S21 (specifically, based on at least one of the position and orientation of the workpiece W held by the end effector 4 calculated based on the image data IMG acquired in step S21). In other words, the image data IMG used in step S23 to determine whether or not at least one of the position and orientation of the workpiece W held by the end effector 4 should be changed is the same as the image data IMG used in step S51 to determine whether or not the workpiece W held by the end effector 4 is unstable. However, the image data IMG used in step S23 to determine whether or not at least one of the position and orientation of the workpiece W held by the end effector 4 should be changed may be different from the image data IMG used in step S51 to determine whether or not the workpiece W held by the end effector 4 is unstable. For example, in step S23, the signal generating unit 312 may determine whether or not to change at least one of the position and the posture of the workpiece W held by the end effector 4, based on the image data IMG generated by the imaging system 2 capturing an image of the workpiece W at a first timing. For example, in step S51, the signal generating unit 312 may determine whether or not the workpiece W held by the end effector 4 is unstable, based on the image data IMG generated by the imaging system 2 capturing an image of the workpiece W at a second timing different from the first timing.As an example, in step S23, the signal generating unit 312 may determine whether or not at least one of the position and posture of the workpiece W held by the end effector 4 should be changed based on image data IMG generated by the imaging system 2 capturing an image of the workpiece W held by the end effector 4 at a first timing after the holding control process has been performed, and in step S51, may determine whether or not the workpiece W held by the end effector 4 is unstable based on image data IMG generated by the imaging system 2 capturing an image of the workpiece W held by the end effector 4 at a second timing after the holding control process has been performed. As another example, in step S23, the signal generating unit 312 may determine whether or not at least one of the position and posture of the workpiece W held by the end effector 4 should be changed based on image data IMG generated by the imaging system 2 capturing an image of the workpiece W that the end effector 4 is about to hold at a first timing before the holding control process is performed, and in step S51, determine whether or not the workpiece W held by the end effector 4 is unstable based on image data IMG generated by the imaging system 2 capturing an image of the workpiece W held by the end effector 4 at a second timing after the holding control process is performed.

[0265] On the other hand, if the result of the determination in step S51 is that the workpiece W held by the end effector 4 is determined to be unstable (step S51: Yes), the control device 3 may determine that a retry process is necessary. In this case, determining whether or not the workpiece W held by the end effector 4 is unstable may be considered equivalent to determining whether or not a retry process is necessary. In this case, the control device 3 may perform a retry process. In other words, the control device 3 may determine that a retry process is necessary.

[0266] Specifically, first, the signal generating unit 312 may generate a robot control signal that controls at least the end effector 4 so that the end effector 4 holding the workpiece W releases (i.e., releases) the workpiece W (step S52). For example, the signal generating unit 312 may generate a robot control signal that controls at least the end effector 4 so that the end effector 4 holding the workpiece W releases the workpiece W to the mounting device T. In this case, the signal generating unit 312 may generate a robot control signal that controls at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holding the workpiece W moves to a position where the end effector 4 should be positioned when releasing the workpiece W to the mounting device T, and then the end effector 4 holding the workpiece W releases the workpiece W to the mounting device T.

[0267] The signal generating unit 312 may generate a robot control signal to control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holding the workpiece W releases the workpiece W onto the placement device T from a desired height relative to the placement device T. For example, the signal generating unit 312 may generate a robot control signal to control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holding the workpiece W releases the workpiece W onto the placement device T from a low position (low height). For example, the signal generating unit 312 may generate a robot control signal to control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holding the workpiece W releases the workpiece W onto the placement device T from a low height so as to reduce the impact caused by a collision between the workpiece W released onto the placement device T and the placement device T. As a result, the possibility of damage to at least one of the workpiece W and the placement device T due to the release of the workpiece W is reduced.

[0268] Thereafter, the control device 3 performs a holding control process for controlling at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holds the workpiece W (step S53). Note that the holding control process in step S53 may be the same as the holding control process in step S1 of Fig. 7. Therefore, in order to avoid redundant explanation, a detailed explanation of the holding control in step S53 will be omitted.

[0269] In this way, the retry process in which the end effector 4 temporarily releases (i.e., releases) the workpiece W held by the end effector 4 and then controls at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holds the workpiece W may be a process that includes the processes of steps S52 and S53 in Figure 15.

[0270] In the holding control process performed in step S53 after the workpiece W is released to the placement device T in step S52, the signal generating unit 312 may generate a robot control signal so that the end effector 4 holds the workpiece W released to the placement device T in step S52. In other words, the workpiece W released to the placement device T in step S52 and the workpiece W held by the end effector 4 in the holding control process performed in step S53 after the workpiece W is released to the placement device T in step S52 may be the same.

[0271] Alternatively, in the holding control process performed in step S53 after the workpiece W is released to the placement device T in step S52, the signal generating unit 312 may generate a robot control signal so that the end effector 4 holds a workpiece W different from the workpiece W released to the placement device T in step S52. In other words, the workpiece W released to the placement device T in step S52 may be different from the workpiece W held by the end effector 4 in the holding control process performed in step S53 after the workpiece W is released to the placement device T in step S52.

[0272] In this way, when the workpiece position and orientation change process including the retry process is performed, if the workpiece W held by the end effector 4 is unstable, the end effector 4 can temporarily let go (i.e., release) the workpiece W held by the end effector 4 in an unstable state. This reduces the possibility that the workpiece W held by the end effector 4 will fall from the end effector 4 before being transferred from the end effector 4 to the position and orientation change device 5. Furthermore, this reduces the possibility that the end effector 4 holding the workpiece W will transfer the workpiece W from the end effector 4 to the position and orientation change device 5 from an unintended position and / or in an unintended orientation. This allows the end effector 4 holding the workpiece W to properly transfer the workpiece W from the end effector 4 to the position and orientation change device 5.

[0273] Furthermore, in this embodiment, because the imaging system 2 is attached to the movable robot 1 (particularly the robot arm 12), the same imaging system 2 can image the workpiece W placed on the mounting device T to generate image data IMG for performing the hold control processing, and can also image the workpiece W held by the end effector 4 to generate image data IMG for performing the retry processing. In other words, the same imaging system 2 can simultaneously or separately generate the image data IMG for performing the hold control processing and the image data IMG for performing the retry processing. If the imaging system 2 were attached to a fixed support device different from the robot 1, because the imaging system 2 is not movable, the robot system SYS might need to separately include a first imaging system 2 for imaging the workpiece W placed on the mounting device T to generate image data IMG for performing the hold control processing, and a second imaging system 2 for imaging the workpiece W held by the end effector 4 to generate image data IMG for performing the retry determination processing. Therefore, in this embodiment, the number of required imaging systems 2 is reduced, and the cost of the robot system SYS can be reduced.

[0274] In addition to the retry process, the control device 3 may perform a workpiece position and posture change process including the above-described workpiece holding determination process (see FIG. 13 ). In this case, the control device 3 may first perform a workpiece holding determination process including the processes of steps S21 and S41 in FIG. 13 . Thereafter, if it is determined in the workpiece holding determination process that the end effector 4 is holding a workpiece W (step S41 in FIG. 13 : Yes), the control device 3 may determine whether the workpiece W held by the end effector 4 is unstable (step S51 in FIG. 15 ). Note that the image data IMG used to determine whether the end effector 4 is holding a workpiece W in step S41 in FIG. 13 may be the same as the image data IMG used to determine whether the workpiece W held by the end effector 4 is unstable in step S51. Alternatively, the image data IMG used to determine whether the end effector 4 is holding a workpiece W in step S41 of Fig. 13 may be different from the image data IMG used to determine whether the workpiece W held by the end effector 4 is unstable in step S51. For example, in step S41 of Fig. 13, the signal generation unit 312 may determine whether the end effector 4 is holding a workpiece W based on image data IMG generated by the imaging system 2 capturing an image of the workpiece holding area WHA at a first timing. For example, in step S51 of Fig. 15, the signal generation unit 312 may determine whether the workpiece W held by the end effector 4 is unstable based on image data IMG generated by the imaging system 2 capturing an image of the workpiece W held by the end effector 4 at a second timing different from the first timing.

[0275] (2-4) Workpiece Position and Orientation Changing Process Including Device Position and Orientation Calculation Process As described above, when the workpiece position and orientation changing process is performed, the workpiece W held by the end effector 4 is transferred from the end effector 4 to the position and orientation changing device 5 (for example, the holding member 53). Here, the position and orientation changing device 5 may be disposed at a known position and in a known orientation, and the control device 3 may generate a robot control signal based on known information regarding at least one of the positions and orientations of the position and orientation changing device 5 and the holding member 53, as described above.

[0276] However, at least one of the position and orientation of the position and orientation change device 5 does not necessarily match at least one of the position and orientation assumed by the control device 3. For example, in a situation where the position and orientation change device 5 is placed at a position known to the control device 3, an unintended impact may be applied to the position and orientation change device 5. In this case, the position and orientation change device 5 may move unintendedly due to the impact. As a result, the position and orientation change device 5 may be placed at a position different from the position known to the control device 3. Similarly, in a situation where the position and orientation change device 5 is placed in a posture known to the control device 3, an unintended impact may be applied to the position and orientation change device 5. In this case, the position and orientation change device 5 may move unintendedly due to the impact. As a result, the position and orientation change device 5 may be placed in a posture different from the posture known to the control device 3. As a result, even if the control device 3 controls at least one of the robot 1, the end effector 4, the robot movable device, and the position and orientation change device 5 so that the end effector 4 passes the workpiece W held by the end effector 4 to the position and orientation change device 5 based on known information regarding at least one of the position and orientation of the position and orientation change device 5, the end effector 4 may not be able to properly pass the workpiece W held by the end effector 4 to the position and orientation change device 5. Alternatively, when the end effector 4 passes the workpiece W held by the end effector 4 to the position and orientation change device 5, the workpiece W passed to the position and orientation change device 5 from an unintended position and / or in an unintended orientation may collide with the position and orientation change device 5. If the workpiece W collides with the position and orientation change device 5, this may result in at least one of breakage of the workpiece W, distortion of the workpiece W, malfunction of the position and orientation change device 5, and / or damage to the position and orientation change device 5. Furthermore, the impact of the collision may cause the workpiece W that the position and orientation change device 5 is trying to hold to fall from the position and orientation change device 5.

[0277] Similarly, at least one of the position and orientation of the holding member 53 included in the position and orientation change device 5 does not necessarily match at least one of the position and orientation assumed by the control device 3. For example, even if the control device 3 generates a position and orientation control signal for controlling the position and orientation change device 5 so as to move the holding member 53 by a desired movement amount, the holding member 53 may move by an amount different from the desired movement amount due to a movement error of the holding member 53, etc. For example, even if the control device 3 generates a position and orientation control signal for controlling the position and orientation change device 5 so as to move the holding member 53 by a desired movement amount, the holding member 53 may move by an amount different from the desired movement amount due to a reaction when the holding member 53 moves and then stops. In this case, at least one of the position and orientation of the holding member 53 that has moved and stopped may differ from at least one of the position and orientation assumed by the control device 3 (i.e., at least one of the position and orientation assumed if the holding member 53 moves by the desired movement amount). As a result, even if the control device 3 controls at least one of the robot 1, the end effector 4, the robot movable device, and the position and orientation changing device 5 so that the end effector 4 passes the workpiece W held by it to the holding member 53 based on known information regarding at least one of the position and orientation of the holding member 53 (i.e., at least one of the position and orientation of the holding member 53 assumed by the control device 3), the end effector 4 may not be able to properly pass the workpiece W held by it to the holding member 53. Alternatively, when the end effector 4 passes the workpiece W held by it to the holding member 53, the workpiece W passed to the holding member 53 from an unintended position and / or in an unintended orientation may collide with the holding member 53. If the workpiece W collides with the holding member 53, this may lead to at least one of damage to the workpiece W, distortion of the workpiece W, failure of the holding member 53, and damage to the holding member 53. Furthermore, there is a possibility that the workpiece W held by the end effector 4 may fall from the end effector 4 due to the impact of the collision.

[0278] Therefore, the control device 3 may perform workpiece position and orientation change processing including device position and orientation calculation processing as an example of workpiece position and orientation change processing. The device position and orientation calculation processing may include processing for calculating at least one of the position and orientation of at least a portion of the position and orientation change device 5. Hereinafter, the workpiece position and orientation change processing including device position and orientation calculation processing will be described with reference to Fig. 18. Fig. 18 is a flowchart showing the flow of workpiece position and orientation change processing including device position and orientation calculation processing.

[0279] 18 , even when a workpiece position and orientation change process including an apparatus position and orientation calculation process is performed, the image capturing device 21 captures an image of at least a portion of the workpiece W held by the end effector 4 after the holding control process is performed (step S61), just as when a workpiece position and orientation change process not including an apparatus position and orientation calculation process is performed. As a result, the position and orientation calculation unit 311 acquires image data IMG (step S61).

[0280] However, when workpiece position and orientation change processing including apparatus position and orientation calculation is performed, the imaging device 21 captures an image of at least a portion of the position and orientation change apparatus 5 in addition to at least a portion of the workpiece W held by the end effector 4 (step S61). That is, the processing of step S61 in Fig. 18 may differ from the processing of step S21 in Fig. 9 described above in that the imaging device 21 captures an image of at least a portion of the position and orientation change apparatus 5 in addition to at least a portion of the workpiece W held by the end effector 4. Other features of the processing of step S61 in Fig. 18 may be the same as other features of the processing of step S21 in Fig. 9 described above.

[0281] For ease of explanation, the following description will be given of an example in which, in step S61, the imaging device 21 captures an image of at least a part of the holding member 53 included in the position and orientation change device 5, as at least a part of the position and orientation change device 5. However, in step S61, the imaging device 21 may capture an image of a part of the position and orientation change device 5 other than the holding member 53, as at least a part of the position and orientation change device 5.

[0282] In step S61, the imaging device 21 may collectively capture images of at least a portion of the workpiece W held by the end effector 4 and at least a portion of the holding member 53. That is, the imaging device 21 may capture images of both at least a portion of the workpiece W held by the end effector 4 and at least a portion of the holding member 53 in a single image. For example, the imaging device 21 may collectively capture images of at least a portion of the workpiece W held by the end effector 4 and at least a portion of the holding member 53 at a timing when the imaging range of the imaging device 21 includes both at least a portion of the workpiece W held by the end effector 4 and at least a portion of the holding member 53.

[0283] In this case, the signal generating unit 312 may generate a robot control signal to move the end effector 4 so that the imaging range of the imaging device 21 includes both at least a portion of the workpiece W held by the end effector 4 and at least a portion of the holding member 53. Thereafter, when the movement of the end effector 4 in accordance with the robot control signal causes the imaging range of the imaging device 21 to include both at least a portion of the workpiece W held by the end effector 4 and at least a portion of the holding member 53, the imaging device 21 may capture an image of at least a portion of the workpiece W held by the end effector 4 and at least a portion of the position and orientation changing device 5 together.

[0284] Alternatively, as described above, in the process of the end effector 4 holding the workpiece W moving toward the transfer position where the end effector 4 should be positioned to transfer the workpiece W held by the end effector 4 to the position and attitude changing device 5, the imaging device 21 may capture images of at least a portion of the workpiece W held by the end effector 4 and at least a portion of the holding member 53 together at the timing when the imaging range of the imaging device 21 includes both at least a portion of the workpiece W held by the end effector 4 and at least a portion of the holding member 53. In this case, the imaging device 21 may capture images of at least a portion of the workpiece W held by the end effector 4 and at least a portion of the holding member 53 during at least a portion of the period during which the end effector 4 moves toward the transfer position. In this case, the robot system SYS can effectively utilize at least a portion of the period during which the end effector 4 holding the workpiece W moves toward the transfer position as at least a portion of the period during which the imaging device 21 captures images of at least a portion of the workpiece W held by the end effector 4 and at least a portion of the holding member 53. As a result, it is possible to reduce the time required to perform the workpiece position / orientation change processing compared to when a dedicated period is ensured for the imaging device 21 to capture images of at least a portion of the workpiece W held by the end effector 4 and at least a portion of the holding member 53, separate from the period during which the end effector 4 holding the workpiece W moves toward the delivery position. However, the imaging device 21 may capture images of at least a portion of the workpiece W held by the end effector 4 and at least a portion of the holding member 53 after the end effector 4 is positioned at the delivery position (i.e., during the period during which the end effector 4 is positioned at the delivery position).

[0285] Alternatively, in step S61, the imaging device 21 may separately capture images of at least a portion of the workpiece W held by the end effector 4 and at least a portion of the holding member 53. For example, the imaging device 21 may capture an image of at least a portion of the workpiece W held by the end effector 4, and then capture an image of at least a portion of the holding member 53. For example, the imaging device 21 may capture an image of at least a portion of the holding member 53, and then capture an image of at least a portion of the workpiece W held by the end effector 4.

[0286] As an example, after holding the workpiece W placed on the placement device T, the end effector 4 moves toward a transfer position where the end effector 4 should be located to transfer the workpiece W held by the end effector 4 to the position / attitude changing device 5. In this case, after the end effector 4 holds the workpiece W placed on the placement device T, the imaging device 21 may first capture an image of at least a portion of the workpiece W held by the end effector 4. This is because, at the time the end effector 4 holds the workpiece W placed on the placement device T, it is highly likely that at least a portion of the workpiece W held by the end effector 4 will be included in the imaging range of the imaging device 21. Thereafter, the signal generating unit 312 may generate a robot control signal so that the end effector 4 holding the workpiece W moves toward the transfer position. Here, the closer the end effector 4 gets to the transfer position, the more likely it is that at least a portion of the holding member 53 will be included in the imaging range of the imaging device 21. Therefore, during the process of the end effector 4 moving toward the transfer position, the imaging device 21 may capture an image of at least a portion of the holding member 53 at a timing when at least a portion of the holding member 53 comes within the imaging range of the imaging device 21. In this case, the imaging device 21 may capture an image of at least a portion of the holding member 53 during at least a portion of the period during which the end effector 4 moving toward the transfer position. In this case, the robot system SYS can also effectively utilize at least a portion of the period during which the end effector 4 holding the workpiece W moves toward the transfer position as at least a portion of the period during which the imaging device 21 captures an image of at least a portion of the workpiece W and at least a portion of the holding member 53 held by the end effector 4. As a result, the time required for the workpiece position and orientation change processing can be shortened compared to when a dedicated period is secured for the imaging device 21 to capture an image of at least a portion of the workpiece W and at least a portion of the holding member 53 held by the end effector 4, separate from the period during which the end effector 4 holding the workpiece W moves toward the transfer position.However, the imaging device 21 may capture an image of at least a portion of the holding member 53 after the end effector 4 is positioned at the delivery position (i.e., while the end effector 4 is positioned at the delivery position).

[0287] Then, the position and orientation calculation unit 311 calculates at least one of the position and orientation of the workpiece W held by the end effector 4 based on the image data IMG acquired in step S61 (particularly, the image data IMG generated by capturing an image of at least a portion of the workpiece W held by the end effector 4) (step S22).

[0288] Furthermore, the position and orientation calculation unit 311 calculates at least one of the position and orientation of the holding member 53 based on the image data IMG acquired in step S61 (particularly, image data IMG generated by capturing an image of at least a portion of the holding member 53) (step S62). Specifically, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the holding member 53 by performing processing similar to the processing for calculating the position of the workpiece W in step S12 of FIG. 8. In other words, the above-described description of the processing for calculating at least one of the position and orientation of the workpiece W in step S12 of FIG. 8 can be used as a description of the processing for calculating at least one of the position and orientation of the holding member 53 by replacing the term "workpiece W" with the term "holding member 53." Therefore, in order to avoid redundant description, a detailed description of the processing for calculating at least one of the position and orientation of the holding member 53 in step S62 of FIG. 18 will be omitted.

[0289] Then, the signal generating unit 312 determines whether or not at least one of the position and posture of the workpiece W held by the end effector 4 should be changed based on the calculation results of at least one of the position and posture of the workpiece W in step S22 (step S23).

[0290] If it is determined in step S23 that at least one of the position and posture of the workpiece W held by the end effector 4 does not need to be changed (step S23: No), the control device 3 may terminate the workpiece position and posture change process shown in Fig. 9. Thereafter, the control device 3 may perform a release control process (step S3 in Fig. 7).

[0291] On the other hand, if the result of the judgment in step S23 is that it is determined that at least one of the position and posture of the workpiece W held by the end effector 4 should be changed (step S23: Yes), the control device 3 changes at least one of the position and posture of the workpiece W held by the end effector 4 (step S63).

[0292] 9 described above in that at least one of the position and orientation of the workpiece W held by the end effector 4 is changed based on the calculation result of at least one of the position and orientation of the holding member 53 in step S62 (i.e., the calculation result of at least one of the position and orientation of at least a part of the position and orientation changing device 5), instead of known information regarding at least one of the positions and orientations of the position and orientation changing device 5 and the holding member 53. More specifically, the process of step S63 differs from the process of step S24 described above in that it includes the process of step S631 instead of the process of step S241 in FIG. 9 described above. Other features of the process of step S63 may be the same as other features of the process of step S24 described above in FIG.

[0293] In step S631, the control device 3 controls at least one of the robot 1, the end effector 4, the robot movable device, and the position / posture change device 5 so that the end effector 4 passes the workpiece W held by the end effector 4 to the position / posture change device 5 (e.g., the holding member 53) based on the calculation result of at least one of the position and posture of the holding member 53 in step S62, instead of known information regarding at least one of the positions and postures of the position / posture change device 5 and the holding member 53. In other words, the processing of step S631 differs from the processing of step S241 in FIG. 9 , which does not require the calculation result of at least one of the position and posture of the holding member 53, in that the calculation result of at least one of the position and posture of the holding member 53 in step S62 is used instead of known information regarding at least one of the positions and postures of the position / posture change device 5 and the holding member 53. Other features of the processing of step S631 may be the same as other features of the processing of step S241 in FIG. 9 described above.

[0294] Specifically, in step S631, as in step S241 in Figure 9, the signal generating unit 312 may generate a robot control signal to move the end effector 4 holding the workpiece W so that, when the end effector 4 holding the workpiece W is positioned at the transfer position, the positional relationship between the workpiece W and the holding member 53 matches the first target positional relationship and / or the posture relationship between the workpiece W and the holding member 53 matches the first target posture relationship. However, in step S631, the signal generating unit 312 may generate a robot control signal for moving the end effector 4 holding the workpiece W based on at least one of the calculation results of the position and posture of the workpiece W held by the end effector 4 in step S22 and at least one of the calculation results of the position and posture of the holding member 53 in step S62, so that the positional relationship between the workpiece W located at the position calculated in step S22 and the holding member 53 located at the position calculated in step S62 matches the first target positional relationship, and / or so that the posture relationship between the workpiece W assuming the posture calculated in step S22 and the holding member 53 assuming the posture calculated in step S62 matches the first target posture relationship.

[0295] As a result, when a workpiece position and orientation change process including an apparatus position and orientation calculation process is performed, even if the position of the position and orientation change device 5 (e.g., the holding member 53) differs from a position known to the control device 3 and / or the orientation of the position and orientation change device 5 (e.g., the holding member 53) differs from an orientation known to the control device 3, the control device 3 can move the end effector 4 holding the workpiece W so that the positional relationship between the workpiece W and the holding member 53 coincides with the first target positional relationship and / or the orientation relationship between the workpiece W and the holding member 53 coincides with the first target orientation relationship when the end effector 4 holding the workpiece W is positioned at the delivery position. For example, FIG. 19A is a cross-sectional view showing the first target positional relationship and the first target orientation relationship between the workpiece W and the holding member 53. 19B , when the holding member 53 is located at a first position known to the control device 3 and / or when the holding member 53 is in a first orientation known to the control device 3, the control device 3 can calculate at least one of the position of the holding member 53 located at the first position and the orientation of the holding member 53 in the first orientation, based on image data IMG generated by capturing an image of the holding member 53. As a result, as shown in FIG. 19B , when the holding member 53 is located at a first position known to the control device 3 and / or when the holding member 53 is in a first orientation known to the control device 3, the control device 3 can move the end effector 4 holding the workpiece W so that the positional relationship between the workpiece W and the holding member 53 coincides with the first target positional relationship and / or the orientation relationship between the workpiece W and the holding member 53 coincides with the first target orientation relationship when the end effector 4 holding the workpiece W is positioned at the delivery position. On the other hand, as shown in Figure 19C, when the holding member 53 is located at a second position different from the first position known to the control device 3 and / or the holding member 53 takes a second posture different from the first posture known to the control device 3, the control device 3 can calculate at least one of the position of the holding member 53 located at the second position and the posture of the holding member 53 taking the second posture based on the image data IMG generated by imaging the holding member 53.That is, the control device 3 can recognize that the holding member 53 is located at a second position different from the first position known to the control device 3 and / or that the holding member 53 has a second posture different from the first posture known to the control device 3. In other words, the control device 3 can recognize that the holding member 53 is not located at the first position known to the control device 3 and / or that the holding member 53 does not have a first posture known to the control device 3. As a result, as shown in FIG. 19C , even when the holding member 53 is located at a second position different from the first position known to the control device 3 and / or when the holding member 53 has a second posture different from the first posture known to the control device 3, the control device 3 can move the end effector 4 holding the workpiece W so that the positional relationship between the workpiece W and the holding member 53 matches the first target positional relationship and / or the posture relationship between the workpiece W and the holding member 53 matches the first target posture relationship when the end effector 4 holding the workpiece W is located at the delivery position.

[0296] In this way, when the workpiece position and orientation change process including the device position and orientation calculation process is performed, the control device 3 calculates at least one of the position and orientation of the position and orientation change device 5 (e.g., the holding member 53) based on image data IMG generated by capturing an image of at least a portion of the position and orientation change device 5 (e.g., the holding member 53), and generates a robot control signal based on the calculation result of at least one of the position and orientation of the position and orientation change device 5 (e.g., the holding member 53). Therefore, regardless of the position and orientation of the position and orientation change device 5 (e.g., the holding member 53), the end effector 4 can appropriately transfer the workpiece W held by the end effector 4 to the position and orientation change device 5 (e.g., the holding member 53). For example, even if the position and orientation change device 5 (e.g., the holding member 53) is located at a position different from the position known to the control device 3 and / or if the position and orientation change device 5 (e.g., the holding member 53) is in an orientation different from the orientation known to the control device 3, the end effector 4 can appropriately transfer the workpiece W held by the end effector 4 to the position and orientation change device 5 (e.g., the holding member 53).

[0297] Furthermore, in this embodiment, because the imaging system 2 is attached to the movable robot 1 (particularly the robot arm 12), the same imaging system 2 can capture an image of the workpiece W held by the end effector 4 and also capture an image of the position and orientation changing device 5. If the imaging system 2 were attached to a fixed support device different from the robot 1, because the imaging system 2 is not movable, the robot system SYS might have to be provided with a first imaging system 2 for capturing an image of the workpiece W held by the end effector 4 and a second imaging system 2 for capturing an image of the position and orientation changing device 5 separately. Therefore, in this embodiment, the number of necessary imaging systems 2 is reduced, and the cost of the robot system SYS can be reduced.

[0298] In addition to the device position and orientation calculation process, the control device 3 may perform a workpiece position and orientation change process that includes at least one of the above-described workpiece holding determination process (see FIG. 13 ) and retry process. In this case, if it is determined in the workpiece holding determination process that the end effector 4 is holding the workpiece W (step S41 in FIG. 13 : Yes) and / or if it is determined that the workpiece W held by the end effector 4 is not unstable (step S51 in FIG. 15 : No), the control device 3 may calculate at least one of the position and orientation of the position and orientation change device 5 (step S62 in FIG. 18 ), and change at least one of the position and orientation of the workpiece W held by the end effector 4 based on the calculation result of at least one of the position and orientation of the position and orientation change device 5 (step S63 in FIG. 18 ). Furthermore, at least one of the image data IMG used to determine whether the end effector 4 is holding a workpiece W in ...

Claims

A control device is provided with a holding device that holds a target object, and generates control information for controlling at least one of a robot that moves the holding device and the holding device, a computing device that generates the control information; a communication device that outputs the control information generated by the arithmetic device; Equipped with The computing device generating the control information based on an imaging result of at least one of at least a part of the target object and at least a part of the modification device by an imaging system before the target object held by the holding device is handed over to a modification device capable of modifying at least one of a position and a posture of the target object; Control device. 　 the control information includes at least one of control information for passing the target object held in the holding device to the change device and control information for processing the target object held in the holding device without passing through the change device; The retained target object is processed based on the control information. The control device according to claim 1 . 　 The target object is processed based on at least one of the position and the orientation of the target object obtained based on the imaging result. The control device according to claim 1 or 2. 　 The computing device determining whether at least one of the position and the orientation of the target object should be changed; generating the control information so as to pass the target object to the modification device when it is determined that at least one of the position and the orientation of the target object should be modified; The control device according to any one of claims 1 to 3. 　 generating the control information so as to pass the target object to the modification device when the position or orientation of the target object differs from at least one of a desired position or orientation; The control device according to claim 4. 　 The computing device determining whether at least one of the position and the orientation of the target object should be changed; When it is determined that at least one of the position and the orientation of the target object should not be changed, the control information is generated so that the target object held by the holding device is processed without going through the change device. The control device according to claim 4 or 5. 　 The imaging result of the target object includes a result of imaging at least a part of the target object held by the holding device after the target object is held by the holding device. The control device according to any one of claims 1 to 6. 　 The imaging results of the target object include results of imaging a plurality of target objects before the target object is held by the holding device. A control device according to any one of claims 1 to 7. 　 The computing device determines the target objects to be held by the holding device based on the imaging results of the plurality of target objects. The control device according to claim 8. 　 The arithmetic device is capable of generating the control information for performing a retry process for the holding device to hold a second target object after the holding device releases a first target object, which is the target object held by the holding device, based on the imaging result of at least a part of the target object held by the holding device. A control device according to any one of claims 1 to 9. 　 The first target object and the second target object are the same. The control device according to claim 10. 　 The computing device acquires information about a change amount of at least one of a position and an orientation of the target object held by the holding device relative to the holding device, based on a plurality of imaging results obtained by the imaging system capturing images of at least a portion of the target object held by the holding device multiple times. The control device according to claim 10 or 11. 　 The computing device acquires information about a difference between at least one of a position and an orientation of the target object held by the holding device relative to the holding device and at least one of a target position and a target orientation of the target object held by the holding device relative to the holding device, based on the imaging result of at least a part of the target object held by the holding device. A control device according to any one of claims 10 to 12. 　 The computing device determining whether the retry process is necessary based on the image capturing result of at least a part of the target object held by the holding device; If it is determined that the retry process is necessary, the control information is generated so that the retry process is performed. A control device according to any one of claims 10 to 13. 　 The computing device When it is determined that the retry process is unnecessary, the control information is generated so that the target object held by the holding device is passed to the change device. The control device according to claim 14. 　 The computing device determining whether at least one of the position or the orientation of the target object held by the holding device should be changed; When it is determined that at least one of the position and the orientation of the target object should be changed, the control information is generated so that the target object held by the holding device is passed to the change device. The control device according to claim 15. 　 When it is determined that at least one of the position and the orientation of the target object should not be changed, the calculation device generates the control information so that the target object held by the holding device is processed without passing through the change device. The control device according to claim 15. 　 The computing device generates the control information so as to transfer the target object to the position changing device when at least one of the position or orientation of the target object held by the holding device differs from at least one of the desired position or orientation.

18. The control device according to claim 16 or 17. 　 The computing device Calculating at least one of a position and an orientation of at least a part of the change device based on an imaging result of at least a part of the change device; generating the control information based on a calculation result of at least one of a position and an orientation of at least a part of the modification device; 19. A control device according to any one of claims 1 to 18. 　 The calculation device generates the control information so that at least one of a positional relationship and an attitude relationship between at least a part of the change device and at least a part of the target object held by the holding device becomes a desired relationship.

20. The control device of claim 19. 　 The arithmetic device generates the control information so that the target object held in the holding device is handed over to the change device after at least one of a positional relationship and an attitude relationship between at least a part of the change device and at least a part of the target object held in the holding device has reached the desired relationship. The control device of claim 20. 　 The imaging system images a holding member provided in the change device for holding the target object as at least a part of the change device.

22. A control device according to any one of claims 19 to 21. 　 The imaging result is a result of imaging at least a part of the target object held by the holding device and at least a part of the modification device by the imaging system.

23. A control device according to any one of claims 1 to 22. 　 After at least a portion of the target object held by the holding device is imaged by the imaging system, at least a portion of the modification device is imaged by the imaging system.

24. The control device of claim 23. 　 After at least a portion of the target object held by the holding device is imaged by the imaging system, the holding device is moved, and after the movement, at least a portion of the changing device is imaged by the imaging system.

25. The control device of claim 24. 　 The imaging system images at least one of at least a part of the change device and at least a part of the target object held by the holding device during at least one of a first period in which the holding device moves toward a position where the change device receives the target object, and a second period in which the holding device, holding the target object, is positioned at a position for transferring the target object to the change device.

26. A control device according to any one of claims 1 to 25. 　 The arithmetic device is capable of generating the control information for at least one of the holding device and the robot that moves the holding device to perform processing, based on at least one of the imaging results of at least one of at least a part of the target object, the position and / or orientation of which has been changed by the changing device, and at least a part of the changing device, by the imaging system.

27. A control device according to any one of claims 1 to 26. 　 A control device is provided with a holding device that holds a target object, and generates control information for controlling at least one of a robot that moves the holding device and the holding device, a computing device that generates the control information; a communication device that outputs the control information generated by the arithmetic device; Equipped with The computing device generating the control information based on an imaging result obtained by an imaging system capturing at least one of at least a part of the target object, the position and / or orientation of which has been changed by a change device that changes at least one of the position and / or orientation of the target object, and at least a part of the change device; Control device. 　 The computing device is capable of generating control information for controlling at least one of the robot and the holding and placing device based on the imaging result.

29. The control device of claim 28. 　 The computing device calculating at least one of a position and an orientation of the target object held by the change device relative to the holding device based on the imaging result; The control information is generated based on a calculation result of at least one of a position and an orientation of the target object held by the change device relative to the holding device.

30. A control device according to claim 28 or 29. 　 The calculation device generates the control information so that at least one of a positional relationship and an attitude relationship between the target object held by the change device and the holding device becomes a desired relationship.

31. The control device of claim 30. 　 The arithmetic device generates the control information so that the holding device receives the target object held by the change device after at least one of a positional relationship and an attitude relationship between the target object held by the change device and the holding device has reached the desired relationship.

32. The control device of claim 31. 　 The imaging system images at least one of at least a part of the change device and at least a part of the target object held by the change device during at least one of a first period during which the change device moves toward a predetermined receiving position where the holding device should be located when receiving the target object held by the change device and a second period during which the holding device is located at the receiving position.

33. A control device according to any one of claims 28 to 32. 　 The imaging result is a result of imaging at least a part of the target object held by the holding device and at least a part of the modification device by the imaging system.

34. A control device according to any one of claims 28 to 33. 　 The imaging system is attached to the robot.

35. A control device according to any one of claims 1 to 34. 　 The target object is held by the holding device and then handed over to the changing device.

36. A control device according to any one of claims 1 to 35. 　 After the holding device receives the target object whose position or attitude has been changed by the change device, the calculation device generates the control information for at least one of the holding device and the robot that moves the holding device to perform processing based on an image capturing result of the target object held by the holding device by the imaging system.

37. A control device according to any one of claims 1 to 36. 　 The processing includes processing the target object held by the holding device.

38. The control device of claim 37. 　 A control device according to any one of claims 1 to 38; the imaging system; A control system comprising: 　 A control device according to any one of claims 1 to 38; the imaging system; The robot A robot system comprising:

1. A control method in which a holding device for holding a target object is provided, and control information is generated for controlling at least one of the holding device and a robot that moves the holding device, the method comprising: generating the control information based on an imaging result of at least one of at least a part of the target object and at least a part of the modification device by an imaging system before the target object held by the holding device is handed over to a modification device capable of modifying at least one of a position and an orientation of the target object; A control method comprising:

1. A control method in which a holding device for holding a target object is provided, and control information is generated for controlling at least one of the holding device and a robot that moves the holding device, the method comprising: After the holding device holds the target object, the target object held by the holding device is transferred to a change device capable of changing at least one of a position or an orientation of the target object; After the change device changes at least one of the position or the orientation of the target object handed over from the holding device, an imaging system generates the control information based on an imaging result of imaging at least one of at least a part of the change device and at least a part of the target object held by the change device; A control method comprising: A computer program that causes a computer to execute the control method according to claim 41 or 42. 　 a robot provided with a holding device for holding a target object and an imaging system, and moving the holding device and the imaging system; a change device capable of changing at least one of the position or the orientation of the target object received from the holding device; A robot system comprising: 　 Further comprising a control device, The control device generates control information for controlling at least one of the robot and the holding device based on the imaging result obtained by the imaging system.

45. The robotic system of claim 44. 　 the imaging system images at least one of at least a portion of the target object held by the holding device and at least a portion of the change device before the target object is transferred to the change device; The imaging results obtained by the imaging system are used to control the robot.

46. ​​A robotic system according to claim 44 or 45. 　 a control device capable of generating control information for controlling at least one of the robot and the holding device based on an image capturing result of at least a part of the target object held by the holding device by the imaging system, so that the holding device performs a retry process to hold a second target object after the holding device releases a first target object, which is the target object held by the holding device.

47. A robotic system according to any one of claims 44 to 46. 　 The first target object and the second target object are the same.

48. The robotic system of claim 47. 　 The control device acquires information about a change amount of at least one of a position and an orientation of the target object held by the holding device relative to the holding device, based on a plurality of imaging results obtained by the imaging system capturing images of at least a portion of the target object held by the holding device multiple times.

49. A robotic system according to claim 47 or 48. 　 The control device acquires information about a difference between at least one of a position and an orientation of the target object held by the holding device relative to the holding device and at least one of a target position and a target orientation of the target object held by the holding device relative to the holding device, based on the imaging result of at least a part of the target object held by the holding device.

50. The robotic system of any one of claims 47 to 49. 　 The control device determining whether the retry process is necessary based on the image capturing result of at least a part of the target object held by the holding device; If it is determined that the retry process is necessary, the control information is generated so that the retry process is performed.

51. The robotic system of any one of claims 47 to 50. 　 The control device When it is determined that the retry process is unnecessary, the control information is generated so that the target object held by the holding device is passed to the change device.

52. The robotic system of claim 51. 　 The control device determining whether at least one of the position or the orientation of the target object held by the holding device should be changed; When it is determined that at least one of the position and the orientation of the target object should be changed, the control information is generated so that the target object held by the holding device is passed to the change device.

53. The robotic system of claim 52. 　 When it is determined that at least one of the position and the orientation of the target object should not be changed, the control device generates the control information so that the target object held by the holding device is processed without passing through the change device.

54. The robotic system of claim 53. 　 The control device generates the control information so as to transfer the target object to the position change device when at least one of the position or the orientation of the target object held by the holding device is different from at least one of the desired position or the orientation.

55. A robotic system according to claim 53 or 54. 　 The apparatus further includes a control device that calculates at least one of a position and an orientation of at least a part of the modification device based on an imaging result of at least a part of the modification device by the imaging system, and generates the control information based on a calculation result of at least one of a position and an orientation of at least a part of the modification device.

56. The robotic system of any one of claims 44 to 55. 　 The control device generates the control information so that at least one of a positional relationship and an attitude relationship between at least a part of the change device and at least a part of the target object held by the holding device becomes a desired relationship.

57. The robotic system of claim 56. 　 The control device generates the control information so that the target object held in the holding device is handed over to the change device after at least one of a positional relationship and an attitude relationship between at least a part of the change device and at least a part of the target object held in the holding device has reached the desired relationship.

58. The robotic system of claim 57. 　 The imaging system images a holding member provided in the change device for holding the target object as at least a part of the change device.

59. A robotic system according to any one of claims 56 to 58. 　 The robot system further includes a control device that generates control information for controlling the robot based on the image capture result by the image capture system.

60. The robotic system of any one of claims 44 to 59. 　 the imaging system acquires an imaging result of at least one of at least a part of the change device and at least a part of the target object held by the change device after the change device changes the posture of the target object handed over from the holding device; The imaging results are used to control the robot.

61. The robotic system of any one of claims 44 to 60. 　 a control device that calculates at least one of a position and an orientation of the target object held by the change device relative to the holding device based on the imaging result, and generates control information for controlling at least one of the robot and the holding device based on the calculation result of at least one of the position and orientation of the target object held by the change device relative to the holding device.

62. The robotic system of claim 61. 　 The control device generates the control information so that at least one of a positional relationship and an attitude relationship between the target object held by the change device and the holding device becomes a desired relationship.

63. The robotic system of claim 62. 　 The control device generates the control information so that the holding device receives the target object held by the change device after at least one of a positional relationship and an attitude relationship between the target object held by the change device and the holding device has reached the desired relationship.

64. The robotic system of claim 63. 　 The imaging system images at least one of at least a part of the change device and at least a part of the target object held by the change device during at least one of a first period during which the change device moves toward a predetermined receiving position where the holding device should be located when receiving the target object held by the change device and a second period during which the holding device is located at the receiving position.

65. The robotic system of any one of claims 62 to 64. 　 A control device is provided with a holding device that holds a target object, and generates control information for controlling at least one of a robot that moves the holding device and the holding device, a computing device that generates the control information; a communication device that outputs the control information generated by the arithmetic device; Equipped with The computing device After performing an operation for holding the target object by the holding device, the control information is generated based on an imaging result of an area in which the target object is expected to exist when the target object is held by the holding device by an imaging system. Control device. 　 The region includes a region where a holding member provided in the holding device for holding the target object is present.

67. The control device of claim 66. 　 A control device is provided with a holding device that holds a target object, and generates control information for controlling at least one of a robot that moves the holding device and the holding device, a computing device that generates the control information; a communication device that outputs the control information generated by the arithmetic device; Equipped with The computing device After the holding device performs an operation to hold the target object, it is determined whether the holding device is holding the target object based on an image capture result by an image capture system. Control device. 　 The calculation device transfers the target object held by the holding device by the operation to a change device that can change at least one of the position and orientation of the target object, and after the holding device performs processing to receive the target object with at least one of the position and orientation changed from the change device, determines whether the holding device is holding the target object based on an imaging result by the imaging system.

69. The control device of claim 68. 　 The arithmetic device generates the control information based on a determination result of whether the holding device is holding the target object based on the imaging result in which at least a part of the holding device is imaged.

70. A control device according to any one of claims 66 to 69. 　 When it is determined that the holding device is not holding a first target object, the calculation device generates the control information so that the holding device performs processing for holding a second target object.

71. The control device of claim 70. 　 When the calculation device determines that the holding device is holding the target object, the calculation device generates the control information so that the holding device passes the target object held by the holding device to the change device.

72. A control device according to any one of claims 70 or 71.

Images