Control device, control system, robot system, control method, and computer program

Abstract

This control device controls a robot which is provided with a holding device for holding a target object and moves the holding device. The control device performs determination processing relating to a holding position by the holding device with respect to the target object, and generates a control signal such that the holding device approaches the holding position determined on the basis of a determination result in the determination processing. The determination processing includes: first determination processing for determining whether to set a first candidate position as a holding position; and second determination processing for determining whether to set a second candidate position, which is different from the first candidate position, as a holding position. The setting conditions set for the first determination processing are different from the setting conditions set for the second determination processing.

Description

Control device, control system, robot system, control method, and computer program

[0001] The present invention relates to the technical fields of a control device, a control system, a robot system, a control method, and a computer program that can generate a control signal for controlling a robot, for example.

[0002] An example of a control device for controlling a robot provided with a holding device capable of holding an object is described in Patent Document 1. Such a control device is required to appropriately control the robot.

[0003] US Patent Application Publication No. 2013 / 0230235

[0004] According to a first aspect, a control device is provided which is provided with a holding device for holding a target object and generates a control signal for controlling a robot which moves the holding device, the control device comprising: an arithmetic unit which generates the control signal; and a communication unit which outputs the control signal generated by the arithmetic unit, wherein the arithmetic unit performs a judgment process regarding the holding position of the holding device relative to the target object, and generates the control signal for controlling the robot so that the holding device approaches the holding position determined based on the judgment result of the judgment process, the judgment process including a first judgment process which judges whether a first candidate position set as a candidate for the holding position is to be the holding position, and a second judgment process which judges whether a second candidate position set as a candidate for the holding position, different from the first candidate position, is to be the holding position, and a control device is provided in which the first setting condition, which is a setting condition set for the first judgment process, is different from the second setting condition, which is a setting condition set for the second judgment process.

[0005] According to a second aspect, there is provided a control system including the control device provided by the first aspect and an imaging system.

[0006] According to a third aspect, there is provided a robot system including the control device provided by the first aspect, an imaging system, and the robot.

[0007] According to a fourth aspect, a control method is provided which is provided with a holding device for holding a target object and generates a control signal for controlling a robot which moves the holding device, the control method including: performing a determination process regarding a holding position of the holding device relative to the target object; and generating the control signal for controlling the robot so that the holding device approaches the holding position determined based on the determination result of the determination process; the determination process including a first determination process which determines whether a first candidate position set as a candidate for the holding position is to be the holding position; and a second determination process which determines whether a second candidate position set as a candidate for the holding position, different from the first candidate position, is to be the holding position; and a control method is provided in which the first setting condition set for the first determination process is different from the second setting condition set for the second determination process.

[0008] According to a fifth aspect, there is provided a computer program that causes a computer to execute the control method provided by the fourth aspect.

[0009] According to a sixth aspect, there is provided a control device comprising: a calculation device that generates a condition display signal for controlling a display device to display on the display device setting conditions set for an interference detection process that determines interference between at least one of a holding device provided on a robot and the robot and a non-held object different from a target object to be held by the holding device, the setting conditions being a third setting condition set for a first interference detection process that determines whether or not at least one of the holding device and the robot will interfere with the non-held object when it is assumed that the robot operates so that the holding device will approach a first candidate position that is set as a candidate for a holding position by the holding device for the target object; and a fourth setting condition that is different from the third setting condition and is set for a second interference detection process that determines whether or not at least one of the holding device and the robot will interfere with the non-held object when it is assumed that the robot operates so that the holding device will approach a second candidate position that is set as a candidate for the holding position and is different from the first candidate position,

[0010] According to a seventh aspect, there is provided a control system including the control device provided by the sixth aspect and an imaging system.

[0011] According to an eighth aspect, there is provided a robot system including the control device provided by the sixth aspect, an imaging system, and the robot.

[0012] According to a ninth aspect, there is provided a control method including generating a condition display signal for controlling a display device so that setting conditions set for an interference detection process for determining interference between at least one of a holding device provided on a robot and the robot and a non-held object different from a target object to be held by the holding device, the setting conditions being a third setting condition set for a first interference detection process for determining whether or not at least one of the holding device and the robot will interfere with the non-held object when it is assumed that the robot operates so that the holding device will approach a first candidate position set as a candidate for a holding position by the holding device for the target object, and a fourth setting condition different from the third setting condition set for a second interference detection process for determining whether or not at least one of the holding device and the robot will interfere with the non-held object when it is assumed that the robot operates so that the holding device will approach a second candidate position set as a candidate for the holding position and different from the first candidate position, and outputting the generated condition display signal to the display device.

[0013] According to a tenth aspect, there is provided a computer program for causing a computer to execute the control method provided by the ninth aspect.

[0014] According to an eleventh aspect, there is provided a control device comprising: a calculation device that generates a condition display signal for controlling a display device so that the display device displays setting conditions set for a judgment process regarding a holding position by a holding device provided on a robot for a target object to be processed by the holding device, the setting condition being a first setting condition set for a first judgment process that judges whether a first candidate position set as a candidate for the holding position is to be the holding position; and a second setting condition that is set for a second judgment process that judges whether a second candidate position set as a candidate for the holding position, different from the first candidate position, is to be the holding position, and that is different from the first setting condition; and a communication device that outputs the condition display signal generated by the calculation device to the display device.

[0015] According to a twelfth aspect, there is provided a control system including the control device provided by the eleventh aspect and an imaging system.

[0016] According to a thirteenth aspect, there is provided a robot system including the control device provided by the eleventh aspect, an imaging system, and the robot.

[0017] According to a fourteenth aspect, there is provided a control method including generating a condition display signal for controlling a display device so that setting conditions set for a judgment process regarding a holding position by a holding device provided on a robot for a target object to be processed by the holding device, the first setting condition being set for a first judgment process that judges whether a first candidate position set as a candidate for the holding position is to be the holding position, and a second setting condition that is set for a second judgment process that judges whether a second candidate position set as a candidate for the holding position, different from the first candidate position, is to be the holding position, and that is different from the first setting condition, and outputting the generated condition display signal to the display device.

[0018] According to a fifteenth aspect, there is provided a computer program for causing a computer to execute the control method provided by the fourteenth aspect.

[0019] The functions and other advantages of the present invention will become apparent from the following detailed description of the preferred embodiments.

[0020] FIG. 1 is a block diagram showing the configuration of a robot system according to this embodiment. FIG. 2 is a side view showing the appearance of a robot according to this embodiment. FIG. 3 is a block diagram showing the configuration of a control device according to this embodiment. FIG. 4 is a block diagram showing the configuration of a robot control device according to this embodiment. FIGS. 5(a) to 5(c) are side views showing the positional relationship between a robot and a workpiece at a certain point during a holding process for holding a workpiece placed on a mounting device. FIG. 6 is a flowchart showing the overall flow of the robot control process. FIG. 7 shows an example of a holding position registration screen. FIGS. 8(a) to 8(e) are cross-sectional views showing holding positions (holding candidate positions HCP) indicated by holding position information using an end effector and a target object. FIG. 9 is a flowchart showing the overall flow of the holding control process. FIG. 10 is a flowchart showing the process of selecting a holding target position in step S23 of FIG. 9. FIG. 11 conceptually shows an end effector approaching a holding candidate position HCP. FIG. 12 conceptually illustrates an end effector model and a robot model that move in accordance with the movement of the end effector and the robot, assuming that at least one of the robot and the robot movable device operates so that the end effector approaches a holding candidate position HCP. FIG. 13 conceptually illustrates an end effector approaching a holding candidate position HCP. FIGS. 14(a) and 14(b) each illustrate a decision space together with a holding candidate position. FIGS. 15(a) and 15(b) each illustrate a decision space together with a holding candidate position. FIGS. 16(a) and 16(b) each illustrate a decision space together with a holding candidate position. FIGS. 17(a) and 17(b) each illustrate a decision space together with a holding candidate position. FIGS. 18(a) and 18(b) each illustrate a decision space together with a holding candidate position. FIGS. 19(a) and 19(b) each illustrate a decision space together with a holding candidate position. Fig. 20(a) and Fig. 20(b) each show a determination space together with a candidate holding position. Fig. 22 shows an example of a condition setting screen. Fig. 22 shows an example of support information displayed together with the condition setting screen. Fig. 23 shows overlap information, which is an example of support information. Fig. 24(a) and Fig. 24(b) each show overlap information, which is an example of support information. Fig. 25 shows accuracy information, which is an example of support information.Fig. 26(a) and Fig. 26(b) each show accuracy information, which is an example of support information. Fig. 27 shows recommended condition information, which is an example of support information. Fig. 28 shows a calculation model. Fig. 29(a) and Fig. 29(b) each show an example of a candidate placement position. Fig. 30 shows an example in which a workpiece W, which is an example of a first target object, is placed on a placement device T, which is an example of a second target object. Fig. 31 is a block diagram showing the configuration of a robot system equipped with a measurement system.

[0021] Next, embodiments of a control device, a control system, a robot system, a control method, and a computer program will be described with reference to the drawings. Hereinafter, the embodiments of a control device, a control system, a robot system, a control method, and a computer program will be described using a robot system SYS.

[0022] (1) Configuration of the Robot System SYS First, the configuration of the robot system SYS will be described.

[0023] (1-1) Overall Configuration of Robot System SYS First, the overall configuration of the robot system SYS will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the overall configuration of the robot system SYS.

[0024] 1, the robot system SYS includes a robot 1, an imaging system 2, a control device 3, and an end effector 4. The imaging system 2 may also be referred to as an imaging unit.

[0025] The robot 1 is a device capable of performing a predetermined process on a target object OBJ. An example of the robot 1 is shown in FIG. 2. FIG. 2 is a side view showing the appearance of the robot 1. As shown in FIG. 2, the robot 1 includes, for example, a base 11, a robot arm 12, and a robot control device 13.

[0026] The base 11 is a component that forms the base of the robot 1. The base 11 is placed on a support surface S such as a floor surface. The base 11 may be fixed to the support surface S. Alternatively, the base 11 may be movable relative to the support surface S. Note that FIG. 2 shows an example in which the base 11 is fixed to the support surface S.

[0027] The robot arm 12 is attached to the base 11. The robot arm 12 is a device in which a plurality of links 121 are connected via joints 122. An actuator is built into the joint 122. The link 121 may be rotatable around an axis defined by the joint 122 by the actuator built into the joint 122. At least one link 121 may be extendable and contractible along the direction in which the link 121 extends. A device including the device in which a plurality of links 121 are connected via joints 122 and the base 11 may be referred to as the robot arm 12.

[0028] An end effector 4 is attached to the robot arm 12. That is, the end effector 4 is attached to the robot 1. In the example shown in FIG. 2 , the end effector 4 is attached to the tip of the robot arm 12. The end effector 4 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the end effector 4. That is, the robot 1 moves the end effector 4.

[0029] The end effector 4 is a device that performs a predetermined process (in other words, a predetermined operation) on the target object OBJ. The end effector 4 that performs a predetermined process on the target object OBJ may also be called a processing device.

[0030] For example, the end effector 4 may perform a holding process to hold the target object OBJ as an example of the predetermined process. In this case, the end effector 4 may be considered to be performing the holding process on the target object OBJ that the end effector 4 is to hold. An end effector 4 capable of performing the holding process may be referred to as a holding device. Note that holding the target object OBJ may be considered to be equivalent to picking up the target object OBJ. The holding process of holding the target object OBJ may be considered to be equivalent to a pick-up process of picking up the target object OBJ.

[0031] For example, holding the target object OBJ may include grasping the target object OBJ. For example, holding the target object OBJ may include gripping the target object OBJ using a hand gripper, which is an example of the end effector 4. Holding the target object OBJ may include attracting the target object OBJ. For example, holding the target object OBJ may include attracting (vacuum suction) the target object OBJ using a vacuum gripper, which is an example of the end effector 4. For example, holding the target object OBJ may include attracting (vacuum suction) the target object OBJ using a magnetic gripper (e.g., a magnetically attractive gripper), which is an example of the end effector 4.

[0032] For example, as an example of the predetermined process, the end effector 4 may perform a release process (in other words, a release operation) to release (i.e., let go of) the target object OBJ that it is holding. In this case, the end effector 4 may be considered to be performing the release process on the target object OBJ that it is holding. An end effector 4 that is capable of performing the release process may be called a release device. The release process may also be called a placement process.

[0033] A hand gripper is an example of an end effector 4 capable of holding and releasing an object OBJ. A hand gripper is an end effector 4 that can hold (e.g., grasp) a target object OBJ by physically pinching the target object OBJ using multiple (e.g., two, three, or four) finger or claw members. Another example of an end effector 4 capable of holding and releasing an object OBJ is a vacuum gripper (e.g., a vacuum suction gripper). A vacuum gripper is an end effector 4 that can hold (e.g., attract) a target object OBJ by vacuum suction. Another example of an end effector 4 capable of holding and releasing an object OBJ is a magnetic gripper (e.g., a magnetic suction gripper). FIG. 2 illustrates an example in which the end effector 4 is a hand gripper. However, the end effector 4 capable of holding and releasing an object OBJ is not limited to the above example, and may be any other existing end effector capable of holding and releasing an object OBJ. For example, the end effector 4 may be a Bernoulli chuck capable of holding the target object OBJ in a non-contact manner. Note that the end effector 4 capable of performing at least one of the holding process and the release process is not limited to the end effector described above, and may be another existing end effector.

[0034] The robot 1 may perform, as an example of a predetermined process, a placement process (in other words, a placement operation) for placing a target object OBJ at a desired position using the end effector 4 capable of performing a holding process and a release process. For example, the robot 1 may use the end effector 4 to hold a first target object OBJ that is a first example of the target object OBJ, and then perform a placement process for placing the first target object OBJ held by the end effector 4 at a desired position of a second target object OBJ that is a second example of the target object OBJ and different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process for the second target object OBJ on which the end effector 4 is to place the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process for the first target object OBJ that the end effector 4 is to release.

[0035] The robot 1 may use the end effector 4 capable of holding and releasing operations to perform a fitting process (i.e., a fitting operation) for fitting a first target object OBJ into a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (i.e., a placement operation). The fitting process may include, for example, a process for fitting (i.e., inserting) the first target object OBJ (e.g., a convex portion of the first target object OBJ) into a concave portion (e.g., a hole) formed in the second target object OBJ. The fitting process may include, for example, a process for fitting the first target object OBJ (e.g., a concave portion of the first target object OBJ) into a convex portion (e.g., a rod) formed in the second target object OBJ. In this case, the second target object OBJ may be an object (workpiece) into which the first target object OBJ is to be fitted.

[0036] The robot 1 may use the end effector 4 capable of performing a holding process and a release process to perform a placing process (in other words, a placing operation) for placing a first target object OBJ on a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placing operation). In this case, the second target object OBJ may be an object (workpiece) onto which the first target object OBJ is to be placed. The robot 1 may use the end effector 4 capable of performing a holding process and a release process to perform a pasting process (in other words, a pasting operation) for pasting the first target object OBJ on a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placing operation). In this case, the second target object OBJ may be an object (workpiece) onto which the first target object OBJ is to be pasted. The robot 1 may use the end effector 4 capable of holding and releasing processes to perform a bonding process (in other words, a bonding operation) for bonding a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation). In this case, the second target object OBJ may be an object (workpiece) to which the first target object OBJ is to be bonded. Alternatively, another end effector capable of dispensing adhesive for the bonding process may be provided on the robot 1 or another robot. The robot 1 may use the end effector 4 capable of holding and releasing processes to perform a welding process (in other words, a welding operation) for welding the first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation). In this case, the second target object OBJ may be an object (workpiece) to which the first target object OBJ is to be welded. Furthermore, another end effector (for example, a processing device for welding with an energy beam) for welding the first target object OBJ and the second target object OBJ may be provided on the robot 1 or another robot.The robot 1 may use the end effector 4 capable of holding and releasing operations to perform a screw tightening operation (i.e., a screw tightening operation) for tightening a first target object OBJ capable of functioning as a screw into a screw hole formed in a second target object OBJ different from the first target object OBJ, as a specific example of a placement operation (i.e., a placement operation). In this case, the first target object OBJ may be a screw member capable of functioning as a screw, such as a bolt or a nut, and the second target object OBJ may be an object (workpiece) to which the first target object OBJ is to be tightened. Furthermore, the end effector 4 may be a tool such as a screwdriver capable of tightening screws. At least one of the adhering operation, the bonding operation, the welding operation, and the screw tightening operation may be referred to as a processing operation.

[0037] The robot 1 may use an end effector 4 capable of holding and releasing processes to perform a process for discarding the target object OBJ (in other words, a discarding operation) as a specific example of a placement process (in other words, a placement operation).

[0038] The end effector 4 may perform a predetermined process on each of the multiple target objects OBJ. That is, the end effector 4 may perform the predetermined process on the multiple target objects OBJ in sequence. In this case, the robot 1 may move the end effector 4 to a first position where the end effector 4 can perform the predetermined process on a first target object OBJ, and after the end effector 4 has moved to the first position, the end effector 4 may perform the predetermined process on the first target object OBJ. Thereafter, the robot 1 may move the end effector 4 to a second position where the end effector 4 can perform the predetermined process on a second target object OBJ different from the first target object OBJ, and after the end effector 4 has moved to the second position, the end effector 4 may perform the predetermined process on the second target object OBJ.

[0039] For example, the end effector 4 may perform a predetermined process on each of multiple portions of a single target object OBJ. That is, the end effector 4 may sequentially perform a predetermined process on multiple portions of a single target object OBJ. In this case, the robot 1 may move the end effector 4 to a third position where the end effector 4 can perform a predetermined process on a first portion of the target object OBJ, and after the end effector 4 has moved to the third position, the end effector 4 may perform the predetermined process on the first portion of the target object OBJ. Thereafter, the robot 1 may move the end effector 4 to a fourth position where the end effector 4 can perform a predetermined process on a second portion of the target object OBJ, and after the end effector 4 has moved to the fourth position, the end effector 4 may perform the predetermined process on the second portion of the target object OBJ.

[0040] As shown in FIG. 2 , the target object OBJ on which the end effector 4 performs a predetermined process may include a workpiece W. The workpiece W may include, for example, a part or member used to manufacture a desired product. The workpiece W may include, for example, a part or member to be processed to manufacture the desired product. The workpiece W may include, for example, a part or member to be transported to manufacture the desired product. The workpiece W may include, for example, a part or member that is moving due to transportation to manufacture the desired product. The workpiece W may include, for example, a part or member that is moving to manufacture the desired product.

[0041] As shown in FIG. 2 , the target object OBJ on which the end effector 4 performs a predetermined process may include a placement device T on which a workpiece W is placed. An example of the placement device T is a container (storage box) CB. The container CB may have a bottom wall BS and a side wall SS protruding upward from the bottom wall BS. The placement device T may be a placement device T in which the workpiece W is placed on the bottom wall BS in a storage space SP enclosed by the bottom wall BS and the side wall SS. The container CB may also have a bottom wall BS and a side wall SS protruding upward from the bottom wall BS. The placement device T may be a placement device T capable of accommodating the workpiece W in the storage space SP enclosed by the bottom wall BS and the side wall SS. However, the container CB does not necessarily have a side wall SS. A container CB without a side wall SS may be referred to as a pallet. Furthermore, the placement device T is not limited to a container CB or a pallet, but may also be an existing object on which the workpiece W can be placed. The mounting device T may also be referred to as a mounting member.

[0042] The placement device T may be disposed on a support surface S. The placement device T may be fixed to the support surface S. Alternatively, at least a portion of the placement device T may be movable relative to the support surface S. A first example in which at least a portion of the placement device T is movable relative to the support surface S is when the placement device T is supported by a transport device that can move (in other words, transport) the placement device T. In this case, the transport device may be, for example, a belt conveyor. A second example in which at least a portion of the placement device T is movable relative to the support surface S is when the placement device T is supported by a movable device (movable placement device) that can move the placement device T. An example of the movable placement device is at least one of an automatic guided vehicle (AGV), an autonomous mobile robot, an unmanned aerial vehicle (e.g., a drone), and a submarine. A third example in which at least a part of the mounting device T is movable relative to the support surface S is an example in which the mounting device T functions as a movable mounting device. That is, a third example in which at least a part of the mounting device T is movable relative to the support surface S is an example in which a movable mounting device is used as the mounting device T. Note that FIG. 2 shows an example in which the mounting device T is self-propelled on the support surface S.

[0043] The robot control device 13 controls the operation of the robot 1 .

[0044] Specifically, the robot control device 13 may control the movement of the robot arm 12. For example, the robot control device 13 may control the movement of the robot arm 12 so that a desired link 121 rotates around an axis defined by a desired joint 122. For example, the robot control device 13 may control the movement of the robot arm 12 so that the end effector 4 attached to the robot arm 12 is positioned at a desired position. For example, the robot control device 13 may control the movement of the robot arm 12 so that the end effector 4 attached to the robot arm 12 moves to a desired position.

[0045] In addition to controlling the operation of the robot 1, the robot control device 13 may also control the operation of the end effector 4 attached to the robot 1 (processing performed by the end effector 4). For example, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 holds the target object OBJ at a desired timing. That is, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 performs a holding process at a desired timing. For example, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 releases the held target object OBJ at a desired timing. That is, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 performs a release process at a desired timing. In order for the end effector 4 to hold or release the target object OBJ, if the end effector 4 is a hand gripper, the robot control device 13 may control the timing of opening and closing the hand gripper. If the end effector 4 is a vacuum gripper, the robot control device 13 may control the timing of turning on / off the vacuum device of the vacuum gripper (or the vacuum suction force).If the end effector 4 is a magnetic gripper, the robot control device 13 may control the timing of turning on / off the magnetic suction device of the magnetic gripper (or the magnetic force).

[0046] Note that FIG. 2 shows an example in which the robot 1 is a robot arm 12 (i.e., a vertical articulated robot). However, the robot 1 may be a robot different from a vertical articulated robot. For example, the robot 1 may be a SCARA robot (i.e., a horizontal articulated robot). For example, the robot 1 may be a parallel link robot. For example, the robot 1 may be a dual-arm robot having two robot arms 12. For example, the robot 1 may be a Cartesian coordinate robot. For example, the robot 1 may be a cylindrical coordinate robot.

[0047] The robot 1 may be installed on a movable device (robot movable device) that can move the robot 1. Examples of the robot movable device include at least one of an automatic guided vehicle (AGV), an autonomous mobile robot, an unmanned aerial vehicle (e.g., a drone), and a submarine. When the robot 1 is installed on a robot movable device different from the robot 1, a device including the robot 1 and the robot movable device on which the robot 1 is installed may be referred to as a movable device (robot movable device). When the robot 1 is installed on a robot movable device, the robot control device 13 may control the operation of the robot movable device on which the robot 1 is installed in addition to controlling the operation of the robot 1. Note that since the robot 1 moves the robot arm 12, the robot 1 itself may be considered to be a movable device.

[0048] 1, the imaging system 2 captures an image of a target object OBJ. To capture an image of the target object OBJ, the imaging system 2 includes an imaging device 21 and an illumination device 23.

[0049] The imaging device 21 is a camera capable of capturing an image of the target object OBJ. For example, the imaging device 21 may capture an image of the target object OBJ under the control of the control device 3. The imaging device 21 generates image data IMG by capturing an image of the target object OBJ. In other words, the imaging device 21 generates image data IMG that is the result of capturing the image of the target object OBJ. The image data IMG generated by the imaging device 21 is output from the imaging device 21 to the control device 3. As a result, the control device 3 acquires the image data IMG acquired by the imaging device 21 capturing an image of the target object OBJ.

[0050] The imaging system 2 may include an imaging device 21 including a monocular camera. In this case, the imaging device 21 may generate image data IMG including one image data item generated by the monocular camera. Alternatively, the imaging system 2 may include an imaging device 21 including a stereo camera including two monocular cameras. In this case, the imaging device 21 may generate image data IMG including two image data items generated by the two monocular cameras.

[0051] Alternatively, the imaging system 2 may include an imaging device 21 including a first monocular camera and a stereo camera including two second monocular cameras different from the first monocular camera. In this case, the imaging device 21 may generate image data IMG including one image data generated by the first monocular camera and image data IMG including two image data generated by the two second monocular cameras. In this case, the imaging system 2 can also be said to be a system including a first imaging device including the first monocular camera and a second imaging device including a stereo camera.

[0052] The imaging system 2 may include an imaging device 21 that includes a camera other than a monocular camera and a stereo camera. For example, the imaging system 2 may include an imaging device 21 that includes three or more monocular cameras. For example, the imaging system 2 may include an imaging device 21 that includes at least one of a light field camera, a plenoptic camera, and a multispectral camera.

[0053] The imaging device 21 may capture an image of the entire target object OBJ. Alternatively, the imaging device 21 may capture an image of a portion of the target object OBJ. In other words, the imaging device 21 may capture an image of a portion of the target object OBJ while not capturing an image of another portion of the target object OBJ.

[0054] The imaging device 21 may capture an image of a single target object OBJ. In other words, a single target object OBJ may appear in the image represented by the image data IMG. Alternatively, the imaging device 21 may capture an image of multiple target objects OBJ. In other words, a plurality of target objects OBJ may appear in the image represented by the image data IMG. In this case, as will be described in detail later, the control device 3 may determine (in other words, select) one of the multiple target objects OBJ captured by the imaging device 21 as the target object OBJ on which the end effector 4 will actually perform a predetermined process. Note that one of the multiple target objects OBJ captured by the imaging device 21 on which the end effector 4 will actually perform a predetermined process may be referred to as a process execution object.

[0055] When the imaging device 21 captures images of multiple target objects OBJ, the multiple target objects OBJ captured by the imaging device 21 may be arranged such that at least two of the multiple target objects OBJ at least partially overlap. As an example, when the target objects OBJ are workpieces W corresponding to components used to manufacture a desired product, the multiple workpieces W (i.e., multiple components) may be arranged such that at least two of the multiple workpieces W at least partially overlap. In this case, the end effector 4 may perform the above-described holding process of holding at least one workpiece W among the multiple workpieces W that are randomly arranged. In other words, the robot 1 may perform bulk picking, in which workpieces W are picked one by one from the multiple workpieces W that are randomly arranged. Note that the multiple workpieces W that are randomly arranged may also be referred to as multiple workpieces W that are irregularly arranged, multiple workpieces W that are casually arranged, or multiple workpieces W that are randomly arranged.

[0056] In this embodiment, the state of "two objects overlapping" may include a state of "two objects overlapping while being in contact with each other." The state of "two objects overlapping" may include a state of "two objects overlapping without being in contact with each other." The state of "two objects overlapping" may include a state of "two objects entangled." The state of "two objects overlapping" may include a state in which "one of the two objects at least partially covers the other of the two objects." The state of "two objects overlapping" may include a state in which "one of the two objects is located on at least a portion of the other of the two objects."

[0057] Alternatively, the multiple target objects OBJ captured by the imaging device 21 may be arranged regularly. As an example, if the target objects OBJ are workpieces W corresponding to parts used to manufacture a desired product, the multiple workpieces W (i.e., multiple parts) may be arranged in a matrix. In this case, the end effector 4 may perform the above-mentioned holding process of holding at least one workpiece W among the multiple regularly arranged workpieces W. In other words, the robot 1 may pick up a workpiece W one by one from the multiple regularly arranged workpieces W. Note that the multiple regularly arranged workpieces W can be rephrased as multiple workpieces W that are orderly arranged, or multiple workpieces W that are arranged according to a certain arrangement rule.

[0058] Note that the placement process for placing the target object OBJ held by the end effector 4 may include a placement process for placing the target object OBJ randomly (in other words, haphazardly), as with the holding process, or a placement process for placing the target object OBJ regularly (in other words, orderly). For example, the robot 1 may use the end effector 4 to hold one of a plurality of workpieces W that are regularly or randomly placed in the first container CB, and then place the one workpiece W held by the end effector 4 regularly or randomly in the second container CB.

[0059] The illumination device 23 is a device capable of irradiating illumination light onto the target object OBJ (for example, at least one target object OBJ when multiple target objects OBJ exist). For example, the illumination device 23 may irradiate the target object OBJ with illumination light under the control of the control device 3. In particular, the illumination device 23 is a device capable of irradiating the target object OBJ with illumination light, thereby illuminating the target object OBJ with illumination light. In this case, the imaging device 21 may capture an image of the target object OBJ illuminated with illumination light. However, the illumination device 23 does not need to irradiate the target object OBJ with illumination light. In this case, the imaging system 2 (robot system SYS) does not need to be equipped with the illumination device 23.

[0060] When the imaging device 21 includes a stereo camera, the illumination device 23 may be a device capable of projecting a desired projection pattern onto the target object OBJ by irradiating the target object OBJ with illumination light. The desired projection pattern may include, for example, a random pattern. The random pattern may include a random dot pattern. The desired projection pattern may include, for example, a one-dimensional or two-dimensional grid pattern. The desired projection pattern may include, for example, a line pattern. The desired projection pattern may include, for example, a stripe pattern. The desired projection pattern may include other projection patterns. However, even when the imaging device 21 does not include a stereo camera (i.e., the imaging device 21 includes a monocular camera), the illumination device 23 may project the desired projection pattern onto the target object OBJ by irradiating the target object OBJ with illumination light. In this case, the illumination device 23 may irradiate the target object OBJ with illumination light having a uniform intensity distribution (e.g., a uniform intensity distribution). The desired projection pattern may also be referred to as light having a desired intensity distribution.

[0061] The imaging system 2 is attached to the robot arm 12, similar to the end effector 4. That is, the imaging device 21 and the lighting device 23 are attached to the robot arm 12. For example, as shown in FIG. 2 , the imaging device 21 and the lighting device 23 may be attached to the tip of the robot arm 12, similar to the end effector 4. In this case, the imaging device 21 and the lighting device 23 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the imaging device 21 and the lighting device 23.

[0062] However, the imaging system 2 does not have to be attached to the robot arm 12. For example, the imaging system 2 may be disposed at any position where it can capture an image of the target object OBJ. Furthermore, for example, the imaging system 2 may be disposed at any position where it can irradiate the target object OBJ with illumination light. For example, the imaging system 2 may be attached to a structure such as a pillar. When the robot 1 is installed on a robot movable device as described above, the imaging system 2 may be installed on the robot movable device on which the robot 1 is installed. Note that either one of the imaging device 21 and the lighting device 23 may be attached to the robot arm 12, and the other of the imaging device 21 and the lighting device 23 may be attached to a location different from the robot arm 12 (for example, a structure such as a pillar or a robot movable device).

[0063] The imaging device 21 may capture an image of the target object OBJ (e.g., at least one target object OBJ when multiple target objects OBJ are present) during a period in which the imaging device 21 and the target object OBJ are displaced relative to each other. Note that the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may refer to a state in which the relative positional relationship between the imaging device 21 and the target object OBJ is changing. The state in which the imaging device 21 and the target object OBJ are displaced relative to each other may refer to a state in which the imaging device 21 and the target object OBJ are moving relative to each other. For example, the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the target object OBJ is moving relative to the imaging device 21. For example, the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the imaging device 21 is moving relative to the target object OBJ. In this case, the imaging device 21 does not need to be stationary to capture an image of the target object OBJ, and the robot system SYS can efficiently perform predetermined processing on the target object OBJ using the end effector 4.

[0064] Alternatively, the imaging device 21 may capture an image of the target object OBJ during a period in which there is no relative displacement between the imaging device 21 and the target object OBJ. Note that the state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the relative positional relationship between the imaging device 21 and the target object OBJ is not changing. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which there is no relative movement between the imaging device 21 and the target object OBJ. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the imaging device 21 and the target object OBJ are stationary. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the imaging device 21 and the target object OBJ are moving at the same moving speed in the same moving direction.

[0065] The control device 3 performs robot control processing. The robot control processing includes processing for generating a robot control signal for controlling the robot 1. Specifically, the control device 3 generates the robot control signal based on image data IMG output from the imaging system 2 (e.g., the imaging device 21). In this embodiment, the control device 3 calculates at least one of the position and orientation of the target object OBJ in the global coordinate system of the robot system SYS based on the image data IMG, and generates the robot control signal based on the calculated at least one of the position and orientation of the target object OBJ.

[0066] The global coordinate system is a coordinate system that serves as the reference for the robot system SYS. For example, the global coordinate system may be a coordinate system that serves as the reference for the robot 1. The global coordinate system can also be said to be a coordinate system used to control the robot 1. For example, a world coordinate system that is defined based on the support surface S on which the robot 1 is placed may be used as the global coordinate system. In other words, a world coordinate system that is fixed with respect to the support surface S on which the robot 1 is placed may be used as the global coordinate system. In the following description, unless otherwise specified, the X-axis, Y-axis, and Z-axis may refer to the X-axis, Y-axis, and Z-axis in the global coordinate system, respectively.

[0067] However, the control device 3 may calculate at least one of the position and orientation of the target object OBJ in a coordinate system different from the global coordinate system based on the image data IMG. The coordinate system different from the global coordinate system may include at least one of the robot coordinate system and the imaging coordinate system. The robot coordinate system may be a coordinate system defined based on the robot 1. That is, the robot coordinate system may be a coordinate system fixed with respect to the robot 1 (for example, fixed with respect to the base 11 of the robot 1). The imaging coordinate system may be a coordinate system defined based on the imaging device 21. That is, the imaging coordinate system may be a coordinate system fixed with respect to the imaging device 21. An example of the imaging coordinate system is a coordinate system defined based on the optical axis AX21 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 21. An example of the imaging coordinate system is a coordinate system in which one of the three coordinate axes constituting the imaging coordinate system is an axis along the optical axis AX21 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 21.

[0068] The control device 3 may perform end effector control processing in addition to or instead of performing robot control processing. The end effector control processing may include processing for generating an end effector control signal for controlling the end effector 4. For example, the control device 3 may generate the end effector control signal based on image data IMG output from the imaging system 2 (e.g., the imaging device 21). Specifically, the control device 3 may generate the end effector control signal based on at least one of the position and orientation of the target object OBJ calculated from the image data IMG.

[0069] The end effector control processing may or may not be included in the robot control processing. In other words, the end effector control signal generated by the control device 3 may or may not be included in the robot control signal. In the following description, for convenience of explanation, an example will be described in which the end effector control processing is included in the robot control processing (in other words, the end effector control signal is included in the robot control signal). Therefore, in the following description, the robot control processing may mean processing for generating at least one of a robot control signal and an end effector control signal. In addition, in the following description, the robot control signal may mean at least one of a signal for controlling the robot 1 and a signal for controlling the end effector 4. The robot control signal may also be simply referred to as a control signal.

[0070] As described above, when the robot 1 is installed on a robotic movable device (e.g., at least one of an automatic guided vehicle, an autonomously traveling transport robot, an unmanned aerial vehicle, and a submarine), the control device 3 may perform a movable device control process in addition to or instead of performing a robot control process. The movable device control process may include a process of generating a movable device control signal for controlling the robotic movable device. For example, the control device 3 may generate the movable device control signal based on image data IMG output from the imaging system 2 (e.g., the imaging device 21). Specifically, the control device 3 may generate the movable device control signal based on at least one of the position and orientation of the target object OBJ calculated from the image data IMG.

[0071] The movable device control process may or may not be included in the robot control process. In other words, the movable device control signal generated by the control device 3 may or may not be included in the robot control signal. In the following description, for convenience of explanation, an example will be described in which the movable device control process is included in the robot control process (i.e., the movable device control signal is included in the robot control signal). Therefore, in the following description, the robot control process may mean a process for generating at least one of a robot control signal, an end effector control signal, and a movable device control signal. In addition, in the following description, the robot control signal may mean at least one of a signal for controlling the robot 1, a signal for controlling the end effector 4, and a signal for controlling the robot movable device.

[0072] In this way, the control device 3 and the imaging system 2 are used to control the robot 1. Therefore, a system including the control device 3 and the imaging system 2 may be referred to as a robot control system or a control system.

[0073] The robot control signal generated by the control device 3 is output to the robot control device 13 of the robot 1. The robot control device 13 controls the operation of the robot 1 based on the robot control signal generated by the control device 3. For this reason, the robot control signal may include a signal for controlling the operation of the robot 1.

[0074] As described above, when the robot control signal includes a signal for controlling the robot arm 12, the robot control device 13 may control the robot arm 12 based on the robot control signal. For example, the robot control device 13 may control the operation of the actuator built into the joint 122 based on the robot control signal, thereby controlling the operation of the robot arm 12.

[0075] For example, as described above, the robot arm 12 moves the end effector 4. In this case, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at a desired position. The robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves to a desired position. The robot control signal may include a signal for controlling the robot arm 12 so that the positional relationship between the end effector 4 and the target object OBJ is a desired positional relationship. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 is located at a desired position. The robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves to a desired position. The robot control device 13 may control the robot arm 12 based on the robot control signal so that the positional relationship between the end effector 4 and the target object OBJ is a desired positional relationship.

[0076] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves toward (i.e., approaches) a first desired position where the end effector 4 can hold the target object OBJ. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at the first desired position. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves toward (i.e., approaches) the first desired position. That is, the robot control signal may control the robot arm 12 so that the end effector 4 is located at the first desired position. Note that, because the end effector 4 located at the first desired position holds the target object OBJ, controlling the robot arm 12 so that the end effector 4 approaches the first desired position may be considered equivalent to controlling the robot arm 12 so that the end effector 4 approaches the target object OBJ that is to be held at the first desired position.

[0077] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the robot arm 12 so that the posture of the end effector 4 becomes a first desired posture that enables the end effector 4 to hold the target object OBJ. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the posture of the end effector 4 becomes the first desired posture.

[0078] As another example, when performing a release process to release the target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves toward (i.e., approaches) a second desired position where the target object OBJ held by the end effector 4 should be released. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at the second desired position. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves toward (i.e., approaches) the second desired position. That is, the robot control signal may control the robot arm 12 so that the end effector 4 is located at the second desired position. Furthermore, since the end effector 4 located at the second desired position releases the first target object OBJ that it is holding to the second target object OBJ, controlling the robot arm 12 so that the end effector 4 approaches the second desired position may be considered equivalent to controlling the robot arm 12 so that at least one of the end effector 4 and the first target object OBJ approaches the second target object OBJ from which the first target object OBJ is released at the second desired position.

[0079] As another example, when performing a release process to release the target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the robot arm 12 so that the posture of the end effector 4 becomes a second desired posture that allows the end effector 4 to release the target object OBJ. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the posture of the end effector 4 becomes the second desired posture.

[0080] As described above, when the robot control signal includes a signal for controlling the end effector 4, the robot control device 13 may control the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the end effector 4 by controlling the operation of an actuator that moves a hand gripper that constitutes the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the end effector 4 by controlling the operation of a vacuum device of a vacuum gripper that constitutes the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the end effector 4 by controlling the operation of a magnetic attraction device of a magnetic gripper that constitutes the end effector 4 based on the robot control signal.

[0081] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the end effector 4 so that the end effector 4 located at the above-mentioned first desired position and / or in the above-mentioned first desired posture holds the target object OBJ. In this case, the robot control device 13 may control the end effector 4 based on the robot control signal so that the end effector 4 located at the first desired position and / or in the first desired posture holds the target object OBJ.

[0082] As another example, when performing a release process to release the target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the end effector 4 to release the target object OBJ held by the end effector 4 located at the above-mentioned second desired position and / or in the above-mentioned second desired orientation. In this case, the robot control device 13 may control the end effector 4 based on the robot control signal to release the target object OBJ held by the end effector 4 located at the second desired position and / or in the second desired orientation.

[0083] As described above, when the robot control signal includes a signal for controlling a robotic movable device (e.g., at least one of an automatic guided vehicle, an autonomously traveling transport robot, an unmanned aerial vehicle, and a submarine) on which the robot 1 is installed, the robot controller 13 may control the robotic movable device based on the robot control signal. For example, the robot controller 13 may control a power source (e.g., a motor or an engine) of the robotic movable device based on the robot control signal so that the robot 1 moves to a target position indicated directly or indirectly by the robot control signal.

[0084] The above description has been given of an example in which the robot 1 having the robot arm 12 is installed on a robot movable apparatus. In this case, the end effector 4 attached to the robot arm 12 may be considered to be installed on the robot movable apparatus via the robot 1 (e.g., via the robot arm 12). On the other hand, the end effector 4 may be installed on the robot movable apparatus without the robot 1 (e.g., without the robot arm 12). In this case, the robot 1 (e.g., the robot arm 12) may not be installed on the robot movable apparatus. In this case, the control device 3 may generate at least one of a movable apparatus control signal for controlling the robot movable apparatus on which the end effector 4 is installed and an end effector control signal for controlling the end effector 4 installed on the robot movable apparatus, without generating a robot control signal for controlling the robot 1.

[0085] Note that if the robot 1 (e.g., the robot arm 12) is not installed on the robot movable apparatus, the imaging system 2 may be installed on the robot movable apparatus. In this case, the control device 3 may generate at least one of a movable apparatus control signal for controlling the robot movable apparatus on which the end effector 4 is installed and an end effector control signal for controlling the end effector 4 installed on the robot movable apparatus, based on the image data IMG generated by the imaging system 2 installed on the robot movable apparatus. Furthermore, if the end effector 4 (and, in some cases, the imaging system 2) is installed on the robot movable apparatus, the control device 3 may also be disposed inside the robot movable apparatus.

[0086] The robot control signal may include a signal that can be used as is by the robot control device 13 to control the operation of the robot 1. The robot control signal may include a signal that can be used as is as a robot drive signal that the robot control device 13 uses to control the operation of the robot 1. In this case, the robot control device 13 may use the robot control signal as is to control the operation of the robot 1. For example, the control device 3 may generate a drive signal for an actuator built into the joint 122 of the robot arm 12 as the robot control signal, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the actuator built into the joint 122 of the robot arm 12.

[0087] The robot control signal may include a signal that can be used directly by the robot control device 13 to control the operation of the end effector 4. The robot control signal may include a signal that can be used directly as an end effector drive signal used by the robot control device 13 to control the operation of the end effector 4. In this case, the robot control device 13 may use the robot control signal directly to control the operation of the end effector 4. For example, the control device 3 may generate, as the robot control signal, a drive signal (end effector drive signal) for an actuator that moves a hand gripper that constitutes the end effector 4, and the robot control device 13 may use the robot control signal generated by the control device 3 directly to control the actuator of the end effector 4. For example, the control device 3 may generate, as the robot control signal, a drive signal (end effector drive signal) for driving a vacuum device of a vacuum gripper that constitutes the end effector 4, and the robot control device 13 may use the robot control signal generated by the control device 3 directly to control the vacuum device of the end effector 4. For example, the control device 3 may generate a drive signal (end effector drive signal) for driving the magnetic adsorption device of the magnetic gripper that constitutes the end effector 4 as a robot control signal, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the magnetic adsorption device of the end effector 4.

[0088] The robot control signal may include a signal that can be used directly by the robot controller 13 to control the operation of a robot movable device on which the robot 1 is installed. The robot control signal may include a signal that can be used directly as a movable device drive signal that the robot controller 13 uses to control the operation of the robot movable device. In this case, the robot controller 13 may use the robot control signal directly to control the operation of the robot movable device. For example, the controller 3 may generate a power source drive signal (movable device drive signal) for driving a power source (e.g., a motor or engine) of the robot movable device as the robot control signal, and the robot controller 13 may use the robot control signal generated by the controller 3 directly to control the power source of the robot movable device.

[0089] As described above, if the robot control signal includes a signal that the robot control device 13 can use to control the operation of at least one of the robot 1, the end effector 4, and the robot movable device, the robot 1 does not need to be equipped with the robot control device 13. In this case, the control device 3 may use the robot control signal to control an actuator built into the joint 122 of the robot arm 12. For example, the control device 3 may use the robot control signal (end effector drive signal) to control an actuator that moves a hand gripper that constitutes the end effector 4. For example, the control device 3 may use the robot control signal (end effector drive signal) to control a vacuum device of a vacuum gripper that constitutes the end effector 4. For example, the control device 3 may use the robot control signal (end effector drive signal) to control a magnetic attraction device of a magnetic gripper that constitutes the end effector 4. For example, the control device 3 may use the robot control signal (end effector drive signal) to control a robot movable device installed on the robot 1.

[0090] Alternatively, the robot control signal may include a signal that can be used by the robot control device 13 to generate a robot drive signal for controlling the operation of the robot 1. In this case, the robot control device 13 may generate a robot drive signal for controlling the operation of the robot 1 based on the robot control signal, and control the operation of the robot 1 based on the generated robot drive signal. For example, the robot control device 13 may generate a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 based on the robot control signal, and control the actuator built into the joint 122 of the robot arm 12 based on the generated robot drive signal.

[0091] The robot control signal may include a signal that the robot control device 13 can use to generate an end effector drive signal for controlling the operation of the end effector 4. In this case, the robot control device 13 may generate an end effector drive signal for controlling the operation of the end effector 4 based on the robot control signal and control the operation of the end effector 4 based on the generated end effector drive signal. For example, if the end effector 4 is a hand gripper, the robot control device 13 may generate an end effector drive signal for driving an actuator of the hand gripper based on the robot control signal and control the actuator of the hand gripper based on the generated end effector drive signal. For example, if the end effector 4 is a magnetic gripper, the robot control device 13 may generate an end effector drive signal for driving a magnetic attraction device of the magnetic gripper based on the robot control signal and control the magnetic attraction device based on the generated end effector drive signal.

[0092] Note that, if the robot system SYS includes a control device for controlling the end effector 4 in addition to the robot control device 13, the robot control signal may include a signal that can be used by the control device for controlling the end effector 4 to generate an end effector drive signal for controlling the operation of the end effector 4. In this case, the control device for controlling the end effector 4 may generate the end effector drive signal for controlling the operation of the end effector 4 based on the robot control signal, and control the operation of the end effector 4 based on the generated end effector drive signal.

[0093] The robot control signal may include a signal usable by the robot controller 13 to generate a movable unit drive signal for controlling the operation of a robot movable unit on which the robot 1 is installed. In this case, the robot controller 13 may generate a movable unit drive signal for controlling the operation of the robot movable unit based on the robot control signal, and control the operation of the robot movable unit based on the generated movable unit drive signal. For example, the robot controller 13 may generate a power source drive signal (movable unit drive signal) for driving a power source (e.g., a motor or an engine) of the robot movable unit based on the robot control signal, and control the power source of the robot movable unit based on the generated movable unit drive signal.

[0094] In addition, if the robot system SYS includes a control device for controlling the robot movable device in addition to the robot control device 13, the robot control signal may include a signal usable by the control device for controlling the robot movable device to generate a movable device drive signal for controlling the operation of the robot movable device. In this case, the control device for controlling the robot movable device may generate a movable device drive signal for controlling the operation of the robot movable device based on the robot control signal, and control the operation of the robot movable device based on the generated movable device drive signal.

[0095] The signals available to the robot controller 13 for generating the robot drive signals may include signals representing at least one of the position and orientation of the target object OBJ in a global coordinate system. The signals available to the robot controller 13 for generating the robot drive signals may include signals representing a desired positional relationship between the robot 1 and the target object OBJ in a global coordinate system.

[0096] The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value (target position) of the position of the end effector 4 in the global coordinate system. An example of a target position is a processing position where the end effector 4 should process the target object OBJ. For example, the target position may include a position where the end effector 4 should hold the target object OBJ. For example, the target position may include a position where the end effector 4 should release the target object OBJ. The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value of the position of the tip of the robot arm 12 (e.g., a tool center point) in the global coordinate system. The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value of the position of the imaging system 2 in the global coordinate system.

[0097] The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value (target posture) of the posture of the end effector 4 in the global coordinate system. An example of a target posture is a posture (processing posture) that the end effector 4 should take when processing the target object OBJ. For example, the target posture may include a posture that the end effector 4 should take when holding the target object OBJ. For example, the target posture may include a posture that the end effector 4 should take when releasing the target object OBJ. The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value of the posture of the tip of the robot arm 12 (e.g., a tool center point) in the global coordinate system. The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value of the posture of the imaging system 2 in the global coordinate system.

[0098] The signal that the robot control device 13 can use to generate the robot drive signal may be a signal that indicates the amount and direction of movement from the current position of the end effector 4 to the target position of the end effector 4 .

[0099] (1-2) Configuration of the Control Device 3 Next, the configuration of the control device 3 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the control device 3.

[0100] 3 , the control device 3 includes a calculation device 31, a storage device 32, and a communication device 33. The control device 3 may further include an input device 34 and an output device 35. However, the control device 3 does not necessarily have to include at least one of the input device 34 and the output device 35. The calculation device 31, the storage device 32, the communication device 33, the input device 34, and the output device 35 may be connected via a data bus 36.

[0101] The arithmetic device 31 is hardware that includes at least a circuit (for example, at least one of an electronic circuit and an electric circuit). For this reason, the arithmetic device 31 may be referred to as a group of circuits.

[0102] The arithmetic device 31 includes at least one processor (i.e., one processor or multiple processors) as hardware. The processor may include, for example, a processor conforming to a von Neumann computer architecture. The processor conforming to the von Neumann computer architecture may include at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor may include, for example, a processor conforming to a non-von Neumann computer architecture. The processor conforming to the non-von Neumann computer architecture may include at least one of an FPGA (Field Programmable Gate Array) and an ASIC (Application Specific Circuit). The processor may be realized by a group of circuits (e.g., at least one of an electronic circuit and an electric circuit).

[0103] The arithmetic device 31 reads a computer program 321 including at least one of computer program code and computer program instructions. For example, the arithmetic device 31 may read the computer program 321 stored in the storage device 32. For example, the arithmetic device 31 may read the computer program 321 stored in a computer-readable, non-transitory storage medium using a storage medium reading device (not shown) included in the control device 3. The computer program 321 read from the storage medium may be stored in the storage device 32. The arithmetic device 31 may acquire (i.e., download or read) the computer program 321 from a device (not shown) located outside the control device 3 via the communication device 33 (or another communication device). The downloaded computer program 321 may be stored in the storage device 32.

[0104] The arithmetic device 31 executes the loaded computer program 321. As a result, logical functional blocks for executing the processing to be performed by the control device 3 (for example, the robot control processing described above) are realized within the arithmetic device 31. In other words, the arithmetic device 31, together with the storage device 32 or the like in which the computer program 321 is recorded (in other words, together with the storage device 32 and the computer program 321 recorded in the storage device 32 or the like), can function as a controller or computer for realizing the logical functional blocks for executing the processing to be performed by the control device 3. In other words, the at least one processor included in the arithmetic device 31, the memory (recording medium) included in the storage device 32 or the like, and the computer program 321 are configured so that the control device 3 performs the processing to be performed by the control device 3 (for example, the robot control processing described above).

[0105] The arithmetic device 31 may include a single processor. In this case, the arithmetic device 31 may use the single processor to perform the processing to be performed by the control device 3 (e.g., the robot control processing described above). For example, if the arithmetic device 31 performs a first operation (e.g., a first process that is part of the robot control processing) and a second operation (e.g., a second process that is another part of the robot control processing), the arithmetic device 31 may use a single processor to perform both the first and second operations. Alternatively, the arithmetic device 31 may include multiple processors. In this case, the arithmetic device 31 may use any one of the multiple processors to perform the processing to be performed by the control device 3 (e.g., the robot control processing described above). For example, if the arithmetic device 31 includes first and second processors and performs the first and second operations, the arithmetic device 31 may use any one of the first and second processors to perform each of the first and second operations. For example, the computing device 31 may perform a first operation using a first processor, may perform a second operation using the first processor, may perform the first operation using a second processor, or may perform the second operation using the second processor.

[0106] A computational model that can be constructed by machine learning may be implemented in the computational device 31 by the computational device 31 executing the computer program 321. An example of a computational model that can be constructed by machine learning is a computational model including a neural network (so-called artificial intelligence (AI)). In this case, learning of the computational model may include learning of parameters of the neural network (e.g., at least one of a weight and a bias). The computational device 31 may execute a robot control process using the computational model. In other words, the operation of executing the robot control process may include the operation of executing the robot control process using the computational model. Note that a computational model that has been constructed by offline machine learning using teacher data may be implemented in the computational device 31. Furthermore, the computational model implemented in the computational device 31 may be updated by online machine learning on the computational device 31. Alternatively, the calculation device 31 may perform robot control processing using a calculation model implemented in a device external to the calculation device 31 (i.e., a device provided outside the control device 3) in addition to or instead of the calculation model implemented in the calculation device 31.

[0107] The recording medium for recording the computer program 321 executed by the arithmetic device 31 may be at least one of the following: a CD-ROM, CD-R, CD-RW, a flexible disk, an MO, a DVD-ROM, a DVD-RAM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, or an optical disk such as Blu-ray (registered trademark), a magnetic medium such as a magnetic tape, a magneto-optical disk, a semiconductor memory such as a USB memory, or any other medium capable of storing a program. The recording medium may include a device capable of recording the computer program 321 (for example, a general-purpose device or a dedicated device in which the computer program 321 is implemented in a state in which it can be executed in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program 321 may be realized by a logical processing block realized within the arithmetic device 31 (i.e., processor) when the arithmetic device 31 executes the computer program 321, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) provided in the arithmetic device 31, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.

[0108] 3 shows an example of logical functional blocks implemented in the arithmetic device 31 for executing robot control processing. As shown in Fig. 3, a position and orientation calculation unit 311 and a signal generation unit 312 are implemented in the arithmetic device 31. Note that the processing performed by the position and orientation calculation unit 311 and the signal generation unit 312 will be described in detail later with reference to Fig. 6 etc., and therefore will not be described here.

[0109] The storage device 32 includes at least one memory capable of storing desired data. In other words, the storage device 32 includes at least one memory containing desired data. The memory may be realized by a group of circuits (e.g., at least one of electronic circuits and electric circuits). For example, the storage device 32 may store a computer program 321 executed by the arithmetic device 31. In this case, the storage device 32 (memory) may be used as the above-mentioned recording medium for recording the computer program 321 executed by the arithmetic device 31. The storage device 32 may temporarily store data used by the arithmetic device 31 when the arithmetic device 31 is executing the computer program 321. The storage device 32 may also store data to be stored long-term by the control device 3. The storage device 32 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. That is, the storage device 32 may include a non-transitory recording medium.

[0110] The communication device 33 is capable of communicating with both the robot 1 and the imaging system 2 via a communication network (not shown). Alternatively, the communication device 33 may be capable of communicating with another device different from the robot 1 and the imaging system 2, in addition to or instead of at least one of the robot 1 and the imaging system 2, via a communication network (not shown). In this embodiment, the communication device 33 may receive (i.e., acquire) image data IMG from the imaging system 2. Furthermore, the communication device 33 may transmit (i.e., output) a robot control signal to the robot 1. Note that the communication device 33 that outputs the robot control signal to the robot 1 may be referred to as an output unit or an output device.

[0111] The input device 34 is a device that accepts information input to the control device 3 from outside the control device 3. For example, the input device 34 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by a user of the control device 3. For example, the input device 34 may include a recording medium reading device that can read information recorded as data on a recording medium that can be externally attached to the control device 3.

[0112] It should be noted that information can be input as data to the control device 3 from a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an input device that accepts information input to the control device 3 from outside the control device 3.

[0113] The output device 35 is a device that outputs information to the outside of the control device 3. For example, the output device 35 may output information as an image. That is, the output device 35 may include a display device (so-called display) 37 that can display an image. In this case, the calculation device 31 may generate a display signal for displaying the image on the display device 37. Specifically, the calculation device 31 may generate, as the display signal, a display control signal for controlling the display device 37 to display the image. The calculation device 31 may output the generated display signal (display control signal) to the display device 37 via the data bus 36. The display device 37 may display an image based on the display signal (display control signal) generated by the calculation device 31.

[0114] 1 , the robot system SYS may include a display device 8 separate from the control device 3. That is, the robot system SYS may include a display device 8 external to the control device 3 in addition to or instead of the display device 37 included in the control device 3. In this case, the arithmetic device 31 may output the generated display signal (display control signal) to the display device 8 external to the control device 3 via the communication device 33. That is, the communication device 33 may output the display signal (display control signal) generated by the arithmetic device 31 to the display device 8 external to the control device 3. The display device 8 may display an image based on the display signal (display control signal) generated by the arithmetic device 31.

[0115] The output device 35 may include an output device different from the display device 37. For example, the output device 35 may output information as sound. That is, the output device 35 may include an audio device (a so-called speaker) capable of outputting sound. For example, the output device 35 may output information on paper. That is, the output device 35 may include a printing device (a so-called printer) capable of printing desired information on paper. For example, the output device 35 may output information as data to a recording medium that can be externally attached to the control device 3.

[0116] The control device 3 can output information as data to a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an output device that outputs information to a device external to the control device 3.

[0117] The robot control device 13 provided in the robot 1 described above may also have the same configuration as the control device 3. That is, as shown in Fig. 4, which is a block diagram showing the configuration of the robot control device 13, the robot control device 13 includes an arithmetic device 131, a storage device 132, and a communication device 133. The robot control device 13 may further include an input device 134 and an output device 135. However, the robot control device 13 does not have to include at least one of the input device 134 and the output device 135. The arithmetic device 131, the storage device 132, the communication device 133, the input device 134, and the output device 135 may be connected via a data bus 136.

[0118] The characteristics of the arithmetic unit 131, the storage device 132, the communication device 133, the input device 134, and the output device 135 may be the same as the characteristics of the arithmetic unit 31, the storage device 32, the communication device 33, the input device 34, and the output device 35, respectively. The above description of the control device 3 can be used as a description of the robot control device 13 by replacing the terms control device 3, arithmetic unit 31, storage device 32, communication device 33, input device 34, and output device 35 with the terms robot control device 13, arithmetic unit 131, storage device 132, communication device 133, input device 134, and output device 135, respectively. Therefore, to avoid redundant description, the description of the robot control device 13 will be omitted.

[0119] (2) Robot Control Processing Next, the robot control processing performed by the control device 3 will be described. For convenience of explanation, the following description will discuss robot control processing for controlling at least one of the robot 1, the end effector 4, and the robot movable device to perform the processing shown in FIGS. 5( a) to 5(c). That is, for convenience of explanation, the following description will discuss an example in which the control device 3 controls at least one of the robot 1, the end effector 4, and the robot movable device to perform the processing shown in FIGS. 5( a) to 5(c). Specifically, as shown in FIGS. 5( a) to 5(c), the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 performs a holding process to hold a target object OBJ (workpiece W in the example shown in FIGS. 5( a) to 5(c)) placed on the placement device T. In this case, as shown in FIG. 5(a), the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 approaches the target object OBJ. That is, the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves toward the target object OBJ. Thereafter, as shown in Fig. 5(b), the control device 3 may control at least the end effector 4 so that the end effector 4 holds the target object OBJ. Thereafter, as shown in Fig. 5(c), the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 holding the target object OBJ moves away from (in other words, moves away from) the placement device T.

[0120] (2-1) Overall Flow of Robot Control Processing First, the overall flow of the robot control processing performed by the control device 3 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the overall flow of the robot control processing.

[0121] 6, the control device 3 (particularly the arithmetic device 31) performs a pre-setting process to control at least one of the robot 1, the end effector 4, and the robot movable device (step S1). The pre-setting process is an initial setting process that must be performed in advance to control at least one of the robot 1, the end effector 4, and the robot movable device. The pre-setting process will be described in more detail later with reference to FIG. 6 etc.

[0122] 6, the control device 3 (particularly, the arithmetic device 31) controls at least one of the robot 1, the end effector 4, and the robot movable device to hold the target object OBJ (step S2). As a result, the end effector 4 approaches the target object OBJ, and then the end effector 4 holds the target object OBJ.

[0123] After the end effector 4 holds the target object OBJ, the control device 3 (particularly, the calculation device 31) may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holding the target object OBJ releases the held target object OBJ. For example, as described above, the control device 3 (particularly, the calculation device 31) may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holding a first target object OBJ, which is an example of the target object OBJ, releases the held first target object OBJ to a second target object OBJ. In other words, the control device 3 (particularly, the calculation device 31) may control at least one of the robot 1, the end effector 4, and the robot movable device to perform the above-mentioned placement processing.

[0124] Thereafter, until it is determined that the robot control process is to be terminated, the control device 3 (particularly, the arithmetic device 31) may repeat the process of controlling at least one of the robot 1, the end effector 4, and the robot movable device to hold the target object OBJ (step S3). For example, the control device 3 (particularly, the arithmetic device 31) may repeat the process of controlling at least one of the robot 1, the end effector 4, and the robot movable device to hold multiple target objects OBJ (e.g., multiple target objects OBJ placed on the placement device T) in turn. As an example, the control device 3 (particularly, the arithmetic device 31) may control at least one of the robot 1, the end effector 4, and the robot movable device to hold (and then release, as necessary) a first target object OBJ of the multiple target objects OBJ. Thereafter, the control device 3 (particularly, the arithmetic device 31) may control at least one of the robot 1, the end effector 4, and the robot movable device to hold (and then release, as necessary) a second target object OBJ different from the first target object OBJ of the multiple target objects OBJ.

[0125] However, the control device 3 (particularly, the calculation device 31) does not have to repeat the process of controlling at least one of the robot 1, the end effector 4, and the robot movable device to hold the target object OBJ. For example, the control device 3 (particularly, the calculation device 31) may control at least one of the robot 1, the end effector 4, and the robot movable device to hold a single target object OBJ (e.g., a single target object OBJ placed on the placement device T). Thereafter, the control device 3 (particularly, the calculation device 31) may end the robot control process.

[0126] (2-2) Pre-setting process Next, the pre-setting process performed by the control device 3 in step S1 of Fig. 6 will be further described. Note that a device other than the control device 3 (for example, a control device other than the control device 3) may perform the pre-setting process.

[0127] In this embodiment, the pre-setting process may include a process of registering (in other words, setting) processing position information. The processing position information is information regarding candidate processing positions for the end effector 4 relative to the target object OBJ. In other words, the processing position information includes information regarding candidate processing positions that can be selected as targets for the processing positions for the end effector 4 relative to the target object OBJ. For example, the processing position information may include information indicating candidate processing positions for the end effector 4 relative to the target object OBJ. In other words, the processing position information may indicate candidate processing positions for the end effector 4 relative to the target object OBJ. As an example, the processing position information may include information directly indicating candidate processing positions for the end effector 4 relative to the target object OBJ. In other words, the processing position information may directly indicate candidate processing positions for the end effector 4 relative to the target object OBJ. As another example, the processing position information may include information indirectly indicating candidate holding positions for the end effector 4 relative to the target object OBJ. In other words, the processing position information may indirectly indicate candidates for processing positions by the end effector 4 on the target object OBJ.

[0128] In this embodiment, the processing position may include a processing position in a linear direction along a predetermined linear axis. In this case, the processing position of the end effector 4 relative to the target object OBJ may directly or indirectly indicate the position of the end effector 4 processing the target object OBJ in the linear direction along the predetermined linear axis. In this case, the processing position may directly or indirectly indicate the positional relationship between the target object OBJ and the end effector 4 processing the target object OBJ in the linear direction along the predetermined linear axis. Furthermore, the processing position may include a processing position in a rotational direction about a predetermined rotational axis in addition to or instead of the processing position in the linear direction along the predetermined linear axis. In this case, the processing position of the end effector 4 relative to the target object OBJ may directly or indirectly indicate the position of the end effector 4 processing the target object OBJ in the rotational direction about the predetermined rotational axis. In this case, the processing position may directly or indirectly indicate the positional relationship between the end effector 4 processing the target object OBJ and the target object OBJ in the rotational direction about the predetermined rotational axis. Furthermore, the position of the end effector 4 that processes the target object OBJ in the rotation direction around a predetermined rotation axis may be considered to be equivalent to the posture of the end effector 4 that processes the target object OBJ in the rotation direction around the predetermined rotation axis.

[0129] In this embodiment, as described with reference to FIGS. 5( a) to 5(c), a robot control process will be described in which at least one of the robot 1, the end effector 4, and the robot movable device is controlled so that the end effector 4 holds the target object OBJ. In this case, the processing position may refer to the holding position of the end effector 4 relative to the target object OBJ. In this case, the presetting process may include a process of registering (in other words, setting) holding position information, which is an example of processing position information. Note that the following description will be given of an example in which the presetting process includes a process of registering holding position information. However, the following description of holding position information can be used as a description of processing position information by replacing the word "holding" with the word "processing."

[0130] The holding position information is information regarding candidate holding positions for the end effector 4 to hold the target object OBJ. In other words, the holding position information includes information regarding candidate holding positions that can be selected as targets for the holding positions for the end effector 4 to hold the target object OBJ. For example, the holding position information may include information indicating candidate holding positions for the end effector 4 to hold the target object OBJ. That is, the holding position information may indicate candidate holding positions for the end effector 4 to hold the target object OBJ. As one example, the holding position information may include information directly indicating candidate holding positions for the end effector 4 to hold the target object OBJ. That is, the holding position information may directly indicate candidate holding positions for the end effector 4 to hold the target object OBJ. As another example, the holding position information may include information indirectly indicating candidate holding positions for the end effector 4 to hold the target object OBJ. That is, the holding position information may indirectly indicate candidate holding positions for the end effector 4 to hold the target object OBJ. In the following description, for convenience of explanation, a candidate holding position will be referred to as a candidate holding position HCP, and a target holding position will be referred to as a target holding position.

[0131] In this embodiment, an example will be described in which the holding position indicates the position on the target object OBJ held by the end effector 4. In other words, in this embodiment, an example will be described in which the holding position indicates the position of a part of the target object OBJ held by the end effector 4.

[0132] As an example, if the end effector 4 holds the target object OBJ by contacting the target object OBJ, the holding position (i.e., the position on the target object OBJ where the end effector 4 holds the target object OBJ) may refer to the position on the target object OBJ where the end effector 4 contacts in order to hold the target object OBJ. For example, if the end effector 4 is a hand gripper, the holding position may indicate the position on the target object OBJ where the finger members or claw members of the hand gripper contact. For example, if the end effector 4 is a vacuum gripper, the holding position information may indicate the position on the target object OBJ where the suction port of the vacuum device of the vacuum gripper contacts. For example, if the end effector 4 is a magnetic gripper, the holding position may indicate the position on the target object OBJ where the magnetic suction device of the magnetic gripper contacts.

[0133] As another example, when the end effector 4 holds the target object OBJ in a non-contact manner, the holding position (i.e., the position on the target object OBJ where the end effector 4 holds the target object OBJ) may refer to a position on the target object OBJ where a holding force applied from the end effector 4 acts in order to hold the target object OBJ in a non-contact manner by the end effector 4. For example, when the end effector 4 is a Bernoulli chuck, the holding position may indicate a position on the target object OBJ where a negative pressure of the Bernoulli chuck is applied (in other words, a position on the target object OBJ where an attraction force of the Bernoulli chuck acts).

[0134] In this case, the holding position information may include information regarding holding candidate positions HCP, which are candidates for positions on the target object OBJ where the end effector 4 will hold the object. In particular, the holding position information may indicate the holding candidate positions HCP, which are candidates for positions on the target object OBJ where the end effector 4 will hold the object, as positions relative to the target object OBJ. In other words, the holding position information may indicate the holding candidate positions HCP, which are candidates for positions on the target object OBJ where the end effector 4 will hold the object, as positions in a coordinate system based on the target object OBJ.

[0135] In this case, the control device 3 may register a position on the target object OBJ where the end effector 4 will hold the target object OBJ as a holding candidate position HCP by performing a pre-setting process in step S1 of Fig. 6. Thereafter, in step S2 of Fig. 6, the control device 3 may select the holding candidate position HCP indicated by the holding position information as a holding target position that is a target for the position on the target object OBJ where the end effector 4 will hold the target object OBJ, convert the selected holding target position (i.e., a relative position with respect to the target object OBJ, which is the holding target position in the coordinate system of the target object OBJ) into a holding target position in the global coordinate system (or the robot coordinate system), and generate a robot control signal for controlling at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 approaches the holding target position in the global coordinate system (or the robot coordinate system). As will be described later, when the holding position information indicates a plurality of holding candidate positions HCP, the control device 3 may select one of the plurality of holding candidate positions HCP indicated by the holding position information as a holding target position that is a target for the position where the end effector 4 holds the target object OBJ, convert the selected holding target position (i.e., a relative position with respect to the target object OBJ, that is, the holding target position in the coordinate system of the target object OBJ) into a holding target position in the global coordinate system (or the robot coordinate system), and generate a robot control signal for controlling at least one of the robot 1 and the robot movable device so that the end effector 4 approaches the holding target position in the global coordinate system (or the robot coordinate system). Thereafter, in step S2 of FIG. 6 , the control device 3 may control the end effector 4 so that it holds the target object OBJ after the end effector 4 is positioned at the holding target position.

[0136] The holding position information may indicate, in addition to or instead of a holding candidate position HCP, which is a candidate for the position on the target object OBJ where the end effector 4 holds the target object OBJ, the positional relationship between the target object OBJ and the end effector 4 for holding the target object OBJ at the holding candidate position HCP. That is, the holding position information may indicate the positional relationship between the target object OBJ and the end effector 4 at the time when the end effector 4 holds the target object OBJ at the holding candidate position HCP. Note that the positional relationship between the target object OBJ and the end effector 4 may mean the relationship between the position of the target object OBJ and the position of the end effector 4. Furthermore, the target object OBJ and the end effector 4 in the positional relationship indicated by the holding position information may be in contact with each other or may be separated from each other. Furthermore, "the time when the end effector 4 holds the target object OBJ" may mean "the time when the end effector 4 starts to hold the target object OBJ." For example, if the end effector 4 is a hand gripper that physically grips the target object OBJ using multiple finger members or claw members, "the point in time when the end effector 4 holds the target object OBJ" may mean "the point in time when the state of the multiple finger members or claw members is switched from an open state to a closed state." For example, if the end effector 4 is a vacuum gripper that vacuum-attracts the target object OBJ, "the point in time when the end effector 4 holds the target object OBJ" may mean "the point in time when the vacuum device of the vacuum gripper is switched from off to on." For example, if the end effector 4 is a magnetic gripper that magnetically attracts the target object OBJ, "the point in time when the end effector 4 holds the target object OBJ" may mean "the point in time when the magnetic attraction device of the magnetic gripper is switched from off to on." For example, if the end effector 4 is a Bernoulli chuck that holds the target object OBJ without contact, "the point in time when the end effector 4 holds the target object OBJ" may mean "the point in time when the vacuum device of the Bernoulli chuck is switched from off to on."

[0137] In this case, the control device 3 may register the positional relationship between the target object OBJ and the end effector 4 by performing a pre-setting process in step S1 of Fig. 6. Thereafter, in step S2 of Fig. 6, the control device 3 may generate a robot control signal for controlling at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 approaches a holding target position in the global coordinate system (or the robot coordinate system) based on the positional relationship indicated by the holding position information (i.e., the positional relationship between the target object OBJ and the end effector 4). For example, the control device 3 may generate a robot control signal for controlling at least one of the robot 1, the end effector 4, and the robot movable device so as to satisfy the condition that "the end effector 4 approaches the holding candidate position HCP and the positional relationship between the target object OBJ and the end effector 4 matches the positional relationship indicated by the holding position information." Alternatively, for example, the control device 3 may generate a robot control signal for controlling at least one of the robot 1, the end effector 4, and the robot movable device so as to satisfy the condition that "as the end effector 4 approaches the holding candidate position HCP, the difference (deviation) between the positional relationship between the target object OBJ and the end effector 4 and the positional relationship indicated by the holding position information becomes equal to or less than a predetermined allowable threshold value." Thereafter, in step S2 of Fig. 6, the control device 3 may control the end effector 4 to hold the target object OBJ after the positional relationship between the target object OBJ and the end effector 4 matches the positional relationship indicated by the holding position information.

[0138] The holding position information may indicate the position of the end effector 4 for holding the target object OBJ, in addition to or instead of the holding candidate position HCP, which is a candidate for the position where the end effector 4 holds the target object OBJ on the target object OBJ. In particular, the holding position information may indicate the position of the end effector 4 for holding the target object OBJ relative to the target object OBJ. In other words, the holding position information may indicate the position of the end effector 4 for holding the target object OBJ in a coordinate system based on the target object OBJ. More specifically, the holding position information may indicate the position of the end effector 4 at the time when the end effector 4 holds the target object OBJ. In particular, the holding position information may indicate the position of the end effector 4 relative to the target object OBJ at the time when the end effector 4 holds the target object OBJ. In other words, the holding position information may indicate the position of the end effector 4 in a coordinate system based on the target object OBJ at the time when the end effector 4 holds the target object OBJ. The end effector 4 located at the position indicated by the holding position information may be in contact with the target object OBJ or may be separated from the target object OBJ. The position of the end effector 4 indicated by the holding position information may be a position on the target object OBJ or may be a position separated from the target object OBJ.

[0139] In this case, the control device 3 may register the position of the end effector 4 relative to the target object OBJ by performing a pre-setting process in step S1 of Fig. 6. Then, in step S2 of Fig. 6, the control device 3 may generate a robot control signal for controlling at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 approaches a holding target position in the global coordinate system (or the robot coordinate system) based on the position indicated by the holding position information (i.e., the position of the end effector 4 relative to the target object OBJ in a coordinate system based on the target object OBJ). For example, the control device 3 may generate a robot control signal for controlling at least one of the robot 1, the end effector 4, and the robot movable device so as to satisfy the condition that "the end effector 4 approaches the holding candidate position HCP and the position of the end effector 4 relative to the target object OBJ coincides with the position indicated by the holding position information." Then, in step S2 of FIG. 6, the control device 3 may control the end effector 4 to hold the target object OBJ after the position of the end effector 4 relative to the target object OBJ coincides with the position indicated by the holding position information.

[0140] The position of the end effector 4 may refer to the position of a reference point of the end effector 4. The tool center point of the end effector 4 may be used as the reference point of the end effector 4. However, any point determined based on the end effector 4, different from the tool center point, may be used as the reference point of the end effector 4. The reference point of the end effector 4 may be set on the end effector 4 or may be set at a position away from the end effector 4. As an example, the center of gravity of the end effector 4 may be used as the reference point of the end effector 4.

[0141] The holding position information may indicate the position of the target object OBJ to be held by the end effector 4, in addition to or instead of the holding candidate position HCP, which is a candidate for the position on the target object OBJ where the end effector 4 holds the target object OBJ. In particular, the holding position information may indicate the position of the target object OBJ to be held by the end effector 4 relative to the end effector 4. In other words, the holding position information may indicate the position of the target object OBJ to be held by the end effector 4 in a coordinate system based on the end effector 4. More specifically, the holding position information may indicate the position of the target object OBJ at the time the end effector 4 holds the target object OBJ. In particular, the holding position information may indicate the position of the target object OBJ relative to the end effector 4 at the time the end effector 4 holds the target object OBJ. In other words, the holding position information may indicate the position of the target object OBJ in a coordinate system based on the end effector 4 at the time the end effector 4 holds the target object OBJ. The target object OBJ located at the position indicated by the holding position information may be in contact with the end effector 4 or may be separated from the end effector 4. The position of the target object OBJ indicated by the holding position information may be a position on the end effector 4 or may be a position separated from the end effector 4.

[0142] In this case, the control device 3 may register the position of the target object OBJ relative to the end effector 4 by performing a presetting process in step S1 of FIG. 6 . Then, in step S2 of FIG. 6 , the control device 3 may generate a robot control signal for controlling at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 approaches a holding target position in the global coordinate system (or the robot coordinate system) based on the position indicated by the holding position information (i.e., the position of the target object OBJ relative to the end effector 4). For example, the control device 3 may generate a robot control signal for controlling at least one of the robot 1, the end effector 4, and the robot movable device so as to satisfy the condition that "the end effector 4 approaches the holding candidate position HCP and the position of the target object OBJ relative to the end effector 4 coincides with the position indicated by the holding position information." Then, in step S2 of FIG. 6 , the control device 3 may control the end effector 4 to hold the target object OBJ after the position of the target object OBJ relative to the end effector 4 coincides with the position indicated by the holding position information.

[0143] The control device 3 may register the holding position information based on an instruction from a user (operator) of the robot system SYS. In this case, the control device 3 may display a holding position registration screen 61 that can be used by the user to input an instruction to register the holding position information. Specifically, the arithmetic device 31 of the control device 3 may generate a display signal (display control signal) for displaying the holding position registration screen 61 and output the generated display signal (display control signal) to the display device 37 of the control device 3. The display device 37 may display the holding position registration screen 61 based on the display signal (display control signal) generated by the arithmetic device 31.

[0144] An example of the holding position registration screen 61 is shown in FIG. 7 . As shown in FIG. 7 , a virtual three-dimensional space 61SP is constructed within the holding position registration screen 61. An object model OM, which is a three-dimensional model of the target object OBJ, may be arranged in the three-dimensional space 61SP. In this case, the user may use the input device 34 to specify on the object model OM a position where the end effector 4 will be held on the target object OBJ (i.e., a holding candidate position HCP). Thereafter, the control device 3 may register information indicating the position specified by the user as holding position information indicating the position where the end effector 4 will be held on the target object OBJ.

[0145] A CAD (Computer Aided Design) model of the target object OBJ may be used as the object model OM. Alternatively, in addition to or instead of a CAD model, a model generated from image data IMG generated by capturing an image of the target object OBJ using an imaging system 2 including the above-described stereo camera (or another imaging system different from the imaging system 2; the same applies hereinafter in this paragraph) may be used as the object model OM. An example of a model generated from the image data IMG is a point cloud model. Another example of a model generated from the image data IMG is at least one of a surface model, a mesh model, and a polygon model, which are further generated from the point cloud model generated from the image data IMG.

[0146] In addition to the object model OM, an end effector model EM, which is a three-dimensional model of the end effector 4, may be placed in the three-dimensional space 61SP. In this case, the user may use the input device 34 to move at least one of the object model OM and the end effector model EM within the three-dimensional space 61SP. The control device 3 may register holding position information based on at least one of the object model OM and the end effector model EM after the user has moved at least one of the object model OM and the end effector model EM. For example, the control device 3 may register information indicating the position where the object model OM and the end effector model EM come into contact as holding position information indicating the position where the end effector 4 will be held on the target object OBJ (i.e., holding candidate position HCP). For example, the control device 3 may register information indicating the position of the portion of the object model OM closest to the end effector model EM as holding position information indicating the position where the end effector 4 will be held on the target object OBJ (i.e., holding candidate position HCP). For example, the control device 3 may register information indicating the positional relationship between the object model OM and the end effector model EM as holding position information indicating the positional relationship between the target object OBJ and the end effector 4 at the time when the end effector 4 holds the target object OBJ. For example, the control device 3 may register information indicating the position of the end effector model EM relative to the object model OM as holding position information indicating the position of the end effector 4 at the time when the end effector 4 holds the target object OBJ. For example, the control device 3 may register information indicating the position of the object model OM relative to the end effector model EM as holding position information indicating the position of the target object OBJ at the time when the end effector 4 holds the target object OBJ.

[0147] A CAD (Computer Aided Design) model of the end effector 4 may be used as the end effector model EM. Alternatively, in addition to or instead of the CAD model, a model generated from image data IMG generated by capturing images of the end effector 4 with the imaging system 2 including the stereo camera described above (or another imaging system different from the imaging system 2; the same applies hereinafter in this paragraph) may be used as the end effector model EM. An example of a model generated from the image data IMG is a point cloud model. Another example of a model generated from the image data IMG is at least one of a surface model, a mesh model, and a polygon model that are further generated from the point cloud model generated from the image data IMG.

[0148] When holding position information is registered using the holding position registration screen 61 on which the object model OM is arranged in this manner, the holding position information (especially the holding candidate position HCP indicated by the holding position information) may be considered to be set with respect to the object model OM. In other words, the holding candidate position HCP indicated by the holding position information as a candidate position on the target object OBJ where the end effector 4 will hold the object OBJ may be considered to be a position set with respect to the object model OM. In other words, the holding candidate position HCP indicated by the holding position information as a candidate position on the target object OBJ where the end effector 4 will hold the object OBJ may be considered to be a position in the coordinate system of the object model OM. Note that, since the object model OM indicates the target object OBJ, the position in the coordinate system of the object model OM may be considered to be equivalent to the position in the coordinate system of the target object OBJ. In this case, the holding candidate position HCP indicated by the holding position information as a candidate position on the target object OBJ where the end effector 4 will hold the object OBJ may be considered to be a position in the coordinate system of the target object OBJ.

[0149] Alternatively, when the holding candidate position HCP is registered using the object model OM and the end effector model EM as described above, the holding candidate position HCP indicated by the holding position information as a candidate for the position where the end effector 4 holds the target object OBJ may be considered to be a position in the coordinate system of the end effector model EM. This is because, when the holding candidate position HCP is registered using the object model OM and the end effector model EM as described above, the holding candidate position HCP can be registered by aligning the object model OM with the end effector model EM. Note that, since the end effector model EM indicates the end effector 4, the position in the coordinate system of the end effector model EM may be considered to be equivalent to the position in the coordinate system of the end effector 4. In this case, the holding candidate position HCP indicated by the holding position information as a candidate for the position where the end effector 4 holds the target object OBJ may be considered to be a position in the coordinate system of the end effector 4.

[0150] The control device 3 may register holding position information including information about a single holding candidate position HCP. In this case, in step S2 of Fig. 6, the control device 3 may select (in other words, determine or set) the single holding candidate position HCP indicated by the holding position information as the holding target position. Thereafter, the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 approaches the selected holding target position.

[0151] Alternatively, the control device 3 may register holding position information including information on a plurality of different holding candidate positions HCP. Examples of a plurality of different holding candidate positions HCP are conceptually shown in FIGS. 8( a) to 8(d). FIG. 8(a) shows a first holding candidate position HCP (i.e., a first candidate position to which the end effector 4 should approach in order to hold the target object OBJ). FIG. 8(b) shows a second holding candidate position HCP (i.e., a second candidate position to which the end effector 4 should approach in order to hold the target object OBJ). FIG. 8(c) shows a third holding candidate position HCP (i.e., a third candidate position to which the end effector 4 should approach in order to hold the target object OBJ). FIG. 8(d) shows a fourth holding candidate position HCP (i.e., a fourth candidate position to which the end effector 4 should approach in order to hold the target object OBJ).

[0152] 8(a) to 8(d) show holding candidate positions HCP that are registered when an end effector 4 that holds the target object OBJ by contacting the target object OBJ is used. However, as already explained, holding candidate positions HCP may also be registered when an end effector 4 that holds the target object OBJ without contacting the target object OBJ is used, as shown in FIG.

[0153] 8A to 8E each show an example in which the holding candidate position HCP is a position on the target object OBJ. However, a position distant from the target object OBJ may also be registered as the holding candidate position HCP. For example, when an end effector 4 that holds the target object OBJ without contact is used, the position of the end effector 4 that holds the target object OBJ from a position distant from the target object OBJ may be registered as the holding candidate position HCP. Alternatively, when an end effector 4 that holds the target object OBJ by contacting the target object OBJ is used, a position distant from the target object OBJ may also be registered as the holding candidate position HCP. For example, the position of a reference portion of the end effector 4 that is distant from the target object OBJ at the time the end effector 4 comes into contact with and holds the target object OBJ may be registered as the holding candidate position HCP.

[0154] 6, the control device 3 may select (in other words, determine or set) one holding candidate position HCP from the plurality of holding candidate positions HCP indicated by the holding position information as the holding target position. The control device 3 may then control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 approaches the selected holding target position (i.e., the selected one holding candidate position HCP). In this case, compared to a case where only a single holding candidate position HCP is registered (i.e., only a single holding position information is registered), even if at least one of the position and orientation of the target object OBJ placed on the placement device T varies (i.e., is not consistent), the selection of an appropriate holding candidate position HCP corresponding to at least one of the position and orientation of the target object OBJ placed on the placement device T as the holding target position increases the likelihood that the end effector 4 can properly hold the target object OBJ. As an example, even if multiple target objects OBJ are stacked in bulk on the mounting device T (e.g., in a container CB), the end effector 4 is more likely to be able to properly hold the multiple target objects OBJ in sequence.

[0155] The control device 3 may assign priorities to the plurality of holding candidate positions HCP indicated by the holding position information. The priorities may refer to priorities for selection as holding target positions. Alternatively, the priorities may refer to priorities used in a holding position determination process (described later) that is performed to select one holding candidate position HCP from the plurality of holding candidate positions HCP as a holding target position.

[0156] The control device 3 may assign priorities to multiple storage candidate locations HCP based on a user instruction. In this case, the user may input an instruction to assign priorities on the storage location registration screen 61. In this case, as shown in FIG. 7 , the storage location registration screen 61 may display an input screen 611 for inputting priorities to be assigned to the storage candidate locations HCP. The input screen 611 may display the priorities assigned by the user. In this way, when priorities are assigned to multiple storage candidate locations HCP based on a user instruction, the user may be considered to be essentially able to change the priorities assigned to the multiple storage candidate locations HCP. The user may be considered to be essentially able to freely set the priorities assigned to the multiple storage candidate locations HCP. Alternatively, for example, the control device 3 may automatically assign priorities to multiple storage candidate locations HCP without based on a user instruction.

[0157] The control device 3 may change the priorities assigned to the multiple holding candidate positions HCP by the pre-setting process based on a user instruction. For example, during the period in which the process (holding control process) of controlling at least one of the robot 1, the end effector 4, and the robot movable device to hold the target object OBJ is being performed in step S2 of FIG. 6 , the control device 3 may change the priorities assigned to the multiple holding candidate positions HCP by the pre-setting process based on a user instruction. In this case, the user can essentially change the priorities assigned to the multiple holding candidate positions HCP. In other words, the user can essentially freely set the priorities assigned to the multiple holding candidate positions HCP.

[0158] (2-3) Holding control process for holding target object OBJ Next, we will explain the process (holding control process) for controlling at least one of the robot 1, the end effector 4, and the robot movable device so as to hold the target object OBJ in step S2 of Figure 6.

[0159] (2-3-1) Overall Flow of Retention Control Processing First, the overall flow of the retention control processing in step S2 of Fig. 6 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the overall flow of the retention control processing in step S2 of Fig. 6.

[0160] As shown in FIG. 9 , the position and orientation calculation unit 311 included in the control device 3 acquires image data IMG from the imaging system 2 using the communication device 33 (step S21). Specifically, the imaging system 2 captures an image of the target object OBJ placed on the mounting device T (e.g., a container CB) at a predetermined imaging rate. For example, the imaging system 2 may capture images of the target object OBJ at an imaging rate of tens to hundreds (e.g., 500 times) per second. As a result, the imaging system 2 generates image data IMG at a period corresponding to the predetermined imaging rate. For example, the imaging system 2 may generate tens to hundreds (e.g., 500) pieces of image data IMG per second. The position and orientation calculation unit 311 acquires the image data IMG each time the imaging system 2 generates image data IMG. In other words, the position and orientation calculation unit 311 may acquire tens to hundreds (e.g., 500) pieces of image data IMG per second.

[0161] However, the imaging system 2 does not have to periodically capture images of the target object OBJ at a desired imaging rate. For example, the imaging system 2 may capture an image of the target object OBJ when receiving a control signal from the control device 3 (or another control device different from the control device 3) for controlling the imaging system 2 to capture an image of the target object OBJ.

[0162] The imaging device 21 may capture images of objects other than the target object OBJ, in addition to the target object OBJ on which the end effector 4 performs a predetermined process (in this case, held by the end effector 4). Note that, because the other objects are not targets on which the robot 1 performs a predetermined process, the other objects other than the target object OBJ are referred to as non-target objects in the following description. An example of a non-target object is at least a portion of the end effector 4. An example of a non-target object is at least a portion of the robot arm 12. An example of a non-target object is at least a portion of a peripheral object, which is an object located around the robot 1. An example of a peripheral object is the mounting device T. For example, if both the target object OBJ and the non-target object are included in the imaging range (field of view) of the imaging device 21, the imaging device 21 may capture images of both the target object OBJ and the non-target object. As a result, the imaging device 21 may generate image data IMG representing an image in which both the target object OBJ and the non-target object are captured. However, the imaging device 21 may capture an image of the target object OBJ without capturing an image of a non-target object. That is, the imaging device 21 may generate image data IMG representing an image in which the target object OBJ is captured but the non-target object is not captured. In either case, the imaging device 21 generates image data IMG representing an image in which at least the target object OBJ is captured. That is, the imaging device 21 generates image data IMG including image data of at least the target object OBJ.

[0163] When a plurality of target objects OBJ are placed on the mounting device T (e.g., a container CB), the imaging system 2 may capture images of at least some of the plurality of target objects OBJ placed on the mounting device T. In other words, the imaging system 2 may capture images of a group of target objects including at least some of the plurality of target objects OBJ placed on the mounting device T. The group of target objects captured by the imaging system 2 may include a plurality of target objects OBJ included in the imaging field of view of the imaging system 2 out of the plurality of target objects OBJ placed on the mounting device T (or, in some cases, one target object OBJ included in the imaging field of view of the imaging system 2).

[0164] When multiple target objects OBJ are placed on the placement device T, the end effector 4 may hold the multiple target objects OBJ in order. In this case, the end effector 4 may hold one of the multiple target objects OBJ during a first period. Furthermore, the end effector 4 may hold another target object OBJ different from the one of the multiple target objects OBJ during a second period following the first period. In this case, among the multiple target objects OBJ, target objects OBJ other than the one target object OBJ on which the end effector 4 performs a predetermined process may be considered not to be processing objects on which the robot 1 performs a predetermined process during the first period. Similarly, among the multiple target objects OBJ, target objects OBJ other than the other target object OBJ#1 on which the end effector 4 performs a predetermined process may be considered not to be processing objects on which the robot 1 performs a predetermined process during the second period.

[0165] Every time the position and orientation calculation unit 311 acquires image data IMG in step S21, the position and orientation calculation unit 311 calculates at least one of the position and orientation of the target object OBJ based on the image data IMG acquired in step S21 (step S22). As a result, the position and orientation calculation unit 311 generates position and orientation data POI that indicates the object position and orientation, which is at least one of the position and orientation of the target object OBJ (step S22).

[0166] In the following description, an example will be described in which the position and orientation calculation unit 311 calculates at least one of the position and orientation of the target object OBJ in the global coordinate system in step S22, as described above. That is, in the following description, an example will be described in which the position and orientation calculation unit 311 generates position and orientation data POI that indicates at least one of the position and orientation of the target object OBJ in the global coordinate system.

[0167] 6, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a matching process using the image data IMG acquired in step S21. Specifically, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a matching process using the image data IMG and a model (template model) that indicates at least a part of the target object OBJ (for example, that indicates the shape of at least a part of the target object OBJ).

[0168] As a first example, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a contour matching process (in other words, an edge matching process), which is an example of a matching process. Specifically, to perform the contour matching process, the position and orientation calculation unit 311 may use image data IMG generated by the imaging device 21 including a monocular camera (i.e., image data IMG including one image generated by the monocular camera, which may hereinafter be referred to as 2D image data IMG_2D as necessary). In this case, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a contour matching process using the 2D image data IMG_2D (i.e., an image indicated by the 2D image data IMG_2D) and a contour model (in other words, an edge model), which is an example of a template model. The contour model may represent at least a portion of the contour of the target object OBJ. In other words, the contour model does not need to represent a portion of the target object OBJ that is surrounded by the contour. The contour of the target object OBJ may be referred to as the edge of the target object OBJ. In this case, the position and orientation calculation unit 311 may perform, as the contour matching process, an object detection process that detects the target object OBJ (in other words, the contour of the target object OBJ) by detecting a contour (edge) corresponding to a contour model within the image indicated by the 2D image data IMG_2D.

[0169] In this case, the position and orientation calculation unit 311 may translate, enlarge, reduce, and / or rotate the contour model in an imaging coordinate system based on the imaging device 21 so that the contour model approaches (e.g., matches) the contour of the target object OBJ captured in the image represented by the 2D image data IMG_2D. As a result, the position and orientation calculation unit 311 can identify the positional relationship between the coordinate system of the contour model and the imaging coordinate system. Then, based on the positional relationship between the coordinate system of the contour model and the imaging coordinate system, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ in the imaging coordinate system from at least one of the position and orientation of the target object OBJ in the coordinate system of the contour model. Then, based on a transformation matrix for transforming three-dimensional coordinates in the imaging coordinate system into three-dimensional coordinates in the global coordinate system (or the robot coordinate system), the position and orientation calculation unit 311 may transform at least one of the position and orientation of the target object OBJ in the imaging coordinate system into at least one of the position and orientation of the target object OBJ in the global coordinate system (or the robot coordinate system).

[0170] As a second example, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a 2D matching process, which is an example of a matching process. Specifically, the position and orientation calculation unit 311 may use the 2D image data IMG_2D to perform the 2D matching process. In this case, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a 2D matching process using the 2D image data IMG_2D (i.e., the image indicated by the 2D image data IMG_2D) and a two-dimensional model, which is an example of a template model. The two-dimensional model may represent the two-dimensional shape of at least a portion of the target object OBJ using both at least a portion of the contour of the target object OBJ and at least a portion of a portion of the target object OBJ surrounded by the contour. In this case, the position and orientation calculation unit 311 may perform, as the 2D matching process, an object detection process to detect the target object OBJ indicated by the two-dimensional model (in other words, an image portion corresponding to the two-dimensional model) within the image indicated by the 2D image data IMG_2D. The 2D matching process (in this case, the object detection process) itself may be the same as an existing 2D matching process. For example, the position and orientation calculation unit 311 may perform the 2D matching process using a well-known method such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed-Up Robust Feature).

[0171] In this case, the position and orientation calculation unit 311 may translate, enlarge, reduce, and / or rotate the two-dimensional model in an imaging coordinate system based on the imaging device 21 so that characteristic locations of the two-dimensional model (e.g., at least one of a feature point and an edge) approach (e.g., coincide with) characteristic locations of the target object OBJ captured in the image represented by the 2D image data IMG_2D. As a result, the position and orientation calculation unit 311 can identify the positional relationship between the coordinate system of the two-dimensional model and the imaging coordinate system. Thereafter, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ in the imaging coordinate system from at least one of the position and orientation of the target object OBJ in the coordinate system of the two-dimensional model based on the positional relationship between the coordinate system of the two-dimensional model and the imaging coordinate system. Thereafter, the position and orientation calculation unit 311 may convert at least one of the position and orientation of the target object OBJ in the imaging coordinate system into at least one of the position and orientation of the target object OBJ in the global coordinate system (or the robot coordinate system) based on a transformation matrix for converting three-dimensional coordinates in the imaging coordinate system into three-dimensional coordinates in the global coordinate system (or the robot coordinate system).

[0172] As a third example, if the imaging device 21 includes a stereo camera, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a 3D matching process, which is an example of a matching process. Specifically, to perform the 3D matching process, the position and orientation calculation unit 311 may use image data IMG generated by the imaging device 21 including a stereo camera (i.e., image data IMG including two images generated by the stereo camera, which may hereinafter be referred to as 3D image data IMG_3D as necessary). In this case, the position and orientation calculation unit 311 may generate three-dimensional position data indicating the three-dimensional position of at least a portion of the target object OBJ based on the 3D image data IMG_3D (i.e., the two images indicated by the 3D image data IMG_3D). In this case, the position and orientation calculation unit 311 may calculate parallax based on the 3D image data IMG_3D (i.e., the two image data indicated by the 3D image data IMG_3D) and generate the three-dimensional position data using the calculated parallax using a well-known method based on the principle of triangulation. The three-dimensional position data may be data indicating the three-dimensional position of each of multiple points on the target object OBJ. The three-dimensional position data may be data indicating the three-dimensional position of each of multiple points on the surface of the target object OBJ. The three-dimensional position data may be data indicating the three-dimensional position of each of multiple points corresponding to multiple portions of the surface of the target object OBJ. In the following description, an example will be described in which point cloud data indicating a point cloud is used as the three-dimensional position data. However, three-dimensional position data other than point cloud data indicating a point cloud (e.g., depth image data indicating a depth image) may also be used. Thereafter, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a 3D matching process using the three-dimensional position data (e.g., point cloud data) and a three-dimensional model, which is an example of a template model. The three-dimensional model may represent the three-dimensional shape of at least a portion of the target object OBJ using both at least a portion of the contour of the target object OBJ and at least a portion of the portion of the target object OBJ surrounded by the contour.In this case, the position and orientation calculation unit 311 may perform, as the 3D matching process, an object detection process that detects a target object OBJ indicated by the 3D model (in other words, a set of points corresponding to the 3D model) within a point cloud indicated by the 3D position data. The 3D matching process (in this case, the object detection process) itself may be the same as an existing 3D matching process. For example, the position and orientation calculation unit 311 may perform the 3D matching process using a well-known method including at least one of RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), ICP (Iterative Closest Point), and DSO (Direct Sparse Odometry).

[0173] Note that even if the imaging device 21 is an imaging device 21 that includes a monocular camera (in other words, an imaging device 21 that does not include a stereo camera), when the imaging device 21 captures an image of a target object OBJ onto which a projection pattern is projected by the lighting device 23, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing 3D matching processing. This is because the position and orientation calculation unit 311 can calculate a parallax that can be used to generate three-dimensional position data based on the projection pattern (specifically, the projection pattern deformed according to the shape of the target object OBJ) captured in the image indicated by the 2D image data IMG_2D.

[0174] When 3D matching processing is performed, the position and orientation calculation unit 311 may translate, enlarge, reduce, and / or rotate the three-dimensional model in the imaging coordinate system so that characteristic locations of the three-dimensional model (e.g., at least one of a feature point and an edge) approach (e.g., coincide with) characteristic locations of the target object OBJ whose three-dimensional position data indicates (e.g., a point cloud corresponding to the target object OBJ indicated by the three-dimensional position data). As a result, the position and orientation calculation unit 311 can identify the positional relationship between the coordinate system of the three-dimensional model and the imaging coordinate system. Thereafter, even when 3D matching processing is performed, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ in the global coordinate system (or the robot coordinate system) by performing processing similar to that when 2D matching processing is performed.

[0175] The position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing at least two of a contour matching process, a 2D matching process, and a 3D matching process. For example, the position and orientation calculation unit 311 may calculate at least a portion of the position and orientation of the target object OBJ by performing at least one of the contour matching process, the 2D matching process, and the 3D matching process, and calculate at least another portion of the position and orientation of the target object OBJ by performing at least another of the contour matching process, the 2D matching process, and the 3D matching process. For example, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by merging at least two of the calculation results of at least one of the position and orientation of the target object OBJ by the contour matching process, the calculation results of at least one of the position and orientation of the target object OBJ by the 2D matching process, and the calculation results of at least one of the position and orientation of the target object OBJ by the 3D matching process. For example, the position and orientation calculation unit 311 may calculate an average value of at least two of the calculation results of at least one of the position and orientation of the target object OBJ obtained by the contour matching process, the calculation results of at least one of the position and orientation of the target object OBJ obtained by the 2D matching process, and the calculation results of at least one of the position and orientation of the target object OBJ obtained by the 3D matching process, and use the calculated average value as at least one of the positions and orientations of the target object OBJ. For example, the position and orientation calculation unit 311 may calculate a median value of at least two of the calculation results of at least one of the position and orientation of the target object OBJ obtained by the contour matching process, the calculation results of at least one of the position and orientation of the target object OBJ obtained by the 2D matching process, and the calculation results of at least one of the position and orientation of the target object OBJ obtained by the 3D matching process, and use the calculated median value as at least one of the positions and orientations of the target object OBJ.

[0176] The position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ based on the image data IMG using an existing method other than matching processing. For example, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ based on the image data IMG using an inference model generated by machine learning. The inference model may be generated by machine learning so that, when the image data IMG is input, it outputs at least one of the position and orientation of the target object OBJ that appears in the image indicated by the image data IMG.

[0177] By performing the matching process, the position and orientation calculation unit 311 may calculate, as the position of the target object OBJ in the global coordinate system (or the robot coordinate system), at least one of a position Tx of the target object OBJ in an X-axis direction parallel to the X-axis of the global coordinate system (or the robot coordinate system), a position Ty of the target object OBJ in a Y-axis direction parallel to the Y-axis of the global coordinate system (or the robot coordinate system), and a position Tz of the target object OBJ in a Z-axis direction parallel to the Z-axis of the global coordinate system (or the robot coordinate system). By performing the matching process, the position and orientation calculation unit 311 may calculate, as the orientation of the target object OBJ in the global coordinate system (or the robot coordinate system), at least one of a rotation amount Rx of the target object OBJ about the X-axis of the global coordinate system, a rotation amount Ry of the target object OBJ about the Y-axis of the global coordinate system (or the robot coordinate system), and a rotation amount Rz of the target object OBJ about the Z-axis of the global coordinate system (or the robot coordinate system). This is because the amount of rotation Rx of the target object OBJ around the X-axis, the amount of rotation Ry of the target object OBJ around the Y-axis, and the amount of rotation Rz of the target object OBJ around the Z-axis are equivalent to the parameters representing the orientation of the target object OBJ around the X-axis, the parameters representing the orientation of the target object OBJ around the Y-axis, and the parameters representing the orientation of the target object OBJ around the Z-axis, respectively. For this reason, in the following description, the amount of rotation Rx of the target object OBJ around the X-axis, the amount of rotation Ry of the target object OBJ around the Y-axis, and the amount of rotation Rz of the target object OBJ around the Z-axis will be referred to as the orientation Rx of the target object OBJ around the X-axis, the orientation Ry of the target object OBJ around the Y-axis, and the orientation Rz of the target object OBJ around the Z-axis, respectively.

[0178] The orientation Rx of the target object OBJ around the X-axis, the orientation Ry of the target object OBJ around the Y-axis, and the orientation Rz of the target object OBJ around the Z-axis may be considered to indicate the position of the target object OBJ in the rotation direction around the X-axis, the position of the target object OBJ in the rotation direction around the Y-axis, and the position of the target object OBJ in the rotation direction around the Z-axis, respectively. In other words, the orientation Rx of the target object OBJ around the X-axis, the orientation Ry of the target object OBJ around the Y-axis, and the orientation Rz of the target object OBJ around the Z-axis may all be considered to be parameters representing the position of the target object OBJ.

[0179] When performing the matching process, the position and orientation calculation unit 311 may calculate a matching similarity, which is the similarity between the template model and the detected target object OBJ. In this case, in step S22 of FIG. 9 , if a target object OBJ whose calculated matching similarity exceeds a predetermined matching determination threshold used in the matching process is detected by the matching process, the position and orientation calculation unit 311 may select (in other words, determine) the target object OBJ as a processing execution object on which the end effector 4 should actually perform a predetermined process. On the other hand, if a target object OBJ whose calculated matching similarity is below the matching determination threshold is detected by the matching process, the position and orientation calculation unit 311 may not select the target object OBJ as a processing execution object on which the end effector 4 should actually perform a predetermined process. Note that in this embodiment, an example is described in which the end effector 4 performs a holding process to hold the target object OBJ as the predetermined process. Therefore, the processing execution object on which the end effector 4 should actually perform a predetermined process may also be referred to as a held execution object to be actually held by the end effector 4.

[0180] The matching determination threshold is a threshold used for detecting the target object OBJ from the image represented by the image data IMG through the matching process. Specifically, the matching determination threshold is a threshold used for distinguishing, from the matching similarity of the object detected through the matching process, a state in which the object detected through the matching process is highly likely to be the target object OBJ and a state in which the object detected through the matching process is unlikely to be the target object OBJ. The matching determination threshold can also be said to be a threshold used for distinguishing, from the matching similarity, a state in which the accuracy of at least one of the position and orientation of the target object OBJ calculated through the matching process is high and a state in which the accuracy of at least one of the position and orientation of the target object OBJ calculated through the matching process is low.

[0181] As described above, the imaging system 2 may capture images of multiple target objects OBJ placed on the mounting device T. In this case, multiple target objects OBJ may appear in the image represented by the image data IMG generated by the imaging system 2. In this case, the position and orientation calculation unit 311 may select (in other words, determine) one of the multiple target objects OBJ appearing in the image represented by the image data IMG as the processing execution object (holding execution object) to be actually held by the end effector 4. For example, the position and orientation calculation unit 311 may select one of the multiple target objects OBJ appearing in the image represented by the image data IMG as the processing execution object based on the matching similarity. As an example, the position and orientation calculation unit 311 may select one of the multiple target objects OBJ whose matching similarity is greater than a matching threshold and is the highest among the multiple target objects OBJ as the processing execution object. As another example, the position and orientation calculation unit 311 may select, as the object to be processed, one target object OBJ among the multiple target objects OBJ whose matching similarity exceeds the matching determination threshold and whose matching similarity is the Nth largest (N is a constant indicating an integer greater than or equal to 2). As another example, the position and orientation calculation unit 311 may select, as the object to be processed, one target object OBJ among the multiple target objects OBJ whose matching similarity exceeds a predetermined matching determination threshold. As another example, the position and orientation calculation unit 311 may select, as the object to be processed, one target object OBJ among the multiple target objects OBJ whose matching similarity exceeds the matching determination threshold and is closest to the end effector 4. As another example, when a plurality of target objects OBJ are contained in a container CB (mounting device T) (for example, when a plurality of objects are stacked in bulk), the position and orientation calculation unit 311 may select, as the object to be processed, one target object OBJ from among the plurality of target objects OBJ that corresponds to a matching similarity that exceeds the matching judgment threshold and has the largest Z coordinate along the Z axis (is located at the highest position).As another example, the position and orientation calculation unit 311 may select, as the processing execution object, one target object OBJ from among multiple target objects OBJ that corresponds to a matching similarity that exceeds the matching judgment threshold and that the end effector 4 can perform a specified processing on.

[0182] Thereafter, the signal generating unit 312 included in the control device 3 selects (in other words, determines or sets) a holding target position to which the end effector 4 should actually approach in order to hold one target object OBJ selected as the processing execution object (step S23). Specifically, as described above, the control device 3 registers holding position information indicating at least one holding candidate position HCP by performing a presetting process. In this embodiment, the signal generating unit 312 may select one holding candidate position HCP as the holding target position from the at least one holding candidate position HCP indicated by the holding position information by performing a holding position determination process on the at least one holding candidate position HCP indicated by the holding position information.

[0183] As described above, at least one holding candidate position HCP indicated by the holding position information indicates, for example, a position on the target object OBJ where the end effector 4 holds the target object OBJ, as a relative position with respect to the target object OBJ. In this case, the holding position determination process for the holding candidate position HCP may be considered to be the holding position determination process for the holding candidate position HCP with respect to the target object OBJ selected as the processing execution object.

[0184] The process of selecting the holding target position by performing the determination process on the holding candidate position HCP will be described in detail later with reference to FIG. 10 and the like, and therefore will not be described in detail here.

[0185] Thereafter, the signal generation unit 312 included in the control device 3 generates a robot control signal for controlling at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holds the target object OBJ selected as the processing execution object, based on the position and orientation data POI generated in step S22 and the holding target position selected in step S23 (step S24). Thereafter, the signal generation unit 312 outputs the robot control signal generated in step S24 to the robot control device 13 using the communication device 33. As a result, the robot control device 13 controls at least one of the robot 1, the end effector 4, and the robot movable device based on the robot control signal.

[0186] For example, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the robot movable device so that the end effector 4 approaches the holding target position selected in step S23. Here, as described above, the holding target position selected in step S23 indicates the position on the target object OBJ where the end effector 4 holds the target object OBJ as a relative position with respect to the target object OBJ (i.e., a position in a coordinate system with the target object OBJ as the reference). Therefore, the signal generating unit 312 may convert the holding target position selected in step S23 into a holding target position in the global coordinate system (or the robot coordinate system) based on the position and orientation data POI generated in step S22 (i.e., at least one of the position and orientation of the target object OBJ in the global coordinate system (or the robot coordinate system)). That is, the signal generation unit 312 may calculate a holding target position to which the end effector 4 should approach in order to hold the target object OBJ in the global coordinate system (or the robot coordinate system) based on the position and orientation data POI generated in step S22 and the holding target position selected in step S23. The signal generation unit 312 may then generate a robot control signal for controlling at least one of the robot 1 and the robot movable device so that the end effector 4 approaches the holding target position in the global coordinate system (or the robot coordinate system). As a result, as shown in FIG. 5A above, the end effector 4 approaches the holding target position selected in step S23 in the global coordinate system (or the robot coordinate system). Furthermore, the signal generation unit 312 may generate a robot control signal for controlling at least the end effector 4 so that the end effector 4 holds the target object OBJ at the holding target position. As a result, as shown in FIG. 5B above, the end effector 4 holds the target object OBJ at the holding target position after moving to a position where it can hold the target object OBJ at the holding target position.Furthermore, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the robot movable device so that the end effector 4 holding the target object OBJ moves away from (in other words, moves away from) the placement device T. For example, the control device 3 may generate a robot control signal for controlling at least one of the robot 1 and the robot movable device so that the end effector 4 holding the target object OBJ retreats from the holding target position to a predetermined retreat position away from the holding target position. As a result, as shown in FIG. 5C described above, the end effector 4 holding the target object OBJ moves away from (in other words, moves away from) the placement device T. For example, the end effector 4 holding the target object OBJ retreats from the holding target position to a retreat position away from the holding target position.

[0187] During a period in which at least one of the robot 1, the end effector 4, and the robot movable device is controlled based on the robot control signal (e.g., during a period in which the end effector 4 is approaching the holding target position), the imaging system 2 may again capture an image of the target object OBJ (particularly, the processing execution object), and the position and orientation calculation unit 311 may again acquire image data IMG generated by the imaging system 2 again capturing an image of the target object OBJ (particularly, the processing execution object) (step S21 in FIG. 9 ). The position and orientation calculation unit 311 may again generate position and orientation data POI based on the reacquired image data IMG (step S22 in FIG. 9 ). In other words, the position and orientation calculation unit 311 may update at least one of the position and orientation of the target object OBJ (particularly, the processing execution object). In other words, the position and orientation calculation unit 311 may update the position and orientation data POI. Thereafter, the signal generation unit 312 may again generate a robot control signal based on the updated position and orientation data POI (step S24 in FIG. 9 ). That is, during a period in which at least one of the end effector 4 and the robot movable device is controlled based on the robot control signal (e.g., during a period in which the end effector 4 is approaching a target holding position), the signal generating unit 312 may repeatedly generate the robot control signal (i.e., may repeatedly generate a movement path for the end effector 4). In other words, during a period in which at least one of the end effector 4 and the robot movable device is controlled based on the robot control signal (e.g., during a period in which the end effector 4 is approaching a target holding position), the signal generating unit 312 may update the robot control signal (i.e., may update the movement path for the end effector 4). That is, during a period in which at least one of the end effector 4 and the robot movable device is controlled based on the robot control signal (e.g., during a period in which the end effector 4 is approaching a target holding position), the control device 3 may repeat the processes of steps S21 to S22 and S24 in FIG. 9 .Furthermore, during a period in which at least one of the end effector 4 and the robot movable device is controlled based on the robot control signal (e.g., a period in which the end effector 4 is approaching the holding target position), the control device 3 may repeat the processing of step S23 in FIG. 9 in addition to the processing of steps S21 to S22 and step S24 in FIG. 9.

[0188] In particular, when the target object OBJ moves during a period in which at least one of the end effector 4 and the robot movable device is controlled based on the robot control signal (e.g., during a period in which the end effector 4 is approaching the holding target position), the control device 3 may repeat the processes of steps S21 to S22 and S24 in FIG. 9 . In this case, the imaging system 2 may repeatedly capture images of the moving target object OBJ. As a result, when the target object OBJ moves during a period in which the end effector 4 is moving, the position and orientation calculation unit 311 can appropriately update the position and orientation data POI of the target object OBJ so that the movement of the target object OBJ is reflected in the position and orientation data POI. As a result, the signal generation unit 312 can generate a robot control signal based on the updated position and orientation data POI. Therefore, the signal generation unit 312 can control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holds the moving target object OBJ.

[0189] As an example, the signal generation unit 312 may convert an already selected holding target position (i.e., a relative position with respect to the target object OBJ, i.e., the holding target position in the coordinate system of the target object OBJ) into a holding target position in the global coordinate system (or the robot coordinate system) based on the updated position and orientation data POI. That is, the signal generation unit 312 may update the holding target position in the global coordinate system (or the robot coordinate system) based on the updated position and orientation data POI without selecting a new holding target position. In this case, particularly when the target object OBJ moves while the end effector 4 is approaching the holding target position, the holding target position in the global coordinate system (or the robot coordinate system) is appropriately updated in accordance with the movement of the target object OBJ. That is, under a situation in which the holding target position in the global coordinate system (or the robot coordinate system) changes in accordance with the movement of the target object OBJ in the global coordinate system (or the robot coordinate system), the signal generation unit 312 can appropriately update the holding target position in the global coordinate system (or the robot coordinate system). Thereafter, the signal generator 312 may regenerate a robot control signal based on the updated holding target position in the global coordinate system (or the robot coordinate system). For example, the signal generator 312 may generate a movement path for the end effector 4 to approach the holding target position in the global coordinate system (or the robot coordinate system) and that does not cause interference (e.g., interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB, which will be described in detail later), and regenerate a robot control signal for controlling at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 moves along the generated movement path. Alternatively, for example, the signal generator 312 may correct (in other words, update) an already generated movement path using the updated holding target position in the global coordinate system (or the robot coordinate system), and regenerate a robot control signal for controlling at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 moves along the updated movement path.

[0190] 9 while the end effector 4 is approaching the target holding position. Alternatively, when performing the processes of steps S21 to S24 in Fig. 9 while the end effector 4 is approaching the target holding position, the control device 3 may temporarily stop the end effector 4. In other words, while the end effector 4 is approaching the target holding position, the control device 3 may alternately perform the process of stopping the moving end effector 4, the process of steps S21 to S24, and the process of moving the stopped end effector 4 again.

[0191] (2-3-2) Selection of Holding Target Position Next, the process of selecting the holding target position in step S23 in Fig. 9 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the flow of the process of selecting the holding target position in step S23 in Fig. 9.

[0192] 10 , the signal generating unit 312 selects one holding candidate position HCP from at least one holding candidate position HCP indicated by the holding position information (step S231). As a first example, if the holding position information indicates a single holding candidate position HCP, the signal generating unit 312 may select the single holding candidate position HCP indicated by the holding position information. As a second example, if the holding position information indicates multiple holding candidate positions HCP, the signal generating unit 312 may select one holding candidate position HCP from the multiple holding candidate positions HCP indicated by the holding position information. For example, the signal generating unit 312 may select one holding candidate position HCP assigned the highest priority from the multiple holding candidate positions HCP indicated by the holding position information.

[0193] Thereafter, the signal generating unit 312 performs a holding position determination process for the holding candidate position HCP selected in step S231 (step S232). More specifically, the signal generating unit 312 performs a holding position determination process for the holding candidate position HCP selected in step S231, which is the holding candidate position HCP for the target object OBJ selected as the processing execution object in step S22 of FIG. 9 (step S232).

[0194] The holding position determination process for the holding candidate position HCP may include a process for determining whether or not the holding candidate position HCP is selectable as a holding target position (in other words, determinable, settable, or available for use). More specifically, the holding position determination process for the holding candidate position HCP may include a process for determining whether or not the holding candidate position HCP is selectable as a holding target position for holding the target object OBJ selected as the processing execution object in step S22 of FIG. In other words, the holding position determination process for the holding candidate position HCP may include a process for determining whether or not the holding candidate position HCP selected in step S231 (i.e., the holding candidate position HCP set as a relative position with respect to an arbitrary target object OBJ) is selectable (in other words, determinable, settable, or usable) as a holding target position for holding the target object OBJ selected as the processing execution object in step S22 of Fig. 9, when the holding candidate position HCP selected in step S231 (i.e., the holding candidate position HCP set as a relative position with respect to an arbitrary target object OBJ) is used as the holding candidate position HCP with respect to the target object OBJ selected as the processing execution object in step S22 of Fig. 9. In the following description, unless otherwise specified, the holding candidate position HCP may mean the holding candidate position HCP with respect to the target object OBJ selected as the processing execution object in step S22 of Fig. 9 (i.e., the holding candidate position HCP set on the target object OBJ selected as the processing execution object).

[0195] In this embodiment, an example of the signal generation unit 312 performing interference detection processing for the holding candidate position HCP will be described as an example of the holding position determination processing for the holding candidate position HCP. The interference detection processing for the holding candidate position HCP includes processing for determining interference between at least one of the end effector 4, the robot 1, and the robot movable device and a non-held object NOB other than the target object OBJ to be held by the end effector 4 (i.e., the target object OBJ selected as the processing execution object), assuming that at least one of the robot 1 and the robot movable device operates so that the end effector 4 approaches the holding candidate position HCP. In the following description, the target object OBJ to be held by the end effector 4 (i.e., the processing execution object) will be referred to as the processing execution object OBJ_tgt as appropriate.

[0196] Note that, because a holding process is an example of a process performed by the end effector 4, the non-held object NOB may also be referred to as a non-processing object. The non-processing object may refer to an object different from the object processed by the end effector 4. For example, when a placement process (release process) is performed in which the end effector 4 holding a first target object OBJ places (releases) the first target object OBJ on a second target object OBJ, the first target object OBJ and the second target object OBJ are objects that are the targets of the placement process. Therefore, in this case, the non-processing object (non-placement destination object) may include an object different from the first target object OBJ and the second target object OBJ.

[0197] In order to perform the collision detection process, the signal generation unit 312 may calculate a holding candidate position HCP for the processing execution object OBJ_tgt selected in step S22 of Fig. 9 based on the position and orientation data POI (i.e., at least one of the position and orientation of the processing execution object OBJ_tgt) generated in step S22 of Fig. 9 and the holding candidate position HCP selected in step S231 (i.e., the holding candidate position HCP set as a relative position with respect to an arbitrary target object OBJ). As an example, when the position and orientation data POI generated in step S22 of Fig. 9 indicates at least one of the position and orientation of the processing execution object OBJ_tgt in the global coordinate system or the robot coordinate system, the signal generation unit 312 may calculate a holding candidate position HCP for the processing execution object OBJ_tgt in the global coordinate system or the robot coordinate system. Then, the signal generating unit 312 may determine interference between the end effector 4, the robot 1, and / or the robot movable device and the non-held object NOB, assuming that the robot 1 and / or the robot movable device operate so that the end effector 4 approaches the holding candidate position HCP for the processing execution object OBJ_tgt.

[0198] The state in which two objects are interfering with each other may include a first object interference state in which the two objects are in contact with each other. Therefore, the state in which at least one of the end effector 4, the robot 1, and the robot movable device is interfering with the non-held object NOB may include a first object interference state in which at least one of the end effector 4, the robot 1, and the robot movable device is in contact with the non-held object NOB.

[0199] The state in which two objects are interfering with each other may include a second object interference state in which two objects are colliding with each other. Therefore, the state in which at least one of the end effector 4, the robot 1, and the robot movable device is interfering with the non-held object NOB may include a second object interference state in which at least one of the end effector 4, the robot 1, and the robot movable device is colliding with the non-held object NOB.

[0200] The state in which two objects are interfering with each other may include a third object interference state in which two objects are adjacent to each other. Therefore, the state in which at least one of the end effector 4, the robot 1, and the robot movable device is interfering with the non-held object NOB may include a third object interference state in which at least one of the end effector 4, the robot 1, and the robot movable device is adjacent to the non-held object NOB.

[0201] The state in which two objects are interfering with each other may include a fourth object interference state in which the distance between the two objects is equal to or less than a predetermined first distance threshold. Therefore, the state in which at least one of the end effector 4, the robot 1, and the robot movable device is interfering with the non-held object NOB may include a fourth object interference state in which the distance between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB is equal to or less than a predetermined first distance threshold. Each of the first to third object interference states described above may be considered equivalent to a fourth object interference state in which the first distance threshold is set to 0. On the other hand, if the first distance threshold is set to a value greater than 0, it may be determined that two objects spaced apart are interfering with each other. In other words, the state in which two objects are interfering with each other may include a fourth object interference state in which the two objects are spaced apart (but the distance between the two objects is equal to or less than the predetermined first distance threshold).

[0202] The state in which two objects are interfering with each other may include a fifth object interference state in which the overlapping area of ​​the two objects is equal to or greater than a predetermined first range threshold. Note that, if the two objects are real objects, the overlapping area of ​​the two objects may refer to the area in which the surfaces of the two objects overlap. Specifically, the overlapping area of ​​the two objects may refer to the area in which the surface of one of the two objects overlaps (i.e., touches) with the surface of the other of the two objects. In this case, the overlapping area of ​​the two objects may refer to the area in which the two objects touch. For example, the state in which two objects are interfering with each other may include a fifth object interference state in which the area of ​​the overlapping area of ​​the two objects is equal to or greater than a predetermined first range threshold (area threshold). Therefore, the state in which at least one of the end effector 4, the robot 1, and the robot movable device is interfering with the non-held object NOB may include a fifth object interference state in which the overlapping area of ​​at least one of the end effector 4, the robot 1, and the robot movable device with the non-held object NOB is equal to or greater than a predetermined first range threshold.

[0203] As shown in FIG. 11 , which conceptually illustrates the end effector 4 approaching the holding candidate position HCP, the non-held object NOB may include a placement device T on which the target object OBJ is placed. Note that FIG. 11 illustrates an example in which the placement device T is a container CB. Although not shown for the sake of simplicity, the non-held object NOB may also include a stand on which the placement device T is placed. Although not shown for the sake of simplicity, the non-held object NOB may also include an object (e.g., at least one of piping, a shelf, a wall, and a ceiling) present in a factory in which the robot system SYS is installed. Although not shown for the sake of simplicity, the non-held object NOB may include an object (e.g., a conveying machine such as a belt conveyor) that constitutes a production line in which the robot system SYS is installed.

[0204] 11 , the non-held object NOB may include other target objects OBJ that are placed on the mounting device T and that have not been selected as the process execution object OBJ_tgt to be held by the end effector 4. In the following description, for the sake of convenience, other target objects OBJ that have not been selected as the process execution object OBJ_tgt to be held by the end effector 4 will be referred to as non-process execution objects OBJ_ntgt. In addition, when the end effector 4 holds a plurality of target objects OBJ in turn, the non-process execution objects OBJ_ntgt that have not been selected as the process execution object OBJ_tgt to be held by the end effector 4 in a first period may be selected as the process execution object OBJ_tgt to be held by the end effector 4 in a second period different from the first period. Similarly, the processing execution object OBJ_tgt to be held by the end effector 4 in the first period may be regarded as a non-processing execution object OBJ_ntgt (i.e., a non-held object NOB) in a second period different from the first period.

[0205] When a single non-held object NOB exists, the interference detection process may include a process for determining interference between at least one of the end effector 4, the robot 1, and the robot movable device and the single non-held object NOB. When a plurality of non-held objects NOB exist, the interference detection process may include a process for determining interference between at least one of the end effector 4, the robot 1, and the robot movable device and the plurality of non-held objects NOB.

[0206] To determine interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB, the signal generator 312 may use the end effector model EM, the robot model RM, the robot movable device model, and the non-held object model NOM. In this case, the signal generator 312 does not need to actually operate at least one of the robot 1 and the robot movable device in real space so that the end effector 4 approaches the holding candidate position HCP in order to determine interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB. However, the signal generator 312 may determine interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB without using the end effector model EM, the robot model RM, the robot movable device model, and the non-held object model NOM.

[0207] Furthermore, since one example of the processing performed by the end effector 4 is holding processing, the non-held object model NOM, which is a model of the non-held object NOB, may also be referred to as a non-processed object model, which means a model of the non-processed object described above.

[0208] The end effector model EM is a three-dimensional model of the end effector 4 attached to the robot arm 12. For example, a CAD model of the end effector 4 may be used as the end effector model EM. However, if the end effector 4 is captured in the image data IMG acquired in step S21 of FIG. 9 , a model based on three-dimensional position data (e.g., point cloud data or a depth image) of the end effector 4 generated from the one image data IMG may be used as the end effector model EM. If, apart from the one image data IMG acquired in step S21 of FIG. 9 , the imaging system 2 (or an imaging system different from the imaging system 2; the same applies hereinafter in the description of the interference detection process) captures an image of the end effector 4 to generate other image data IMG, a model based on three-dimensional position data (e.g., point cloud data or a depth image) of the end effector 4 generated from the other image data IMG may be used as the end effector model EM. Examples of a model based on three-dimensional position data (e.g., point cloud data or a depth image) include at least one of a point cloud model, a surface model, a mesh model, and a polygon model. Note that a point cloud model may also be referred to as a point cloud.

[0209] The robot model RM is a three-dimensional model of the robot 1. For example, a CAD model of the robot 1 may be used as the robot model RM. However, if the robot 1 is captured in the image data IMG acquired in step S21 of FIG. 9 , a model based on three-dimensional position data (e.g., point cloud data or depth image) of the robot 1 generated from the image data IMG may be used as the robot model RM. If the imaging system 2 captures images of the robot 1 and generates other image data IMG separate from the image data IMG acquired in step S21 of FIG. 9 , a model based on three-dimensional position data (e.g., point cloud data or depth image) of the robot 1 generated from the other image data IMG may be used as the robot model RM. Examples of models based on three-dimensional position data (e.g., point cloud data or depth image) include at least one of a point cloud model, a surface model, a mesh model, and a polygon model. Note that the point cloud model may also be referred to as a point cloud.

[0210] The robot movable device model is a three-dimensional model of the robot movable device. For example, a CAD model of the robot movable device may be used as the robot movable device model. However, if the robot movable device is captured in the image data IMG acquired in step S21 of FIG. 9 , a model based on three-dimensional position data (e.g., point cloud data or depth image) of the robot movable device generated from the image data IMG may be used as the robot movable device model. If the imaging system 2 generates other image data IMG by capturing an image of the robot movable device, separate from the image data IMG acquired in step S21 of FIG. 9 , a model based on three-dimensional position data (e.g., point cloud data or depth image) of the robot movable device generated from the other image data IMG may be used as the robot movable device model. Examples of models based on three-dimensional position data (e.g., point cloud data or depth image) include at least one of a point cloud model, a surface model, a mesh model, and a polygon model. Note that the point cloud model may also be referred to as a point cloud.

[0211] The non-held object model NOM is a three-dimensional model of the non-held object NOB. For example, a CAD model of the non-held object NOB may be used as the non-held object model NOM. However, if the non-held object NOB is reflected in the image data IMG acquired in step S21 of FIG. 9 , a model based on three-dimensional position data (e.g., point cloud data or depth image) of the non-held object NOB generated from the image data IMG may be used as the non-held object model NOM. If the imaging system 2 generates other image data IMG by capturing the non-held object NOB, separate from the image data IMG acquired in step S21 of FIG. 9 , a model based on three-dimensional position data (e.g., point cloud data or depth image) of the non-held object NOB generated from the other image data IMG may be used as the non-held object model NOM. Examples of models based on three-dimensional position data (e.g., point cloud data or depth image) include at least one of a point cloud model, a surface model, a mesh model, and a polygon model. The point cloud model may also be referred to as a point cloud.

[0212] When the non-held object NOB includes a non-processing execution object OBJ_ntgt that is not selected as the processing execution object OBJ_tgt to be held by the end effector 4, a model based on three-dimensional position data (e.g., point cloud data or depth image) of the non-processing execution object OBJ_ntgt generated from the image data IMG may be used as the non-held object model NOM, which is a three-dimensional model of the non-processing execution object OBJ_ntgt. However, a CAD model of the non-processing execution object OBJ_ntgt (CAD model of the target object OBJ) may also be used as the non-held object model NOM, which is a three-dimensional model of the non-processing execution object OBJ_ntgt. In the following description, the non-retained object model NOM, which is a model of the non-processing execution object OBJ_ntgt, will be referred to as the non-processing execution object model OM_ntgt as necessary to distinguish it from the non-retained object model NOM, which is a model of the non-retained object NOB other than the non-processing execution object OBJ_ntgt. However, unless otherwise specified, the non-retained object model NOM includes both the non-retained object model NOM, which is a model of the non-processing execution object OBJ_ntgt (i.e., the non-processing execution object model OM_ntgt), and the non-retained object model NOM, which is a model of the non-retained object NOB other than the non-processing execution object OBJ_ntgt.

[0213] In this manner, when at least one of the end effector model EM, robot model RM, robot movable device model, and non-held object model NOM is generated based on the image data IMG, the signal generation unit 312 may consider that it is performing the interference detection process based on the image data IMG. In this case, the signal generation unit 312 may acquire the image data IMG before performing the interference detection process, generate at least one of the end effector model EM, robot model RM, robot movable device model, and non-held object model NOM based on the acquired image data IMG, and perform the interference detection process based on at least one of the generated end effector model EM, robot model RM, robot movable device model, and non-held object model NOM. For example, the signal generation unit 312 may generate at least one of the end effector model EM, robot model RM, robot movable device model, and non-held object model NOM based on the image data IMG acquired in step S21 of FIG. 9, and perform the interference detection process based on at least one of the generated end effector model EM, robot model RM, robot movable device model, and non-held object model NOM. For example, the signal generation unit 312 may acquire image data IMG generated by the imaging system 2 separately from the image data IMG acquired in step S21 of Figure 9, generate at least one of the end effector model EM, robot model RM, robot movable device model, and non-held object model NOM based on the acquired image data IMG, and perform interference detection processing based on at least one of the generated end effector model EM, robot model RM, robot movable device model, and non-held object model NOM.

[0214] For ease of explanation, the following description will be given of an example in which CAD models are used for the end effector model EM, the robot model RM, and the robot movable device model, and a model generated based on image data IMG is used for the non-held object model NOM. That is, for ease of explanation, the following description will be given of an example in which the signal generation unit 312 performs the collision detection process based on image data IMG generated by capturing an image of at least a target object OBJ that will not be selected as the processing execution object OBJ_tgt later (i.e., a non-processing execution object OBJ_ntgt that is a non-held object NOB). That is, the following description will be given of an example in which the signal generation unit 312 performs the collision detection process using a non-processing execution object model OM_ntgt generated based on image data IMG generated by capturing an image of at least the non-processing execution object OBJ_ntgt. In this case, the imaging system 2 may capture an image of at least the non-processing execution object OBJ_ntgt before the collision detection process is performed. For example, the imaging system 2 may capture at least the non-processing execution object OBJ_ntgt in order to generate the image data IMG acquired in step S21 of Fig. 9. For example, the imaging system 2 may capture at least the non-processing execution object OBJ_ntgt in order to generate image data IMG separate from the image data IMG acquired in step S21 of Fig. 9.

[0215] As described above, the imaging system 2 typically captures images of at least some of the multiple target objects OBJ placed on the mounting device T. In this case, one of the multiple target objects OBJ captured by the imaging system 2 is selected as the processing object OBJ_tgt, while the target objects OBJ captured by the imaging system 2 other than the one selected as the processing object OBJ_tgt are non-processing objects OBJ_ntgt. Therefore, when generating the image data IMG acquired in step S21 of FIG. 9 , the imaging system 2 is typically likely to capture not only the target object OBJ that will later be selected as the processing object OBJ_tgt, but also the target object OBJ that will later become the non-processing object OBJ_ntgt. Therefore, the signal generator 312 can appropriately perform the collision detection process based on the non-processing object model OM_ntgt generated based on the image data IMG.

[0216] Since the non-processing execution object OBJ_ntgt is the target object OBJ, there is a possibility that three-dimensional position data of the non-processing execution object OBJ_ntgt will be generated by the matching process (step S22 in FIG. 9 ) performed by the position and orientation calculation unit 311. For example, when the position and orientation calculation unit 311 performs 3D matching process, the position and orientation calculation unit 311 generates three-dimensional position data of the non-processing execution object OBJ_ntgt in order to perform the 3D matching process. In this case, the signal generation unit 312 may generate the non-processing execution object model OM_ntgt by utilizing the three-dimensional position data of the non-processing execution object OBJ_ntgt generated by the position and orientation calculation unit 311. However, the signal generation unit 312 may generate three-dimensional position data of the non-processing execution object OBJ_ntgt based on the image data IMG without reusing the three-dimensional position data of the non-processing execution object OBJ_ntgt generated by the position and orientation calculation unit 311, and may generate a non-processing execution object model OM_ntgt based on the generated three-dimensional position data.

[0217] However, the three-dimensional position data of the non-processing object OBJ_ntgt generated based on the image data IMG may include, in addition to data indicating the three-dimensional positions of multiple points on the non-processing object OBJ_ntgt, noise-related data that does not indicate the three-dimensional positions of multiple points on the non-processing object OBJ_ntgt as part of the three-dimensional position data. In this case, the signal generation unit 312 may perform an invalidation process to invalidate the noise-related data in the three-dimensional position data of the non-processing object OBJ_ntgt. For example, if the three-dimensional position data is point cloud data, a set of multiple points that satisfies the condition that the number of points per unit volume is equal to or less than a predetermined noise threshold (i.e., a sparse or non-dense set of multiple points) may be noise that is not multiple points on the non-processing object OBJ_ntgt. Note that the noise threshold may be a threshold set to invalidate part of the three-dimensional position data as noise-related data. In other words, the noise threshold is a threshold set to determine whether to invalidate part of the three-dimensional position data as noise-related data. In this case, the invalidation process may include invalidating a set of multiple points that satisfies the condition that the number of points per unit volume is equal to or less than a predetermined noise threshold. Here, invalidating noise-related data may include deleting noise-related data (i.e., invalidated data) from the three-dimensional position data. Invalidating noise-related data may include not using noise-related data (i.e., invalidated data) included in the three-dimensional position data in the collision detection process. The signal generation unit 312 may then generate a non-processed object model OM_ntgt based on the three-dimensional position data that has been invalidated. Even when the three-dimensional position data is depth image data, the signal generation unit 312 may perform an invalidation process to invalidate noise-related data in the depth image data of the non-processed object OBJ_ntgt, and generate a non-processed object model OM_ntgt based on the depth image data that has been invalidated.Furthermore, even when at least one of the end effector model EM, the robot model RM, the robot movable device model, and the non-held object model NOM different from the non-processing execution object model OM_ntgt is generated based on image data IMG, the signal generation unit 312 may generate at least one of the end effector model EM, the robot model RM, the robot movable device model, and the non-held object model NOM different from the non-processing execution object model OM_ntgt based on three-dimensional position data that has been subjected to the invalidation process.

[0218] 12 , the signal generating unit 312 may place an end effector model EM, a robot model RM, a robot movable device model (not shown in FIG. 12 ), and a non-held object model NOM in a predetermined virtual space. For example, the signal generating unit 312 may place the end effector model EM, the robot model RM, the robot movable device model, and the non-held object model NOM in the predetermined virtual space so that the positional relationship between the end effector model EM, the robot model RM, the robot movable device model, and the non-held object model NOM is the same as the positional relationship between the end effector 4, the robot, the robot movable device, and the non-held object NOB. Here, because the control device 3 controls the operations of the end effector 4, the robot, and the robot movable device, at least one of the positions and orientations of the end effector 4, the robot, and the robot movable device is known information to the control device 3. For example, when the control device 3 can acquire information regarding at least one of the positions and orientations of the end effector 4, the robot, and the robot movable device from the robot control device 13, the control device 3 may consider at least one of the positions and orientations of the end effector 4, the robot, and the robot movable device to be known information to the control device 3. In this case, the control device 3 may place the end effector model EM, the robot model RM, and the robot movable device model, respectively, in a predetermined virtual space at positions and / or orientations of the end effector 4, the robot, and the robot movable device known to the control device 3. Alternatively, when the imaging system 2 captures images of at least one of the end effector 4, the robot, and the robot movable device, the control device 3 may calculate at least one of the positions and orientations of the end effector 4, the robot, and the robot movable device by performing the above-described matching process based on the image data IMG. In this case, the control device 3 may place the end effector model EM, the robot model RM, and the robot movable device model, respectively, in a position and / or orientation calculated by the control device 3 in a predetermined virtual space.

[0219] Furthermore, because the non-processing object OBJ_ntgt is the target object OBJ, the position and orientation calculation unit 311 may perform the above-described matching process (step S22 in FIG. 9 ) to calculate at least one of the position and orientation of the non-processing object OBJ_ntgt in addition to at least one of the position and orientation of the processing object OBJ_tgt. Alternatively, in step S232 in FIG. 10 , the signal generation unit 312 may perform the above-described matching process (step S22 in FIG. 9 ) based on the image data IMG to calculate at least one of the position and orientation of the non-processing object OBJ_ntgt. In this case, for example, the control device 3 may place the non-processing object model OM_ntgt in a predetermined virtual space at the position of the non-processing object OBJ_ntgt calculated by the matching process. For example, the control device 3 may place a non-processing execution object model OM_ntgt in a predetermined virtual space in the orientation of the non-processing execution object OBJ_ntgt calculated by the matching process. Furthermore, the control device 3 may acquire information on at least one of the position and orientation of a non-held object NOB (e.g., a placement device T) other than the non-processing execution object OBJ_ntgt in the predetermined virtual space, and place a non-held object model NOM corresponding to a non-held object NOB other than the non-processing execution object OBJ_ntgt in the virtual space based on the acquired information.

[0220] The signal generating unit 312 may then perform a simulation in which the end effector model EM, the robot model RM, and the robot movable device model are moved in a virtual space in accordance with the movements of the end effector 4, the robot, and the robot movable device, assuming that at least one of the robot 1 and the robot movable device operates so that the end effector 4 approaches the holding candidate position HCP for the processing execution object OBJ_tgt. In this case, the signal generating unit 312 may determine interference between at least one of the end effector model EM, the robot model RM, and the robot movable device model and the non-held object model NOM based on the results of the simulation, thereby determining interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB. For example, if it is determined that at least one of the end effector model EM, the robot model RM, and the robot movable device model will interfere with the non-held object model NOM, the signal generating unit 312 may determine that at least one of the end effector 4, the robot 1, and the robot movable device will interfere with the non-held object NOB. For example, if it is determined that at least one of the end effector model EM, the robot model RM, and the robot movable device model does not interfere with the non-held object model NOM, the signal generating unit 312 may determine that at least one of the end effector 4, the robot 1, and the robot movable device does not interfere with the non-held object NOB.

[0221] As described above, the state in which two objects interfere may include a first object interference state in which the two objects are in contact. In this case, the state in which two models corresponding to the two objects interfere may include a first model interference state in which the two models are in contact. In other words, the state in which the two models interfere may include a first model interference state corresponding to the first object interference state. Therefore, the state in which at least one of the end effector model EM, the robot model RM, and the robot movable device model interferes with the non-held object model NOM may include a first model interference state in which at least one of the end effector model EM, the robot model RM, and the robot movable device model contacts the non-held object model NOM.

[0222] As described above, the state in which two objects interfere may include a second object interference state in which the two objects collide. In this case, the state in which two models corresponding to the two objects interfere may include a second model interference state in which the two models collide. In other words, the state in which the two models interfere may include a second model interference state corresponding to the second object interference state. Therefore, the state in which at least one of the end effector model EM, the robot model RM, and the robot movable device model interferes with the non-held object model NOM may include a second model interference state in which at least one of the end effector model EM, the robot model RM, and the robot movable device model collide with the non-held object model NOM.

[0223] As described above, the state in which two objects interfere may include a third object interference state in which the two objects are adjacent. In this case, the state in which two models corresponding to the two objects interfere may include a third model interference state in which the two models are adjacent. In other words, the state in which two models interfere may include a third model interference state corresponding to the third object interference state. Therefore, the state in which at least one of the end effector model EM, the robot model RM, and the robot movable device model interferes with the non-held object model NOM may include a third model interference state in which at least one of the end effector model EM, the robot model RM, and the robot movable device model is adjacent to the non-held object model NOM.

[0224] As described above, the state in which two objects interfere may include a fourth object interference state in which the distance between the two objects is equal to or less than a predetermined first distance threshold. In this case, the state in which two models corresponding to two objects interfere may include a fourth model interference state in which the distance between the two models is equal to or less than a predetermined second distance threshold. In other words, the state in which two models interfere may include a fourth model interference state corresponding to the fourth object interference state. Therefore, the state in which at least one of the end effector model EM, the robot model RM, and the robot movable device model interferes with the non-held object model NOM may include a fourth model interference state in which the distance between at least one of the end effector model EM, the robot model RM, and the robot movable device model and the non-held object model NOM is equal to or less than a predetermined second distance threshold. Note that each of the first to third model interference states described above may be considered equivalent to a fourth model interference state in which the second distance threshold is set to 0. On the other hand, if the second distance threshold is set to a value greater than 0, two models that are spaced apart may be determined to be interfering. In other words, the state in which two models are interfering may include a fourth model interference state in which the two models are separated (but the distance between the two models is less than or equal to a predetermined second distance threshold).

[0225] The second distance threshold may be a distance determined based on the first distance threshold described above. This is because, as described above, the fourth object interference state using the first distance threshold and the fourth model interference state using the second distance threshold correspond to each other. For example, the second distance threshold may be set to a distance that satisfies the condition that, under a situation in which two models corresponding to two objects are arranged in a virtual space such that the positional relationship between the two models is the same as the positional relationship between the two objects and the posture relationship between the two models is the same as the posture relationship between the two objects, the distance between the two models becomes equal to or less than the second distance threshold, thereby determining that the distance between the two objects is equal to or less than the first distance threshold. The second distance threshold may be the same as or different from the first distance threshold described above. In this case, determining whether two models are interfering based on the second distance threshold may be considered equivalent to determining whether two objects corresponding to the two models are interfering based on the first distance threshold. For example, determining whether or not at least one of the end effector model EM, the robot model RM, and the robot movable device model interferes with the non-held object model NOM based on the second distance threshold may be considered equivalent to determining whether or not at least one of the end effector 4, the robot 1, and the robot movable device interferes with the non-held object NOB based on the first distance threshold.

[0226] As described above, the state in which two objects are interfering may include a fifth object interference state in which the overlapping area of ​​the two objects is equal to or less than a predetermined first range threshold. In this case, the state in which two models corresponding to two objects are interfering may include a fifth model interference state in which the overlapping area of ​​the surfaces of the two models is equal to or greater than a predetermined second range threshold. In other words, the state in which two models are interfering may include a fifth model interference state corresponding to the fifth object interference state. Note that the overlapping area of ​​the surfaces of the two models may refer to the area in which the surface of one of the two models overlaps (i.e., touches) with the surface of the other of the two models. In this case, the overlapping area of ​​the two models may refer to the area in which the two models are in contact. As an example, the state in which two models are interfering may include a fifth model interference state in which the area in which the surfaces of the two models are overlapping is equal to or greater than a predetermined second range threshold. Therefore, the state in which at least one of the end effector model EM, robot model RM, and robot movable device model interferes with the non-held object model NOM may include a fifth model interference state in which the range in which the surface of at least one of the end effector model EM, robot model RM, and robot movable device model overlaps with the surface of the non-held object model NOM is greater than or equal to a predetermined second range threshold.

[0227] The second range threshold may be a range determined based on the first range threshold described above. This is because, as described above, the fifth object interference state using the first range threshold and the fifth model interference state using the second range threshold correspond to each other. For example, the second range threshold may be set to a range that satisfies the condition that, under a situation in which two models corresponding to two objects are arranged in a virtual space such that the positional relationship between the two models is the same as the positional relationship between the two objects and the orientation relationship between the two models is the same as the orientation relationship between the two objects, the overlapping area of ​​the surfaces of the two models is equal to or greater than the second range threshold, thereby causing the overlapping area of ​​the two objects to be equal to or greater than the first range threshold. The second range threshold may be the same as or different from the first range threshold described above. In this case, determining whether two models are interfering based on the second range threshold may be considered equivalent to determining whether two objects corresponding to the two models are interfering based on the first range threshold. For example, determining whether or not at least one of the end effector model EM, the robot model RM, and the robot movable device model interferes with the non-held object model NOM based on the second range threshold may be considered equivalent to determining whether or not at least one of the end effector 4, the robot 1, and the robot movable device interferes with the non-held object NOB based on the first range threshold.

[0228] The state in which two models corresponding to two objects interfere with each other may include a sixth model interference state in which the overlapping range of the interiors (i.e., inner surfaces) of the two models is equal to or greater than a predetermined third range threshold. The overlapping range of the interiors of the two models may refer to the overlapping range of the interior of one of the two models with the interior of the other of the two models. As one example, the state in which two models interfere with each other may include a sixth model interference state in which the volume of the overlapping range of the two models is equal to or greater than a predetermined third range threshold. As another example, the state in which two point cloud models interfere with each other may include a sixth model interference state in which the number of points included in the overlapping range of the two point cloud models is equal to or greater than a predetermined third range threshold. Therefore, the state in which at least one of the end effector model EM, the robot model RM, and the robot movable device model interferes with the non-held object model NOM may include a sixth model interference state in which the overlapping range of the interior of at least one of the end effector model EM, the robot model RM, and the robot movable device model with the non-held object model NOM is equal to or greater than a predetermined third range threshold.

[0229] In the following explanation, unless otherwise specified, the state in which "two models overlap" will mean at least one of the state in which "the surfaces of two models overlap" assumed in the fifth model interference state and the state in which "the interiors (inner surfaces) of two models overlap" assumed in the sixth model interference state.

[0230] If the interference detection process determines that at least one of the end effector 4, the robot 1, and the robot movable device will interfere with the non-held object NOB, when at least one of the robot 1 and the robot movable device actually operates so that the end effector 4 actually approaches the holding candidate position HCP, there is a high possibility that at least one of the end effector 4, the robot 1, and the robot movable device will actually interfere with the non-held object NOB. Therefore, in this case, the signal generation unit 312 may determine that the holding candidate position HCP selected in step S231 cannot be selected as the holding target position.

[0231] Alternatively, if the interference determination process determines that at least one of the end effector 4, the robot 1, and the robot movable device will interfere with the non-held object NOB, the signal generation unit 312 may change the movement path of the end effector 4 as it moves closer to the holding candidate position HCP, and then repeat the process of determining interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB. As a result, if it is not possible to generate a movement path for the end effector 4 as it moves closer to the holding candidate position HCP without interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB, there is a high possibility that at least one of the end effector 4, the robot 1, and the robot movable device will actually interfere with the non-held object NOB when at least one of the robot 1 and the robot movable device actually operates so that the end effector 4 actually approaches the holding candidate position HCP. In other words, if it is determined that at least one of the end effector 4, the robot 1, and the robot movable device will interfere with the non-held object NOB even if the end effector 4 moves to approach the holding candidate position HCP along each of multiple possible movement paths of the moving end effector 4, then when at least one of the robot 1 and the robot movable device actually operates so that the end effector 4 actually approaches the holding candidate position HCP, there is a high possibility that at least one of the end effector 4, the robot 1, and the robot movable device will actually interfere with the non-held object NOB. Therefore, in this case, the signal generation unit 312 may determine that the holding candidate position HCP selected in step S231 cannot be selected as the holding target position.

[0232] On the other hand, if the interference detection process determines that at least one of the end effector 4, the robot 1, and the robot movable device will not interfere with the non-held object NOB (i.e., it is possible to generate a movement path for the end effector 4 that moves toward the holding candidate position HCP without interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB), then when at least one of the robot 1 and the robot movable device actually operates so that the end effector 4 actually approaches the holding candidate position HCP, there is a low possibility that at least one of the end effector 4, the robot 1, and the robot movable device will actually interfere with the non-held object NOB. Therefore, in this case, the signal generation unit 312 may determine that the holding candidate position HCP selected in step S231 can be selected as the holding target position.

[0233] The interference detection process may include a process for determining interference between at least one of the end effector 4, the robot 1, and the robot movable device and a process-executing object OBJ_tgt that is to be held by the end effector 4. Here, interference between the end effector 4 and the process-executing object OBJ_tgt that is to be held by the end effector 4 may refer to interference between a portion of the end effector 4 that should not interfere (e.g., contact) with the process-executing object OBJ_tgt and the process-executing object OBJ_tgt that is to be held by the end effector 4. For example, when the end effector 4 holds the process-executing object OBJ_tgt by bringing a first portion of the end effector 4 into contact with the process-executing object OBJ_tgt, interference between the end effector 4 and the process-executing object OBJ_tgt may refer to interference between a second portion of the end effector 4, different from the first portion, and the process-executing object OBJ_tgt. As an example, if the end effector 4 is a magnetic gripper, interference between the end effector 4 and the processing execution object OBJ_tgt may mean interference between the processing execution object OBJ_tgt and a part of the magnetic gripper other than the magnet part (specifically, the part that adheres to the processing execution object OBJ_tgt by magnetic force).

[0234] When the interference detection process includes a process for determining interference between at least one of the end effector 4, the robot 1, and the robot movable device and the process execution object OBJ_tgt, the signal generation unit 312 may perform a simulation of moving the end effector model EM, the robot model RM, and the robot movable device model in a virtual space in which the end effector model EM, the robot model RM, the robot movable device model (not shown in FIG. 12), and the object model OM representing the three-dimensional shape of the process execution object OBJ_tgt are arranged, as shown in Fig. 12. Note that the object model OM has already been described, so its description will be omitted here.

[0235] Furthermore, when the imaging system 2 is attached to the robot 1, the imaging system 2 also moves in accordance with the movement of the end effector 4. In this case, the interference detection process may include a process of determining interference between the imaging system 2 and at least one of the non-held object NOB and the processing execution object OBJ_tgt to be held by the end effector 4.

[0236] If the interference detection process includes a process for determining interference between the imaging system 2 and at least one of the non-retained object NOB and the processing execution object OBJ_tgt, the signal generation unit 312 may perform a simulation of moving the imaging system model in a virtual space in which the imaging system model representing the three-dimensional shape of the imaging system 2 and at least one of the non-retained object model NOM and the object model OM are arranged. The imaging system model is a three-dimensional model of the imaging system 2. For example, a CAD model of the imaging system 2 may be used as the imaging system model. If the image data IMG is generated by an imaging system different from the imaging system 2 capturing images of the imaging system 2, three-dimensional position data (e.g., a point cloud model) of the imaging system 2 generated from the image data IMG may be used as the imaging system model. A three-dimensional model (e.g., at least one of a surface model, a mesh model, and a polygon model) generated from the three-dimensional position data (e.g., a point cloud model) of the imaging system 2 may be used as the imaging system model. Note that the point cloud model may also be referred to as a point cloud.

[0237] To place the imaging system 2 in the virtual space, the control device 3 may calculate at least one of the position and orientation of the imaging system 2 by performing a matching process based on the image data IMG obtained by capturing an image of the imaging system 2 using an imaging system different from the imaging system 2. The control device 3 may then place an imaging system model in a predetermined virtual space at the position and / or orientation calculated by the control device 3. Alternatively, the control device 3 may acquire information regarding at least one of the positions and orientations of the end effector 4, the robot, and the robot movable device from the robot control device 13. The control device 3 may then calculate at least one of the position and orientation of the imaging system 2 based on information regarding the positional relationship between the imaging system 2 and at least one of the end effector 4, the robot, and the robot movable device, and information acquired from the robot control device 13. The control device 3 may then place the imaging system model in a predetermined virtual space at the position and / or orientation calculated by the control device 3.

[0238] As described above, after the end effector 4 holds the processing execution object OBJ_tgt, the end effector 4 holding the processing execution object OBJ_tgt may move to retreat from the holding target position. In this case, the interference determination process may include a process of determining interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB, assuming that at least one of the robot 1 and the robot movable device operates to retreat the end effector 4 from the holding candidate position HCP. In this case, the signal generation unit 312 may determine interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB by performing a simulation in which the end effector model EM, the robot model RM, and the robot movable device model are moved in a virtual space in accordance with the movements of the end effector 4, the robot, and the robot movable device, assuming that at least one of the robot 1 and the robot movable device operates to retreat the end effector 4 from the holding candidate position HCP.

[0239] As an example, the signal generating unit 312 may determine interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB, assuming that at least one of the robot 1 and the robot movable device operates to move the end effector 4 from the holding candidate position HCP to a predetermined target position. The predetermined target position may include a predetermined retreat position to which the end effector 4 holding the processing execution object OBJ_tgt should retreat. The predetermined target position may also include a placement position to which the end effector 4 holding the processing execution object OBJ_tgt should place the processing execution object OBJ_tgt. If, as a result of such interference detection processing, it is determined that at least one of the end effector 4, the robot 1, and the robot movable device will interfere with the non-held object NOB, there is a high possibility that at least one of the end effector 4, the robot 1, and the robot movable device will actually interfere with the non-held object NOB if at least one of the robot 1 and the robot movable device actually operates to retreat the end effector 4 from the holding candidate position HCP. Alternatively, if the interference determination process determines that at least one of the end effector 4, the robot 1, and the robot movable device will interfere with the non-held object NOB, the signal generation unit 312 may change the movement path of the end effector 4 as it moves to retreat from the holding candidate position HCP, and then repeat the process of determining interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB. As a result, if it is not possible to generate a movement path for the end effector 4 as it moves to retreat from the holding candidate position HCP without interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB, there is a high possibility that at least one of the end effector 4, the robot 1, and the robot movable device will actually interfere with the non-held object NOB when at least one of the robot 1 and the robot movable device actually operates to cause the end effector 4 to retreat from the holding candidate position HCP.In other words, if it is determined that at least one of the end effector 4, the robot 1, and the robot movable device will interfere with the non-held object NOB even if the end effector 4 moves to retreat from the holding candidate position HCP along each of multiple possible movement paths that can be assumed as the movement path of the moving end effector 4, then when the robot 1 and at least one of the robot movable device actually operate to actually cause the end effector 4 to retreat from the holding candidate position HCP, there is a high possibility that at least one of the end effector 4, the robot 1, and the robot movable device will actually interfere with the non-held object NOB. On the other hand, if the interference detection process determines that at least one of the end effector 4, robot 1, and robot movable device will not interfere with the non-held object NOB (i.e., a movement path can be generated for the end effector 4 to move away from the holding candidate position HCP without interference between at least one of the end effector 4, robot 1, and robot movable device and the non-held object NOB), then when the robot 1 and at least one of the robot movable device actually operate to actually cause the end effector 4 to retreat from the holding candidate position HCP, there is a low possibility that at least one of the end effector 4, robot 1, and robot movable device will actually interfere with the non-held object NOB.

[0240] The signal generating unit 312 may determine that the holding candidate position HCP selected in step S231 can be selected as the holding target position based on the result of the interference determination process when it is assumed that at least one of the robot 1 and the robot movable device will operate so that the end effector 4 will approach the holding candidate position HCP and the result of the interference determination process when it is assumed that at least one of the robot 1 and the robot movable device will operate so that the end effector 4 will retreat from the holding candidate position HCP. For example, if it is determined that no interference will occur by both the interference determination process when it is assumed that the end effector 4 will approach the holding candidate position HCP and the interference determination process when it is assumed that the end effector 4 will retreat from the holding candidate position HCP, the signal generating unit 312 may determine that the holding candidate position HCP selected in step S231 can be selected as the holding target position. On the other hand, for example, even if the interference detection process determines that no interference will occur when the end effector 4 approaches the holding candidate position HCP, if the interference detection process determines that interference will occur when the end effector 4 retreats from the holding candidate position HCP, the signal generation unit 312 may determine that the holding candidate position HCP selected in step S231 cannot be selected as the holding target position.

[0241] Incidentally, the interference detection process when it is assumed that at least one of the robot 1 and the robot movable device operates to retract the end effector 4 from the holding candidate position HCP may include a process of determining interference between the processing execution object OBJ_tgt held by the end effector 4 and the non-held object NOB, assuming that the end effector 4 is holding the processing execution object OBJ_tgt. In this case, the signal generation unit 312 may add an object model OM indicating the three-dimensional shape of the processing execution object OBJ_tgt held by the end effector 4 to the end effector model EF, and then perform a simulation of moving the end effector model EF to which the object model OM has been added.

[0242] However, when the end effector 4 retreats from the holding candidate position HCP, the end effector 4 is already holding the processing execution object OBJ_tgt. Therefore, the signal generating unit 312 may add an object model OM indicating the three-dimensional shape of the processing execution object OBJ_tgt to the end effector model EM, and then perform a simulation of moving the end effector model EM with the added object model OM in a virtual space.

[0243] 10 , the signal generator 312 then determines whether a holding position determination termination condition is met (step S233). The holding position determination termination condition may include a first termination condition that "the holding position determination process (in this embodiment, the interference determination process, the same applies hereinafter) for all holding candidate positions HCP indicated by the holding position information has been completed." The holding position determination termination condition may include a second termination condition that "the holding position determination process has determined that at least one holding candidate position HCP can be selected as a holding target position." Note that when an interference determination process is performed as the holding position determination process, the holding position determination termination condition may be referred to as an interference determination termination condition.

[0244] If it is determined in step S233 that the holding position determination termination condition is not satisfied (step S233: No), the signal generation unit 312 repeats the processes from step S231 to step S232. That is, the signal generation unit 312 selects a new holding candidate position HCP that has not yet been selected for the holding position determination process from among at least one holding candidate position HCP indicated by the holding position information (step S231). For example, the signal generation unit 312 may select a new holding candidate position HCP that has not yet been selected for the holding position determination process and that has the highest priority from among multiple holding candidate positions HCP indicated by the holding position information. Then, the signal generation unit 312 performs the holding position determination process for the holding candidate position HCP newly selected in step S231 (step S232).

[0245] In this way, the signal generating unit 312 may repeat the holding position determination process for determining whether or not the holding candidate position HCP can be selected as the holding target position until it is determined that the holding position determination termination condition is satisfied. As an example, the signal generating unit 312 may perform the holding position determination process for determining whether or not the first holding candidate position HCP indicated by the holding position information can be selected as the holding target position, and may perform the holding position determination process for determining whether or not the second holding candidate position HCP indicated by the holding position information and different from the first holding candidate position HCP can be selected as the holding target position.

[0246] As described above, in this embodiment, an example is described in which interference determination processing is performed as the holding position determination processing. In this case, the signal generation unit 312 may perform interference determination processing to determine whether a first holding candidate position HCP indicated by the holding position information can be selected as a holding target position, and may perform interference determination processing to determine whether a second holding candidate position HCP indicated by the holding position information and different from the first holding candidate position HCP can be selected as a holding target position. The interference determination processing to determine whether the first holding candidate position HCP indicated by the holding position information can be selected as a holding target position may include processing to determine interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB, assuming that at least one of the robot 1 and the robot movable device operates so that the end effector 4 approaches the first holding candidate position HCP. The interference detection process for determining whether the second holding candidate position HCP indicated by the holding position information can be selected as the holding target position may include a process for determining interference between the end effector 4, the robot 1, and at least one of the robot movable devices and the non-held object NOB, assuming that the robot 1 and at least one of the robot movable devices operate so as to bring the end effector 4 closer to the second holding candidate position HCP.

[0247] Note that, when the signal generating unit 312 selects the storage candidate positions HCP in order of priority in step S231, the signal generating unit 312 may be considered to be performing the storage candidate position determination process for the storage candidate positions HCP in order of priority of the storage candidate positions HCP in step S232. In this case, it may be considered that a priority is set (assigned) to the storage candidate positions HCP. In other words, it may be considered that a priority is set (assigned) to the storage candidate positions HCP in the storage candidate position determination process for the storage candidate positions HCP. For example, if the priority of a first storage candidate position HCP indicated by the storage position information is higher than the priority of a second storage candidate position HCP indicated by the storage position information, the priority of the storage candidate position determination process for the first storage candidate position HCP may be considered to be higher than the priority of the storage candidate position determination process for the second storage candidate position HCP.

[0248] Furthermore, even if a priority is not assigned to the holding candidate position HCP, the signal generating unit 312 may repeat the holding position determination process to determine whether or not the holding candidate position HCP can be selected as a holding target position until it is determined that the holding position determination termination condition (particularly, the first termination condition that "the holding position determination process (in this embodiment, the interference determination process; the same applies hereinafter) for all holding candidate positions HCP indicated by the holding position information has been completed") is met.

[0249] On the other hand, if the result of the determination in step S233 indicates that the holding position determination termination condition is met (step S233: Yes), the signal generating unit 312 selects a holding target position based on the result of the holding position determination process (step S234). Specifically, the signal generating unit 312 does not select, as a holding target position, a holding candidate position HCP determined by the holding position determination process to be incapable of being selected as a holding target position. The signal generating unit 312 selects, as a holding target position, a holding candidate position HCP determined by the holding position determination process to be selectable as a holding target position. For example, if the holding position determination process determines that a single holding candidate position HCP is selectable as a holding target position, the signal generating unit 312 may select, as a holding target position, the single holding candidate position HCP determined to be selectable as a holding target position. For example, if the holding position determination process determines that multiple holding candidate positions HCP can be selected as holding target positions, the signal generation unit 312 may select one of the multiple holding candidate positions HCP determined to be selectable as holding target positions as the holding target position. For example, the signal generation unit 312 may select, as the holding target position, one holding candidate position HCP assigned the highest priority among the multiple holding candidate positions HCP determined to be selectable as holding target positions. For example, the signal generation unit 312 may select, as the holding target position, one holding candidate position HCP that satisfies the condition that "the likelihood of interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB is lowest" among the multiple holding candidate positions HCP determined to be selectable as holding target positions. In addition, the condition "the possibility of interference between at least one of the end effector 4, robot 1 and robot movable device and the non-held object NOB is the smallest" may also be the condition "the distance between at least one of the end effector 4, robot 1 and robot movable device and the non-held object NOB is the shortest when at least one of the end effector 4, robot 1 and robot movable device and the non-held object NOB are closest to each other."For example, the signal generating unit 312 may select, as the holding target position, one holding candidate position HCP that satisfies the condition that "the movement path to the holding candidate position HCP is the shortest" from among multiple holding candidate positions HCP that have been determined to be selectable as the holding target position.

[0250] As described above, in this embodiment, an example is described in which the interference detection process is performed as the holding position determination process. In this case, in step S234, the signal generation unit 312 may select, as the holding target position, one holding candidate position HCP that satisfies the condition that "the interference detection process has determined that at least one of the end effector 4, the robot 1, and the robot movable device will not interfere with the non-held object NOB."

[0251] As described above, the holding position determination process (interference determination process in this embodiment) in step S232 may be performed multiple times. In this case, in step S234, the signal generation unit 312 may consider that the holding target position is selected based on the determination results of the multiple holding position determination processes. For example, an example will be described in which the signal generation unit 312 performs the holding position determination process for the first holding candidate position HCP and the holding position determination process for the second holding candidate position HCP in step S232. In this case, the signal generation unit 312 may consider that the holding target position is selected based on the determination results of the holding position determination process for the first holding candidate position HCP and the determination results of the holding position determination process for the second holding candidate position HCP. For example, if the holding position determination process for the first holding candidate position HCP determines that the first holding candidate position HCP can be selected as a holding target position, the signal generation unit 312 may select the first holding candidate position HCP as a holding target position based on the result of the determination in the holding position determination process for the first holding candidate position HCP. On the other hand, if the holding position determination process for the first holding candidate position HCP determines that the first holding candidate position HCP cannot be selected as a holding target position, the signal generation unit 312 may perform holding position determination process for the second holding candidate position HCP. As a result, if the holding position determination process for the second holding candidate position HCP determines that the second holding candidate position HCP can be selected as a holding target position, the signal generation unit 312 may select the second holding candidate position HCP as a holding target position based on the result of the determination in the holding position determination process for the second holding candidate position HCP. On the other hand, if the signal generating unit 312 determines through the holding position determination process for the second holding candidate position HCP that the second holding candidate position HCP cannot be selected as a holding target position, the signal generating unit 312 may further perform holding position determination process for the third holding candidate position HCP.

[0252] (2-3-3) Two Types of Interference Detection Processing In this embodiment, in step S232 of FIG. 10 , the signal generation unit 312 may perform at least two types of holding position determination processing with different setting conditions as the holding position determination processing. Here, the setting conditions are conditions set for the holding position determination processing. Below, an example will be described in which the setting conditions include a determination condition that is a condition related to the holding position determination processing. The determination condition may be a condition that defines the content of the holding position determination processing. Also, below, for convenience of explanation, an example will be described in which the signal generation unit 312 performs two types of holding position determination processing with different setting conditions. However, the signal generation unit 312 may perform three or more types of holding position determination processing with different setting conditions.

[0253] As described above, in this embodiment, the signal generation unit 312 performs the interference detection process as the holding position determination process. Therefore, in step S232 of Fig. 10, the signal generation unit 312 may perform two types of interference detection processes with different setting conditions as the holding position determination process. In this case, the setting conditions are conditions set for the interference detection process.

[0254] In the following description, an example will be described in which the signal generating unit 312 performs, as the collision detection process, a first collision detection process based on one piece of image data IMG generated by the imaging system 2 capturing an image of at least a portion of the target objects OBJ placed on the mounting device T (particularly, a target object OBJ that will later become a non-processing execution object OBJ_ntgt), and a second collision detection process based on multiple pieces of image data IMG generated by the imaging system 2 capturing an image of at least a portion of the target objects OBJ placed on the mounting device T (particularly, a target object OBJ that will later become a non-processing execution object OBJ_ntgt). In this case, the first collision detection process and the second collision detection process may be considered to be two types of collision detection process in which the imaging system 2 captures images a different number of times to generate the image data IMG used in the collision detection process. In other words, the first collision detection process and the second collision detection process may be considered to be two types of collision detection process in which the “number of times the imaging system 2 captures images (conditions related to the number of times the imaging system 2 captures images),” which is an example of a setting condition, is different from each other. Specifically, the first interference detection process may be an interference detection process in which the condition regarding the number of times images are taken is set to "one image taken," and the second interference detection process may be an interference detection process in which the condition regarding the number of times images are taken is set to "multiple images taken (two or more)."

[0255] As described above, the image data IMG used in the collision detection process may be the image data IMG acquired in step S21 of FIG. 9 and image data IMG generated by the imaging system 2 (or another imaging system; the same applies hereinafter) separately from the image data IMG acquired in step S21 of FIG. 9 . Therefore, the image data IMG used in the first collision detection process may be the image data IMG acquired in step S21 of FIG. 9 , or image data IMG generated by the imaging system 2 separately from the image data IMG acquired in step S21 of FIG. 9 . Furthermore, because the second collision detection process uses multiple image data IMG, the imaging system 2 may generate multiple image data IMG and the control device 3 may acquire multiple image data IMG in step S21 of FIG. 9 . In this case, the multiple image data IMG acquired in step S21 may be used as the multiple image data IMG used in the second collision detection process. Some of the plurality of image data IMG acquired in step S21 may be used as the plurality of image data IMG used in the second collision detection process. The same applies when image data IMG generated by the imaging system 2 is used separately from the image data IMG acquired in step S21 of Fig. 9. Note that if the control device 3 acquires the plurality of image data IMG in step S21 of Fig. 9, in step S22 of Fig. 9, the control device 3 may calculate at least one of the position and orientation of the target object OBJ based on one of the plurality of image data IMG acquired in step S21 of Fig. 9.

[0256] The image data IMG used in the first collision detection process may be used as one of the plurality of image data IMG used in the second collision detection process, or each of the plurality of image data IMG used in the second collision detection process may be generated (acquired) separately from the image data IMG used in the first collision detection process.

[0257] As described above, the image data IMG acquired for performing the collision detection processing is used to generate a non-processing execution object model OM_ntgt used in the simulation of the collision detection processing. In this case, when performing the first collision detection processing based on one image data IMG, the signal generation unit 312 may generate a non-processing execution object model OM_ntgt based on the one image data IMG and perform the collision detection processing based on the generated non-processing execution object model OM_ntgt. On the other hand, when performing the second collision detection processing based on multiple image data IMG, the signal generation unit 312 may generate multiple non-processing execution object models OM_ntgt based on the multiple image data IMG and perform multiple collision detection processing based on the generated multiple non-processing execution object models OM_ntgt.

[0258] Here, a non-processing execution object model OM_ntgt generated based on image data IMG may have a loss of information related to the non-processing execution object OBJ_ntgt. For example, if the non-processing execution object model OM_ntgt is a point cloud model, a non-processing execution object model OM_ntgt generated based on image data IMG may have a loss of points corresponding to a portion of the non-processing execution object OBJ_ntgt. Similarly, even when a non-processing execution object model OM_ntgt corresponding to a surface model, mesh model, or polygon model is generated based on image data IMG, each model may have a loss of a portion corresponding to a portion of the non-processing execution object OBJ_ntgt. On the other hand, the location where the loss of information related to the non-processing execution object OBJ_ntgt occurs may differ between multiple non-processing execution object models OM_ntgt generated based on multiple image data IMG. Therefore, the plurality of collision detection processes based on the plurality of non-processing object models OM_ntgt, each generated based on the plurality of image data IMG, may be collision detection processes based on the plurality of non-processing object models OM_ntgt in which the portions of the non-processing object OBJ_ntgt that are missing information differ from one another. In this case, for example, the plurality of collision detection processes may include a collision detection process based on a first non-processing object model OM_ntgt in which information about a first portion of the processing object OBJ_ntgt is missing, and a collision detection process based on a second non-processing object model OM_ntgt in which information about a second portion of the processing object OBJ_ntgt that is different from the first portion is missing. As a result, the plurality of collision detection processes based on the plurality of non-processing object models OM_ntgt, each generated based on the plurality of image data IMG, are essentially equivalent to collision detection processes based on non-processing object models OM_ntgt with fewer missing information. As a result, compared to when a first interference detection process is performed based on a single image data IMG, the signal generation unit 312 can accurately determine interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB (especially the non-processing execution object OBJ_ntgt) by performing a second interference detection process based on multiple image data IMG.For example, compared to when the first collision detection process is performed based on a single piece of image data IMG, by performing the second collision detection process based on a plurality of image data IMG, the signal generation unit 312 is less likely to erroneously determine that at least one of the end effector 4, the robot 1, and the robot movable device will not interfere with the non-processing execution object OBJ_ntgt in a scene where at least one of the end effector 4, the robot 1, and the robot movable device will actually interfere with the non-processing execution object OBJ_ntgt. For example, compared to when the first collision detection process is performed based on a single piece of image data IMG, by performing the second collision detection process based on a plurality of image data IMG, the signal generation unit 312 is more likely to accurately determine that at least one of the end effector 4, the robot 1, and the robot movable device will interfere with the non-processing execution object OBJ_ntgt in a scene where at least one of the end effector 4, the robot 1, and the robot movable device will actually interfere with the non-processing execution object OBJ_ntgt. Similarly, compared to when the first collision detection process is performed based on a single piece of image data IMG, by performing the second collision detection process based on a plurality of image data IMG, the signal generation unit 312 is less likely to erroneously determine that at least one of the end effector 4, the robot 1, and the robot movable device will interfere with the non-processing execution object OBJ_ntgt in a situation where at least one of the end effector 4, the robot 1, and the robot movable device will not interfere with the non-processing execution object OBJ_ntgt in reality. For example, compared to when the first collision detection process is performed based on a single piece of image data IMG, by performing the second collision detection process based on a plurality of image data IMG, the signal generation unit 312 is more likely to accurately determine that at least one of the end effector 4, the robot 1, and the robot movable device will not interfere with the non-processing execution object OBJ_ntgt in a situation where at least one of the end effector 4, the robot 1, and the robot movable device will not interfere with the non-processing execution object OBJ_ntgt in reality.

[0259] For this reason, the first collision detection process based on one image data IMG may be considered to be a collision detection process capable of detecting interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-processing execution object OBJ_ntgt with a first accuracy (in other words, with a first sensitivity). On the other hand, the first collision detection process based on multiple image data IMG may be considered to be a collision detection process capable of detecting interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-processing execution object OBJ_ntgt with a second accuracy higher than the first accuracy (in other words, with a second sensitivity higher than the first sensitivity). For this reason, in the following description, the first collision detection process based on one image data IMG will be referred to as a normal collision detection process, and the second collision detection process based on multiple image data IMG will be referred to as a high-sensitivity collision detection process, as necessary. For convenience of explanation, the following description will be given assuming that the first collision detection process is a normal collision detection process and the second collision detection process is a high-sensitivity collision detection process. However, the first collision detection processing does not have to be limited to the normal collision detection processing, and the second collision detection processing does not have to be limited to the high-sensitivity collision detection processing. The following description of the normal collision detection processing and the high-sensitivity collision detection processing can be reused as a description of the first collision detection processing and the second collision detection processing by replacing the terms "normal collision detection processing" and "high-sensitivity collision detection processing" with the terms "first collision detection processing" and "second collision detection processing," respectively.

[0260] In addition, when performing the second collision detection process based on a plurality of image data IMG (i.e., high-sensitivity collision detection process), the signal generation unit 312 may generate a plurality of non-processed object models OM_ntgt based on the plurality of image data IMG, merge (in other words, combine or synthesize) the generated plurality of non-processed object models OM_ntgt to generate a single non-processed object model OM_ntgt, and perform the collision detection process based on the single merged non-processed object model OM_ntgt. For example, if the non-processed object model OM_ntgt is a point cloud model, the signal generation unit 312 may synthesize the respective point clouds generated based on the plurality of image data IMG to generate a composite point cloud as a single non-processed object model OM_ntgt, and perform the collision detection process based on the generated single non-processed object model OM_ntgt. Even in this case, compared to a non-processing execution object model OM_ntgt generated based on a single image data IMG, a single non-processing execution object model OM_ntgt generated based on a plurality of image data IMG will have less missing information about the non-processing execution object OBJ_ntgt. Therefore, even in this case, compared to when a first collision detection process is performed based on a single image data IMG, the signal generation unit 312 can more accurately determine collision between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB (particularly the non-processing execution object OBJ_ntgt) by performing the second collision detection process based on the plurality of image data IMG.

[0261] When the imaging device 21 of the imaging system 2 images multiple target objects OBJ (especially target objects OBJ that will later become non-processing execution objects OBJ_ntgt) multiple times to generate multiple image data IMG to be used in the high-sensitivity interference detection process, the signal generation unit 312 may control the illumination device 23 so that the intensity of illumination light from the illumination device 23 changes each time the imaging system 2 images multiple target objects OBJ. For example, the signal generation unit 312 may control the illumination device 23 so that the imaging device 21 images multiple target objects OBJ illuminated with illumination light of a first intensity at least once, and so that the imaging device 21 images multiple target objects OBJ illuminated with illumination light of a second intensity different from the first intensity at least once. As a result, the signal generating unit 312 may acquire image data IMG generated by the imaging device 21 capturing at least one image of multiple target objects OBJ illuminated with illumination light of a first intensity, and image data IMG generated by the imaging device 21 capturing at least one image of multiple target objects OBJ illuminated with illumination light of a second intensity. In this case, compared to when the intensity of the illumination light is fixed (i.e., not changed), missing information regarding the non-processing object OBJ_ntgt can be more appropriately complemented because multiple target objects OBJ are captured under multiple different imaging environments.

[0262] In this case, the high-sensitivity collision detection process and the normal collision detection process may be considered to be two types of collision detection process in which the intensities of illumination light that illuminates the multiple target objects OBJ to generate the image data IMG used in the collision detection process are different from each other. In other words, the high-sensitivity collision detection process and the normal collision detection process may be considered to be two types of collision detection process in which the "intensity of illumination light (condition related to the intensity of illumination light)," which is an example of a set condition, is different from each other. Specifically, the high-sensitivity collision detection process may be a collision detection process in which the condition related to the intensity of illumination light is set to the condition that "the intensity of illumination light is changed (e.g., the intensity of illumination light is changed each time the imaging device 21 captures images of the multiple target objects OBJ)," and the normal collision detection process may be a collision detection process in which the condition related to the intensity of illumination light is set to the condition that "the intensity of illumination light does not need to be changed."

[0263] The condition related to the intensity of the illumination light may include a maximum value of the intensity of the illumination light (i.e., maximum intensity). The condition related to the intensity of the illumination light may include a minimum value of the intensity of the illumination light (i.e., minimum intensity). The condition related to the intensity of the illumination light may include an average value of the intensity of the illumination light (i.e., average intensity). The condition related to the intensity of the illumination light may include an intensity distribution of the illumination light. Note that the intensity of the illumination light may mean at least one of the maximum value of the intensity of the illumination light (i.e., maximum intensity), the minimum value of the intensity of the illumination light (i.e., minimum intensity), and the average value of the intensity of the illumination light (i.e., average intensity). Note that the intensity of the illumination light may include at least one of light amount, luminous flux, illuminance, luminous intensity, and brightness.

[0264] Note that the conditions related to the intensity of the illumination light may be considered to be an example of the illumination conditions related to the illumination device 23 (illumination conditions related to the illumination of the illumination device 23). In this case, the high-sensitivity interference detection process and the normal interference detection process may be considered to be two types of interference detection process that differ from each other in "illumination conditions," which are an example of set conditions. For example, the high-sensitivity interference detection process and the normal interference detection process may be two types of interference detection process that satisfy the condition that the illumination conditions used in at least one of the multiple imaging operations performed in the high-sensitivity interference detection process are different from the illumination conditions used in the imaging operation performed in the normal interference detection process. In this case, the illumination conditions used in at least one other of the multiple imaging operations performed in the high-sensitivity interference detection process may be the same as the illumination conditions used in the imaging operation performed in the normal interference detection process. For example, the high-sensitivity interference detection process may be an interference detection process in which the illumination conditions are changed (e.g., the illumination conditions are changed each time the imaging device 21 images multiple target objects OBJ a predetermined number of times (e.g., once or twice or more)). The illumination conditions may include, in addition to or instead of the condition related to the intensity of the illumination light, at least one of a condition related to the wavelength of the illumination light, a condition related to the degree of polarization of the illumination light, a condition related to the amount of the illumination light, and a condition related to the amount of irradiation of the illumination light. When pulsed light is used as the illumination light, the illumination conditions may include at least one of a condition related to the emission cycle of the illumination light, a condition related to the number of times the illumination light is emitted (e.g., the emission time per unit time), and a condition related to the pulse width of the illumination light.

[0265] When the intensity of the illumination light is changed, the signal generation unit 312 may change the gain of the image capture device 21 accordingly. For example, the signal generation unit 312 may lower the gain of the image capture device 21 as the intensity of the illumination light increases. For example, the signal generation unit 312 may increase the gain of the image capture device 21 as the intensity of the illumination light decreases. As a result, even when the intensity of the illumination light is changed, the image capture device 21 can appropriately capture images of multiple target objects OBJ. Alternatively, even when the intensity of the illumination light is not changed, the signal generation unit 312 may change the gain of the image capture device 21.

[0266] In this case, the high-sensitivity interference detection process and the normal interference detection process may be considered to be two types of interference detection process in which the gain of the imaging device 21 (i.e., the gain of the imaging system 2) used to generate the image data IMG used in the interference detection process is different from each other. In other words, the high-sensitivity interference detection process and the normal interference detection process may be considered to be two types of interference detection process in which the “gain of the imaging device 21 (condition related to the gain of the imaging device 21),” which is an example of a setting condition, is different from each other. For example, the high-sensitivity interference detection process and the normal interference detection process may be two types of interference detection process that satisfy the condition that the gain used in at least one of the multiple imaging operations performed in the high-sensitivity interference detection process is different from the gain used in the imaging operation performed in the normal interference detection process. In this case, the gain used in at least one other of the multiple imaging operations performed in the high-sensitivity interference detection process may be the same as the gain used in the imaging operation performed in the normal interference detection process. For example, the high-sensitivity interference detection process may be an interference detection process in which the gain is changed (e.g., the gain is changed each time the imaging device 21 images multiple target objects OBJ a predetermined number of times (e.g., once or twice or more)).

[0267] When the imaging device 21 of the imaging system 2 images multiple target objects OBJ (especially target objects OBJ that will later become non-processing execution objects OBJ_ntgt) multiple times to generate multiple image data IMG to be used in the high-sensitivity interference detection process, the signal generation unit 312 may control the imaging device 21 so that the exposure amount of the imaging device 21 changes each time the imaging device 21 images the multiple target objects OBJ, in addition to or instead of controlling the illumination device 23 so that the intensity of the illumination light from the illumination device 23 changes each time the imaging device 21 images the multiple target objects OBJ. For example, the signal generation unit 312 may control the illumination device 23 so that the imaging device 21 whose exposure amount is set to a first exposure amount images the multiple target objects OBJ at least once, and so that the imaging device 21 whose exposure amount is set to a second exposure amount different from the first exposure amount images the multiple target objects OBJ at least once. As a result, the signal generating unit 312 may acquire image data IMG generated by capturing images of multiple target objects OBJ at least once using the image capturing device 21 whose exposure amount is set to the first exposure amount, and image data IMG generated by capturing images of multiple target objects OBJ at least once using the image capturing device 21 whose exposure amount is set to the first exposure amount. In this case, compared to when the exposure amount of the image capturing device 21 is fixed (i.e., not changed), missing information regarding the non-processing execution object OBJ_ntgt can be more appropriately complemented. This is because multiple target objects OBJ are captured under multiple different imaging environments.

[0268] In this case, the high-sensitivity interference detection process and the normal interference detection process may be considered to be two types of interference detection process in which the exposure amount of the imaging device 21 (i.e., the exposure amount of the imaging system 2) used to generate the image data IMG used in the interference detection process is different from each other. In other words, the high-sensitivity interference detection process and the normal interference detection process may be considered to be two types of interference detection process in which the "exposure amount of the imaging device 21 (conditions related to the exposure amount of the imaging device 21)," which is an example of a setting condition, is different from each other. For example, the high-sensitivity interference detection process and the normal interference detection process may be two types of interference detection process that satisfy the condition that the exposure amount used in at least one of the multiple imaging operations performed in the high-sensitivity interference detection process is different from the exposure amount used in the imaging operation performed in the normal interference detection process. In this case, the exposure amount used in at least one other of the multiple imaging operations performed in the high-sensitivity interference detection process may be the same as the exposure amount used in the imaging operation performed in the normal interference detection process. For example, the high-sensitivity interference detection process may be an interference detection process in which the exposure amount is changed (for example, the exposure amount is changed each time the imaging device 21 images multiple target objects OBJ a predetermined number of times (for example, once or twice or more)).

[0269] When the imaging device 21 of the imaging system 2 images multiple target objects OBJ (especially target objects OBJ that will later become non-processing execution objects OBJ_ntgt) multiple times to generate multiple image data IMG to be used in the high-sensitivity interference detection process, the signal generation unit 312 may control the imaging device 21 so that the exposure time of the imaging device 21 changes each time the imaging device 21 images multiple target objects OBJ, in addition to or instead of controlling the illumination device 23 so that the intensity of illumination light from the illumination device 23 changes each time the imaging device 21 images multiple target objects OBJ. For example, the signal generation unit 312 may control the illumination device 23 so that the imaging device 21 whose exposure time is set to a first exposure time images multiple target objects OBJ at least once, and so that the imaging device 21 whose exposure time is set to a second exposure time different from the first exposure time images multiple target objects OBJ at least once. As a result, the signal generating unit 312 may acquire image data IMG generated by capturing images of multiple target objects OBJ at least once by the image capturing device 21 whose exposure time is set to the first exposure time, and image data IMG generated by capturing images of multiple target objects OBJ at least once by the image capturing device 21 whose exposure time is set to the first exposure time. In this case, compared to when the exposure time of the image capturing device 21 is fixed (i.e., does not change), missing information regarding the non-processing execution object OBJ_ntgt can be more appropriately complemented. This is because multiple target objects OBJ are captured under multiple different imaging environments.

[0270] In this case, the high-sensitivity collision detection process and the normal collision detection process may be considered to be two types of collision detection process in which the exposure time of the imaging device 21 (i.e., the exposure time of the imaging system 2) for generating the image data IMG used in the collision detection process is different from each other. In other words, the high-sensitivity collision detection process and the normal collision detection process may be considered to be two types of collision detection process in which the “exposure time of the imaging device 21 (conditions related to the exposure time of the imaging device 21),” which is an example of a setting condition, is different from each other. For example, the high-sensitivity collision detection process and the normal collision detection process may be two types of collision detection process that satisfy the condition that the exposure time used in at least one of the multiple imaging operations performed in the high-sensitivity collision detection process is different from the exposure time used in the imaging operation performed in the normal collision detection process. In this case, the exposure time used in at least one other of the multiple imaging operations performed in the high-sensitivity collision detection process may be the same as the exposure time used in the imaging operation performed in the normal collision detection process. For example, the high-sensitivity collision detection process may be a collision detection process in which the exposure time is changed (e.g., the exposure time is changed each time the imaging device 21 images multiple target objects OBJ a predetermined number of times (e.g., once or twice or more)).

[0271] The number of times the imaging system 2 captures images, the gain of the imaging system 2, the exposure amount of the imaging system 2, and the exposure time of the imaging system 2 may all be considered to be conditions related to imaging by the imaging system 2 (imaging conditions). In other words, the set conditions may include imaging conditions. Furthermore, because the imaging system 2 captures images of the non-processing execution object OBJ_ntgt illuminated by illumination light, the above-mentioned illumination conditions (e.g., conditions related to the intensity of the illumination light) may also be considered to be examples of imaging conditions. In this case, the high-sensitivity collision detection process and the normal collision detection process may be considered to be two types of collision detection process in which the imaging conditions of the imaging system 2 for generating the image data IMG used in the collision detection process are different from each other. For example, the high-sensitivity collision detection process and the normal collision detection process may be two types of collision detection process in which the imaging conditions used in at least one of the multiple imaging operations performed in the high-sensitivity collision detection process are different from the imaging conditions used in the imaging performed in the normal collision detection process. In this case, the imaging conditions used in at least one of the multiple imaging operations performed in the high-sensitivity interference detection processing may be the same as the imaging conditions used in the imaging performed in the normal interference detection processing. For example, the high-sensitivity interference detection processing may be an interference detection processing in which the imaging conditions are changed (for example, the imaging conditions are changed each time the imaging device 21 images multiple target objects OBJ a predetermined number of times (for example, once or twice or more)).

[0272] As described above, when generating the non-processing object model OM_ntgt based on the image data IMG, the signal generation unit 312 may perform an invalidation process to invalidate noise-related data in the three-dimensional position data of the non-processing object OBJ_ntgt. In this case, the high-sensitivity interference detection process and the normal interference detection process may be two types of interference detection process with different invalidation conditions (invalidation conditions). That is, the high-sensitivity interference detection process and the normal interference detection process may be two types of interference detection process with different "invalidation conditions," which are examples of set conditions. As an example, the high-sensitivity interference detection process may be an interference detection process in which an invalidation process is performed to prioritize accurate detection of interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB. On the other hand, the normal interference detection process may be an interference detection process in which an invalidation process is not performed to prioritize shortening the time required to perform the interference detection process. As another example, the high-sensitivity interference detection process may be an interference detection process in which the invalidation level is higher than that of the normal interference detection process (i.e., the amount of data invalidated is greater than that of the normal interference detection process) in order to prioritize accurate detection of interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB. In other words, the normal interference detection process may be an interference detection process in which the invalidation level is lower than that of the high-sensitivity interference detection process (i.e., the amount of data invalidated is less than that of the normal interference detection process). The amount of data invalidated increases as the noise threshold described above decreases. Therefore, the high-sensitivity interference detection process may be an interference detection process in which invalidation is performed using a noise threshold lower than that of the normal interference detection process in order to prioritize accurate detection of interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB. In other words, the normal interference detection process may be an interference detection process in which invalidation is performed using a noise threshold higher than that of the high-sensitivity interference detection process.

[0273] The invalidation process may include, in addition to or instead of invalidating noise-related data from the three-dimensional position data of the non-processed execution object OBJ_ntgt, invalidating a portion of the three-dimensional position data of the non-processed execution object OBJ_ntgt that is not used to generate the non-processed execution object model OM_ntgt. In other words, the invalidation process may include invalidating a portion of the three-dimensional position data of the non-processed execution object OBJ_ntgt that is not noise-related data but is not used to generate the non-processed execution object model OM_ntgt.

[0274] The invalidation process may be considered to be post-processing performed on the image data IMG after the imaging system 2 generates the image data IMG. In this case, the invalidation conditions for the invalidation process may be considered to be an example of conditions (post-processing conditions for the image data IMG, post-processing conditions for the imaging results of the imaging system 2) related to post-processing of the image data IMG (i.e., post-processing of the imaging results of the imaging system 2). In other words, the set conditions may include post-processing conditions. The post-processing conditions may include conditions related to post-processing different from the invalidation process. In this case, the high-sensitivity collision detection process and the normal collision detection process may be two types of collision detection process that have different conditions (post-processing conditions) related to post-processing different from the invalidation process. In other words, the high-sensitivity collision detection process and the normal collision detection process may be two types of collision detection process that have different "post-processing conditions," which are an example of set conditions. For example, the high-sensitivity collision detection process and the normal collision detection process may be two types of collision detection process that satisfy the condition that the post-processing conditions for the results of at least one of the multiple imaging operations performed in the high-sensitivity collision detection process are different from the post-processing conditions for the results of imaging performed in the normal collision detection process. In this case, the post-processing conditions for the results of at least one of the multiple image captures performed in the high-sensitivity interference detection processing may be the same as the post-processing conditions for the results of image capture performed in the normal interference detection processing. For example, the high-sensitivity interference detection processing may be an interference detection processing in which the post-processing conditions are changed (for example, the post-processing conditions are changed each time the image capture device 21 captures images of multiple target objects OBJ a predetermined number of times (for example, once or twice or more)).

[0275] Furthermore, in the above description, the lighting conditions related to the lighting of the lighting device 23, the imaging conditions related to the imaging of the imaging system 2, and the invalidation conditions related to the invalidation process have been exemplified as setting conditions for the high-sensitivity interference detection processing and the normal interference detection processing. Here, the lighting of the lighting device 23, the imaging conditions related to the imaging of the imaging system 2, and the invalidation process may all be considered to be performed as pre-processing steps of the interference detection processing. In this case, the lighting conditions related to the lighting of the lighting device 23, the imaging conditions related to the imaging of the imaging system 2, and the invalidation conditions related to the invalidation process may each be considered to be examples of conditions related to pre-processing steps (pre-processing conditions) of the interference detection processing. In other words, the setting conditions may include pre-processing conditions. In this case, the high-sensitivity interference detection processing and the normal interference detection processing may be considered to be two types of interference detection processing with different pre-processing conditions. For example, the high-sensitivity interference detection processing and the normal interference detection processing may be two types of interference detection processing that satisfy the condition that the pre-processing conditions of at least one of the multiple interference detection processing steps performed using multiple image data IMG in the high-sensitivity interference detection processing are different from the pre-processing conditions of the normal interference detection processing. In this case, the pre-processing conditions of at least one other of the plurality of collision detection processes performed using each of the plurality of image data IMG in the high-sensitivity collision detection processing may be the same as the pre-processing conditions of the normal collision detection processing. For example, the high-sensitivity collision detection processing may be a collision detection processing in which the pre-processing conditions are changed (for example, the pre-processing conditions are changed each time a predetermined number (for example, one or more) of collision detection processes performed using each of the plurality of image data IMG in the high-sensitivity collision detection processing are performed).

[0276] The high-sensitivity interference detection processing and the normal interference detection processing may be considered to be two types of interference detection processing that have different conditions (threshold conditions) related to the interference detection threshold used to determine whether or not at least one of the end effector 4, the robot 1, and the robot movable device interferes with the non-held object NOB. For example, the high-sensitivity interference detection processing and the normal interference detection processing may be two types of interference detection processing that satisfy the condition that the threshold condition (e.g., the interference detection threshold) used in at least one of the multiple interference detection processings performed using multiple image data IMG in the high-sensitivity interference detection processing is different from the threshold condition (e.g., the interference detection threshold) used in the normal interference detection processing. In this case, the threshold condition (e.g., the interference detection threshold) used in at least another of the multiple interference detection processings performed using multiple image data IMG in the high-sensitivity interference detection processing may be the same as the threshold condition (e.g., the interference detection threshold) used in the normal interference detection processing. For example, the high-sensitivity interference detection process may be an interference detection process in which the threshold conditions are changed (for example, the threshold conditions are changed each time a predetermined number (for example, one or more) of multiple interference detection processes performed using multiple image data IMG in the high-sensitivity interference detection process are performed).

[0277] When the normal interference detection process and the high-sensitivity interference detection process are performed, the signal generation unit 312 may select a holding target position based on the results of the normal interference detection process and the high-sensitivity interference detection process in step S234 of Fig. 10. Specifically, the signal generation unit 312 may not select, as a holding target position, a holding candidate position HCP determined to be incapable of being selected as a holding target position by at least one of the normal interference detection process and the high-sensitivity interference detection process. On the other hand, the signal generation unit 312 may select, as a holding target position, a holding candidate position HCP determined to be selectable as a holding target position by both the normal interference detection process and the high-sensitivity interference detection process. In this case, compared to when one holding candidate position HCP determined to be selectable as the holding target position by only one of the normal interference detection process and the high-sensitivity interference detection process is selected as the holding target position, the signal generation unit 312 can select, as the holding target position, a holding candidate position HCP that can further reduce the possibility of actual interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB. As a result, the signal generation unit 312 can further reduce the possibility of interference between at least one of the end effector 4, the robot 1, and the robot movable device and the non-held object NOB.

[0278] The signal generation unit 312 may first perform normal interference detection processing in step S232 of FIG. 10 . If the normal interference detection processing determines that at least one of the end effector 4, the robot 1, and the robot movable device will interfere with the non-held object NOB (i.e., the holding candidate position HCP cannot be used as the holding target position), the signal generation unit 312 may terminate the interference detection processing of step S232 without performing high-sensitivity interference detection processing. On the other hand, if the normal interference detection processing determines that at least one of the end effector 4, the robot 1, and the robot movable device will not interfere with the non-held object NOB (i.e., the holding candidate position HCP can be used as the holding target position), the signal generation unit 312 may further perform high-sensitivity interference detection processing. In other words, the signal generation unit 312 may determine whether to perform high-sensitivity interference detection processing based on the determination result of the normal interference detection processing. As a result, the time required to perform the collision detection processing in step S232 of FIG. 10 can be shortened compared to when high-sensitivity collision detection processing is always performed regardless of the determination results of the normal collision detection processing. In particular, because the high-sensitivity collision detection processing includes multiple collision detection processing steps performed based on multiple image data IMG, the time required to perform the high-sensitivity collision detection processing is longer than the time required to perform the normal collision detection processing, which includes a single collision detection processing step performed based on a single image data IMG. In this case, when it is determined whether to perform the high-sensitivity collision detection processing based on the determination results of the normal collision detection processing, the number of times the high-sensitivity collision detection processing is performed can be reduced, thereby significantly reducing the time required to perform the collision detection processing in step S232 of FIG. 10.

[0279] In this case, the setting conditions for the normal collision detection processing may include a condition regarding whether or not to execute high-sensitivity collision detection processing, which is performed subsequent to the normal collision detection processing. Specifically, the setting conditions for the normal collision detection processing may include a condition that determines whether or not to execute high-sensitivity collision detection processing based on the detection results of the normal collision detection processing. On the other hand, the setting conditions for the high-sensitivity collision detection processing do not need to include a condition regarding whether or not to execute normal collision detection processing, which is performed prior to the high-sensitivity collision detection processing.

[0280] At least one of the setting conditions for the normal collision detection processing and the setting conditions for the high-sensitivity collision detection processing may be set manually by the user. That is, the control device 3 may set at least one of the setting conditions for the normal collision detection processing and the setting conditions for the high-sensitivity collision detection processing based on a user instruction. At least one of the setting conditions for the normal collision detection processing and the setting conditions for the high-sensitivity collision detection processing may be automatically set by the control device 3. That is, the control device 3 may automatically set at least one of the setting conditions for the normal collision detection processing and the setting conditions for the high-sensitivity collision detection processing without using a user instruction.

[0281] The process of setting at least one of the setting conditions for the normal collision detection process and the setting conditions for the high-sensitivity collision detection process may be included in the pre-setting process (step...

Claims

1. A control device provided with a holding device for holding a target object and generating a control signal for controlling a robot that moves the holding device, comprising: an arithmetic unit that generates the control signal; and a communication unit that outputs the control signal generated by the arithmetic unit, wherein the arithmetic unit performs a determination process regarding a holding position of the holding device relative to the target object, and generates the control signal for controlling the robot so that the holding device approaches the holding position determined based on the determination result of the determination process, wherein the determination process includes a first determination process that determines whether a first candidate position set as a candidate for the holding position is to be the holding position, and a second determination process that determines whether a second candidate position set as a candidate for the holding position, different from the first candidate position, is to be the holding position, and the first setting condition that is set for the first determination process is different from a second setting condition that is set for the second determination process.

2. The control device according to claim 1, wherein the arithmetic device determines the target object to be held by the holding device based on the imaging results of the imaging system, and performs the determination process regarding the holding position for the determined target object.

3. The control device according to claim 2, wherein the imaging result is a first imaging result, and the arithmetic device performs the determination process based on a second imaging result obtained by the imaging system.

4. The control device according to claim 3, wherein the determination process is performed based on the first imaging result as the second imaging result.

5. A control device according to any one of claims 1 to 4, wherein the arithmetic device determines the first candidate position or the second candidate position as the holding position based on the judgment result of the first judgment process and the judgment result of the second judgment process.

6. The control device according to claim 5, wherein the arithmetic device executes the second determination process when it determines that the first candidate position is not to be the holding position as a result of the first determination process.

7. A control device as described in claim 5 or 6, wherein the first determination process and the second determination process are each set with a priority for holding by the holding device, and the priority of the first determination process is higher than the priority of the second determination process.

8. The control device according to claim 7, wherein the priority of the first determination process and the priority of the second determination process are changeable.

9. The control device according to any one of claims 1 to 8, wherein the judgment process includes an interference judgment process that judges interference between a non-held object different from the target object to be held by the holding device and at least one of the holding device and the robot, the first judgment process includes, as the interference judgment process, a first interference judgment process that judges interference between the non-held object and at least one of the holding device and the robot when it is assumed that the robot operates so that the holding device approaches the first candidate position, the second judgment process includes, as the interference judgment process, a second interference judgment process that judges interference between the non-held object and at least one of the holding device and the robot when it is assumed that the robot operates so that the holding device approaches the second candidate position, a third setting condition that is a setting condition set for the first interference judgment process is different from a fourth setting condition that is a setting condition set for the second interference judgment process, and the calculation device determines the first candidate position or the second candidate position as the holding position based on the judgment result of the first interference judgment process and the judgment result of the second interference judgment process.

10. The control device according to claim 9, wherein the third set condition includes at least one of the first set conditions, and the fourth set condition includes at least one of the second set conditions.

11. A control device as described in claim 9 or 10, wherein each of the third setting condition and the fourth setting condition includes a plurality of conditions, and at least one condition among the third setting conditions is different from the at least one condition among the fourth setting conditions.

12. A control device according to any one of claims 9 to 11, wherein at least one of the third set condition and the fourth set condition includes an interference determination condition related to the interference determination process.

13. The control device according to claim 12, wherein the interference determination conditions include spatial conditions relating to the space in which the interference determination processing is performed.

14. The control device according to claim 13, wherein the spatial conditions include at least one of a condition regarding a position in the space, a condition regarding an attitude in the space, and a condition regarding a size in the space.

15. A control device according to any one of claims 12 to 14, wherein the interference determination conditions include a threshold condition relating to a threshold for determining whether or not the non-held object will interfere with at least one of the holding device and the robot.

16. The control device described in claim 15, wherein the threshold conditions include at least one of a condition regarding a threshold for the overlap range between the non-held object and at least one of the holding device and the robot, when it is assumed that the robot operates so that the holding device approaches the candidate position, and a condition regarding a threshold for the distance between the non-held object and at least one of the holding device and the robot, when it is assumed that the robot operates so that the holding device approaches the candidate position.

17. The control device according to claim 16, wherein the interference detection process determines whether or not a non-held object model based on three-dimensional position data of multiple points on the non-held object interferes with at least one of a holding device model that is a model of the holding device and a robot model that is a model of the robot, thereby determining interference between the non-held object and at least one of the holding device model and the robot, the condition regarding the threshold of the overlap range includes a condition regarding the range in which the non-held object model overlaps with at least one of the holding device model and the robot model, and the condition regarding the distance threshold includes a condition regarding the distance between the non-held object model and at least one of the holding device model and the robot model.

18. A control device according to any one of claims 9 to 17, wherein at least one of the third setting condition and the fourth setting condition includes a pre-processing condition relating to a pre-processing of the interference detection processing.

19. The control device described in claim 18, wherein the interference detection process determines interference between the non-held object and at least one of the holding device and the robot based on an imaging result of at least the non-held object being imaged by an imaging system, and the pre-processing conditions include at least one of imaging conditions related to imaging by the imaging system and post-processing conditions related to post-processing of the imaging results.

20. The control device according to claim 19, wherein the imaging conditions include at least one of a condition regarding the exposure amount of the imaging system, a condition regarding the exposure time of the imaging system, a condition regarding the gain of the imaging system, and a number of imaging times of the imaging system.

21. A control device as described in claim 19 or 20, wherein the imaging system is equipped with a lighting device, the interference detection process determines interference between the non-held object and at least one of the holding device and the robot based on the imaging results obtained by imaging at least the non-held object illuminated with illumination light from the lighting device using the imaging system, and the imaging conditions include lighting conditions related to the illumination of the lighting device when imaging using the imaging system.

22. The control device according to claim 21, wherein the illumination conditions include a condition related to the intensity of the illumination light.

23. A control device as described in any one of claims 19 to 22, wherein the interference detection process includes: an invalidation process that invalidates some of the three-dimensional position data indicating the three-dimensional positions of multiple points on the non-held object, which is generated based on the imaging results; and a process that determines whether or not the non-held object will interfere with at least one of the holding device and the robot, based on the three-dimensional position data after the invalidation process; and the post-processing conditions include an invalidation condition related to the invalidation process.

24. The control device according to claim 23, wherein the invalidation conditions include a condition relating to a threshold value for invalidating a portion of the three-dimensional position data.

25. The control device described in any one of claims 9 to 24, wherein the first interference detection process includes: a first process for determining interference between the non-held object and at least one of the holding device and the robot, assuming that the robot operates so that the holding device approaches the first candidate position; and a second process for determining interference between the non-held object and at least one of the holding device and the robot, assuming that the robot operates so that the holding device approaches the first candidate position; and a fifth setting condition, which is a setting condition set for the first process, is different from a sixth setting condition, which is a setting condition set for the second process.

26. The control device according to claim 25, wherein the fifth set condition and the sixth set condition each include a plurality of conditions, and at least one condition of the fifth set condition is different from the at least one condition of the sixth set condition.

27. A control device according to claim 25 or 26, wherein at least one of the fifth set condition and the sixth set condition includes an interference determination condition related to the interference determination process.

28. A control device according to claim 27, wherein the fifth set condition includes the interference determination condition, and the interference determination condition included in the fifth set condition includes a condition regarding whether or not the second process is executed.

29. A control device according to any one of claims 25 to 28, wherein at least one of the fifth setting condition and the sixth setting condition includes a pre-processing condition relating to a pre-processing of the interference detection processing.

30. The control device described in claim 29, wherein the interference detection process determines interference between the non-held object and at least one of the holding device and the robot based on the imaging results of at least the non-held object imaged by an imaging system, and the pre-processing conditions include at least one of imaging conditions related to imaging by the imaging system and post-processing conditions related to post-processing of the imaging results.

31. The control device according to any one of claims 9 to 30, wherein the non-held objects include a plurality of objects different from the target object to be held by the holding device.

32. The control device described in claim 31, wherein the target object to be held by the holding device is one of a plurality of target objects placed on a mounting member, and the non-held object includes a plurality of target objects that are at least a portion of the plurality of target objects placed on the mounting member excluding the target object to be held by the holding device.

33. The control device according to claim 32, wherein the non-held object includes the placement member.

34. A control device according to any one of claims 9 to 33, wherein the target object to be held by the holding device is one of a plurality of target objects placed on a mounting member, the target object to be held by the holding device is a first holding execution object, and the arithmetic device, after the first holding execution object has been held by the holding device based on the control signal, determines the first candidate position or the second candidate position as the holding position based on a determination result of at least one of the first interference determination process based on the third set condition and the second interference determination process based on the fourth set condition for a second holding execution object which is the target object to be held by the holding device among the plurality of target objects placed on the mounting member and is different from the first holding execution object, and generates the control signal for controlling the robot so that the holding device approaches the determined first candidate position or the second candidate position.

35. A control device according to any one of claims 1 to 8, wherein the judgment process is a judgment process regarding the holding position for the target object to be held by the holding device, the target object to be held by the holding device is one of a plurality of target objects placed on a mounting member, the target object to be held by the holding device is a first holding execution object, and after the holding device has held the first holding execution object based on the control signal, the arithmetic device determines the first candidate position or the second candidate position as the holding position based on a judgment result of at least one of the first judgment process based on the first set condition and the second judgment process based on the second set condition for a second holding execution object which is the target object to be held by the holding device among the plurality of target objects placed on the mounting member and is different from the first holding execution object, and generates the control signal for controlling the robot so that the holding device approaches the determined first candidate position or the second candidate position.

36. A control device according to any one of claims 32 to 35, wherein each of the plurality of target objects placed on the placement member is placed irregularly on the placement member.

37. A control device according to any one of claims 1 to 36, wherein each of the first candidate position and the second candidate position is set with respect to a target object model that is a model of the target object.

38. A control device according to any one of claims 1 to 37, wherein the setting conditions are set based on instructions from a user.

39. A control device according to any one of claims 1 to 38, wherein the setting conditions are set by the arithmetic unit.

40. A control device as described in any one of claims 1 to 8, wherein each of the first set conditions and the second set conditions includes a plurality of conditions, and at least one condition among the first set conditions is different from the at least one condition among the second set conditions.

41. A control device according to any one of claims 1 to 7 and 39, wherein at least one of the first set condition and the second set condition includes a judgment condition relating to the judgment process.

42. A control device according to any one of claims 1 to 7, 40 and 41, wherein at least one of the first set condition and the second set condition includes a pre-process condition relating to a pre-process of the determination process.

43. The control device described in claim 42, wherein the judgment process is a judgment process regarding the holding position for the target object to be held by the holding device, based on an imaging result obtained by imaging a non-held object different from the target object to be held by the holding device with an imaging system, and the pre-processing conditions include at least one of an imaging condition regarding imaging by the imaging system and a post-processing condition regarding post-processing of the imaging results.

44. A control device according to any one of claims 1 to 43, wherein the arithmetic unit generates a condition display signal for controlling the display device so that the first setting condition and the second setting condition are displayed on the display device, and the communication unit outputs the display control signal to the display device.

45. A control device according to claim 44, wherein the condition display signal includes a signal for controlling the display device so that the first set condition and the first candidate position are associated and displayed on the display device, and a signal for controlling the display device so that the second set condition and the second candidate position are associated and displayed on the display device.

46. ​​A control device according to any one of claims 8 to 34, wherein the arithmetic unit generates a condition display signal for controlling the display device so that the third setting condition and the fourth setting condition are displayed on the display device, and the communication unit outputs the display control signal to the display device.

47. A control device according to claim 46, wherein the condition display signal includes a signal for controlling the display device so that the third set condition and the first candidate position are associated and displayed on the display device, and a signal for controlling the display device so that the fourth set condition and the second candidate position are associated and displayed on the display device.

48. A control device according to any one of claims 25 to 30, wherein the arithmetic unit generates a condition display signal for controlling the display device so that the fifth setting condition and the sixth setting condition are displayed on the display device, and the communication unit outputs the display control signal to the display device.

49. The control device according to claim 48, wherein the condition display signal includes a signal for controlling the display device so that at least one of the fifth set condition and the sixth set condition is displayed on the display device in association with the first candidate position.

50. A control device according to any one of claims 1 to 43, wherein the arithmetic unit generates an assistance display signal for controlling the display device so that assistance information for assisting a user in setting the setting conditions is displayed on the display device.

51. The control device according to claim 50, wherein the assistance display signal includes a signal for controlling the display device so that the assistance information is displayed on the display device together with a target object model that is a model of the target object.

52. A control device according to claim 50 or 51, wherein the support information includes information relating to the accuracy of the determination process for each position on the target object.

53. The control device according to claim 52, wherein the support information includes information regarding the accuracy of the determination process for at least one of the first candidate position and the second candidate position.

54. A control device according to any one of claims 50 to 53, wherein the assistance information includes information relating to the likelihood of object overlap at each position on the target object.

55. The control device according to claim 54, wherein the assistance information includes information regarding the possibility of object overlap at least one of the first candidate position and the second candidate position.

56. The control device according to claim 54 or 55, wherein the target object is a first target object, and the objects include a second target object that is a target object different from the first target object.

57. A control device according to any one of claims 50 to 56, wherein the support information includes information relating to recommended conditions that the user is recommended to set as the setting conditions.

58. A control device according to any one of claims 50 to 57, wherein the computing device generates the assistance information based on information about the shape of the target object.

59. A control device according to any one of claims 50 to 58, wherein the arithmetic unit generates the assistance information based on image data generated by an imaging system capturing an image of at least the target object.

60. A control device according to any one of claims 50 to 59, wherein the calculation device generates the assistance information based on ambient environment information relating to the ambient environment of the target object.

61. The control device according to claim 60, wherein the surrounding environment information includes information about the brightness of the surrounding environment of the target object.

62. A control device according to any one of claims 50 to 61, wherein the calculation device generates the assistance information using a calculation model generated by machine learning so as to output the assistance information when information usable for generating the assistance information is input.

63. A control device according to any one of claims 2 to 4, 19 to 30, 43 and 59, wherein the imaging system is provided on the robot.

64. A control system comprising: a control device according to any one of claims 1 to 63; and an imaging system.

65. A robot system comprising: a control device according to any one of claims 1 to 63; an imaging system; and the robot.

66. A control method for generating a control signal for controlling a robot provided with a holding device for holding an object and for moving the holding device, the control method comprising: performing a determination process regarding a holding position of the holding device relative to the object; and generating the control signal for controlling the robot so that the holding device approaches the holding position determined based on the determination result of the determination process; the determination process comprising a first determination process for determining whether a first candidate position set as a candidate for the holding position is to be the holding position; and a second determination process for determining whether a second candidate position set as a candidate for the holding position, different from the first candidate position, is to be the holding position; and a first setting condition set for the first determination process is different from a second setting condition set for the second determination process.

67. A computer program that causes a computer to execute the control method according to claim 66.

68. A control device comprising: a computing device that generates a condition display signal for controlling a display device to display on the display device setting conditions set for an interference detection process that determines whether at least one of a holding device provided on a robot and the robot will interfere with an unheld object different from the target object being held by the holding device, the setting conditions being a third setting condition set for a first interference detection process that determines whether at least one of the holding device and the robot will interfere with the unheld object when it is assumed that the robot operates so that the holding device approaches a first candidate position that is set as a candidate for a holding position by the holding device with respect to the target object; and a fourth setting condition different from the third setting condition that is set for a second interference detection process that determines whether at least one of the holding device and the robot will interfere with the unheld object when it is assumed that the robot operates so that the holding device approaches a second candidate position that is set as a candidate for the holding position and is different from the first candidate position.

69. A control device according to claim 68, wherein the condition display signal includes a signal for controlling the display device so that the third set condition and the first candidate position are associated and displayed on the display device, and a signal for controlling the display device so that the fourth set condition and the second candidate position are associated and displayed on the display device.

70. A control device as described in claim 68 or 69, wherein each of the third set conditions and the fourth set conditions includes a plurality of conditions, and at least one condition among the third set conditions is different from the at least one condition among the fourth set conditions.

71. A control device according to any one of claims 68 to 70, wherein at least one of the third set condition and the fourth set condition includes an interference determination condition related to the interference determination process.

72. A control device according to claim 71, wherein the interference determination conditions include at least one of a spatial condition relating to the space in which the interference determination process is performed, and a threshold condition relating to a threshold for determining whether or not at least one of the holding device and the robot has interfered with the non-held object.

73. A control device according to any one of claims 68 to 72, wherein at least one of the third set condition and the fourth set condition includes a pre-process condition relating to a pre-process of the interference detection process.

74. The control device described in claim 73, wherein the interference detection process determines interference between at least one of the holding device and the robot and the non-held object based on the imaging results of at least the non-held object imaged by an imaging system, and the pre-processing conditions include at least one of imaging conditions related to imaging by the imaging system and post-processing conditions related to post-processing of the imaging results.

75. A control device as described in any one of claims 68 to 74, wherein the calculation device generates an assistance display signal for controlling the display device so that assistance information for assisting a user in setting at least one of the third setting condition and the fourth setting condition is displayed on the display device, and the communication device outputs the generated assistance display signal to the display device.

76. A control system comprising: a control device according to any one of claims 68 to 75; and an imaging system.

77. A robot system comprising: a control device according to any one of claims 68 to 75; an imaging system; and the robot.

78. A control method comprising: generating a condition display signal for controlling a display device to display on the display device a third set condition set for a first interference detection process that determines whether at least one of the holding device and the robot will interfere with the non-held object when it is assumed that the robot operates so that the holding device approaches a first candidate position that is set as a candidate for a holding position by the holding device with respect to the target object; and a fourth set condition set for a second interference detection process that determines whether at least one of the holding device and the robot will interfere with the non-held object when it is assumed that the robot operates so that the holding device approaches a second candidate position that is set as a candidate for the holding position and is different from the first candidate position, the fourth set condition being different from the third set condition; and outputting the generated condition display signal to the display device.

79. A computer program that causes a computer to execute the control method according to claim 78.

80. A control device comprising: a computing device that generates a condition display signal for controlling a display device to display setting conditions set for a judgment process regarding the holding position of a holding device provided on a robot for a target object to be processed by the holding device, the first setting condition being set for a first judgment process that judges whether a first candidate position set as a candidate for the holding position should be the holding position; and a second setting condition that is set for a second judgment process that judges whether a second candidate position set as a candidate for the holding position, different from the first candidate position, should be the holding position, and that is different from the first setting condition; and a communication device that outputs the condition display signal generated by the computing device to the display device.

81. A control device according to claim 80, wherein the condition display signal includes a signal for controlling the display device so that the first set condition and the first candidate position are associated and displayed on the display device, and a signal for controlling the display device so that the second set condition and the second candidate position are associated and displayed on the display device.

82. A control device as described in claim 80 or 81, wherein each of the first set conditions and the second set conditions includes a plurality of conditions, and at least one of the first set conditions is different from the at least one of the second set conditions.

83. A control device according to any one of claims 80 to 82, wherein at least one of the first set condition and the second set condition includes a judgment condition that is a condition for the judgment process.

84. A control device according to any one of claims 80 to 83, wherein at least one of the first set condition and the second set condition includes a pre-process condition relating to a pre-process of the determination process.

85. The control device described in claim 84, wherein the judgment process is a judgment process regarding the holding position for the target object to be held by the holding device based on an imaging result obtained by imaging a non-held object different from the target object to be held by the holding device with an imaging system, and the pre-processing conditions include at least one of imaging conditions regarding imaging by the imaging system and post-processing conditions regarding post-processing of the imaging results.

86. A control device as described in any one of claims 80 to 85, wherein the calculation device generates an assistance display signal for controlling the display device so that assistance information for assisting a user in setting at least one of the first setting condition and the second setting condition is displayed on the display device, and the communication device outputs the generated assistance display signal to the display device.

87. A control system comprising: a control device according to any one of claims 80 to 86; and an imaging system.

88. A robot system comprising: a control device according to any one of claims 80 to 86; an imaging system; and the robot.

89. A control method comprising: generating a condition display signal for controlling a display device to display setting conditions set for a judgment process regarding a holding position by a holding device provided on a robot for a target object to be processed by the holding device, the setting condition being a first setting condition set for a first judgment process that judges whether a first candidate position set as a candidate for the holding position should be the holding position; and a second setting condition set for a second judgment process that judges whether a second candidate position set as a candidate for the holding position, different from the first candidate position, should be the holding position, the second setting condition being different from the first setting condition; and outputting the generated condition display signal to the display device.

90. A computer program that causes a computer to execute the control method of claim 89.

Images