Control device, control system, robot system, control method, and computer program

Abstract

This control device: generates a first control signal for controlling at least one among a robot and a holding device provided to the robot so that a second target object is lowered into a container when the second target object is held by the holding device together with a first target object at the time that the first target object stored in the container is held by the holding device; and generates a second control signal for controlling at least one among the robot and the holding device so that a fourth target object is lowered into the container from a position lower than the position at which the lowering of the second target object into the container is started, when the fourth target object is held by the holding device together with a third target object at the time that the third target object is held by the holding device after the second target object has been lowered into the container on the basis of the first control signal.

Description

Control device, control system, robot system, control method, and computer program

[0001] The present invention relates to the technical fields of a control device, a control system, a robot system, a control method, and a computer program that can generate a control signal for controlling a robot, for example.

[0002] An example of a control device that controls a robot to hold an object is described in Patent Document 1. Such a control device is required to control the robot to hold the object appropriately.

[0003] US Patent Application Publication No. 2013 / 0230235

[0004] According to a first aspect, a control device is provided which is provided with a holding device capable of holding a target object and an imaging system which images the target object, and which generates a control signal for controlling at least one of a robot which moves the holding device and the imaging system and the holding device, and which includes a calculation device which generates the control signal and a communication device which outputs the control signal generated by the calculation device, and the calculation device is configured to, based on an imaging result obtained by the imaging system of a target object group including at least some of the target objects contained in a container, hold a first target object which is one of the target objects in the target object group in the holding device, when a second target object among the multiple target objects contained in the container is held by the holding device together with the first target object, at least a control device that generates as the control signal a first control signal for controlling at least one of the robot and the holding device so that the second target object falls into the container, and that generates as the control signal a second control signal for controlling at least one of the robot and the holding device so that, after the at least second target object falls into the container based on the first control signal, when a third target object among the multiple target objects contained in the container is held by the holding device and a fourth target object among the multiple target objects contained in the container is held by the holding device together with the third target object, at least the fourth target object falls into the container from a position lower than the position at which the at least second target object starts to fall into the container.

[0005] According to a second aspect, there is provided a control system including the control device provided by the first aspect and the imaging system.

[0006] According to a third aspect, there is provided a robot system including the control device provided by the first aspect, the imaging system, and the robot.

[0007] According to a fourth aspect, there is provided a control method for generating a control signal for controlling a robot that is provided with a holding device capable of holding a target object and an imaging system that images the target object, and that moves the holding device and the imaging system, and at least one of the holding devices, the control method comprising: a step of: based on an imaging result obtained by imaging a target object group including at least some of the target objects contained in a container using the imaging system, holding a first target object that is the target object of the target object group by the holding device; and a step of controlling the robot and the imaging system such that, when a second target object of the multiple target objects contained in the container is held by the holding device together with the first target object, at least the second target object falls into the container. A control method is provided that includes generating a first control signal as the control signal for controlling at least one of the holding devices, and generating a second control signal as the control signal for controlling at least one of the robot and the holding device so that, after the at least second target object falls into the container based on the first control signal, when a third target object among the plurality of target objects contained in the container is held by the holding device and a fourth target object among the plurality of target objects contained in the container is held by the holding device together with the third target object, at least the fourth target object falls into the container from a position lower than a position at which the at least second target object starts to fall into the container.

[0008] According to a fifth aspect, there is provided a computer program that causes a computer to execute the control method provided by the fourth aspect.

[0009] According to a sixth aspect, there is provided a control device which is provided with a holding device capable of holding a target object and an imaging system which images the target object, and which generates a control signal for controlling a robot which moves the holding device and the imaging system, and at least one of the holding devices, and which comprises an arithmetic device which generates the control signal, and a communication device which outputs the control signal generated by the arithmetic device, and the control device generates a first control signal as the control signal for controlling at least one of the robot and the holding device based on a holding state of a group of held target objects held in the holding device which is detected based on a second imaging result which images a group of held target objects, which are the multiple target objects held in the holding device, by the imaging system after the holding operation of the target objects held in the container is performed by the holding device.

[0010] According to a seventh aspect, there is provided a control system including the control device provided by the sixth aspect and the imaging system.

[0011] According to an eighth aspect, there is provided a robot system including the control device provided by the sixth aspect, the imaging system, and the robot.

[0012] According to a ninth aspect, there is provided a control method for generating a control signal for controlling at least one of a robot that moves the holding device and the imaging system, and the holding device, which is provided with a holding device capable of holding a target object and an imaging system that images the target object, and the holding device, the control method including generating a first control signal as the control signal for controlling at least one of the robot and the holding device based on a holding state of a group of held target objects held in the holding device, which is detected based on a second imaging result in which the imaging system images a group of held target objects, which are the multiple target objects held in the holding device, after the holding operation by the holding device has been performed on the target objects contained in the container, based on a first imaging result in which the imaging system images at least a portion of the multiple target objects contained in the container.

[0013] According to a tenth aspect, there is provided a computer program for causing a computer to execute the control method provided by the ninth aspect.

[0014] According to an eleventh aspect, there is provided a control device which is provided with a holding device capable of holding a target object and an imaging system which images the target object, and which generates a control signal for controlling a robot which moves the holding device and the imaging system, and at least one of the holding devices, and which comprises an arithmetic device which generates the control signal, and a communication device which outputs the control signal generated by the arithmetic device, wherein the arithmetic device performs a holding operation on the target objects contained in a container using the holding device based on a first imaging result in which at least some of the target objects contained in the container are imaged by the imaging system, and then determines the number of target objects held by the holding device based on a second imaging result in which the target objects held in the holding device are imaged, and generates the control signal for controlling at least one of the robot and the holding device so that the control mode of at least one of the robot and the holding device differs depending on the determined number.

[0015] According to a twelfth aspect, there is provided a control system including the control device provided by the eleventh aspect and the imaging system.

[0016] According to a thirteenth aspect, there is provided a robot system including the control device provided by the eleventh aspect, the imaging system, and the robot.

[0017] According to a fourteenth aspect, there is provided a control method for generating a control signal for controlling a robot that moves the holding device and the imaging system, and at least one of the holding devices, the control method including: performing a holding operation on the target objects contained in a container using the holding device based on a first imaging result in which at least some of the target objects contained in the container are imaged using the imaging system; determining the number of target objects held by the holding device based on a second imaging result in which the target objects held in the holding device are imaged; and generating the control signal for controlling at least one of the robot and the holding device so that the control mode of at least one of the robot and the holding device differs depending on the determined number.

[0018] According to a fifteenth aspect, there is provided a computer program for causing a computer to execute the control method provided by the fourteenth aspect.

[0019] The functions and other advantages of the present invention will become apparent from the following detailed description of the preferred embodiments.

[0020] FIG. 1 is a block diagram showing the configuration of a robot system according to this embodiment. FIG. 2 is a side view showing the appearance of a robot according to this embodiment. FIG. 3 is a block diagram showing the configuration of a control device according to this embodiment. FIG. 3 is a block diagram showing the configuration of a robot control device according to this embodiment. FIG. 5 is a flowchart showing the flow of robot control processing. FIGS. 6(a) to 6(d) are cross-sectional views showing an end effector holding a non-processing object as well as a processing object. FIGS. 7(a) to 7(c) are cross-sectional views showing the positional relationship between multiple target objects contained in a container and an end effector holding a group of holding target objects. FIG. 8 is a cross-sectional view showing a processing object and a non-processing object dropping from the end effector. FIG. 9 is a cross-sectional view showing a non-processing object dropping from the end effector. FIG. 10 is a cross-sectional view showing a non-processing object dropping from the end effector. FIG. 11 is a cross-sectional view showing a rotationally moving end effector. FIG. 12 is a cross-sectional view showing a rotationally moving end effector. FIG. 13 is a cross-sectional view showing a processing object and a non-processing object dropping from the end effector. FIG. 14 is a cross-sectional view showing the height of a group of target objects contained in a container. FIGS. 15(a) to 15(c) are cross-sectional views showing the drop height. FIGS. 16(a) to 16(b) are cross-sectional views showing an end effector holding a non-processing object. FIG. 17 is a cross-sectional view showing a non-processing object dropping from the end effector. FIG. 18 is a cross-sectional view showing an imaging system that images a target object. FIG. 19 is a cross-sectional view showing a weight sensor that measures the weight of a target object. FIG. 20 is a cross-sectional view showing a container in which an object detection sensor that detects a target object is disposed. FIG. 21 is a cross-sectional view showing a container in which a marker is disposed. FIG. 22 is a flowchart showing the flow of robot control processing in the second modified example. FIG. 23 is a cross-sectional view showing an example of a state in which a desired portion of a non-processing object faces a desired direction. FIG. 24 is a cross-sectional view conceptually showing an imaging device that moves before imaging a non-processing object. FIG. 25 is a block diagram showing the configuration of a robot system including a measurement system.

[0021] Next, embodiments of a control device, a control system, a robot system, a control method, and a computer program will be described with reference to the drawings. Hereinafter, the embodiments of a control device, a control system, a robot system, a control method, and a computer program will be described using a robot system SYS.

[0022] (1) Configuration of the Robot System SYS First, the configuration of the robot system SYS will be described.

[0023] (1-1) Overall Configuration of Robot System SYS First, the overall configuration of the robot system SYS will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the overall configuration of the robot system SYS.

[0024] 1, the robot system SYS includes a robot 1, an imaging system 2, a control device 3, and an end effector 4. The imaging system 2 may also be referred to as an imaging unit.

[0025] The robot 1 is a device capable of performing a predetermined process on a target object OBJ. An example of the robot 1 is shown in FIG. 2. FIG. 2 is a side view showing the appearance of the robot 1. As shown in FIG. 2, the robot 1 includes, for example, a base 11, a robot arm 12, and a robot control device 13.

[0026] The base 11 is a component that forms the base of the robot 1. The base 11 is placed on a support surface S such as a floor surface. The base 11 may be fixed to the support surface S. Alternatively, the base 11 may be movable relative to the support surface S. Note that FIG. 2 shows an example in which the base 11 is fixed to the support surface S.

[0027] The robot arm 12 is attached to the base 11. The robot arm 12 is a device in which a plurality of links 121 are connected via joints 122. An actuator is built into the joint 122. The link 121 may be rotatable around an axis defined by the joint 122 by the actuator built into the joint 122. At least one link 121 may be extendable and contractible along the direction in which the link 121 extends. A device including the device in which a plurality of links 121 are connected via joints 122 and the base 11 may be referred to as the robot arm 12.

[0028] An end effector 4 is attached to the robot arm 12. That is, the end effector 4 is attached to the robot 1. In the example shown in FIG. 2 , the end effector 4 is attached to the tip of the robot arm 12. The end effector 4 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the end effector 4. That is, the robot 1 moves the end effector 4.

[0029] The end effector 4 is a device that performs a predetermined process (in other words, a predetermined operation) on the target object OBJ. The end effector 4 that performs a predetermined process on the target object OBJ may also be called a processing device.

[0030] For example, the end effector 4 may perform a holding process to hold the target object OBJ as an example of the predetermined process. In this case, the end effector 4 may be considered to be performing the holding process on the target object OBJ that the end effector 4 is to hold. An end effector 4 capable of performing the holding process may be referred to as a holding device. Note that holding the target object OBJ may be considered to be equivalent to picking up the target object OBJ. The holding process of holding the target object OBJ may be considered to be equivalent to a pick-up process of picking up the target object OBJ.

[0031] For example, holding the target object OBJ may include grasping the target object OBJ. For example, holding the target object OBJ may include gripping the target object OBJ using a hand gripper, which is an example of the end effector 4. Holding the target object OBJ may include attracting the target object OBJ. For example, holding the target object OBJ may include attracting (vacuum suction) the target object OBJ using a vacuum gripper, which is an example of the end effector 4. For example, holding the target object OBJ may include attracting (vacuum suction) the target object OBJ using a magnetic gripper (e.g., a magnetically attractive gripper), which is an example of the end effector 4.

[0032] For example, as an example of the predetermined process, the end effector 4 may perform a release process (in other words, a release operation) to release (i.e., let go of) the target object OBJ that it is holding. In this case, the end effector 4 may be considered to be performing the release process on the target object OBJ that it is holding. An end effector 4 that is capable of performing the release process may be referred to as a release device.

[0033] A hand gripper is an example of an end effector 4 capable of holding and releasing an object OBJ. A hand gripper is an end effector 4 that can hold (e.g., grasp) a target object OBJ by physically pinching the target object OBJ using multiple (e.g., two, three, or four) finger or claw members. Another example of an end effector 4 capable of holding and releasing an object OBJ is a vacuum gripper (e.g., a vacuum suction gripper). A vacuum gripper is an end effector 4 that can hold (e.g., attract) a target object OBJ by vacuum suction. Another example of an end effector 4 capable of holding and releasing an object OBJ is a magnetic gripper (e.g., a magnetic suction gripper). FIG. 2 illustrates an example in which the end effector 4 is a hand gripper. However, the end effector 4 capable of holding and releasing an object OBJ is not limited to the above example, and may be any other existing end effector capable of holding and releasing an object OBJ.

[0034] The robot 1 may perform, as an example of a predetermined process, a placement process (in other words, a placement operation) for placing a target object OBJ at a desired position using the end effector 4 capable of performing a holding process and a release process. For example, the robot 1 may hold a first target object OBJ using the end effector 4, and then perform a placement process for placing the first target object OBJ held by the end effector 4 at a desired position of a second target object OBJ different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process for the second target object OBJ on which the end effector 4 is to place the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process for the first target object OBJ to be released by the end effector 4.

[0035] The robot 1 may use the end effector 4 capable of holding and releasing operations to perform a fitting process (i.e., a fitting operation) for fitting a first target object OBJ into a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (i.e., a placement operation). The fitting process may include, for example, a process for fitting (i.e., inserting) the first target object OBJ (e.g., a convex portion of the first target object OBJ) into a concave portion (e.g., a hole) formed in the second target object OBJ. The fitting process may include, for example, a process for fitting the first target object OBJ (e.g., a concave portion of the first target object OBJ) into a convex portion (e.g., a rod) formed in the second target object OBJ.

[0036] The robot 1 may use the end effector 4 capable of performing a holding process and a release process to perform a pasting process (in other words, a pasting operation) for pasting a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation). The robot 1 may use the end effector 4 capable of performing a holding process and a release process to perform a bonding process (in other words, a bonding operation) for adhering the first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation). The robot 1 may use the end effector 4 capable of performing a holding process and a release process to perform a welding process (in other words, a welding operation) for welding the first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation). The robot 1 may use the end effector 4 capable of holding and releasing processes to perform a screw tightening process (in other words, a screw tightening operation) for tightening a first target object OBJ capable of functioning as a screw into a screw hole formed in a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation). In this case, the first target object OBJ may be a screw member capable of functioning as a screw, such as a bolt or a nut. Note that at least one of the pasting process, adhesive process, welding process, and screw tightening process may be referred to as a processing process.

[0037] The robot 1 may use an end effector 4 capable of holding and releasing processes to perform a process for discarding the target object OBJ (in other words, a discarding operation) as a specific example of a placement process (in other words, a placement operation).

[0038] For example, the end effector 4 may perform a predetermined process on each of multiple target objects OBJ. That is, the end effector 4 may perform the predetermined process on the multiple target objects OBJ in sequence. In this case, the robot 1 may move the end effector 4 to a first position where the end effector 4 can perform the predetermined process on a first target object OBJ, and after the end effector 4 has moved to the first position, the end effector 4 may perform the predetermined process on the first target object OBJ. Thereafter, the robot 1 may move the end effector 4 to a second position where the end effector 4 can perform the predetermined process on a second target object OBJ different from the first target object OBJ, and after the end effector 4 has moved to the second position, the end effector 4 may perform the predetermined process on the second target object OBJ.

[0039] For example, the end effector 4 may perform a predetermined process on each of multiple portions of a single target object OBJ. That is, the end effector 4 may sequentially perform a predetermined process on multiple portions of a single target object OBJ. In this case, the robot 1 may move the end effector 4 to a third position where the end effector 4 can perform a predetermined process on a first portion of the target object OBJ, and after the end effector 4 has moved to the third position, the end effector 4 may perform the predetermined process on the first portion of the target object OBJ. Thereafter, the robot 1 may move the end effector 4 to a fourth position where the end effector 4 can perform a predetermined process on a second portion of the target object OBJ, and after the end effector 4 has moved to the fourth position, the end effector 4 may perform the predetermined process on the second portion of the target object OBJ.

[0040] The target object OBJ on which the end effector 4 performs a predetermined process may include a workpiece W, as shown in FIG. 2 . The workpiece W may include, for example, a part or member used to manufacture a desired product. The workpiece W may include, for example, a part or member to be processed to manufacture the desired product. The workpiece W may include, for example, a part or member to be transported to manufacture the desired product. The workpiece W may include, for example, a part or member that is moving due to transportation to manufacture the desired product. The workpiece W may include, for example, a part or member that is moving to manufacture the desired product.

[0041] As shown in FIG. 2 , the target object OBJ on which the end effector 4 performs a predetermined process may include a placement device T on which a workpiece W is placed. An example of the placement device T is a container (storage box) CB. The container CB may have a bottom wall BS and a side wall SS protruding upward from the bottom wall BS. The placement device T may be a placement device T in which the workpiece W is placed on the bottom wall BS in a storage space SP enclosed by the bottom wall BS and the side wall SS. The container CB may also have a bottom wall BS and a side wall SS protruding upward from the bottom wall BS. The placement device T may be a placement device T capable of accommodating the workpiece W in the storage space SP enclosed by the bottom wall BS and the side wall SS. However, the container CB may not have a side wall SS. A container CB without a side wall SS may be referred to as a pallet. Furthermore, the placement device T is not limited to a container CB or a pallet, but may also be an existing object on which the workpiece W can be placed. The placement device T may also be referred to as a placement member. The placement device T may be disposed on the support surface S. The placement device T may be fixed to the support surface S. Alternatively, at least a portion of the placement device T may be movable relative to the support surface S. A first example in which at least a portion of the placement device T is movable relative to the support surface S is when the placement device T is supported by a transport device that can move (in other words, transport) the placement device T. In this case, the transport device may be, for example, a belt conveyor. A second example in which at least a portion of the placement device T is movable relative to the support surface S is when the placement device T itself is movable relative to the support surface S. In this case, the transport device may be at least one of an automatic guided vehicle, an autonomously traveling transport robot, and an unmanned aerial vehicle. Note that FIG. 2 shows an example in which the placement device T is self-propelled on the support surface S. Note that at least one of the automatic guided vehicle, the autonomously traveling transport robot, and the unmanned aerial vehicle may also be referred to as a movable device.

[0042] The robot control device 13 controls the operation of the robot 1 .

[0043] Specifically, the robot control device 13 may control the movement of the robot arm 12. For example, the robot control device 13 may control the movement of the robot arm 12 so that a desired link 121 rotates around an axis defined by a desired joint 122. For example, the robot control device 13 may control the movement of the robot arm 12 so that the end effector 4 attached to the robot arm 12 is positioned at a desired position. For example, the robot control device 13 may control the movement of the robot arm 12 so that the end effector 4 attached to the robot arm 12 moves to a desired position.

[0044] In addition to controlling the operation of the robot 1, the robot control device 13 may also control the operation of the end effector 4 attached to the robot 1 (processing performed by the end effector 4). For example, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 holds the target object OBJ at a desired timing. That is, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 performs a holding process at a desired timing. For example, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 releases the held target object OBJ at a desired timing. That is, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 performs a release process at a desired timing. In order for the end effector 4 to hold or release the target object OBJ, if the end effector 4 is a hand gripper, the robot control device 13 may control the timing of opening and closing the hand gripper. If the end effector 4 is a vacuum gripper, the robot control device 13 may control the timing of turning on / off the vacuum device of the vacuum gripper (or the vacuum suction force).If the end effector 4 is a magnetic gripper, the robot control device 13 may control the timing of turning on / off the magnetic suction device of the magnetic gripper (or the magnetic force).

[0045] FIG. 2 illustrates an example in which the robot 1 is a robot arm 12 (i.e., a vertically articulated robot). However, the robot 1 may be a robot different from a vertically articulated robot. For example, the robot 1 may be a SCARA robot (i.e., a horizontally articulated robot). For example, the robot 1 may be a parallel link robot. For example, the robot 1 may be a dual-arm robot including two robot arms 12. For example, the robot 1 may be a Cartesian coordinate robot. For example, the robot 1 may be a cylindrical coordinate robot. The robot 1 may be referred to as a mobile device. In addition to the robot 1, the mobile device may include at least one of an automatic guided vehicle, an autonomously traveling transport robot, and an unmanned aerial vehicle. For example, the robot 1 may be installed on at least one of an automatic guided vehicle, an autonomously traveling transport robot, and an unmanned aerial vehicle. When the robot 1 is installed on a mobile device (e.g., at least one of an automatic guided vehicle, an autonomously traveling transport robot, and an unmanned aerial vehicle), the robot control device 13 may control the operation of the mobile device on which the robot 1 is installed, in addition to controlling the operation of the robot 1.

[0046] 1, the imaging system 2 captures an image of a target object OBJ. To capture an image of the target object OBJ, the imaging system 2 includes an imaging device 21 and an illumination device 23.

[0047] The imaging device 21 is a camera capable of capturing an image of the target object OBJ. For example, the imaging device 21 may capture an image of the target object OBJ under the control of the control device 3. The imaging device 21 generates image data IMG by capturing an image of the target object OBJ. In other words, the imaging device 21 generates image data IMG that is the result of capturing the image of the target object OBJ. The image data IMG generated by the imaging device 21 is output from the imaging device 21 to the control device 3. As a result, the control device 3 acquires the image data IMG acquired by the imaging device 21 capturing an image of the target object OBJ.

[0048] The imaging system 2 may include an imaging device 21 including a monocular camera. In this case, the imaging device 21 may generate image data IMG including one image data item generated by the monocular camera. Alternatively, the imaging system 2 may include an imaging device 21 including a stereo camera including two monocular cameras. In this case, the imaging device 21 may generate image data IMG including two image data items generated by the two monocular cameras. Alternatively, the imaging system 2 may include an imaging device 21 including a first monocular camera and a stereo camera including two second monocular cameras different from the first monocular camera. In this case, the imaging device 21 may generate image data IMG including one image data item generated by the first monocular camera and image data IMG including two image data items generated by the two second monocular cameras.

[0049] The imaging system 2 may include an imaging device 21 that includes a camera other than a monocular camera and a stereo camera. For example, the imaging system 2 may include an imaging device 21 that includes three or more monocular cameras. For example, the imaging system 2 may include an imaging device 21 that includes at least one of a light field camera, a plenoptic camera, and a multispectral camera.

[0050] The imaging device 21 may capture an image of the entire target object OBJ. Alternatively, the imaging device 21 may capture an image of a portion of the target object OBJ. In other words, the imaging device 21 may capture an image of a portion of the target object OBJ while not capturing an image of another portion of the target object OBJ.

[0051] The imaging device 21 may capture an image of a single target object OBJ. In other words, a single target object OBJ may appear in the image represented by the image data IMG. Alternatively, the imaging device 21 may capture an image of multiple target objects OBJ. In other words, a plurality of target objects OBJ may appear in the image represented by the image data IMG. In this case, as will be described in detail later, the control device 3 may determine (in other words, select) one of the multiple target objects OBJ captured by the imaging device 21 as the target object OBJ on which the end effector 4 will actually perform a predetermined process. Note that one of the multiple target objects OBJ captured by the imaging device 21 on which the end effector 4 will actually perform a predetermined process may be referred to as a process execution object.

[0052] When the imaging device 21 captures images of multiple target objects OBJ, the multiple target objects OBJ captured by the imaging device 21 may be arranged such that at least two of the multiple target objects OBJ at least partially overlap. As an example, when the target objects OBJ are workpieces W corresponding to components used to manufacture a desired product, the multiple workpieces W (i.e., multiple components) may be arranged such that at least two of the multiple workpieces W at least partially overlap. In this case, the end effector 4 may perform the above-described holding process of holding at least one workpiece W among the multiple workpieces W that are randomly arranged. In other words, the robot 1 may perform bulk picking, in which workpieces W are picked one by one from the multiple workpieces W that are randomly arranged. Note that the multiple workpieces W that are randomly arranged may also be referred to as multiple workpieces W that are irregularly arranged, multiple workpieces W that are casually arranged, or multiple workpieces W that are randomly arranged.

[0053] Alternatively, the multiple target objects OBJ captured by the imaging device 21 may be arranged regularly. As an example, if the target objects OBJ are workpieces W corresponding to parts used to manufacture a desired product, the multiple workpieces W (i.e., multiple parts) may be arranged in a matrix. In this case, the end effector 4 may perform the above-described holding process of holding at least one workpiece W among the multiple regularly arranged workpieces W. That is, the robot 1 may pick up a workpiece W one by one from the multiple regularly arranged workpieces W. Note that the multiple regularly arranged workpieces W can also be rephrased as multiple workpieces W that are orderly arranged, or multiple workpieces W that are arranged according to a certain arrangement rule. Note that, as another example of the multiple target objects OBJ captured by the imaging device 21 being arranged regularly, the multiple target objects OBJ may be arranged (e.g., fitted) in an object different from the target object OBJ.

[0054] Note that the placement process for placing the target object OBJ held by the end effector 4 may include a placement process for placing the target object OBJ randomly (in other words, haphazardly), as with the holding process, or a placement process for placing the target object OBJ regularly (in other words, orderly). For example, the robot 1 may use the end effector 4 to hold one of a plurality of workpieces W that are regularly or randomly placed in the first container CB, and then place the one workpiece W held by the end effector 4 regularly or randomly in the second container CB.

[0055] The illumination device 23 is a device capable of irradiating illumination light onto the target object OBJ (for example, at least one target object OBJ when multiple target objects OBJ exist). For example, the illumination device 23 may irradiate the target object OBJ with illumination light under the control of the control device 3. In particular, the illumination device 23 is a device capable of irradiating the target object OBJ with illumination light, thereby illuminating the target object OBJ with illumination light. In this case, the imaging device 21 may capture an image of the target object OBJ illuminated with illumination light. However, the illumination device 23 does not need to irradiate the target object OBJ with illumination light. In this case, the imaging system 2 (robot system SYS) does not need to be equipped with the illumination device 23.

[0056] When the imaging device 21 includes a stereo camera, the illumination device 23 may be a device capable of projecting a desired projection pattern onto the target object OBJ by irradiating the target object OBJ with illumination light. The desired projection pattern may include, for example, a random pattern. The random pattern may include a random dot pattern. The desired projection pattern may include, for example, a one-dimensional or two-dimensional grid pattern. The desired projection pattern may include, for example, a line pattern. The desired projection pattern may include, for example, a stripe pattern. The desired projection pattern may include other projection patterns. However, even when the imaging device 21 does not include a stereo camera (i.e., the imaging device 21 includes a monocular camera), the illumination device 23 may project the desired projection pattern onto the target object OBJ by irradiating the target object OBJ with illumination light. In this case, the illumination device 23 may irradiate the target object OBJ with illumination light having a uniform intensity distribution (e.g., a uniform intensity distribution). The desired projection pattern may also be referred to as light having a desired intensity distribution.

[0057] The imaging system 2 is attached to the robot arm 12, similar to the end effector 4. That is, the imaging device 21 and the lighting device 23 are attached to the robot arm 12. For example, as shown in FIG. 2 , the imaging device 21 and the lighting device 23 may be attached to the tip of the robot arm 12, similar to the end effector 4. In this case, the imaging device 21 and the lighting device 23 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the imaging device 21 and the lighting device 23.

[0058] However, the imaging system 2 does not have to be attached to the robot arm 12. For example, the imaging system 2 may be disposed at any position where it can capture an image of the target object OBJ. Furthermore, for example, the imaging system 2 may be disposed at any position where it can irradiate the target object OBJ with illumination light. For example, the imaging system 2 may be attached to a structure such as a pillar. Note that either the imaging device 21 or the lighting device 23 may be attached to the robot arm 12, and the other either the imaging device 21 or the lighting device 23 may be attached to a location different from the robot arm 12 (for example, a structure such as a pillar).

[0059] The imaging device 21 may capture an image of the target object OBJ (e.g., at least one target object OBJ when multiple target objects OBJ are present) during a period in which the imaging device 21 and the target object OBJ are displaced relative to each other. Note that the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may refer to a state in which the relative positional relationship between the imaging device 21 and the target object OBJ is changing. The state in which the imaging device 21 and the target object OBJ are displaced relative to each other may refer to a state in which the imaging device 21 and the target object OBJ are moving relative to each other. For example, the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the target object OBJ is moving relative to the imaging device 21. For example, the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the imaging device 21 is moving relative to the target object OBJ. In this case, the imaging device 21 does not need to be stationary to capture an image of the target object OBJ, and the robot system SYS can efficiently perform predetermined processing on the target object OBJ using the end effector 4.

[0060] Alternatively, the imaging device 21 may capture an image of the target object OBJ during a period in which there is no relative displacement between the imaging device 21 and the target object OBJ. Note that the state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the relative positional relationship between the imaging device 21 and the target object OBJ is not changing. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which there is no relative movement between the imaging device 21 and the target object OBJ. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the imaging device 21 and the target object OBJ are stationary. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the imaging device 21 and the target object OBJ are moving at the same moving speed in the same moving direction.

[0061] The control device 3 performs robot control processing. The robot control processing includes processing for generating a robot control signal for controlling the robot 1. Specifically, the control device 3 generates the robot control signal based on image data IMG output from the imaging system 2. In this embodiment, the control device 3 calculates at least one of the position and orientation of the target object OBJ in the global coordinate system of the robot system SYS based on the image data IMG, and generates the robot control signal based on the calculated at least one of the position and orientation of the target object OBJ.

[0062] The global coordinate system is a coordinate system that serves as the reference for the robot system SYS. For example, the global coordinate system may be a coordinate system that serves as the reference for the robot 1. The global coordinate system can also be said to be a coordinate system used to control the robot 1. For example, a world coordinate system that is defined based on the support surface S on which the robot 1 is placed may be used as the global coordinate system. In other words, a world coordinate system that is fixed with respect to the support surface S on which the robot 1 is placed may be used as the global coordinate system. In the following description, unless otherwise specified, the X-axis, Y-axis, and Z-axis may refer to the X-axis, Y-axis, and Z-axis in the global coordinate system, respectively.

[0063] However, the control device 3 may calculate at least one of the position and orientation of the target object OBJ in a coordinate system different from the global coordinate system based on the image data IMG. The coordinate system different from the global coordinate system may include at least one of the robot coordinate system and the imaging coordinate system. The robot coordinate system may be a coordinate system defined based on the robot 1. That is, the robot coordinate system may be a coordinate system fixed with respect to the robot 1 (for example, fixed with respect to the base 11 of the robot 1). The imaging coordinate system may be a coordinate system defined based on the imaging device 21. That is, the imaging coordinate system may be a coordinate system fixed with respect to the imaging device 21. An example of the imaging coordinate system is a coordinate system defined based on the optical axis AX21 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 21. An example of the imaging coordinate system is a coordinate system in which one of the three coordinate axes constituting the imaging coordinate system is an axis along the optical axis AX21 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 21.

[0064] The control device 3 may perform end effector control processing in addition to performing robot control processing. The end effector control processing may include processing for generating an end effector control signal for controlling the end effector 4. Specifically, the control device 3 may generate the end effector control signal based on at least one of the calculated position and orientation of the target object OBJ.

[0065] The end effector control processing may or may not be included in the robot control processing. In other words, the end effector control signal generated by the control device 3 may or may not be included in the robot control signal. In the following description, for convenience of explanation, an example will be described in which the end effector control processing is included in the robot control processing (in other words, the end effector control signal is included in the robot control signal). Therefore, in the following description, the robot control processing may mean processing for generating at least one of a robot control signal and an end effector control signal. In addition, in the following description, the robot control signal may mean at least one of a signal for controlling the robot 1 and a signal for controlling the end effector 4. The robot control signal may also be simply referred to as a control signal.

[0066] As described above, when the robot 1 is installed on a movable device (for example, at least one of an automatic guided vehicle, an autonomous transport robot, and an unmanned aerial vehicle), the control device 3 may perform a movable device control process in addition to performing a robot control process. The movable device control process may include a process of generating a movable device control signal for controlling the movable device. Specifically, the control device 3 may generate the movable device control signal based on at least one of the calculated position and orientation of the target object OBJ.

[0067] The movable device control process may or may not be included in the robot control process. In other words, the movable device control signal generated by the control device 3 may or may not be included in the robot control signal. In the following description, for convenience of explanation, an example will be described in which the movable device control process is included in the robot control process (i.e., the movable device control signal is included in the robot control signal). Therefore, in the following description, the robot control process may mean a process for generating at least one of a robot control signal, an end effector control signal, and a movable device control signal. In addition, in the following description, the robot control signal may mean at least one of a signal for controlling the robot 1, a signal for controlling the end effector 4, and a signal for controlling the movable device.

[0068] In this way, the control device 3 and the imaging system 2 are used to control the robot 1. Therefore, a system including the control device 3 and the imaging system 2 may be referred to as a robot control system or a control system.

[0069] The robot control signal generated by the control device 3 is output to the robot control device 13 of the robot 1. The robot control device 13 controls the operation of the robot 1 based on the robot control signal generated by the control device 3. For this reason, the robot control signal may include a signal for controlling the operation of the robot 1.

[0070] As described above, when the robot control signal includes a signal for controlling the robot arm 12, the robot control device 13 may control the robot arm 12 based on the robot control signal. For example, the robot control device 13 may control the operation of the actuator built into the joint 122 based on the robot control signal, thereby controlling the operation of the robot arm 12.

[0071] For example, as described above, the robot arm 12 moves the end effector 4. In this case, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at a desired position. The robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves to a desired position. The robot control signal may include a signal for controlling the robot arm 12 so that the positional relationship between the end effector 4 and the target object OBJ is a desired positional relationship. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 is located at a desired position. The robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves to a desired position. The robot control device 13 may control the robot arm 12 based on the robot control signal so that the positional relationship between the end effector 4 and the target object OBJ is a desired positional relationship.

[0072] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves toward (i.e., approaches) a holding target position where the end effector 4 can hold the target object OBJ. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at the holding target position. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves toward (i.e., approaches) the holding target position. That is, the robot control signal may control the robot arm 12 so that the end effector 4 is located at the holding target position.

[0073] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the robot arm 12 so that the posture of the end effector 4 becomes a holding target posture that allows the end effector 4 to hold the target object OBJ. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the posture of the end effector 4 becomes the holding target posture.

[0074] As another example, when performing a release process to release a target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves toward (i.e., approaches) a release target position where the target object OBJ held by the end effector 4 should be released. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at the release target position. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves toward (i.e., approaches) the release target position. That is, the robot control signal may control the robot arm 12 so that the end effector 4 is located at the release target position.

[0075] As another example, when performing a release process to release a target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the robot arm 12 so that the posture of the end effector 4 becomes a release target posture that allows the end effector 4 to release the target object OBJ. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the posture of the end effector 4 becomes the release target posture.

[0076] As described above, when the robot control signal includes a signal for controlling the end effector 4, the robot control device 13 may control the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the end effector 4 by controlling the operation of an actuator that moves a hand gripper that constitutes the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the end effector 4 by controlling the operation of a vacuum device of a vacuum gripper that constitutes the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the end effector 4 by controlling the operation of a magnetic attraction device of a magnetic gripper that constitutes the end effector 4 based on the robot control signal.

[0077] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the end effector 4 so that the end effector 4 located at the above-mentioned holding target position and / or in the above-mentioned holding target posture holds the target object OBJ. In this case, the robot control device 13 may control the end effector 4 based on the robot control signal so that the end effector 4 located at the above-mentioned holding target position and / or in the above-mentioned holding target posture holds the target object OBJ.

[0078] As described above, when the robot control signal includes a signal for controlling a movable device (e.g., at least one of an automatic guided vehicle, an autonomous transport robot, and an unmanned aerial vehicle) on which the robot 1 is installed, the robot control device 13 may control the movable device based on the robot control signal. For example, the robot control device 13 may control a power source (e.g., a motor or an engine) of the movable device based on the robot control signal so that the robot 1 moves to a target position indicated directly or indirectly by the robot control signal.

[0079] As another example, when performing a release process to release the target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the end effector 4 to release the target object OBJ held by the end effector 4 located at the above-mentioned release target position and / or in the above-mentioned release target posture. In this case, the robot control device 13 may control the end effector 4 based on the robot control signal to release the target object OBJ held by the end effector 4 located at the release target position and / or in the release target posture.

[0080] The robot control signal may include a signal that can be used as is by the robot control device 13 to control the operation of the robot 1. The robot control signal may include a signal that can be used as is as a robot drive signal that the robot control device 13 uses to control the operation of the robot 1. In this case, the robot control device 13 may use the robot control signal as is to control the operation of the robot 1. For example, the control device 3 may generate a drive signal for an actuator built into the joint 122 of the robot arm 12 as the robot control signal, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the actuator built into the joint 122 of the robot arm 12.

[0081] The robot control signal may include a signal that can be used directly by the robot control device 13 to control the operation of the end effector 4. The robot control signal may include a signal that can be used directly as an end effector drive signal used by the robot control device 13 to control the operation of the end effector 4. In this case, the robot control device 13 may use the robot control signal directly to control the operation of the end effector 4. For example, the control device 3 may generate, as the robot control signal, a drive signal (end effector drive signal) for an actuator that moves a hand gripper that constitutes the end effector 4, and the robot control device 13 may use the robot control signal generated by the control device 3 directly to control the actuator of the end effector 4. For example, the control device 3 may generate, as the robot control signal, a drive signal (end effector drive signal) for driving a vacuum device of a vacuum gripper that constitutes the end effector 4, and the robot control device 13 may use the robot control signal generated by the control device 3 directly to control the vacuum device of the end effector 4. For example, the control device 3 may generate a drive signal (end effector drive signal) for driving the magnetic adsorption device of the magnetic gripper that constitutes the end effector 4 as a robot control signal, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the magnetic adsorption device of the end effector 4.

[0082] The robot control signal may include a signal that can be used directly by the robot controller 13 to control the operation of a movable device on which the robot 1 is installed (e.g., at least one of an automatic guided vehicle, an autonomous transport robot, and an unmanned aerial vehicle). The robot control signal may include a signal that can be used directly as a movable device drive signal that the robot controller 13 uses to control the operation of the movable device. In this case, the robot controller 13 may use the robot control signal directly to control the operation of the movable device. For example, the controller 3 may generate a power source drive signal (movable device drive signal) for driving a power source (e.g., a motor or an engine) of the movable device as the robot control signal, and the robot controller 13 may use the robot control signal generated by the controller 3 directly to control the power source of the movable device 1.

[0083] As described above, if the robot control signal includes a signal that can be directly used by the robot control device 13 to control the operation of at least one of the robot 1, the end effector 4, and the movable device, the robot 1 does not need to be equipped with the robot control device 13. In this case, the control device 3 may use the robot control signal to control an actuator built into the joint 122 of the robot arm 12. For example, the control device 3 may use the robot control signal (end effector drive signal) to control an actuator that moves a hand gripper that constitutes the end effector 4. For example, the control device 3 may use the robot control signal (end effector drive signal) to control a vacuum device of a vacuum gripper that constitutes the end effector 4. For example, the control device 3 may use the robot control signal (end effector drive signal) to control a magnetic attraction device of a magnetic gripper that constitutes the end effector 4. For example, the control device 3 may use the robot control signal (end effector drive signal) to control a movable device installed on the robot 1.

[0084] Alternatively, the robot control signal may include a signal that can be used by the robot control device 13 to generate a robot drive signal for controlling the operation of the robot 1. In this case, the robot control device 13 may generate a robot drive signal for controlling the operation of the robot 1 based on the robot control signal, and control the operation of the robot 1 based on the generated robot drive signal. For example, the robot control device 13 may generate a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 based on the robot control signal, and control the actuator built into the joint 122 of the robot arm 12 based on the generated robot drive signal.

[0085] The robot control signal may include a signal that the robot control device 13 can use to generate an end effector drive signal for controlling the operation of the end effector 4. In this case, the robot control device 13 may generate an end effector drive signal for controlling the operation of the end effector 4 based on the robot control signal and control the operation of the end effector 4 based on the generated end effector drive signal. For example, if the end effector 4 is a hand gripper, the robot control device 13 may generate an end effector drive signal for driving an actuator of the hand gripper based on the robot control signal and control the actuator of the hand gripper based on the generated end effector drive signal. For example, if the end effector 4 is a magnetic gripper, the robot control device 13 may generate an end effector drive signal for driving a magnetic attraction device of the magnetic gripper based on the robot control signal and control the magnetic attraction device based on the generated end effector drive signal.

[0086] The robot control signal may include a signal usable by the robot controller 13 to generate a movable device drive signal for controlling the operation of a movable device on which the robot 1 is installed. In this case, the robot controller 13 may generate a movable device drive signal for controlling the operation of the movable device based on the robot control signal, and control the operation of the movable device based on the generated movable device drive signal. For example, the robot controller 13 may generate a power source drive signal (movable device drive signal) for driving a power source (e.g., a motor or an engine) of the movable device based on the robot control signal, and control the power source of the movable device based on the generated mobile device drive signal.

[0087] The signals available to the robot controller 13 for generating the robot drive signals may include signals representing at least one of the position and orientation of the target object OBJ in a global coordinate system. The signals available to the robot controller 13 for generating the robot drive signals may include signals representing a desired positional relationship between the robot 1 and the target object OBJ in a global coordinate system.

[0088] The signals available to the robot controller 13 for generating the robot drive signals may include signals representing a desired position of the end effector 4 in the global coordinate system. An example of a desired position is a processing position where the end effector 4 should process the target object OBJ. For example, the desired position may include a target holding position where the end effector 4 should hold the target object OBJ. For example, the desired position may include a target release position where the end effector 4 should release the target object OBJ. The signals available to the robot controller 13 for generating the robot drive signals may include signals representing a desired position of the tip of the robot arm 12 (e.g., a tool center point) in the global coordinate system. The signals available to the robot controller 13 for generating the robot drive signals may include a signal representing a desired position of the imaging system 2 in the global coordinate system.

[0089] The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a desired posture of the end effector 4 in the global coordinate system. An example of a desired posture is a posture (processing posture) that the end effector 4 should assume when processing the target object OBJ. For example, the desired posture may include a posture (holding target posture) that the end effector 4 should assume when holding the target object OBJ. For example, the desired posture may include a posture (release target posture) that the end effector 4 should assume when releasing the target object OBJ. The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a desired posture of the tip of the robot arm 12 (e.g., tool center point) in the global coordinate system. The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a desired posture of the imaging system 2 in the global coordinate system.

[0090] The signal that the robot controller 13 can use to generate the robot drive signal may be a signal that represents the amount and direction of movement from the current position of the end effector 4 to the desired position of the end effector 4 .

[0091] (1-2) Configuration of the Control Device 3 Next, the configuration of the control device 3 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the control device 3.

[0092] 3 , the control device 3 includes a calculation device 31, a storage device 32, and a communication device 33. The control device 3 may further include an input device 34 and an output device 35. However, the control device 3 does not necessarily have to include at least one of the input device 34 and the output device 35. The calculation device 31, the storage device 32, the communication device 33, the input device 34, and the output device 35 may be connected via a data bus 36.

[0093] The arithmetic device 31 is hardware that includes at least a circuit (for example, at least one of an electronic circuit and an electric circuit). For this reason, the arithmetic device 31 may be referred to as a group of circuits.

[0094] The arithmetic device 31 includes at least one processor (i.e., one processor or multiple processors) as hardware. The processor may include, for example, a processor conforming to a von Neumann computer architecture. The processor conforming to the von Neumann computer architecture may include at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor may include, for example, a processor conforming to a non-von Neumann computer architecture. The processor conforming to the non-von Neumann computer architecture may include at least one of an FPGA (Field Programmable Gate Array) and an ASIC (Application Specific Circuit). The processor may be realized by a group of circuits (e.g., at least one of an electronic circuit and an electric circuit).

[0095] The arithmetic device 31 reads a computer program 321 including at least one of computer program code and computer program instructions. For example, the arithmetic device 31 may read the computer program 321 stored in the storage device 32. For example, the arithmetic device 31 may read the computer program 321 stored in a computer-readable, non-transitory storage medium using a storage medium reading device (not shown) included in the control device 3. The computer program 321 read from the storage medium may be stored in the storage device 32. The arithmetic device 31 may acquire (i.e., download or read) the computer program 321 from a device (not shown) located outside the control device 3 via the communication device 33 (or another communication device). The downloaded computer program 321 may be stored in the storage device 32.

[0096] The arithmetic device 31 executes the loaded computer program 321. As a result, logical functional blocks for executing the processing to be performed by the control device 3 (for example, the robot control processing described above) are realized within the arithmetic device 31. In other words, the arithmetic device 31, together with the storage device 32 or the like in which the computer program 321 is recorded (in other words, together with the storage device 32 and the computer program 321 recorded in the storage device 32 or the like), can function as a controller or computer for realizing the logical functional blocks for executing the processing to be performed by the control device 3. In other words, the at least one processor included in the arithmetic device 31, the memory (recording medium) included in the storage device 32 or the like, and the computer program 321 are configured so that the control device 3 performs the processing to be performed by the control device 3 (for example, the robot control processing described above).

[0097] A computational model that can be constructed by machine learning may be implemented in the computational device 31 by the computational device executing the computer program 321. An example of a computational model that can be constructed by machine learning is a computational model including a neural network (so-called artificial intelligence (AI)). In this case, learning of the computational model may include learning of parameters of the neural network (e.g., at least one of a weight and a bias). The computational device 31 may execute a robot control process using the computational model. In other words, the operation of executing the robot control process may include the operation of executing the robot control process using the computational model. Note that a computational model that has been constructed by offline machine learning using teacher data may be implemented in the computational device 31. Furthermore, the computational model implemented in the computational device 31 may be updated by online machine learning on the computational device 31. Alternatively, the calculation device 31 may perform robot control processing using a calculation model implemented in a device external to the calculation device 31 (i.e., a device provided outside the control device 3) in addition to or instead of the calculation model implemented in the calculation device 31.

[0098] The recording medium for recording the computer program 321 executed by the arithmetic device 31 may be at least one of the following: a CD-ROM, CD-R, CD-RW, a flexible disk, an MO, a DVD-ROM, a DVD-RAM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, or an optical disk such as Blu-ray (registered trademark), a magnetic medium such as a magnetic tape, a magneto-optical disk, a semiconductor memory such as a USB memory, or any other medium capable of storing a program. The recording medium may include a device capable of recording the computer program 321 (for example, a general-purpose device or a dedicated device in which the computer program 321 is implemented in a state in which it can be executed in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program 321 may be realized by a logical processing block realized within the arithmetic device 31 (i.e., processor) when the arithmetic device 31 executes the computer program 321, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) provided in the arithmetic device 31, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.

[0099] 3 shows an example of logical functional blocks realized in the arithmetic device 31 for executing robot control processing. As shown in Fig. 3, a position and orientation calculation unit 311, a signal generation unit 312, and an erroneous holding detection unit 313 are realized in the arithmetic device 31. Note that the processing performed by each of the position and orientation calculation unit 311, the signal generation unit 312, and the erroneous holding detection unit 313 will be described in detail later with reference to Fig. 5 etc., and therefore will not be described here.

[0100] The storage device 32 includes at least one memory capable of storing desired data. In other words, the storage device 32 includes at least one memory containing desired data. The memory may be realized by a group of circuits (e.g., at least one of electronic circuits and electric circuits). For example, the storage device 32 may store a computer program 321 executed by the arithmetic device 31. In this case, the storage device 32 (memory) may be used as the above-mentioned recording medium for recording the computer program 321 executed by the arithmetic device 31. The storage device 32 may temporarily store data used by the arithmetic device 31 when the arithmetic device 31 is executing the computer program 321. The storage device 32 may also store data to be stored long-term by the control device 3. The storage device 32 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. That is, the storage device 32 may include a non-transitory recording medium.

[0101] The communication device 33 is capable of communicating with both the robot 1 and the imaging system 2 via a communication network (not shown). Alternatively, the communication device 33 may be capable of communicating with another device different from the robot 1 and the imaging system 2, in addition to or instead of at least one of the robot 1 and the imaging system 2, via a communication network (not shown). In this embodiment, the communication device 33 may receive (i.e., acquire) image data IMG from the imaging system 2. Furthermore, the communication device 33 may transmit (i.e., output) a robot control signal to the robot 1. The communication device 33 that outputs the robot control signal to the robot 1 may be referred to as an output unit.

[0102] The input device 34 is a device that accepts information input to the control device 3 from outside the control device 3. For example, the input device 34 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by a user of the control device 3. For example, the input device 34 may include a recording medium reading device that can read information recorded as data on a recording medium that can be externally attached to the control device 3.

[0103] It should be noted that information can be input as data to the control device 3 from a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an input device that accepts information input to the control device 3 from outside the control device 3.

[0104] The output device 35 is a device that outputs information to the outside of the control device 3. For example, the output device 35 may output information as an image. That is, the output device 35 may include a display device (a so-called display) that can display an image. For example, the output device 35 may output information as sound. That is, the output device 35 may include an audio device (a so-called speaker) that can output sound. For example, the output device 35 may output information on paper. That is, the output device 35 may include a printing device (a so-called printer) that can print desired information on paper. For example, the output device 35 may output information as data to a recording medium that can be externally attached to the control device 3.

[0105] The control device 3 can output information as data to a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an output device that outputs information to a device external to the control device 3.

[0106] The robot control device 13 provided in the robot 1 described above may also have a configuration similar to that of the control device 3. That is, as shown in Fig. 4, which is a block diagram showing the configuration of the robot control device 13, the robot control device 13 includes an arithmetic device 131, a storage device 132, and a communication device 133. The robot control device 13 may further include an input device 134 and an output device 135. However, the robot control device 13 does not have to include at least one of the input device 134 and the output device 135. The arithmetic device 131, the storage device 132, the communication device 133, the input device 134, and the output device 135 may be connected via a data bus 136.

[0107] The arithmetic device 131 includes at least one processor (i.e., one processor or multiple processors) as hardware. The processor may include, for example, a processor conforming to a von Neumann computer architecture. The processor conforming to the von Neumann computer architecture may include at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor may include, for example, a processor conforming to a non-von Neumann computer architecture. The processor conforming to the non-von Neumann computer architecture may include at least one of an FPGA (Field Programmable Gate Array) and an ASIC (Application Specific Circuit).

[0108] The arithmetic device 131 reads a computer program 1321 including at least one of computer program code and computer program instructions. For example, the arithmetic device 131 may read the computer program 1321 stored in the storage device 132. For example, the arithmetic device 131 may read the computer program 1321 stored in a computer-readable, non-transitory storage medium using a storage medium reading device (not shown) included in the robot control device 13. The computer program 1321 read from the storage medium may be stored in the storage device 132. The arithmetic device 131 may acquire (i.e., download or read) the computer program 1321 from a device (not shown) located outside the robot control device 13 via the communication device 133 (or another communication device). The downloaded computer program 1321 may be stored in the storage device 132.

[0109] The arithmetic device 131 executes the loaded computer program 1321. As a result, logical functional blocks for executing processing to be performed by the robot control device 13 (e.g., processing to control at least one of the robot 1 and the end effector 4) are realized within the arithmetic device 131. In other words, the arithmetic device 131, together with the storage device 132 or the like in which the computer program 1321 is recorded (in other words, together with the storage device 132 and the computer program 1321 recorded in the storage device 132 or the like), can function as a controller or computer for realizing the logical functional blocks for executing processing to be performed by the robot control device 13. In other words, the at least one processor included in the arithmetic device 131, the memory (recording medium) included in the storage device 132 or the like, and the computer program 1321 are configured so that the robot control device 13 performs processing to be performed by the robot control device 13.

[0110] A computational model that can be constructed by machine learning may be implemented in the computational device 131 by the computational device executing the computer program 1321. An example of a computational model that can be constructed by machine learning is a computational model including a neural network (so-called artificial intelligence (AI)). In this case, learning of the computational model may include learning of parameters of the neural network (e.g., at least one of a weight and a bias). The computational device 131 may control at least one of the robot 1 and the end effector 4 using the computational model. In other words, the operation of controlling at least one of the robot 1 and the end effector 4 may include the operation of controlling at least one of the robot 1 and the end effector 4 using the computational model. The computational device 131 may be implemented with a computational model that has been constructed by offline machine learning using training data. Furthermore, the computational model implemented in the computational device 131 may be updated by online machine learning on the computational device 131. Alternatively, the computing device 131 may execute the robot control process using a computational model implemented in a device external to the computing device 131 (i.e., a device provided outside the robot control device 13) in addition to or instead of the computational model implemented in the computing device 131.

[0111] The recording medium for recording the computer program 1321 executed by the arithmetic device 131 may be at least one of a CD-ROM, CD-R, CD-RW, flexible disk, MO, DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW, and Blu-ray (registered trademark) optical disk, magnetic medium such as magnetic tape, magneto-optical disk, semiconductor memory such as USB memory, and any other medium capable of storing a program. The recording medium may include a device capable of recording the computer program 1321 (for example, a general-purpose device or dedicated device in which the computer program 1321 is implemented in a state in which it can be executed in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program 1321 may be realized by a logical processing block realized within the arithmetic device 131 when the arithmetic device 131 (i.e., processor) executes the computer program 1321, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) provided in the arithmetic device 131, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.

[0112] The storage device 132 includes at least one memory capable of storing desired data. In other words, the storage device 132 includes at least one memory containing desired data. For example, the storage device 132 may store a computer program 1321 executed by the arithmetic device 131. In this case, the storage device 132 (memory) may be used as the above-mentioned recording medium for recording the computer program 1321 executed by the arithmetic device 131. The storage device 132 may temporarily store data used by the arithmetic device 131 when the arithmetic device 131 is executing the computer program 1321. The storage device 132 may also store data to be stored long-term by the robot control device 13. The storage device 132 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. That is, the storage device 132 may include a non-transitory recording medium.

[0113] The communication device 133 is capable of communicating with at least one of the imaging system 2 and the control device 3 via a communication network (not shown). Alternatively, the communication device 133 may be capable of communicating with another device different from the imaging system 2 and the control device 3 via a communication network (not shown), in addition to or instead of at least one of the imaging system 2 and the control device 3. In this embodiment, the communication device 133 may receive (i.e., acquire) a robot control signal from the control device 3.

[0114] The input device 134 is a device that accepts information input to the robot control device 13 from outside the robot control device 13. For example, the input device 134 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by a user of the robot control device 13. For example, the input device 134 may include a recording medium reader that can read information recorded as data on a recording medium that can be externally attached to the robot control device 13.

[0115] It should be noted that information can be input as data to the robot control device 13 from a device external to the robot control device 13 via the communication device 133. In this case, the communication device 133 may function as an input device that accepts information input to the robot control device 13 from outside the robot control device 13.

[0116] The output device 135 is a device that outputs information to the outside of the robot control device 13. For example, the output device 135 may output information as an image. That is, the output device 135 may include a display device (a so-called display) capable of displaying an image. For example, the output device 135 may output information as sound. That is, the output device 135 may include an audio device (a so-called speaker) capable of outputting sound. For example, the output device 135 may output information on paper. That is, the output device 135 may include a printing device (a so-called printer) capable of printing desired information on paper. For example, the output device 135 may output information as data to a recording medium that can be attached externally to the robot control device 13.

[0117] The robot control device 13 can output information as data to a device external to the robot control device 13 via the communication device 133. In this case, the communication device 133 may function as an output device that outputs information to a device external to the robot control device 13.

[0118] (2) Robot Control Processing Next, the robot control processing performed by the control device 3 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the flow of the robot control processing.

[0119] (2-1) Generation of Robot Control Signals for Performing Predetermined Processing on the Target Object OBJ As shown in FIG. 5 , the position and orientation calculation unit 311 included in the control device 3 acquires image data IMG from the imaging device 21 using the communication device 33 (step S11). Specifically, the imaging device 21 captures an image of the target object OBJ at a predetermined imaging rate. For example, the imaging device 21 may capture an image of the target object OBJ at an imaging rate of tens to hundreds of times (e.g., 500 times) per second. Note that, when the target object OBJ is contained in a container CB, capturing an image of the target object OBJ may be considered equivalent to capturing an image of the interior of the container CB (i.e., the above-mentioned storage space SP). The image data IMG resulting from capturing the image of the target object OBJ may be considered equivalent to the image data IMG resulting from capturing an image of the interior of the container CB (i.e., the above-mentioned storage space SP). As a result, the imaging device 21 generates image data IMG at a period corresponding to the predetermined imaging rate. For example, the imaging device 21 may generate tens to hundreds (for example, 500) of image data IMG per second. The position and orientation calculation unit 311 acquires image data IMG every time the imaging device 21 generates image data IMG. In other words, the position and orientation calculation unit 311 may acquire tens to hundreds (for example, 500) of image data IMG per second.

[0120] However, the imaging device 21 does not have to periodically capture images of the target object OBJ at a desired imaging rate. For example, the imaging device 21 may capture an image of the target object OBJ when receiving a control signal from the control device 3 to control the imaging device 21 to capture an image of the target object OBJ.

[0121] The imaging device 21 may capture images of objects other than the target object OBJ, in addition to the target object OBJ on which the robot 1 performs a predetermined process. Note that, because the other objects are not targets for the robot 1 to perform the predetermined process, the other objects other than the target object OBJ are referred to as non-target objects in the following description. An example of a non-target object is at least a portion of the end effector 4. An example of a non-target object is at least a portion of the robot arm 12. An example of a non-target object is at least a portion of a peripheral object, which is an object located around the robot 1. An example of a peripheral object is the mounting device T (in other words, a mounting member, for example, a container CB). For example, if both the target object OBJ and the non-target object are included in the imaging range (field of view) of the imaging device 21, the imaging device 21 may capture images of both the target object OBJ and the non-target object. As a result, the imaging device 21 may generate image data IMG representing an image in which both the target object OBJ and the non-target object are captured. However, the imaging device 21 may capture an image of the target object OBJ without capturing an image of a non-target object. That is, the imaging device 21 may generate image data IMG representing an image in which the target object OBJ is captured but the non-target object is not captured. In either case, the imaging device 21 generates image data IMG representing an image in which at least the target object OBJ is captured. That is, the imaging device 21 generates image data IMG including image data of at least the target object OBJ.

[0122] As described above, when a plurality of target objects OBJ are contained in the container CB, the image capturing device 21 may capture an image of at least some of the plurality of target objects OBJ contained in the container CB. In other words, the image capturing device 21 may capture an image of a group of target objects including at least some of the plurality of target objects OBJ contained in the container CB. The group of target objects captured by the image capturing device 21 may include a plurality of target objects OBJ contained in the container CB that are included in the imaging field of view of the image capturing device 21 (or, in some cases, one target object OBJ included in the imaging field of view of the image capturing device 21).

[0123] As described above, the imaging device 21 may capture images of multiple target objects OBJ on which the end effector 4 sequentially performs a predetermined process. In this case, the end effector 4 may perform the predetermined process on a first target object OBJ among the multiple target objects OBJ during a first period, and may perform the predetermined process on a second target object OBJ different from the first target object OBJ among the multiple target objects OBJ during a second period following the first period. In this case, during the first period, among the multiple target objects OBJ, target objects OBJ other than the first target object OBJ on which the end effector 4 performs the predetermined process (e.g., the second target object OBJ) may be considered to be target objects OBJ that are not processing execution objects on which the robot 1 performs the predetermined process during the first period (or non-target objects). Similarly, during the second period, among the multiple target objects OBJ, target objects OBJ other than the second target object OBJ for which the end effector 4 performs a predetermined process (e.g., the first target object OBJ) may be considered to be target objects OBJ (or non-target objects) that are not processing execution objects for which the robot 1 performs a predetermined process during the second period.

[0124] Every time the position and orientation calculation unit 311 acquires image data IMG in step S11, the position and orientation calculation unit 311 calculates at least one of the position and orientation of the target object OBJ based on the image data IMG acquired in step S121 (step S12). As a result, the position and orientation calculation unit 311 generates position and orientation data POI that indicates at least one of the position and orientation of the target object OBJ (step S12).

[0125] In the following, as described above, an example will be described in which the position and orientation calculation unit 311 calculates at least one of the position and orientation of the target object OBJ in the global coordinate system in step S12. That is, in the following, an example will be described in which the position and orientation calculation unit 311 generates position and orientation data POI that indicates at least one of the position and orientation of the target object OBJ in the global coordinate system.

[0126] The position and orientation calculation unit 311 may calculate, as the position of the target object OBJ in the global coordinate system, at least one of a position Tx of the target object OBJ in an X-axis direction parallel to the X-axis of the global coordinate system, a position Ty of the target object OBJ in a Y-axis direction parallel to the Y-axis of the global coordinate system, and a position Tz of the target object OBJ in a Z-axis direction parallel to the Z-axis of the global coordinate system. The position and orientation calculation unit 311 may calculate, as the orientation of the target object OBJ in the global coordinate system, at least one of a rotation amount Rx of the target object OBJ about the X-axis of the global coordinate system, a rotation amount Ry of the target object OBJ about the Y-axis of the global coordinate system, and a rotation amount Rz of the target object OBJ about the Z-axis of the global coordinate system. This is because the amount of rotation Rx of the target object OBJ around the X-axis, the amount of rotation Ry of the target object OBJ around the Y-axis, and the amount of rotation Rz of the target object OBJ around the Z-axis are equivalent to the parameters representing the orientation of the target object OBJ around the X-axis, the parameters representing the orientation of the target object OBJ around the Y-axis, and the parameters representing the orientation of the target object OBJ around the Z-axis, respectively. For this reason, in the following description, the amount of rotation Rx of the target object OBJ around the X-axis, the amount of rotation Ry of the target object OBJ around the Y-axis, and the amount of rotation Rz of the target object OBJ around the Z-axis will be referred to as the orientation Rx of the target object OBJ around the X-axis, the orientation Ry of the target object OBJ around the Y-axis, and the orientation Rz of the target object OBJ around the Z-axis, respectively.

[0127] The orientation Rx of the target object OBJ around the X-axis, the orientation Ry of the target object OBJ around the Y-axis, and the orientation Rz of the target object OBJ around the Z-axis may be considered to indicate the position of the target object OBJ in the rotation direction around the X-axis, the position of the target object OBJ in the rotation direction around the Y-axis, and the position of the target object OBJ in the rotation direction around the Z-axis, respectively. In other words, the orientation Rx of the target object OBJ around the X-axis, the orientation Ry of the target object OBJ around the Y-axis, and the orientation Rz of the target object OBJ around the Z-axis may all be considered to be parameters representing the position of the target object OBJ.

[0128] 5, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a matching process using the image data IMG acquired in step S11. Specifically, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a matching process using the image data IMG and a template model that indicates the shape of at least a part of the target object OBJ.

[0129] For example, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a 2D matching process, which is an example of a matching process. Specifically, to perform the 2D matching process, the position and orientation calculation unit 311 may use image data IMG generated by the imaging device 21 including a monocular camera (i.e., image data IMG including one image generated by the monocular camera, which may hereinafter be referred to as 2D image data IMG_2D as necessary). In this case, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a 2D matching process using the 2D image data IMG_2D (i.e., an image indicated by the 2D image data IMG_2D) and a template model indicating the two-dimensional shape of at least a portion of the target object OBJ. In this case, the position and orientation calculation unit 311 may perform, as the 2D matching process, an object detection process that detects the target object OBJ indicated by the template model (in other words, an image portion corresponding to the template model) within the image indicated by the 2D image data IMG_2D. The 2D matching process (in this case, the object detection process) itself may be the same as an existing 2D matching process. For example, the position and orientation calculation unit 311 may perform the 2D matching process using a well-known method such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed-Up Robust Feature).

[0130] In this case, the position and orientation calculation unit 311 may translate, enlarge, reduce, and / or rotate the template model in an imaging coordinate system based on the imaging device 21 so that characteristic features of the two-dimensional template model (e.g., at least one of feature points and edges) approach (e.g., coincide with) characteristic features of the target object OBJ captured in the image represented by the 2D image data IMG_2D. As a result, the position and orientation calculation unit 311 can specify the positional relationship between the coordinate system of the template model and the imaging coordinate system. Thereafter, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ in the imaging coordinate system from at least one of the position and orientation of the target object OBJ in the coordinate system of the template model based on the positional relationship between the coordinate system of the template model and the imaging coordinate system. Thereafter, the position and orientation calculation unit 311 may convert at least one of the position and orientation of the target object OBJ in the imaging coordinate system into at least one of the position and orientation of the target object OBJ in the global coordinate system based on a transformation matrix for converting three-dimensional coordinates in the imaging coordinate system into three-dimensional coordinates in either of the other global coordinate systems.

[0131] The template model is not limited to a model showing at least a part of the two-dimensional shape of the target object OBJ, but may be a model showing at least a part of the edge of the target object OBJ. The template model may be generated in advance from a CAD model of the target object, or may be generated from image data obtained by capturing an image of the reference target object OBJ with the imaging device 21 or another imaging device.

[0132] The position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ based on the 2D image data IMG_2D using an existing method other than the 2D matching process. For example, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ based on the 2D image data IMG_2D using an inference model generated by machine learning. The inference model may be generated by machine learning so as to output at least one of the position and orientation of the target object OBJ appearing in the image indicated by the image data IMG when the 2D image data IMG_2D is input.

[0133] For example, if the imaging device 21 includes a stereo camera, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing 3D matching processing, which is an example of matching processing, instead of 2D matching processing. Specifically, to perform the 3D matching processing, the position and orientation calculation unit 311 may use image data IMG generated by the imaging device 21 including a stereo camera (i.e., image data IMG including two images generated by the stereo camera, which may hereinafter be referred to as 3D image data IMG_3D as necessary). In this case, the position and orientation calculation unit 311 may generate three-dimensional position data indicating the three-dimensional position of at least a part of the target object OBJ reflected in the image indicated by the 3D image data IMG_3D, based on the 3D image data IMG_3D (i.e., the two images indicated by the 3D image data IMG_3D). In this case, the position and orientation calculation unit 311 may calculate the parallax based on the 3D image data IMG_3D (i.e., the two images indicated by the 3D image data IMG_3D) and generate three-dimensional position data using the calculated parallax using a well-known method based on the principle of triangulation. The three-dimensional position data may be data indicating the three-dimensional position of each of multiple points on the target object OBJ. The three-dimensional position data may be data indicating the three-dimensional position of each of multiple points on the surface of the target object OBJ. The three-dimensional position data may be data indicating the three-dimensional position of each of multiple points corresponding to multiple parts on the surface of the target object OBJ. In the following description, an example will be described in which point cloud data indicating a point cloud is used as the three-dimensional position data. However, three-dimensional position data other than point cloud data indicating a point cloud (e.g., depth image data indicating a depth image) may also be used. Thereafter, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a 3D matching process using the three-dimensional position data (e.g., point cloud data) and a template model that indicates the three-dimensional shape of at least a part of the target object OBJ. In this case, the position and orientation calculation unit 311 may perform, as the 3D matching process, an object detection process that detects the target object OBJ indicated by the three-dimensional template model (in other words, a set of points corresponding to the template model) within the point cloud indicated by the three-dimensional position data.The 3D matching process (in this case, the object detection process) itself may be the same as an existing 3D matching process. For example, the position and orientation calculation unit 311 may perform the 3D matching process using a well-known method including at least one of RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), ICP (Iterative Closest Point), and DSO (Direct Sparse Odometry).

[0134] In addition, the template model showing the three-dimensional shape of at least a part of the target object OBJ may be generated in advance from a CAD model of the target object OBJ, or may be generated from image data obtained by capturing an image of the reference target object OBJ using the imaging device 21 or another imaging device.

[0135] The position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ based on the image data IMG (i.e., the two images indicated by the image data IMG) using an existing method other than 3D matching processing. For example, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ based on the 3D image data IMG_3D using an inference model generated by machine learning. The inference model may be generated by machine learning so as to output at least one of the position and orientation of the target object OBJ appearing in the images indicated by the image data IMG when the 3D image data IMG_3D is input.

[0136] Note that even if the imaging device 21 includes a monocular camera (in other words, an imaging device 21 that does not include a stereo camera), when the imaging device 21 captures an image of a target object OBJ onto which a projection pattern is projected by the lighting device 23, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing 3D matching processing, which is an example of matching processing, in addition to or instead of 2D matching processing. This is because the position and orientation calculation unit 311 can calculate a parallax that can be used to generate three-dimensional position data based on the projection pattern (specifically, the projection pattern deformed according to the shape of the target object OBJ) captured in the image indicated by the 2D image data IMG_2D.

[0137] When 3D matching processing is performed, the position and orientation calculation unit 311 may translate, enlarge, reduce, and / or rotate the template model in the imaging coordinate system so that characteristic locations (e.g., at least one of feature points and edges) of the three-dimensional template model approach (e.g., coincide with) characteristic locations of the target object OBJ whose three-dimensional position data indicates (e.g., a point cloud corresponding to the target object OBJ indicated by the three-dimensional position data). As a result, the position and orientation calculation unit 311 can identify the positional relationship between the coordinate system of the template model and the imaging coordinate system. Thereafter, even when 3D matching processing is performed, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ in the global coordinate system by performing processing similar to that when 2D matching processing is performed.

[0138] The position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing both 2D matching processing and 3D matching processing. For example, the position and orientation calculation unit 311 may calculate at least a portion of the position and orientation of the target object OBJ by performing 2D matching processing, and calculate at least another portion of the position and orientation of the target object OBJ by performing 3D matching processing. For example, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by merging the calculation result of at least one of the position and orientation of the target object OBJ obtained by the 2D matching processing and the calculation result of at least one of the position and orientation of the target object OBJ obtained by the 3D matching processing.

[0139] When performing the matching process, the position and orientation calculation unit 311 may calculate a matching similarity, which is the similarity between the template model and a target object OBJ that appears in an image indicated by the image data IMG (or a target object OBJ whose three-dimensional position is indicated by three-dimensional position data generated from the image data IMG). When a target object OBJ whose calculated matching similarity exceeds a predetermined matching determination threshold used in the matching process is detected by the matching process, the position and orientation calculation unit 311 may select the target object OBJ as a processing execution object on which the end effector 4 should actually perform a predetermined process. On the other hand, when a target object OBJ whose calculated matching similarity is below the matching determination threshold is detected by the matching process, the position and orientation calculation unit 311 does not have to select the target object OBJ as a processing execution object on which the end effector 4 should actually perform a predetermined process.

[0140] The matching determination threshold is a threshold used for detecting a target object OBJ from an image represented by image data IMG through a matching process. Specifically, the matching determination threshold is a threshold used for distinguishing, from the matching similarity of the object detected from the image represented by image data IMG, a state in which the object detected from the image represented by image data IMG is highly likely to be the target object OBJ and a state in which the object detected from the image represented by image data IMG is low likely to be the target object OBJ. The matching determination threshold can also be said to be a threshold used for distinguishing, from the matching similarity, a state in which the accuracy of at least one of the position and orientation of the target object OBJ calculated through the matching process using the image data IMG is high and a state in which the accuracy of at least one of the position and orientation of the target object OBJ calculated through the matching process using the image data IMG is low.

[0141] As described above, the imaging device 21 may capture images of multiple target objects OBJ. In this case, multiple target objects OBJ may appear in the image represented by the image data IMG generated by the imaging device 21. In this case, the position and orientation calculation unit 311 may select one of the multiple target objects OBJ reflected in the image represented by the image data IMG as a processing execution object on which the end effector 4 should actually perform a predetermined process. For example, the position and orientation calculation unit 311 may select one of the multiple target objects OBJ reflected in the image represented by the image data IMG as a processing execution object on which the end effector 4 should actually perform a predetermined process, based on the matching similarity. As an example, the position and orientation calculation unit 311 may select one of the multiple target objects OBJ whose matching similarity is greater than a matching threshold and is the highest, as the processing execution object. As another example, the position and orientation calculation unit 311 may select, as the object to be processed, one target object OBJ among the multiple target objects OBJ whose matching similarity exceeds the matching determination threshold and whose matching similarity is the Nth largest (N is a constant indicating an integer greater than or equal to 2). As another example, the position and orientation calculation unit 311 may select, as the object to be processed, one target object OBJ among the multiple target objects OBJ whose matching similarity exceeds a predetermined matching determination threshold. As another example, the position and orientation calculation unit 311 may select, as the object to be processed, one target object OBJ among the multiple target objects OBJ whose matching similarity exceeds the matching determination threshold and is closest to the end effector 4. As another example, when a plurality of target objects OBJ are contained in a container CB (mounting device T) (for example, when a plurality of objects are stacked in bulk), the position and orientation calculation unit 311 may select, as the object to be processed, one target object OBJ from among the plurality of target objects OBJ that corresponds to a matching similarity that exceeds the matching judgment threshold and has the largest Z coordinate along the Z axis (is located at the highest position).As another example, the position and orientation calculation unit 312 may select, as the processing execution object, one target object OBJ from among multiple target objects OBJ that corresponds to a matching similarity that exceeds the matching judgment threshold and that the end effector 4 can perform a specified processing on.

[0142] However, if the end effector 4 is capable of simultaneously performing a predetermined process on two or more target objects OBJ, the position and orientation calculation unit 311 may select at least two of the multiple target objects OBJ as the process execution objects. In other words, the position and orientation calculation unit 311 may select at least two process execution objects on which the end effector 4 should simultaneously perform a predetermined process. For convenience of explanation, the following description will be given of an example in which the position and orientation calculation unit 311 selects one target object OBJ of the multiple target objects OBJ as the process execution object.

[0143] Thereafter, the signal generating unit 312 generates a robot control signal using the position and orientation data POI generated in step S12 (i.e., the calculation result of at least one of the position and orientation of the processing execution object, i.e., the result of the matching process) (step S13). For example, the signal generating unit 312 may generate a robot control signal so that the end effector 4 can perform a predetermined process on the target object OBJ (particularly, the processing execution object). For example, the signal generating unit 312 may generate a robot control signal so that the end effector 4 approaches the target object OBJ (particularly, the processing execution object) in order to perform a predetermined process on the target object OBJ (particularly, the processing execution object). For example, the signal generating unit 312 may generate a robot control signal so that the positional relationship between the end effector 4 and the target object OBJ (particularly, the processing execution object) becomes a desired holding target positional relationship. For example, the signal generating unit 312 may generate a robot control signal for controlling the operation of the robot arm 12 so that the positional relationship between the end effector 4 and the target object OBJ (particularly, the processing execution object) becomes a target positional relationship. For example, the signal generating unit 312 may generate a robot control signal for controlling the operation of the robot arm 12 so that the posture relationship between the end effector 4 and the target object OBJ (particularly, the processing execution object) becomes a desired target posture relationship. For example, the signal generating unit 312 may generate a robot control signal for controlling the operation of the robot arm 12 so that the posture relationship between the end effector 4 and the target object OBJ (particularly, the processing execution object) becomes a target posture relationship.

[0144] In this case, the holding target positional relationship and the holding target posture relationship may each be set (registered) in advance. The signal generating unit 312 may calculate at least one of the holding target position and holding target posture of the end effector 4 described above based on the previously set holding target positional relationship and holding target posture relationship and the position and posture data POI generated in step S12 so that the positional relationship between the end effector 4 and the target object OBJ (particularly, the processing execution object) becomes the holding target positional relationship and / or the posture relationship between the end effector 4 and the target object OBJ (particularly, the processing execution object) becomes the holding target posture relationship. At least one of the holding target position and holding target posture of the end effector 4 may mean at least one of the target position and target posture of a reference point (e.g., a tool center point) of the end effector 4. Then, the signal generating unit 312 may generate a robot control signal so that the position of the end effector 4 approaches the holding target position and / or the posture of the end effector 4 approaches the holding target posture. As a result, the end effector 4 approaches the target object OBJ (particularly, the processing execution object).

[0145] While the end effector 4 is approaching the target object OBJ (particularly, the processing execution object), the imaging system 2 may again capture an image of the target object OBJ (particularly, the processing execution object), and the position and orientation calculation unit 311 may again acquire image data IMG generated by the imaging system 2 again capturing an image of the target object OBJ (particularly, the processing execution object) (step S11 in FIG. 5 ). The position and orientation calculation unit 311 may again generate position and orientation data POI based on the reacquired image data IMG (step S12 in FIG. 5 ). That is, the position and orientation calculation unit 311 may update at least one of the position and orientation of the target object OBJ (particularly, the processing execution object). In other words, the position and orientation calculation unit 311 may update the position and orientation data POI. Thereafter, the signal generation unit 312 may again generate a robot control signal based on the updated position and orientation data POI. That is, the signal generation unit 312 may update the robot control signal based on the updated position and orientation data POI. That is, while the end effector 4 is approaching the target object OBJ (particularly, the processing execution object), the control device 3 may repeat the processing from step S11 to step S13 in FIG.

[0146] Thereafter, the signal generating unit 312 may generate a robot control signal so that the end effector 4 performs a predetermined process on the target object OBJ (particularly, the process execution object) when the positional relationship between the end effector 4 and the target object OBJ (particularly, the process execution object) becomes a desired positional relationship. For example, the signal generating unit 312 may generate a robot control signal for controlling the operation of the end effector 4 so that a predetermined process is performed on the target object OBJ (particularly, the process execution object) when the positional relationship between the end effector 4 and the target object OBJ (particularly, the process execution object) becomes a desired positional relationship.

[0147] Thereafter, the signal generator 312 outputs the robot control signal generated in step S13 to the robot 1 (particularly, the robot control device 13) using the communication device 33. As a result, the robot control device 13 controls at least one of the operation of the robot 1 (for example, the operation of the robot arm 12) and the operation of the end effector 4 based on the robot control signal.

[0148] In this embodiment, an example of the predetermined process will be described in which the end effector 4 performs a holding process in which one target object OBJ selected as a process execution object from among multiple target objects OBJ contained in a container CB is held by the end effector 4. In this case, the signal generating unit 312 may generate a robot control signal for controlling at least one of the operation of the robot 1 (e.g., the operation of the robot arm 12) and the operation of the end effector 4 so that the end effector 4 holds the process execution object.

[0149] When performing a holding process, the robot 1 may move the robot arm 12 based on a robot control signal so that the end effector 4 moves toward (i.e., approaches) a holding target position where the end effector 4 can hold the processing execution object. After the end effector 4 reaches the holding target position, the end effector 4 may hold the processing execution object based on the robot control signal.

[0150] When performing a holding process, the robot 1 may move the robot arm 12 based on a robot control signal so that the end effector 4 moves (i.e., moves) until the posture of the end effector 4 becomes a holding target posture that allows the end effector 4 to hold the processing execution object. After the posture of the end effector 4 becomes the holding target posture, the end effector 4 may hold the processing execution object based on the robot control signal.

[0151] When the end effector 4 performs a holding process to hold the target object OBJ, the robot system SYS may be considered to be performing a holding operation to hold the target object OBJ. For example, the robot system SYS may perform a holding process to hold the target object OBJ using the end effector 4 as at least a part of the holding operation. In other words, the holding operation may mainly include a holding process performed by the end effector 4. For example, the robot system SYS may perform a movement process to move the end effector 4 so that the end effector 4 approaches the target object OBJ (particularly, the processing execution object) as at least a part of the holding operation. For example, the robot system SYS may perform a movement process to move the end effector 4 so that the end effector 4 approaches a holding target position as at least a part of the holding operation. For example, the robot system SYS may perform a movement process to move the end effector 4 so that the posture of the end effector 4 becomes the holding target posture as at least a part of the holding operation. That is, the holding operation may include a movement process mainly performed by the robot 1 in addition to the holding process described above.

[0152] After the end effector 4 holds the processing execution object, the end effector 4 may perform, as an example of the predetermined processing, a release process for releasing the processing execution object held by the end effector 4. In other words, by performing the holding process and the release process, the end effector 4 may perform, as an example of the predetermined processing, a placement process (in other words, a placement operation) for placing the target object OBJ.

[0153] When performing the release process, the robot 1 may move the robot arm 12 so that the end effector 4 holding the processing object moves toward (i.e., approaches) a release target position where the end effector 4 can release the processing object. The release target position may be set outside the container CB. In this case, the robot 1 may be considered to be moving the robot arm 12 so that the processing object held by the end effector 4 is transported out of the container CB. Thereafter, after the end effector 4 reaches the release target position, the end effector 4 may release the processing object held by the end effector 4.

[0154] The control device 3 may generate a robot control signal for controlling the robot 1 so that the end effector 4 moves toward the release target position. In this case, the robot control device 13 may move the robot arm 12 based on the robot control signal so that the end effector 4 moves toward the release target position. Alternatively, the robot control device 13 may move the robot arm 12 based on another robot control method such as teaching so that the end effector 4 moves toward the release target position.

[0155] Similarly, the control device 3 may generate a robot control signal for controlling the end effector 4 so that the end effector 4 located at the release target position releases the processing object. In this case, the end effector 4 may release the processing object held by the end effector 4 based on the robot control signal. Alternatively, the end effector 4 may release the processing object held by the end effector 4 based on another robot control method such as teaching.

[0156] (2-2) Processing When the End Effector 4 Holds a Target Object OBJ Different from the Processing-Execution Object Here, as described above, when one of the multiple target objects OBJ contained in the container CB (particularly the multiple target objects OBJ imaged by the imaging device 21 in step S11) is selected as the processing-execution object, the end effector 4 should normally hold the one target object OBJ selected as the processing-execution object. However, in some cases, the end effector 4 may hold at least one target object OBJ that is not selected as the processing-execution object, along with the target object OBJ selected as the processing-execution object. In other words, the end effector 4 may hold at least one target object OBJ that is contained in the container CB and selected as the processing-execution object, along with at least one target object OBJ that is contained in the container CB and not selected as the processing-execution object. In the following explanation, for convenience of explanation, a process execution object to be held by the end effector 4 during a certain period (i.e., a target object OBJ selected as a process execution object) will be referred to as a "process execution object OBJ_tgt," and a target object OBJ held by the end effector 4 together with the process execution object OBJ_tgt during the same period will be referred to as a "non-process execution object OBJ_ntgt." Note that a target object OBJ that becomes the process execution object OBJ_tgt to be held by the end effector 4 during a certain period may be a target object OBJ that is not a process execution object OBJ_tgt, or may be a target object OBJ that becomes a non-process execution object OBJ_ntgt, during a period different from the certain period. Similarly, a target object OBJ that becomes a non-processing object OBJ_ntgt held by the end effector 4 together with a processing object OBJ_tgt during a certain period may be a target object OBJ that is not a non-processing object OBJ_ntgt, or may be a target object OBJ that becomes a processing object OBJ_tgt during another period different from the certain period.

[0157] For example, the end effector 4 may erroneously hold a non-processing object OBJ_ntgt together with a processing object OBJ_tgt. For example, the end effector 4 may unintentionally hold a non-processing object OBJ_ntgt together with a processing object OBJ_tgt. For example, the end effector 4 may unnecessarily (in other words, excessively) hold a non-processing object OBJ_ntgt together with a processing object OBJ_tgt. For example, the end effector 4 may unintentionally hold a non-processing object OBJ_ntgt together with a processing object OBJ_tgt.

[0158] The state in which the end effector 4 holds the non-processing object OBJ_ntgt together with the processing object OBJ_tgt may include a state in which the end effector 4 directly holds the non-processing object OBJ_ntgt.

[0159] The state in which "the end effector 4 directly holds one target object OBJ" may include a state in which the end effector 4 holds one target object OBJ without going through any other object different from the end effector 4 and the one target object OBJ. In this case, the state in which the end effector 4 directly holds the non-processing object OBJ_ntgt may include a state in which the end effector 4 holds the non-processing object OBJ_ntgt without going through any other object different from the non-processing object OBJ_ntgt (for example, the processing object OBJ_tgt).

[0160] The state in which "the end effector 4 directly holds one target object OBJ" may include a state in which the end effector 4 holds one target object OBJ as a result of the holding force of the end effector 4 for holding the target object OBJ acting directly on the one target object OBJ. In this case, the state in which the end effector 4 directly holds the non-processing object OBJ_ntgt may include a state in which the end effector 4 holds the non-processing object OBJ_ntgt as a result of the holding force of the end effector 4 for holding the target object OBJ acting directly on the non-processing object OBJ_ntgt.

[0161] If the end effector 4 is a hand gripper, the holding force of the end effector 4 may include a frictional force acting between the end effector 4 (e.g., the finger members or claw members of the hand gripper) and the target object OBJ in contact with the end effector 4. If the end effector 4 is a vacuum gripper, the holding force of the end effector 4 may include a suction force (in other words, a degree of vacuum) generated by a vacuum device of the end effector 4. If the end effector 4 is a magnetic gripper, the holding force of the end effector 4 may include a magnetic force (or an electromagnetic force) generated by a magnetic attraction device of the end effector 4.

[0162] FIG. 6A shows an example of a state in which the end effector 4 directly holds the non-processing object OBJ_ntgt. In the following description, an example will be described in which the end effector 4 is a magnetic gripper, as shown in FIG. 6A. As shown in FIG. 6A, the state in which the end effector 4 directly holds the non-processing object OBJ_ntgt may include a state in which the end effector 4 directly holds the processing object OBJ_tgt and also directly holds the non-processing object OBJ_ntgt. The state in which the end effector 4 directly holds the non-processing object OBJ_ntgt may include a state in which the end effector 4 directly holds the processing object OBJ_tgt by the direct action of the holding force of the end effector 4 and also directly holds the non-processing object OBJ_ntgt by the direct action of the holding force of the end effector 4.

[0163] Note that, although Figure 6(a) shows an example in which the end effector 4 directly holds one non-processing execution object OBJ_ntgt, the state in which the end effector 4 directly holds a non-processing execution object OBJ_ntgt may also include a state in which the end effector 4 directly holds multiple non-processing execution objects OBJ_ntgt.

[0164] The state in which the end effector 4 holds the non-processing object OBJ_ntgt together with the processing object OBJ_tgt may include a state in which the end effector 4 indirectly holds the non-processing object OBJ_ntgt.

[0165] The state in which "the end effector 4 indirectly holds one target object OBJ" may include a state in which the end effector 4 holds one target object OBJ via another object different from the end effector 4 and the one target object OBJ. In this case, the state in which the end effector 4 indirectly holds the non-processing object OBJ_ntgt may include a state in which the end effector 4 holds the non-processing object OBJ_ntgt via another object different from the non-processing object OBJ_ntgt (for example, the processing object OBJ_tgt).

[0166] The state in which "the end effector 4 indirectly holds one target object OBJ" may include a state in which the end effector 4 holds one target object OBJ with a force different from the holding force of the end effector 4 for holding the target object OBJ. In this case, the state in which the end effector 4 indirectly holds the non-processing object OBJ_ntgt may include a state in which the end effector 4 holds the non-processing object OBJ_ntgt with a force different from the holding force of the end effector 4 for holding the target object OBJ.

[0167] An example of a state in which the end effector 4 indirectly holds the non-processing object OBJ_ntgt is shown in Fig. 6(b). As shown in Fig. 6(b), the state in which the end effector 4 indirectly holds the non-processing object OBJ_ntgt may include a state in which the end effector 4 directly holds the processing object OBJ_tgt and the non-processing object OBJ_ntgt is held by the processing object OBJ_tgt held by the end effector 4. The state in which the end effector 4 indirectly holds the non-processing object OBJ_ntgt may include a state in which the end effector 4 directly holds the processing object OBJ_tgt using the holding force of the end effector 4 and holds the non-processing object OBJ_ntgt by a force different from the holding force of the end effector 4. For example, the state in which the end effector 4 indirectly holds the non-processing execution object OBJ_ntgt may include a state in which the end effector 4 directly holds the processing execution object OBJ_tgt using the holding force of the end effector 4, and the non-processing execution object OBJ_ntgt is held by a frictional force acting between the processing execution object OBJ_tgt held by the end effector 4 and the non-processing execution object OBJ_ntgt.

[0168] Note that Figure 6(b) shows an example in which the end effector 4 indirectly holds one non-processing execution object OBJ_ntgt, but the state in which the end effector 4 indirectly holds a non-processing execution object OBJ_ntgt may include a state in which the end effector 4 indirectly holds multiple non-processing execution objects OBJ_ntgt.

[0169] An example of a state in which the end effector 4 indirectly holds multiple non-processing objects OBJ_ntgt is shown in Fig. 6(c). Fig. 6(c) shows a state in which the end effector 4 indirectly holds two non-processing objects OBJ_ntgt#1 and OBJ_ntgt#2. As shown in Fig. 6(c), the state in which the end effector 4 indirectly holds multiple non-processing objects OBJ_ntgt may include a state in which the end effector 4 directly holds a processing object OBJ_tgt, and the non-processing objects OBJ_ntgt#1 and OBJ_ntgt#2 are held by the processing object OBJ_tgt held by the end effector 4. The state in which the end effector 4 indirectly holds multiple non-processing execution objects OBJ_ntgt may include a state in which the end effector 4 directly holds the processing execution object OBJ_tgt using the holding force of the end effector 4, and also holds the non-processing execution objects OBJ_ntgt#1 and OBJ_tgt#2 using a force different from the holding force of the end effector 4. For example, the state in which the end effector 4 indirectly holds multiple non-processing execution objects OBJ_ntgt may include a state in which the end effector 4 directly holds the processing execution object OBJ_tgt using the holding force of the end effector 4, the non-processing execution object OBJ_ntgt#1 is held by a frictional force acting between the processing execution object OBJ_tgt held by the end effector 4 and the non-processing execution object OBJ_ntgt#1, and the non-processing execution object OBJ_ntgt#2 is held by a frictional force acting between the processing execution object OBJ_tgt held by the end effector 4 and the non-processing execution object OBJ_ntgt#2.

[0170] An example of a state in which the end effector 4 indirectly holds multiple non-processing objects OBJ_ntgt is shown in Fig. 6(d). Fig. 6(d) shows a state in which the end effector 4 indirectly holds two non-processing objects OBJ_ntgt#1 and OBJ_ntgt#2. As shown in Fig. 6(d), the state in which the end effector 4 indirectly holds multiple non-processing objects OBJ_ntgt may include a state in which the end effector 4 directly holds a processing object OBJ_tgt, a non-processing object OBJ_ntgt#1 is held by the processing object OBJ_tgt held by the end effector 4, and a non-processing object OBJ_ntgt#2 is held by the non-processing object OBJ_ntgt#1 held by the processing object OBJ_tgt. The state in which the end effector 4 indirectly holds multiple non-processing execution objects OBJ_ntgt may include a state in which the end effector 4 directly holds the processing execution object OBJ_tgt using the holding force of the end effector 4, and also holds the non-processing execution objects OBJ_ntgt#1 and OBJ_tgt#2 using a force different from the holding force of the end effector 4. For example, a state in which the end effector 4 indirectly holds multiple non-processing execution objects OBJ_ntgt may include a state in which the end effector 4 directly holds the processing execution object OBJ_tgt using the holding force of the end effector 4, and in which the non-processing execution object OBJ_ntgt#1 is held by a frictional force acting between the processing execution object OBJ_tgt held by the end effector 4 and the non-processing execution object OBJ_ntgt#1, and in which the non-processing execution object OBJ_ntgt#2 is held by a frictional force acting between the non-processing execution object OBJ_ntgt#1 and the non-processing execution object OBJ_ntgt#2.

[0171] In addition, the state in which the end effector 4 holds a non-processing execution object OBJ_ntgt together with a processing execution object OBJ_tgt may include a state in which the end effector 4 directly holds at least one non-processing execution object OBJ_ntgt and indirectly holds at least one other non-processing execution object OBJ_ntgt.

[0172] 6(a) to 6(d) show an example in which the end effector 4 holds one process-execution object OBJ_tgt. However, when multiple process-execution objects OBJ_tgt are selected as described above (when the end effector 4 is capable of holding multiple process-execution objects OBJ_tgt), the end effector 4 may hold multiple process-execution objects OBJ_tgt. In this case, the state in which the end effector 4 holds both the process-execution object OBJ_tgt and the non-processing object OBJ_ntgt may include a state in which the end effector 4 directly holds multiple process-execution objects OBJ_tgt and also directly or indirectly holds at least one non-processing object OBJ_ntgt.

[0173] In this manner, when the end effector 4 holds the processing object OBJ_tgt and the non-processing object OBJ_ntgt is held by the end effector 4, the control device 3 may control at least one of the robot 1 and the end effector 4 to return at least the non-processing object OBJ_ntgt to the container CB before the end effector 4 performs a release process to release the processing object OBJ_tgt. That is, the control device 3 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 to return at least the non-processing object OBJ_ntgt to the container CB. In other words, the control device 3 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 to perform a process to return at least the non-processing object OBJ_ntgt to the container CB.

[0174] For example, the control device 3 may control at least one of the robot 1 and the end effector 4 so that at least the non-processing execution object OBJ_ntgt falls into the container CB (as a result, at least the non-processing execution object OBJ_ntgt returns to the container CB). In other words, the control device 3 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to perform a process of dropping at least the non-processing execution object OBJ_ntgt into the container CB as a process of returning at least the non-processing execution object OBJ_ntgt to the container CB. In this case, the non-processing execution object OBJ_ntgt that is dropped into the container CB may be referred to as a dropping target object.

[0175] Furthermore, for example, the control device 3 may control at least one of the robot 1 and the end effector 4 so that at least the non-processing execution object OBJ_ntgt is placed in the container CB without dropping at least the non-processing execution object OBJ_ntgt (as a result, at least the non-processing execution object OBJ_ntgt returns to the container CB). In other words, the control device 3 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to perform a process of placing at least the non-processing execution object OBJ_ntgt in the container CB without dropping at least the non-processing execution object OBJ_ntgt, as the process of returning at least the non-processing execution object OBJ_ntgt to the container CB. In this case, compared to the case where the non-processing execution object OBJ_ntgt falls into the container CB, there is a lower possibility that at least one of the dropped non-processing execution object OBJ_ntgt, the container CB into which the non-processing execution object OBJ_ntgt has fallen, and the target object OBJ contained in the container CB into which the non-processing execution object OBJ_ntgt has fallen will be damaged due to the impact of the drop of the non-processing execution object OBJ_ntgt. Furthermore, compared to the case where the non-processing execution object OBJ_ntgt falls into the container CB, the control device 3 can return at least the non-processing execution object OBJ_ntgt to the container CB more quickly.

[0176] It should be noted that the process of returning the non-processing object OBJ_ntgt to the container CB and the release process of releasing the process object OBJ_tgt may be considered to be common in that they are both processes of separating the target object OBJ held by the end effector 4 from the end effector 4. However, at least the process of returning the non-processing object OBJ_ntgt to the container CB may be considered to be different from the release process, which is a process of separating the target object OBJ held by the end effector 4 from the end effector 4 in order to release the target object OBJ held by the end effector 4 at the above-mentioned release target position, in that the process of returning the non-processing object OBJ_ntgt to the container CB is a process of separating the target object OBJ held by the end effector 4 from the end effector 4 in order to release the target object OBJ held by the end effector 4 at a position different from the above-mentioned release target position. At least the process of returning the non-processed object OBJ_ntgt to the container CB may be considered different from the release process, which is a process of releasing the target object OBJ held by the end effector 4 from the end effector 4 in order to place the target object OBJ held by the end effector 4 at a position different from the container CB, in that the process is a process of releasing the target object OBJ held by the end effector 4 from the end effector 4 in order to place the target object OBJ held by the end effector 4 in the container CB. At least the process of returning the non-processing object OBJ_ntgt to the container CB may be considered different from the release process, which is a process of releasing the target object OBJ held by the end effector 4 from the end effector 4 in order to release the processing execution object OBJ_tgt that the end effector 4 should have held at a desired release target position, in that it is a process of releasing the target object OBJ held by the end effector 4 from the end effector 4 in order to return the non-processing execution object OBJ_ntgt, which the end effector 4 did not originally need to hold, to the container CB. At least the process of returning the non-processing execution object OBJ_ntgt to the container CB may be considered different from the release process, which is a process of partially reversing the steps of the placement process that the robot 1 has progressed (i.e., a series of processes including the holding process and the release process).

[0177] In the following description, for convenience of explanation, an example will be described in which at least the non-processing execution object OBJ_ntgt is dropped into the container CB in order to return at least the non-processing execution object OBJ_ntgt to the container CB. However, even in the case in which at least the non-processing execution object OBJ_ntgt is placed without being dropped into the container CB in order to return at least the non-processing execution object OBJ_ntgt to the container CB, the operation described below may be performed.

[0178] 5, after at least one of the robot 1 and the end effector 4 is controlled based on the robot control signal generated in step S13 to drop at least the non-processing object OBJ_ntgt into the container CB, the control device 3 determines whether the end effector 4 is holding the non-processing object OBJ_ntgt in addition to the processing object OBJ_tgt (steps S21 to S22). That is, when the end effector 4 is indirectly or directly holding at least the non-processing object OBJ_ntgt, after the end effector 4 has held the processing object OBJ_tgt in accordance with the robot control signal generated in step S13 to drop at least the non-processing object OBJ_ntgt into the container CB, the control device 3 determines whether the end effector 4 is holding the non-processing object OBJ_ntgt in addition to the processing object OBJ_tgt (steps S21 to S22). In this case, the control device 3 may determine whether the end effector 4 is holding a non-processing object OBJ_ntgt at the timing when the end effector 4 holds the processing object OBJ_tgt. In other words, the control device 3 may determine whether the end effector 4 is holding a non-processing object OBJ_ntgt at the same time when the end effector 4 holds the processing object OBJ_tgt. Alternatively, the control device 3 may determine whether the end effector 4 is holding a non-processing object OBJ_ntgt after a predetermined time has elapsed since the end effector 4 held the processing object OBJ_tgt. Thereafter, when it is determined that the end effector 4 is holding the non-processing object OBJ_ntgt in addition to the processing object OBJ_tgt, the control device 3 generates a robot control signal for controlling at least one of the robot 1 and the end effector 4 to return at least the non-processing object OBJ_ntgt to the container CB (steps S23 to S24).

[0179] If the end effector 4 is not holding any target object OBJ after at least one of the robot 1 and the end effector 4 has been controlled based on the robot control signal generated in step S13, the control device 3 may determine, as a result of the determination as to whether the end effector 4 is holding a non-processing object OBJ_ntgt in addition to the processing object OBJ_tgt, that the end effector 4 is not holding any target object OBJ. In other words, if the end effector 4 is not holding not only the non-processing object OBJ_ntgt that the end effector 4 should not have held, but also the processing object OBJ_tgt that the end effector 4 should have held, the control device 3 may determine, as a result of the determination as to whether the end effector 4 is holding a non-processing object OBJ_ntgt in addition to the processing object OBJ_tgt, that the end effector 4 is not holding any target object OBJ. In this case, the control device 3 may generate a robot control signal so that the end effector 4 holds the process-execution object OBJ_tgt that the end effector 4 should have held but did not. That is, the control device 3 may generate a robot control signal so that the end effector 4 holds the same process-execution object OBJ_tgt (the same target object OBJ) without changing the process-execution object OBJ_tgt to another target object OBJ. Alternatively, the control device 3 may select, as a new process-execution object OBJ_tgt, a target object OBJ that is different from the process-execution object OBJ_tgt that the end effector 4 should have held but did not. That is, the control device 3 may select, as a new process-execution object OBJ_tgt, a target object OBJ that is different from the previously selected process-execution object OBJ_tgt and whose matching similarity satisfies the above-described condition for selecting a process-execution object OBJ_tgt. Alternatively, if there is no target object OBJ that is different from the selected processing execution object OBJ_tgt and whose matching similarity satisfies the above-mentioned conditions regarding the selection of the processing execution object OBJ_tgt, the control device 3 may generate a robot control signal to move the imaging system 2.For example, the control device 3 may generate a robot control signal for moving the imaging system 2 so as to change at least one of the position and orientation of the imaging system 2. The control device 3 may then acquire image data IMG generated by the moved imaging system 2 capturing an image of the target object OBJ, and select, based on the image data IMG, a target object OBJ whose matching similarity satisfies the above-described conditions for selecting the processing execution object OBJ_tgt as a new processing execution object OBJ_tgt. The control device 3 may then generate a robot control signal for causing the end effector 4 to hold the newly selected processing execution object OBJ_tgt. Note that in this case, the control device 3 does not need to generate a robot control signal for moving the imaging system 2. The control device 3 may acquire image data IMG generated by capturing an image of the target object OBJ at the same position and orientation as the previous time, and select, based on the image data IMG, a target object OBJ whose matching similarity satisfies the above-described conditions for selecting the processing execution object OBJ_tgt as a new processing execution object OBJ_tgt.

[0180] (2-2-1) Determining Whether the End Effector 4 is Holding a Non-Processing Object Specifically, first, after the end effector 4 holds a processing execution object OBJ_tgt in accordance with the robot control signal generated in step S13, the imaging device 21 images at least one target object OBJ held by the end effector 4 (step S21). For example, the imaging device 21 may image an area in which the target object OBJ held by the end effector 4 exists, assuming that the end effector 4 is holding the target object OBJ (e.g., at least one of the processing execution object OBJ_tgt and the non-processing execution object OBJ_ntgt). In other words, the imaging device 21 may image an area in which the target object OBJ held by the end effector 4 exists, assuming that the end effector 4 is holding the target object OBJ, with the imaging range (imaging angle of view) of the imaging device 21 positioned to include the area in which the target object OBJ held by the end effector 4 exists, assuming that the end effector 4 is holding the target object OBJ. However, at the stage when the processing of step S21 is performed, it is not known whether the end effector 4 is actually holding a target object OBJ, but if the end effector 4 is not holding any target objects OBJ at the stage when the processing of step S21 is performed, the imaging device 21 does not need to image the target objects OBJ in step S21. Note that the imaging device 21 that images the target object OBJ held by the end effector 4 may be considered to function as a measuring device that measures the target object OBJ held by the end effector 4.

[0181] Thereafter, the erroneous holding detection unit 313 of the control device 3 acquires, using the communication device 33, image data IMG generated by the imaging device 21 capturing an image of at least one target object OBJ held by the end effector 4 (step S21). Thereafter, the erroneous holding detection unit 313 determines whether or not the end effector 4 is holding a non-processing execution object OBJ_ntgt based on the image data IMG acquired in step S21 (step S22).

[0182] Specifically, the erroneous holding detection unit 313 may control the position and orientation calculation unit 311 to perform a matching process (particularly, an object detection process) based on the image data IMG acquired in step S21, thereby calculating the number of target objects OBJ actually held by the end effector 4. In other words, the erroneous holding detection unit 313 may control the position and orientation calculation unit 311 to perform a matching process (particularly, an object detection process) based on the image data IMG acquired in step S21, thereby calculating the number of target objects OBJ actually held by the end effector 4.

[0183] Alternatively, the erroneous holding detection unit 313 may control the position and orientation calculation unit 311 to perform matching processing (particularly, object detection processing) based on the image data IMG acquired in step S21, thereby calculating at least one of the positions and orientations of the target objects OBJ actually held by the end effector 4. When at least one of the positions and orientations of the target objects OBJ actually held by the end effector 4 is determined, the number of target objects OBJ actually held by the end effector 4 is also necessarily determined. Therefore, the erroneous holding detection unit 313 may calculate the number of target objects OBJ actually held by the end effector 4 based on the calculation results of at least one of the positions and orientations of the target objects OBJ actually held by the end effector 4.

[0184] The matching process performed in step S22 may be the same as the matching process performed in step S12.

[0185] Thereafter, the erroneous holding detection unit 313 determines whether or not the number of target objects OBJ actually held by the end effector 4 is the same as a predetermined number of held objects, which is the number of target objects OBJ that the end effector 4 should originally hold (i.e., the number of process execution objects OBJ_tgt that the end effector 4 should originally hold). For example, as described above, when one target object OBJ out of the multiple target objects OBJ contained in the container CB is selected as the process execution object OBJ_tgt, the erroneous holding detection unit 313 determines whether or not the number of target objects OBJ actually held by the end effector 4 is 1, which is the predetermined number of held objects. For example, as described above, when at least two target objects OBJ out of the multiple target objects OBJ contained in the container CB are selected as the processing execution objects OBJ_tgt, the erroneous holding detection unit 313 determines whether the number of target objects OBJ actually held by the end effector 4 is the same as the predetermined number of held objects, which is the number of target objects OBJ selected as the processing execution objects OBJ_tgt (i.e., an integer greater than or equal to 2).

[0186] When the number of target objects OBJ actually held by the end effector 4 is the same as the predetermined number of held objects, it is unlikely that the end effector 4 is holding both the process object OBJ_tgt and the non-process object OBJ_ntgt. On the other hand, when the number of target objects OBJ actually held by the end effector 4 is greater than the predetermined number of held objects, it is highly likely that the end effector 4 is holding both the process object OBJ_tgt and the non-process object OBJ_ntgt. Therefore, when the number of target objects OBJ actually held by the end effector 4 is the same as the predetermined number of held objects, the erroneous holding detection unit 313 may determine that the end effector 4 is not holding the non-process object OBJ_ntgt. On the other hand, when the number of target objects OBJ actually held by the end effector 4 is greater than the predetermined number of held objects, the erroneous holding detection unit 313 may determine that the end effector 4 is holding the non-process object OBJ_ntgt.

[0187] Here, if the image represented by the image data IMG generated by the imaging device 21 in step S21 includes at least one target object OBJ remaining in the container CB in addition to at least one target object OBJ held by the end effector 4, the matching process (particularly, the object detection process) performed in step S22 may erroneously detect the target object OBJ that remains in the container CB and is therefore not being held by the end effector 4 as the target object OBJ held by the end effector 4. However, even in this case, if the size of the at least one target object OBJ remaining in the container CB in the image represented by the image data IMG generated by the imaging device 21 is smaller than the size of the at least one target object OBJ held by the end effector 4, the matching process (e.g., the object detection process) performed in step S22 is less likely to erroneously detect the target object OBJ that remains in the container CB and is therefore not being held by the end effector 4 as the target object OBJ held by the end effector 4. Furthermore, if at least one target object OBJ remaining in the container CB is blurred in the image represented by the image data IMG generated by the imaging device 21, the matching process performed in step S22 is less likely to erroneously detect a target object OBJ that remains in the container CB and is therefore not being held by the end effector 4 as a target object OBJ being held by the end effector 4. This is because the matching similarity of the at least one target object OBJ remaining in the container CB decreases.

[0188] Therefore, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the imaging device 21 moves away from the container CB (particularly, moves away from the at least one target object OBJ contained in the container CB) from the first camera position, which is the position of the imaging device 21 at the time the end effector 4 holds the process execution object OBJ_tgt, before the imaging device 21 captures an image of the at least one target object OBJ held by the end effector 4 in step S21. In other words, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the imaging device 21, which is located at the first camera position, moves away from the container CB (particularly, moves away from the at least one target object OBJ contained in the container CB) before the imaging device 21 captures an image of the at least one target object OBJ held by the end effector 4 in step S21. 7A and 7B, the signal generator 312 may generate a robot control signal for controlling the robot 1 to move the imaging device 21 to a second camera position that is higher than the first camera position. Note that the height here may refer to the height from the container CB (i.e., the distance to the container CB). That is, the signal generator 312 may generate a robot control signal for controlling the robot 1 to move the imaging device 21 to a second camera position that is farther from the container CB (particularly, farther from at least one target object OBJ contained in the container CB) than the first camera position. Then, based on the robot control signal, the robot 1 may move the robot arm 12 away from the at least one target object OBJ contained in the container CB from the first camera position, which is the position of the imaging device 21 when the end effector 4 holds the processing execution object OBJ_tgt, before the imaging device 21 captures an image of the at least one target object OBJ held by the end effector 4 in step S21.In other words, based on the robot control signal, the robot 1 may move the robot arm 12 so that the imaging device 21, located at the first camera position, moves away from the at least one target object OBJ contained in the container CB before the imaging device 21 captures an image of the at least one target object OBJ held by the end effector 4 in step S21. As an example, the robot 1 may move the robot arm 12 so that the imaging device 21 moves to a second camera position that is higher than the first camera position. After the imaging device 21 moves, the imaging device 21 may capture an image of the at least one target object OBJ held by the end effector 4. As a result, even if the imaging device 21 captures an image of the at least one target object OBJ remaining in the container CB in addition to the at least one target object OBJ held by the end effector 4, the size of the at least one target object OBJ remaining in the container CB will be smaller than the size of the at least one target object OBJ held by the end effector 4 in the image represented by the image data IMG generated by the imaging device 21. Furthermore, even if the imaging device 21 captures an image of at least one target object OBJ remaining in the container CB in addition to at least one target object OBJ held by the end effector 4, there is a high possibility that the at least one target object OBJ remaining in the container CB will be blurred in the image represented by the image data IMG generated by the imaging device 21. As a result, there is a low possibility that the erroneous holding detection unit 313 will erroneously detect a target object OBJ that remains in the container CB and is therefore not being held by the end effector 4 as a target object OBJ being held by the end effector 4. This improves the accuracy of determining whether the end effector 4 is holding a non-processing execution object OBJ_ntgt.

[0189] When the imaging device 21 moves from the first camera position to the second camera position, the end effector 4, which moves together with the imaging device 21, also moves from the first tool position corresponding to the first camera position to the second tool position corresponding to the second camera position. That is, the end effector 4 moves from the first tool position, which is the position of the end effector 4 at the time the end effector 4 holds the processing execution object OBJ_tgt, to move away from the container CB (particularly, to move away from at least one target object OBJ contained in the container CB). In other words, the end effector 4 located at the first tool position moves away from the container CB (particularly, to move away from at least one target object OBJ contained in the container CB). As an example, as shown in FIGS. 7A and 7B, the end effector 4 moves to the second tool position, which is higher than the first tool position. That is, the end effector 4 moves to a second tool position that is farther from the container CB (particularly, farther from at least one target object OBJ contained in the container CB) than the first tool position.

[0190] Furthermore, when the end effector 4 moves from the first tool position to the second tool position, at least one target object OBJ held by the end effector 4 also moves from a first object position corresponding to the first tool position to a second object position corresponding to the second tool position. That is, at least one target object OBJ held by the end effector 4 moves from a first object position, which is the position of the at least one target object OBJ held by the end effector 4 at the time the end effector 4 holds the processing execution object OBJ_tgt, to move away from the container CB (particularly, away from at least one target object OBJ contained in the container CB). In other words, at least one target object OBJ located at the first object position moves away from the container CB (particularly, away from at least one target object OBJ contained in the container CB). As an example, as shown in FIGS. 7A and 7B, at least one target object OBJ held by the end effector 4 moves to a second object position that is higher than the first object position. In other words, at least one target object OBJ held by the end effector 4 moves to a second object position that is farther from the container CB than the first object position (particularly, farther from at least one target object OBJ contained in the container CB).

[0191] However, before the imaging device 21 captures an image of the at least one target object OBJ held by the end effector 4, the imaging device 21 does not have to move away from the at least one target object OBJ contained in the container CB from the first camera position, which is the position of the imaging device 21 at the time when the end effector 4 holds the processing execution object OBJ_tgt. In other words, the imaging device 21 located at the first camera position may capture an image of the at least one target object OBJ held by the end effector 4 from the first camera position.

[0192] In this way, when the imaging device 21 does not need to move away from at least one target object OBJ contained in the container CB before imaging at least one target object OBJ held by the end effector 4 (i.e., does not need to move away from the container CB), the control device 3 can return at least the non-processing execution object OBJ_ntgt to the container CB faster than if at least the non-processing execution object OBJ_ntgt were dropped into the container CB by placing (i.e., returning) at least the non-processing execution object OBJ_ntgt in the container CB without dropping at least the non-processing execution object OBJ_ntgt into the container CB. This is because, if the imaging device 21 does not move away from the container CB, the end effector 4 also does not move away from the container CB, and therefore the end effector 4 holding the non-processing execution object OBJ_ntgt simply releases (let go of) the non-processing execution object OBJ_ntgt at that position (specifically, the first tool position holding the non-processing execution object OBJ_ntgt).

[0193] Note that as the above-described holding process and release process are repeated, the number of target objects OBJ included in the group of target objects contained in the container CB decreases. As a result, the height of the group of target objects contained in the container CB may decrease. In this case, before the imaging device 21 captures an image of the group of target objects contained in the container CB in step S11 of FIG. 5 , the control device 3 may control the height of the imaging device 21 (e.g., the height from the container CB or the distance from the container CB) to match the height of the group of target objects contained in the container CB. For example, the control device 3 may control the height of the imaging device 21 so that the height of the imaging device 21 decreases as the height of the group of target objects contained in the container CB decreases. As a result, the imaging device 21 can properly focus on the group of target objects contained in the container CB even if the height of the group of target objects contained in the container CB decreases. Therefore, the imaging device 21 can properly capture an image of the group of target objects contained in the container CB even if the height of the group of target objects contained in the container CB decreases. In this case, the height at which the imaging device 21 images the at least one target object OBJ held by the end effector 4 during one period may be lower than the height at which the imaging device 21 images the at least one target object OBJ held by the end effector 4 during another period before the one period. In other words, the height at which the imaging device 21 images the at least one target object OBJ held by the end effector 4 during one period when the height of the group of target objects contained in the container CB is a first height may be lower than the height at which the imaging device 21 images the at least one target object OBJ held by the end effector 4 during another period when the height of the group of target objects contained in the container CB is a second height higher than the first height.

[0194] 5, if it is determined in step S22 that the end effector 4 is not holding the non-processing object OBJ_ntgt (step S22: No), the control device 3 may not perform the process of dropping at least the non-processing object OBJ_ntgt into the container CB. In this case, the end effector 4 may perform a release process of releasing the process object OBJ_tgt held by the end effector 4 (step S33). For example, as described above, the process object OBJ_tgt held by the end effector 4 may be carried out of the container CB, and the process object OBJ_tgt held by the end effector 4 may be released outside the container CB.

[0195] On the other hand, if it is determined in step S22 that the end effector 4 is holding a non-processing object OBJ_ntgt (step S22: No), it is assumed that the end effector 4 is holding a group of holding objects OBG_h including both the processing object OBJ_tgt and the non-processing object OBJ_ntgt. In this case, it may be assumed that the imaging device 21 is capturing an image of the group of holding objects OBG_h held by the end effector 4 in the above-mentioned step S21. In this case, the control device 3 performs a process of dropping at least the non-processing object OBJ_ntgt held by the end effector 4 into the container CB (steps S23 to S24).

[0196] In this way, when the end effector 4 holds the non-processing execution object OBJ_ntgt, at least the process of dropping the non-processing execution object OBJ_ntgt held by the end effector 4 into the container CB is performed, while when the end effector 4 does not hold the non-processing execution object OBJ_ntgt, at least the process of dropping the non-processing execution object OBJ_ntgt held by the end effector 4 into the container CB does not have to be performed. Therefore, the control device 3 may be considered to have changed at least one control mode of the robot 1 and the end effector 4 in accordance with the determination result of whether or not the end effector 4 holds the non-processing execution object OBJ_ntgt. Since the number of target objects OBJ actually held by the end effector 4 is used to determine whether or not the end effector 4 holds the non-processing execution object OBJ_ntgt, the control device 3 may be considered to have changed at least one control mode of the robot 1 and the end effector 4 in accordance with the number of target objects OBJ actually held by the end effector 4.

[0197] (2-2-2-1) Process for identifying the holding state In order to perform the process of dropping the non-processing execution object OBJ_ntgt held by the end effector 4 into the container CB, the erroneous holding detection unit 313 may identify the holding state of the target object OBJ (e.g., the group of objects to be held OBG_h, the non-processing execution object OBJ_ntgt, or the processing execution object OBJ_tgt) held by the end effector 4 based on the image data IMG acquired in step S21 (step S23).

[0198] The holding state may include at least one of the position and orientation of the group of holding objects OBG_h held by the end effector 4. Specifically, because the end effector 4 holds the process execution object OBJ_tgt, the holding state may include at least one of the position and orientation of the process execution object OBJ_tgt held by the end effector 4. In other words, at least one of the positions and orientations of the group of holding objects OBG_h may include at least one of the position and orientation of the process execution object OBJ_tgt held by the end effector 4. Furthermore, because the end effector 4 holds the non-processing execution object OBJ_ntgt, the holding state may include at least one of the position and orientation of the non-processing execution object OBJ_ntgt held by the end effector 4 in addition to or instead of at least one of the position and orientation of the process execution object OBJ_tgt held by the end effector 4. In other words, at least one of the position and orientation of the group of objects to be held OBG_h may include at least one of the position and orientation of the non-processing execution object OBJ_ntgt held by the end effector 4 .

[0199] In order to calculate at least one of the position and orientation of the holding target object group OBG_h, the erroneous holding detection unit 313 may control the position and orientation calculation unit 311 to perform matching processing using the image data IMG acquired in step S21. Note that the matching processing performed in step S23 may be the same as the matching processing performed in step S12. Therefore, a detailed description of the matching processing performed in step S23 will be omitted.

[0200] As described above, a matching process may be performed in step S22 to determine whether the end effector 4 is holding the non-processing execution object OBJ_ntgt. In this case, as described above, the erroneous holding detection unit 313 may control the position and orientation calculation unit 311 to calculate at least one of the position and orientation of at least one target object OBJ held by the end effector 4 in step S22. In this case, the erroneous holding detection unit 313 may not need to recalculate at least one of the position and orientation of the held target object group OBG_h in step S23. In step S23, the erroneous holding detection unit 313 may use at least one of the position and orientation of at least one target object OBJ held by the end effector 4 calculated in step S22 as the holding state.

[0201] The holding state may include a holding manner of the non-processing execution object OBJ_ntgt by the end effector 4. Specifically, the holding state may include a state in which the end effector 4 directly holds the non-processing execution object OBJ_ntgt and a state in which the end effector 4 indirectly holds the non-processing execution object OBJ_ntgt. In other words, the holding state may include a determination result as to whether the holding manner of the non-processing execution object OBJ_ntgt by the end effector 4 is a first holding manner in which the end effector 4 directly holds the non-processing execution object OBJ_ntgt, or a second holding manner in which the end effector 4 indirectly holds the non-processing execution object OBJ_ntgt.

[0202] In order to identify the manner in which the non-processing execution object OBJ_ntgt is held by the end effector 4, the erroneous holding detection unit 313 may calculate the positional relationship between the multiple target objects OBJ included in the holding target object group OBG_h based on the calculation results of at least one of the position and orientation of the holding target object group OBG_h. Thereafter, the erroneous holding detection unit 313 may identify the manner in which the non-processing execution object OBJ_ntgt is held by the end effector 4 based on the positional relationship between the multiple target objects OBJ included in the holding target object group OBG_h.

[0203] In order to identify the manner in which the non-processing execution object OBJ_ntgt is being held by the end effector 4, the erroneous holding detection unit 313 may control the position and orientation calculation unit 311 to calculate at least one of the position and orientation of the end effector 4 in addition to at least one of the position and orientation of the holding target object group OBG_h. In this case, the erroneous holding detection unit 313 may calculate at least one of the position and orientation of the end effector 4 by performing a matching process using the image data IMG acquired in step S21 and a template model representing the shape (two-dimensional shape and / or three-dimensional shape) of at least a portion of the end effector 4. Note that the matching process using the template model representing the shape of the end effector 4 may be the same as the matching process using the template model representing the shape of the target object OBJ described above. Note that the template model is not limited to a model representing the shape of at least a portion of the end effector 4, but may also be a model representing at least a portion of the edges of the end effector 4. Furthermore, the template model may be generated in advance from a CAD model of the end effector 4, or may be generated from image data obtained by capturing an image of the reference end effector 4 using the imaging device 21 or another imaging device. At least one of the calculated positions and orientations of the end effector 4 may be at least one of the position and orientation of the tip portion of the end effector 4 (particularly the portion that holds the target object OBJ), or at least one of the position and orientation of a reference point (e.g., a tool center point) of the end effector 4. Alternatively, the erroneous holding detection unit 313 may acquire information regarding at least one of the position and orientation of the end effector 4 from the robot 1 (from the robot control device 13). For example, the erroneous holding detection unit 313 may acquire, as information regarding at least one of the position and orientation of the end effector 4, a measurement result from a position and orientation measurement device (e.g., an encoder) provided in the robot 1 for detecting at least one of the position and orientation of the end effector 4. The information regarding at least one of the position and orientation of the end effector 4 may mean information regarding at least one of the position and orientation of a reference point (e.g., a tool center point) of the end effector 4 .Thereafter, the erroneous holding detection unit 313 may calculate the positional relationship between the group of holding objects OBG_h and the end effector 4 based on at least one of the calculation results of the position and orientation of the group of holding objects OBG_h and at least one of the calculation results of the position and orientation of the end effector 4. Thereafter, the erroneous holding detection unit 313 may identify the holding manner of the non-processing execution object OBJ_ntgt by the end effector 4 based on the positional relationship between the group of holding objects OBG_h and the end effector 4. For example, if there is no target object OBJ located at a position distant from the end effector 4 (see FIG. 6A), there is a low possibility that there is a target object OBJ indirectly held by the end effector 4. Therefore, in this case, the erroneous holding detection unit 313 may determine that the end effector 4 is directly holding the non-processing execution object OBJ_ntgt. For example, when there is a target object OBJ located at a position distant from the end effector 4 (see FIGS. 6(b) to 6(d)), there is a high possibility that there is a target object OBJ indirectly held by the end effector 4. Therefore, in this case, the erroneous holding detection unit 313 may determine that the end effector 4 is indirectly holding the non-processing execution object OBJ_ntgt. For example, when there is a target object OBJ in contact with the end effector 4 (see FIG. 6(a)), there is a high possibility that there is a target object OBJ directly held by the end effector 4. Therefore, in this case, the erroneous holding detection unit 313 may determine that the end effector 4 is directly holding the non-processing execution object OBJ_ntgt.

[0204] The holding state may include "the number of target objects OBJ actually held by the end effector 4" used to determine whether the end effector 4 is holding the non-processing execution object OBJ_ntgt in step S22 described above. In this case, "the number of target objects OBJ actually held by the end effector 4" calculated in step S22 may be used as at least a part of the holding state.

[0205] (2-2-2-2) Generation of robot control signal for dropping at least the non-processing execution object into the container Then, based on the holding state identified in step S23, the signal generation unit 312 generates a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to drop at least the non-processing execution object OBJ_ntgt held by the end effector 4 into the container CB (step S24).

[0206] For example, when the holding state includes a calculation result of at least one of the position and orientation of the holding target object group OBG_h, the signal generation unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 based on the calculation result of at least one of the position and orientation of the non-processing execution object OBJ_ntgt included in the holding target object group OBG_h to drop the non-processing execution object OBJ_ntgt present at the calculated position and / or in the calculated orientation into the container CB. For example, the signal generation unit 312 may generate a robot control signal for moving the robot arm 12 so that the non-processing execution object OBJ_ntgt present at the calculated position and / or in the calculated orientation drops toward the container CB. For example, the signal generation unit 312 may generate a robot control signal for moving the robot arm 12 to move the non-processing execution object OBJ_ntgt present at the calculated position and / or in the calculated orientation to place the non-processing execution object OBJ_ntgt present at the calculated position and / or in the calculated orientation at a predetermined position in the container CB.

[0207] For example, when the holding state includes at least one calculation result of the position and orientation of the holding target object group OBG_h, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 based on at least one calculation result of the position and orientation of the processing execution object OBJ_tgt included in the holding target object group OBG_h to drop into the container CB the processing execution object OBJ_tgt that exists at the calculated position and / or in the calculated orientation and the non-processing execution object OBJ_ntgt held by the end effector 4. For example, the signal generating unit 312 may generate a robot control signal for moving the robot arm 12 so that the non-processing execution object OBJ_ntgt held by the processing execution object OBJ_tgt that exists at the calculated position and / or in the calculated orientation drops toward the container CB. For example, the signal generating unit 312 may generate a robot control signal for moving the robot arm 12 to move the non-processing execution object OBJ_ntgt held by the processing execution object OBJ_tgt, which is present at the calculated position and / or in the calculated posture, in order to place the non-processing execution object OBJ_ntgt at a predetermined position in the container CB.

[0208] For example, when the holding state includes the holding manner of the non-processing execution object OBJ_ntgt by the end effector 4, the signal generating unit 312 may change the control manner of at least one of the robot 1 and the end effector 4 for dropping the non-processing execution object OBJ_ntgt into the container CB in accordance with the holding manner of the non-processing execution object OBJ_ntgt by the end effector 4. In other words, the signal generating unit 312 may generate a robot control signal so that the control manner of at least one of the robot 1 and the end effector 4 for dropping the non-processing execution object OBJ_ntgt into the container CB when the end effector 4 directly holds the non-processing execution object OBJ_ntgt is different from the control manner of at least one of the robot 1 and the end effector 4 for dropping the non-processing execution object OBJ_ntgt into the container CB when the end effector 4 indirectly holds the non-processing execution object OBJ_ntgt. In other words, the signal generating unit 312 may generate a robot control signal so that at least one control mode of the robot 1 and the end effector 4 for dropping the non-processing execution object OBJ_ntgt that is directly held by the end effector 4 into the container CB is different from at least one control mode of the robot 1 and the end effector 4 for dropping the non-processing execution object OBJ_ntgt that is indirectly held by the end effector 4 into the container CB.

[0209] For example, if the holding state includes the number of target objects OBJ actually held by the end effector 4, the signal generating unit 312 may calculate the number of non-processing execution objects OBJ_ntgt to be dropped into the container CB based on the number of target objects OBJ actually held by the end effector 4. For example, the signal generating unit 312 may calculate the number of non-processing execution objects OBJ_ntgt to be dropped into the container CB by subtracting the number of processing execution objects OBJ_tgt that the end effector 4 should originally hold from the number of target objects OBJ actually held by the end effector 4. Then, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 to drop the same number of non-processing execution objects OBJ_ntgt into the container CB as the calculated number.

[0210] (2-2-2-3) Generation of Robot Control Signal for Dropping Non-Processing Execution Object Directly Held by End Effector 4 into Container When the end effector 4 directly holds the non-processing execution object OBJ_ntgt, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to drop both the non-processing execution object OBJ_ntgt held by the end effector 4 and the processing execution object OBJ_tgt held by the end effector 4 into the container CB, as shown in Fig. 8. In other words, when the end effector 4 directly holds the non-processing execution object OBJ_ntgt, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to drop the group of holding target objects OBG_h held by the end effector 4 into the container CB, as shown in Fig. 8.

[0211] The signal generating unit 312 may generate a robot control signal for controlling the end effector 4 so that the end effector 4 releases the group of objects to be held OBG_h by weakening the holding force with which the end effector 4 holds the group of objects to be held OBG_h (as a result, the group of objects to be held OBG_h falls). For example, if the end effector 4 is a hand gripper, the signal generating unit 312 may generate a robot control signal for controlling the end effector 4 so that the end effector 4 releases the group of objects to be held OBG_h by weakening the force with which the hand gripper (e.g., finger members or claw members of the hand gripper) pinches the group of objects to be held OBG_h (as a result, weakening the holding force acting between the hand gripper and the group of objects to be held OBG_h). For example, if the end effector 4 is a vacuum gripper, the signal generating unit 312 may generate a robot control signal for controlling the end effector 4 so that the end effector 4 releases the group of objects to be held OBG_h by weakening the suction force generated by the end effector 4. For example, if the end effector 4 is a magnetic gripper, the signal generating unit 312 may generate a robot control signal for controlling the end effector 4 so that the end effector 4 releases the group of objects to be held OBG_h by weakening the magnetic force (or electromagnetic force) generated by the end effector 4.

[0212] Weakening the holding force of the end effector 4 may include weakening the holding force while the end effector 4 continues to generate the holding force. In other words, weakening the holding force of the end effector 4 may include applying from the end effector 4 to the group of objects to be held OBG_h a holding force that is weaker than the holding force that was acting on the group of objects to be held OBG_h at the time the end effector 4 held the group of objects to be held OBG_h. In other words, weakening the holding force of the end effector 4 may include switching the state of the end effector 4 from a state in which the end effector 4 applies a holding force of a first strength to the group of objects to be held OBG_h to a state in which the end effector 4 applies a holding force of a second strength that is weaker than the first strength to the group of objects to be held OBG_h.

[0213] Weakening the holding force of the end effector 4 may include reducing the holding force of the end effector 4 to zero. In other words, weakening the holding force of the end effector 4 may include switching the state of the end effector 4 from a state in which the end effector 4 applies a holding force to the group of objects to be held OBG_h to a state in which the end effector 4 does not apply a holding force to the group of objects to be held OBG_h.

[0214] The signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the group of objects to be held OBG_h falls from the end effector 4 by moving the end effector 4. For example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the group of objects to be held OBG_h falls from the end effector 4 by translationally moving the end effector 4 along a predetermined translational axis (e.g., at least one of the X-axis, the Y-axis, and the Z-axis). For example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the group of objects to be held OBG_h falls from the end effector 4 by rotationally moving the end effector 4 around a predetermined rotational axis (e.g., at least one of the rotational axis along the X-axis, the Y-axis, and the Z-axis).

[0215] As a first example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the group of objects to be held OBG_h falls from the end effector 4 by vibrating the end effector 4. As a second example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the group of objects to be held OBG_h falls from the end effector 4 by shaking the end effector 4. As a third example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the group of objects to be held OBG_h falls from the end effector 4 due to friction between the group of objects to be held OBG_h and another object (e.g., the container CB and the target object OBJ contained in the container CB) by moving the end effector 4 so that the group of objects to be held OBG_h comes into contact with the other object.

[0216] The signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the holding force with which the end effector 4 holds the group of holding objects OBG_h is weakened and the end effector 4 is moved (for example, by translating and / or rotating as described above) to cause the group of holding objects OBG_h to fall from the end effector 4. Alternatively, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the holding target objects OBG_h is dropped from the end effector 4 by moving the end effector 4 without weakening the holding force with which the end effector 4 holds the group of holding objects OBG_h. For example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the holding target objects OBG_h is dropped from the end effector 4 by moving the end effector 4 while maintaining the holding force with which the end effector 4 holds the group of holding objects OBG_h.

[0217] 9 , when the end effector 4 directly holds the non-processing execution object OBJ_ntgt, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to drop the other of the process execution object OBJ_tgt and the non-processing execution object OBJ_ntgt held by the end effector 4 into the container CB while the end effector 4 holds either the process execution object OBJ_tgt or the non-processing execution object OBJ_ntgt. In other words, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to drop the other of the process execution object OBJ_tgt and the non-processing execution object OBJ_ntgt held by the end effector 4 into the container CB while the end effector 4 holds either the process execution object OBJ_tgt or the non-processing execution object OBJ_ntgt. As one example, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so that, while the end effector 4 is holding the process-execution object OBJ_tgt, the end effector 4 drops the non-processing object OBJ_ntgt held by the end effector 4 into the container CB. In this case, after the non-processing object OBJ_ntgt has dropped into the container CB, the end effector 4 may perform a release process for releasing the process-execution object OBJ_tgt held by the end effector 4 (step S33). As another example, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so that, while the end effector 4 is holding the non-processing object OBJ_ntgt, the end effector 4 drops the process-execution object OBJ_tgt held by the end effector 4 into the container CB. In this case, after the processing execution object OBJ_tgt has fallen into the container CB, the end effector 4 may perform a release process to release the non-processing execution object OBJ_ntgt held by the end effector 4 (step S33).That is, the non-processing object OBJ_ntgt that is still held by the end effector 4 after the processing object OBJ_tgt has fallen into the container CB may be released as a new processing object OBJ_tgt.

[0218] In the following explanation, an example will be described in which the signal generating unit 312 generates a robot control signal for controlling at least one of the robot 1 and the end effector 4 so that the end effector 4 drops the non-processing object OBJ_ntgt held by the end effector 4 into the container CB while the end effector 4 holds the processing object OBJ_tgt.

[0219] The signal generating unit 312 may generate a robot control signal for controlling the end effector 4 so that the end effector 4 releases the non-processing execution object OBJ_ntgt by weakening the holding force with which the end effector 4 holds the non-processing execution object OBJ_ntgt (as a result, the non-processing execution object OBJ_ntgt falls). Note that since weakening the holding force of the end effector 4 has already been explained, detailed explanation thereof will be omitted.

[0220] The signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the non-processing execution object OBJ_ntgt falls from the end effector 4 by moving the end effector 4 (for example, by translating and / or rotating as described above). As a first example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the non-processing execution object OBJ_ntgt falls from the end effector 4 by vibrating the end effector 4. As a second example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the non-processing execution object OBJ_ntgt falls from the end effector 4 by shaking the end effector 4. As a third example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 to move the end effector 4 so that the non-processing execution object OBJ_ntgt comes into contact with another object (e.g., a container CB and a target object OBJ contained in the container CB), thereby causing the non-processing execution object OBJ_ntgt to fall from the end effector 4 due to friction between the non-processing execution object OBJ_ntgt and the other object.

[0221] The signal generating unit 312 may generate a robot control signal for controlling the robot 1 to weaken the holding force with which the end effector 4 holds the non-processing object OBJ_ntgt and move the end effector 4 (for example, by translating and / or rotating as described above) so that the group of objects to be held OBG_h falls from the end effector 4. For example, if the end effector 4 is a magnetic gripper, the holding force (for example, magnetic force) acting on the non-processing object OBJ_ntgt to hold the non-processing object OBJ_ntgt that the end effector 4 should not have held may be weaker than the holding force (for example, magnetic force) acting on the process execution object OBJ_tgt to hold the process execution object OBJ_tgt that the end effector 4 should have held. This is because, while at least one of the relative positional relationship and orientation relationship between the end effector 4 and the non-processing object OBJ_ntgt that the end effector 4 should not have originally held is significantly different from at least one of the ideal positional relationship and orientation relationship for the end effector 4 to properly hold the target object OBJ, at least one of the relative positional relationship and orientation relationship between the end effector 4 and the processing object OBJ_tgt that the end effector 4 should have originally held may become the same as at least one of the ideal positional relationship and orientation relationship for properly holding the target object OBJ by the end effector 4. For this reason, if the holding force of the end effector 4 weakens, there is a high possibility that the non-processing object OBJ_ntgt held by the end effector 4 will fall into the container CB while the end effector 4 is still holding the processing object OBJ_tgt.

[0222] Alternatively, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the end effector 4 moves to cause the non-processing execution object OBJ_ntgt to fall from the end effector 4 without weakening the holding force with which the end effector 4 holds the non-processing execution object OBJ_ntgt. For example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the end effector 4 moves to cause the non-processing execution object OBJ_ntgt to fall from the end effector 4 while maintaining the holding force with which the end effector 4 holds the non-processing execution object OBJ_ntgt.

[0223] 8 and 9 show an example in which the end effector 4 directly holds a single non-processing execution object OBJ_ntgt. However, as described above, the end effector 4 may directly hold multiple non-processing execution objects OBJ_ntgt. In this case, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 to drop all of the multiple non-processing execution objects OBJ_ntgt into the container CB. Alternatively, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 to drop some of the multiple non-processing execution objects OBJ_ntgt into the container CB. In other words, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so that the end effector 4 holds a part of the plurality of non-processing execution objects OBJ_ntgt while dropping another part of the plurality of non-processing execution objects OBJ_ntgt held by the end effector 4 into the container CB.

[0224] 8 and 9 show an example in which the end effector 4 directly holds a single processing execution object OBJ_tgt. However, as described above, the end effector 4 may directly hold multiple processing execution objects OBJ_tgt. In this case, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 to drop all of the processing execution objects OBJ_tgt into the container CB. Alternatively, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 to drop some of the processing execution objects OBJ_tgt into the container CB. In other words, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 to drop other parts of the processing execution objects OBJ_tgt held by the end effector 4 into the container CB while the end effector 4 continues to hold some of the processing execution objects OBJ_tgt.

[0225] (2-2-2-4) Generation of robot control signal for dropping non-processing execution object indirectly held by end effector 4 into container When the end effector 4 indirectly holds the non-processing execution object OBJ_ntgt, as shown in FIG. 10, the signal generation unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to drop the non-processing execution object OBJ_ntgt held by the end effector 4 into the container CB while the end effector 4 continues to hold the processing execution object OBJ_tgt. That is, when the end effector 4 indirectly holds the non-processing execution object OBJ_ntgt, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to drop the non-processing execution object OBJ_ntgt indirectly held by the end effector 4 into the container CB while the end effector 4 holds the process execution object OBJ_tgt, as shown in Fig. 10. In other words, when the end effector 4 indirectly holds the non-processing execution object OBJ_ntgt, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to drop only the non-processing execution object OBJ_ntgt indirectly held by the end effector 4 into the container CB, without dropping the process execution object OBJ_tgt held by the end effector 4 into the container CB, as shown in Fig. 10. In other words, when the end effector 4 indirectly holds the non-processing execution object OBJ_ntgt, as shown in Figure 10, the signal generation unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to drop only the non-processing execution object OBJ_ntgt held by the end effector 4 into the container CB.

[0226] As described above, when the end effector 4 indirectly holds the non-processing object OBJ_ntgt, the non-processing object OBJ_ntgt may be considered to be held by the processing object OBJ_tgt. In this case, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so that the non-processing object OBJ_ntgt drops from the processing object OBJ_tgt into the container CB while the end effector 4 holds the processing object OBJ_tgt. In other words, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so that the non-processing object OBJ_ntgt drops from the processing object OBJ_tgt into the container CB while the end effector 4 holds the processing object OBJ_tgt.

[0227] The signal generating unit 312 may generate a robot control signal for controlling the end effector 4 so that the non-processing object OBJ_ntgt falls by weakening the holding force of the end effector 4 while the end effector 4 is holding the processing object OBJ_tgt. Note that since weakening the holding force of the end effector 4 has already been explained, a detailed explanation thereof will be omitted.

[0228] The signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the non-processing object OBJ_ntgt falls from the processing object OBJ_tgt by moving the end effector 4 (for example, by translating and / or rotating as described above) while the end effector 4 is holding the processing object OBJ_tgt. As a first example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the non-processing object OBJ_ntgt falls from the processing object OBJ_tgt by vibrating the end effector 4. As a second example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the non-processing object OBJ_ntgt falls from the processing object OBJ_tgt by swinging the end effector 4. As a third example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 to move the end effector 4 so that the non-processing execution object OBJ_ntgt comes into contact with another object (e.g., a container CB and a target object OBJ contained in the container CB), thereby causing the non-processing execution object OBJ_ntgt to fall from the processing execution object OBJ_tgt due to friction between the non-processing execution object OBJ_ntgt and the other object.

[0229] The signal generating unit 312 may generate a robot control signal for controlling the robot 1 to weaken the holding force of the end effector 4 and move the end effector 4 (for example, by translating and / or rotating as described above) so that the non-processing object OBJ_ntgt falls from the processing object OBJ_tgt. For example, if the end effector 4 is a magnetic gripper, the holding force (for example, magnetic force) by which the end effector 4 indirectly holds the non-processing object OBJ_tgt is likely to be weaker than the holding force (for example, magnetic force) by which the end effector 4 directly holds the processing object OBJ_tgt. This is because the end effector 4 and the processing object OBJ_tgt are in direct contact with each other, while the end effector 4 and the non-processing object OBJ_ntgt are not in direct contact with each other. Therefore, if the holding force of the end effector 4 weakens, there is a high possibility that the non-processing object OBJ_ntgt held by the end effector 4 will fall into the container CB while the end effector 4 is still holding the processing object OBJ_tgt.

[0230] Alternatively, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the non-processing execution object OBJ_ntgt falls from the processing execution object OBJ_tgt by moving the end effector 4 without weakening the holding force of the end effector 4. For example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the non-processing execution object OBJ_ntgt falls from the processing execution object OBJ_tgt by moving the end effector 4 while maintaining the holding force of the end effector 4.

[0231] 11 , when the non-processing object OBJ_ntgt is dropped into the container CB by moving the end effector 4, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the non-processing object OBJ_ntgt drops from the process execution object OBJ_tgt by rotating the end effector 4. That is, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the non-processing object OBJ_ntgt drops from the process execution object OBJ_tgt by changing the posture of the end effector 4. In this case, compared to when the posture of the end effector 4 does not change, the contact surface CTP where the non-processing object OBJ_ntgt and the process execution object OBJ_tgt come into contact is more likely to be inclined with respect to the horizontal plane. In particular, compared to when the orientation of the end effector 4 does not change, there is a high possibility that the angle at which the contact surface CTP at which the non-processing object OBJ_ntgt and the processing object OBJ_tgt intersect with the horizontal plane (i.e., the tilt angle) will be larger. In this case, compared to when the orientation of the end effector 4 does not change, there is a high possibility that the frictional force acting between the non-processing object OBJ_ntgt and the processing object OBJ_tgt will be weaker. Alternatively, compared to when the orientation of the end effector 4 does not change, in addition to or instead of the frictional force acting between the non-processing object OBJ_ntgt and the processing object OBJ_tgt being weaker, there is a high possibility that a force acting in the opposite direction to the frictional force acting between the non-processing object OBJ_ntgt and the processing object OBJ_tgt (specifically, a force component of gravity acting in the opposite direction to the frictional force acting between the non-processing object OBJ_ntgt and the processing object OBJ_tgt) will be greater. As a result, the non-processing object OBJ_ntgt is more likely to fall off the processing object OBJ_tgt.

[0232] When rotating the end effector 4, as shown in FIG. 12 , the signal generating unit 312 may determine the rotation direction of the end effector 4 based on the positional relationship between the end MFP of the processing execution object OBJ_tgt held by the end effector 4 and the reference point TCP of the end effector 4. The end MFP of the processing execution object OBJ_tgt may be the end of the processing execution object OBJ_tgt that is located farthest from the reference point TCP of the end effector 4. The reference point TCP of the end effector 4 may be a reference point that defines the position of the end effector 4. An example of the reference point TCP is the tool center point of the end effector 4. Another example of the reference point TCP is a point set on the end effector 4 by the user. Note that the tool center point of the end effector 4 may be set by the user.

[0233] Specifically, the signal generating unit 312 may determine the rotation direction RD from the end MFP of the process execution object OBJ_tgt toward the reference point TCP of the end effector 4 as the rotation direction of the reference point TCP of the end effector 4. In other words, the signal generating unit 312 may determine the rotation direction RD from the end MFP of the process execution object OBJ_tgt toward the reference point TCP of the end effector 4 as the rotation direction of the end effector 4. In this case, compared to when a rotation direction different from the rotation direction RD is determined as the rotation direction of the end effector 4, there is a higher possibility that the contact surface CTP where the non-processing object OBJ_ntgt and the process execution object OBJ_tgt come into contact will be inclined with respect to the horizontal plane. In particular, there is a higher possibility that the angle at which the contact surface CTP where the non-processing object OBJ_ntgt and the process execution object OBJ_tgt come into contact with the horizontal plane (i.e., the inclination angle) will be large. In this case, the frictional force acting between the non-processing object OBJ_ntgt and the process execution object OBJ_tgt is likely to be weakened. Alternatively, in addition to or instead of the frictional force acting between the non-processing object OBJ_ntgt and the process execution object OBJ_tgt being weakened, a force in the opposite direction to the frictional force acting between the non-processing object OBJ_ntgt and the process execution object OBJ_tgt (specifically, a force component of gravity acting in the opposite direction to the frictional force acting between the non-processing object OBJ_ntgt and the process execution object OBJ_tgt) is likely to be increased. As a result, the non-processing object OBJ_ntgt is likely to fall off the process execution object OBJ_tgt.

[0234] 12 , when the end effector 4 rotates (i.e., when the posture of the end effector 4 changes), the height of the end effector 4 changes in accordance with the rotation of the end effector 4. Therefore, the rotation of the end effector 4 may include changing the height of the end effector 4 in accordance with the rotation of the end effector 4.

[0235] When rotating the end effector 4, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that a non-processing object OBJ_ntgt that has moved in a direction different from the rotation direction of the end effector 4 (i.e., the direction in which the posture of the end effector 4 is changed) falls from the processing object OBJ_tgt, as shown in Fig. 12. In the example shown in Fig. 12, the rotation direction of the end effector 4 is a rotation direction RD from the end MFP of the processing object OBJ_tgt toward the reference point TCP of the end effector 4. However, even when the rotation direction of the end effector 4 is different from the rotation direction RD from the end MFP of the process execution object OBJ_tgt toward the reference point TCP of the end effector 4, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the non-processing object OBJ_ntgt that has moved in a direction different from the rotation direction of the end effector 4 falls from the process execution object OBJ_tgt. In other words, the signal generating unit 312 may rotate the end effector 4 in a rotation direction different from the rotation direction RD from the end MFP of the process execution object OBJ_tgt toward the reference point TCP of the end effector 4, thereby generating a robot control signal for controlling the robot 1 so that the non-processing object OBJ_ntgt that has moved in a direction different from the rotation direction falls from the process execution object OBJ_tgt.

[0236] For example, as shown in Fig. 12, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that a non-processing object OBJ_ntgt that has moved in a direction opposite to the rotation direction of the end effector 4 falls from the process execution object OBJ_tgt. In the example shown in Fig. 12, the rotation direction of the end effector 4 is a rotation direction toward the -Y side and / or the +Z side. In this case, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that a non-processing object OBJ_ntgt that has moved toward the +Y side and / or the -Z side falls from the process execution object OBJ_tgt.

[0237] However, the signal generating unit 312 does not have to generate a robot control signal for controlling the robot 1 so that the non-processing object OBJ_ntgt that has moved in a direction different from the rotation direction of the end effector 4 falls from the processing object OBJ_tgt. Even in this case, the rotational movement of the end effector 4 may result in the non-processing object OBJ_ntgt that has moved in a direction different from the rotation direction of the end effector 4 falling from the processing object OBJ_tgt.

[0238] When rotating the end effector 4, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that a non-processing object OBJ_ntgt that moves in a direction different from the direction in which the end effector 4 exists falls from the processing object OBJ_tgt, as shown in FIG. 12 . For example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that a non-processing object OBJ_ntgt that moves in a direction opposite to the direction in which the end effector 4 exists falls from the processing object OBJ_tgt, as shown in FIG. 12 . In the example shown in FIG. 12 , the end effector 4 is located on the −Y side and / or +Z side of the non-processing object OBJ_ntgt. In this case, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that a non-processing object OBJ_ntgt that moves in the +Y side and / or −Z side falls from the processing object OBJ_tgt.

[0239] However, the signal generating unit 312 does not have to generate a robot control signal for controlling the robot 1 so that a non-processing object OBJ_ntgt that has moved in a direction different from the direction in which the end effector 4 exists falls from the processing object OBJ_tgt. Even in this case, the rotational movement of the end effector 4 may result in a non-processing object OBJ_ntgt that has moved in a direction different from the direction in which the end effector 4 exists falling from the processing object OBJ_tgt.

[0240] However, when the end effector 4 indirectly holds the non-processing execution object OBJ_ntgt, a robot control signal may be generated to control at least one of the robot 1 and the end effector 4 so as to drop into the container CB both the non-processing execution object OBJ_ntgt indirectly held by the end effector 4 and the processing execution object OBJ_tgt directly held by the end effector 4. In other words, when the end effector 4 indirectly holds the non-processing execution object OBJ_ntgt, the signal generating unit 312 may generate a robot control signal to control at least one of the robot 1 and the end effector 4 so as to drop into the container CB the group of holding target objects OBG_h held by the end effector 4, as shown in FIG.

[0241] The signal generating unit 312 may generate a robot control signal for controlling the end effector 4 so that the end effector 4 releases the process execution object OBJ_tgt by weakening the holding force with which the end effector 4 holds the process execution object OBJ_tgt. As a result, the process execution object OBJ_tgt falls from the end effector 4, and the non-process execution object OBJ_ntgt held by the process execution object OBJ_tgt also falls. Note that since weakening the holding force of the end effector 4 has already been explained, a detailed explanation thereof will be omitted.

[0242] The signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the process execution object OBJ_tgt falls from the end effector 4 by moving (for example, translating and / or rotating) the end effector 4. As a result, the non-processing execution object OBJ_ntgt held by the process execution object OBJ_tgt also falls as the process execution object OBJ_tgt falls from the end effector 4. As a first example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the process execution object OBJ_tgt falls from the end effector 4 by vibrating the end effector 4 (as a result, the non-processing execution object OBJ_ntgt also falls). As a second example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the process execution object OBJ_tgt falls from the end effector 4 by swinging the end effector 4 (as a result, the non-process execution object OBJ_ntgt also falls). As a third example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the process execution object OBJ_tgt falls from the end effector 4 due to friction between the process execution object OBJ_tgt and the other object (as a result, the non-process execution object OBJ_ntgt also falls) by moving the end effector 4 so that the process execution object OBJ_tgt comes into contact with another object (e.g., a container CB and a target object OBJ contained in the container CB).

[0243] The signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the end effector 4 weakens the holding force with which it holds the process-execution object OBJ_tgt and moves the end effector 4 (for example, by translating and / or rotating as described above) to cause the process-execution object OBJ_tgt to fall from the end effector 4. Alternatively, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the end effector 4 moves to cause the process-execution object OBJ_tgt to fall from the end effector 4 without weakening the holding force with which the end effector 4 holds the process-execution object OBJ_tgt. For example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the end effector 4 moves to cause the process-execution object OBJ_tgt to fall from the end effector 4 while maintaining the holding force with which the end effector 4 holds the process-execution object OBJ_tgt.

[0244] The signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the non-processing execution object OBJ_ntgt falls from the end effector 4 by moving the end effector 4 (for example, by translating and / or rotating), and thereafter the process execution object OBJ_tgt falls from the end effector 4. As a first example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the non-processing execution object OBJ_ntgt and the process execution object OBJ_tgt fall from the end effector 4 in this order by vibrating the end effector 4. As a second example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the non-processing execution object OBJ_ntgt and the process execution object OBJ_tgt fall from the end effector 4 in this order by swinging the end effector 4. As a third example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the non-processing object OBJ_ntgt falls from the end effector 4 due to the frictional force between the non-processing object OBJ_ntgt and the other object by moving the end effector 4 so that the non-processing object OBJ_ntgt comes into contact with another object (e.g., a container CB and a target object OBJ contained in the container CB), and then the process execution object OBJ_tgt falls from the end effector 4 due to the frictional force between the process execution object OBJ_tgt and the other object by moving the end effector 4 so that the process execution object OBJ_tgt comes into contact with the other object (e.g., a container CB and a target object OBJ contained in the container CB).

[0245] The signal generating unit 312 may generate a robot control signal for controlling the robot 1 to weaken the holding force with which the end effector 4 holds the non-processing object OBJ_ntgt and move the end effector 4 (for example, by translating and / or rotating as described above) so that the non-processing object OBJ_ntgt falls from the end effector 4, and thereafter the process execution object OBJ_tgt falls from the end effector 4. For example, if the end effector 4 is a magnetic gripper, the holding force (magnetic force) acting on the non-processing object OBJ_ntgt to hold the non-processing object OBJ_ntgt that the end effector 4 should not have held may be weaker than the holding force (magnetic force) acting on the process execution object OBJ_tgt to hold the process execution object OBJ_tgt that the end effector 4 should have held. This is because, while at least one of the relative positional relationship and orientation relationship between the end effector 4 and the non-processing object OBJ_ntgt that the end effector 4 should not have originally held is significantly different from at least one of the ideal positional relationship and orientation relationship for the end effector 4 to properly hold the target object OBJ, at least one of the relative positional relationship and orientation relationship between the end effector 4 and the processing object OBJ_tgt that the end effector 4 should have originally held may become the same as at least one of the ideal positional relationship and orientation relationship for properly holding the target object OBJ by the end effector 4. For this reason, if the holding force of the end effector 4 weakens, there is a high possibility that the non-processing object OBJ_ntgt held by the end effector 4 will fall into the container CB while the end effector 4 is still holding the processing object OBJ_tgt. As a result, there is a high possibility that the non-processing object OBJ_ntgt will fall from the end effector 4 first, and then the processing object OBJ_tgt will fall from the end effector 4 .

[0246] Alternatively, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the end effector 4 is moved without weakening the holding force with which the end effector 4 holds the non-processing execution object OBJ_ntgt, thereby causing the non-processing execution object OBJ_ntgt to fall from the end effector 4, and thereafter the process execution object OBJ_tgt to fall from the end effector 4. For example, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the end effector 4 is moved while maintaining the holding force with which the end effector 4 holds the non-processing execution object OBJ_ntgt, thereby causing the non-processing execution object OBJ_ntgt to fall from the end effector 4, and thereafter the process execution object OBJ_tgt to fall from the end effector 4.

[0247] 10 to 12 show an example in which the end effector 4 indirectly holds a single non-processing execution object OBJ_ntgt. However, as described above, the end effector 4 may indirectly hold multiple non-processing execution objects OBJ_ntgt. In this case, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 to drop all of the multiple non-processing execution objects OBJ_ntgt into the container CB. Alternatively, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 to drop some of the multiple non-processing execution objects OBJ_ntgt into the container CB. In other words, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so that the end effector 4 holds a part of the plurality of non-processing execution objects OBJ_ntgt while dropping another part of the plurality of non-processing execution objects OBJ_ntgt held by the end effector 4 into the container CB.

[0248] When dropping a plurality of non-processing execution objects OBJ_ntgt into the container CB, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to drop the plurality of non-processing execution objects OBJ_ntgt collectively into the container CB. Specifically, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to drop the plurality of non-processing execution objects OBJ_ntgt collectively into the container CB in a manner similar to the method for generating a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to drop a single non-processing execution object OBJ_ntgt into the container CB described above.

[0249] Alternatively, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 to drop the plurality of non-processing execution objects OBJ_ntgt one by one into the container CB in sequence. Specifically, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 to drop one non-processing execution object OBJ_ntgt of the plurality of non-processing execution objects OBJ_ntgt individually into the container CB in a manner similar to the method for generating a robot control signal for controlling at least one of the robot 1 and the end effector 4 to drop a single non-processing execution object OBJ_ntgt into the container CB described above. Furthermore, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 to individually drop another non-processing execution object OBJ_ntgt, which is different from one of the multiple non-processing execution objects OBJ_ntgt, into the container CB in a manner similar to the method for generating a robot control signal for controlling at least one of the robot 1 and the end effector 4 to drop the single non-processing execution object OBJ_ntgt described above into the container CB. As an example, as shown in Figure 6 (d) above, in the case where the end effector 4 directly holds the processing execution object OBJ_tgt, the processing execution object OBJ_tgt holds the non-processing execution object OBJ_ntgt#1, and the non-processing execution object OBJ_ntgt#2 is held by the non-processing execution object OBJ_ntgt#1, the signal generation unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so that the non-processing execution object OBJ_ntgt#2 falls from the non-processing execution object OBJ_ntgt#1, and then the non-processing execution object OBJ_ntgt#1 falls from the processing execution object OBJ_tgt.For example, the signal generating unit 312 may generate a first robot control signal for controlling at least one of the robot 1 and the end effector 4 so that the non-processing execution object OBJ_ntgt#2 drops from the non-processing execution object OBJ_ntgt#1, and a second robot control signal for controlling at least one of the robot 1 and the end effector 4 so that the non-processing execution object OBJ_ntgt#1 drops from the processing execution object OBJ_tgt. Specifically, the signal generating unit 312 may generate a first robot control signal for controlling at least one of the robot 1 and the end effector 4 so that the non-processing execution objects OBJ_ntgt#1 to OBJ_ntgt#2 drop individually into the container CB in a manner similar to the method for generating a robot control signal for controlling at least one of the robot 1 and the end effector 4 so that the single non-processing execution object OBJ_ntgt drops into the container CB described above. Furthermore, the signal generating unit 312 may generate a second robot control signal, separate from the first robot control signal, for controlling at least one of the robot 1 and the end effector 4 to individually drop the non-processing execution object OBJ_ntgt#1 from the processing execution object OBJ_tgt into the container CB in a manner similar to the method for generating a robot control signal for controlling at least one of the robot 1 and the end effector 4 to drop the single non-processing execution object OBJ_ntgt into the container CB described above.

[0250] 10 to 12 show an example in which the end effector 4 indirectly holds a single process-execution object OBJ_tgt. However, as described above, the end effector 4 may directly hold multiple process-execution objects OBJ_tgt. In this case, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to drop the non-processing object OBJ_ntgt into the container CB while holding all of the multiple process-execution objects OBJ_tgt. Alternatively, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to drop another part of the multiple process-execution objects OBJ_tgt held by the end effector 4 into the container CB while the end effector 4 holds some of the multiple process-execution objects OBJ_tgt. Alternatively, the signal generating unit 312 may generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 so as to drop all of the multiple processing execution objects OBJ_tgt held by the end effector 4 into the container CB.

[0251] 5 , thereafter, the signal generating unit 312 outputs the robot control signal generated in step S24 to the robot 1 (particularly, the robot control device 13) using the communication device 33. As a result, the robot control device 13 controls at least one of the operation of the robot 1 (e.g., the operation of the robot arm 12) and the operation of the end effector 4 based on the robot control signal. In other words, the robot control device 13 controls at least one of the operation of the robot 1 (e.g., the operation of the robot arm 12) and the operation of the end effector 4 based on the robot control signal so that at least the non-processing execution object OBJ_ntgt falls into the container CB.

[0252] 7A and 7B, after the end effector 4 holds the group of holding target objects OBG_h in step S13 and before the imaging device 21 captures an image of the group of holding target objects OBG_h in step S21, the imaging device 21, which is located at the first camera position, may move to a second camera position that is higher than the first camera position. Consequently, as described above with reference to FIGS. 7A and 7B, the end effector 4, which is located at the first tool position, may move to a second tool position that is higher than the first tool position. In this case, before control of the operation of the robot 1 is started based on the robot control signal generated in step S24 so that at least the non-processing execution object OBJ_ntgt falls into the container CB, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 to move the end effector 4 from the second tool position toward the container CB (particularly, toward at least one target object OBJ contained in the container CB). In other words, before control of the operation of the robot 1 is started based on the robot control signal generated in step S24 so that at least the non-processing execution object OBJ_ntgt falls into the container CB, the signal generation unit 312 may generate a robot control signal for controlling the robot 1 to move the end effector 4 located at the second tool position closer to the container CB (particularly, closer to at least one target object OBJ contained in the container CB). As an example, as shown in FIG. 7C , the signal generation unit 312 may generate a robot control signal for controlling the robot 1 to move the end effector 4 to a third tool position that is lower than the second tool position. That is, the signal generation unit 312 may generate a robot control signal for controlling the robot 1 to move the end effector 4 to a third tool position that is closer to the container CB (particularly, closer to at least one target object OBJ contained in the container CB) than the second tool position. Thereafter, control of the operation of the robot 1 may be started based on the robot control signal generated in step S24 so that at least the non-processing execution object OBJ_ntgt falls into the container CB.In this case, the drop position from which at least the non-processing execution object OBJ_ntgt starts to drop into the container CB becomes lower. For example, at least the non-processing execution object OBJ_ntgt drops into the container CB from a drop position lower than the drop position (not shown) corresponding to the second tool position (i.e., a drop position corresponding to the third tool position which is lower than the second tool position). As a result, the impact of the non-processing execution object OBJ_ntgt dropping into the container CB becomes smaller. This reduces the possibility of damage to the dropped non-processing execution object OBJ_ntgt or other target objects OBJ onto which the non-processing execution object OBJ_ntgt has fallen.

[0253] 7C, when the end effector 4 moves from the second tool position to the third tool position, the group of objects to be held OBG_h held by the end effector 4 also moves from the second object position corresponding to the second tool position to a third object position corresponding to the third tool position (i.e., the third object position is lower than the second object position and therefore closer to the container CB than the second object position). In this case, at least the non-processing object OBJ_ntgt may be considered to fall into the container CB from the third object position, which is lower than the second object position. Similarly, when the end effector 4 moves from the second tool position to the third tool position, the imaging system 2 moving together with the end effector 4 also moves from the second camera position corresponding to the second tool position to a third camera position corresponding to the third tool position (i.e., the third camera position is lower than the second camera position and therefore closer to the container CB than the second camera position). In this case, it may be assumed that at least the non-processing execution object OBJ_ntgt falls into the container CB from a third camera position that is lower than the second camera position.

[0254] 5 , after control of the operation of the robot 1 is started based on the robot control signal generated in step S24, the erroneous holding detection unit 313 determines whether at least the non-processing execution object OBJ_ntgt has fallen into the container CB (step S31). For example, during the period in which the operation of the robot 1 is controlled based on the robot control signal generated in step S24, the erroneous holding detection unit 313 may determine whether at least the non-processing execution object OBJ_ntgt has fallen into the container CB. For example, after control of the operation of the robot 1 is completed based on the robot control signal generated in step S24, the erroneous holding detection unit 313 may determine whether at least the non-processing execution object OBJ_ntgt has fallen into the container CB.

[0255] In order to determine whether at least the non-processing object OBJ_ntgt has fallen into the container CB, after control of the robot 1's operation is started based on the robot control signal generated in step S24, the imaging device 21 may capture an image of an area where the target object OBJ held by the end effector 4 is located, assuming that the end effector 4 is holding the target object OBJ. For example, the imaging device 21 may capture an image of at least a portion of the end effector 4 (particularly, the portion of the end effector 4 that holds the target object OBJ), thereby capturing an image of an area where the target object OBJ held by the end effector 4 is located, assuming that the end effector 4 is holding the target object OBJ. In this case, if the processing object OBJ_tgt falls from the end effector 4, the imaging device 21 may capture an image of the end effector 4 from which the processing object OBJ_tgt has fallen. In other words, the imaging device 21 may capture an image of the end effector 4 that is not holding the processing object OBJ_tgt. On the other hand, if the process execution object OBJ_tgt has not fallen from the end effector 4, the imaging device 21 may image the process execution object OBJ_tgt held by the end effector 4. If the non-process execution object OBJ_ntgt has fallen from the end effector 4, the imaging device 21 may image the end effector 4 from which the non-process execution object OBJ_ntgt has fallen. In other words, the imaging device 21 may image the end effector 4 that is not holding the non-process execution object OBJ_ntgt. On the other hand, if the non-process execution object OBJ_ntgt has not fallen from the end effector 4, the imaging device 21 may image the non-process execution object OBJ_ntgt held by the end effector 4. Thereafter, the erroneous holding detection unit 313 of the control device 3 may acquire, using the communication device 33, the image data IMG generated by the imaging device 21 capturing an image of the end effector 4.

[0256] Thereafter, the erroneous holding detection unit 313 determines whether or not at least the non-processing execution object OBJ_ntgt has fallen into the container CB based on the image data IMG.

[0257] As a first example of an operation for determining whether at least the non-processing execution object OBJ_ntgt has fallen into the container CB based on the image data IMG, the erroneous holding detection unit 313 may control the position and orientation calculation unit 311 to perform a matching process (e.g., an object detection process) based on the image data IMG, thereby calculating the number of target objects OBJ actually held by the end effector 4. Alternatively, the erroneous holding detection unit 313 may control the position and orientation calculation unit 311 to perform a matching process based on the image data IMG, thereby calculating at least one of the positions and orientations of the target objects OBJ actually held by the end effector 4. In this case, as described above, once at least one of the positions and orientations of the target objects OBJ actually held by the end effector 4 is determined, the number of target objects OBJ actually held by the end effector 4 is also determined. Therefore, the erroneous holding detection unit 313 may calculate the number of target objects OBJ actually held by the end effector 4 based on the calculation results of at least one of the positions and orientations of the target objects OBJ actually held by the end effector 4. The matching process itself performed in step S31 may be the same as the matching process performed in steps S12 and S22. Thereafter, the erroneous holding detection unit 313 may determine whether at least the non-processing execution object OBJ_ntgt has fallen into the container CB based on the number of target objects OBJ actually held by the end effector 4.

[0258] For example, as described above, when at least one of the robot 1 and the end effector 4 is controlled so that the non-processing object OBJ_ntgt falls into the container CB while the end effector 4 holds the processing object OBJ_tgt, the erroneous holding detection unit 313 may determine whether or not the number of target objects OBJ actually held by the end effector 4 is the same as a predetermined number of held objects, which is the number of target objects OBJ that the end effector 4 should originally hold (i.e., the number of processing objects OBJ_tgt that the end effector 4 should originally hold). Note that the process of determining in step S31 whether the number of target objects OBJ actually held by the end effector 4 is the same as the predetermined number of held objects may be the same as the process of determining in step S22 whether the number of target objects OBJ actually held by the end effector 4 is the same as the predetermined number of held objects. In this case, if the number of target objects OBJ actually held by the end effector 4 is the same as the predetermined number of held objects, it is highly likely that the non-processing object OBJ_ntgt held by the end effector 4 has fallen, and the end effector 4 is still holding the processing object OBJ_tgt. On the other hand, if the number of target objects OBJ actually held by the end effector 4 is greater than the predetermined number of held objects, it is highly likely that the end effector 4 is still holding at least one non-processing object OBJ_ntgt. Therefore, if the number of target objects OBJ actually held by the end effector 4 is the same as the predetermined number of held objects, the erroneous holding detection unit 313 may determine that the non-processing object OBJ_ntgt has fallen into the container CB. On the other hand, if the number of target objects OBJ actually held by the end effector 4 is greater than the predetermined number of held objects, the erroneous holding detection unit 313 may determine that at least one non-processing execution object OBJ_ntgt has not fallen into the container CB (specifically, there remains a non-processing execution object OBJ_ntgt that has not fallen into the container CB).

[0259] For example, when at least one of the robot 1 and the end effector 4 is controlled so that both the process execution object OBJ_tgt and the non-process execution object OBJ_ntgt fall into the container CB as described above, the erroneous holding detection unit 313 may determine whether the number of target objects OBJ actually held by the end effector 4 is zero. In this case, if the number of target objects OBJ actually held by the end effector 4 is zero, it is highly likely that both the process execution object OBJ_tgt and the non-process execution object OBJ_ntgt held by the end effector 4 have fallen. On the other hand, if the number of target objects OBJ actually held by the end effector 4 is greater than zero, it is highly likely that the end effector 4 is still holding at least one of the process execution object OBJ_tgt and the non-process execution object OBJ_ntgt. Therefore, when the number of target objects OBJ actually held by the end effector 4 is zero, the erroneous holding detection unit 313 may determine that the process execution object OBJ_tgt and the non-process execution object OBJ_ntgt have fallen into the container CB. On the other hand, when the number of target objects OBJ actually held by the end effector 4 is greater than zero, the erroneous holding detection unit 313 may determine that at least one of the process execution object OBJ_tgt and the non-process execution object OBJ_ntgt has not fallen into the container CB (specifically, at least one of the process execution object OBJ_tgt and the non-process execution object OBJ_ntgt that has not fallen into the container CB remains).

[0260] As a second example of an operation for determining whether at least the non-processing execution object OBJ_ntgt has dropped into the container CB based on the image data IMG, the erroneous holding detection unit 313 may control the position and orientation calculation unit 311 to perform matching processing based on the image data IMG, thereby calculating at least one of the position and orientation of the target object OBJ actually held by the end effector 4. Furthermore, the erroneous holding detection unit 313 may control the position and orientation calculation unit 311 to perform matching processing based on the image data IMG, thereby calculating at least one of the position and orientation of the end effector 4. Note that the matching processing itself for calculating at least one of the position and orientation of the end effector 4 in step S31 may be the same as the matching processing for identifying the holding mode of the non-processing execution object OBJ_ntgt by the end effector 4 in step S23 described above. Thereafter, the erroneous holding detection unit 313 may determine whether at least the non-processing execution object OBJ_ntgt has fallen into the container CB based on at least one of the calculation results of the position and orientation of the target object OBJ actually held by the end effector 4 and at least one of the calculation results of the position and orientation of the end effector 4. For example, in a situation where it is determined in step S23 that the end effector 4 is indirectly holding the non-processing execution object OBJ_ntgt, if the result of the matching process in step S31 indicates that there is no target object OBJ located at a position away from the end effector 4 (see FIG. 6A), it is unlikely that there is a non-processing execution object OBJ_ntgt indirectly held by the end effector 4. Therefore, in this case, the erroneous holding detection unit 313 may determine that at least the non-processing execution object OBJ_ntgt has fallen into the container CB. For example, in a situation where it is determined in step S23 that the end effector 4 is indirectly holding a non-processing execution object OBJ_ntgt, if the result of the matching process in step S31 shows that there is a target object OBJ located at a position away from the end effector 4 (see Figures 6(b) to 6(d)), there is a high possibility that there is a non-processing execution object OBJ_ntgt that is indirectly held by the end effector 4.Therefore, in this case, the erroneous holding detection unit 313 may determine that at least the non-processing execution object OBJ_ntgt has not fallen into the container CB.

[0261] If the result of the judgment in step S31 is that at least the non-processing execution object OBJ_ntgt has not fallen into the container CB (step S31: No), the erroneous holding detection unit 313 may control the output device 35 to output an alert to notify the user (operator) of the robot system SYS that at least the non-processing execution object OBJ_ntgt could not be dropped into the container CB (step S32).

[0262] After the alert is output, the control device 3 may temporarily suspend the robot control process shown in Fig. 5. In this case, the user may manually return the non-processing object OBJ_ntgt that did not fall from the end effector 4 to the container CB. Thereafter, the temporarily suspended robot control process may be resumed. For example, the end effector 4 may perform a release process to release the processing object OBJ_tgt held by the end effector 4 (step S33).

[0263] However, even if it is determined that at least the non-processing execution object OBJ_ntgt has not fallen into the container CB, the erroneous holding detection unit 313 does not need to output an alert to notify the user (operator) of the robot system SYS that at least the non-processing execution object OBJ_ntgt could not be dropped into the container CB.

[0264] On the other hand, if it is determined in step S31 that at least the non-processing object OBJ_ntgt has fallen into the container CB (step S31: Yes), the erroneous holding detection unit 313 does not need to control the output device 35 to output an alert. In this case, the end effector 4 may perform a release process to release the processing object OBJ_tgt held by the end effector 4 (step S33). For example, as described above, the processing object OBJ_tgt held by the end effector 4 may be carried out from the container CB, and the processing object OBJ_tgt held by the end effector 4 may be released outside the container CB.

[0265] However, as described above, when at least one of the robot 1 and the end effector 4 is controlled so that both the processing-execution object OBJ_tgt and the non-processing-execution object OBJ_ntgt fall into the container CB, the end effector 4 does not hold any target objects OBJ. In this case, the control device 3 may repeat the series of processes from step S11 to step S33 (however, the processes of step S32 and step S33 are performed exclusively) without the end effector 4 performing the release process. In other words, the imaging device 21 may newly capture images of some of the multiple target objects OBJ contained in the container CB, and the control device 3 may newly acquire image data IMG from the imaging device 21. In this case, the signal generation unit 312 may generate a robot control signal for controlling the robot 1 so that the imaging device 21 captures images of some of the multiple target objects OBJ contained in the container CB. The robot 1 may move the robot arm 12 so that the imaging device 21 captures images of some of the target objects OBJ contained in the container CB based on the robot control signal generated by the signal generator 312. Alternatively, the robot 1 may move the robot arm 12 so that the imaging device 21 captures images of some of the target objects OBJ contained in the container CB based on another robot control method such as teaching. Thereafter, the control device 3 may perform the series of processes from step S12 to step S33 again (however, the processes of step S32 and step S33 are performed exclusively) using the image data IMG newly acquired in step S11.

[0266] However, when at least one of the robot 1 and the end effector 4 is controlled so that both the process-execution object OBJ_tgt and the non-processing object OBJ_ntgt fall into the container CB, the image capture device 21 does not need to re-image some of the multiple target objects OBJ contained in the container CB. For example, in step S12 described above, if multiple target objects OBJ whose matching similarity exceeds the matching determination threshold have been detected based on the already acquired image data IMG, the position and orientation calculation unit 311 may select one target object OBJ that has not yet been selected as the process-execution object OBJ_tgt from among the multiple target objects OBJ whose matching similarity exceeds the matching determination threshold as a new process-execution object OBJ_tgt. Thereafter, the signal generation unit 312 may generate a robot control signal so as to perform a predetermined process on the newly selected process-execution object OBJ_tgt (step S13). Thereafter, the control device 3 may perform the series of processes from step S21 to step S33 again (however, the process of step S32 and the process of step S33 are performed exclusively).

[0267] After that, after an alert is output (step S32) or the processing execution object OBJ_tgt is released (step S33), the control device 3 repeats the series of processes from step S11 to step S33 (however, the processes of step S32 and step S33 are performed exclusively) until there are no more target objects OBJ in the container CB on which the end effector 4 should perform a predetermined process (step S41). However, if multiple target objects OBJ whose matching similarity exceeds the matching determination threshold have been detected based on the already acquired image data IMG in step S12, the control device 3 may not need to re-acquire image data IMG in step S11. The imaging device 21 may not need to re-image some of the multiple target objects OBJ contained in the container CB in step S11. In this case, the position and orientation calculation unit 311 may select one target object OBJ that has not yet been selected as the processing execution object OBJ_tgt from among the multiple target objects OBJ whose matching similarity exceeds the matching determination threshold as a new processing execution object OBJ_tgt. Thereafter, the signal generation unit 312 may generate a robot control signal so as to perform a predetermined process on the newly selected processing execution object OBJ_tgt (step S13). Thereafter, the control device 3 may again perform the series of processes from step S21 to step S33 (however, the process of step S32 and the process of step S33 are performed exclusively).

[0268] (3) Technical Effects As described above, in this embodiment, when the end effector 4 holds the non-processing object OBJ_ntgt together with the processing object OBJ_tgt, the control device 3 can control at least one of the robot 1 and the end effector 4 so that at least the non-processing object OBJ_ntgt falls into (i.e., returns to) the container CB. Therefore, the control device 3 can switch the state of the end effector 4 from an abnormal state in which the end effector 4 holds the processing object OBJ_tgt and the non-processing object OBJ_ntgt to a normal state in which the end effector 4 holds the processing object OBJ_tgt without holding the non-processing object OBJ_ntgt. Therefore, the control device 3 can control the robot 1 so that the end effector 4 properly holds the processing object OBJ_tgt.

[0269] In addition, if the end effector 4 is holding a non-processing object OBJ_ntgt, when a release process is performed to release the process-executing object OBJ_tgt held by the end effector 4 to a desired position outside the container CB, the non-processing object OBJ_ntgt held by the end effector 4 together with the process-executing object OBJ_tgt may become an obstacle to the release process. As an example, when the process-executing object OBJ_tgt is released to a desired position outside the container CB, the process-executing object OBJ_tgt may interfere with the non-processing object OBJ_ntgt, causing the process-executing object OBJ_tgt to be released to a position different from the desired position. As another example, the non-processing object OBJ_ntgt may collide with an object (e.g., placement device T) from which the process-executing object OBJ_tgt is to be released, potentially damaging the object (e.g., placement device T) from which the process-executing object OBJ_tgt is to be released. However, in this embodiment, such a technical problem does not occur because the non-processing object OBJ_ntgt falls (i.e., returns) into the container CB. Therefore, the end effector 4 can appropriately place the held processing object OBJ_tgt at a desired position outside the container CB.

[0270] In addition, if an operation to drop the non-processing execution object OBJ_ntgt held by the end effector 4 is not performed, the robot control process shown in FIG. 5 is temporarily interrupted, and then the user must manually return the non-processing execution object OBJ_ntgt that did not fall from the end effector 4 to the container CB, and then resume the temporarily interrupted robot control process. This may result in a decrease in throughput of the robot control process. In other words, the takt time of the robot control process may be extended. However, in this embodiment, because an operation to drop the non-processing execution object OBJ_ntgt held by the end effector 4 is performed, the decrease in throughput (i.e., an increase in takt time) is minimized.

[0271] Additionally, the control device 3 can control at least one of the robot 1 and the end effector 4 so that at least the non-processing execution object OBJ_ntgt falls into (i.e., returns to) the container CB based on the calculation results of at least one of the position and orientation of the target object OBJ (e.g., the group of held objects OBG_h, the non-processing execution object OBJ_ntgt, or the processing execution object OBJ_tgt) held by the end effector 4 as an example of the holding state. For example, the control device 3 can generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 in a control manner that takes into consideration the calculation results of at least one of the position and orientation of the non-processing execution object OBJ_ntgt, whose position and / or orientation is known, so as to drop the non-processing execution object OBJ_ntgt into the container CB. The control device 3 can generate a robot control signal for controlling at least one of the robot 1 and the end effector 4 in a control mode that takes into account the calculation results of at least one of the position and orientation of the processing object OBJ_tgt so that the non-processing object OBJ_ntgt falls appropriately (e.g., reliably) into the container CB from the processing object OBJ_tgt whose position and / or orientation is known. If the position and orientation of the target object OBJ (e.g., the group of held objects OBG_h, the non-processing object OBJ_ntgt, or the processing object OBJ_tgt) held by the end effector 4 are not known, it is not possible to control at least one of the robot 1 and the end effector 4 in a control mode that takes into account the calculation results of at least one of the position and orientation of the target object OBJ held by the end effector 4, and therefore the control mode for dropping at least the non-processing object OBJ_ntgt may become a control mode that is, in a sense, left to chance. However, in this embodiment, the control device 3 can adopt, as at least one control mode of the robot 1 and the end effector 4, a control mode that increases the likelihood that at least the non-processing execution object OBJ_ntgt will fall properly (e.g., reliably) into the container CB, based on at least one calculation result of the position and posture of the target object OBJ (e.g., the group of objects to be held OBG_h, the non-processing execution object OBJ_ntgt, or the processing execution object OBJ_tgt) held by the end effector 4.Therefore, the control device 3 can drop at least the non-processing execution object OBJ_ntgt into the container CB appropriately.

[0272] (4) Modifications Next, modifications of the robot system SYS will be described. Note that each modification described below can be combined with other modifications. Alternatively, detailed descriptions of components that have already been described will be omitted in the following description. Furthermore, detailed descriptions of processes that have already been described will be omitted in the following description. In other words, the following description will particularly describe components that are different from components that have already been described and processes that are different from processes that have already been described. Therefore, unless otherwise specified, the robot system SYS may also be provided with components that have already been described, and the robot system SYS may perform processes that have already been described, even in the modifications described below.

[0273] (4-1) First Modification Example (4-1-1) Control of Drop Height Hd In the first modification example, when generating a robot control signal to drop at least the non-processing execution object OBJ_ntgt, the erroneous holding detection unit 313 may control the drop height Hd, which indicates the height (i.e., the position in the Z-axis direction) at which at least the non-processing execution object OBJ_ntgt starts to drop into the container CB.

[0274] (4-1-1-1) Definition of Drop Height Hd The drop height Hd may refer to the height from a predetermined height reference plane (i.e., the distance in the Z-axis direction). In the first modified example, an example will be described in which the bottom surface (upper surface) of the bottom wall BS of the container CB is used as the height reference plane.

[0275] However, a reference plane other than the bottom surface of the bottom wall BS of the container CB may be used as the height reference plane. For example, the placement surface of the placement device T on which the target object OBJ is placed may be used as the height reference plane. Alternatively, the drop height Hd may refer to a coordinate value of the Z axis in the global coordinate system. In other words, the drop height Hd may refer to the height from the coordinate origin of the global coordinate system. Note that the drop height Hd may refer to a coordinate value of the Z axis in a coordinate system (e.g., a robot coordinate system) different from the global coordinate system. In other words, the drop height Hd may refer to the height from the coordinate origin of a coordinate system (e.g., a robot coordinate system) different from the global coordinate system.

[0276] As a first example, the drop height Hd may refer to the height of either the non-processing object OBJ_ntgt or the processing object OBJ_tgt at the time when at least the non-processing object OBJ_ntgt starts to drop into the container CB (in other words, the time when at least the non-processing object OBJ_ntgt drops into the container CB; the same applies below). Note that, as described above, in the case where the non-processing object OBJ_ntgt drops into the container CB while the processing object OBJ_tgt does not drop into the container CB (i.e., the processing object OBJ_tgt continues to be held by the end effector 4), "the time when at least the non-processing object OBJ_ntgt starts to drop into the container CB" may refer to "the time when the non-processing object OBJ_ntgt starts to drop into the container CB." As described above, when a processing object OBJ_tgt falls into the container CB in addition to a non-processing object OBJ_ntgt, "the point at which at least the non-processing object OBJ_ntgt starts to fall into the container CB" may mean "the point at which only the non-processing object OBJ_ntgt starts to fall into the container CB" or "the point at which both the non-processing object OBJ_ntgt and the processing object OBJ_tgt start to fall into the container CB."

[0277] As a second example, the drop height Hd may be the height of the non-processing object OBJ_ntgt and the processing object OBJ_tgt at least when the non-processing object OBJ_ntgt starts to fall into the container CB. More specifically, the drop height Hd may be a height determined based on the heights of the non-processing object OBJ_ntgt and the processing object OBJ_tgt at least when the non-processing object OBJ_ntgt starts to fall into the container CB. For example, the drop height Hd may mean the average value (i.e., intermediate height) of the height of the non-processing object OBJ_ntgt at least when the non-processing object OBJ_ntgt starts to fall into the container CB and the height of the processing object OBJ_tgt at least when the non-processing object OBJ_ntgt starts to fall into the container CB. For example, the drop height Hd may mean the higher of the height of the non-processing object OBJ_ntgt when at least the non-processing object OBJ_ntgt starts to drop into the container CB and the height of the processing object OBJ_tgt when at least the non-processing object OBJ_ntgt starts to drop into the container CB. For example, the drop height Hd may mean the lower of the height of the non-processing object OBJ_ntgt when at least the non-processing object OBJ_ntgt starts to drop into the container CB and the height of the processing object OBJ_tgt when at least the non-processing object OBJ_ntgt starts to drop into the container CB. For example, the fall height Hd may mean any height between the height of the non-processed object OBJ_ntgt when the non-processed object OBJ_ntgt starts to fall into the container CB and the height of the processed object OBJ_tgt when the non-processed object OBJ_ntgt starts to fall into the container CB.

[0278] As a third example, the drop height Hd may refer to the height of an object other than the non-processing object OBJ_ntgt and the processing object OBJ_tgt when at least the non-processing object OBJ_ntgt starts to drop into the container CB. For example, the drop height Hd may refer to the height of the imaging system 2 (particularly the imaging device 21) when at least the non-processing object OBJ_ntgt starts to drop into the container CB. For example, the drop height Hd may refer to the height of the end effector 4 when at least the non-processing object OBJ_ntgt starts to drop into the container CB. The height of the end effector 4 may refer to the height of the reference point TCP of the end effector 4 (e.g., the above-mentioned tool center point or a point set on the end effector 4 by the user), or may refer to the height of any portion of the end effector 4.

[0279] As a fourth example, the drop height Hd may refer to the height of the robot arm 12 at least when the non-processing execution object OBJ_ntgt starts to drop into the container CB. In this case, the height of the robot arm 12 may be the height of the flange of the robot arm 12 (the portion where the end effector 4 is attached to the robot arm 12).

[0280] (4-1-1-2) Control of Drop Height Hd Based on Height of Target Objects Containered in Container CB In order to control the drop height Hd, the erroneous holding detection unit 313 may use the object height Ho, which is the height of the multiple target objects OBJ contained in the container CB, as shown in Fig. 14. That is, the erroneous holding detection unit 313 may control the drop height Hd by setting (in other words, determining) the drop height Hd based on the object height Ho.

[0281] Note that the object height Ho may mean the height from a predetermined height reference plane (i.e., the distance in the Z-axis direction), similar to the drop height Hd described above. Alternatively, the object height Ho may mean the Z-axis coordinate value in the global coordinate system. In other words, the object height Ho may mean the height from the coordinate origin of the global coordinate system. Note that the object height Ho may mean the Z-axis coordinate value in a coordinate system (e.g., a robot coordinate system) different from the global coordinate system. In other words, the object height Ho may mean the height from the coordinate origin of a coordinate system (e.g., a robot coordinate system) different from the global coordinate system.

[0282] The object height Ho may refer to the height of the target object group OBG_cb that includes all of the multiple target objects OBJ contained in the container CB. Alternatively, the object height Ho may refer to the height of the target object group OBG_cb that includes some of the multiple target objects OBJ contained in the container CB. In other words, the object height Ho may refer to the height of the target object group OBG_cb that includes at least some of the multiple target objects OBJ contained in the container CB.

[0283] The height of the highest portion of the target object group OBG_cb may be used as the object height Ho. For example, the height of one target object OBJ located at the highest position among the multiple target objects OBJ included in the target object group OBG_cb may be used as the object height Ho. For example, the height of the highest portion of one target object OBJ located at the highest position among the multiple target objects OBJ included in the target object group OBG_cb may be used as the object height Ho. For example, the height of an arbitrary portion of one target object OBJ located at the highest position among the multiple target objects OBJ included in the target object group OBG_cb may be used as the object height Ho.

[0284] The height of the highest portion of a portion of the target object group OBG_cb may be used as the object height Ho. For example, the height of at least one target object OBJ selected based on desired selection criteria from among the multiple target objects OBJ included in the target object group OBG_cb may be used as the object height Ho. For example, the height of the highest portion of at least one target object OBJ selected based on desired selection criteria from among the multiple target objects OBJ included in the target object group OBG_cb may be used as the object height Ho. For example, the height of an arbitrary portion of at least one target object OBJ selected based on desired selection criteria from among the multiple target objects OBJ included in the target object group OBG_cb may be used as the object height Ho.

[0285] The average height of the target object group OBG_cb may be used as the object height Ho. The average height of the target object group OBG_cb may mean the average value of the heights of multiple portions of the target object group OBG_cb. The average height of the target object group OBG_cb may mean the average value of the heights of multiple target objects OBJ included in the target object group OBG_cb. When multiple target objects OBJ are piled up (i.e., loosely stacked), the average height of the target object group OBG_cb may mean the average value of the heights of some target objects OBJ that are not piled up on top of other target objects OBJ (i.e., some target objects OBJ located in the uppermost layer) among the multiple target objects OBJ included in the target object group OBG_cb. When multiple target objects OBJ are stacked (i.e., piled up loosely), the average height of the target object group OBG_cb may mean the average value of the heights of some target objects OBJ (i.e., some target objects OBJ located in the top layer) among the multiple target objects OBJ included in the target object group OBG_cb, of which at least a portion of the upper part is exposed upward.

[0286] The object height Ho may be an average height of a portion of the target object group OBG_cb. The average height of the portion of the target object group OBG_cb may mean an average value of the heights of at least one target object OBJ selected based on a desired selection criterion from among the multiple target objects OBJ included in the target object group OBG_cb.

[0287] The erroneous holding detection unit 313 may set the drop height Hd based on the object height Ho so that the drop height Hd used when the object height Ho is a first height is lower than the drop height Hd used when the object height Ho is a second height that is higher than the first height. In other words, the erroneous holding detection unit 313 may determine the drop height Hd based on the object height Ho so that the drop height Hd used when the object height Ho is the first height is higher than the drop height Hd used when the object height Ho is a third height that is lower than the first height.

[0288] In particular, the erroneous holding detection unit 313 may set the drop height Hd based on the object height Ho such that the drop height Hd decreases as the object height Ho decreases. In other words, the erroneous holding detection unit 313 may set the drop height Hd based on the object height Ho such that the drop height Hd increases as the object height Ho increases.

[0289] 15(a) and 15(b) show examples of the drop height Hd set based on the object height Ho. FIG. 15(a) shows an example in which the end effector 4 holds a process-to-be-processed object OBJ_tgt#1 and a non-processing-to-be-processed object OBJ_ntgt#1. Furthermore, FIG. 15(a) shows an example in which, at the time when the end effector 4 holds the process-to-be-processed object OBJ_tgt#1 and the non-processing-to-be-processed object OBJ_ntgt#1, a target object group _cb#1 is contained in the container CB, and the object height Ho of the target object group OBG_cb#1 is the first object height Ho#1. In this case, the erroneous holding detection unit 313 may set the drop height Hd to the first drop height Hd#1 based on the first object height Ho#1, which is the object height Ho of the target object group OBG_cb#1. For example, the erroneous holding detection unit 313 may set the drop height Hd to a first drop height Hd#1, which is a height calculated by adding a predetermined first margin height to the first object height Ho#1.

[0290] The first margin height may be set in advance or may be set when setting the drop height Hd. The first margin height may be set manually by the user or automatically by the control device 3.

[0291] Thereafter, the erroneous holding detection unit 313 may generate a robot control signal based on the first drop height Hd#1.

[0292] For example, if the drop height Hd refers to the height of the non-processing execution object OBJ_ntgt, the signal generating unit 312 may generate a robot control signal so that the non-processing execution object OBJ_ntgt#1 is located at the first drop height Hd#1 at least when the non-processing execution object OBJ_ntgt#1 starts to drop into the container CB. In other words, the signal generating unit 312 may generate a robot control signal so that the non-processing execution object OBJ_ntgt#1 is located at the Z position indicated by the first drop height Hd#1 at least when the non-processing execution object OBJ_ntgt#1 starts to drop into the container CB. In this case, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the difference between the current height of the non-processing execution object OBJ_ntgt#1 and the first drop height Hd#1 becomes small (e.g., becomes zero). The position (particularly the Z position) of the non-processing execution object OBJ_ntgt#1 calculated to identify the holding state in step S23 of FIG. 5 may be used as the current height of the non-processing execution object OBJ_ntgt#1. Alternatively, the signal generation unit 312 may calculate (estimate) the current height of the non-processing execution object OBJ_ntgt#1 based on at least one of the position and orientation of the end effector 4 calculated to identify the holding state in step S23 of FIG. 5. Alternatively, the signal generation unit 312 may acquire information regarding at least one of the current position and orientation of the reference point (e.g., the tool center point) of the end effector 4 from the robot control device 13, and calculate (estimate) the current height of the non-processing execution object OBJ_ntgt#1 based on the acquired information. As a result, the robot 1 moves the end effector 4 so that the non-processing execution object OBJ_ntgt#1 moves to the first drop height Hd#1. That is, the robot 1 changes (in other words, controls or adjusts) the height of the end effector 4 so that the non-processing execution object OBJ_ntgt#1 moves to the first drop height Hd#1. Then, at least one of the robot 1 and the end effector 4 is controlled so that at least the non-processing execution object OBJ_ntgt#1 falls. As a result, at least the non-processing execution object OBJ_ntgt#1 falls from the first drop height Hd#1 (in other words, from the Z position indicated by the first drop height Hd#1).

[0293] For example, if the drop height Hd refers to the height of the processing-execution object OBJ_tgt, the signal generating unit 312 may generate a robot control signal so that the processing-execution object OBJ_tgt#1 is located at the first drop height Hd#1 at least when the non-processing-execution object OBJ_ntgt#1 starts to drop into the container CB. In other words, the signal generating unit 312 may generate a robot control signal so that the processing-execution object OBJ_tgt#1 is located at the Z position indicated by the first drop height Hd#1 at least when the non-processing-execution object OBJ_ntgt#1 starts to drop into the container CB. In this case, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the difference between the current height of the processing-execution object OBJ_tgt#1 and the first drop height Hd#1 becomes small (e.g., becomes zero). The position (particularly the Z position) of the process execution object OBJ_tgt#1 calculated to identify the holding state in step S23 of FIG. 5 may be used as the current height of the process execution object OBJ_tgt#1. Alternatively, the signal generation unit 312 may calculate (estimate) the current height of the process execution object OBJ_tgt#1 based on at least one of the position and orientation of the end effector 4 calculated to identify the holding state in step S23 of FIG. 5. Alternatively, the signal generation unit 312 may acquire information regarding at least one of the current position and orientation of the reference point (e.g., the tool center point) of the end effector 4 from the robot control device 13, and calculate (estimate) the current height of the process execution object OBJ_tgt#1 based on the acquired information. As a result, the robot 1 moves the end effector 4 so that the process execution object OBJ_tgt#1 moves to the first drop height Hd#1. That is, the robot 1 changes (in other words, controls or adjusts) the height of the end effector 4 so that the process-execution object OBJ_tgt#1 moves to the first drop height Hd#1. Then, at least one of the robot 1 and the end effector 4 is controlled so that at least the non-processing object OBJ_ntgt#1 falls. As a result, under the condition that the process-execution object OBJ_tgt#1 is located at the first drop height Hd#1 (in other words, the Z position indicated by the first drop height Hd#1), at least the non-processing object OBJ_ntgt#1 falls.Furthermore, the state in which "a non-processing object OBJ_ntgt#1 falls when a processing object OBJ_tgt#1 is located at the first drop height Hd#1" may be considered equivalent to the state in which "a non-processing object OBJ_ntgt#1 falls from the first drop height Hd#1."

[0294] For example, if the drop height Hd refers to a height determined based on the heights of the non-processing execution object OBJ_ntgt and the processing execution object OBJ_tgt, the signal generating unit 312 may generate a robot control signal so that the height determined based on the heights of the non-processing execution object OBJ_ntgt#1 and the processing execution object OBJ_tgt#1 coincides with the first drop height Hd#1 at least when the non-processing execution object OBJ_ntgt#1 starts to fall into the container CB. As a result, the robot 1 moves the end effector 4 so that the height determined based on the heights of the non-processing execution object OBJ_ntgt#1 and the processing execution object OBJ_tgt#1 coincides with the first drop height Hd#1. That is, the robot 1 changes (in other words, controls or adjusts) the height of the end effector 4 so that the height determined based on the heights of the non-processing execution object OBJ_ntgt#1 and the process execution object OBJ_tgt#1 matches the first drop height Hd#1. In this case, the signal generating unit 312 may calculate the current heights of the non-processing execution object OBJ_ntgt#1 and the process execution object OBJ_tgt#1 by using the position (particularly, the Z position) of the non-processing execution object OBJ_ntgt#1 and the position (particularly, the Z position) of the process execution object OBJ_tgt#1 calculated to identify the holding state in step S23 of FIG. 5 as the current height of the non-processing execution object OBJ_ntgt#1 and the current height of the process execution object OBJ_tgt#1, respectively. Alternatively, the signal generating unit 312 may calculate (estimate) the current heights of the non-processing execution object OBJ_ntgt#1 and the processing execution object OBJ_tgt#1 based on at least one of the position and orientation of the end effector 4 calculated to identify the holding state in step S23 of Fig. 5. Alternatively, the signal generating unit 312 may acquire information regarding at least one of the current position and orientation of the reference point (e.g., tool center point) of the end effector 4 from the robot control device 13, and calculate (estimate) the current heights of the non-processing execution object OBJ_ntgt#1 and the processing execution object OBJ_tgt#1 based on the acquired information.Thereafter, the signal generating unit 312 may generate a robot control signal for controlling the robot 1 so that the difference between the current heights of the non-processing execution object OBJ_ntgt#1 and the process execution object OBJ_tgt#1 and the first drop height Hd#1 becomes small (for example, becomes zero). Then, at least one of the robot 1 and the end effector 4 is controlled so that at least the non-processing execution object OBJ_ntgt#1 falls. As a result, at least the non-processing execution object OBJ_ntgt#1 falls under a condition where the height determined based on the heights of the non-processing execution object OBJ_ntgt#1 and the process execution object OBJ_tgt#1 matches the first drop height Hd#1. Furthermore, the state in which "non-processing execution object OBJ_ntgt#1 falls under a situation in which the height determined based on the heights of non-processing execution object OBJ_ntgt#1 and processing execution object OBJ_tgt#1 is the same as the first drop height Hd#1" may be considered equivalent to the state in which "non-processing execution object OBJ_ntgt#1 falls from the first drop height Hd#1."

[0295] For example, if the drop height Hd refers to the height of the imaging system 2 (particularly, the imaging device 21), the signal generation unit 312 may generate a robot control signal so that the imaging system 2 is located at the first drop height Hd#1 when at least the non-processing execution object OBJ_ntgt#1 starts to drop into the container CB. In other words, the signal generation unit 312 may generate a robot control signal so that the imaging system 2 is located at the Z position indicated by the first drop height Hd#1 when at least the non-processing execution object OBJ_ntgt#1 starts to drop into the container CB. In this case, the signal generation unit 312 may calculate the current height of the imaging system 2. For example, the signal generation unit 312 may calculate (estimate) the current height of the imaging system 2 from the positions (particularly, the Z position) of at least one of the non-processing execution object OBJ_ntgt#1 and the processing execution object OBJ_tgt#1 calculated to identify the holding state in step S23 of FIG. 5 . Alternatively, the signal generator 312 may calculate (estimate) the current height of the imaging system 2 based on at least one of the position and orientation of the end effector 4 calculated to identify the holding state in step S23 of FIG. 5 . Alternatively, the signal generator 312 may acquire information regarding at least one of the current position and orientation of a reference point (e.g., a tool center point) of the end effector 4 from the robot control device 13, and calculate (estimate) the current height of the imaging system 2 based on the acquired information. The signal generator 312 may then generate a robot control signal for controlling the robot 1 so that the difference between the current height of the imaging system 2 and the first drop height Hd#1 becomes small (e.g., becomes zero). As a result, the robot 1 moves the end effector 4 so that the imaging system 2 moves to the first drop height Hd#1. In other words, the robot 1 changes (in other words, controls or adjusts) the height of the end effector 4 so that the imaging system 2 moves to the first drop height Hd#1. Thereafter, at least one of the robot 1 and the end effector 4 is controlled so that at least the non-processing execution object OBJ_ntgt#1 falls.As a result, at least the non-processing execution object OBJ_ntgt#1 falls when the imaging system 2 is located at the first drop height Hd#1 (in other words, the Z position indicated by the first drop height Hd#1). Note that the state in which "the non-processing execution object OBJ_ntgt#1 falls when the imaging system 2 is located at the first drop height Hd#1" may be considered equivalent to the state in which "the non-processing execution object OBJ_ntgt#1 falls from the first drop height Hd#1."

[0296] For example, if the drop height Hd refers to the height of the end effector 4 (e.g., the height of the reference point TCP of the end effector 4), the signal generation unit 312 may generate a robot control signal so that the end effector 4 is located at the first drop height Hd#1 at least when the non-processing execution object OBJ_ntgt#1 starts to drop into the container CB. That is, the signal generation unit 312 may generate a robot control signal so that the end effector 4 is located at the Z position indicated by the first drop height Hd#1 at least when the non-processing execution object OBJ_ntgt#1 starts to drop into the container CB. In this case, the signal generation unit 312 may generate a robot control signal for controlling the robot 1 so that the difference between the current height of the end effector 4 and the first drop height Hd#1 becomes small (e.g., becomes zero). The position (particularly the Z position) of the end effector 4 calculated to identify the holding state in step S23 of FIG. 5 may be used as the current height of the end effector 4. Alternatively, the signal generation unit 312 may calculate (estimate) the current height of the end effector 4 based on at least one of the position and orientation of at least one of the non-processing execution object OBJ_ntgt#1 and the processing execution object OBJ_tgt#1 calculated to identify the holding state in step S23 of FIG. 5 . Alternatively, the signal generation unit 312 may acquire information regarding at least one of the current position and orientation of a reference point (e.g., a tool center point) of the end effector 4 from the robot control device 13, and calculate (estimate) the current height of the end effector 4 based on the acquired information. As a result, the robot 1 moves the end effector 4 so that the end effector 4 moves to the first drop height Hd#1. In other words, the robot 1 changes (in other words, controls or adjusts) the height of the end effector 4 so that the end effector 4 moves to the first drop height Hd#1. Thereafter, at least one of the robot 1 and the end effector 4 is controlled so that at least the non-processing execution object OBJ_ntgt#1 falls. As a result, at least the non-processing execution object OBJ_ntgt#1 falls while the end effector 4 is positioned at the first drop height Hd#1 (in other words, the Z position indicated by the first drop height Hd#1).Furthermore, the state in which "non-processing execution object OBJ_ntgt#1 falls when end effector 4 is positioned at the first drop height Hd#1" may be considered equivalent to the state in which "non-processing execution object OBJ_ntgt#1 falls from the first drop height Hd#1."

[0297] For example, if the drop height Hd refers to the height of the robot arm 12 (e.g., the height of the flange of the robot arm 12), the signal generation unit 312 may generate a robot control signal so that the robot arm 12 is located at the first drop height Hd#1 when at least the non-processing execution object OBJ_ntgt#1 starts to drop into the container CB. In other words, the signal generation unit 312 may generate a robot control signal so that the robot arm 12 is located at the Z position indicated by the first drop height Hd#1 when at least the non-processing execution object OBJ_ntgt#1 starts to drop into the container CB. In this case, the signal generation unit 312 may calculate the current height of the robot arm 12. For example, the signal generation unit 312 may calculate (estimate) the current height of the robot arm 12 from the positions (particularly the Z position) of at least one of the non-processing execution object OBJ_ntgt#1 and the processing execution object OBJ_tgt#1 calculated to identify the holding state in step S23 of FIG. 5 . Alternatively, the signal generator 312 may calculate (estimate) the current height of the robot arm 12 based on at least one of the position and orientation of the end effector 4 calculated to identify the holding state in step S23 of FIG. 5 . Alternatively, the signal generator 312 may acquire information regarding at least one of the current position and orientation of a reference point (e.g., a tool center point) of the end effector 4 from the robot control device 13, and calculate (estimate) the current height of the robot arm 12 based on the acquired information. The signal generator 312 may then generate a robot control signal for controlling the robot 1 so that the difference between the current height of the robot arm 12 and the first drop height Hd#1 becomes small (e.g., becomes zero). As a result, the robot 1 moves the end effector 4 so that the robot arm 12 moves to the first drop height Hd#1. In other words, the robot 1 changes (in other words, controls or adjusts) the height of the end effector 4 so that the robot arm 12 moves to the first drop height Hd#1. Thereafter, at least one of the robot 1 and the end effector 4 is controlled so that at least the non-processing execution object OBJ_ntgt#1 falls.As a result, at least the non-processing execution object OBJ_ntgt#1 falls when the robot arm 12 is located at the first drop height Hd#1 (in other words, the Z position indicated by the first drop height Hd#1). Note that the state in which "the non-processing execution object OBJ_ntgt#1 falls when the robot arm 12 is located at the first drop height Hd#1" may be considered equivalent to the state in which "the non-processing execution object OBJ_ntgt#1 falls from the first drop height Hd#1."

[0298] After at least the non-processing object OBJ_ntgt#1 has fallen into the container CB, the end effector 4 newly holds the process-execution object OBJ_tgt#2, as shown in Fig. 15(b). In this case, an example will be described in which the end effector 4 holds the non-processing object OBJ_ntgt#2 together with the process-execution object OBJ_tgt#2, as shown in Fig. 15(b). Furthermore, Fig. 15(b) shows an example in which, at the time when the end effector 4 is holding the non-processing object OBJ_ntgt#2 together with the process-execution object OBJ_tgt#2, a target object group OBG_cb#2 is contained in the container CB, and the object height Ho of the target object group OBG_cb#2 is the first object height Ho#2.

[0299] Note that Figure 15(b) may be regarded as showing an example in which, after at least the non-processing object OBJ_ntgt#1 has fallen into the container CB, the end effector 4 holds the non-processing object OBJ_ntgt#2 together with the processing object OBJ_tgt#2 without ever performing a placement process in which it holds another processing object OBJ_tgt newly selected from the multiple target objects OBJ contained in the container CB and releases the other processing object OBJ_tgt that it has held outside the container CB. Alternatively, Figure 15 (b) may be considered to show an example in which, after at least the non-processing object OBJ_ntgt#1 falls into the container CB, the end effector 4 holds another processing object OBJ_tgt newly selected from the multiple target objects OBJ contained in the container CB and performs a placement process at least once to release the other processing object OBJ_tgt that it has held outside the container CB, and then holds the non-processing object OBJ_ntgt#2 together with the processing object OBJ_tgt#2.

[0300] The target object group OBG_cb#2 may be different from the target object group OBG_cb#1. For example, after at least the non-processing object OBJ_ntgt#1 has fallen into the container CB, and after performing at least one placement process of holding another process-execution object OBJ_tgt newly selected from the plurality of target objects OBJ contained in the container CB and releasing the held other process-execution object OBJ_tgt out of the container CB, if the end effector 4 is holding both the process-execution object OBJ_tgt#2 and the non-processing object OBJ_ntgt#2, at least one target object OBJ included in the target object group OBG_cb#1 has been carried out of the container CB. In this case, at least one target object OBJ included in the target object group OBG_cb#1 does not have to be included in the target object group OBG_cb#2. For example, even if the end effector 4 holds both the process execution object OBJ_tgt#2 and the non-process execution object OBJ_ntgt#2 without performing any placement processing after at least the non-process execution object OBJ_ntgt#1 has fallen into the container CB, at least one of the process execution object OBJ_tgt#2 and the non-process execution object OBJ_ntgt#2 is a target object OBJ that was included in the target object group OBG_cb#1. Therefore, even in this case, at least one target object OBJ that was included in the target object group OBG_cb#1 does not have to be included in the target object group OBG_cb#2.

[0301] In this way, the number of target objects OBJ included in the target object group OBG_cb#2 is likely to be smaller than the number of target objects OBJ included in the target object group OBG_cb#1. As a result, as shown in FIG. 15B, the second object height Ho#2, which is the object height Ho#2 of the target object group OBG_cb#2, is likely to be lower than the second object height Ho#1, which is the object height Ho#2 of the target object group OBG_cb#2. In this case, as shown in FIG. 15B, the erroneous holding detection unit 313 may set the drop height Hd to a second drop height H#2 that is lower than the first drop height Hd#1, based on the second object height Ho#2, which is the object height Ho of the target object group OBG_cb#2. For example, the erroneous holding detection unit 313 may set the drop height Hd to a second drop height Hd#2, which is a height calculated by adding a predetermined second margin height to the second object height Ho#2.

[0302] The second margin height may be set in advance or may be set when the drop height Hd is set. The second margin height may be set manually by a user or automatically by the control device 3. The second margin height used to set the second drop height Hd#2 may be the same as or different from the first margin height used to set the first drop height Hd#1. When the second margin height and the first margin height are the same, a common margin height that can be used as each of the first and second margin heights may be set. The common margin height may be set in advance or may be set when the drop height Hd is set. The common margin height may be set manually by a user or automatically by the control device 3. The erroneous holding detection unit 313 may then generate a robot control signal based on the second drop height Hd#2.

[0303] For example, if the drop height Hd refers to the height of the non-processing execution object OBJ_ntgt, the signal generating unit 312 may generate a robot control signal so that the non-processing execution object OBJ_ntgt#2 is located at the second drop height Hd#2 at least when the non-processing execution object OBJ_ntgt#2 starts to drop into the container CB. In other words, the signal generating unit 312 may generate a robot control signal so that the non-processing execution object OBJ_ntgt#2 is located at the Z position indicated by the second drop height Hd#2 at least when the non-processing execution object OBJ_ntgt#2 starts to ...

Claims

1. A control device that includes a holding device capable of holding a target object, an imaging system that images the target object, a robot that moves the holding device and the imaging system, and generates a control signal for controlling at least one of the holding device. The control device includes an arithmetic unit that generates the control signal and a communication device that outputs the control signal generated by the arithmetic unit. The arithmetic unit generates, as the control signal, a first control signal for controlling at least one of the robot and the holding device so that at least a second target object among a plurality of target objects accommodated in a container drops into the container when a first target object, which is the target object among the target object group, is held by the holding device based on an imaging result obtained by imaging, by the imaging system, a target object group including at least a part of the plurality of target objects accommodated in the container. After at least the second target object drops into the container based on the first control signal, when a third target object among the plurality of target objects accommodated in the container is held by the holding device and a fourth target object among the plurality of target objects accommodated in the container is held by the holding device together with the third target object, the arithmetic unit generates, as the control signal, a second control signal for controlling at least one of the robot and the holding device so that at least the fourth target object drops into the container from a position lower than a position where at least the second target object starts to drop into the container.

2. The control device according to claim 1, wherein the second control signal includes at least one of a signal for controlling at least one of the robot and the holding device so that at least the fourth target object located at the lower position drops from the holding device into the container and a signal for controlling at least one of the robot and the holding device so that at least the fourth target object drops from the holding device located at the lower position into the container.

3. The height of at least a part of the plurality of target objects contained in the container when at least the fourth target object starts to fall based on the second control signal is lower than the height of at least a part of the plurality of target objects contained in the container when at least the second target object starts to fall based on the first control signal. The control device according to claim 1 or 2.

4. The number of the target objects contained in the container when at least the fourth target object starts to fall is smaller than the number of the target objects contained in the container when at least the second target object starts to fall. The control device according to any one of claims 1 to 3.

5. The arithmetic device determines a first drop height for at least the second target object to start falling into the container based on an imaging result obtained by imaging at least a part of the plurality of target objects contained in the container with the imaging system, and the first control signal includes a signal for controlling at least one of the robot and the holding device so that at least the second target object falls into the container from the determined first drop height. The control device according to any one of claims 1 to 4.

6. The first drop height is the height of the second target object when at least the second target object starts to fall, the height of the first target object when at least the second target object starts to fall, the height of the first and second target objects when at least the second target object starts to fall, the height of the imaging system when at least the second target object starts to fall, or the height of the holding device when at least the second target object starts to fall. The control device according to claim 5.

7. The imaging result used to determine the first drop height is the imaging result obtained by imaging the group of target objects with the imaging system. The control device according to claim 5 or 6.

8. When the group of target objects is the first group of target objects, the second control signal is based on an imaging result obtained by imaging a second group of target objects including at least a part of the plurality of target objects accommodated in the container by the imaging system. When a fourth target object is held by the holding device together with the third target object when the third target object in the second group of target objects is held by the holding device, from the low position, at least the fourth target object The control device according to any one of claims 1 to 7, including a signal for controlling at least one of the robot and the holding device so as to fall into the container.

9. The height of the second group of target objects is lower than the height of the first group of target objects. The control device according to claim 8.

10. The arithmetic unit determines a second drop height for at least the fourth target object to start falling into the container based on an imaging result obtained by imaging at least a part of the plurality of target objects accommodated in the container by the imaging system. The second control signal includes a signal for controlling at least one of the robot and the holding device so that at least the fourth target object falls into the container from the determined second drop height. The control device according to claim 8 or 9.

11. The second drop height is the height of the fourth target object when at least the fourth target object starts to fall, the height of the third target object when at least the fourth target object starts to fall, the third and fourth target objects when at least the fourth target object starts to fall. The height, the height of the imaging system when at least the fourth target object starts to fall, or the height of the holding device when at least the fourth target object starts to fall. The control device according to claim 10.

12. The imaging result used to determine the second drop height is the imaging result obtained by imaging the second group of target objects by the imaging system. The control device according to claim 10 or 11.

13. The state in which the second target object is held by the holding device together with the first target object includes a state in which the second target object is indirectly or directly held by the holding device. The state in which the fourth target object is held by the holding device together with the third target object includes a state in which the fourth target object is indirectly or directly held by the holding device. The control device according to any one of claims 1 to 12.

14. The state in which the second target object is indirectly held by the holding device is a state in which the second target object is held by the first target object held by the holding device. The state in which the second target object is directly held by the holding device is a state in which each of the first target object and the second target object is directly held by the holding device. The state in which the fourth target object is indirectly held by the holding device is a state in which the fourth target object is held by the third target object held by the holding device. The state in which the fourth target object is directly held by the holding device is a state in which each of the third target object and the fourth target object is directly held by the holding device. The control device according to claim 13.

15. In the case of the state in which the second target object is held by the first target object held by the holding device, the arithmetic unit generates, as the first control signal, a signal for controlling at least one of the robot and the holding device so that the second target object falls from the first target object while the first target object is held by the holding device. The control device according to claim 14.

16. The first control signal includes a signal for controlling the robot so that the second target object falls from the first target object by moving the holding device while holding the first target object. The control device according to claim 15.

17. Moving the holding device includes changing the posture of the holding device. The control device according to claim 16.

18. Changing the posture of the holding device by the first control signal includes rotationally moving the reference point of the holding device in a direction from the end of the first target object held by the holding device toward the reference point of the holding device. The control device according to claim 17.

19. The end portion of the first target object is the portion that is farthest from the reference point of the holding device among the end portions of the first target object. The control device according to claim 18.

20. The second target object falling from the first target object by the first control signal includes the second target object moving in a direction different from the direction in which the posture of the holding device is changed based on the first control signal and then falling from the first target object. The control device according to any one of claims 17 to 19.

21. The second target object falling from the first target object by the first control signal includes the second target object moving in a direction different from the direction in which the holding device exists and then falling from the first target object by changing the posture of the holding device based on the first control signal. The control device according to any one of claims 17 to 20.

22. Moving the holding device includes changing the height of the holding device. The control device according to any one of claims 16 to 21.

23. Moving the holding device includes changing the posture of the holding device, and the first control signal includes a signal for controlling the robot so that the second target object falls from the first target object at a position higher than the position where the at least fourth target object starts to fall into the container by changing the height of the holding device according to the amount of change in the posture of the holding device. The control device according to claim 22.

24. The first control signal includes a signal for controlling the holding device so that the second target object falls from the first target object at a position higher than the position where the at least fourth target object starts to fall into the container by changing the height of the holding device while changing the posture of the holding device. The control device according to claim 22 or 23.

25. The first control signal includes a signal for controlling the holding device so that the second target object falls from the first target object by reducing the holding force of the holding device while holding the first target object. The control device according to any one of claims 15 to 24.

26. After the second target object has fallen from the first target object into the container while the first target object is held by the holding device based on the first control signal, the first target object held by the holding device is carried out of the container and released from the holding device outside the container. The control device according to any one of claims 15 to 24.

27. When each of the first target object and the second target object is directly held by the holding device, the arithmetic unit generates, as the first control signal, a signal for controlling at least one of the robot and the holding device so that the first target object and the second target object fall from the holding device. The control device according to any one of claims 14 to 26.

28. The first control signal includes a signal for controlling the holding device so that the first target object and the second target object fall from the holding device. The control device according to claim 27.

29. The first control signal includes a signal for controlling the robot so that the first target object and the second target object fall from the holding device by moving the holding device. The control device according to claim 27 or 28.

30. The first control signal includes a signal for controlling at least one of the robot and the holding device so that the first target object and the second target object fall from the holding device at a position higher than the position where the at least fourth target object starts to fall into the container by changing the height of the holding device. The control device according to any one of claims 27 to 29.

31. The first control signal includes a first signal for controlling at least one of the robot and the holding device so that the at least second target object falls into the container. When, based on the first signal, the arithmetic unit controls at least one of the robot and the holding device but the at least second target object has not fallen, the arithmetic unit generates, as a partial signal of the first control signal, a second signal for controlling at least one of the robot and the holding device in a control mode different from the control mode based on the first signal so that the at least second target object falls. The control device according to any one of claims 1 to 30.

32. The control mode according to claim 31, including at least one of the control mode of the robot and the control mode of the holding device.

33. The control mode of the robot according to claim 32, including at least one of the moving direction, moving amount, moving speed, and moving acceleration of the holding device by the control of the robot.

34. The control mode of the holding device according to claim 32 or 33, including the holding force of the holding device.

35. After the arithmetic unit holds the first target object with the holding device based on the imaging result of the target object group imaged by the imaging system, the first control signal is generated. The control device according to any one of claims 1 to 34.

36. The arithmetic unit generates the first control signal based on the measurement result of the measuring device. The control device according to claim 35.

37. The arithmetic unit generates the first control signal for controlling at least one of the robot and the holding device so that the posture of the at least second target object located in the container becomes a desired posture by falling based on the measurement result. The control device according to claim 36.

38. The desired posture according to claim 37 includes a posture in which a desired part of the at least second target object located in the container by falling faces a desired direction.

39. The desired part according to claim 38 includes a holding target part that is at least a part of the target object held by the holding device and is preset.

40. The first target object is held by the holding device from among the plurality of target objects accommodated in the container based on the result of a matching process using a model showing at least a part of the shape of the target object and the imaging result of the target object group imaged by the imaging system. The desired part includes a part corresponding to the model in the at least second target object. The control device according to claim 38 or 39.

41. The desired part according to any one of claims 38 to 40 includes a part designated by the user in the second target object.

42. The control device according to any one of claims 38 to 41, wherein the desired direction includes a direction in which the imaging system can be positioned to image the at least second target object located in the container by falling.

43. The control device according to any one of claims 38 to 42, wherein the desired direction includes a direction in which the desired part of the at least second target object located in the container by falling faces the imaging system that images the at least second target object located in the container by falling.

44. An opening is formed in the container for the holding device to enter the container under the control of the robot to hold the target object accommodated in the container, and the desired direction includes the direction of the opening as seen from the at least second target object located in the container by falling. The control device according to any one of claims 38 to 43.

45. The control device according to any one of claims 38 to 44, wherein the desired direction includes an upward direction as seen from the at least second target object located in the container by falling.

46. The control device according to any one of claims 38 to 45, wherein the desired direction includes a direction different from the direction in which at least a part of the container exists as seen from the at least second target object located in the container by falling.

47. The arithmetic unit generates the first control signal for controlling at least one of the robot and the holding device so that the at least second target object falls to a desired position in the container based on the measurement result. The control device according to any one of claims 36 to 46.

48. The control device according to claim 47, wherein the desired position includes a position where the target object does not exist in the container.

49. The arithmetic unit determines whether to drop only the second target object or both the first target object and the second target object held by the holding device into the container based on the measurement result, and generates the first control signal based on the result of the determination. The control device according to any one of claims 36 to 48.

50. The control device according to claim 49, wherein when the arithmetic unit determines that only the second target object is to be dropped into the container, the first control signal including a signal for controlling at least one of the robot and the holding device is generated so that only the second target object drops into the container while the first target object is held.

51. The control device according to claim 50, wherein the first control signal includes a signal for controlling the robot so that only the second target object drops into the container by moving the holding device while holding the first target object.

52. After the at least second target object starts to drop into the container based on the first control signal, the first target object held by the holding device is carried out of the container and released from the holding device outside the container. The control device according to claim 50 or 51.

53. The control device according to any one of claims 49 to 52, wherein when the arithmetic unit determines that the first target object and the second target object are to be dropped into the container, the first control signal including a signal for controlling at least one of the robot and the holding device is generated so that the first target object and the second target object held by the holding device drop into the container.

54. The control device according to claim 53, wherein the first control signal includes a signal for controlling the holding device so that the first target object and the second target object drop into the container after controlling the robot to change the height of the holding device.

55. After the first target object and the second target object drop into the container based on the first control signal, the target object among the plurality of target objects accommodated in the container is held by the holding device, or at least a part of the plurality of target objects accommodated in the container is imaged by the imaging system. The control device according to claim 53 or 54.

56. The measurement device includes the imaging system, and the arithmetic unit generates the first control signal based on the imaging result of the imaging system after holding the first target object by the holding device based on the imaging result of the target object group imaged by the imaging system. The control device according to any one of claims 36 to 55.

57. The imaging result used for generating the first control signal includes the result of imaging at least one of the first target object and the second target object held by the holding device by the imaging system. The control device according to claim 56.

58. After the arithmetic unit holds the first target object by the holding device based on the imaging result of imaging the target object group by the imaging system, the holding device moves away from the plurality of target objects accommodated in the container from the position when the holding device holds the first target object. A signal for controlling the robot is generated as a part of the first control signal, and at least one of the first target object and the second target object held by the holding device that has moved away from the plurality of target objects accommodated in the container based on the signal of the part of the first control signal is imaged by the imaging system. The control device according to claim 57.

59. The signal of the part of the first control signal is a signal for controlling the robot so that the holding device moves to a position higher than the position when the holding device holds the first target object. The control device according to claim 58.

60. The first control signal includes a signal for controlling the robot so that the holding device approaches the container from the position when the holding device images at least one of the first target object and the second target object by the imaging system. The control device according to claim 58 or 59.

61. The first control signal includes a signal for controlling at least one of the robot and the holding device so that the at least second target object falls after the holding device approaches the container from the position when the holding device images at least one of the first target object and the second target object by the imaging system. The control device according to any one of claims 58 to 60.

62. The imaging result used for generating the first control signal includes the result of imaging at least a part of the plurality of target objects accommodated in the container by the imaging system. The control device according to any one of claims 56 to 61.

63. The imaging result used for generating the first control signal is the imaging result of imaging the target object group by the imaging system. The control device according to claim 62.

64. A control system comprising the control device according to any one of claims 1 to 63 and the imaging system.

65. A robot system comprising the control device according to any one of claims 1 to 63, the imaging system, and the robot.

66. A control method for generating a control signal for controlling at least one of a holding device capable of holding a target object and an imaging system for imaging the target object, and a robot for moving the holding device and the imaging system, the method comprising: generating, as the control signal, a first control signal for controlling at least one of the robot and the holding device such that at least a second target object among a plurality of target objects accommodated in a container falls into the container when a first target object, which is the target object among the target object group, is held by the holding device based on an imaging result obtained by imaging, by the imaging system, a target object group including at least a part of the plurality of target objects accommodated in the container, and the second target object is held by the holding device together with the first target object; and generating, as the control signal, a second control signal for controlling at least one of the robot and the holding device such that at least a fourth target object among the plurality of target objects accommodated in the container falls into the container from a position lower than a position where the at least second target object starts to fall into the container when a third target object, which is one of the plurality of target objects accommodated in the container, is held by the holding device and a fourth target object among the plurality of target objects accommodated in the container is held by the holding device together with the third target object after the at least second target object has fallen into the container based on the first control signal.

67. A computer program for causing a computer to execute the control method according to claim 66.

68. A control device that is provided with a holding device capable of holding a target object and an imaging system that images the target object, and a robot that moves the holding device and the imaging system, and generates a control signal for controlling at least one of the holding device, the control device includes: an arithmetic device that generates the control signal; and a communication device that outputs the control signal generated by the arithmetic device. The arithmetic device is configured to, based on a first imaging result obtained by imaging at least a part of a plurality of the target objects accommodated in a container by the imaging system, after a holding operation by the holding device is performed on the target objects accommodated in the container, based on a holding state of a holding target object group, which is a plurality of the target objects held by the holding device, detected based on a second imaging result obtained by imaging the holding target object group held by the holding device by the imaging system, generate a first control signal for controlling at least one of the robot and the holding device as the control signal.

69. The arithmetic device is configured to, after the holding operation by the holding device is performed on the target objects accommodated in the container based on the first imaging result, generate, as a part of the first control signal, a signal for controlling the robot so that the holding device moves away from a plurality of the target objects accommodated in the container from a position where the holding device holds the holding target object group. The holding state is detected based on the second imaging result obtained by imaging, by the imaging system, the holding target object group held by the holding device that has moved away from a plurality of the target objects accommodated in the container based on the part of the signal of the first control signal. The control device according to claim 68.

70. The part of the signal of the first control signal includes a signal for controlling the robot so that the holding device moves to a position higher than a position where the holding device holds the holding target object group. The control device according to claim 69.

71. The arithmetic unit generates the first control signal for controlling at least one of the robot and the holding device so that at least one of the target objects in the object group to be held falls into the container based on the holding state. The position where at least one of the target objects in the object group to be held starts to fall into the container is closer to the container than the position where the imaging system images the object group to be held to obtain the second imaging result. The control device according to any one of claims 68 to 70.

72. The first control signal includes a signal for controlling at least one of the robot and the holding device so that at least one of the target objects in the object group to be held falls into the container from a position lower than the position where the imaging system images the object group to be held to obtain the second imaging result. The control device according to claim 71.

73. The holding operation includes an operation for holding one of the target objects among the plurality of target objects accommodated in the container with the holding device. The target objects other than the one target object in the object group to be held held by the holding device are extra target objects held by the holding device when holding the one target object by the holding operation. The control device according to any one of claims 68 to 72.

74. The holding operation includes an operation for holding one of the target objects included in a part of the plurality of target objects accommodated in the container imaged by the imaging system as the first imaging result with the holding device. The target objects other than the one target object in the object group to be held held by the holding device are extra target objects held by the holding device when holding the one target object by the holding operation. The control device according to any one of claims 68 to 73.

75. The holding state includes a state in which at least one of the target objects in the object group to be held held by the holding device is indirectly held by the holding device, and a state in which two or more of the target objects in the object group to be held held by the holding device are directly held by the holding device. The control device according to any one of claims 68 to 74.

76. The arithmetic unit discriminates between the indirectly held state and the directly held state based on the second imaging result, and generates the first control signal based on the discrimination result. The control device according to claim 75.

77. The indirectly held state includes a state in which one of the target objects among the target object group held by the holding device holds another target object, and the directly held state includes a state in which each of two or more of the target objects among the target object group is directly held by the holding device. The control device according to claim 75 or 76.

78. When the detected holding state is a state in which one of the target objects held by the holding device holds another target object, the first control signal includes a signal for controlling at least one of the robot and the holding device so that the other target object falls into the container while holding the one target object. The control device according to claim 77.

79. The first control signal includes a signal for controlling the robot so that the other target object falls into the container by moving the holding device while holding the one target object. The control device according to claim 78.

80. When the other target object falls into the container based on the first control signal, the one target object held by the holding device is carried out of the container and released from the holding device outside the container. The control device according to claim 78 or 79.

81. The arithmetic unit generates, as the control signal, a third control signal for controlling at least one of the robot and the holding device so that the one target object held by the holding device is carried out of the container and released from the holding device outside the container. The control device according to any one of claims 78 to 80.

82. The other target object is one of the target objects. When the one target object is a first target object and the other target object is a second target object, the first control signal includes a signal for controlling the robot so that the second target object falls from the first target object into the container by moving the holding device while holding the first target object. The control device according to any one of claims 78 to 81.

83. The control device according to claim 82, wherein moving the holding device includes changing the posture of the holding device.

84. The control device according to claim 83, wherein changing the posture of the holding device by the first control signal includes rotationally moving the reference point of the holding device in a direction from an end of the first target object held by the holding device toward the reference point of the holding device.

85. The control device according to claim 84, wherein the end of the first target object is the portion of the end of the first target object that is farthest from the reference point of the holding device.

86. The control device according to any one of claims 83 to 85, wherein the second target object falling from the first target object by the first control signal includes the second target object moving in a direction different from the direction in which the posture of the holding device is changed based on the first control signal and falling from the first target object.

87. The control device according to any one of claims 83 to 86, wherein the second target object falling from the first target object by the first control signal includes the second target object moving in a direction different from the direction in which the holding device is located by changing the posture of the holding device based on the first control signal and falling from the first target object.

88. The control device according to any one of claims 82 to 87, wherein the first control signal includes a signal for controlling the robot so that the second target object falls from the first target object at a first falling height by moving the holding device while holding the first target object.

89. The control device according to claim 88, wherein the first falling height is the height of the second target object when falling from the first target object, the height of the first target object when the second target object falls from the first target object, or the height of the holding device when the second target object falls from the first target object.

90. The control device according to claim 88 or 89, wherein moving the holding device includes changing the height of the holding device.

91. Moving the holding device includes changing the posture of the holding device, and the first control signal includes a signal for controlling the robot so that the second target object falls from the first target object at the first dropping height by changing the height of the holding device according to the amount of change in the posture of the holding device. The control device according to claim 90.

92. The first control signal includes a signal for controlling the robot so that the second target object falls from the first target object at the first dropping height by changing the height of the holding device while changing the posture of the holding device. The control device according to claim 90 or 91.

93. When the detected holding state is a state in which each of two or more of the target objects in the group of target objects held by the holding device is directly held by the holding device, the first control signal includes a signal for controlling at least one of the robot and the holding device so that the group of target objects held by the holding device falls into the container. The control device according to any one of claims 77 to 92.

94. The first control signal includes a signal for controlling the holding device so that the group of target objects held by the holding device falls into the container. The control device according to claim 93.

95. After the group of target objects held by the holding device has fallen into the container based on the first control signal, a target object among the plurality of target objects accommodated in the container is held by the holding device, or at least a part of the plurality of target objects accommodated in the container is imaged by the imaging system. The control device according to claim 93 or 94.

96. The arithmetic unit generates, as the control signal, a third control signal for controlling at least one of the robot and the holding device so that, after the group of target objects held by the holding device has fallen into the container based on the first control signal, a target object among the plurality of target objects accommodated in the container is held by the holding device, or at least a part of the plurality of target objects accommodated in the container is imaged by the imaging system. The control device according to any one of claims 93 to 95.

97. The object group to be held by the holding device is a first object and a second object, and the first control signal includes a signal for controlling at least one of the robot and the holding device so that the first object and the second object held by the holding device fall from the holding device into the container. The control device according to any one of claims 93 to 96.

98. The first control signal includes a signal for controlling at least one of the robot and the holding device so that the first object and the second object held by the holding device fall from the holding device into the container at a first falling height. The control device according to claim 97.

99. The first control signal includes a signal for controlling the robot so that at least one of the holding device, the first object, and the second object is positioned at the first falling height by changing the height of the holding device, and a signal for controlling the holding device so that the first object and the second object fall from the holding device at the first falling height. The control device according to claim 98.

100. The first falling height is the height of at least one of the first object and the second object when the first object and the second object fall, or the height of the holding device when the first object and the second object fall. The control device according to claim 98 or 99.

101. The arithmetic device determines the first falling height based on measurement results of at least a part of a plurality of the objects accommodated in the container by a measuring device. The control device according to any one of claims 88 to 92 and 98 to 100.

102. At the determined first falling height, the position where at least the second object falls into the container is included in a measurement range including at least a part of a plurality of the objects accommodated in the container measured by the measuring device. The control device according to claim 101.

103. The arithmetic device determines the first falling height based on at least one of the height and the quantity of at least a part of a plurality of the objects accommodated in the container, which is calculated based on measurement results of at least a part of a plurality of the objects accommodated in the container by the measuring device. The control device according to claim 101 or 102.

104. The measuring device is the imaging system, and the arithmetic device determines the first drop height based on the first imaging result. The control device according to any one of claims 101 to 103.

105. The measuring device is the imaging system, and after the holding operation by the holding device, the arithmetic device determines the first drop height based on a third imaging result obtained by imaging at least a part of the plurality of target objects accommodated in the container by the imaging system. The control device according to any one of claims 101 to 104.

106. The arithmetic device generates the first control signal based on measurement results of at least a part of the plurality of target objects accommodated in the container by the measuring device. The control device according to any one of claims 68 to 105.

107. The first control signal includes a signal for controlling at least one of the robot and the holding device so that a drop target object, which is at least one of the objects to be held, drops from the first drop height by moving the holding device while holding a first target object, which is at least one of the objects to be held. The arithmetic device determines the first drop height based on the measurement results. The control device according to claim 106.

108. The arithmetic device generates the first control signal for controlling at least one of the robot and the holding device so that a drop target object, which is at least one of the target objects of the group of objects to be held held by the holding device, drops to a desired position in the container based on the measurement results by the measuring device. The control device according to claim 106 or 107.

109. The desired position includes a position where no target object exists in the container. The control device according to claim 108.

110. The arithmetic device generates the first control signal based on at least one of the height and the quantity of at least a part of the plurality of target objects accommodated in the container, which is calculated based on the measurement results by the measuring device. The control device according to any one of claims 106 to 109.

111. The measuring device can directly or indirectly measure at least one of the height and the quantity. The control device according to claim 109 or 110.

112. The imaging system is a first imaging system, the measuring device includes at least one of the first imaging system and a second imaging system different from the first imaging system, and the arithmetic unit calculates at least one of the height or the quantity based on a third imaging result obtained by at least one of the first and second imaging systems. The control device according to claim 103, 110, or 111.

113. The third imaging result includes a result of imaging at least a part of a plurality of the target objects accommodated in the container by at least one of the first and second imaging systems. The control device according to claim 112.

114. The third imaging result includes an imaging result of at least one marker arranged at a position facing a space on the surface of the container where at least a part of a plurality of the target objects is located. The control device according to claim 112 or 113.

115. The measuring device includes a weight sensor capable of measuring the weight of at least a part of a plurality of the target objects accommodated in the container. The control device according to any one of claims 101 to 114.

116. The measuring device includes at least one sensor capable of detecting at least one of the target objects among a plurality of the target objects accommodated in the container. The control device according to any one of claims 101 to 115.

117. The at least one sensor is arranged on the container. The control device according to claim 116.

118. The arithmetic unit determines the first drop height based on the number of the target objects carried out from the container and released from the holding device outside the container. The control device according to any one of claims 88 to 92, 98 to 105, and 107.

119. The arithmetic unit generates the first control signal based on the number of the target objects carried out from the container and released from the holding device outside the container. The control device according to any one of claims 68 to 118.

120. The first control signal includes a signal for controlling at least one of the robot and the holding device so that a falling target object, which is at least one of the target objects in the group of objects to be held held by the holding device, falls into the container, the signal being generated by the arithmetic unit based on the holding state of the group of objects to be held. The control device according to any one of claims 68 to 119.

121. The holding state includes at least one of the position and orientation of at least the falling target object among the group of objects to be held. The control device according to claim 120.

122. The first control signal includes a signal for controlling at least one of the robot and the holding device so that the orientation of the falling target object located in the container by falling becomes a desired orientation, the signal being generated by the arithmetic unit based on the holding state. The control device according to claim 120 or 121.

123. The desired orientation includes an orientation in which a desired part of the falling target object located in the container by falling faces a desired direction. The control device according to claim 122.

124. The desired part includes a holding target part that is at least a part of the target object held by the holding device and is preset. The control device according to claim 123.

125. The holding operation is performed based on the result of a matching process using a model showing at least a part of the shape of the target object and the first imaging result, and the desired part includes a part of the falling target object corresponding to the model. The control device according to claim 123 or 124.

126. The desired part includes a part specified by the user in the falling target object. The control device according to any one of claims 123 to 125.

127. The desired direction includes a direction in which the imaging system can be located to image the falling target object located in the container by falling. The control device according to any one of claims 123 to 126.

128. The desired direction includes a direction in which the desired part of the falling target object located in the container by falling faces directly the imaging system for imaging the falling target object located in the container by falling. The control device according to any one of claims 123 to 127.

129. In the container, an opening is formed through which the holding device enters the container under the control of the robot in order to hold the target object accommodated in the container. The desired direction includes the direction of the opening as viewed from the object to be dropped located in the container by dropping. The control device according to any one of claims 123 to 128.

130. The desired direction includes an upward direction as viewed from the object to be dropped located in the container by dropping. The control device according to any one of claims 123 to 129.

131. The desired direction includes a direction different from the direction in which at least a part of the container exists as viewed from the object to be dropped located in the container by dropping. The control device according to any one of claims 123 to 130.

132. The arithmetic unit generates the first control signal for controlling at least one of the robot and the holding device so that the object to be dropped drops into the container based on the measurement result by the measuring device and the holding state of the group of objects to be held held by the holding device. The control device according to any one of claims 123 to 131.

133. The measuring device includes the imaging system. The measurement result includes the imaging result by the imaging system of at least a part of the plurality of target objects accommodated in the container. The control device according to claim 132.

134. The first control signal includes a first signal for controlling at least one of the robot and the holding device so that an object to be dropped, which is at least one of the group of objects to be held, drops into the container. The arithmetic unit determines whether or not the object to be dropped is held by the holding device based on the measurement result by the measuring device while or after at least one of the robot and the holding device is being controlled based on the first signal. The control device according to any one of claims 68 to 133.

135. The measuring device is the imaging system. The measurement result is the imaging result by the imaging system. The control device according to claim 134.

136. The first signal is a signal for controlling at least one of the robot and the holding device so that the object to be dropped drops into the container. When the arithmetic unit determines that at least one of the objects to be dropped is held by the holding device, the arithmetic unit controls at least one of the robot and the holding device in a control mode different from the control mode based on the first signal so that at least one of the objects to be dropped drops, and generates a second signal for this purpose as a part of the first control signal. The control device according to claim 134 or 135.

137. The control mode includes at least one of the control mode of the robot and the control mode of the holding device. The control mode of the robot includes at least one of the moving direction, moving amount, moving speed, and moving acceleration of the holding device by the control of the robot. The control mode of the holding device includes the holding force of the holding device. The control device according to claim 136.

138. The arithmetic unit generates the first control signal for controlling at least one of the robot and the holding device so that at least one of the objects to be held drops into the container based on the holding state, estimates at least one of the position and orientation of the object that has dropped into the container based on the first control signal based on the second imaging result, and based on at least one of the estimated position and orientation of the object, a holding operation by the holding device is performed on the object. The control device according to any one of claims 1 to 137.

139. The other object is two objects, and the arithmetic unit includes, in the control signal, a signal for controlling the robot and the holding device so that the two objects drop into the container while the one object is held by the holding device. The control device according to any one of claims 77 to 92.

140. The other target objects are the first target object and the second target object. The state in which the other target object is held by the one target object held by the holding device is a state in which the first target object is held by the one target object held by the holding device, and the second target object is held by the first target object held by the one target object. The control signal is a signal for controlling at least one of the robot and the holding device so that the second target object falls from the first target object into the container in a state where the first target object is held by the one target object held by the holding device, and a signal for controlling at least one of the robot and the holding device so that the first target object falls from the one target object into the container in a state where the one target object is held by the holding device. The control device according to claim 139.

141. The group of objects to be held consists of three of the objects. When two or three of the objects in the group of objects to be held are directly held by the holding device, the control signal includes a signal for controlling the robot and the holding device so that the two or three objects directly held by the holding device while holding the one object fall into the container. The control device according to any one of claims 77 and 93 to 120.

142. A control system comprising the control device according to any one of claims 68 to 141 and the imaging system.

143. A robot system comprising the control device according to any one of claims 1 to 141, the imaging system, and the robot.

144. A control method for generating a control signal for controlling at least one of a holding device capable of holding a target object, an imaging system for imaging the target object, a robot for moving the holding device and the imaging system, and the holding device, the method comprising: based on a first imaging result obtained by imaging at least a part of a plurality of the target objects contained in a container by the imaging system, after a holding operation by the holding device is performed on the target objects contained in the container, generating, as the control signal, a first control signal for controlling at least one of the robot and the holding device based on a holding state of a group of target objects to be held, which are a plurality of the target objects held by the holding device, detected based on a second imaging result obtained by imaging the group of target objects to be held by the imaging system.

145. A computer program for causing a computer to execute the control method according to claim 144.

146. A control device for generating a control signal for controlling at least one of a holding device capable of holding a target object, an imaging system for imaging the target object, a robot for moving the holding device and the imaging system, and the holding device, the control device comprising: an arithmetic unit for generating the control signal; and a communication unit for outputting the control signal generated by the arithmetic unit, wherein the arithmetic unit: determines the number of the target objects held by the holding device based on a second imaging result obtained by imaging the target objects held by the holding device, after a holding operation by the holding device is performed on the target objects contained in the container based on a first imaging result obtained by imaging at least a part of a plurality of the target objects contained in the container by the imaging system; and generates the control signal for controlling at least one of the robot and the holding device such that a control mode of at least one of the robot and the holding device is different according to the determined number.

147. The control device according to claim 146, wherein the arithmetic unit generates the control signal for controlling at least one of the robot and the holding device such that the control mode is different when the number is determined to be one and when the number is determined to be plural.

148. When it is determined that the number is plural, the arithmetic unit controls at least one of the robot and the holding device so that the control mode is different between a state in which at least one of the plurality of target objects held in the holding device is indirectly held by the holding device and a state in which two or more of the target objects in the group of target objects to be held held in the holding device are directly held by the holding device. The control device according to claim 146 or 147, which generates a control signal for this purpose.

149. When it is determined that the number is one, the arithmetic unit controls at least one of the robot and the holding device so that one target object held by the holding device is carried out of the container and released from the holding device outside the container. The control device according to any one of claims 146 to 148, which generates a control signal for this purpose.

150. The control mode of the robot includes at least one of a moving direction, a moving amount, a moving speed, and a moving acceleration of the holding device by the control of the robot. The control mode of the holding device includes a holding force of the holding device. The control device according to any one of claims 146 to 149.

151. A control system including the control device according to any one of claims 146 to 150 and the imaging system.

152. A robot system including the control device according to any one of claims 146 to 150, the imaging system, and the robot.

153. A holding device capable of holding an object to be imaged, an imaging system for imaging the object to be imaged, a robot for moving the holding device and the imaging system, and a control method for generating a control signal for controlling at least one of the holding device, the control method comprising: performing a holding operation on the object to be imaged in the container by the holding device based on a first imaging result obtained by imaging at least a part of a plurality of the objects to be imaged contained in the container by the imaging system; determining the number of the objects to be imaged held by the holding device based on a second imaging result obtained by imaging the object to be imaged held by the holding device; and generating the control signal for controlling at least one of the robot and the holding device such that a control mode of at least one of the robot and the holding device is different according to the determined number.

154. A computer program for causing a computer to execute the control method according to claim 153.

Images