Control device, control system, robot system, control method, and computer program

Abstract

This control device: generates a first control signal for controlling a robot such that a holding device approaches a first object, which is one of a group of first objects accommodated in a container, in order to hold the first object, on the basis of an imaging result obtained by imaging the group of first objects from a first imaging height by means of an imaging system; and, after the holding device has approached the first object on the basis of the first control signal and the first object held by the holding device has been released outside the container, generates a second control signal for controlling the robot such that the holding device approaches a second object, which is one of a group of second objects accommodated in the container, in order to hold the second object, on the basis of an imaging result obtained by imaging the second object group from a second imaging height lower than the first imaging height by means of the imaging system.

Description

Control device, control system, robot system, control method, and computer program

[0001] The present invention relates to the technical fields of a control device, a control system, a robot system, a control method, and a computer program that can generate a control signal for controlling a robot, for example.

[0002] An example of a control device that controls a robot equipped with a processing device capable of processing an object based on the image capture results of the object captured by an imaging system is described in Patent Document 1. In such a control device, the imaging system is required to capture an appropriate image of the object.

[0003] US Patent Application Publication No. 2013 / 0230235

[0004] According to a first aspect, there is provided a control device that generates a control signal for controlling a robot that is provided with a holding device capable of holding a target object and an imaging system that images the target object, and that moves the holding device and the imaging system, the control device comprising: a calculation device that generates the control signal; and a communication device that outputs the control signal generated by the calculation device, wherein the calculation device controls the robot to move the holding device toward a first target object, which is one of the target objects in the first target object group, based on an imaging result obtained by the imaging system from a first imaging height to hold the first target object. a control device that generates as the control signal a first control signal for controlling the robot so that the holding device approaches the first target object based on the first control signal, and generates as the control signal a second control signal for controlling the robot so that the holding device approaches the second target object in order to hold the second target object, which is one of the second target object group, based on an imaging result obtained by the imaging system imaging a second target object group including at least a part of the first target object group contained in the container from a second imaging height lower than the first imaging height after the first target object held by the holding device is removed from the container and released from the holding device outside the container.

[0005] According to a second aspect, there is provided a control system including the control device provided by the first aspect and the imaging system.

[0006] According to a third aspect, there is provided a robot system including the control device provided by the first aspect, the imaging system, and the robot.

[0007] According to a fourth aspect, there is provided a control method for generating a control signal for controlling a robot that is provided with a holding device capable of holding a target object and an imaging system that images the target object and that moves the holding device and the imaging system, the control signal including the steps of: generating, as the control signal, a first control signal for controlling the robot so that the holding device approaches a first target object, which is one of the first target object group, in order to hold the first target object, based on an imaging result obtained by imaging a first target object group including at least some of the plurality of target objects contained in a container using the imaging system from a first imaging height; a control method including generating, as the control signal, a second control signal for controlling the robot so that the holding device approaches the first target object in order to hold the second target object, which is one of the second target object group, based on an imaging result obtained by the imaging system imaging a second target object group including at least a part of the first target object group contained in the container from a second imaging height lower than the first imaging height, after the first target object held by the holding device is removed from the container and released from the holding device outside the container.

[0008] According to a fifth aspect, there is provided a computer program that causes a computer to execute the control method provided by the fourth aspect.

[0009] According to a sixth aspect, there is provided a control device that is provided with a processing device that processes target objects and an imaging system that images the target objects, and that generates a control signal for controlling a robot that moves the processing device and the imaging system, the control device comprising: an arithmetic device that generates the control signal; and a communication device that outputs the control signal generated by the arithmetic device, wherein the arithmetic device generates, as the control signal, a first control signal for controlling the robot to move to an imaging height where the imaging system images the target objects, based on an imaging result by the imaging system of a first target object group that includes at least some of the target objects contained in the container, and generates, as the control signal, a second control signal for controlling the robot to approach a first target object, which is one of the second target object group, in order to process the first target object, based on an imaging result by the imaging system of a second target object group that includes at least some of the first target object group contained in the container at the imaging height.

[0010] According to a seventh aspect, there is provided a control system including the control device provided by the sixth aspect and the imaging system.

[0011] According to an eighth aspect, there is provided a robot system including the control device provided by the sixth aspect, the imaging system, and the robot.

[0012] According to a ninth aspect, there is provided a control method for generating a control signal for controlling a robot that is provided with a processing device for processing target objects and an imaging system for imaging the target objects and that moves the processing device and the imaging system, the control method including: generating, as the control signal, a first control signal for controlling the robot to move to an imaging height where the imaging system images the target objects, based on imaging results by the imaging system of a first target object group including at least some of the target objects contained in a container; and generating, as the control signal, a second control signal for controlling the robot to approach a first target object, which is one of the second target object group, in order to process the first target object, based on imaging results obtained by the imaging system of a second target object group including at least some of the first target object group contained in the container at the imaging height.

[0013] According to a tenth aspect, there is provided a computer program for causing a computer to execute the control method provided by the ninth aspect.

[0014] According to an eleventh aspect, there is provided a control device that is provided with a holding device capable of holding a target object and an imaging system that images the target object, and that generates a control signal for controlling a robot that moves the holding device and the imaging system, the control device comprising: an arithmetic unit that generates the control signal; and a communication unit that outputs the control signal generated by the arithmetic unit, wherein a first target object, which is one of a plurality of target objects contained in a container, is held in the holding device, and the robot is controlled so that the holding device holding the first target object moves from the container to a position outside the container at which the first target object is released, and the arithmetic unit generates a first control signal as the control signal for controlling the robot from a position after the first target object has been released from the holding device, based on an imaging result obtained by the imaging system capturing an image of a first group of target objects including at least some of the plurality of target objects contained in the container on a movement path of the imaging system accompanying the movement of the holding device from the container to the position outside the container at which the first target object is released.

[0015] According to a twelfth aspect, there is provided a control system including the control device provided by the eleventh aspect and the imaging system.

[0016] According to a thirteenth aspect, there is provided a robot system including the control device provided by the eleventh aspect, the imaging system, and the robot.

[0017] According to a fourteenth aspect, there is provided a control method for generating a control signal for controlling a robot that is provided with a holding device capable of holding a target object and an imaging system that images the target object and that moves the holding device and the imaging system, wherein a first target object, which is one of a plurality of target objects contained in a container, is held in the holding device, and the robot is controlled so that the holding device holding the first target object moves from the container to a position outside the container at which the first target object is released, the control method including generating a first control signal as the control signal for controlling the robot from a position after the first target object has been released from the holding device based on an imaging result obtained by the imaging system capturing an image of a first group of target objects including at least some of the plurality of target objects contained in the container on a movement path of the imaging system accompanying the movement of the holding device from the container to the position outside the container at which the first target object is released.

[0018] According to a fifteenth aspect, there is provided a computer program for causing a computer to execute the control method provided by the fourteenth aspect.

[0019] According to a 16th aspect, there is provided a control device that is provided with a holding device capable of holding a target object and an imaging system that images the target object, and that generates a control signal for controlling a robot that moves the holding device and the imaging system, the control device comprising: an arithmetic device that generates the control signal; and a communication device that outputs the control signal generated by the arithmetic device, wherein the arithmetic device generates a first control signal as the control signal for controlling the robot after the holding device releases the first target object, based on the imaging results of the imaging system capturing images of the inside of the container during the period from when a first target object, which is one of a plurality of target objects contained in a container, is held in the holding device until the first target object held in the holding device is transported out of the container and released from the holding device outside the container.

[0020] According to a seventeenth aspect, there is provided a control system including the control device provided by the sixteenth aspect and the imaging system.

[0021] According to an eighteenth aspect, there is provided a robot system including the control device provided by the sixteenth aspect, the imaging system, and the robot.

[0022] According to a 19th aspect, there is provided a control method for generating a control signal for controlling a robot that is provided with a holding device capable of holding a target object and an imaging system that images the target object and that moves the holding device and the imaging system, the control method including generating a first control signal as the control signal for controlling the robot after the holding device releases the first target object based on imaging results of the imaging system capturing images of the inside of the container during the period from when a first target object, which is one of a plurality of target objects contained in a container, is held in the holding device until the first target object held in the holding device is transported out of the container and released from the holding device outside the container.

[0023] According to a twentieth aspect, there is provided a computer program that causes a computer to execute the control method provided by the nineteenth aspect.

[0024] According to a 21st aspect, there is provided a control device that is provided with a processing device that processes target objects and an imaging system that images the target objects, and that generates a control signal for controlling a robot that moves the processing device and the imaging system, the control device comprising: an arithmetic device that generates the control signal; and a communication device that outputs the control signal generated by the arithmetic device, wherein the arithmetic device generates, as the control signal, a first control signal for controlling the robot to move to an imaging height where the imaging system images the target objects, based on at least one of the height of a first target object group that includes at least a portion of the multiple target objects contained in a container and the amount of the first target object group contained in the container, and generates, as the control signal, a second control signal for controlling the robot to approach one target object of the second target object group in order to process the one target object of the second target object group, based on an imaging result obtained by the imaging system imaging a second target object group that includes at least a portion of the first target object group contained in the container at the imaging height.

[0025] According to a twenty-second aspect, there is provided a control system including the control device provided by the twenty-first aspect and the imaging system.

[0026] According to a twenty-third aspect, there is provided a robot system including the control device provided by the twenty-first aspect, the imaging system, and the robot.

[0027] According to the 24th aspect, there is provided a control method for generating a control signal for controlling a robot that is provided with a holding device capable of holding a target object and an imaging system that images the target object and that moves the holding device and the imaging system, the control method including generating a first control signal as the control signal for controlling the robot after the holding device releases the first target object based on imaging results of the imaging system capturing images of the inside of the container during the period from when a first target object, which is one of a plurality of target objects contained in a container, is held in the holding device until the first target object held in the holding device is transported out of the container and released from the holding device outside the container.

[0028] According to a twenty-fifth aspect, there is provided a computer program for causing a computer to execute the control method provided by the twenty-fourth aspect.

[0029] The functions and other advantages of the present invention will become apparent from the following detailed description of the preferred embodiments.

[0030] FIG. 1 is a block diagram showing the configuration of a robot system according to this embodiment. FIG. 2 is a side view showing the appearance of a robot according to this embodiment. FIG. 3 is a block diagram showing the configuration of a control device according to this embodiment. FIG. 4 is a flowchart showing the flow of robot control processing. FIGS. 5( a) to 5(d) are side views showing the positional relationship between a robot and a workpiece at a certain point in time during a holding process for holding a workpiece placed on a mounting device moving on a support surface. FIGS. 6( a) to 6(d) are side views showing the positional relationship between a robot and a workpiece at a certain point in time during a release process for placing a workpiece on a mounting device moving on a support surface. FIGS. 7( a) to 7(b) are side views showing the positional relationship between a robot and a workpiece at a certain point in time during a holding process for holding a workpiece placed on a mounting device stationary on a support surface. FIGS. 8( c) to 8(e) are side views showing the positional relationship between a robot and a workpiece at a certain point in time during a release process for placing a workpiece on a mounting device stationary on a support surface. 8(a) to 8(e) are side views showing the positional relationship between the robot and the workpieces at a certain time during a period in which a holding process for holding multiple workpieces placed on the placement device one by one in order and a release process for placing the multiple workpieces one by one on the placement device are being performed. FIGS. 9(a) to 9(e) are side views showing the positional relationship between the robot and the workpieces at a certain time during a period in which a holding process for holding multiple workpieces placed on the placement device one by one in order and a release process for placing the multiple workpieces one by one on the placement device are being performed. FIG. 10 is a cross-sectional view showing multiple target objects contained in a container and an imaging system. FIG. 11 is a cross-sectional view showing a first example of an imaging allowable range set based on the field of view of the imaging system. FIG. 12 is a cross-sectional view showing a second example of an imaging allowable range set based on the field of view of the imaging system. FIG. 13(a) shows the relationship between the field of view and partial field of view of the left camera and the image generated by the left camera, and FIG. 13(b) shows the relationship between the field of view and partial field of view of the right camera and the image generated by the right camera.FIG. 14(a) shows an imaging device that captures an image of a target object included in an allowable imaging range determined based on a search area, and an image generated by the imaging device. FIG. 14(b) shows an imaging device that captures an image of a target object that is not included in an allowable imaging range determined based on a search area, and an image generated by the imaging device. FIG. 15 is a cross-sectional view showing the height of a target object group including multiple target objects contained in a container. FIGS. 16(a) and 16(b) are cross-sectional views showing an example of a method for calculating the height of a target object group. FIG. 17 is a cross-sectional view showing multiple target objects contained in a container and an imaging system. FIGS. 18(a) and 18(b) are cross-sectional views showing multiple target objects contained in a container and an imaging system. FIG. 19 is a cross-sectional view showing multiple target objects contained in a container and an imaging system. FIGS. 20(a) and 20(b) are cross-sectional views showing multiple target objects contained in a container and an imaging system. FIGS. 21(a) and 21(b) are cross-sectional views showing multiple target objects housed in a container and an imaging system. FIGS. 22(a) and 22(b) are cross-sectional views showing multiple target objects housed in a container and an imaging system. FIG. 23 is a cross-sectional view showing multiple target objects housed in a container and an imaging system. FIG. 24 is a flowchart showing the flow of robot control processing in a first modified example. FIGS. 25(a) to 25(c) are cross-sectional views showing an imaging system 2 that images target objects housed in a container after the end effector 4 holds the target objects. FIGS. 26(a) and 26(b) are cross-sectional views showing multiple target objects housed in a container and an imaging system. FIGS. 27(a) and 27(b) are cross-sectional views showing multiple target objects housed in a container and an imaging system. FIGS. 28(a) and 28(b) are cross-sectional views showing multiple target objects housed in a container and an imaging system. Fig. 29 is a cross-sectional view showing a plurality of target objects contained in a container and an imaging system. Fig. 30 is a cross-sectional view showing an imaging system that images target objects. Fig. 31 is a cross-sectional view showing a weight sensor that measures the weight of a target object. Fig. 32 is a cross-sectional view showing a container in which an object detection sensor that detects target objects is arranged. Fig. 33 is a cross-sectional view showing a container in which a marker is arranged.Fig. 34(a) and Fig. 34(b) are cross-sectional views conceptually showing a process for adjusting the horizontal imaging position of the imaging system. Fig. 35(a) and Fig. 35(b) are cross-sectional views conceptually showing a process for adjusting the imaging attitude of the imaging system. Fig. 36 is a side view showing an imaging system provided in a part different from the robot. Fig. 37(a) and Fig. 37(b) are cross-sectional views showing a plurality of target objects contained in a container and the imaging system. Fig. 38 is a block diagram showing an imaging system equipped with a focus adjustment optical system. Fig. 39 is a block diagram showing the configuration of a robot system equipped with a measurement system.

[0031] Next, embodiments of a control device, a control system, a robot system, a control method, and a computer program will be described. Hereinafter, embodiments of a control device, a control system, a robot system, a control method, and a computer program will be described using a robot system SYS.

[0032] (1) Configuration of the Robot System SYS First, the configuration of the robot system SYS will be described.

[0033] (1-1) Overall Configuration of Robot System SYS First, the overall configuration of the robot system SYS will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the overall configuration of the robot system SYS.

[0034] 1, the robot system SYS includes a robot 1, an imaging system 2, a control device 3, and an end effector 4. The imaging system 2 may also be referred to as an imaging unit.

[0035] The robot 1 is a device capable of performing a predetermined process on a target object OBJ. An example of the robot 1 is shown in FIG. 2. FIG. 2 is a side view showing the appearance of the robot 1. As shown in FIG. 2, the robot 1 includes, for example, a base 11, a robot arm 12, and a robot control device 13.

[0036] The base 11 is a component that forms the base of the robot 1. The base 11 is placed on a support surface S, such as a floor surface. The base 11 may be fixed to the support surface S. Alternatively, the base 11 may be movable relative to the support surface S. As an example, the base 11 may be self-propelled on the support surface S. In this case, the base 11 may be installed on an automatic guided vehicle (AGV). When the base 11 is installed on an automatic guided vehicle, the automatic guided vehicle may be considered to be part of the robot 1. Alternatively, the automatic guided vehicle (AGV) may be used as the base 11. Note that FIG. 2 shows an example in which the base 11 is fixed to the support surface S.

[0037] The robot arm 12 is attached to the base 11. The robot arm 12 is a device in which a plurality of links 121 are connected via joints 122. An actuator is built into the joint 122. The link 121 may be rotatable around an axis defined by the joint 122 by the actuator built into the joint 122. At least one link 121 may be extendable and contractible along the direction in which the link 121 extends. A device including the device in which a plurality of links 121 are connected via joints 122 and the base 11 may be referred to as the robot arm 12.

[0038] An end effector 4 is attached to the robot arm 12. That is, the end effector 4 is attached to the robot 1. In the example shown in FIG. 2 , the end effector 4 is attached to the tip of the robot arm 12. The end effector 4 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the end effector 4. That is, the robot 1 moves the end effector 4.

[0039] The end effector 4 is a device that performs a predetermined process (in other words, a predetermined operation) on the target object OBJ. The end effector 4 that performs a predetermined process on the target object OBJ may also be called a processing device.

[0040] For example, the end effector 4 may perform a holding process for holding a target object OBJ as an example of the predetermined process. In this case, the end effector 4 may be considered to be performing the holding process on the target object OBJ that is to be held by the end effector 4. An end effector 4 capable of performing the holding process may be referred to as a holding device.

[0041] For example, holding the target object OBJ may include grasping the target object OBJ. For example, holding the target object OBJ may include gripping the target object OBJ using a hand gripper (described later), which is an example of the end effector 4. Holding the target object OBJ may include attracting the target object OBJ. For example, holding the target object OBJ may include attracting (vacuum suction) the target object OBJ using a vacuum gripper (described later), which is an example of the end effector 4. For example, holding the target object OBJ may include attracting the target object OBJ using a magnetic attraction gripper (described later), which is an example of the end effector 4.

[0042] For example, as an example of the predetermined process, the end effector 4 may perform a release process (in other words, a release operation) to release (i.e., let go of) the target object OBJ that it is holding. In this case, the end effector 4 may be considered to be performing the release process on the target object OBJ that it is holding. An end effector 4 that is capable of performing the release process may be referred to as a release device.

[0043] A hand gripper is an example of an end effector 4 capable of holding and releasing an object OBJ. A hand gripper is an end effector 4 that can hold a target object OBJ by physically pinching the target object OBJ using multiple (e.g., two, three, or four) finger or claw members. Another example of an end effector 4 capable of holding and releasing an object OBJ is a vacuum gripper. A vacuum gripper is an end effector 4 that can hold a target object OBJ by vacuum-suctioning the target object OBJ. Another example of an end effector 4 capable of holding and releasing an object OBJ is a magnetically attractive gripper. FIG. 2 illustrates an example in which the end effector 4 is a hand gripper. However, the end effector 4 capable of holding and releasing an object OBJ may be any other existing end effector.

[0044] The robot 1 may perform, as an example of a predetermined process, a placement process (in other words, a placement operation) for placing a target object OBJ at a desired position using the end effector 4 capable of performing a holding process and a release process. For example, the robot 1 may hold a first target object OBJ using the end effector 4, and then perform a placement process for placing the first target object OBJ held by the end effector 4 at a desired position of a second target object OBJ different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process for the second target object OBJ on which the end effector 4 is to place the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process for the first target object OBJ to be released by the end effector 4.

[0045] The robot 1 may use the end effector 4 capable of performing a holding process and a release process to perform a fitting process (in other words, a fitting operation) for fitting a first target object OBJ into a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation). For example, the robot 1 may hold the first target object OBJ using the end effector 4, and then perform a fitting process for fitting the first target object OBJ held by the end effector 4 into a second target object OBJ different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process on the second target object OBJ into which the end effector 4 should fit the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process on the first target object OBJ to be released by the end effector 4. The fitting process may also be referred to as a processing process.

[0046] As an example, the fitting process may include a process for fitting (i.e., inserting) the first target object OBJ into a hole formed in the second target object OBJ. For example, the robot 1 may hold the first target object OBJ using the end effector 4, and then perform a fitting process for fitting (i.e., inserting) the first target object OBJ held by the end effector 4 into a hole formed in the second target object OBJ. In this case, the end effector 4 may be considered to be performing a release process for the second target object OBJ, which has a hole formed therein into which the first target object OBJ should be fitted (i.e., inserted). Similarly, the end effector 4 may be considered to be performing a release process for the first target object OBJ that the end effector 4 should release.

[0047] The robot 1 may use the end effector 4 capable of performing a holding process and a release process to perform a pasting process (in other words, a pasting operation) for pasting a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation). For example, the robot 1 may hold the first target object OBJ using the end effector 4, and then perform a pasting process for pasting the first target object OBJ held by the end effector 4 to a second target object OBJ different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process on the second target object OBJ to which the end effector 4 is to paste the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process on the first target object OBJ to be released by the end effector 4. The pasting process may also be referred to as a processing process.

[0048] The robot 1 may use the end effector 4 capable of performing a holding process and a release process to perform a bonding process (i.e., a bonding operation) for bonding a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (i.e., a placement operation). For example, the robot 1 may hold the first target object OBJ using the end effector 4, and then perform a bonding process for bonding the first target object OBJ held by the end effector 4 to a second target object OBJ different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process on the second target object OBJ to which the end effector 4 is to bond the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process on the first target object OBJ to which the end effector 4 is to release. The bonding process may also be referred to as a processing process.

[0049] The robot 1 may use the end effector 4 capable of holding and releasing operations to perform a welding process (i.e., a welding operation) for welding a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (i.e., a placement operation). For example, the robot 1 may hold the first target object OBJ using the end effector 4, and then perform a welding process for welding the first target object OBJ held by the end effector 4 to a second target object OBJ different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process on the second target object OBJ to which the end effector 4 is to weld the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process on the first target object OBJ to which the end effector 4 is to release. The welding process may also be referred to as a processing process.

[0050] The robot 1 may use the end effector 4 capable of holding and releasing operations to perform a screw tightening process (i.e., a screw tightening operation) for tightening a first target object OBJ capable of functioning as a screw into a screw hole formed in a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (i.e., a placement operation). For example, the robot 1 may hold the first target object OBJ using the end effector 4, and then perform a screw tightening process for tightening the first target object OBJ held by the end effector 4 into a second target object OBJ different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process for the second target object OBJ into which the end effector 4 is to tighten the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process for the first target object OBJ to be released by the end effector 4. The screw tightening process may also be referred to as a processing process.

[0051] The robot 1 may use the end effector 4 capable of performing holding and release processes to perform a process for discarding a target object OBJ (in other words, a discarding operation) as a specific example of a placement process (in other words, a placement operation). For example, the robot 1 may hold the target object OBJ using the end effector 4, and then perform a discarding process to release (i.e., place) the target object OBJ held by the end effector 4 to a disposal location for discarding the target object OBJ. In this case, the end effector 4 may be considered to be performing a release process on the target object OBJ to be discarded by the end effector 4.

[0052] For example, the end effector 4 may perform a predetermined process on each of multiple target objects OBJ. That is, the end effector 4 may perform the predetermined process on the multiple target objects OBJ in sequence. In this case, the robot 1 may move the end effector 4 to a first position where the end effector 4 can perform the predetermined process on a first target object OBJ, and after the end effector 4 has moved to the first position, the end effector 4 may perform the predetermined process on the first target object OBJ. Thereafter, the robot 1 may move the end effector 4 to a second position where the end effector 4 can perform the predetermined process on a second target object OBJ different from the first target object OBJ, and after the end effector 4 has moved to the second position, the end effector 4 may perform the predetermined process on the second target object OBJ.

[0053] For example, the end effector 4 may perform a predetermined process on each of multiple portions of a single target object OBJ. That is, the end effector 4 may sequentially perform a predetermined process on multiple portions of a single target object OBJ. In this case, the robot 1 may move the end effector 4 to a third position where the end effector 4 can perform a predetermined process on a first portion of the target object OBJ, and after the end effector 4 has moved to the third position, the end effector 4 may perform the predetermined process on the first portion of the target object OBJ. Thereafter, the robot 1 may move the end effector 4 to a fourth position where the end effector 4 can perform a predetermined process on a second portion of the target object OBJ, and after the end effector 4 has moved to the fourth position, the end effector 4 may perform the predetermined process on the second portion of the target object OBJ.

[0054] As an example of an operation in which the robot 1 performs a predetermined process on each of multiple portions of a single target object OBJ, the robot 1 may repeatedly perform a fitting process on a second target object OBJ having multiple holes formed therein, fitting the first target object OBJ into each hole. Note that a portion of the target object OBJ on which the end effector 4 performs a predetermined process may be referred to as a target portion or a target member. For example, each portion (each member) of the second target object OBJ in which each hole is formed may be referred to as a target portion or a target member. Note that when the target object OBJ is composed of multiple parts, each part may be referred to as a target portion or a target member. When the target object OBJ is composed of multiple members, each member may be considered as a target portion or a target member.

[0055] As shown in FIG. 2 , the target object OBJ on which the end effector 4 performs a predetermined process may include a workpiece W. The workpiece W may include, for example, a part or member used to manufacture a desired product. The workpiece W may include, for example, a part or member to be processed to manufacture the desired product. The workpiece W may include, for example, a part or member to be transported to manufacture the desired product. The workpiece W may include, for example, a part or member that is moving due to transportation to manufacture the desired product. The workpiece W may include, for example, a part or member that is moving to manufacture the desired product.

[0056] As shown in FIG. 2 , the target object OBJ on which the end effector 4 performs a predetermined process may include a placement device T on which a workpiece W is placed. An example of the placement device T is a container (storage box) CB. The container CB may have a bottom wall BS and a side wall SS protruding upward from the bottom wall BS. The placement device T may be a placement device T in which the workpiece W is placed on the bottom wall BS in a storage space SP enclosed by the bottom wall BS and the side wall SS. The container CB may also have a bottom wall BS and a side wall SS protruding upward from the bottom wall BS. The placement device T may be a placement device T capable of accommodating the workpiece W in the storage space SP enclosed by the bottom wall BS and the side wall SS. However, the container CB may not have a side wall SS. A container CB without a side wall SS may be referred to as a pallet. The placement device T may also be a pallet. Furthermore, the placement device T is not limited to a container CB or a pallet, but may be an existing object on which a workpiece W can be placed. The mounting device T may be referred to as a mounting member. The mounting device T may be disposed on a support surface S. The mounting device T may be fixed to the support surface S. Alternatively, at least a portion of the mounting device T may be movable relative to the support surface S.

[0057] A first example of a case in which at least a portion of the placement device T is movable relative to the support surface S is an example in which the placement device T is supported by a transport device that can move (in other words, transport) the placement device T. A first example of the transport device is a transport device that is self-propelled on the support surface S. In this case, the placement device T may be placed on the self-propelled transport device or may be held by the self-propelled transport device. Note that the self-propelled transport device may also be referred to as an automatic guided vehicle (AGV). A second example of the transport device is a belt conveyor. In this case, the placement device T may be placed on a transport belt that is part of the belt conveyor and is movable relative to the support surface S. A third example of the transport device is a flying device that can fly above the support surface S. In this case, the placement device T may be placed on the flying transport device or may be held by the flying transport device. The flying transport device may be referred to as an unmanned aerial vehicle or a drone. A fourth example of the transport device is another robot arm that is different from the robot arm 12 and that is equipped with an end effector that can hold the mounting device T. In this case, the mounting device T may be held by the end effector attached to the other robot arm.

[0058] A second example of a placement device T in which at least a portion thereof is movable relative to the support surface S is where the placement device T itself is movable relative to the support surface S. A first example of a movable placement device T is a placement device T that is self-propelled on the support surface S. The self-propelled placement device T may be referred to as an automatic guided vehicle (AGV). A second example of a movable placement device T is a belt conveyor. In this case, a transport belt that is part of the belt conveyor and on which the workpiece W is placed may move relative to the support surface S. A third example of a movable placement device T is a placement device T that can fly above the support surface S. In this case, the placement device T may be movable relative to the support surface S by flying above the support surface S. The flyable placement device T may be referred to as an unmanned aerial vehicle or a drone. A fourth example of the movable mounting device T is another robot arm that is different from the robot arm 12 and is equipped with an end effector that can hold the workpiece W. In this case, the end effector attached to the other robot arm may move relative to the support surface S.

[0059] When the transport device moves the placement device T, the workpiece W placed on the placement device T also moves relative to the support surface S. For this reason, a transport device that can move the placement device T may be considered equivalent to a transport device that can transport the workpiece W. Similarly, when the placement device T moves, the workpiece W placed on the placement device T also moves relative to the support surface S. For this reason, the movable placement device T may be considered equivalent to a moving device that can move the workpiece W (in other words, a transport device that can transport the workpiece W). Note that FIG. 2 shows an example in which the placement device T is self-propelled on the support surface S.

[0060] In addition, when at least a part of the placement device T is movable relative to the support surface S, the first placement device T on which the workpiece W is placed may be placed on a second placement device T that is movable relative to the support surface S. In this case, the device including the first placement device T and the second placement device T may be referred to as the placement device T.

[0061] However, the target object OBJ does not have to include the placement device T. Furthermore, the workpiece W does not have to be placed on the placement device T, and the placement device T may not exist. For example, the workpiece W may be placed on a support surface S.

[0062] When the target object OBJ includes a workpiece W and a mounting device T, the holding process described above may include a process of holding a workpiece W placed on a stationary or moving mounting device T. The holding process described above may include a process of holding a workpiece W placed on a support surface S. The release process described above may include a process of releasing a workpiece W held by the end effector 4 in order to place the workpiece W held by the end effector 4 at a desired position on a stationary or moving mounting device T. The release process described above may include a process of releasing a workpiece W held by the end effector 4 in order to place the workpiece W held by the end effector 4 at a desired position on the support surface S. The release process described above may include a process of releasing a first workpiece W held by the end effector 4 in order to fit the first workpiece W held by the end effector 4 into a second workpiece W placed on a stationary or moving mounting device T. The above-described release process may include a process of releasing a first workpiece W held by the end effector 4 in order to fit the first workpiece W held by the end effector 4 into a second workpiece W placed on the support surface S. The above-described release process may include a process of releasing the first workpiece W held by the end effector 4 in order to fit (i.e., insert) the first workpiece W held by the end effector 4 into a hole formed in a second workpiece W placed on a stationary or moving mounting device T. The above-described release process may include a process of releasing the first workpiece W held by the end effector 4 in order to fit (i.e., insert) the first workpiece W held by the end effector 4 into a hole formed in the second workpiece W placed on the support surface S. Note that the target object OBJ may include the workpiece W or the mounting device T.

[0063] The robot control device 13 controls the operation of the robot 1 .

[0064] Specifically, the robot control device 13 may control the movement of the robot arm 12. For example, the robot control device 13 may control the movement of the robot arm 12 so that a desired link 121 rotates around an axis defined by a desired joint 122. For example, the robot control device 13 may control the movement of the robot arm 12 so that the end effector 4 attached to the robot arm 12 is positioned at a desired position. For example, the robot control device 13 may control the movement of the robot arm 12 so that the end effector 4 attached to the robot arm 12 moves to a desired position.

[0065] In addition to controlling the operation of the robot 1, the robot control device 13 may also control the operation of the end effector 4 attached to the robot 1. For example, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 holds the target object OBJ at a desired timing. That is, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 performs a holding process at a desired timing. For example, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 releases the held target object OBJ at a desired timing. That is, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 performs a release process at a desired timing. If the end effector 4 is a hand gripper, the robot control device 13 may control the timing of opening and closing the hand gripper. If the end effector 4 is a vacuum gripper, the robot control device 13 may control the timing of turning on and off a vacuum device of the vacuum gripper. If the end effector 4 is a magnetic attraction gripper, the robot control device 13 may control the timing of turning on and off the magnetic attraction device of the magnetic attraction gripper.

[0066] Note that FIG. 2 shows an example in which the robot 1 is a robot arm 12 (i.e., a vertical articulated robot). However, the robot 1 may be a robot different from a vertical articulated robot. For example, the robot 1 may be a SCARA robot (i.e., a horizontal articulated robot). For example, the robot 1 may be a parallel link robot. For example, the robot 1 may be a dual-arm robot having two robot arms 12. For example, the robot 1 may be a Cartesian coordinate robot. For example, the robot 1 may be a cylindrical coordinate robot. The robot 1 may be referred to as a movable device. The movable device may include, in addition to the robot 1, at least one of an automatic guided vehicle and an unmanned aerial vehicle. For example, the robot 1 may be installed in at least one of an automatic guided vehicle and an unmanned aerial vehicle.

[0067] 1 again, the imaging system 2 captures an image of a target object OBJ. To capture an image of the target object OBJ, the imaging system 2 includes an imaging device 21, an imaging device 22, and a projection device 23. The imaging system 2 may also be referred to as an imaging unit.

[0068] The imaging device 21 is a camera capable of capturing an image of the target object OBJ. In this embodiment, the imaging device 21 is a monocular camera. The imaging device 21 captures an image of the target object OBJ to generate image data IMG_2D. That is, the imaging device 21 generates image data IMG_2D, which is the result of capturing the image of the target object OBJ. The image data IMG_2D generated by the imaging device 21 is output from the imaging device 21 to the control device 3. As a result, the control device 3 acquires the image data IMG_2D acquired by the imaging device 21 capturing the image of the target object OBJ. In this embodiment, the imaging device 21 is a monocular camera. Specifically, the imaging device 21 is capable of capturing an image of the target object OBJ using a monocular camera (in other words, an image sensor). Specifically, because the imaging device 21 is a monocular camera, the imaging device 21 generates one piece of image data generated by the monocular camera as the image data IMG_2D. In this case, the image data IMG_2D corresponding to one image data (i.e., showing one image) may be referred to as monocular image data. The image capturing device 21 is not limited to a monocular camera. The image capturing device 21 may be a stereo camera capable of capturing an image of the target object OBJ using two monocular cameras, or may include three or more monocular cameras. The image capturing device 21 may be at least one of a light field camera, a plenoptic camera, and a multispectral camera.

[0069] Like the imaging device 21, the imaging device 22 is a camera capable of capturing an image of the target object OBJ. In this embodiment, the imaging device 22 is a stereo camera. Specifically, the imaging device 22 is a stereo camera capable of capturing an image of the target object OBJ using two monocular cameras (in other words, two image sensors). The imaging device 22 captures an image of the target object OBJ to generate image data IMG_3D. That is, the imaging device 22 generates image data IMG_3D that is the result of capturing the image of the target object OBJ. Specifically, because the imaging device 22 is a stereo camera, the imaging device 22 generates image data IMG_3D that includes two pieces of image data generated by the two monocular cameras. In this case, the image data IMG_3D that includes the two pieces of image data (i.e., shows two images) may be referred to as stereo image data. The image data IMG_3D generated by the imaging device 22 is output from the imaging device 22 to the control device 3. As a result, the control device 3 acquires image data IMG_3D acquired by capturing an image of the target object OBJ using the imaging device 22. Note that the imaging device 22 is not limited to a stereo camera. The imaging device 22 may be a monocular camera or may include three or more monocular cameras. Note that the imaging device 22 may be at least one of a light field camera, a plenoptic camera, and a multispectral camera.

[0070] The projection device 23 is a device capable of irradiating projection light onto the target object OBJ. In particular, the projection device 23 is a device capable of projecting a desired projection pattern onto the target object OBJ by irradiating the target object OBJ with projection light. The projection pattern may also be referred to as the intensity distribution of light projected onto the target object OBJ. If the intensity distribution of the projected light changes, the projection pattern also changes. The desired projection pattern may include, for example, a random pattern. The random pattern may be a projection pattern having a different pattern for each unit irradiation area. The random pattern may include a random dot pattern. The desired projection pattern may include, for example, a one-dimensional or two-dimensional grid pattern. The desired projection pattern may include, for example, a line pattern. The desired projection pattern may include, for example, a stripe pattern. The desired projection pattern may include other projection patterns. The imaging device 22 captures an image of the target object OBJ onto which the projection pattern from the projection device 23 is projected. In this case, the image represented by the image data IMG_3D includes a target object OBJ onto which the projection pattern is projected. On the other hand, the imaging device 21 does not need to capture an image of the target object OBJ onto which the projection pattern is projected. The imaging device 21 may capture an image of a target object OBJ onto which no projection pattern is projected. In this case, the image represented by the image data IMG_2D does not need to include a target object OBJ onto which the projection pattern is projected. The image represented by the image data IMG_2D may include a target object OBJ onto which no projection pattern is projected. The projection light for projecting a desired projection pattern onto the target object OBJ may be referred to as pattern light or structured light. In this case, the projection light may include pattern light or structured light. The projection light may be light of a uniform projection pattern (i.e., a uniform intensity distribution).

[0071] Note that the projection device 23 may be considered to illuminate the target object OBJ with the projection light by irradiating the target object OBJ with the projection light. In this case, the projection device 23 may function as an illumination device that illuminates the target object OBJ. When the projection device 23 functions as an illumination device, the projection light may be referred to as illumination light. The projection light (illumination light) used as an illumination device may be light with a uniform intensity distribution. When the projection device 23 functions as an illumination device, the projection light does not need to be light that can project a desired projection pattern onto the target object OBJ. When the projection device 23 functions as an illumination device, the projection light (i.e., illumination light) may be any light that can illuminate the target object OBJ.

[0072] The imaging device 21 may image the entire target object OBJ. Alternatively, the imaging device 21 may image a portion of the target object OBJ. That is, the imaging device 21 may image a portion of the target object OBJ while not imaging another portion of the target object OBJ. Similarly, the imaging device 22 may image the entire target object OBJ. Alternatively, the imaging device 22 may image a portion of the target object OBJ. That is, the imaging device 22 may image a portion of the target object OBJ while not imaging another portion of the target object OBJ.

[0073] The imaging device 21 may capture an image of a single target object OBJ. In other words, a single target object OBJ may appear in the image represented by the image data IMG_2D. Alternatively, the imaging device 21 may capture an image of multiple target objects OBJ. In other words, a plurality of target objects OBJ may appear in the image represented by the image data IMG_2D. In this case, as will be described in detail later, the control device 3 may determine (in other words, select) one of the multiple target objects OBJ captured by the imaging device 21 as the target object OBJ on which the end effector 4 will actually perform a predetermined process. Note that one of the multiple target objects OBJ captured by the imaging device 21 on which the end effector 4 will actually perform a predetermined process may be referred to as a process execution object.

[0074] The imaging device 22 may capture an image of a single target object OBJ. In other words, a single target object OBJ may appear in the image represented by the image data IMG_3D. Alternatively, the imaging device 22 may capture an image of multiple target objects OBJ. In other words, a plurality of target objects OBJ may appear in the image represented by the image data IMG_3D. In this case, as will be described in detail later, the control device 3 may determine (in other words, select) one of the multiple target objects OBJ captured by the imaging device 22 as the target object OBJ on which the end effector 4 will actually perform a predetermined process. Note that one of the multiple target objects OBJ captured by the imaging device 22 on which the end effector 4 will actually perform a predetermined process may be referred to as a process execution object.

[0075] When each of the imaging devices 21 and 22 captures an image of a plurality of target objects OBJ, the plurality of target objects OBJ captured by the imaging device 21 may be the same as the plurality of target objects OBJ captured by the imaging device 22. That is, the plurality of target objects OBJ captured in the image represented by the image data IMG_2D may be the same as the plurality of target objects OBJ captured in the image represented by the image data IMG_3D. Alternatively, at least one of the plurality of target objects OBJ captured by the imaging device 21 may be different from at least one of the plurality of target objects OBJ captured by the imaging device 22. That is, at least one of the plurality of target objects OBJ captured in the image represented by the image data IMG_2D may be the same as at least one of the plurality of target objects OBJ captured in the image represented by the image data IMG_3D.

[0076] The multiple target objects OBJ captured by at least one of the imaging devices 21 and 22 may be arranged such that at least two of the multiple target objects OBJ at least partially overlap. As an example, if the target objects OBJ are workpieces W corresponding to parts used to manufacture a desired product, the multiple workpieces W (i.e., multiple parts) may be arranged such that at least two of the multiple workpieces W at least partially overlap. In this case, the robot 1 may perform bulk picking, in which the robot 1 picks workpieces W one by one from the multiple workpieces W that are arranged in a disorderly manner.

[0077] The imaging system 2 is attached to the robot arm 12, similar to the end effector 4. That is, the imaging devices 21 and 22 and the projection device 23 are attached to the robot arm 12. For example, as shown in FIG. 2 , the imaging devices 21 and 22 and the projection device 23 may be attached to the tip of the robot arm 12, similar to the end effector 4. In this case, the imaging devices 21 and 22 and the projection device 23 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the imaging devices 21 and 22 and the projection device 23.

[0078] However, as will be described in detail later in a sixth modified example, the imaging system 2 does not have to be attached to the robot arm 12. The imaging system 2 may be provided at any position where it can irradiate the target object OBJ with projection light and capture an image of the target object OBJ. In this case, for example, the imaging system 2 may be attached to a structure such as a pillar so that it can irradiate the target object OBJ with projection light and capture an image of the target object OBJ. Note that at least one of the imaging devices 21, 22, and projection device 23 may be attached to the robot arm 12, and at least one of the imaging devices 21, 22, and projection device 23 may be attached to a location different from the robot arm 12 (for example, a structure such as a pillar). When at least one of the imaging devices 21 and 22 is attached to a location different from the robot arm 12, at least one of the imaging devices 21 and 22 may be attached to a structure such as a pillar arranged so that it can capture an image of the target object OBJ. Furthermore, when the projection device 23 is attached to a location other than the robot arm 12, the projection device 23 may be attached to a structure such as a pillar arranged so that it can irradiate the projection light onto the target object OBJ.

[0079] The imaging device 21 may capture an image of the target object OBJ while the imaging device 21 and the target object OBJ are displaced relative to each other. Note that the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may refer to a state in which the relative positional relationship between the imaging device 21 and the target object OBJ is changing. The state in which the imaging device 21 and the target object OBJ are displaced relative to each other may refer to a state in which the imaging device 21 and the target object OBJ are moving relative to each other. For example, the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the target object OBJ is moving relative to the imaging device 21. For example, the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the imaging device 21 is moving relative to the target object OBJ. In this case, the imaging device 21 does not need to be stationary to capture an image of the target object OBJ, and the robot system SYS can efficiently perform a predetermined process on the target object OBJ using the end effector 4.

[0080] Alternatively, the imaging device 21 may capture an image of the target object OBJ during a period in which there is no relative displacement between the imaging device 21 and the target object OBJ. Note that the state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the relative positional relationship between the imaging device 21 and the target object OBJ is not changing. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which there is no relative movement between the imaging device 21 and the target object OBJ. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the imaging device 21 and the target object OBJ are stationary. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the imaging device 21 and the target object OBJ are moving at the same moving speed in the same moving direction.

[0081] The imaging device 22 may capture an image of the target object OBJ while the imaging device 22 and the target object OBJ are displaced relative to each other. Note that the state in which the imaging device 22 and the target object OBJ are displaced relative to each other may refer to a state in which the relative positional relationship between the imaging device 22 and the target object OBJ is changing. The state in which the imaging device 22 and the target object OBJ are displaced relative to each other may refer to a state in which the imaging device 22 and the target object OBJ are moving relative to each other. For example, the state in which the imaging device 22 and the target object OBJ are displaced relative to each other may include a state in which the target object OBJ is moving relative to the imaging device 22. For example, the state in which the imaging device 22 and the target object OBJ are displaced relative to each other may include a state in which the imaging device 22 is moving relative to the target object OBJ. In this case, the imaging device 22 does not need to be stationary to capture an image of the target object OBJ, and the robot system SYS can efficiently perform a predetermined process on the target object OBJ using the end effector 4.

[0082] Alternatively, the imaging device 22 may capture an image of the target object OBJ during a period in which there is no relative displacement between the imaging device 22 and the target object OBJ. Note that the state in which there is no relative displacement between the imaging device 22 and the target object OBJ may refer to a state in which the relative positional relationship between the imaging device 22 and the target object OBJ is not changing. The state in which there is no relative displacement between the imaging device 22 and the target object OBJ may refer to a state in which there is no relative movement between the imaging device 22 and the target object OBJ. The state in which there is no relative displacement between the imaging device 22 and the target object OBJ may refer to a state in which the imaging device 22 and the target object OBJ are stationary. The state in which there is no relative displacement between the imaging device 22 and the target object OBJ may refer to a state in which the imaging device 22 and the target object OBJ are moving at the same moving speed in the same moving direction.

[0083] The imaging devices 21 and 22 may capture images of the target object OBJ in synchronization with each other. For example, the imaging devices 21 and 22 may capture images of the target object OBJ simultaneously. That is, the imaging devices 21 and 22 may capture images of the target object OBJ such that the 2D imaging time at which the imaging device 21 captures the target object OBJ and the 3D imaging time at which the imaging device 22 captures the target object OBJ are the same. The imaging devices 21 and 22 may capture images of the target object OBJ such that the 2D imaging time at which the imaging device 21 captures the target object OBJ to generate image data IMG_2D and the 3D imaging time at which the imaging device 22 captures the target object OBJ to generate image data IMG_3D are the same.

[0084] The imaging devices 21 and 22 may capture images of the target object OBJ under the control of the control device 3. In this case, the time (in other words, the timing) at which each of the imaging devices 21 and 22 captures an image of the target object OBJ may be controlled by the control device 3. For example, the control device 3 may control the imaging devices 21 and 22 so that the imaging devices 21 and 22 capture images of the target object OBJ in synchronization with each other. For example, the control device 3 may control the imaging devices 21 and 22 so that the imaging devices 21 and 22 capture images of the target object OBJ simultaneously. In other words, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same time.

[0085] Here, the state in which "the 2D imaging time and the 3D imaging time are the same" may include a state in which "the 2D imaging time and the 3D imaging time are literally exactly the same." The state in which "the 2D imaging time and the 3D imaging time are the same" may include a state in which "the 2D imaging time and the 3D imaging time are not exactly the same, but the time difference between the 2D imaging time and the 3D imaging time is smaller than a first allowable upper limit, so the 2D imaging time and the 3D imaging time can be considered to be substantially the same." Here, the first allowable upper limit may be a first allowable upper limit based on a control error of the robot arm 12. For example, a time difference between the 2D imaging time and the 3D imaging time may cause an error in the calculation result of at least one of the position and orientation of the target object OBJ (described later) (i.e., a case in which the accuracy of at least one of the calculated position and orientation of the target object OBJ is reduced). In this case, an error in the calculation result of at least one of the position and orientation of the target object OBJ may cause a control error of the robot arm 12. A control error of the robot arm 12 may result in a movement error of the end effector 4, which may prevent the end effector 4 from properly performing a predetermined process on the target object OBJ. The first allowable upper limit may be set to an appropriate value that can avoid a situation in which the end effector 4 is unable to properly perform a predetermined process on the target object OBJ due to such a control error of the robot arm 12. The first allowable upper limit may be considered equivalent to the allowable upper limit of the movement error of the end effector 4 by the robot arm 12. Furthermore, even if there is a time difference between the 2D imaging time and the 3D imaging time due to a synchronization error in the imaging processes of the imaging devices 21 and 22, the 2D imaging time and the 3D imaging time may be considered to be substantially the same time. The synchronization error in the imaging processes of the imaging devices 21 and 22 may be a synchronization control error of the imaging processes of the imaging devices 21 and 22 by the control device 3.

[0086] However, the imaging devices 21 and 22 do not have to image the target object OBJ at the same time. In other words, the imaging devices 21 and 22 may image the target object OBJ such that the 2D imaging time at which the imaging device 21 images the target object OBJ and the 3D imaging time at which the imaging device 22 images the target object OBJ are different times. Note that the state in which "the 2D imaging time and the 3D imaging time are different times" may include a state in which "the 2D imaging time and the 3D imaging time cannot be considered to be substantially the same time because the time difference between the 2D imaging time and the 3D imaging time is larger than a first allowable upper limit value."

[0087] In the present embodiment, when the imaging devices 21 and 22 capture images of the target object OBJ during a period in which the imaging devices 21 and 22 are displaced relative to the target object OBJ, the imaging devices 21 and 22 may capture images of the target object OBJ such that the 2D imaging time and the 3D imaging time are the same. In other words, when the imaging devices 21 and 22 capture images of the target object OBJ during a period in which the imaging devices 21 and 22 are displaced relative to the target object OBJ, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same.

[0088] On the other hand, when the imaging devices 21 and 22 capture images of the target object OBJ during a period in which there is no relative displacement between each of the imaging devices 21 and 22 and the target object OBJ, the imaging devices 21 and 22 do not need to capture images of the target object OBJ so that the 2D imaging time and the 3D imaging time are the same. That is, the control device 3 does not need to control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same. For example, the imaging devices 21 and 22 may capture images of the target object OBJ so that the 2D imaging time and the 3D imaging time are different times. That is, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are different times. However, when the imaging devices 21 and 22 capture images of the target object OBJ during a period when there is no relative displacement between the imaging devices 21 and 22 and the target object OBJ, the imaging devices 21 and 22 may capture images of the target object OBJ so that the 2D imaging time and the 3D imaging time are the same. In other words, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same.

[0089] The control device 3 performs robot control processing. The robot control processing includes processing for generating a robot control signal for controlling the robot 1. Specifically, the control device 3 generates the robot control signal based on at least one of the image data IMG_2D and IMG_3D output from the imaging system 2. In this embodiment, the control device 3 calculates at least one of the position and orientation of the target object OBJ in the global coordinate system of the robot system SYS based on at least one of the image data IMG_2D and IMG_3D, and generates the robot control signal based on the calculated at least one of the position and orientation of the target object OBJ.

[0090] The global coordinate system is a coordinate system that serves as the reference for the robot system SYS. For example, the global coordinate system may be a coordinate system that serves as the reference for the robot 1. The global coordinate system can also be said to be a coordinate system used to control the robot 1. For example, the global coordinate system may be a world coordinate system that is determined based on the support surface S on which the robot 1 is placed. In other words, the global coordinate system may be a world coordinate system that is fixed with respect to the support surface S on which the robot 1 is placed.

[0091] However, the control device 3 may calculate at least one of the position and orientation of the target object OBJ in a coordinate system different from the global coordinate system based on at least one of the image data IMG_2D and IMG_3D.

[0092] As a first example, the control device 3 may calculate at least one of the position and orientation of the target object OBJ in the robot coordinate system based on at least one of the image data IMG_2D and IMG_3D. The robot coordinate system may be a coordinate system defined based on the robot 1. In other words, the robot coordinate system may be a coordinate system fixed with respect to the robot 1 (for example, fixed with respect to the base 11 of the robot 1). Since the robot coordinate system is a coordinate system defined based on the robot 1, it may be referred to as a global coordinate system. The robot coordinate system can also be said to be a coordinate system used to control the robot 1.

[0093] As a second example, the control device 3 may calculate at least one of the position and orientation of the target object OBJ in a 2D imaging coordinate system based on at least one of the image data IMG_2D and IMG_3D. The 2D imaging coordinate system may be a coordinate system defined based on the imaging device 21. In other words, the 2D imaging coordinate system may be a coordinate system fixed with respect to the imaging device 21. An example of the 2D imaging coordinate system is a coordinate system defined based on the optical axis AX21 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 21. An example of the 2D imaging coordinate system is a coordinate system in which one of the three coordinate axes constituting the 2D imaging coordinate system is an axis aligned with the optical axis AX21 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 21.

[0094] As a third example, the control device 3 may calculate at least one of the position and orientation of the target object OBJ in a 3D imaging coordinate system based on at least one of the image data IMG_2D and IMG_3D. The 3D imaging coordinate system may be a coordinate system defined based on the imaging device 22. In other words, the 3D imaging coordinate system may be a coordinate system fixed with respect to the imaging device 22. An example of the 3D imaging coordinate system is a coordinate system defined based on the optical axis AX22 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 22. An example of the 3D imaging coordinate system is a coordinate system in which one of the three coordinate axes constituting the 3D imaging coordinate system is an axis aligned with the optical axis AX22 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 22.

[0095] The control device 3 may perform end effector control processing in addition to performing robot control processing. The end effector control processing may include processing for generating an end effector control signal for controlling the end effector 4. Specifically, the control device 3 may generate the end effector control signal based on at least one of the calculated position and orientation of the target object OBJ.

[0096] The end effector control processing may or may not be included in the robot control processing. In other words, the end effector control signal generated by the control device 3 may or may not be included in the robot control signal. In the following description, for convenience of explanation, an example will be described in which the end effector control processing is included in the robot control processing (in other words, the end effector control signal is included in the robot control signal). Therefore, in the following description, the robot control processing may mean processing for generating at least one of a robot control signal and an end effector control signal. In addition, in the following description, the robot control signal may mean at least one of a signal for controlling the robot 1 and a signal for controlling the end effector 4. The robot control signal may also be simply referred to as a control signal.

[0097] In this way, the control device 3 and the imaging system 2 are used to control the robot 1. Therefore, a system including the control device 3 and the imaging system 2 may be referred to as a robot control system or a control system.

[0098] The robot control signal generated by the control device 3 is output to the robot control device 13 of the robot 1. The robot control device 13 controls the operation of the robot 1 based on the robot control signal generated by the control device 3. For this reason, the robot control signal may include a signal for controlling the operation of the robot 1.

[0099] As described above, when the robot control signal includes a signal for controlling the robot arm 12, the robot control device 13 may control the robot arm 12 based on the robot control signal. For example, the robot control device 13 may control the operation of the actuator built into the joint 122 based on the robot control signal, thereby controlling the operation of the robot arm 12.

[0100] For example, as described above, the robot arm 12 moves the end effector 4. In this case, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at a desired position. The robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves to a desired position. The robot control signal may include a signal for controlling the robot arm 12 so that the positional relationship between the end effector 4 and the target object OBJ is a desired positional relationship. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 is located at a desired position. The robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves to a desired position. The robot control device 13 may control the robot arm 12 based on the robot control signal so that the positional relationship between the end effector 4 and the target object OBJ is a desired positional relationship.

[0101] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves toward (i.e., approaches) a holding position where the end effector 4 can hold the target object OBJ. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at the holding position. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves toward (i.e., approaches) the holding position. That is, the robot control signal may control the robot arm 12 so that the end effector 4 is located at the holding position.

[0102] As another example, when performing a release process to release the target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves toward (i.e., approaches) a release position where the target object OBJ held by the end effector 4 should be released. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at the release position. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves toward (i.e., approaches) the release position. That is, the robot control signal may control the robot arm 12 so that the end effector 4 is located at the release position.

[0103] As described above, when the robot control signal includes a signal for controlling the end effector 4, the robot control device 13 may control the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the end effector 4 by controlling the operation of an actuator that moves a hand gripper that constitutes the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the vacuum device of a vacuum gripper that constitutes the end effector 4 based on the robot control signal.

[0104] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the end effector 4 so that the end effector 4 located at the holding position holds the target object OBJ. In this case, the robot control device 13 may control the end effector 4 based on the robot control signal so that the end effector 4 located at the holding position holds the target object OBJ.

[0105] As another example, when performing a release process to release the target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the end effector 4 to release the target object OBJ held by the end effector 4 located at the release position described above. In this case, the robot control device 13 may control the end effector 4, based on the robot control signal, to release the target object OBJ held by the end effector 4 located at the release position.

[0106] The robot control signal may include a signal that can be used as is by the robot control device 13 to control the operation of the robot 1. The robot control signal may include a signal that can be used as is as a robot drive signal that the robot control device 13 uses to control the operation of the robot 1. In this case, the robot control device 13 may use the robot control signal as is to control the operation of the robot 1. For example, the control device 3 may generate a drive signal for an actuator built into the joint 122 of the robot arm 12 as the robot control signal, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the actuator built into the joint 122 of the robot arm 12.

[0107] The robot control signal may include a signal that can be used directly by the robot control device 13 to control the operation of the end effector 4. The robot control signal may include a signal that can be used directly as an end effector drive signal used by the robot control device 13 to control the operation of the end effector 4. In this case, for example, the control device 3 may generate, as the robot control signal, a drive signal (end effector drive signal) for an actuator that moves a hand gripper that constitutes the end effector 4, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the actuator of the end effector 4. For example, the control device 3 may generate, as the robot control signal, a drive signal (end effector drive signal) for driving a vacuum device of a vacuum gripper that constitutes the end effector 4, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the vacuum device of the end effector 4.

[0108] As described above, if the robot control signal includes a signal that can be used directly by the robot control device 13 to control the operation of at least one of the robot 1 and the end effector 4, the robot 1 does not need to be equipped with the robot control device 13.

[0109] Alternatively, the robot control signal may include a signal that can be used by the robot control device 13 to generate a robot drive signal for controlling the operation of the robot 1. In this case, the robot control device 13 may generate a robot drive signal for controlling the operation of the robot 1 based on the robot control signal, and control the operation of the robot 1 based on the generated robot drive signal. For example, the robot control device 13 may generate a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 based on the robot control signal, and control the actuator built into the joint 122 of the robot arm 12 based on the generated robot drive signal.

[0110] The robot control signal may include a signal usable by the robot control device 13 to generate an end effector drive signal for controlling the operation of the end effector 4. In this case, the robot control device 13 may generate an end effector drive signal for controlling the operation of the end effector 4 based on the end effector control signal, and control the operation of the end effector 4 based on the generated end effector drive signal. For example, if the end effector 4 is a hand gripper, the robot control device 13 may generate an end effector drive signal for driving an actuator of the hand gripper based on the robot control signal, and control the actuator of the hand gripper based on the generated end effector drive signal. For example, if the end effector 4 is a magnetically attractive gripper, the robot control device 13 may generate an end effector drive signal for driving a magnetically attractive device of the magnetically attractive gripper based on the robot control signal, and control the magnetically attractive device based on the generated end effector drive signal.

[0111] The signals that the robot control device 13 can use to generate the robot drive signal may include signals that represent at least one of the position and orientation of the target object OBJ in the global coordinate system. In this case, for example, the robot control device 13 may generate, based on the robot control signal, a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 so that the end effector 4 approaches the target object OBJ, whose position and / or orientation in the global coordinate system has been calculated by the robot control signal (i.e., so that the positional relationship between the robot 1 (end effector 4) and the target object OBJ becomes a desired positional relationship), and control the operation of the robot arm 12 based on the generated robot drive signal.

[0112] The signals that the robot controller 13 can use to generate the robot drive signal may include a signal that represents a desired positional relationship between the robot 1 and the target object OBJ in a global coordinate system. In this case, for example, the robot controller 13 may generate a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 based on the robot control signal so that the positional relationship between the robot 1 (end effector 4) and the target object OBJ becomes the desired positional relationship represented by the robot control signal, and control the operation of the robot arm 12 based on the generated robot drive signal.

[0113] The signals that the robot control device 13 can use to generate the robot drive signal may include a signal representing a desired position of the end effector 4 in a global coordinate system. In this case, for example, the robot control device 13 may generate a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 based on the robot control signal so that the end effector 4 is positioned at the desired position represented by the robot control signal (i.e., so that the positional relationship between the robot 1 (end effector 4) and the target object OBJ is the desired positional relationship), and control the operation of the robot arm 12 based on the generated robot drive signal. An example of the desired position is a processing position where the end effector 4 should process the target object OBJ. A specific example of the desired position is a holding position where the end effector 4 should hold the target object OBJ. In this case, the robot control device 13 may generate, based on the robot control signal, a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 so that the end effector 4 moves to the holding position represented by the robot control signal, and may control the operation of the robot arm 12 based on the generated robot drive signal. Another example of a desired position is a release position where the end effector 4 should release the target object OBJ. In this case, the robot control device 13 may generate, based on the robot control signal, a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 so that the end effector 4 is positioned at the release position represented by the robot control signal, and may control the operation of the robot arm 12 based on the generated robot drive signal.

[0114] The signals that the robot controller 13 can use to generate the robot drive signal may include, for example, a signal that represents a desired position of the tip of the robot arm 12 (e.g., a tool center point) in the global coordinate system, or a signal that represents a desired position of the imaging system 2 in the global coordinate system.The signals that the robot controller 13 can use to generate the robot drive signal may also be signals that represent the amount and direction of movement from the current position of the end effector 4 to a desired position of the end effector 4.

[0115] The coordinate system that serves as the reference for the robot control signal may be a coordinate system other than the global coordinate system (for example, a robot coordinate system, a 2D imaging coordinate system, or a 3D imaging coordinate system).

[0116] (1-2) Configuration of the Control Device 3 Next, the configuration of the control device 3 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the control device 3.

[0117] 3 , the control device 3 includes a calculation device 31, a storage device 32, and a communication device 33. The control device 3 may further include an input device 34 and an output device 35. However, the control device 3 does not necessarily have to include at least one of the input device 34 and the output device 35. The calculation device 31, the storage device 32, the communication device 33, the input device 34, and the output device 35 may be connected via a data bus 36.

[0118] The arithmetic device 31 includes, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a field programmable gate array (FPGA). The arithmetic device 31 reads a computer program. For example, the arithmetic device 31 may read a computer program stored in the storage device 32. For example, the arithmetic device 31 may read a computer program stored in a computer-readable, non-transitory storage medium using a storage medium reading device (not shown) included in the control device 3. The arithmetic device 31 may acquire (i.e., download or read) the computer program from a device (not shown) located outside the control device 3 via the communication device 33 (or another communication device). The arithmetic device 31 executes the read computer program. As a result, a logical function block for executing the processing to be performed by the control device 3 (for example, the robot control processing described above) is realized within the arithmetic device 31. In other words, the arithmetic device 31 can function as a controller for realizing the logical function block for executing the processing to be performed by the control device 3.

[0119] A computational model that can be constructed by machine learning may be implemented in the computational device 31 by the computational device executing a computer program. An example of a computational model that can be constructed by machine learning is a computational model including a neural network (so-called artificial intelligence (AI)). In this case, learning of the computational model may include learning of parameters of the neural network (e.g., at least one of a weight and a bias). The computational device 31 may execute a robot control process using the computational model. In other words, the operation of executing the robot control process may include the operation of executing the robot control process using the computational model. Note that a computational model that has been constructed by offline machine learning using teacher data may be implemented in the computational device 31. Furthermore, the computational model implemented in the computational device 31 may be updated by online machine learning on the computational device 31. Alternatively, the calculation device 31 may perform robot control processing using a calculation model implemented in a device external to the calculation device 31 (i.e., a device provided outside the control device 3) in addition to or instead of the calculation model implemented in the calculation device 31.

[0120] The recording medium for recording the computer program executed by the arithmetic unit 31 may be at least one of the following: a CD-ROM, CD-R, CD-RW, a flexible disk, an MO, a DVD-ROM, a DVD-RAM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, and an optical disk such as Blu-ray (registered trademark), a magnetic medium such as a magnetic tape, a magneto-optical disk, a semiconductor memory such as a USB memory, and any other medium capable of storing a program. The recording medium may also include a device capable of recording a computer program (for example, a general-purpose device or a dedicated device in which a computer program is implemented in an executable state in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program may be realized by a logical processing block realized within the arithmetic device 31 (i.e., a computer) when the arithmetic device 31 executes the computer program, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) included in the arithmetic device 31, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.

[0121] 3 shows an example of logical functional blocks realized in the arithmetic device 31 for executing robot control processing. As shown in FIG. 3, a three-dimensional position data generation unit 311, a position and orientation calculation unit 312, and a signal generation unit 313 are realized in the arithmetic device 31. Note that the processes performed by the three-dimensional position data generation unit 311, the position and orientation calculation unit 312, and the signal generation unit 313 will be described in detail later with reference to FIG. 4 etc., and therefore will not be described here. Note that the arithmetic device 31 may also be referred to as a calculation unit.

[0122] The storage device 32 can store desired data. For example, the storage device 32 may temporarily store a computer program executed by the arithmetic device 31. The storage device 32 may temporarily store data that the arithmetic device 31 temporarily uses when the arithmetic device 31 is executing a computer program. The storage device 32 may also store data that the control device 3 stores long-term. The storage device 32 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 32 may include a non-temporary recording medium.

[0123] The communication device 33 is capable of communicating with both the robot 1 and the imaging system 2 via a communication network (not shown). Alternatively, the communication device 33 may be capable of communicating with another device different from the robot 1 and the imaging system 2, in addition to or instead of at least one of the robot 1 and the imaging system 2, via a communication network (not shown). In this embodiment, the communication device 33 may receive (i.e., acquire) at least one of the image data IMG_2D and IMG_3D from the imaging system 2. Furthermore, the communication device 33 may transmit (i.e., output) a robot control signal to the robot 1. The communication device 33 that outputs the robot control signal to the robot 1 may be referred to as an output unit.

[0124] The input device 34 is a device that accepts information input to the control device 3 from outside the control device 3. For example, the input device 34 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by a user of the control device 3. For example, the input device 34 may include a recording medium reading device that can read information recorded as data on a recording medium that can be externally attached to the control device 3.

[0125] It should be noted that information can be input as data to the control device 3 from a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an input device that accepts information input to the control device 3 from outside the control device 3.

[0126] The output device 35 is a device that outputs information to the outside of the control device 3. For example, the output device 35 may output information as an image. That is, the output device 35 may include a display device (a so-called display) that can display an image. For example, the output device 35 may output information as sound. That is, the output device 35 may include an audio device (a so-called speaker) that can output sound. For example, the output device 35 may output information on paper. That is, the output device 35 may include a printing device (a so-called printer) that can print desired information on paper. For example, the output device 35 may output information as data to a recording medium that can be externally attached to the control device 3.

[0127] The control device 3 can output information as data to a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an output device that outputs information to a device external to the control device 3.

[0128] (2) Robot Control Processing Next, the robot control processing performed by the control device 3 will be described.

[0129] (2-1) Flow of Robot Control Processing First, the flow of the robot control processing will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of the robot control processing.

[0130] As shown in FIG. 4 , the control device 3 acquires image data IMG_3D from the imaging device 22 using the communication device 33 (step S1). Specifically, the imaging device 22 captures an image of the target object OBJ at a predetermined 3D imaging rate. For example, the imaging device 22 may capture an image of the target object OBJ at a 3D imaging rate of tens to hundreds of times (e.g., 500 times) per second. Note that, when the target object OBJ is contained in a container CB, capturing an image of the target object OBJ may be considered equivalent to capturing an image of the interior of the container CB (i.e., the above-mentioned storage space SP). Image data resulting from capturing the image of the target object OBJ may be considered equivalent to image data resulting from capturing an image of the interior of the container CB (i.e., the above-mentioned storage space SP). As a result, the imaging device 22 generates image data IMG_3D at a period corresponding to the predetermined 3D imaging rate. For example, the imaging device 22 may generate tens to hundreds (for example, 500) of pieces of image data IMG_3D per second. The control device 3 acquires the image data IMG_3D every time the imaging device 22 generates the image data IMG_3D. In other words, the control device 3 may acquire tens to hundreds (for example, 500) of pieces of image data IMG_3D per second.

[0131] However, the imaging device 22 does not have to periodically capture images of the target object OBJ at a desired 3D imaging rate. For example, the imaging device 22 may capture an image of the target object OBJ when receiving a control signal from the control device 3 to control the imaging device 22 to capture an image of the target object OBJ.

[0132] The imaging device 22 may capture images of objects other than the target object OBJ, in addition to the target object OBJ on which the robot 1 performs a predetermined process. Because the other objects are not targets for the robot 1 to perform a predetermined process, the other objects other than the target object OBJ are referred to as non-target objects in the following description. Examples of non-target objects include at least a portion of the end effector 4, at least a portion of the robot arm 12, and at least one of peripheral objects located around the robot 1. Examples of peripheral objects include at least one of the placement device T and the container CB. For example, if both the target object OBJ and the non-target object are included in the imaging range (field of view) of the imaging device 22, the imaging device 22 may capture images of both the target object OBJ and the non-target object. As a result, the imaging device 22 may generate image data IMG_3D representing an image in which both the target object OBJ and the non-target object are captured. However, the imaging device 22 may capture images of the target object OBJ but not the non-target object. That is, the image capturing device 22 may generate image data IMG_3D representing an image in which the target object OBJ is captured but no non-target objects are captured. In either case, the image capturing device 22 generates image data IMG_3D representing an image in which at least the target object OBJ is captured. That is, the image capturing device 22 generates image data IMG_3D including image data of at least the target object OBJ.

[0133] As described above, the imaging device 22 may capture images of multiple target objects OBJ on which the end effector 4 sequentially performs a predetermined process. In this case, the end effector 4 may perform the predetermined process on a first target object OBJ among the multiple target objects OBJ during a first period, and may perform the predetermined process on a second target object OBJ different from the first target object OBJ among the multiple target objects OBJ during a second period following the first period. In this case, among the multiple target objects OBJ, target objects OBJ other than the first target object OBJ on which the end effector 4 performs the predetermined process may be considered to be target objects OBJ that are not processing objects on which the robot 1 performs the predetermined process during the first period (or non-target objects). Similarly, among the multiple target objects OBJ, target objects OBJ other than the second target object OBJ on which the end effector 4 performs the predetermined process during the second period may be considered to be target objects OBJ that are not processing objects on which the robot 1 performs the predetermined process during the second period (or non-target objects). The first target object OBJ and the second target object OBJ may be different target objects OBJ among the plurality of target objects OBJ captured by the imaging device 22 .

[0134] The control device 3 may further use the communication device 33 to acquire image data IMG_2D from the imaging device 21 in addition to or instead of acquiring image data IMG_3D from the imaging device 22 (step S1). Specifically, the imaging device 21 captures an image of the target object OBJ at a predetermined 2D imaging rate. The 2D imaging rate is the same as the 3D imaging rate. However, the 2D imaging rate may differ from the 3D imaging rate. For example, the imaging device 21 may capture an image of the target object OBJ at a 2D imaging rate that captures an image of the target object OBJ tens to hundreds of times (e.g., 500 times) per second. As a result, the imaging device 21 generates image data IMG_2D at a period corresponding to the predetermined 2D imaging rate. For example, the imaging device 21 may generate tens to hundreds of pieces of image data IMG_2D per second (e.g., 500 pieces). The control device 3 acquires the image data IMG_2D every time the imaging device 21 generates the image data IMG_2D. That is, the control device 3 may acquire several tens to several hundreds (for example, 500) of image data IMG_2D per second.

[0135] However, the imaging device 21 does not have to periodically capture images of the target object OBJ at a desired 2D imaging rate. For example, the imaging device 21 may capture an image of the target object OBJ when receiving a control signal from the control device 3 to control the imaging device 21 to capture an image of the target object OBJ.

[0136] The image data IMG_2D may be referred to as two-dimensional position data indicating the two-dimensional position of the target object OBJ reflected in the image data IMG_2D. The image data IMG_2D (two-dimensional position data) may be data indicating the two-dimensional position of at least a portion of the surface of the target object OBJ. In particular, the image data IMG_2D (two-dimensional position data) is data indicating the two-dimensional position of each of multiple points on the surface of the target object OBJ. For example, the image data IMG_2D (two-dimensional position data) may be data indicating the two-dimensional position of each of multiple points on the surface of the target object OBJ. For example, the image data IMG_2D (two-dimensional position data) may be data indicating the two-dimensional position of each of multiple points corresponding to multiple portions on the surface of the target object OBJ.

[0137] The imaging device 21 may capture an image of a non-target object different from the target object OBJ, in addition to the target object OBJ on which the robot 1 performs a predetermined process. For example, if both the target object OBJ and the non-target object are included in the imaging range (field of view) of the imaging device 21, the imaging device 21 may capture images of both the target object OBJ and the non-target object. As a result, the imaging device 21 may generate image data IMG_2D representing an image in which both the target object OBJ and the non-target object are captured. However, the imaging device 21 may capture an image of the target object OBJ but not the non-target object. In other words, the imaging device 21 may generate image data IMG_2D representing an image in which the target object OBJ is captured but the non-target object is not captured. In either case, the imaging device 21 generates image data IMG_2D representing an image in which at least the target object OBJ is captured. In other words, the imaging device 21 generates image data IMG_2D including image data of at least the target object OBJ.

[0138] As described above, the imaging device 21 may capture images of multiple target objects OBJ on which the end effector 4 sequentially performs a predetermined process. In this case, the end effector 4 may perform a predetermined process on a third target object OBJ among the multiple target objects OBJ during a third period, and then perform a predetermined process on a fourth target object OBJ different from the third target object OBJ among the multiple target objects OBJ during a fourth period following the third period. In this case, during the third period, the target objects OBJ other than the third target object OBJ on which the end effector 4 performs a predetermined process may be considered to be target objects OBJ that are not processing objects on which the robot 1 performs a predetermined process during the third period (or non-target objects). Similarly, during the fourth period, the target objects OBJ other than the fourth target object OBJ on which the end effector 4 performs a predetermined process may be considered to be target objects OBJ that are not processing objects on which the robot 1 performs a predetermined process during the fourth period (or non-target objects). The third target object OBJ and the fourth target object OBJ may be different target objects OBJ among the plurality of target objects OBJ captured by the imaging device 21 .

[0139] When the control device 3 acquires image data IMG_3D, the three-dimensional position data generator 311 generates three-dimensional position data WSD based on the acquired image data IMG_3D each time the control device 3 acquires image data IMG_3D (step S2). Furthermore, the three-dimensional position data generator 311 outputs the generated three-dimensional position data WSD to the position and orientation calculation unit 312. However, if the control device 3 has not acquired image data IMG_3D (for example, if the control device 3 has acquired image data IMG_2D), the three-dimensional position data generator 311 does not need to generate the three-dimensional position data WSD. In this case, the control device 3 does not need to perform the operation of step S2.

[0140] The control device 3 may generate the three-dimensional position data WSD based on each of some image data IMG_3D among the plurality of image data IMG_3D generated by the imaging device 22. The control device 3 may generate the three-dimensional position data WSD without using other some image data IMG_3D among the plurality of image data IMG_3D generated by the imaging device 22.

[0141] The three-dimensional position data WSD is data indicating the three-dimensional position of the target object OBJ captured in the image data IMG_3D. For example, the three-dimensional position data WSD may be data indicating the three-dimensional position of at least a portion of the surface of the target object OBJ. In particular, the three-dimensional position data WSD is data indicating the three-dimensional position of each of multiple points on the target object OBJ. In other words, the three-dimensional position data WSD is data indicating the three-dimensional position of each of multiple points on the target object OBJ captured by the imaging device 22. For example, the three-dimensional position data WSD may be data indicating the three-dimensional position of each of multiple points on the surface of the target object OBJ. For example, the three-dimensional position data WSD may be data indicating the three-dimensional position of each of multiple points corresponding to multiple portions on the surface of the target object OBJ. In the following explanation, unless otherwise specified, the three-dimensional position of the target object OBJ may mean at least one of the three-dimensional position of at least a portion of the surface of the target object OBJ, the three-dimensional position of each of multiple points on the target object OBJ, the three-dimensional position of each of multiple points on the surface of the target object OBJ, and the three-dimensional position of each of multiple points corresponding to multiple parts of the surface of the target object OBJ.

[0142] Specifically, as described above, the image represented by the image data IMG_3D reflects the target object OBJ onto which the projection pattern is projected. In this case, the projection pattern reflected in the image represented by the image data IMG_3D reflects the three-dimensional shape of at least a portion of the surface of the target object OBJ onto which the projection pattern is projected. The shape of the projection pattern reflected in the image represented by the image data IMG_3D reflects the three-dimensional shape of at least a portion of the surface of the target object OBJ onto which the projection pattern is projected. Therefore, the three-dimensional position data generation unit 311 can calculate the three-dimensional shape of at least a portion of the surface of the target object OBJ based on the projection pattern reflected in the image represented by the image data IMG_3D. The three-dimensional shape of at least a portion of the surface of the target object OBJ essentially indicates the three-dimensional position of each of multiple points on the target object OBJ. This is because each of the multiple points on the target object OBJ is contained within the surface of the target object OBJ. Therefore, the process of calculating the three-dimensional shape of at least a part of the surface of the target object OBJ may be considered to be substantially equivalent to the process of calculating the three-dimensional position of each of multiple points on the target object OBJ. Therefore, the three-dimensional position data generator 311 can generate the three-dimensional position data WSD based on the image data IMG_3D.

[0143] Furthermore, considering that each of the multiple points of the target object OBJ is contained within the surface of the target object OBJ, the three-dimensional position data WSD indicating the three-dimensional position of each of the multiple points of the target object OBJ may be considered to be essentially equivalent to three-dimensional shape data indicating the three-dimensional shape of at least a portion of the target object OBJ (in particular, the three-dimensional shape of at least a portion of the surface of the target object OBJ).

[0144] To generate the three-dimensional position data WSD, the three-dimensional position data generation unit 311 may calculate the parallax by associating each portion (e.g., each pixel) of the images represented by each of the two image data included in the image data IMG_3D. Specifically, in this association, the three-dimensional position data generation unit 311 may calculate the parallax by associating each portion of the projection pattern reflected in the images represented by each of the two image data (i.e., each portion of the projection pattern reflected in each image). The three-dimensional position data generation unit 311 may calculate the three-dimensional position of each of multiple points on the target object OBJ using a well-known method based on the principle of triangulation using the calculated parallax. As a result, three-dimensional position data WSD indicating the three-dimensional position of each of the multiple points on the target object OBJ is generated. In this case, the accuracy of the parallax calculation is higher when associating each portion of the images in which the projection pattern is reflected (i.e., each portion of the reflected projection pattern) rather than when associating each portion of the images in which the projection pattern is not reflected. Therefore, the accuracy of the generated three-dimensional position data WSD (that is, the accuracy of calculating the three-dimensional positions of each of the multiple points of the target object OBJ) is improved.

[0145] The three-dimensional position data WSD may be any data as long as it can indicate the three-dimensional position of each of the multiple points of the target object OBJ. In other words, the three-dimensional position data WSD may be any data that directly or indirectly indicates the three-dimensional position of each of the multiple points of the target object OBJ. For example, the three-dimensional position data WSD may include coordinate information indicating the three-dimensional position of each of the multiple points of the target object OBJ. For example, the three-dimensional position data WSD may include information indicating the three-dimensional shape of at least a portion of the target object OBJ.

[0146] An example of three-dimensional position data WSD is depth image data. Depth image data is image data in which depth information is associated with each pixel of a depth image represented by the depth image data, in addition to or instead of brightness information. Depth information is information indicating the distance (i.e., depth) between each part of the target object OBJ reflected in each pixel and the imaging device 22. The distance (i.e., depth) between each part of the target object OBJ reflected in each pixel and the imaging device 22 can be calculated from the above-mentioned parallax. Note that the depth image data may be image data in which the brightness information of each pixel indicates the depth of each part of the target object OBJ (i.e., the distance between each part of the target object OBJ and the imaging device 22). The three-dimensional position data generation unit 311 may calculate the distance between each part of the target object OBJ reflected in the image indicated by the image data IMG_3D and the imaging device 22 based on the projection pattern reflected in the image indicated by the image data IMG_3D, and generate a depth image by associating the calculated distance with each pixel of the image indicated by the image data IMG_3D as depth information.

[0147] Another example of the three-dimensional position data WSD is point cloud data. Point cloud data is data that indicates a collection of points in three-dimensional space that correspond to each part of the target object OBJ that appears in the image indicated by the image data IMG_3D. The three-dimensional position data generator 311 may generate the point cloud data based on the depth image data and the camera parameters of the imaging device 22. Note that the following description will discuss an example in which point cloud data is used as the three-dimensional position data WSD.

[0148] As described above, when the imaging range (field of view) of the imaging device 22 includes an object other than the target object OBJ, in addition to the target object OBJ on which the robot 1 performs a predetermined process, the image data IMG_3D represents an image in which both the target object OBJ and the other object are captured. In this case, the three-dimensional position data WSD may be data indicating the three-dimensional position of the target object OBJ captured in the image data IMG_3D, as well as the three-dimensional position of the other object captured in the image data IMG_3D. In other words, the three-dimensional position data WSD may include data indicating the three-dimensional positions of each of multiple points of the other object captured by the imaging device 22. Even in this case, as long as the three-dimensional position data WSD includes data indicating the three-dimensional position of the target object OBJ, the three-dimensional position data WSD may be considered to be data indicating the three-dimensional positions of each of the multiple points of the target object OBJ. This is because even if the three-dimensional position data WSD contains data indicating the three-dimensional position of another object, the three-dimensional position data WSD still contains data indicating the three-dimensional position of the target object OBJ.

[0149] Thereafter, the position and orientation calculation unit 312 calculates at least one of the position and orientation of the target object OBJ based on at least one of the image data IMG_2D acquired in step S1 and the three-dimensional position data WSD generated in step S2 (step S3). As a result, the position and orientation calculation unit 312 generates position and orientation data POI indicating at least one of the position and orientation of the target object OBJ (step S3). The position and orientation data POI may include position data indicating the position of the target object OBJ. The position and orientation data POI may include orientation data indicating the orientation of the target object OBJ in addition to or instead of the position data indicating the position of the target object OBJ. In other words, the position and orientation data POI may include at least one of position data and orientation data.

[0150] Note that the image data IMG_2D usable for generating the position and orientation data POI corresponds to the imaging result of the imaging device 21. Similarly, the three-dimensional position data WSD usable for generating the position and orientation data POI is generated from image data IMG_3D corresponding to the imaging result of the imaging device 22. Therefore, the position and orientation calculation unit 312 may be considered to calculate the position and orientation data POI based on at least one of the imaging result of the imaging device 21 and the imaging result of the imaging device 22. In other words, the position and orientation calculation unit 312 may be considered to calculate the position and orientation data POI based on the imaging result of the imaging system 2. Also, in the following description, processing (operation) based on the image data IMG_2D may be considered equivalent to processing (operation) based on the imaging result of the imaging device 21 (i.e., the imaging result of the imaging system 2). Processing (operation) based on the image data IMG_3D may be considered equivalent to processing (operation) based on the imaging result of the imaging device 22 (i.e., the imaging result of the imaging system 2). The processing (operation) based on the three-dimensional position data WSD generated from the image data IMG_3D may be considered equivalent to the processing (operation) based on the imaging results of the imaging device 22 (i.e., the imaging results of the imaging system 2).

[0151] In step S3, the position and orientation calculation unit 312 calculates at least one of the position and orientation of the target object OBJ in the global coordinate system. That is, the position and orientation calculation unit 312 generates position and orientation data POI indicating at least one of the position and orientation of the target object OBJ in the global coordinate system. For example, the robot control device 13 may control the robot arm 12 so that the end effector 4 is positioned at a desired position in the global coordinate system. The global coordinate system is a coordinate system defined by an X-axis (GL), a Y-axis (GL), and a Z-axis (GL) that are orthogonal to each other. The X-axis (GL) may be an axis along a horizontal plane. The Y-axis (GL) may be an axis along a horizontal plane. The Z-axis (GL) may be an axis perpendicular to the horizontal plane. The Z-axis (GL) may be an axis extending in the direction of gravity. The X-axis, Y-axis, and Z-axis shown in FIG. 2 may be the X-axis (GL), the Y-axis (GL), and the Z-axis (GL), respectively. The origin of the global coordinate system does not have to be the origin of the X-axis (GL), Y-axis (GL), and Z-axis (GL) shown in Fig. 2. For example, the origin of the global coordinate system may be set at any position on the base 11 in Fig. 2. For example, the origin of the global coordinate system may be set at any position on the support surface S. For example, the origin of the global coordinate system may be set at any position on the contact surface of the base 11 with the support surface S (for example, the center or center of gravity of the contact surface).

[0152] The position and orientation calculation unit 312 may calculate, as the position of the target object OBJ in the global coordinate system, at least one of a position Tx(GL) of the target object OBJ in an X-axis direction (GL) parallel to the X-axis (GL), a position Ty(GL) of the target object OBJ in a Y-axis direction (GL) parallel to the Y-axis (GL), and a position Tz(GL) of the target object OBJ in a Z-axis direction (GL) parallel to the Z-axis (GL). The position and orientation calculation unit 312 may calculate, as the orientation of the target object OBJ in the global coordinate system, at least one of a rotation amount Rx(GL) of the target object OBJ around the X-axis (GL), a rotation amount Ry(GL) of the target object OBJ around the Y-axis (GL), and a rotation amount Rz(GL) of the target object OBJ around the Z-axis (GL). This is because the rotation amount Rx(GL) of the target object OBJ around the X-axis (GL), the rotation amount Ry(GL) of the target object OBJ around the Y-axis (GL), and the rotation amount Rz(GL) of the target object OBJ around the Z-axis (GL) are equivalent to the parameters representing the orientation of the target object OBJ around the X-axis (GL), the parameters representing the orientation of the target object OBJ around the Y-axis (GL), and the parameters representing the orientation of the target object OBJ around the Z-axis (GL), respectively. For this reason, in the following description, the rotation amount Rx(GL) of the target object OBJ around the X-axis (GL), the rotation amount Ry(GL) of the target object OBJ around the Y-axis (GL), and the rotation amount Rz(GL) of the target object OBJ around the Z-axis (GL) will be referred to as the orientation Rx(GL) of the target object OBJ around the X-axis (GL), the orientation Ry(GL) of the target object OBJ around the Y-axis (GL), and the orientation Rz(GL) of the target object OBJ around the Z-axis (GL), respectively.

[0153] The orientation Rx(GL) of the target object OBJ around the X-axis (GL), the orientation Ry(GL) of the target object OBJ around the Y-axis (GL), and the orientation Rz(GL) of the target object OBJ around the Z-axis (GL) may be considered to indicate the position of the target object OBJ in the rotation direction around the X-axis (GL), the position of the target object OBJ in the rotation direction around the Y-axis (GL), and the position of the target object OBJ in the rotation direction around the Z-axis (GL), respectively. In other words, the orientation Rx(GL) of the target object OBJ around the X-axis (GL), the orientation Ry(GL) of the target object OBJ around the Y-axis (GL), and the orientation Rz(GL) of the target object OBJ around the Z-axis (GL) may all be considered to be parameters representing the position of the target object OBJ.

[0154] In this way, in step S3 of FIG. 4, the position and orientation calculation unit 312 may calculate at least one of the position Tx(GL), position Ty(GL), position Tz(GL), orientation Rx(GL), orientation Ry(GL), and orientation Rz(GL) as at least one of the position and orientation of the target object OBJ in the global coordinate system.

[0155] 4, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ by performing a matching process using at least one of the image data IMG_2D and the three-dimensional position data WSD. For convenience of explanation, the following description will mainly focus on an example in which the position and orientation calculation unit 312 calculates at least one of the position and orientation of the target object OBJ by performing a matching process using the three-dimensional position data WSD. Note that in the following description, the matching process using the three-dimensional position data WSD will be referred to as a 3D matching process.

[0156] Specifically, the position and orientation calculation unit 312 may perform 3D matching processing using the three-dimensional position data WSD and three-dimensional model data WMD indicating a three-dimensional model that serves as a reference for the target object OBJ. In this case, the 3D matching processing may be considered to be, for example, a matching processing using a point cloud indicated by the three-dimensional position data WSD and a three-dimensional model indicated by the three-dimensional model data. Note that the 3D matching processing itself may be the same as an existing matching processing. For example, the position and orientation calculation unit 312 may perform the 3D matching processing using a well-known method including at least one of RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), ICP (Iterative Closest Point), and DSO (Direct Sparse Odometry).

[0157] The three-dimensional model based on the three-dimensional position data WSD may include a point cloud model, which is a three-dimensional model that indicates the three-dimensional shape of the target object OBJ using the point cloud indicated by the three-dimensional position data WSD. The three-dimensional model based on the three-dimensional position data WSD may include a polygon model or a mesh model that is generated by connecting multiple points that make up the point cloud indicated by the three-dimensional position data WSD with surfaces (particularly, polygonal surfaces with each point as a vertex). The three-dimensional model based on the three-dimensional position data WSD may include a solid model that is generated by connecting multiple points that make up the point cloud indicated by the three-dimensional position data WSD with smooth surfaces.

[0158] The three-dimensional model data WMD is data representing a template model TM3, which is a three-dimensional model of the target object OBJ. In other words, the three-dimensional model data WMD is data representing the template model TM3, which is a three-dimensional model having a three-dimensional shape that serves as a reference for the target object OBJ. The template model may be a CAD model of the target object OBJ. The template model may be a three-dimensional model having a shape similar to the three-dimensional shape of the target object OBJ obtained by measuring the three-dimensional shape of the actual target object OBJ in advance. The template model may be a polygon model, a mesh model, or a solid model. The three-dimensional model data WMD may be point cloud data representing the template model TM3 of the target object OBJ. The three-dimensional model data WMD may be polygon model data, mesh model data, or solid model data generated from the point cloud data. Note that the actual target object OBJ measured in advance to generate the three-dimensional model data WMD may be a reference or non-defective target object OBJ.

[0159] In this case, the position and orientation calculation unit 312 may perform 3D matching processing (in other words, template matching processing) on ​​the three-dimensional position data WSD using the template model TM3 indicated by the three-dimensional model data WMD as a template. Specifically, the position and orientation calculation unit 312 may translate, enlarge, reduce, and / or rotate the template model TM3 within the 3D imaging coordinate system so that characteristic locations (e.g., at least one of feature points and edges) of the template model TM3 indicated by the three-dimensional model data WMD approach (e.g., match) characteristic locations of the target object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD (e.g., a point cloud corresponding to the target object OBJ indicated by the three-dimensional position data WSD, or a three-dimensional model (e.g., a point cloud model) reflecting the actual three-dimensional position of the target object OBJ). In other words, the position and orientation calculation unit 312 may change the positional relationship between the coordinate system of the three-dimensional model data WMD (e.g., the coordinate system of a CAD model) and the 3D imaging coordinate system so that characteristic locations of the target object OBJ, whose three-dimensional position is indicated by the three-dimensional position data WSD (e.g., a point cloud corresponding to the target object OBJ indicated by the three-dimensional position data WSD, or a three-dimensional model (e.g., a point cloud model) reflecting the actual three-dimensional position of the target object OBJ) approach (e.g., coincide) with characteristic locations of the template model TM3. As a result, the position and orientation calculation unit 312 can specify the positional relationship between the coordinate system of the three-dimensional model data WMD and the 3D imaging coordinate system. Thereafter, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ in the 3D imaging coordinate system from at least one of the position and orientation of the template model TM3 in the coordinate system of the three-dimensional model data WMD based on the positional relationship between the coordinate system of the three-dimensional model data WMD and the 3D imaging coordinate system.

[0160] Then, the position and orientation calculation unit 312 converts at least one of the position and orientation of the target object OBJ in the 3D imaging coordinate system into at least one of the position and orientation of the target object OBJ in the robot coordinate system based on a transformation matrix for converting three-dimensional coordinates in either the robot coordinate system or the 3D imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the 3D imaging coordinate system. When the robot coordinate system is used as the global coordinate system, at least one of the position and orientation of the target object OBJ in the robot coordinate system corresponds to at least one of the position and orientation of the target object OBJ in the global coordinate system. However, when a coordinate system other than the robot coordinate system is used as the global coordinate system, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ in the global coordinate system from at least one of the position and orientation of the target object OBJ in the robot coordinate system using a transformation matrix for converting three-dimensional coordinates in either the robot coordinate system or the global coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the global coordinate system.

[0161] When the position and orientation calculation unit 312 brings characteristic locations of the three-dimensional model indicated by the three-dimensional model data WMD closer to characteristic locations of the target object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD, the position and orientation calculation unit 312 may calculate a matching similarity, which is the similarity between the three-dimensional model of the target object OBJ indicated by the three-dimensional model data WMD and the target object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD (e.g., a point cloud of the target object OBJ). In the following description, the matching similarity, which is the similarity between the three-dimensional model of the target object OBJ indicated by the three-dimensional model data WMD and the target object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD (e.g., a point cloud of the target object OBJ), may be simply referred to as the matching similarity of the target object OBJ. The position and orientation calculation unit 312 may translate, enlarge, reduce, and / or rotate the three-dimensional model indicated by the three-dimensional model data WMD to maximize the matching similarity. The matching similarity may be considered to be equivalent to the correlation between the three-dimensional model indicated by the three-dimensional model data WMD and the target object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD. The correlation may also be considered an index showing the correlation between the three-dimensional model indicated by the three-dimensional model data WMD and the target object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD. The matching similarity may also be called a matching score.

[0162] When a target object OBJ whose calculated matching similarity exceeds a predetermined matching determination threshold is detected by the 3D matching process, the position and orientation calculation unit 312 may select the target object OBJ as one target object OBJ on which the end effector 4 should perform a predetermined process (i.e., a processing execution object). On the other hand, when a target object OBJ whose calculated matching similarity is below the matching determination threshold is detected by the 3D matching process, the position and orientation calculation unit 312 may not select the target object OBJ as one target object OBJ on which the end effector 4 should perform a predetermined process (i.e., a processing execution object).

[0163] Depending on the three-dimensional position data WSD, the three-dimensional position data WSD may indicate the respective positions of multiple target objects OBJ. For example, if multiple workpieces W are placed on the mounting device T in a disorderly or orderly manner, the imaging device 22 may capture images of the multiple workpieces W and generate image data IMG_3D in which the multiple workpieces W are captured as multiple target objects OBJ. For example, if multiple workpieces W are placed in a disorderly or orderly manner in a container (storage box) CB, which is an example of the mounting device T, the imaging device 22 may capture images of the multiple workpieces W and generate image data IMG_3D in which the multiple workpieces W are captured as multiple target objects OBJ. As a result, the position and orientation calculation unit 312 may generate three-dimensional position data WSD indicating the respective positions of the multiple target objects OBJ. In this case, the position and orientation calculation unit 312 may perform the above-described 3D matching process on each of the multiple target objects OBJ. For example, the position and orientation calculation unit 312 may sequentially perform the 3D matching process on the multiple target objects OBJ. Alternatively, for example, the position and orientation calculation unit 312 may simultaneously perform multiple 3D matching processes for multiple target objects OBJ, respectively. Thereafter, the position and orientation calculation unit 312 may select, from the multiple target objects OBJ, one target object OBJ whose matching similarity exceeds the matching determination threshold and has the largest matching similarity, as a processing execution object on which the end effector 4 should perform a predetermined process. Alternatively, the position and orientation calculation unit 312 may select, from the multiple target objects OBJ, one target object OBJ whose matching similarity exceeds the matching determination threshold and has the Nth largest matching similarity (where N is a constant representing an integer greater than or equal to 2), as a processing execution object on which the end effector 4 should perform a predetermined process. Alternatively, the position and orientation calculation unit 312 may select, from the multiple target objects OBJ, one target object OBJ whose matching similarity exceeds a predetermined matching determination threshold, as a processing execution object. Alternatively, the position and orientation calculation unit 312 may select, as the processing execution object, one target object OBJ from among the plurality of target objects OBJ, which corresponds to a matching similarity that exceeds the matching determination threshold and is closest to the end effector 4.Alternatively, for example, when a plurality of target objects OBJ are contained in the container CB (mounting device T) (e.g., when a plurality of objects are stacked in bulk), the position and orientation calculation unit 312 may select, as the processing execution object, one target object OBJ among the plurality of target objects OBJ that corresponds to a matching similarity that exceeds the matching determination threshold and has the largest Z coordinate along the Z axis (located at the highest position). Note that the Z coordinate of each of the plurality of target objects OBJ may be a Z position along the Z axis of the global coordinate system of the robot 1, calculated by the above-described 3D matching process. Alternatively, the position and orientation calculation unit 312 may select, as the processing execution object, one target object OBJ among the plurality of target objects OBJ that corresponds to a matching similarity that exceeds the matching determination threshold and on which the end effector 4 can perform a predetermined process.

[0164] However, if the end effector 4 is capable of simultaneously performing a predetermined process on two or more target objects OBJ, the position and orientation calculation unit 312 may select at least two of the multiple target objects OBJ as the process execution objects. In other words, the position and orientation calculation unit 312 may select at least two process execution objects on which the end effector 4 should simultaneously perform a predetermined process. For convenience of explanation, the following description will be given of an example in which the position and orientation calculation unit 312 selects one target object OBJ of the multiple target objects OBJ as the process execution object.

[0165] The position and orientation calculation unit 312 may calculate the matching similarity of multiple target objects OBJ, then select one target object OBJ on which the end effector 4 is to perform a predetermined process based on the matching similarity of the multiple target objects OBJ, and then calculate at least one of the position and orientation of the selected target object OBJ. In this case, the position and orientation calculation unit 312 does not need to calculate the position and orientation of other target objects OBJ that are not selected as the one target object OBJ on which the end effector 4 is to perform a predetermined process. Alternatively, because at least one of the position and orientation of the target object OBJ is calculated as a result of the 3D matching process as described above, the position and orientation calculation unit 312 may be considered to have calculated the above-mentioned matching similarity together with at least one of the position and orientation of the target object OBJ. In this case, the position and orientation calculation unit 312 may calculate the matching similarity of the multiple target objects OBJ together with at least one of the positions and orientations of each of the multiple target objects OBJ, and then, based on the matching similarity of the multiple target objects OBJ, select one target object OBJ on which the end effector 4 should perform a predetermined process. Alternatively, the position and orientation calculation unit 312 may calculate at least one of the positions and orientations of the multiple target objects OBJ, then calculate the matching similarity of the multiple target objects OBJ, and then, based on the calculated matching similarity, select one target object OBJ on which the end effector 4 should perform a predetermined process.

[0166] 4, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ by performing a matching process using the image data IMG_2D in addition to or instead of performing the 3D matching process using the three-dimensional position data WSD. In the following description, the matching process using the image data IMG_2D will be referred to as a 2D matching process.

[0167] Specifically, the position and orientation calculation unit 312 may perform a 2D matching process using the image data IMG_2D (two-dimensional position data) and two-dimensional model data indicating a two-dimensional model serving as a reference for the target object OBJ. Specifically, as the 2D matching process, the position and orientation calculation unit 312 may perform an object detection process in which, by using a two-dimensional model indicated by the two-dimensional model data as a template image within the image indicated by the image data IMG_2D, the target object OBJ indicated by the template image is detected. In other words, as the matching process, the position and orientation calculation unit 312 may perform an object detection process in which, by detecting a similar image portion within the image indicated by the image data IMG_2D that is similar to the template image, the target object OBJ is detected within the image indicated by the image data IMG_2D. Note that the 2D matching process (in this case, the object detection process) itself may be the same as an existing matching process. For example, the position and orientation calculation unit 312 may perform 2D matching processing using a well-known method such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed-Up Robust Feature).

[0168] The two-dimensional model data may be data based on three-dimensional model data of the target object OBJ. Here, the two-dimensional model data may be generated from CAD (Computer Aided Design) model data of the target object OBJ, or may be generated from model data (data representing a polygon model or mesh model) of the target object OBJ generated based on prior measurement results using a well-known three-dimensional shape measurement device or the imaging device 22. For example, the two-dimensional model data may be two-dimensional image data representing at least a portion of a two-dimensional model of the target object OBJ generated by virtually projecting at least a portion of a three-dimensional model represented by the three-dimensional model data of the target object OBJ onto a virtual plane. For example, the two-dimensional model data may be two-dimensional image data representing at least a portion of an edge of the two-dimensional model of the target object OBJ generated by virtually projecting at least a portion of a three-dimensional model represented by the three-dimensional model data of the target object OBJ onto a virtual plane. Note that the two-dimensional model data does not have to be data based on the three-dimensional model data of the target object OBJ. For example, the two-dimensional model data may be image data IMG_2D generated in advance by capturing an image of the target object OBJ using the imaging device 21. For example, the two-dimensional model data may be a characteristic location on an image represented by the image data IMG_2D generated in advance by capturing an image of the target object OBJ using the imaging device 21. For example, the characteristic location on the image represented by the image data IMG_2D may be at least one of a feature point and an edge of the target object OBJ that appears in the image represented by the image data IMG_2D. Note that if the characteristic location on the image represented by the image data IMG_2D includes an edge of the target object OBJ, the two-dimensional model may be considered to be a model of at least a portion of the edge of the target object OBJ (edge ​​model). The characteristic location on the image represented by the image data IMG_2D may be at least one of a plurality of feature points of the target object OBJ that appear in the image represented by the image data IMG_2D and at least a portion of the edges of the target object OBJ. Note that the characteristic portions on the image indicated by the image data IMG_2D may be detected by well-known image processing.

[0169] The position and orientation calculation unit 312 may translate, enlarge, reduce, and / or rotate the two-dimensional model of the target object OBJ reflected in the template image so that characteristic features (e.g., at least one of feature points and edges) of the two-dimensional model of the target object OBJ reflected in the template image approach (e.g., match) characteristic features of the target object OBJ reflected in the image represented by the image data IMG_2D. As a result, the position and orientation calculation unit 312 can identify the positional relationship between the coordinate system of the two-dimensional model data and the 2D imaging coordinate system. Thereafter, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ in the 2D imaging coordinate system from at least one of the position and orientation of the target object OBJ in the coordinate system of the two-dimensional model data based on the positional relationship between the coordinate system of the two-dimensional model data and the 2D imaging coordinate system.

[0170] Then, the position and orientation calculation unit 312 converts at least one of the position and orientation of the target object OBJ in the 2D imaging coordinate system into at least one of the position and orientation of the target object OBJ in the robot coordinate system based on a transformation matrix for converting three-dimensional coordinates in either the robot coordinate system or the 2D imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the 2D imaging coordinate system. When the robot coordinate system is used as the global coordinate system, at least one of the position and orientation of the target object OBJ in the robot coordinate system corresponds to at least one of the position and orientation of the target object OBJ in the global coordinate system. However, when a coordinate system other than the robot coordinate system is used as the global coordinate system, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ in the global coordinate system from at least one of the position and orientation of the target object OBJ in the robot coordinate system using a transformation matrix for converting three-dimensional coordinates in either the robot coordinate system or the global coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the global coordinate system.

[0171] When performing 2D matching processing, the position and orientation calculation unit 312 may calculate matching similarity, as in the case of 3D matching processing. The matching similarity calculated in the 2D matching processing is the similarity between the template image (i.e., the two-dimensional model of the target object OBJ) and the image indicated by the image data IMG_2D (particularly, the image portion to which the target object OBJ of the template image is fitted). In the following description, the matching similarity, which is the similarity between the template image (i.e., the two-dimensional model of the target object OBJ) and the image indicated by the image data IMG_2D (particularly, the image portion to which the target object OBJ of the template image is fitted), may simply be referred to as the matching similarity of the target object OBJ. Note that the use of the matching similarity calculated in the 2D matching processing may be the same as the use of the matching similarity calculated in the above-mentioned 3D matching processing, and therefore a detailed description thereof will be omitted.

[0172] When performing 2D matching processing, as in the case of 3D matching processing, the position and orientation calculation unit 312 may perform the 2D matching processing sequentially or simultaneously on multiple target objects OBJ captured in the image represented by the image data IMG_2D. Then, as in the case of 3D matching processing, the position and orientation calculation unit 312 may select at least one of the multiple target objects OBJ as the processing execution object. In this case, the position and orientation calculation unit 312 may select at least one of the multiple target objects OBJ as the processing execution object based on the matching similarity. Note that in step S3 of FIG. 4 , if the position and orientation calculation unit 312 calculates at least one of the position and orientation of the target object OBJ by performing matching processing using the image data IMG_2D instead of performing 3D matching processing using the three-dimensional position data WSD, the robot control processing shown in FIG. 4 does not need to include step S2 of FIG. 4 .

[0173] Alternatively, in step S3 of FIG. 4, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ by performing a matching process using both the three-dimensional position data WSD and the image data IMG_2D, in addition to or instead of performing at least one of a 3D matching process using the three-dimensional position data WSD and a 2D matching process using the image data IMG_2D.

[0174] For example, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ in the global coordinate system from a portion of the position and orientation of the target object OBJ in the 2D imaging coordinate system calculated based on the image data IMG_2D and a portion of the position and orientation of the target object OBJ in the 3D imaging coordinate system calculated based on the image data IMG_3D. For example, the position and orientation calculation unit 312 may set at least one of the initial position and initial orientation of a three-dimensional model (template model TM3) used when performing the 3D matching process based on at least one of the position and orientation of the target object OBJ in the 2D imaging coordinate system calculated based on the image data IMG_2D, calculate at least one of the position and orientation of the target object OBJ in the 3D imaging coordinate system by performing the 3D matching process using the three-dimensional model arranged at the set initial position and / or the set initial orientation, and calculate at least one of the position and orientation of the target object OBJ in the global coordinate system from at least one of the position and orientation of the target object OBJ in the 3D imaging coordinate system.

[0175] When calculating at least one of the position and orientation of the target object OBJ using both the image data IMG_2D and the three-dimensional position data WSD, the position and orientation calculation unit 312 may select, as the target object OBJ on which the end effector 4 should perform a predetermined process, a target object OBJ that satisfies a condition that the sum or product of the matching similarity calculated by the 2D matching process (hereinafter referred to as 2D matching similarity) and the matching similarity calculated by the 3D matching process (hereinafter referred to as 3D matching similarity) is maximized. The position and orientation calculation unit 312 may select, as the target object OBJ on which the end effector 4 should perform a predetermined process, a target object OBJ that satisfies a condition that the 2D matching similarity exceeds a matching determination threshold and is maximized. The position and orientation calculation unit 312 may select, as the target object OBJ on which the end effector 4 should perform a predetermined process, a target object OBJ that satisfies a condition that the 3D matching similarity exceeds a matching determination threshold and is maximized. The position and orientation calculation unit 312 may select a target object OBJ that satisfies the condition that both the 2D matching similarity and the 3D matching similarity exceed the matching determination threshold as a target object OBJ on which the end effector 4 should perform a predetermined process. The position and orientation calculation unit 312 may select a target object OBJ that satisfies the condition that both the 2D matching similarity and the 3D matching similarity exceed the matching determination threshold and is closest to the end effector 4 as a target object OBJ on which the end effector 4 should perform a predetermined process. The position and orientation calculation unit 312 may select a target object OBJ that satisfies the condition that both the 2D matching similarity and the 3D matching similarity exceed the matching determination threshold and has the largest Z coordinate along the Z axis (is located at the highest position) as a target object OBJ on which the end effector 4 should perform a predetermined process.

[0176] Note that if the position and orientation calculation unit 312 performs matching processing using the three-dimensional position data WSD but does not perform matching processing using the image data IMG_2D, the control device 3 does not need to acquire the image data IMG_2D in step S1. If the control device 3 does not acquire the image data IMG_2D, the imaging device 21 does not need to capture an image of the target object OBJ. If the control device 3 does not acquire the image data IMG_2D, the imaging system 2 does not need to include the imaging device 21.

[0177] If the position and orientation calculation unit 312 performs matching processing using the image data IMG_2D but does not perform matching processing using the three-dimensional position data WSD, the control device 3 does not need to acquire the image data IMG_3D in step S1. If the control device 3 does not acquire the image data IMG_3D, the imaging device 22 does not need to capture an image of the target object OBJ. If the control device 3 does not acquire the image data IMG_3D, the imaging system 2 does not need to include the imaging device 22. If the control device 3 does not acquire the image data IMG_3D, the control device 3 does not need to include the three-dimensional position data generation unit 311.

[0178] For simplicity of explanation, the following description will mainly focus on an example in which the position and orientation calculation unit 312 performs 3D matching processing in step S3 of FIG.

[0179] Thereafter, the signal generating unit 313 generates a robot control signal using the position and orientation data POI generated in step S3 (i.e., the calculation result of at least one of the position and orientation of the processing execution object) (step S4). For example, the signal generating unit 313 may generate a robot control signal so that the end effector 4 can perform a predetermined process on the target object OBJ (particularly, the processing execution object). For example, the signal generating unit 313 may generate a robot control signal so that the end effector 4 approaches the target object OBJ (particularly, the processing execution object) in order to perform a predetermined process on the target object OBJ (particularly, the processing execution object). For example, the signal generating unit 313 may generate a robot control signal so that the positional relationship between the end effector 4 and the target object OBJ (particularly, the processing execution object) is a desired positional relationship. For example, the signal generating unit 313 may generate a robot control signal to control the operation of the robot arm 12 so that the positional relationship between the end effector 4 and the target object OBJ (particularly, the processing execution object) is a desired positional relationship. For example, the signal generating unit 313 may generate a robot control signal so that the end effector 4 performs a predetermined process on the target object OBJ (particularly, the process execution object) when the positional relationship between the end effector 4 and the target object OBJ (particularly, the process execution object) reaches a desired positional relationship. For example, the signal generating unit 313 may generate a robot control signal for controlling the operation of the end effector 4 so that a predetermined process is performed on the target object OBJ (particularly, the process execution object) when the positional relationship between the end effector 4 and the target object OBJ (particularly, the process execution object) reaches a desired positional relationship. As already described above, the robot control signal for controlling the operation of the end effector 4 may be referred to as an end effector control signal.

[0180] The signal generator 313 may use the position and orientation data POI generated in step S3 to generate a robot control signal to perform feedback control of the robot 1. For example, the signal generator 313 may use the position and orientation data POI to generate a robot control signal to perform feedback control including P (Proportional) control. For example, the signal generator 313 may use the position and orientation data POI to generate a robot control signal to perform feedback control including PI (Proportional-Integral) control. For example, the signal generator 313 may use the position and orientation data POI to generate a robot control signal to perform feedback control including PID (Proportional-Integral-Differential) control.

[0181] As an example, each of Figures 5(a) to 5(d) is a side view showing the positional relationship between the robot 1 and the workpiece W at a certain point during a holding process for holding a workpiece W (i.e., an example of a target object OBJ) placed on a placement device T#1, such as an AGV (Automatic Guided Vehicle), moving on a support surface S. In this case, the control device 3 may generate position and orientation data POI indicating at least one of the position and orientation of the workpiece W (i.e., an example of a target object OBJ in the holding process) that is the target of the holding process, and generate a robot control signal using the generated position and orientation data POI. For example, as shown in Figure 5(a), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 moves toward the space directly above the moving workpiece W. As an example, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 is positioned (in other words, approaches) in a desired orientation at the sequentially updated movement target point while sequentially updating the movement target point of the end effector 4 in accordance with the moving workpiece W based on at least one of the position and orientation of the workpiece W sequentially calculated by the position and orientation calculation unit 312. As shown in FIG. 5( b), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 located directly above the moving workpiece W approaches the workpiece W until it can hold the workpiece W while remaining positioned directly above the workpiece W. As shown in FIG. 5( c), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4 located in a position capable of holding the moving workpiece W holds the workpiece W while following the moving workpiece W. As shown in Figure 5 (d), the signal generating unit 313 may generate a robot control signal to control the operation of the robot arm 12 and the end effector 4 so that the end effector 4 holding the workpiece W moves away from the moving mounting device T#1 while holding the workpiece W.

[0182] When the robot 1 holds the workpiece W placed on the placement device T#1 using the end effector 4, the robot 1 may perform a release process to release the held workpiece W so that the held workpiece W is placed on a placement device T#2 (i.e., an example of the target object OBJ in the release process) different from the placement device T#1. That is, the robot 1 may perform a placement process to place the workpiece W on the placement device T#2 by successively performing the holding process and the release process. In this case, the control device 3 may generate position and orientation data POI indicating at least one of the position and orientation of the placement device T#2 that is the target of the release process, and may generate a robot control signal using the generated position and orientation data POI. For example, each of FIGS. 6( a) to 6(d) is a side view showing the positional relationship between the robot 1 and the workpiece W at a certain point during a release process to place the workpiece W on the placement device T#2 moving on the support surface S. 6A, the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 holding the workpiece W moves to the space directly above the moving mounting device T#2 while still holding the workpiece W. As an example, the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 is positioned in a desired orientation at (in other words, moves closer to) the sequentially updated movement target point, while sequentially updating the movement target point of the end effector 4 in accordance with the moving mounting device T#2, based on at least one of the position and orientation of the mounting device T#2 sequentially calculated by the position and orientation calculating unit 312. As shown in Figure 6 (b), the signal generating unit 313 may generate a robot control signal to control the operation of the robot arm 12 so that the end effector 4, which is positioned directly above the moving mounting device T#2 and holding the workpiece W, approaches the mounting device T#2 while remaining positioned directly above the mounting device T#2 and holding the workpiece W, until the workpiece W can be placed on the mounting device T#2.6(c), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4, which is positioned at a position where it can place the workpiece W on the placement device T#2, places the workpiece W on the placement device T#2 while following the moving placement device T#2 (i.e., releases the workpiece W that it was holding). As shown in FIG. 6(d), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4, which has finished placing the workpiece W on the placement device T#2, moves away from the placement device T#2.

[0183] When a release process is performed (particularly when the release process is performed to perform a placement process), the control device 3 may further generate position and orientation data POI indicating at least one of the position and orientation of the workpiece W that is a second target of the release process (i.e., an example of the target object OBJ in the release process), in addition to position and orientation data POI indicating at least one of the position and orientation of the placement device T#2 that is a first target of the release process (i.e., an example of the target object OBJ in the release process). That is, the control device 3 may generate position and orientation data POI indicating at least one of the position and orientation of the workpiece W that is not yet held by the end effector 4 before the end effector 4 holds the workpiece W, and may also generate position and orientation data POI indicating at least one of the position and orientation of the workpiece W that is held by the end effector 4 after the end effector 4 holds the workpiece W.

[0184] When a release process is performed, before releasing the workpiece W held by the end effector 4 through the release process, the control device 3 may generate a robot control signal using, in addition to the position and orientation data POI regarding the mounting device T#2, the position and orientation data POI indicating at least one of the position and orientation of the workpiece W held by the end effector 4. In other words, the control device 3 may generate a robot control signal using the position and orientation data POI indicating at least one of the position and orientation of the workpiece W held by the end effector 4 during at least a portion of the period during which the end effector 4 holds the workpiece W.

[0185] For example, the signal generating unit 313 may generate a robot control signal for controlling the robot arm 12 to move the workpiece W held by the end effector 4 to a desired position (e.g., a position where the workpiece W should be released). In this case, compared to when the position and orientation data POI related to the workpiece W is not used, the robot 1 can appropriately place the workpiece W held by the end effector 4 at a desired position on the mounting device T (the robot 1 can execute the placement process). This is because the position of the workpiece W held by the end effector 4 becomes known information for the control device 3. If the position of the workpiece W is not known information, a technical problem may arise in that the workpiece W collides with the mounting device T. Alternatively, if the position of the workpiece W is not known information, the above-described placement process may result in a technical problem in that the workpiece W is placed at an unintended position on the mounting device T (or another object). Alternatively, if the position of the workpiece W is not known information, the above-described fitting process may result in a technical problem in that the workpiece W is fitted into an unintended position on another object. Alternatively, if the position of the workpiece W is not known information, the above-described pasting process may result in a technical problem in which the workpiece W is pasted in an unintended position on another object. Alternatively, if the position of the workpiece W is not known information, the above-described bonding process may result in a technical problem in which the workpiece W is adhered in an unintended position on another object. Alternatively, if the position of the workpiece W is not known information, the above-described welding process may result in a technical problem in which the workpiece W is welded to an unintended position on another object. However, in this embodiment, because the position of the workpiece W is known information, the possibility of such technical problems occurring is low or none.

[0186] Furthermore, for example, the signal generating unit 313 may generate a robot control signal for controlling the end effector 4 to change the posture of the workpiece W held by the end effector 4 to a desired posture. In this case, compared to when the position and posture data POI related to the workpiece W is not used, the robot 1 can use the end effector 4 to place the workpiece W in a desired posture on the mounting device T (the mounting process can be executed). This is because the posture of the workpiece W held by the end effector 4 becomes known information to the control device 3. If the posture of the workpiece W is not the desired posture, a technical problem may occur in that the workpiece W collides with the mounting device T. Alternatively, if the posture of the workpiece W is not the desired posture, a technical problem may occur in that the workpiece W is placed on the mounting device T (or another object) in an unintended posture as a result of the above-described mounting process. Alternatively, if the posture of the workpiece W is not the desired posture, a technical problem may occur in that the workpiece W is fitted into another object in an unintended posture as a result of the above-described fitting process. Alternatively, if the orientation of the workpiece W is not the desired orientation, the above-described attaching process may result in a technical problem in which the workpiece W is attached to another object in an unintended orientation. Alternatively, if the orientation of the workpiece W is not the desired orientation, the above-described bonding process may result in a technical problem in which the workpiece W is attached to another object in an unintended orientation. Alternatively, if the orientation of the workpiece W is not the desired orientation, the above-described welding process may result in a technical problem in which the workpiece W is welded to another object in an unintended orientation. However, in this embodiment, because the robot 1 can be controlled so that the orientation of the workpiece W is the desired orientation, the possibility of such technical problems occurring is low or none.

[0187] In addition, not only when a release process is performed, but also in any situation where the end effector 4 is holding the workpiece W, the control device 3 may generate a robot control signal using position and orientation data POI that indicates at least one of the position and orientation of the workpiece W held by the end effector 4.

[0188] At least one of the placement device T#1 shown in FIGS. 5( a) to 5(d) and the placement device T#2 shown in FIGS. 6( a) to 6(d) does not necessarily have to move on the support surface S. For example, FIGS. 7( a) and 7(b) are side views showing the positional relationship between the robot 1 and the workpiece W at a certain point during a holding process for holding the workpiece W placed on the placement device T#1 that is stationary on the support surface S. In this case, as shown in FIG. 7( a), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 approaches the stationary workpiece W until it can hold the stationary workpiece W. As an example, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 based on at least one of the position and orientation of the workpiece W calculated by the position and orientation calculation unit 312 so that the end effector 4 is positioned in a desired orientation (in other words, approaches) at a position where the end effector 4 can hold the workpiece W, at least one of the position and orientation of which has been calculated. As shown in FIG. 7( b), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4, which is located at a position where it can hold the stationary workpiece W, holds the workpiece W. Furthermore, each of FIGS. 7( c) to 7( e) is a side view showing the positional relationship between the robot 1 and the workpiece W at a certain point during a period in which a release process is being performed to place the workpiece W on the placement device T#2 that is stationary on the support surface S. In this case, as shown in FIG. 7( c), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 holding the workpiece W approaches the placement device T#2 while still holding the workpiece W until it is able to place the workpiece W on the placement device T#2.7(d), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4, which is positioned at a position where it can place the workpiece W on the placement device T#2, places the workpiece W on the stationary placement device T#2 (i.e., releases the workpiece W that it was holding). As shown in FIG. 7(e), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4, which has finished placing the workpiece W on the placement device T#2, moves away from the placement device T#2.

[0189] A plurality of workpieces W may be placed on the placement device T#1. For example, the plurality of workpieces W may be placed on the placement device T#1 so that they are arranged according to a certain standard on the placement device T#1. For example, the plurality of workpieces W may be placed on the placement device T#1 so that they are randomly stacked on the placement device T#1. In this case, the robot 1 may perform a holding process to selectively hold a desired workpiece W from the plurality of workpieces W placed on the placement device T#1. In particular, the robot 1 may perform a holding process to hold the plurality of workpieces W placed on the placement device T#1 one by one in order.

[0190] Furthermore, the robot 1 may perform a release process to place multiple workpieces W on the placement device T#2. That is, the robot 1 may perform a placement process to place multiple workpieces W placed on the placement device T#1 one by one on the placement device T#2 (or another placement device) by performing a holding process and a release process. In this case, the robot 1 may perform a release process to place multiple workpieces W one by one on the placement device T#2 so that the multiple workpieces W are lined up according to a certain standard on the placement device T#2. The robot 1 may perform a release process to place multiple workpieces W one by one on the placement device T#2 so that the multiple workpieces W are randomly stacked on the placement device T#2.

[0191] 8(a) to 8(e) and 9(a) to 9(e) are side views showing the positional relationship between the robot 1 and the workpiece W at a certain point during a period in which a holding process for holding two workpieces W#1 and W#2 placed on the placement device T#1 in order one by one and a release process for placing the two workpieces W#1 and W#2 in order one by one on the placement device T#2 are being performed. In this case, as shown in FIG. 8(a), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 approaches workpiece W#2 until it is able to hold either one of workpieces W#1 or W#2 (workpiece W#2 in the example shown in FIG. 8(a)). As an example, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 is positioned (in other words, approaches) in a desired orientation at a position where the end effector 4 can hold the workpiece W#2, whose position and orientation have been calculated, based on at least one of the position and orientation of the workpiece W#2 calculated by the position and orientation calculation unit 312. Thereafter, as shown in FIG. 8B, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4, positioned at a position where it can hold the workpiece W#2, holds the workpiece W#2. Thereafter, as shown in FIG. 8C, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 holding the workpiece W#2 approaches the placement device T#2 until it can place the workpiece W#2 on the placement device T#2, while still holding the workpiece W#2. As an example, the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 based on at least one of the position and posture of the mounting device T#2 calculated by the position and posture calculating unit 312 so that the end effector 4 is positioned in the desired posture (in other words, approaches) at a position where the end effector 4 can place the workpiece W#2 on the mounting device T#2, at least one of the position and posture of which has been calculated.8(d), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4, which is positioned to place the workpiece W#2 on the placement device T#2, places the workpiece W#2 on the placement device T#2 (i.e., releases the workpiece W#2 that it was holding). Then, as shown in FIG. 8(e), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4, which has placed the workpiece W#2 on the placement device T#2, moves away from the placement device T#2. Then, as shown in FIG. 9(a), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 approaches the remaining workpiece W#1 until it can hold the remaining one of the workpieces W#1 and W#2 (workpiece W#1 in the example shown in FIG. 9(a)). As an example, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 is positioned (in other words, approaches) in a desired orientation at a position where the end effector 4 can hold the workpiece W#1, whose position and orientation have been calculated, based on at least one of the position and orientation of the workpiece W#1 calculated by the position and orientation calculation unit 312. Thereafter, as shown in FIG. 9B , the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4, positioned at a position where it can hold the workpiece W#1, holds the workpiece W#1. Thereafter, as shown in FIG. 9C , the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4, holding the workpiece W#1, approaches the placement device T#2 until it can place the workpiece W#1 on the placement device T#2, while still holding the workpiece W#1.As an example, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 is positioned in a desired orientation (i.e., approaches) the placement device T#2, at least one of whose position and orientation is calculated, based on at least one of the position and orientation of the placement device T#2 calculated by the position and orientation calculation unit 312. Thereafter, as shown in FIG. 9D, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4, which is positioned at a position where it can place the workpiece W#1 on the placement device T#2, places the workpiece W#1 on the placement device T#2 (i.e., releases the workpiece W#1 it was holding). Thereafter, as shown in FIG. 9E, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4, which has placed the workpiece W#1 on the placement device T#2, moves away from the placement device T#2.

[0192] The robot 1 may perform a holding process to hold, one by one, a plurality of workpieces W placed on a placement device T#1 that moves on the support surface S. Alternatively, the robot 1 may perform a holding process to hold, one by one, a plurality of workpieces W placed on a placement device T#1 that is stationary on the support surface S. The robot 1 may perform a release process to place, one by one, a plurality of workpieces W on a placement device T#2 that moves on the support surface S. The robot 1 may perform a release process to place, one by one, a plurality of workpieces W on a placement device T#2 that is stationary on the support surface S.

[0193] 4, the signal generator 313 outputs the robot control signal generated in step S4 to the robot 1 (particularly, the robot control device 13) using the communication device 33. As a result, the robot control device 13 controls at least one of the operation of the robot 1 (for example, the operation of the robot arm 12) and the operation of the end effector 4 based on the robot control signal.

[0194] Thereafter, the control device 3 repeats the series of processes from step S1 to step S4 until it is determined that the robot 1 has completed the predetermined process for the target object OBJ selected as the process execution object in step S3 (step S5). For example, when the end effector 4 performs a holding process as shown in Figures 5(a) to 5(d), the control device 3 repeats the series of processes from step S1 to step S4 until it is determined that the end effector 4 has held the target object OBJ selected as the process execution object in step S3 (step S5). For example, when the end effector 4 performs a release process as shown in Figures 6(a) to 6(d), the control device 3 repeats the series of processes from step S1 to step S4 until it is determined that the end effector 4 has released the target object OBJ that it was holding (step S5).

[0195] If it is determined that the robot 1 has not completed the predetermined process for one target object OBJ (step S5: No), the series of processes from step S1 to step S4 is repeated, and therefore the control device 3 continues to acquire at least one of the image data IMG_2D and IMG_3D from the imaging devices 21 and 22, even during the period when the operation of at least one of the robot arm 12 and the end effector 4 is being controlled based on the robot control signal. For example, when the end effector 4 performs a holding process as shown in Figures 5(a) to 5(d), the control device 3 may repeat the series of processes from step S1 to step S4 until the end effector 4 holds the workpiece W (and, if necessary, until the end effector 4 holding the workpiece W leaves the placement device T#1). For example, when the end effector 4 performs a release process as shown in Figures 6(a) to 6(d), the control device 3 may repeat the series of processes from step S1 to step S4 until the end effector 4 places the workpiece W on the placement device T#2 (and, if necessary, until the end effector 4, having finished placing the workpiece W on the placement device T#2, leaves the placement device T#2).

[0196] In this case, because the operation of at least one of the robot arm 12 and the end effector 4 is controlled based on the robot control signal as described above, each of the imaging devices 21 and 22 may capture an image of the target object OBJ during relative movement between the target object OBJ and the imaging devices 21 and 22. For example, each of the imaging devices 21 and 22 may capture an image of the target object OBJ while the target object OBJ is stationary and the imaging devices 21 and 22 are moving. For example, each of the imaging devices 21 and 22 may capture an image of the target object OBJ while the target object OBJ is moving and the imaging devices 21 and 22 are stationary. For example, each of the imaging devices 21 and 22 may capture an image of the target object OBJ while the target object OBJ is moving and the imaging devices 21 and 22 are moving. 4 (or may repeat the robot control process shown in FIG. 4 ) during the relative movement between the target object OBJ and the image capturing devices 21 and 22 (i.e., while at least one of the image capturing devices 21 and 22 and the target object OBJ is moving). As a result, even during the period in which the operation of the robot 1 is being controlled based on the robot control signal, the control device 3 can newly generate (i.e., update) position and orientation data POI indicating at least one of the position and orientation of the target object OBJ based on newly acquired image data IMG_2D and IMG_3D.

[0197] In this case, particularly when at least one of the position and orientation of the target object OBJ changes due to movement of the target object OBJ, the control device 3 can recalculate at least one of the position and orientation of the target object OBJ. That is, the control device 3 can regenerate position and orientation data POI that reflects at least one change in the position and orientation of the target object OBJ due to movement of the target object OBJ. Therefore, it is unlikely that at least one of the position and orientation indicated by the position and orientation data POI will deviate significantly from at least one of the actual position and actual orientation of the target object OBJ. Therefore, the control device 3 can appropriately control the operation of the robot arm 12 so that the end effector 4 can appropriately approach the target object OBJ (particularly a moving target object OBJ).

[0198] However, each of the imaging devices 21 and 22 may capture an image of the target object OBJ while the target object OBJ is stationary and the imaging devices 21 and 22 are stationary. The control device 3 may perform the robot control process shown in Fig. 4 while the imaging devices 21 and 22 and the target object OBJ are stationary. Note that the control device 3 may repeat the robot control process shown in Fig. 4 while the imaging devices 21 and 22 and the target object OBJ are stationary.

[0199] Alternatively, if the target object OBJ is stationary, it is unlikely that at least one of the position and orientation of the target object OBJ calculated in step S3 will change over time. Therefore, there is little need to recalculate at least one of the position and orientation of the target object OBJ by performing the series of processes from step S1 to step S3 anew. Therefore, if the target object OBJ is stationary, the control device 3 may not need to repeat the series of processes from step S1 to step S3 even if it is determined that the robot 1 has not completed the predetermined process for one target object OBJ. In this case, the control device 3 may continue to control the robot 1 based on the robot control signal already generated in step S4. In other words, the control device 3 may continue to control the robot 1 based on the robot control signal already generated in step S4 without generating a new robot control signal in step S4. In other words, the control device 3 may continue to use the robot control signal already generated in step S4 to control the robot 1 without generating a new robot control signal in step S4.

[0200] As described above, the position and orientation calculation unit 312 does not have to calculate at least one of the position and orientation of the target object OBJ in the global coordinate system in step S3. In this case, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ in a coordinate system different from the global coordinate system (e.g., a robot coordinate system, a 2D imaging coordinate system, or a 3D imaging coordinate system) in step S3 of FIG. 4 . That is, the position and orientation calculation unit 312 may generate position and orientation data POI indicating at least one of the position and orientation of the target object OBJ in a coordinate system different from the global coordinate system (e.g., a robot coordinate system, a 2D imaging coordinate system, or a 3D imaging coordinate system). In this case, the signal generation unit 313 may generate a robot control signal in step S4 using the position and orientation data POI indicating at least one of the position and orientation of the target object OBJ in the coordinate system different from the global coordinate system calculated in step S3.

[0201] Thereafter, when it is determined that the robot 1 has completed the predetermined process for one target object OBJ (step S5: Yes), the control device 3 determines whether to terminate the robot control process shown in FIG. 4 (step S6). For example, when the robot 1 is to perform the predetermined process for another target object OBJ, the control device 3 may determine not to terminate the robot control process shown in FIG. 4. For example, when there are still target objects OBJ for which the robot 1 is to perform the predetermined process, the control device 3 may determine not to terminate the robot control process shown in FIG. 4. For example, when the robot 1 is not required to perform the predetermined process for another target object OBJ, the control device 3 may determine to terminate the robot control process shown in FIG. 4. For example, when there are no target objects OBJ for which the robot 1 is to perform the predetermined process, the control device 3 may determine to terminate the robot control process shown in FIG. 4.

[0202] As a result of the determination in step S6, if it is determined that the robot control process shown in FIG. 4 is to be ended (step S6: Yes), the control device 3 ends the robot control process shown in FIG.

[0203] 4 is not to be ended (step S6: No), the control device 3 performs the series of processes from step S1 to step S5 again. That is, the control device 3 performs the series of processes from step S1 to step S5 again so that the robot 1 newly performs a predetermined process on another target object OBJ different from the one target object OBJ on which the robot 1 has already performed a predetermined process.

[0204] However, in this embodiment, the control device 3 (particularly, the signal generation unit 313) adjusts the height of the imaging system 2 before performing the series of processes from step S1 to step S5 again so that the robot 1 can perform the predetermined process on another target object OBJ after completing the predetermined process on one target object OBJ (step S7). Specifically, the control device 3 may adjust the height at which the imaging system 2 captures an image of the target object OBJ (step S7). In the following description, the height of the imaging system 2 (i.e., the height at which the imaging system 2 captures an image of the target object OBJ) is referred to as the "imaging height CH." It can also be said that the imaging height CH is the height at which the imaging system 2 performs the process of capturing an image of the target object OBJ (imaging process). For this reason, the imaging height CH may also be referred to as the imaging process height.

[0205] In this embodiment, "height" refers to a position along the Z-axis direction of the global coordinate system. In this case, height may be a coordinate value of the Z-axis in the global coordinate system. However, height may also refer to the position of the imaging system 2 along the Z-axis direction of a coordinate system different from the global coordinate system. In this case, height may be a coordinate value of the Z-axis in a coordinate system different from the global coordinate system. Examples of coordinate systems different from the global coordinate system include at least one of the robot coordinate system, the 2D imaging coordinate system, and the 3D imaging coordinate system described above. Alternatively, height may refer to a position along the direction of the optical axis of the imaging system 2. Note that at least one of the optical axis AX21 of the imaging device 21 and the optical axis AX22 of the imaging device 22 may be used as the optical axis of the imaging system 2. Note that the direction of the optical axis of the imaging system 2 may be considered equivalent to the imaging direction of the imaging system 2.

[0206] Furthermore, in this embodiment, "height" may refer to the height from a reference point. An example of the reference point may be the coordinate origin of a global coordinate system. An example of the reference point may be the coordinate origin of a coordinate system different from the global coordinate system. An example of the reference point may be a reference portion of a reference object. The target object OBJ may be used as the reference object, or another object different from the target object OBJ (for example, the mounting device T or the container CB) may be used.

[0207] The imaging height CH may refer to the height of the imaging system 2 from the target object OBJ. In this case, the imaging height CH may be considered equivalent to the distance between the imaging system 2 and the target object OBJ. Specifically, the imaging height CH may be considered equivalent to the distance between the imaging system 2 that captures the target object OBJ and the target object OBJ. Therefore, the imaging height CH may be referred to as the imaging distance. As described above, the imaging height CH may be considered equivalent to the distance between the imaging system 2 and the target object OBJ. In this case, the imaging height CH may be considered equivalent to the distance between the imaging system 2 and the target object OBJ in the Z-axis direction of the global coordinate system. For example, the imaging height CH may be considered equivalent to the distance between the imaging system 2 and the target object OBJ in a Z-axis direction different from the global coordinate system. For example, the imaging height CH may be considered equivalent to the distance between the imaging system 2 and the target object OBJ in the direction of the optical axis of the imaging system 2.

[0208] Furthermore, in this embodiment, the "height of the imaging system 2" may refer to the height of at least one of the imaging devices 21 and 22. Therefore, in this embodiment, the "imaging height CH of the imaging system 2" may refer to the height at which at least one of the imaging devices 21 and 22 images the target object OBJ. For example, if the image data IMG_2D is used to calculate the position and orientation data POI in step S3, the "height of the imaging system 2" may refer to the height of the imaging device 21. In other words, the "imaging height CH of the imaging system 2" may refer to the height at which the imaging device 21 images the target object OBJ. In this case, the "imaging height CH of the imaging system 2" may refer to the height of a point representing the imaging device 21. For example, if the image data IMG_3D (specifically, the three-dimensional position data WSD generated from the image data IMG_3D) is used to calculate the position and orientation data POI in step S3, the "height of the imaging system 2" may refer to the height of the imaging device 22. In other words, the "imaging height CH of the imaging system 2" may refer to the height at which the imaging device 22 images the target object OBJ. In this case, the "imaging height CH of the imaging system 2" may mean the height of a point representing the imaging device 22. For example, if both the image data IMG_2D and the image data IMG_3D (specifically, the three-dimensional position data WSD generated from the image data IMG_3D) are used to calculate the position and orientation data POI in step S3, the "height of the imaging system 2" may mean the height of both the imaging devices 21 and 22. In other words, the "imaging height CH of the imaging system 2" may mean the height at which both the imaging devices 21 and 22 capture images of the target object OBJ. In this case, the "imaging height CH of the imaging system 2" may mean the height of a point representing both the imaging devices 21 and 22.

[0209] After adjusting the imaging height CH of the imaging system 2, the control device 3 performs the series of processes from step S1 to step S5 again. Specifically, after adjusting the imaging height CH of the imaging system 2 in step S7, the imaging system 2 re-images the target object OBJ to re-generate at least one of the image data IMG_2D and IMG_3D (step S1). Thereafter, the control device 3 may re-acquire at least one of the image data IMG_2D and IMG_3D re-generated by the imaging system 2 (step S1). Thereafter, the control device 3 may re-generate three-dimensional position data WSD based on the image data IMG_3D re-acquired in step S1 (step S2). Thereafter, the control device 3 may re-generate position and orientation data POI based on at least one of the image data IMG_2D re-acquired in step S1 and the three-dimensional position data WSD re-generated in step S2 (step S3). That is, the control device 3 may generate position and orientation data POI indicating at least one of the position and orientation of another target object OBJ (i.e., the next processing execution object) on which the end effector 4 is to perform a predetermined process next (step S3). The control device 3 may then generate a new robot control signal based on the position and orientation data POI reoriented in step S3 (step S4). For example, the control device 3 may generate a robot control signal so that the end effector 4 approaches the next processing execution object in order to perform the predetermined process on the next processing execution object. The control device 3 then repeats the series of processes from step S1 to step S4 until it is determined that the robot 1 has completed the predetermined process on the next processing execution object (step S5).

[0210] 4 is terminated (Step S6: Yes), the control device 3 alternately repeats the process of controlling the robot 1 to perform a predetermined process on the process execution object (Steps S1 to S5) and the process of adjusting the imaging height CH of the imaging system 2 (Step S7). In other words, the control device 3 adjusts the imaging height CH of the imaging system 2 every time the robot 1 completes a predetermined process on the process execution object. However, as will be described in detail later, the control device 3 does not have to adjust the imaging height CH of the imaging system 2 every time the robot 1 completes a predetermined process on one process execution object. The control device 3 may adjust the imaging height CH of the imaging system 2 every time the robot 1 completes a predetermined process on two or more process execution objects.

[0211] (2-2) Processing for Adjusting the Imaging Height CH of the Imaging System 2 Next, the processing for adjusting the imaging height CH of the imaging system 2 in step S7 of FIG. 4 will be described in further detail.

[0212] In order to adjust the imaging height CH of the imaging system 2, the control device 3 may determine a target value of the imaging height CH of the imaging system 2. In the following description, the target value of the imaging height CH of the imaging system 2 will be referred to as "target imaging height CH_target."

[0213] 10 , which is a cross-sectional view showing multiple target objects OBJ and the imaging system 2, the control device 3 may determine the target imaging height CH_target so as to satisfy the condition that at least one target object OBJ is included in the imaging allowable range CAR of the imaging system 2 positioned at the target imaging height CH_target. Specifically, the control device 3 may determine the target imaging height CH_target so as to satisfy the condition that at least one target object OBJ contained in the container CB is included in the imaging allowable range CAR of the imaging system 2 positioned at the target imaging height CH_target. Note that the imaging system 2 positioned at the target imaging height CH_target may refer to an imaging system 2 whose imaging height CH coincides with the target imaging height CH_target. In particular, the control device 3 may determine the target imaging height CH_target so as to satisfy the condition that at least one target object OBJ on which the robot 1 (end effector 4) is to perform a predetermined process next is included in the imaging allowable range CAR of the imaging system 2 positioned at the target imaging height CH_target.

[0214] Here, when multiple target objects OBJ are arranged so that at least two of the multiple target objects OBJ at least partially overlap each other (e.g., the multiple target objects OBJ are arranged randomly) as shown in FIG. 10 , the robot 1 is likely to typically perform a predetermined process on the multiple target objects OBJ in order from top to bottom. For example, the robot 1 may perform a predetermined process on a target object OBJ that is positioned higher than the other target objects OBJ. For example, the robot 1 may perform a predetermined process on the topmost target object OBJ. For example, the robot 1 may perform a predetermined process on a target object OBJ that is not covered by other target objects OBJ. For example, the robot 1 is likely to perform a predetermined process on a target object OBJ whose upper surface is relatively small and is covered by other target objects OBJ. For this reason, the control device 3 may determine the target imaging height CH_target so as to satisfy the condition that at least one target object OBJ that is higher than the other target objects OBJ is included in the imaging allowable range CAR of the imaging system 2 positioned at the target imaging height CH_target. The control device 3 may determine the target imaging height CH_target so as to satisfy a condition that the uppermost target object OBJ is included in the permissible imaging range CAR of the imaging system 2 located at the target imaging height CH_target. The control device 3 may determine the target imaging height CH_target so as to satisfy a condition that the permissible imaging range CAR of the imaging system 2 located at the target imaging height CH_target includes at least one target object OBJ that is not covered by other target objects OBJ. The control device 3 may determine the target imaging height CH_target so as to satisfy a condition that the permissible imaging range CAR of the imaging system 2 located at the target imaging height CH_target includes at least one target object OBJ whose area covered by other target objects OBJ is relatively small.

[0215] The permissible imaging range CAR may be a range determined based on the depth of field of the imaging system 2. For example, the permissible imaging range CAR may be the same range as the depth of field of the imaging system 2. For example, the permissible imaging range CAR may be a range that at least partially overlaps with the depth of field of the imaging system 2. In other words, the permissible imaging range CAR may be a range that includes part or all of the depth of field of the imaging system 2. In this case, the permissible imaging range CAR may be a range narrower than the depth of field of the imaging system 2, or may be a range wider than the depth of field of the imaging system 2. Note that the depth of field of the imaging system 2 may refer to the depth of field of at least one of the imaging devices 21 and 22.

[0216] The depth of field of the imaging system 2 may be, for example, a range extending on both sides along the optical axis of the imaging system 2 (on the object side of the imaging system 2) centered on the best focus position BFP of the objective optical system (e.g., the objective lens) included in the imaging system 2. Therefore, the permissible imaging range CAR may be a range determined based on the best focus position BFP of the objective optical system included in the imaging system 2. For example, the permissible imaging range CAR may be a range extending on both sides along the optical axis of the imaging system 2 centered on the best focus position BFP of the objective optical system included in the imaging system 2. The objective optical system of the imaging system 2 may refer to the objective optical system included in at least one of the imaging devices 21 and 22. The best focus position BFP may also be referred to as a just-in-focus position. The permissible imaging range CAR may also be the best focus position BFP. In other words, the permissible imaging range CAR may coincide with the best focus position BFP. In this case, the permissible imaging range CAR may also be referred to as a permissible imaging position.

[0217] The permissible imaging range CAR may be a range determined based on the depth of focus of the imaging system 2. For example, the permissible imaging range CAR may be the same range as the depth of focus of the imaging system 2. For example, the permissible imaging range CAR may be a range that at least partially overlaps with the depth of focus of the imaging system 2. In other words, the permissible imaging range CAR may be a range that includes part or all of the depth of focus of the imaging system 2. In this case, the permissible imaging range CAR may be a range narrower than the depth of focus of the imaging system 2 or may be a range wider than the depth of field of the imaging system 2. Note that the depth of focus of the imaging system 2 may refer to the depth of focus of at least one of the imaging devices 21 and 22. The depth of focus of the imaging system 2 may refer to the depth of focus on the image plane side of the projection device 23 that projects the projection pattern.

[0218] The permissible imaging range CAR may be a range that satisfies the accuracy condition that "the accuracy of the position and orientation data POI based on at least one of the image data IMG_2D and IMG_3D generated by the imaging system 2 by capturing an image of the target object OBJ included in the permissible imaging range CAR (i.e., the calculation accuracy of at least one of the position and orientation of the target object OBJ) is equal to or greater than a predetermined accuracy threshold." Specifically, as described above, the control device 3 (particularly the position and orientation calculation unit 312) performs a position and orientation calculation including a matching process to generate position and orientation data POI of the target object OBJ whose matching similarity exceeds a matching determination threshold. Even in this case, it is expected that the higher the matching similarity of the target object OBJ, the higher the accuracy of the position and orientation data POI of the target object OBJ (i.e., the calculation accuracy of at least one of the position and orientation). For this reason, the permissible imaging range CAR may be a range that satisfies the accuracy condition that "the matching similarity of the target object OBJ, the position and orientation data POI of which is generated based on at least one of the image data IMG_2D and IMG_3D generated by the imaging system 2 by imaging the target object OBJ included in the permissible imaging range CAR, exceeds both a matching determination threshold and a predetermined accuracy threshold that is different from the matching determination threshold." Note that even in the case where the permissible imaging range CAR is determined based on at least one of the depth of field, best focus position BFP, and depth of focus of the imaging system 2 as described above, the permissible imaging range CAR may also be determined based on at least one of the depth of field, best focus position BFP, and depth of focus of the imaging system 2 so as to satisfy the accuracy condition.

[0219] The permissible imaging range CAR may be a range determined based on the field of view (imaging range, which may be expressed as an angle of view or a field of view) of the imaging system 2. Note that the field of view of the imaging system 2 may refer to at least one of the field of view (imaging range) of the imaging device 21 and the field of view (imaging range) of the imaging device 22. In particular, when the imaging system 2 includes the imaging device 22 and the position and orientation data POI is generated based on the three-dimensional position data WSD generated from the image data IMG_3D generated by the imaging device 22 in the robot control process of FIG. 4 , the permissible imaging range CAR may be a range determined based on the field of view (imaging range) of the imaging device 22. For convenience of explanation, two examples of the permissible imaging range CAR determined based on the field of view of the imaging device 22 that is a stereo camera (i.e., includes two monocular cameras) will be described below.

[0220] FIG. 11 is a cross-sectional view showing a first example of an allowable imaging range CAR determined based on the field of view of the imaging device 22. The allowable imaging range CAR may be a range determined based on the field of view of two monocular cameras included in the imaging device 22. Note that in the following description, the two monocular cameras included in the imaging device 22 are referred to as the left camera 22R and the right camera 22R, respectively. Specifically, the allowable imaging range CAR may be a range determined based on a field of view overlap region 221O where the field of view 221L of the left camera 22L and the field of view 221R of the right camera 22R overlap. For example, the allowable imaging range CAR may be the same range as the field of view overlap region 221O. For example, the allowable imaging range CAR may be a range that includes a portion of the field of view overlap region 221O. For example, the allowable imaging range CAR may be a range that includes the entire field of view overlap region 221O. For example, the allowable imaging range CAR may be a range that is included in the field of view overlap region 221O.

[0221] 11 , both the left camera 22L and the right camera 22R can capture images of the target object OBJ. As a result, the control device 3 can appropriately calculate the parallax based on the image data IMG_3D representing the image generated by the left camera 22L and the image generated by the right camera 22R. Therefore, the control device 3 can appropriately generate the three-dimensional position data WSD based on the parallax calculation result.

[0222] The larger the field of view of the imaging device 22 (i.e., at least one of the field of view 221L of the left camera 22L and the field of view 221R of the right camera 22R) is, the larger the size of the permissible imaging range CAR may be. Therefore, the field of view of the imaging device 22 may be expanded to set a desired size of permissible imaging range CAR. However, if the field of view of the imaging device 22 is expanded while the number of pixels of the imaging element included in the imaging device 22 remains fixed, the resolution of the imaging device 22 decreases. As a result, the accuracy of the position and orientation data POI may deteriorate. Therefore, when the field of view of the imaging device 22 is expanded, the number of pixels of the imaging element included in the imaging device 22 may be increased. In other words, the field of view of the imaging device 22 and the number of pixels of the imaging element included in the imaging device 22 may be adjusted in conjunction with each other so that the resolution of the imaging device 22 meets a desired standard. Note that when the field of view of the imaging device 22 is expanded, in addition to or instead of increasing the number of pixels of the imaging element included in the imaging device 22, the pixel size of the imaging element included in the imaging device 22 may be reduced. In other words, the field of view of the imaging device 22 and the pixel size of the imaging element included in the imaging device 22 may be adjusted in conjunction with each other so that the resolution of the imaging device 22 satisfies a desired standard.

[0223] 12 is a cross-sectional view showing a second example of the allowable imaging range CAR determined based on the field of view of the imaging device 22. In the second example, the allowable imaging range CAR may be a range determined based on a partial field of view, which is a part of the field of view of the imaging device 22. Specifically, the allowable imaging range CAR may be a range determined based on a field of view overlap region 222O where a partial field of view 222L, which is a part of the field of view 221L of the left camera 22L, and a partial field of view 222R, which is a part of the field of view 221R of the right camera 22R, overlap. For example, the allowable imaging range CAR may be the same range as the field of view overlap region 222O. For example, the allowable imaging range CAR may be a range that includes a part of the field of view overlap region 222O. For example, the allowable imaging range CAR may be a range that includes the entire field of view overlap region 222O. For example, the allowable imaging range CAR may be a range that is included in the field of view overlap region 222O.

[0224] When the imaging device 22 captures an image of the target object OBJ included in the permissible imaging range CAR shown in Fig. 12, both the left camera 22L and the right camera 22R can capture images of the target object OBJ, similar to the case where the imaging device 22 captures an image of the target object OBJ included in the permissible imaging range CAR shown in Fig. 11. As a result, the control device 3 can appropriately calculate the parallax based on the image data IMG_3D indicating the image generated by the left camera 22L and the image generated by the right camera 22R. Therefore, the control device 3 can appropriately generate the three-dimensional position data WSD based on the parallax calculation result.

[0225] The partial fields of view 222L and 222R shown in FIG. 12 may correspond to the search areas 224L and 224R used in stereo matching when calculating disparity based on the image data IMG_3D. In this case, the allowable capture range CAR may be considered to be a range determined based on the search areas 224L and 224R. The search areas 224L and 224R used in stereo matching will be described below with reference to FIGS. 13(a) and 13(b). Specifically, FIG. 13(a) shows the relationship between the field of view 221L and partial field of view 222L of the left camera 22L and the image 223L generated by the left camera 22L. As shown in FIG. 13(a), the left camera 22L generates an image 223L that includes an object included in the field of view 221L. Furthermore, as shown in FIG. 13(a), the image 223L includes a search area 224L that includes an object included in the partial field of view 222L, which is part of the field of view 221L. On the other hand, Figure 13(b) shows the relationship between the field of view 221R and partial field of view 222R of the right camera 22R and the image 223R generated by the right camera 22R. As shown in Figure 13(b), the right camera 22R generates an image 223R in which an object included in the field of view 221R is captured. Furthermore, as shown in Figure 13(b), the image 223R includes a search area 224R in which an object included in the partial field of view 222R, which is part of the field of view 221R, is captured. In order to calculate the parallax based on the image data IMG_3D representing the images 223L and 223R, the control device 3 performs stereo matching to search for the same portion BP of the target object OBJ in each of the search area 224L and the search area 224R, and calculates the deviation between the position of the portion BP detected in the search area 224L and the position of the same portion BP detected in the search area 224R as the parallax. The control device 3 may calculate a plurality of parallaxes by calculating the respective deviations of a plurality of different parts BP between the search area 224L and the search area 224R. That is, the control device 3 may calculate the parallax of each of a plurality of parts of the target object OBJ (or any object). Any part of the target object OBJ may be used as the part BP. A characteristic part of the target object OBJ may be used as the part BP.The portion BP may be a projection pattern projected onto the target object OBJ by the projection device 23. In this manner, the partial fields of view 222L and 222R may be fields of view corresponding to the search regions 224L and 224R, respectively.

[0226] When the imaging device 22 captures an image of a target object OBJ included in the allowable imaging range CAR under such conditions that the partial fields of view 222L and 222R correspond to the search areas 224L and 224R, respectively, the object OBJ (particularly, the portion BP of the target object OBJ) is included in both the partial fields of view 222L and 222R, as shown in the upper part of FIG. 14( a). As a result, as shown in the lower part of FIG. 14( a), the same portion BP of the target object OBJ appears in both the search area 224L of the image 223L and the search area 224R of the image 223R. This allows the control device 3 to appropriately calculate the parallax based on the image data IMG_3D representing the images 223L and 223R. This allows the control device 3 to appropriately generate the three-dimensional position data WSD based on the parallax calculation results.

[0227] On the other hand, if the imaging device 22 captures an image of a target object OBJ that is not included in the allowable imaging range CAR under such circumstances where the partial fields of view 222L and 222R correspond to the search areas 224L and 224R, respectively, there is a possibility that the object OBJ (particularly, a portion BP of the target object OBJ) is not included in at least one of the partial fields of view 222L and 222R, as shown in the upper part of FIG. 14(b). Note that FIG. 14(b) illustrates an example in which the object OBJ (particularly, a portion BP of the target object OBJ) is not included in both the partial fields of view 222L and 222R. As a result, as shown in the lower part of FIG. 14(b), there is a possibility that the same portion BP of the target object OBJ is not captured in at least one of the search area 224L of the image 223L and the search area 224R of the image 223R. In this case, the control device 3 may not be able to calculate the parallax based on the image data IMG_3D representing the images 223L and 223R. Therefore, the control device 3 may not be able to generate the three-dimensional position data WSD based on the calculation result of the parallax.

[0228] 14B, by enlarging at least one of the search areas 224R and 224R, it is possible that the same portion BP of the target object OBJ will appear in both the search area 224L in the image 223L and the search area 224R in the image 223R. However, enlarging at least one of the search areas 224R and 224R increases the time required to search for the portion BP within the search areas 224R and 224R. In other words, it increases the time required to generate the three-dimensional position data WSD.

[0229] However, in this embodiment, the imaging height CH of the imaging system 2 is adjusted so that the target object OBJ is included in the imaging allowable range CAR determined based on the search areas 224R and 224R. Therefore, the control device 3 does not need to expand at least one of the search areas 224R and 224R to ensure that the same portion BP of the target object OBJ is captured in both the search area 224L of the image 223L and the search area 224R of the image 223R. This reduces the increase in the time required to generate the three-dimensional position data WSD. In other words, because the imaging height CH of the imaging system 2 is adjusted so that the target object OBJ is included in the imaging allowable range CAR determined based on the search areas 224R and 224R, the time required to generate the three-dimensional position data WSD can be shortened. In other words, the time required to generate the three-dimensional position data WSD can be shortened compared to when the imaging system 2 captures the target object OBJ as shown in FIG. 14(b).

[0230] The permissible imaging range CAR may be automatically set by the control device 3. For example, the control device 3 may acquire information on the specifications of the imaging system 2 and set the permissible imaging range CAR based on the acquired information. For example, the control device 3 may acquire information on the depth of field of the imaging system 2 as information on the specifications of the imaging system 2 and set the permissible imaging range CAR based on the acquired information on the depth of field. For example, the control device 3 may acquire information on the best focus position BFP of the objective optical system provided in the imaging system 2 as information on the specifications of the imaging system 2 and set the permissible imaging range CAR based on the acquired information on the best focus position BFP. For example, the control device 3 may acquire information on the depth of focus of the imaging system 2 as information on the specifications of the imaging system 2 and set the permissible imaging range CAR based on the acquired information on the depth of focus. For example, the control device 3 may acquire information about the field of view of the imaging system 2 (e.g., the field of view 221L of the left camera 22L and the field of view 221R of the right camera 22R) as information about the specifications of the imaging system 2, and set the permissible imaging range CAR based on the acquired information about the field of view. For example, the control device 3 may acquire information about a partial field of view of the imaging system 2 (e.g., the partial field of view 222L of the left camera 22L and the partial field of view 222R of the right camera 22R) as information about the specifications of the imaging system 2, and set the permissible imaging range CAR based on the acquired information about the partial field of view. For example, the control device 3 may acquire information about the search regions 224L and 224R used in stereo matching as information about the specifications of the imaging system 2, and set the permissible imaging range CAR based on the acquired information about the search regions 224L and 224R.

[0231] The permissible imaging range CAR may be set based on information input by the user. For example, the user may input information regarding the specifications of the imaging system 2 to the control device 3, and the permissible imaging range CAR may be set based on the input information. For example, the user may input information regarding the depth of field of the imaging system 2 as information regarding the specifications of the imaging system 2, and the permissible imaging range CAR may be set based on the input information regarding the depth of field. For example, the user may input information regarding the best focus position BFP of the objective optical system provided in the imaging system 2 as information regarding the specifications of the imaging system 2, and the permissible imaging range CAR may be set based on the input information regarding the best focus position BFP. For example, the user may input information regarding the depth of focus of the imaging system 2 as information regarding the specifications of the imaging system 2, and the permissible imaging range CAR may be set based on the input information regarding the depth of focus. For example, the user may input information about the field of view of the imaging system 2 (e.g., the field of view 221L of the left camera 22L and the field of view 221R of the right camera 22R) as information about the specifications of the imaging system 2, and set the allowable imaging range CAR based on the input information about the field of view. For example, the user may input information about a partial field of view of the imaging system 2 (e.g., the partial field of view 222L of the left camera 22L and the partial field of view 222R of the right camera 22R) as information about the specifications of the imaging system 2, and set the allowable imaging range CAR based on the input information about the partial field of view. For example, the user may input information about the search areas 224L and 224R used in stereo matching as information about the specifications of the imaging system 2, and set the allowable imaging range CAR based on the input information about the search areas 224L and 224R. Alternatively, for example, the user may input information for directly or indirectly specifying the allowable imaging range CAR to the control device 3, and the allowable imaging range CAR may be set based on the input information.

[0232] To determine the target imaging height CH_target, the control device 3 may use at least one of the image data IMG_2D and IMG_3D generated by the imaging system 2 capturing an image of at least one target object OBJ contained in the container CB. That is, the control device 3 may adjust the imaging height CH of the imaging system 2 based on at least one of the image data IMG_2D and IMG_3D. Specifically, the control device 3 may determine the target imaging height CH_target based on at least one of the image data IMG_2D and IMG_3D last acquired in step S1 of FIG. 4 . However, as will be described in detail later in a first modified example, the control device 3 may acquire at least one of the image data IMG_2D and IMG_3D that is different from the at least one of the image data IMG_2D and IMG_3D acquired in step S1 of FIG. 4 , and determine the target imaging height CH_target based on the acquired at least one of the image data IMG_2D and IMG_3D.

[0233] Note that, when the control device 3 determines the target imaging height CH_target based on at least one of the image data IMG_2D and IMG_3D last acquired in step S1 of Fig. 4, the control device 3 may determine the target imaging height CH_target at any timing after at least one of the image data IMG_2D and IMG_3D is acquired in step S1 of Fig. 4. For example, the control device 3 may determine the target imaging height CH_target in advance after at least one of the image data IMG_2D and IMG_3D is acquired in step S1 of Fig. 4 and before starting the process of adjusting the imaging height CH in step S7 of Fig. 4. As will be described later, when the three-dimensional position data WSD generated in step S2 of Fig. 4 is used to determine the target imaging height CH_target (specifically, to calculate the height OH of the target object group OBG), the control device 3 may determine the target imaging height CH_target at any timing after the three-dimensional position data WSD is generated in step S2 of Fig. 4. For example, the control device 3 may determine the target imaging height CH_target in advance after generating the three-dimensional position data WSD in step S2 of Fig. 4 and before starting the process of adjusting the imaging height CH in step S7 of Fig. 4. Alternatively, the control device 3 may determine the target imaging height CH_target after starting the process of adjusting the imaging height CH in step S7 of Fig. 4.

[0234] To determine the target imaging height CH_target based on at least one of the image data IMG_2D and IMG_3D, the control device 3 first calculates the height OH of the target object OBJ contained in the container CB based on at least one of the image data IMG_2D and IMG_3D. As described above, the imaging height CH of the imaging system 2 is adjusted when the robot 1 sequentially performs predetermined processes on the multiple target objects OBJ contained in the container CB, as shown in FIG. 15 . Therefore, the control device 3 may calculate the height OH of the multiple target objects OBJ contained in the container CB based on at least one of the image data IMG_2D and IMG_3D.

[0235] The height OH of the multiple target objects OBJ contained in the container CB may refer to the height OH of the target object group OBG that includes all of the multiple target objects OBJ contained in the container CB. Alternatively, the height OH of the multiple target objects OBJ contained in the container CB may refer to the height OH of the target object group OBG that includes some of the multiple target objects OBJ contained in the container CB. In other words, the height OH of the multiple target objects OBJ contained in the container CB may refer to the height OH of the target object group OBG that includes at least some of the multiple target objects OBJ contained in the container CB.

[0236] In this embodiment, the target object group OBG is a term used to treat a plurality of target objects OBJ as one group. The plurality of target objects OBJ included in the target object group OBG may include at least two target objects OBJ concentratedly distributed in one local area. The plurality of target objects OBJ included in the target object group OBG may include at least two target objects OBJ that are separated by a predetermined distance or more. In other words, the plurality of target objects OBJ included in the target object group OBG may include at least two target objects OBJ that are each distributed in at least two areas that are separated by a predetermined distance or more.

[0237] Considering that at least one of the image data IMG_2D and IMG_3D is used to calculate the height OH of the target object group OBG as described above, the height OH of the target object group OBG corresponds to the height OH of the multiple target objects OBJ that appear in the image represented by at least one of the image data IMG_2D and IMG_3D. In other words, the height OH of the target object group OBG corresponds to the height OH of the target object group OBG that includes at least a portion of the multiple target objects OBJ contained in the container CB that are included in the imaging field of view of the imaging system 2.

[0238] The height OH of the target object group OBG may mean the height OH of the target object group OBG from a reference point, similar to the imaging height CH of the imaging system 2 described above. Note that the height OH of the target object group OBG may be the Z-axis coordinate value of the target object group OBG in the global coordinate system. The height OH of the target object group OBG may be the Z-axis coordinate value of the target object group OBG in a coordinate system (e.g., a robot coordinate system) different from the global coordinate system.

[0239] Note that the reference point for the height OH of the target object OBG may be the same as the reference point for the imaging height CH of the imaging system 2. However, the reference point for the height OH of the target object OBG may be different from the reference point for the imaging height CH of the imaging system 2. For example, the height of the target object OBG may mean the height from a first reference point (e.g., a point on the bottom surface of the bottom wall BS of the container CB), and the height of the target object OBG may mean the height from a second reference point (e.g., a point on the target object OBJ) that is different from the first reference point.

[0240] The height OH of the target object group OBG may be the height of the highest portion of the target object group OBG. For example, the height of one target object OBJ located at the highest position among the multiple target objects OBJ included in the target object group OBG may be used as the height OH of the target object group OBG. For example, the height of the highest portion of one target object OBJ located at the highest position among the multiple target objects OBJ included in the target object group OBG may be used as the height OH of the target object group OBG. For example, the height OH of the target object group OBG may be the height of an arbitrary portion of one target object OBJ located at the highest position among the multiple target objects OBJ included in the target object group OBG.

[0241] The height OH of the target object group OBG may be the height of the highest portion of a portion of the target object group OBG. For example, the height of at least one target object OBJ selected based on desired selection criteria from among the multiple target objects OBJ included in the target object group OBG may be used as the height OH of the target object group OBG. For example, the height of the highest portion of at least one target object OBJ selected based on desired selection criteria from among the multiple target objects OBJ included in the target object group OBG may be used as the height OH of the target object group OBG. For example, the height of an arbitrary portion of at least one target object OBJ selected based on desired selection criteria from among the multiple target objects OBJ included in the target object group OBG may be used as the height OH of the target object group OBG.

[0242] The average height of the target object group OBG may be used as the height OH of the target object group OBG. The average height of the target object group OBG may mean the average value of the heights of multiple portions of the target object group OBG. The average height of the target object group OBG may mean the average value of the heights of multiple target objects OBJ included in the target object group OBG. When multiple target objects OBJ are piled up (i.e., loosely stacked), the average height of the target object group OBG may mean the average value of the heights of some target objects OBJ (i.e., some target objects OBJ located in the uppermost layer) among the multiple target objects OBJ included in the target object group OBG that have no other target objects OBJ stacked on top of them. When multiple target objects OBJ are piled up (i.e., loosely stacked), the average height of the target object group OBG may mean the average value of the heights of some target objects OBJ (i.e., some target objects OBJ located in the uppermost layer) among the multiple target objects OBJ included in the target object group OBG that have at least a portion of their upper part exposed upward (i.e., some target objects OBJ located in the uppermost layer).

[0243] The height OH of the target object group OBG may be an average height of a portion of the target object group OBG. The average height of the portion of the target object group OBG may mean an average value of the heights of at least one target object OBJ selected based on a desired selection criterion from among the multiple target objects OBJ included in the target object group OBG.

[0244] The control device 3 calculates the height OH of the target object group OBG using at least one of the image data IMG_2D and IMG_3D as described above. For example, the control device 3 may calculate the height OH of the target object group OBG by analyzing an image represented by at least one of the image data IMG_2D and IMG_3D. Alternatively, the control device 3 may calculate at least one of the position and orientation of the target object OBJ included in the target object group OBG by performing a 2D matching process using the image data IMG_2D, and calculate the height OH of the target object group OBG based on the calculation results of at least one of the position and orientation of the target object OBJ. Alternatively, the control device 3 may place a three-dimensional model of the target object OBJ in a virtual simulation space that simulates the three-dimensional space in which the target object group OBG is located, based on the calculation results of at least one of the position and orientation of the target object OBJ obtained by the 2D matching process, and calculate the height of the three-dimensional model as the height OH of the target object group OBG. Alternatively, in addition to or instead of analyzing the image represented by at least one of the image data IMG_2D and IMG_3D, the control device 3 may calculate the height OH of the target object group OBG using three-dimensional position data WSD generated from the image data IMG_3D. As one example, the control device 3 may calculate at least one of the position and orientation of the target object OBJ included in the target object group OBG by performing a 3D matching process using the three-dimensional position data WSD, and calculate the height OH of the target object group OBG based on the calculation result of at least one of the position and orientation of the target object OBJ. As another example, the control device 3 may place a three-dimensional model of the target object OBJ in a virtual simulation space that simulates the three-dimensional space in which the target object group OBG is located, based on the calculation result of at least one of the position and orientation of the target object OBJ obtained by the 3D matching process, and calculate the height of the three-dimensional model as the height OH of the target object group OBG. Furthermore, when three-dimensional position data WSD is used to calculate the height OH of the target object group OBG, the control device 3 may use the three-dimensional position data WSD generated in step S2 of Figure 4 to calculate the height OH of the target object group OBG.The control device 3 may generate new three-dimensional position data WSD separate from the three-dimensional position data WSD generated in step S2 of Figure 4, and calculate the height OH of the target object group OBG based on the newly generated three-dimensional position data WSD.

[0245] When calculating the height OH of the target object group OBG using the three-dimensional position data WSD, the control device 3 may calculate the height of the point cloud indicated by the three-dimensional position data WSD (particularly the height of the point cloud corresponding to the target object group OBG) as the height OH of the target object group OBG. Specifically, the control device 3 may calculate the height of at least some of the multiple points included in the point cloud indicated by the three-dimensional position data WSD as the height OH of the target object group OBG. For example, as shown in FIG. 16( a), which is a cross-sectional view showing the point cloud indicated by the three-dimensional position data WSD, the control device 3 may calculate the height of the highest point among the multiple points included in the point cloud indicated by the three-dimensional position data WSD as the height OH of the target object group OBG. In this case, the control device 3 may calculate the height of the highest part of the target object group OBG as the height OH of the target object group OBG. Alternatively, for example, the control device 3 may calculate the height of the Nth (where N is a constant indicating an integer greater than or equal to 2) highest point among the multiple points included in the point cloud indicated by the three-dimensional position data WSD as the height OH of the target object group OBG. In this case, the control device 3 may calculate the height of one target object OBJ that is the Nth highest among the multiple target objects OBJ included in the target object group OBG as the height OH of the target object group OBG. Alternatively, for example, the control device 3 may calculate the average value of the heights of at least some of the multiple points included in the point cloud indicated by the three-dimensional position data WSD as the height OH of the target object group OBG. In this case, the control device 3 may calculate the average height of at least some of the target object group OBG as the height OH of the target object group OBG. As an example, the control device 3 may calculate the average value of all the heights of the multiple points included in the point cloud indicated by the three-dimensional position data WSD as the height OH of the target object group OBG. As another example, as shown in Figure 16 (b), which is a cross-sectional view showing a point cloud indicated by the three-dimensional position data WSD, the control device 3 may calculate the average value of the heights of a portion of the points included in a predetermined point cloud space PCSP among the multiple points included in the point cloud indicated by the three-dimensional position data WSD as the height OH of the target object group OBG.16(b), an example of the predetermined point cloud space PCSP is a space including the point located at the highest (or the Nth highest) position among the multiple points included in the point cloud indicated by the three-dimensional position data WSD. For example, the predetermined point cloud space PCSP may be a three-dimensional space including the space between a virtual plane (e.g., a horizontal plane) including the point (highest point) located at the highest (or the Nth highest) position among the multiple points included in the point cloud indicated by the three-dimensional position data WSD, and a virtual plane (e.g., a horizontal plane) located at a position a predetermined height lower along the Z-axis direction from the point (highest point).

[0246] When calculating the height OH of the target object group OBG using the image data IMG_2D, the control device 3 may perform segmentation using the image data IMG_2D to calculate the height of the target object OBJ reflected in the image represented by the image data IMG_2D as the height OH of the target object group OBG. Alternatively, the control device 3 may perform 2D matching processing using the image data IMG_2D to calculate the position and orientation of the target object OBJ, and then calculate the calculated position (particularly, the position and height in the Z-axis direction) of the target object OBJ as the height OH of the target object group OBG. Alternatively, the control device 3 may calculate the height (i.e., the position in the Z-axis direction) of the target object group OBG using the position and orientation of the target object OBJ calculated by performing 2D matching processing and a template model TM3 of the target object OBJ.

[0247] After calculating the height OH of the target object group OBG, the control device 3 may determine the target imaging height CH_target based on the calculated height OH of the target object group OBG. Specifically, the control device 3 may determine the target imaging height CH_target based on the calculated height OH of the target object group OBG and the permissible imaging range CAR of the imaging system 2. For example, as shown in FIG. 17 , which is a cross-sectional view showing the relationship between the height OH of the target object group OBG and the permissible imaging range CAR of the imaging system 2, the control device 3 may determine the target imaging height CH_target so that the calculated height OH is included in the permissible imaging range CAR of the imaging system 2 located at the target imaging height CH_target. In other words, as shown in FIG. 17 , the control device 3 may determine the target imaging height CH_target so that the portion of the target object group OBG located at the calculated height OH is included in the permissible imaging range CAR of the imaging system 2 located at the target imaging height CH_target. The control device 3 may determine the target imaging height CH_target so that at least a portion of the target object OBJ that is included in the target object group OBG and that is located at the calculated height OH is included in the permissible imaging range CAR of the imaging system 2 located at the target imaging height CH_target, as shown in Fig. 17. As a result, the target imaging height CH_target is determined so as to satisfy the condition that at least one target object OBJ is included in the permissible imaging range CAR of the imaging system 2 located at the target imaging height CH_target, as shown in Fig. 17. Note that the control device 3 may determine the target imaging height CH_target so that at least a portion of the target object OBJ that is located at the calculated height OH is located at the best focus position BFP of the imaging system 2 located at the target imaging height CH_target.

[0248] After the target imaging height CH_target is determined, the control device 3 (particularly, the signal generation unit 313) generates a robot control signal for controlling the robot 1 so that the imaging system 2 is positioned at the determined target imaging height CH_target. That is, the control device 3 (particularly, the signal generation unit 313) generates a robot control signal for controlling the robot 1 so that the imaging height CH of the imaging system 2 coincides with the determined target imaging height CH_target. In other words, the control device 3 (particularly, the signal generation unit 313) generates a robot control signal for controlling the robot 1 so that the imaging system 2 moves to the determined target imaging height CH_target. As a result, the robot 1 moves the imaging system 2 in accordance with the robot control signal, thereby completing the adjustment of the imaging height CH of the imaging system 2. That is, the imaging system 2 moves in accordance with the robot control signal, thereby completing the adjustment of the imaging height CH of the imaging system 2.

[0249] The control device 3 performs the above-described process of adjusting the imaging height CH of the imaging system 2 each time the robot 1 completes a predetermined process on the processing object. Here, if the predetermined process performed by the robot 1 includes a process of holding one of the multiple target objects OBJ contained in the container CB and a release process of holding the target object OBJ and then releasing the held target object OBJ to a release position outside the container CB, the number of target objects OBJ contained in the container CB decreases each time the robot 1 completes a predetermined process on the processing object. Therefore, it is highly likely that the height OH of the target object group OBG calculated in step S7 of FIG. 4 will decrease over time. As a result, it is highly likely that the target imaging height CH_target calculated based on the height OH of the target object group OBG in step S7 of FIG. 4 will also decrease over time.

[0250] 18(a) shows an example in which the height OH of the target objects group OBG contained in the container CB at a first time is a first height OH#1. In this case, in step S7 of Fig. 4, the control device 3 may adjust the imaging height CH of the imaging system 2 so that the imaging height CH of the imaging system 2 becomes a first target imaging height CH_target#1 determined based on the first height OH#1. As shown in Fig. 18(a), the first target imaging height CH_target#1 satisfies the condition that the first height OH#1 is included in the imaging allowable range CAR of the imaging system 2 located at the first target imaging height CH_target#1.

[0251] Thereafter, between the first time point and a second time point that is later than the first time point, the robot 1 holds at least one target object OBJ contained in the container CB and releases the held target object OBJ to a release position outside the container CB. That is, the robot 1 carries at least one target object OBJ that was contained in the container CB out of the container CB. As a result, as shown in FIG. 18( b ), which shows the target object group OBG contained in the container CB at the second time point, the number of target objects OBJ included in the target object group OBG contained in the container CB at the second time point is less than the number of target objects OBJ included in the target object group OBG contained in the container CB at the first time point. As a result, it is highly likely that the height OH of the target object group OBG contained in the container CB at the second time point will be a second height OH#2 that is lower than the first height OH#2, which is the height of the target objects OBJ included in the target object group OBG contained in the container CB at the first time point.

[0252] 18(b), if the imaging height CH of the imaging system 2 is not adjusted at the second time, the imaging height CH of the imaging system 2 will be maintained at the first target imaging height CH_target #1. As a result, as shown in FIG. 18(b), at the second time, the target objects OBG contained in the container CB may no longer be included in the permissible imaging range CAR of the imaging system 2 located at the first target imaging height CH_target #1. This is because the target objects OBJ that were included in the permissible imaging range CAR of the imaging system 2 located at the first target imaging height CH_target #1 between the first time and the second time are carried out of the container CB by the robot 1. As a result, there is a possibility that the accuracy of the position and orientation data POI generated based on at least one of the image data IMG_2D and IMG_3D generated by the imaging system 2 capturing images of the target object group OBG in a situation where the target object group OBG contained in the container CB is no longer included in the imaging allowable range CAR of the imaging system 2 will deteriorate. In other words, there is a possibility that the accuracy of calculating at least one of the position and orientation of the target object OBJ included in the target object group OBG will deteriorate. The technical reasons for this will be explained below.

[0253] First, the technical reasons for the deterioration in accuracy of the position and orientation data POI generated based on the image data IMG_3D will be described. As described above, in order to generate the position and orientation data POI based on the image data IMG_3D, the control device 3 first acquires image data IMG_3D representing the image 223L captured by the left camera 22L and the image 223R captured by the right camera 22R, as shown in Figures 13(a) and 13(b) described above. Then, the control device 3 searches for the same part BP of the target object OBJ in each of the search areas 224L and 224R in the image 223L and calculates the deviation between the position of the part BP detected in the search area 224L and the position of the same part BP detected in the search area 224R as the parallax. Here, when the target object OBJ captured by the imaging system 2 is not included in the permissible imaging range CAR, the target object OBJ may not appear clearly in each of the images 223L and 223R compared to when the target object OBJ captured by the imaging system 2 is included in the permissible imaging range CAR. For example, the target object OBJ may appear blurred in each of the images 223L and 223R. Therefore, when the target object OBJ captured by the imaging system 2 is not included in the permissible imaging range CAR, the accuracy of stereo matching is lower compared to when the target object OBJ captured by the imaging system 2 is included in the permissible imaging range CAR. When the accuracy of stereo matching is lower, the accuracy of calculating the parallax based on the partial BP detected by stereo matching is also lower. As a result, the three-dimensional position data WSD generated based on the low-accuracy parallax may not accurately represent the three-dimensional position of the target object OBJ. As a result, the position and orientation data POI generated based on the low-accuracy three-dimensional position data WSD may not accurately represent at least one of the position and orientation of the target object OBJ.

[0254] Next, technical reasons for the deterioration in accuracy of the position and orientation data POI generated based on the image data IMG_2D will be explained. In this case, too, when the target object OBJ captured by the imaging system 2 is not included in the permissible imaging range CAR, the target object OBJ may not appear clearly in the image represented by the image data IMG_2D, compared to when the target object OBJ captured by the imaging system 2 is included in the permissible imaging range CAR. For example, the target object OBJ may appear blurred in the image represented by the image data IMG_2D. Therefore, when the target object OBJ captured by the imaging system 2 is not included in the permissible imaging range CAR, the accuracy of the 2D matching process that searches for the same image portion as the template image in the image represented by the image data IMG_2D is lower, compared to when the target object OBJ captured by the imaging system 2 is included in the permissible imaging range CAR. As a result, the position and orientation data POI generated as a result of a low-accuracy 2D matching process may not accurately represent at least one of the position and orientation of the target object OBJ.

[0255] However, in this embodiment, as shown in Fig. 19 , the imaging height CH of the imaging system 2 is adjusted in accordance with changes in the height OH of the target object group OBG. Specifically, in step S7 of Fig. 4 , the control device 3 adjusts the imaging height CH of the imaging system 2 so that the imaging height CH of the imaging system 2 changes from a first target imaging height CH_target#1 determined based on the first height OH#1 to a second target imaging height CH_target#2 determined based on the second height OH#2. As shown in Fig. 19 , the second target imaging height CH_target#2 satisfies the condition that the second height OH#2 is included in the imaging allowable range CAR of the imaging system 2 located at the second target imaging height CH_target#2. As a result, as shown in Fig. 19 , at the second time, at least a portion of the target object group OBG contained in the container CB is included in the imaging allowable range CAR of the imaging system 2 located at the second target imaging height CH_target#2. In other words, the target object OBJ that was not included in the imaging allowable range CAR of the imaging system 2 located at the first target imaging height CH_target #1 will be included in the imaging allowable range CAR of the imaging system 2 located at the second target imaging height CH_target #2. As a result, the control device 3 can generate position and orientation data POI that accurately indicates at least one of the position and orientation of the target object OBJ included in the target object group OBG based on at least one of the image data IMG_2D and IMG_3D that are generated by the imaging system 2 capturing an image of the target object group OBG under circumstances in which the target object group OBG contained in the container CB is included in the imaging allowable range CAR of the imaging system 2. For example, when the position and orientation data POI is generated based on the image data IMG_3D, because the target object OBJ captured by the imaging system 2 is included in the imaging allowable range CAR, the target object OBJ appears more clearly in each of the images 223L and 223R compared to when the target object OBJ captured by the imaging system 2 is not included in the imaging allowable range CAR.Therefore, when the target object OBJ captured by the imaging system 2 is included in the permissible imaging range CAR, the accuracy of stereo matching is higher than when the target object OBJ captured by the imaging system 2 is not included in the permissible imaging range CAR. Higher accuracy in stereo matching increases the accuracy of calculating the parallax based on the partial BP detected by stereo matching. As a result, the control device 3 can generate three-dimensional position data WSD indicating the three-dimensional position of the target object OBJ with higher accuracy based on the highly accurate parallax. As a result, the control device 3 can generate position and orientation data POI indicating at least one of the position and orientation of the target object OBJ with higher accuracy based on the highly accurate three-dimensional position data WSD. As a result, the control device 3 can generate position and orientation data POI indicating at least one of the position and orientation of the target object OBJ with higher accuracy based on the highly accurate position and orientation data POI.

[0256] For example, when the position and orientation data POI is generated based on the image data IMG_2D, the target object OBJ captured by the imaging system 2 is included in the permissible imaging range CAR, and therefore the target object OBJ is more clearly reflected in the image represented by the image data IMG_2D than when the target object OBJ captured by the imaging system 2 is not included in the permissible imaging range CAR. Therefore, when the target object OBJ captured by the imaging system 2 is included in the permissible imaging range CAR, the accuracy of the 2D matching process is higher than when the target object OBJ captured by the imaging system 2 is not included in the permissible imaging range CAR. As a result, the control device 3 can generate position and orientation data POI that indicates with higher accuracy at least one of the position and orientation of the target object OBJ, which is generated as a result of the highly accurate 2D matching process.

[0257] As described above, in this embodiment, because the imaging height CH of the imaging system 2 is adjusted in accordance with changes in the height OH of the target object group OBG, even when the height OH of the target object group OBG changes, the control device 3 can generate position and orientation data POI that indicates with higher accuracy at least one of the position and orientation of the target object OBJ. For example, even when the height OH of the target object group OBG changes as a result of the robot 1 sequentially carrying out multiple target objects OBJ contained in the container CB out of the container CB, the control device 3 can generate position and orientation data POI that indicates with higher accuracy at least one of the position and orientation of the target object OBJ.

[0258] When position and orientation data POI indicating at least one of the position and orientation of the target object OBJ with high accuracy is generated in this manner, the control device 3 can control the robot 1 so that the end effector 4 appropriately performs a predetermined process on at least one target object OBJ included in the target object group OBG. For example, when the height OH of the target object group OBG changes as the robot 1 sequentially carries multiple target objects OBJ out of the container CB, the control device 3 can control the robot 1 so that the end effector 4 appropriately performs a predetermined process on at least one target object OBJ included in the target object group OBG. As an example, because at least one of the position and orientation of the target object OBJ is calculated with high accuracy, the control device 3 can control the robot 1 so that the actual positional relationship between the target object OBJ and the end effector 4 appropriately approaches the ideal positional relationship when the end effector 4 performs a predetermined process on the target object OBJ. As an example, because at least one of the position and orientation of the target object OBJ is calculated with high accuracy, the control device 3 can control the robot 1 so as to reduce the deviation between the actual position of the end effector 4 relative to the target object OBJ and the ideal position of the end effector 4 relative to the target object OBJ when performing a specified processing on the target object OBJ.

[0259] In the robot control process shown in FIG. 4 , the control device 3 adjusts the imaging height CH of the imaging system 2 each time the robot 1 completes a predetermined process on one target object OBJ selected as the processing execution object. However, simply removing one target object OBJ from the container CB (i.e., the robot 1 holding one target object OBJ and releasing it to a release position outside the container CB) may not significantly change the height OH of the group of target objects OBG contained in the container CB. In this case, there is not much need to adjust the imaging height CH of the imaging system 2 in accordance with changes in the height OH of the group of target objects OBG. Therefore, the control device 3 does not necessarily need to adjust the imaging height CH of the imaging system 2 each time the robot 1 completes a predetermined process on one processing execution object. Specifically, the control device 3 may adjust the imaging height CH of the imaging system 2 each time the robot 1 completes a predetermined process on two or more processing execution objects. Even in this case, the above-described technical effect can be achieved.

[0260] The user may use the input device 34 to input the interval for adjusting the imaging height CH of the imaging system 2 (step S7). The input interval may be, for example, the number of target objects for which processing has been completed since the adjustment of the imaging height CH of the imaging system 2 was completed, or the elapsed time since the adjustment of the imaging height CH of the imaging system 2 was completed. The user may use the input device 34 to input the timing for adjusting the imaging height CH of the imaging system 2 (step S7).

[0261] 4 , after the robot control process is started, the imaging height CH of the imaging system 2 is not adjusted before the imaging system 2 captures an image of the target object group OBG, which includes at least some of the target objects OBJ contained in the container CB, for the first time. In particular, the imaging height CH of the imaging system 2 is not adjusted to match the height OH of the target object group OBG. In this case, the control device 3 may adjust the imaging height CH of the imaging system 2 so that the imaging height CH of the imaging system 2 becomes the initial imaging height. The initial imaging height may be set automatically by the control device 3 or by the user. Alternatively, the imaging system 2 may image the target object group OBG to generate at least one of image data IMG_2D and IMG_3D used for initially adjusting the imaging height CH of the imaging system 2, separately from at least one of image data IMG_2D and IMG_3D acquired by the control device 3 in step S1 of Fig. 4, and the control device 3 may initially adjust the imaging height CH of the imaging system 2 based on at least one of the image data IMG_2D and IMG_3D used for initially adjusting the imaging height CH. Note that the process of initially adjusting the imaging height CH of the imaging system 2 may be the same as the process of adjusting the imaging height CH of the imaging system 2 in step S7 of Fig. 4.

[0262] (2-3) Specific Example of Robot Control Processing Next, a specific example of the robot control processing will be described. In the following, a specific example of the robot control processing will be described, which is performed when a container CB contains a plurality of target objects OBJ, and the robot 1 repeatedly holds one of the plurality of target objects OBJ contained in the container CB and releases the held target object OBJ to a release position outside the container CB.

[0263] After the robot control process shown in FIG. 4 is started, as shown in FIG. 20A, the imaging system 2 captures an image of a first target object group OBG#11 including at least some of the multiple target objects OBJ contained in the container CB (step S1 in FIG. 4). In this case, as shown in FIG. 20A, the imaging height CH of the imaging system 2 may be set to a first target imaging height CH_target#11, which is an initial imaging height. In other words, the imaging system 2 positioned at the first target imaging height CH_target#11 may capture an image of the first target object group OBG#11. As a result, the control device 3 acquires image data IMG_3D (referred to as image data IMG_3D#11) from the imaging system 2, which indicates an image in which the first target object group OBG#11 is captured (step S1 in FIG. 4). Note that, for convenience of explanation, in the following description, the target object OBJ included in the first target object group OBG#11 will be referred to as the first target object OBJ#11. Thereafter, the position and orientation calculation unit 312 generates position and orientation data POI (referred to as position and orientation data POI#11) of one first target object OBJ#11 selected as the processing execution object from the multiple first target objects OBJ#11 based on the image data IMG_3D#11 (specifically, based on the three-dimensional position data WSD generated from the image data IMG_3D#11) (steps S2 to S3 in FIG. 4). In the following description, the one first target object OBJ#11 selected as the processing execution object will be referred to as first target object OBJ#11_target.

[0264] The position and orientation calculation unit 312 is likely to select the first target object OBJ#11, which is included in the permissible imaging range CAR of the imaging system 2 located at the first target imaging height CH_target#11, as the first target object OBJ#11_target, which is the processing execution object. This is because the matching similarity of the target object OBJ included in the permissible imaging range CAR is likely to be higher than the matching similarity of the target object OBJ not included in the permissible imaging range CAR. In this case, the position and orientation calculation unit 312 can generate position and orientation data POI that accurately indicates at least one of the position and orientation of the first target object OBJ#11 included in the permissible imaging range CAR of the imaging system 2 located at the first target imaging height CH_target#11. This is because the higher the matching similarity of the target object OBJ, the higher the likelihood that the accuracy of the position and orientation data POI of the target object OBJ will be. However, the position and orientation calculation unit 312 may select the first target object OBJ#11, which is not included in the imaging allowable range CAR of the imaging system 2 located at the first target imaging height CH_target#11, as the first target object OBJ#11_target, which is the object to be processed.

[0265] Thereafter, the signal generating unit 313 generates a robot control signal for controlling the robot 1 to bring the end effector 4 closer to the first target object OBJ#11_target based on the position and orientation data POI#11 of the first target object OBJ#11_target (step S4 in FIG. 4 ). After the end effector 4 approaches the first target object OBJ#11_target in accordance with the robot control signal generated in step S4, the robot 1 performs a predetermined process on the first target object OBJ#11_target in step S5. That is, as the predetermined process, the robot 1 may hold the first target object OBJ#11_target contained in the container CB, carry the held first target object OBJ#11_target out of the container CB, and release it to a release position outside the container CB.

[0266] Thereafter, after the robot 1 performs a predetermined process on the first target object OBJ#11_target (step S5 in FIG. 4 : Yes), the control device 3 adjusts the imaging height CH of the imaging system 2 (step S7 in FIG. 4 ). As described above, the control device 3 may adjust the imaging height CH of the imaging system 2 after the robot 1 performs a predetermined process on a single first target object OBJ#11. Alternatively, as described above, the control device 3 may adjust the imaging height CH of the imaging system 2 after the robot 1 performs a predetermined process on each of the multiple first target objects OBJ#11. For convenience of explanation, the following describes an example in which the control device 3 adjusts the imaging height CH of the imaging system 2 after the robot 1 performs a predetermined process on a single first target object OBJ#11.

[0267] In order to adjust the imaging height CH of the imaging system 2, the control device 3 calculates the height OH of the first target object group OBG#11 using the image data IMG_3D#11 (specifically, the three-dimensional position data WSD generated from the image data IMG_3D#11) already acquired in step S1 of Fig. 4 to generate a robot control signal for controlling the robot 1 to bring the end effector 4 closer to the first target object OBJ#11_target. As a result, as shown in Fig. 20(b), the control device 3 calculates a first height OH#11 as the height OH of the first target object group OBG#11.

[0268] 20(a), at the time when the first target object group OBG#11 is imaged, the first target object OBJ#11_target, which has not yet been carried out of the container CB by the robot 1, is located at the highest position among the first target object group OBG#11. Therefore, the control device 3 calculates the height of the first target object OBJ#11_target as the height OH of the first target object group OBG#11. However, as shown in FIG. 20(b), at the time when the imaging height CH of the imaging system 2 is actually adjusted based on the height OH of the first target object group OBG#11, the first target object OBJ#11_target has already been carried out of the container CB. For this reason, when the robot control process shown in Fig. 4 is performed, there is a possibility that the height OH of the first target object group OBG#11 calculated in step S7 of Fig. 4 contains an error with respect to the height OH of the target object group OBG actually contained in the container CB at the time when the imaging height CH of the imaging system 2 is actually adjusted based on the height OH of the first target object group OBG#11. However, even in this case, the error contained in the height OH of the first target object group OBG#11 calculated in step S7 of Fig. 4 is unlikely to be large enough to adversely affect the adjustment of the imaging height CH of the imaging system 2. However, even when the robot control process shown in Fig. 4 is performed, there is a possibility that the height OH of the first target object group OBG#11 calculated in step S7 of Fig. 4 will match the height OH of the target object group OBG actually contained in the container CB at the time when the imaging height CH of the imaging system 2 is actually adjusted based on the height OH of the first target object group OBG#11.

[0269] Note that, if at least one of the image data IMG_2D and IMG_3D used to calculate the height OH of the target object group OBG includes information about the target objects OBJ that have already been taken out of the container CB at the time when the imaging height CH of the imaging system 2 is actually adjusted based on the height OH of the target object group OBG, the control device 3 may perform a removal process to remove information about the target objects OBJ that have already been taken out of the container CB. For example, if the height OH of the target object group OBG is calculated based on three-dimensional position data WSD (e.g., point cloud data) generated from the image data IMG_3D, the control device 3 may perform a removal process to remove, from the three-dimensional position data WSD (e.g., point cloud data), three-dimensional position data corresponding to the target objects OBJ that have already been taken out of the container CB at the time when the imaging height CH of the imaging system 2 is actually adjusted based on the height OH of the target object group OBG. In this case, the three-dimensional position data WSD after the removal process will be substantially the same as the three-dimensional position data WSD generated from the image data IMG_3D acquired in step S8a of FIG. 24 to adjust the imaging height CH of the imaging system 2 in a first modified example described below. The control device 3 may then calculate the height OH of the target object group OBG based on the three-dimensional position data WSD after the removal process (e.g., point cloud data). In this case, the error in the calculation result of the height OH of the target object group OBG will be smaller than when the removal process is not performed. It should be noted that the control device 3 may perform a removal process to remove information about the target objects OBJ that have already been removed from the container CB in any situation in which the height OH of the target object group OBG is calculated, not limited to the situations shown in FIGS. 20( a) and 20(b).

[0270] The control device 3 then determines a target imaging height CH_target of the imaging system 2 based on the calculated first height OH#11 and the imaging allowable range CAR of the imaging system 2. Specifically, as shown in FIG. 20B , the control device 3 may determine, as the second target imaging height CH_target#12, a target imaging height CH_target that satisfies the condition that "the calculated first height OH#11 is included in the imaging allowable range CAR of the imaging system 2 located at the target imaging height CH_target." The control device 3 then generates a robot control signal for controlling the robot 1 so that the imaging height CH of the imaging system 2 becomes the second target imaging height CH_target#12 determined based on the first height OH#11. As a result, the imaging system 2 is positioned at the second target imaging height CH_target#12.

[0271] 21(a), the imaging system 2 positioned at the second target imaging height CH_target#12 images a second target object group OBG#12 including at least some of the target objects OBJ contained in the container CB (step S1 in FIG. 4). As a result, the control device 3 acquires image data IMG_3D (referred to as image data IMG_3D#12) representing an image in which the second target object group OBG#12 appears from the imaging system 2 (step S1 in FIG. 4). In the following description, for convenience of explanation, the target object OBJ included in the second target object group OBG#12 will be referred to as the second target object OBJ#12.

[0272] The second target object group OBG#12 imaged by the imaging system 2 located at the second target imaging height CH_target#12 may be different from the first target object group OBG#11 imaged by the imaging system 2 to determine the second target imaging height CH_target#12. Specifically, at least one of the multiple first target objects OBJ#11 included in the first target object group OBG#11 does not have to be included in the second target object group OBG#12. As an example, at the time the imaging system 2 located at the second target imaging height CH_target#12 images the second target object group OBG#12, at least one of the multiple first target objects OBJ#11 included in the first target object group OBG#11 should have already been carried out of the container CB by the robot 1. In this case, at least one first target object OBJ#11 that has already been removed from the container CB among the multiple first target objects OBJ#11 included in the first target object group OBG#11 may not be included in the second target object group OBG#12. As another example, the width of the field of view of the imaging system 2 for the target object group OBG contained in the container CB may change due to adjustment of the imaging height CH of the imaging system 2. In this case, the second target object group OBG#12 may be different from the first target object group OBG#11.

[0273] However, at least one first target object OBJ#11 among the multiple first target objects OBJ#11 included in the first target object group OBG#11 that has not yet been removed from the container CB may be included in the second target object group OBG#12. In other words, the first target object group OBG#11 and the second target object group OBG#12 may partially overlap. In this case, because the second target imaging height CH_target#12 of the imaging system 2 that images the second target object group OBG#12 is determined based on the first height OH#11 of the first target object group OBG#11, as shown in FIG. 21( a ), the imaging permissible range CAR of the imaging system 2 located at the second target imaging height CH_target#12 includes at least a portion of the second target object group OBG#12 that partially overlaps with the first target object group OBG#11. That is, the imaging allowable range CAR of the imaging system 2 located at the second target imaging height CH_target#12 includes at least one of the multiple second target objects OBJ#12 included in the second target object group OBG#12. Therefore, the imaging system 2 can properly image at least one of the multiple second target objects OBJ#12 included in the second target object group OBG#12. However, all of the multiple second target objects OBJ#12 included in the second target object group OBG#12 may be different from the multiple first target objects OBJ#11 included in the first target object group OBG#11.

[0274] Furthermore, as described above, since at least one first target object OBJ#11 has been removed from the container CB at the time the imaging system 2 images the second target object group OBG#12 from the second target imaging height CH_target#12, the number of target objects OBJ contained in the container CB at the time the imaging system 2 images the second target object group OBG#12 from the second target imaging height CH_target#12 may be less than the number of target objects OBJ contained in the container CB at the time the imaging system 2 images the first target object group OBG#11 from the first target imaging height CH_target#11. The number of second target objects OBJ#12 included in the second target object group OBG#12 imaged by the imaging system 2 from the second target imaging height CH_target#12 may be smaller than the number of first target objects OBJ#11 included in the first target object group OBG#11 imaged by the imaging system 2 from the first target imaging height CH_target#11. The weight of the container CB containing multiple target objects OBJ at the time the imaging system 2 images the second target object group OBG#12 from the second target imaging height CH_target#12 may be lighter than the weight of the container CB containing multiple target objects OBJ at the time the imaging system 2 images the first target object group OBG#11 from the first target imaging height CH_target#11. It should be noted that the "weight of the container CB containing multiple target objects OBJ" referred to here may be a weight that includes both the weight of the container CB itself and the weight of the multiple target objects OBJ contained in the container CB.

[0275] Thereafter, the position and orientation calculation unit 312 generates position and orientation data POI (referred to as position and orientation data POI#12) of one second target object OBJ#12 selected as the processing execution object from the plurality of second target objects OBJ#12 based on the image data IMG_3D#12 (specifically, based on the three-dimensional position data WSD generated from the image data IMG_3D#12) (steps S2 to S3 in FIG. 4). In the following description, the one second target object OBJ#12 selected as the processing execution object will be referred to as second target object OBJ#12_target.

[0276] The position and orientation calculation unit 312 is likely to select the second target object OBJ#12, which is included in the permissible imaging range CAR of the imaging system 2 located at the second target imaging height CH_target#12, as the second target object OBJ#12_target, which is the object to be processed. This is because the matching similarity of the position and orientation data POI of the target object OBJ included in the permissible imaging range CAR is likely to be higher than the matching similarity of the target object OBJ not included in the permissible imaging range CAR. In this case, the position and orientation calculation unit 312 can generate position and orientation data POI that accurately indicates at least one of the position and orientation of the second target object OBJ#12 included in the permissible imaging range CAR of the imaging system 2 located at the second target imaging height CH_target#12. This is because the higher the matching similarity of the target object OBJ, the higher the likelihood that the accuracy of the position and orientation data POI of the target object OBJ will be. Conversely, the second target imaging height CH_target#12 may be an imaging height CH that satisfies the condition that "the second target object OBJ#12 selected as the processing execution object is included in the imaging allowable range CAR of the imaging system 2 located at the second target imaging height CH_target#12." In other words, the control device 3 may determine the second target imaging height CH_target#12 so as to satisfy the condition that "the second target object OBJ#12 selected as the processing execution object is included in the imaging allowable range CAR of the imaging system 2 located at the second target imaging height CH_target#12." However, the position and orientation calculation unit 312 may select the second target object OBJ#12 that is not included in the imaging allowable range CAR of the imaging system 2 located at the second target imaging height CH_target#12 as the processing execution object, second target object OBJ#12_target.

[0277] On the other hand, the second target object OBJ#12_target selected as the processing execution object does not have to be included in the permissible imaging range CAR of the imaging system 2 before the imaging height CH is adjusted. In other words, the second target object OBJ#12_target selected as the processing execution object does not have to be included in the permissible imaging range CAR of the imaging system 2 located at the first target imaging height CH_target #11. This is because, as explained with reference to FIGS. 18( a), 18(b), and 19, the imaging height CH of the imaging system 2 is adjusted in accordance with the height OH of the group of target objects OBG contained in the container CB. Therefore, even if one target object OBJ was not included in the permissible imaging range CAR of the imaging system 2 at the first time point, the one target object OBJ will be included in the permissible imaging range CAR of the imaging system 2 at the second time point that is later than the first time point, and as a result, the one target object OBJ can be selected as the processing execution object. In other words, even if a target object OBJ is not included in the imaging allowable range CAR of the imaging system 2 located at the first target imaging height CH_target #11, over time the target object OBJ will be included in the imaging allowable range CAR of the imaging system 2 located at the second target imaging height CH_target #12, and as a result, the target object OBJ can be selected as the object to be processed.

[0278] The second target object OBJ#12_target selected as the processing execution object may be the first target object OBJ#11 that was not included in the imaging allowable range CAR of the imaging system 2 located at the first target imaging height CH_target#11. Alternatively, the second target object OBJ#12_target selected as the processing execution object may be the first target object OBJ#11 that was included in the imaging allowable range CAR of the imaging system 2 located at the first target imaging height CH_target#11 but was not selected as the processing execution object.

[0279] Thereafter, the signal generating unit 313 generates a robot control signal for controlling the robot 1 to bring the end effector 4 closer to the second target object OBJ#12_target based on the position and orientation data POI#12 of the second target object OBJ#12_target (step S4 in FIG. 4 ). After the end effector 4 approaches the second target object OBJ#12_target in accordance with the robot control signal generated in step S4, the robot 1 performs a predetermined process on the second target object OBJ#12_target in step S5. That is, the robot 1 holds the second target object OBJ#12_target contained in the container CB, carries the held second target object OBJ#12_target out of the container CB, and releases it to a release position outside the container CB.

[0280] Thereafter, after the robot 1 performs a predetermined process on the second target object OBJ#12_target (step S5 in FIG. 4 : Yes), the control device 3 adjusts the imaging height CH of the imaging system 2 (step S7 in FIG. 4 ). As described above, the control device 3 may adjust the imaging height CH of the imaging system 2 after the robot 1 performs a predetermined process on a single second target object OBJ#12. Alternatively, as described above, the control device 3 may adjust the imaging height CH of the imaging system 2 after the robot 1 performs a predetermined process on each of the multiple second target objects OBJ#12. For convenience of explanation, the following describes an example in which the control device 3 adjusts the imaging height CH of the imaging system 2 after the robot 1 performs a predetermined process on a single second target object OBJ#12.

[0281] In order to adjust the imaging height CH of the imaging system 2, the control device 3 calculates the height OH of the second target object group OBG#12 using the image data IMG_3D#12 (specifically, the three-dimensional position data WSD generated from the image data IMG_3D#12) already acquired in step S1 of Fig. 4 to generate a robot control signal for controlling the robot 1 to bring the end effector 4 closer to the second target object OBJ#12_target. As a result, as shown in Fig. 21(b), the control device 3 calculates the second height OH#12 as the height OH of the second target object group OBG#12.

[0282] As described above, at the time the second target object group OBG#12 is imaged, at least one of the multiple first target objects OBJ#11 included in the first target object group OBG#11 has been carried out of the container CB by the robot 1. Therefore, at least one of the multiple first target objects OBJ#11 included in the first target object group OBG#11 is not included in the second target object group OBG#12. In other words, the number of second target objects OBJ#12 included in the second target object group OBG#12 is fewer than the number of first target objects OBJ#11 included in the first target object group OBG#11. As a result, the second height OH#12 of the second target object group OBG#12 is usually lower than the first height OH#11 of the first target object group OBG#11. However, in some cases, the second height OH#12 of the second target object group OBG#12 may be the same as the first height OH#11 of the first target object group OBG#11.

[0283] 21( a), at the time when the second target object group OBG#12 is imaged, the second target object OBJ#12_target, which has not yet been carried out of the container CB by the robot 1, is located at the highest position in the second target object group OBG#12. Therefore, the control device 3 calculates the height of the second target object OBJ#12_target as the height OH of the second target object group OBG#12. However, as shown in FIG. 21( b), at the time when the imaging height CH of the imaging system 2 is actually adjusted based on the height OH of the second target object group OBG#12, the second target object OBJ#12_target has already been carried out of the container CB. For this reason, when the robot control process shown in Fig. 4 is performed, there is a possibility that the height OH of the second target object group OBG#12 calculated in step S7 of Fig. 4 contains an error with respect to the height OH of the target object group OBG actually contained in the container CB at the time when the imaging height CH of the imaging system 2 is actually adjusted based on the height OH of the second target object group OBG#12. However, even in this case, the error contained in the height OH of the second target object group OBG#12 calculated in step S7 of Fig. 4 is unlikely to be large enough to adversely affect the adjustment of the imaging height CH of the imaging system 2. However, even when the robot control process shown in Fig. 4 is performed, there is a possibility that the height OH of the second target object group OBG#12 calculated in step S7 of Fig. 4 will match the height OH of the target object group OBG actually contained in the container CB at the time when the imaging height CH of the imaging system 2 is actually adjusted based on the height OH of the second target object group OBG#12.

[0284] The control device 3 then determines a target imaging height CH_target of the imaging system 2 based on the calculated second height OH#12 and the imaging allowable range CAR of the imaging system 2. Specifically, as shown in FIG. 21B , the control device 3 may determine, as the third target imaging height CH_target#13, a target imaging height CH_target that satisfies the condition that "the calculated second height OH#12 is included in the imaging allowable range CAR of the imaging system 2 located at the target imaging height CH_target." The control device 3 then generates a robot control signal for controlling the robot 1 so that the imaging height CH of the imaging system 2 becomes the third target imaging height CH_target#13 determined based on the second height OH#12. As a result, the imaging system 2 is positioned at the third target imaging height CH_target#13.

[0285] As described above, the second height OH#12 of the second object group OBG#12 is usually lower than the first height OH#11 of the first object group OBG#11. In this case, the third target imaging height CH_target#13 determined based on the second height OH#12 may also be lower than the second target imaging height CH_target#12 determined based on the first height OH#11.

[0286] However, in some cases, the second height OH#12 of the second target object group OBG#12 may normally be higher than the first height OH#11 of the first target object group OBG#11. For example, in a situation where multiple first target objects OBJ#11 included in the first target object group OBG#11 are arranged so that they partially overlap (e.g., arranged randomly), if one first target object OBJ#11 with another first target object OBJ#11 overlapping it is held by the end effector 4, the pile of first target objects OBG#11 (i.e., second target object group OBG#12) from which one first target object OBJ#11 has been removed may collapse. As a result, the orientation of at least one second target object OBJ#12 included in the second target object group OBG#12 may change. As a result, for example, if the orientation of the second target object OBJ#12 changes such that its longitudinal direction changes from a direction aligned with the horizontal direction to a direction aligned with the vertical direction, the height OH of the second target object OBJ#12 (i.e., the height of the second target object group OBG#12) may increase. In this case, as will be described in a first modified example described in detail later with reference to FIG. 24 etc., after the end effector 4 holds one first target object OBJ#11 (step S5 in FIG. 4 : Yes), the imaging system 2 may image the second target object group OBG#12 (step S8a in FIG. 24 ), and the control device 3 may calculate the second height OH#12 of the second target object group OBG#12 based on the imaging results of the second target object group OBG#12 (step S7 in FIG. 4 ). Note that this is not limited to the case where the second height OH#12 of the second target object group OBG#12 is calculated, but is also the same when the height OH of any target object group OBG is calculated.

[0287] 22(a), the imaging system 2 positioned at the third target imaging height CH_target#13 images a third target object group OBG#13 including at least some of the target objects OBJ contained in the container CB (step S1 in FIG. 4). As a result, the control device 3 acquires image data IMG_3D (referred to as image data IMG_3D#13) representing an image in which the third target object group OBG#13 appears from the imaging system 2 (step S1 in FIG. 4). In the following description, for convenience of explanation, the target object OBJ included in the third target object group OBG#13 will be referred to as the third target object OBJ#13.

[0288] The third target object group OBG#13 imaged by the imaging system 2 located at the third target imaging height CH_target#13 may be different from the second target object group OBG#12 imaged by the imaging system 2 to determine the third target imaging height CH_target#13. Specifically, at least one of the multiple second target objects OBJ#12 included in the second target ...

Claims

1. A control device that generates a control signal for controlling a robot that is provided with a holding device capable of holding a target object and an imaging system that images the target object, and that moves the holding device and the imaging system, comprising: a calculation device that generates the control signal; and a communication device that outputs the control signal generated by the calculation device, wherein the calculation device generates a first control signal as the control signal for controlling the robot to move the holding device toward a first target object in order to hold a first target object, which is one of the first target object group, based on an imaging result obtained by imaging a first target object group including at least a portion of a plurality of target objects contained in a container from a first imaging height using the imaging system, A control device that generates as the control signal a second control signal for controlling the robot so that the holding device approaches the first target object based on the first control signal, and after the first target object held by the holding device is removed from the container and released from the holding device outside the container, based on an imaging result obtained by the imaging system imaging a second target object group including at least a portion of the first target object group contained in the container from a second imaging height lower than the first imaging height, in order to control the robot so that the holding device approaches the second target object in order to hold a second target object, which is one of the target objects of the second target object group.

2. The control device according to claim 1, wherein the height of the second group of target objects is lower than the height of the first group of target objects.

3. A control device as described in claim 1 or 2, wherein the number of target objects contained in the container when the first group of target objects is imaged from the second imaging height is less than the number of target objects contained in the container when the first group of target objects is imaged from the first imaging height.

4. A control device according to any one of claims 1 to 3, wherein the number of the target objects included in the second target object group is smaller than the number of the target objects included in the first target object group.

5. A control device as described in any one of claims 1 to 4, wherein the weight of the container containing the multiple target objects when the second group of target objects is imaged from the second imaging height is less than the weight of the container containing the multiple target objects when the second group of target objects is imaged from the first imaging height.

6. A control device as described in any one of claims 1 to 5, wherein the first imaging height is a height at which the first target object is included in the imaging allowable range of the imaging system positioned at the first imaging height, and the second imaging height is a height at which the second target object is included in the imaging allowable range of the imaging system positioned at the second imaging height.

7. The control device according to claim 6, wherein the first imaging height is a height at which the second target object is not included in the imaging allowable range of the imaging system positioned at the first imaging height.

8. The control device according to claim 6 or 7, wherein the imaging allowable range is at least one of the focal depth and the depth of field of the imaging system.

9. A control device according to any one of claims 6 to 8, wherein the imaging tolerance range is set based on a field of view of the imaging system.

10. A control device according to any one of claims 6 to 9, wherein the permissible imaging range is set by a user's input.

11. A control device as described in any one of claims 1 to 10, wherein the control signal is a third control signal for controlling the robot so that the holding device approaches the second target object based on the second control signal, and after the second target object held by the holding device is removed from the container and released from the holding device outside the container, a third target object group including at least a part of the second target object group contained in the container is generated based on an imaging result obtained by the imaging system from a third imaging height lower than the second imaging height, in order to control the robot so that the holding device approaches the third target object in order to hold the third target object, which is one of the target objects in the third target object group.

12. The control device according to claim 11, wherein the height of the third group of target objects is lower than the height of the second group of target objects.

13. A control device as described in claim 11 or 12, wherein the second imaging height is a height at which the second target object is included in the imaging allowable range of the imaging system positioned at the second imaging height, but the third target object is not included, and the third imaging height is a height at which the third target object is included in the imaging allowable range of the imaging system positioned at the third imaging height.

14. The control device according to any one of claims 1 to 13, wherein the imaging system includes at least one of a monocular camera and a stereo camera having two monocular cameras different from the monocular camera.

15. A control system comprising: a control device according to any one of claims 1 to 14; and the imaging system.

16. A robot system comprising: a control device according to any one of claims 1 to 14; the imaging system; and the robot.

17. A control method for generating a control signal for controlling a robot provided with a holding device capable of holding a target object and an imaging system for imaging the target object and for moving the holding device and the imaging system, comprising the steps of: generating, as the control signal, a first control signal for controlling the robot to move the holding device toward a first target object, which is one of the first target object group, in order to hold a first target object, the first target object being one of the first target object group, based on an imaging result obtained by imaging a first target object group including at least some of the target objects contained in a container from a first imaging height using the imaging system; The control method includes generating a second control signal as the control signal for controlling the robot so that the holding device approaches the first target object based on the first control signal, and after the first target object held by the holding device is removed from the container and released from the holding device outside the container, based on an imaging result in which the imaging system images a second target object group including at least a portion of the first target object group contained in the container from a second imaging height lower than the first imaging height, in order to control the robot so that the holding device approaches the second target object in order to hold the second target object, which is one of the target objects of the second target object group.

18. A computer program for causing a computer to execute the control method according to claim 17.

19. A control device provided with a processing device for processing a target object and an imaging system for imaging the target object, and configured to generate a control signal for controlling a robot for moving the processing device and the imaging system, the control device comprising: an arithmetic device for generating the control signal; and a communication device for outputting the control signal generated by the arithmetic device, wherein the arithmetic device generates, based on an imaging result of the imaging system of a first target object group including at least a portion of a plurality of the target objects contained in a container, a first control signal as the control signal for controlling the robot to move to an imaging height at which the imaging system images the target objects, and generates, based on an imaging result of the imaging system imaging a second target object group including at least a portion of the first target object group contained in the container at the imaging height, a second control signal as the control signal for controlling the robot to approach the first target object, which is one of the second target object group, in order to process the first target object.

20. The control device described in claim 19, wherein the calculation device determines the imaging height based on the imaging results of the first group of target objects by the imaging system, and generates the first control signal for controlling the robot so that the imaging system moves to the determined imaging height.

21. The control device according to claim 19 or 20, wherein the calculation device calculates the height of the first target object group based on the imaging result, and determines the imaging height based on the height of the first target object group and an imaging tolerance range of the imaging system.

22. A control device according to any one of claims 19 to 21, wherein the imaging height is a height at which the first target object is included in an imaging allowable range of the imaging system located at the imaging height.

23. The control device described in any one of claims 19 to 22, wherein the processing device is a holding device capable of holding the target object, and when the imaging height is a first imaging height, the holding device approaches the first target object based on the second control signal, and after the first target object held by the holding device is removed from the container and released from the holding device outside the container, the calculation device generates as the control signal a third control signal for controlling the robot so that the imaging system moves to a second imaging height lower than the first imaging height, based on the imaging results by the imaging system of a third target object group including at least a portion of the second target object group contained in the container.

24. The control device described in claim 23, wherein the third group of target objects is the second group of target objects, and the imaging results of the third group of target objects by the imaging system used to generate the third control signal are the imaging results of the second group of target objects by the imaging system used to generate the second control signal.

25. A control device according to claim 23 or 24, wherein the calculation device determines the second imaging height based on the imaging results of the third group of target objects by the imaging system, and generates the third control signal for controlling the robot so that the imaging system moves to the determined second imaging height.

26. A control device as described in any one of claims 23 to 25, wherein the calculation device generates, as the control signal, a fourth control signal for controlling the robot so that the holding device approaches a second target object, which is one of the target objects in the fourth target object group, in order to hold the second target object, which is one of the target objects in the fourth target object group, based on an imaging result obtained by the imaging system at the second imaging height of a fourth target object group including at least some of the target objects in the third target object group contained in the container.

27. The control device according to claim 26, wherein the height of the fourth group of objects is lower than the height of the second group of objects.

28. A control device as described in claim 26 or 27, wherein the first imaging height is a height at which the first target object is included in the imaging allowable range of the imaging system located at the first imaging height, but the second target object is not included, and the second imaging height is a height at which the second target object is included in the imaging allowable range of the imaging system located at the second imaging height.

29. The control device according to claim 21, 22 or 28, wherein the imaging allowable range is at least one of the focal depth and the depth of field of the imaging system.

30. A control device according to claim 21, 22, 28 or 29, wherein the imaging tolerance range is set based on the field of view of the imaging system.

31. The control device according to any one of claims 21, 22 and 28 to 30, wherein the permissible imaging range is set by a user's input.

32. A control device as described in any one of claims 19 to 31, wherein the calculation device generates the first control signal for controlling the robot so that the imaging system moves to at least one of an imaging attitude, which is an attitude at which the imaging system images the target objects, and an imaging horizontal position, which is a position on a horizontal plane at which the imaging system images the target objects, based on the imaging results by the imaging system of the first group of target objects.

33. The control device described in claim 32, wherein the calculation device determines at least one of the imaging attitude and the imaging horizontal position based on the imaging results of the imaging system of the first group of target objects, and generates the first control signal for controlling the robot so that the imaging system moves to the determined at least one of the imaging attitude and the imaging horizontal position.

34. The control device according to any one of claims 19 to 33, wherein the imaging system includes at least one of a monocular camera and a stereo camera having two monocular cameras different from the monocular camera.

35. A control system comprising: a control device according to any one of claims 19 to 33; and the imaging system.

36. A robot system comprising: a control device according to any one of claims 19 to 33; the imaging system; and the robot.

37. A control method for generating a control signal for controlling a robot provided with a processing device for processing a target object and an imaging system for imaging the target object, the control method comprising: generating, based on an imaging result of a first target object group by the imaging system, the first target object group including at least a part of the target objects contained in a container, a first control signal for controlling the robot to move to an imaging height at which the imaging system images the target objects; and generating, based on an imaging result of an imaging system imaging a second target object group including at least a part of the first target object group contained in the container at the imaging height, a second control signal for controlling the robot to approach the first target object, the first target object being one of the second target object group, in order to process the first target object.

38. A computer program for causing a computer to execute the control method according to claim 37.

39. A control device provided with a holding device capable of holding a target object and an imaging system that images the target object, and which generates a control signal for controlling a robot that moves the holding device and the imaging system, the control device comprising: a calculation device that generates the control signal; and a communication device that outputs the control signal generated by the calculation device, wherein a first target object which is one of a plurality of target objects contained in a container is held by the holding device, and the robot is controlled so that the holding device holding the first target object moves from the container to a position outside the container at which the first target object is released, and the calculation device generates as the control signal a first control signal for controlling the robot from a position after the first target object is released from the holding device, based on an imaging result obtained by the imaging system imaging a first target object group including at least a part of the plurality of target objects contained in the container on a movement path of the imaging system accompanying the movement of the holding device from the container to the position outside the container at which the first target object is released.

40. The control device according to claim 39, wherein the imaging results of the first group of target objects are the results of the imaging system imaging the first group of target objects contained in the container while moving along the movement path.

41. The control device described in claim 39 or 40, wherein the imaging system, which has moved based on the first control signal, generates as the control signal a second control signal for controlling the robot so that the holding device approaches the second target object in order to hold a second target object, which is one of the second target object group, based on an imaging result of the imaging system imaging a second target object group including at least some of the target objects of the first target object group contained in the container.

42. A control device as described in any one of claims 39 to 41, wherein the first control signal is a signal for controlling the robot so that the imaging system moves from a position after the first target object is released from the holding device to an imaging height at which the imaging system images the target object contained in the container.

43. The control device according to claim 42, wherein the first control signal is a signal for controlling the robot to move the imaging system from a position where the first target object is released from the holding device to the imaging height.

44. A control device as described in claim 42 or 43, wherein the calculation device determines the imaging height based on the imaging results of the first group of target objects by the imaging system, and generates the first control signal for controlling the robot so that the imaging system moves to the determined imaging height.

45. A control device as described in any one of claims 42 to 44, wherein the calculation device calculates the height of the first target object group based on the imaging results of the first target object group by the imaging system, and determines the imaging height based on the height of the first target object group and an imaging tolerance range of the imaging system.

46. ​​A control device as claimed in any one of claims 42 to 45, wherein the calculation device generates as the control signal a second control signal for controlling the robot so that the holding device approaches the second target object, which is one of the target objects in the second target object group, in order to hold the second target object, which is one of the target objects in the second target object group, based on an imaging result obtained by the imaging system, which has been moved to the imaging height based on the first control signal, imaging a second target object group including at least some of the target objects in the first target object group contained in the container.

47. The control device according to claim 46, wherein the imaging height is a height at which the second target object is included in an imaging allowable range of the imaging system located at the imaging height.

48. The control device according to claim 46 or 47, wherein, when the imaging height is a first imaging height, the second target object is held by the holding device based on the second control signal, the robot is controlled so that the holding device holding the second target object moves from the container to a position outside the container at which the second target object is released, and the computing device generates, as the control signal, a third control signal for controlling the robot so that the imaging system moves from a position after the second target object is released from the holding device to a second imaging height lower than the first imaging height, based on an imaging result of the imaging system imaging a third target object group including at least some of the target objects of the second target object group contained in the container on a movement path of the imaging system accompanying the movement of the holding device from the container to the position outside the container at which the second target object is released.

49. The control device described in claim 48, wherein the imaging results of the third group of target objects are the result of the imaging system imaging the third group of target objects contained in the container while moving along the movement path of the imaging system as the holding device moves from the container to the position where the second target objects are released outside the container.

50. A control device as described in claim 48 or 49, wherein the third control signal is a signal for controlling the robot so that the imaging system moves from a position where the second target object is released from the holding device to the second imaging height.

51. A control device as described in any one of claims 48 to 50, wherein the calculation device determines the second imaging height based on the imaging results of the third group of target objects by the imaging system, and generates the third control signal for controlling the robot so that the imaging system moves to the determined second imaging height.

52. A control device as described in any one of claims 48 to 51, wherein the calculation device generates, as the control signal, a fourth control signal for controlling the robot so that the holding device approaches the third target object, which is one of the target objects in the fourth target object group, in order to hold the third target object, which is one of the target objects in the fourth target object group, based on an imaging result obtained by the imaging system at the second imaging height of a fourth target object group including at least some of the target objects in the third target object group contained in the container.

53. The control device according to claim 52, wherein the height of the fourth group of objects is lower than the height of the second group of objects.

54. A control device as described in claim 52 or 53, wherein the first imaging height is a height at which the second target object is included in the imaging allowable range of the imaging system located at the first imaging height and the third target object is not included, and the second imaging height is a height at which the third target object is included in the imaging allowable range of the imaging system located at the second imaging height.

55. The control device according to claim 45, 47 or 54, wherein the imaging tolerance is at least one of the focal depth and the depth of field of the imaging system.

56. A control device as claimed in claim 45, 47, 54 or 55, wherein the imaging tolerance range is set based on the field of view of the imaging system.

57. A control device according to any one of claims 45, 47 and 54 to 56, wherein the imaging tolerance range is set by a user's input.

58. A control device as described in any one of claims 39 to 57, wherein the first control signal is a signal for controlling the robot to move the imaging system to at least one of an imaging attitude, which is an attitude at which the imaging system images the target object contained in the container from a position after the first target object is released from the holding device, and an imaging horizontal position, which is a position on a horizontal plane.

59. The control device described in claim 58, wherein the calculation device determines at least one of the imaging attitude and the imaging horizontal position based on the imaging results obtained when the imaging system images the first group of target objects contained in the container on the movement path of the imaging system, and generates the first control signal for controlling the robot so that the imaging system moves to the determined at least one of the imaging attitude and the imaging horizontal position.

60. A control device according to any one of claims 39 to 59, wherein the imaging system includes at least one of a monocular camera and a stereo camera having two monocular cameras different from the monocular camera.

61. A control system comprising: a control device according to any one of claims 39 to 60; and the imaging system.

62. A robot system comprising: a control device according to any one of claims 39 to 60; the imaging system; and the robot.

63. A control method for generating a control signal for controlling a robot provided with a holding device capable of holding a target object and an imaging system for imaging the target object, and for moving the holding device and the imaging system, wherein a first target object, which is one of a plurality of target objects contained in a container, is held by the holding device, and the robot is controlled so that the holding device holding the first target object moves from the container to a position outside the container at which the first target object is released, the control method including: generating a first control signal as the control signal for controlling the robot from a position after the first target object is released from the holding device, based on an imaging result obtained by the imaging system imaging a first target object group including at least a part of the plurality of target objects contained in the container, on a movement path of the imaging system accompanying the movement of the holding device from the container to the position outside the container at which the first target object is released.

64. A computer program for causing a computer to execute the control method according to claim 63.

65. A control device provided with a holding device capable of holding a target object and an imaging system that images the target object, and which generates a control signal for controlling a robot that moves the holding device and the imaging system, the control device comprising: a calculation device that generates the control signal; and a communication device that outputs the control signal generated by the calculation device, wherein the calculation device generates a first control signal as the control signal for controlling the robot after the holding device releases the first target object, based on an imaging result obtained by the imaging system capturing an image of the inside of the container during a period from when a first target object, which is one of a plurality of target objects contained in a container, is held in the holding device to when the first target object held in the holding device is removed from the container and released from the holding device outside the container.

66. The control device according to claim 65, wherein the imaging results of imaging the interior of the container include imaging results of imaging a first group of target objects including at least a portion of the plurality of target objects contained in the container.

67. The control device described in claim 66, wherein the imaging system, which has moved based on the first control signal, generates as the control signal a second control signal for controlling the robot so that the holding device approaches the second target object, which is one of the target objects in the second target object group, in order to hold the second target object, which is one of the target objects in the second target object group, based on an imaging result of the imaging system imaging a second target object group including at least some of the target objects in the first target object group contained in the container.

68. The control device according to claim 67, wherein the first control signal is a signal for controlling the robot to move the imaging system to an imaging height at which the imaging system images the second group of targets after releasing the first target object from the holding device.

69. The control device described in claim 68, wherein the calculation device determines the imaging height based on the imaging result of the first group of target objects by the imaging system, and generates the first control signal for controlling the robot so that the imaging system moves to the determined imaging height.

70. A control device as described in claim 68 or 69, wherein the calculation device calculates the height of the first target object group based on the imaging results of the first target object group by the imaging system, and determines the imaging height based on the height of the first target object group and an imaging tolerance range of the imaging system.

71. A control device according to any one of claims 68 to 70, wherein the imaging height is a height at which the second target object is included in an imaging allowable range of the imaging system located at the imaging height.

72. A control device as described in any one of claims 67 to 71, wherein the control signal is a third control signal for controlling the robot after the holding device releases the second object, based on the imaging results of the imaging system capturing images of the inside of the container during the period from when the second object is held by the holding device based on the second control signal, and when the second object held by the holding device is removed from the container and released from the holding device outside the container.

73. A control device as described in any one of claims 68 to 71, wherein when the imaging height is a first imaging height, a third control signal is generated as the control signal for controlling the robot to move to a second imaging height lower than the first imaging height at which the imaging system images the second target object group after the second target object is released from the holding device, based on an imaging result obtained by the imaging system imaging a third target object group including at least some of the target objects of the second target object group contained in the container during a period from when the second target object is held in the holding device based on the second control signal, and when the second target object held in the holding device is removed from the container and released from the holding device outside the container.

74. The control device described in claim 73, wherein the calculation device determines the second imaging height based on the imaging results of the third group of target objects by the imaging system, and generates the third control signal for controlling the robot so that the imaging system moves to the determined second imaging height.

75. The control device according to claim 73 or 74, wherein the calculation device generates, as the control signal, a fourth control signal for controlling the robot so that the holding device approaches the third target object, which is one of the target objects in the fourth target object group, in order to hold the third target object, which is one of the target objects in the fourth target object group, based on an imaging result obtained by the imaging system at the second imaging height of a fourth target object group including at least some of the target objects in the third target object group contained in the container.

76. The control device according to claim 75, wherein the height of the fourth group of objects is lower than the height of the second group of objects.

77. A control device as described in claim 75 or 76, wherein the first imaging height is a height at which the second target object is included in the imaging allowable range of the imaging system located at the first imaging height and the third target object is not included, and the second imaging height is a height at which the third target object is included in the imaging allowable range of the imaging system located at the second imaging height.

78. A control device according to claim 70, 71 or 77, wherein the imaging tolerance is at least one of the depth of focus and the depth of field of the imaging system.

79. A control device as claimed in claim 70, 71, 77 or 78, wherein the imaging tolerance range is set based on a field of view of the imaging system.

80. A control device according to any one of claims 70, 71 and 77 to 79, wherein the imaging tolerance range is set by a user's input.

81. A control device as described in any one of claims 65 to 80, wherein the first control signal is a signal for controlling the robot so that the imaging system moves to at least one of an imaging attitude, which is an attitude at which the imaging system captures an image, and an imaging horizontal position, which is a position on a horizontal plane, after the first target object is released from the holding device.

82. The control device described in claim 81, wherein the calculation device determines at least one of the imaging attitude and the imaging horizontal position based on the imaging results obtained by the imaging system imaging the inside of the container during a period from when the first target object contained in the container is held by the holding device until the first target object held by the holding device is removed from the container and released from the holding device outside the container, and generates the first control signal for controlling the robot so that the imaging system moves to the determined at least one of the imaging attitude and the imaging horizontal position.

83. The control device according to any one of claims 65 to 82, wherein the imaging system includes at least one of a monocular camera and a stereo camera having two monocular cameras different from the monocular camera.

84. A control system comprising: a control device according to any one of claims 65 to 83; and the imaging system.

85. A robot system comprising: a control device according to any one of claims 65 to 83; the imaging system; and the robot.

86. A control method for generating a control signal for controlling a robot provided with a holding device capable of holding a target object and an imaging system for imaging the target object, and for moving the holding device and the imaging system, the control method including: generating a first control signal as the control signal for controlling the robot after the holding device releases the first target object, based on an imaging result obtained by the imaging system imaging the inside of the container during a period from when a first target object, which is one of a plurality of target objects contained in a container, is held by the holding device until the first target object held by the holding device is removed from the container and released from the holding device outside the container.

87. A computer program for causing a computer to execute the control method according to claim 86.

88. A control device provided with a processing device for processing a target object and an imaging system for imaging the target object, and configured to generate a control signal for controlling a robot for moving the processing device and the imaging system, the control device comprising: a calculation device for generating the control signal; and a communication device for outputting the control signal generated by the calculation device, wherein the calculation device generates, as the control signal, a first control signal for controlling the robot to move to an imaging height at which the imaging system images the target objects, based on at least one of a height of a first target object group including at least a portion of a plurality of the target objects contained in a container and an amount of the first target object group contained in the container, and generates, as the control signal, a second control signal for controlling the robot to move to an imaging height at which the imaging system images the target objects, based on an imaging result of the imaging system imaging a second target object group including at least a portion of the first target object group contained in the container at the imaging height, so as to cause the processing device to approach one target object in the second target object group in order to process the one target object.

89. The control device according to claim 88, wherein the second group of objects is identical to the first group of objects.

90. A control device as described in claim 88 or 89, wherein the imaging height is a height at which at least one of the first target object group is included in the imaging allowable range of the imaging system located at the imaging height.

91. A control device as described in any one of claims 88 to 90, wherein the one of the second group of target objects is one of the first group of target objects included in the imaging tolerance range of the imaging system located at the imaging height.

92. A control device as described in any one of claims 88 to 91, wherein the calculation device determines the imaging height based on at least one of the height of the first group of target objects contained in the container and the amount of the first group of target objects contained in the container, and generates the first control signal for controlling the robot so that the imaging system moves to the determined imaging height.

93. A control device as described in any one of claims 88 to 92, wherein the calculation device calculates a height of the first group of target objects contained in the container based on an imaging result of the first group of target objects contained in the container by the imaging system, and generates the first control signal based on the calculated height.

94. A control device as described in any one of claims 88 to 93, wherein at least one of the height of the first group of target objects contained in the container and the amount of the first group of target objects contained in the container is calculated directly or indirectly based on measurement results of a measurement device.

95. The control device described in claim 94, wherein the imaging system is a first imaging system, the measurement device includes at least one of the first imaging system and a second imaging system different from the first imaging system, and the calculation device calculates at least one of the height of the first group of target objects contained in the container and the amount of the first group of target objects contained in the container based on the imaging results of at least one of the first and second imaging systems as the measurement result.

96. The control device according to claim 95, wherein the imaging results of at least one of the first and second imaging systems are imaging results of the first group of target objects contained in the container.

97. A control device as described in claim 95 or 96, wherein the imaging results of at least one of the first and second imaging systems are imaging results of at least one marker formed on the surface of the container at a position facing the space in which the first group of target objects is located.

98. The control device described in claim 97, wherein the calculation device calculates at least one of the height of the first group of target objects contained in the container and the amount of the first group of target objects contained in the container based on the number of markers detected based on the imaging results of the markers.

99. A control device as described in any one of claims 94 to 98, wherein the measurement device includes a weight sensor capable of measuring the weight of the first group of target objects contained in the container.

100. A control device as described in any one of claims 94 to 99, wherein the measurement device includes at least one sensor capable of detecting at least one of the first group of targets within the container.

101. The control device according to claim 100, wherein the calculation device calculates at least one of the height of the first group of target objects contained in the container and the amount of the first group of target objects contained in the container based on the number of sensors that do not detect at least one of the first group of target objects.

102. The control device according to claim 100 or 101, wherein the at least one sensor is disposed on the container.

103. A control device according to any one of claims 88 to 102, wherein the arithmetic device calculates at least one of the height of the first group of target objects contained in the container and the amount of the first group of target objects contained in the container based on the number of target objects on which the processing device has performed the processing.

104. A control device as described in any one of claims 88 to 103, wherein the calculation device generates the first control signal for controlling the robot so that the imaging system moves to at least one of an imaging attitude, which is an attitude at which the imaging system images the target objects, and an imaging horizontal position, which is a position on a horizontal plane at which the imaging system images the target objects, based on at least one of a height of the first group of target objects contained in the container and a quantity of the first group of target objects contained in the container.

105. The control device described in claim 104, wherein the calculation device determines at least one of the imaging attitude and the imaging horizontal position based on at least one of the height of the first group of target objects contained in the container and the amount of the first group of target objects contained in the container, and generates the first control signal for controlling the robot so that the imaging system moves to the determined at least one of the imaging attitude and the imaging horizontal position.

106. A control device according to any one of claims 88 to 105, wherein the imaging system includes at least one of a monocular camera and a stereo camera having two monocular cameras different from the monocular camera.

107. A control system comprising: a control device according to any one of claims 88 to 106; and said imaging system.

108. A robot system comprising: a control device according to any one of claims 88 to 106; the imaging system; and the robot.

109. A control method for generating a control signal for controlling a robot provided with a holding device capable of holding a target object and an imaging system for imaging the target object, the control method comprising: generating, as the control signal, a first control signal for controlling the robot to move to an imaging height at which the imaging system images the target objects, based on at least one of a height of a first target object group including at least a portion of a plurality of the target objects contained in a container and an amount of the first target object group contained in the container; and generating, as the control signal, a second control signal for controlling the robot to move to an imaging height at which the imaging system images the target objects, based on an imaging result of the imaging system imaging a second target object group including at least a portion of the first target object group contained in the container at the imaging height, the processing device to process one of the target objects of the second target object group.

110. A computer program for causing a computer to execute the control method recited in claim 109.

Images