Control device, control system, robot system, control method, and computer program

Abstract

This control device controls a robot that moves a processing device, which performs processing on an object, and an imaging system. When it is determined, on the basis of first image data which is generated by imaging a target object by the imaging system when the positional relationship between the object and the imaging system is a first relationship, that there is no target object that can be processed by the processing device, the control device generates a first control signal for controlling the robot so that the positional relationship between the target object and the imaging system changes to a second positional relationship, on the basis of second image data which is generated by imaging the target object by the imaging system, and generates a second control signal for controlling the robot, on the basis of the target object-related position and posture data which is generated on the basis of third image data which is generated by imaging the target object by the imaging system when the positional relationship between the target object and the imaging system is the second positional relationship.

Description

Control device, control system, robot system, control method, and computer program

[0001] The present invention relates to the technical fields of a control device, a control system, a robot system, a control method, and a computer program that can generate a control signal for controlling a robot, for example.

[0002] An example of a control device that calculates at least one of the position and orientation of an object that is the target of processing by a robot and controls the robot based on the calculated at least one of the position and orientation is described in Patent Document 1. In such a control device, it is required to calculate at least one of the position and orientation of the object in order to control the robot to perform processing on the object.

[0003] US Patent Application Publication No. 2013 / 0230235

[0004] According to a first aspect, there is provided a control device that is provided with a processing device that performs processing on a target object and an imaging system, and that generates a control signal for controlling a robot that moves the processing device and the imaging system, the control device comprising: an arithmetic device that generates the control signal; and a communication device that outputs the control signal generated by the arithmetic device, wherein the arithmetic device, when it determines that there is no target object that can be processed by the processing device based on first image data generated by the imaging system imaging the target object when the positional relationship between the target object and the imaging system is a first positional relationship, generates a first control signal as the control signal based on second image data generated by the imaging system imaging the target object to control the robot so that the positional relationship changes to a second positional relationship different from the first positional relationship, and generates the second control signal as the control signal based on position data and orientation data indicating the position and orientation of the target object that are generated based on third image data generated by the imaging system imaging the target object when the positional relationship is the second positional relationship.

[0005] According to a second aspect, there is provided a control system including the control device provided by the first aspect and the imaging system.

[0006] According to a third aspect, there is provided a robot system including the control device provided by the first aspect, the imaging system, and the robot.

[0007] According to a fourth aspect, there is provided a control method for generating a control signal for controlling a robot that is provided with a processing device that performs processing on a target object and an imaging system and that moves the processing device and the imaging system, the control method including: when it is determined that there is no target object that can be processed by the processing device based on first image data generated by the imaging system imaging the target object when the positional relationship between the target object and the imaging system is a first positional relationship, generating a first control signal as the control signal based on second image data generated by the imaging system imaging the target object to control the robot so that the positional relationship changes to a second positional relationship different from the first positional relationship; and generating a second control signal as the control signal based on position data and orientation data indicating the position and orientation of the target object that are generated based on third image data generated by the imaging system imaging the target object when the positional relationship is the second positional relationship.

[0008] According to a fifth aspect, there is provided a computer program that causes a computer to execute the control method provided by the fourth aspect.

[0009] According to a sixth aspect, there is provided a control device that is provided with a processing device that performs processing on a target object and an imaging system, and that generates a control signal for controlling a robot that moves the processing device and the imaging system, the control device comprising: an arithmetic unit that generates the control signal; and a communication unit that outputs the control signal generated by the arithmetic unit, wherein the arithmetic unit determines, based on first image data generated by the imaging system imaging a first group of target objects including a plurality of the target objects when the imaging system is at a first position and a first attitude, that there are no target objects among the first group of target objects that can be processed by the processing device, and generates a first control signal as the control signal based on second image data generated by the imaging system imaging a second group of target objects including a plurality of the target objects, for controlling the robot so that the imaging system changes to a second position and a second attitude different from the first position and the first attitude.

[0010] According to a seventh aspect, there is provided a control system including the control device provided by the sixth aspect and the imaging system.

[0011] According to an eighth aspect, there is provided a robot system including the control device provided by the sixth aspect, the imaging system, and the robot.

[0012] According to a ninth aspect, there is provided a control method for generating a control signal for controlling a robot that is provided with a processing device that performs processing on a target object and an imaging system and that moves the processing device and the imaging system, the control method including: when it is determined, based on first image data generated by the imaging system imaging a first group of target objects including a plurality of the target objects when the imaging system is at a first position and a first attitude, that there are no target objects among the first group of target objects that can be processed by the processing device, generating a first control signal as the control signal based on second image data generated by the imaging system imaging a second group of target objects including the plurality of the target objects, for controlling the robot to change the imaging system to a second position and a second attitude different from the first position and the first attitude.

[0013] According to a tenth aspect, there is provided a computer program for causing a computer to execute the control method provided by the ninth aspect.

[0014] According to an eleventh aspect, there is provided a control device that is provided with a processing device that performs processing on target objects and an imaging system, and that generates a control signal for controlling a robot that moves the processing device and the imaging system, the control device comprising: an arithmetic unit that generates the control signal based on first three-dimensional position data indicating the three-dimensional position of each of multiple points in a first object group that includes multiple target objects, and second three-dimensional position data that has less three-dimensional position data than the first three-dimensional position data, generated from the imaging results of the first object group including multiple target objects by the imaging system; and a communication unit that outputs the control signal generated by the arithmetic unit, wherein the arithmetic unit generates multiple first position data and first orientation data indicating the position and orientation of each of the target objects in a second object group that consists of multiple target objects from the first object group based on the second three-dimensional position data, generates second position data and second orientation data indicating the position and orientation of one of the target objects in the second object group based on the generated multiple first position data and first orientation data and the first three-dimensional position data, and generates the control signal based on the generated second position data and second orientation data.

[0015] According to a twelfth aspect, there is provided a control system including the control device provided by the eleventh aspect and the imaging system.

[0016] According to a thirteenth aspect, there is provided a control system including the control device provided by the eleventh aspect and the imaging system.

[0017] According to a fourteenth aspect, there is provided a control method for generating a control signal for controlling a robot that is provided with a processing device that performs processing on target objects and an imaging system and that moves the processing device and the imaging system, the control method including generating the control signal based on first three-dimensional position data indicating the three-dimensional position of each of a plurality of points of a first object group that includes a plurality of the target objects, and second three-dimensional position data that has less three-dimensional position data than the first three-dimensional position data, the first three-dimensional position data being generated from the imaging results of the first object group that includes a plurality of the target objects, and the generating the control signal includes generating, based on the second three-dimensional position data, a plurality of first position data and first orientation data indicating the position and orientation of each of the target objects of a second object group that includes a plurality of the target objects of the first object group; generating, based on the generated plurality of first position data and first orientation data and the first three-dimensional position data, second position data and second orientation data indicating the position and orientation of one of the target objects of the second object group; and generating the control signal based on the generated second position data and second orientation data.

[0018] According to a fifteenth aspect, there is provided a computer program for causing a computer to execute the control method provided by the thirteenth aspect.

[0019] According to a sixteenth aspect, there is provided a control device that is provided with a processing device and an imaging system that perform processing on target objects contained in a container, and that generates a control signal for controlling a robot that moves the processing device and the imaging system, the control device comprising: an arithmetic unit that generates the control signal; and a communication unit that outputs the control signal generated by the arithmetic unit, wherein the arithmetic unit generates position and orientation data indicating at least one of the position and orientation of the container based on first image data generated by the imaging system imaging at least a portion of the container, generates three-dimensional position data indicating the three-dimensional positions of each of a plurality of points in the group of objects based on second image data generated by the imaging system imaging a group of objects including at least a portion of the plurality of target objects contained in the container, selects a portion of the three-dimensional position data as selection data based on the position and orientation data, and generates the control signal for controlling the robot to cause the processing device to approach one of the group of target objects in order to process the one target object with the processing device, based on the selection data.

[0020] According to a seventeenth aspect, there is provided a control system including the control device provided by the sixteenth aspect and the imaging system.

[0021] According to an eighteenth aspect, there is provided a robot system including the control device provided by the sixteenth aspect, the imaging system, and the robot.

[0022] According to a 19th aspect, there is provided a control method for generating a control signal for controlling a robot that is provided with a processing device and an imaging system that perform processing on a target object and that moves the processing device and the imaging system, the control method including: generating position and orientation data indicating at least one of the position and orientation of the container based on first image data generated by the imaging system imaging at least a portion of the container; generating three-dimensional position data indicating the three-dimensional positions of each of a plurality of points of the group of objects based on second image data generated by the imaging system imaging a group of objects including at least some of the plurality of target objects contained in the container; selecting a portion of the three-dimensional position data as selection data based on the position and orientation data; and generating the control signal for controlling the robot to approach one of the group of target objects so that the processing device can process the one target object in the group of target objects based on the selection data.

[0023] According to a twentieth aspect, there is provided a computer program that causes a computer to execute the control method provided by the nineteenth aspect.

[0024] According to a 21st aspect, there is provided a control device that is provided with a processing device that processes target objects and an imaging system, and that generates control signals for controlling a robot that moves the processing device and the imaging system, the control device comprising: an arithmetic unit that generates the control signals; and a communication unit that outputs the control signals generated by the arithmetic unit, wherein the arithmetic unit generates the control signal for controlling the robot so that at least one of the position and orientation of the imaging system relative to one of the target objects in the target object group is changed when it is determined, based on the imaging results by the imaging system of a target object group including a plurality of the target objects, that at least one of the target objects in the target object group cannot be processed by the processing device.

[0025] The functions and other advantages of the present invention will become apparent from the following detailed description of the preferred embodiments.

[0026] FIG. 1 is a block diagram showing the configuration of a robot system according to this embodiment. FIG. 2 is a side view showing the appearance of a robot according to this embodiment. FIG. 3 is a block diagram showing the configuration of a control device according to this embodiment. FIG. 4 is a flowchart showing the flow of robot control processing. FIG. 5 is a schematic diagram showing 3D matching processing. FIGS. 6(a) to 6(d) are each side views showing the positional relationship between a robot and a workpiece at a certain point in time during a holding process for holding a workpiece placed on a mounting device moving on a support surface. FIGS. 7(a) to 7(d) are each side views showing the positional relationship between a robot and a workpiece at a certain point in time during a release process for placing the workpiece on a mounting device moving on a support surface. 8(a) and 8(b) are side views showing the positional relationship between the robot and the workpiece at a certain time during a holding process for holding a workpiece placed on a mounting device that is stationary on a support surface, and FIGS. 8(c) to 8(e) are side views showing the positional relationship between the robot and the workpiece at a certain time during a release process for placing a workpiece on a mounting device that is stationary on a support surface. FIGS. 9(a) to 9(e) are side views showing the positional relationship between the robot and the workpiece at a certain time during a holding process for holding multiple workpieces placed on the mounting device one by one in order and a release process for placing the multiple workpieces one by one on the mounting device. FIGS. 10(a) to 10(e) are side views showing the positional relationship between the robot and the workpiece at a certain time during a holding process for holding multiple workpieces placed on the mounting device one by one in order and a release process for placing the multiple workpieces one by one on the mounting device. Fig. 11(a) and Fig. 11(b) each show an example of a model generation screen. Fig. 12 shows a plurality of partial models. Fig. 13(a) and Fig. 13(c) each show an example of a target object, and Fig. 13(b) and Fig. 13(d) each show an example of a template model. Fig. 14 is a block diagram showing the configuration of a control device in the second modified example. Fig. 15 is a flowchart showing the flow of robot control processing in the second modified example.FIGS. 16(a) and 16(b) each show the positional relationship between a target object and an imaging device. FIG. 17(a) shows an imaging device moving rotationally, and FIG. 17(b) shows an imaging device moving translationally. FIGS. 18(a) and 18(b) each show an imaging device moving. FIG. 19(a) shows an example of a target object, and FIGS. 19(b) and 19(c) each show an example of a point cloud included in point cloud data (three-dimensional position data) generated from image data generated by an imaging system before movement capturing the target object shown in FIG. 19(a). FIG. 19(d) shows an overall model of the target object shown in FIG. 19(a), and FIG. 19(e) shows a partial model of the target object shown in FIG. 19(a). FIGS. 20(a) and 20(b) each show an imaging system capturing images of multiple target objects. FIGS. 21(a) and 21(b) each show an imaging system capturing images of multiple target objects. FIG. 22 shows multiple reset positions. FIG. 23 shows multiple reset postures. FIGS. 24(a) and 24(b) show a method for setting multiple reset positions. FIG. 25 is a block diagram showing the configuration of a control device in a third modified example. FIG. 26 conceptually shows resizing processing. FIG. 27 is a flowchart showing the flow of robot control processing in the third modified example. FIGS. 28(a) and 28(b) each conceptually show attention data. FIG. 29 shows an imaging system that captures images of multiple target objects. FIG. 30 is a flowchart showing the flow of robot control processing in the third modified example. FIGS. 31(a) to 31(c) each conceptually show multiple template models. FIG. 32 is a flowchart showing the flow of robot control processing in the third modified example. FIG. 33 is a block diagram showing the configuration of a control device in the third modified example. FIG. 34 shows an image in which a held object is captured. FIGS. 35(a) to 35(c) each show an initial setting operation. Each of Figures 36(a) to 36(c) conceptually illustrates selected data. Figure 37 illustrates an area designation screen. Figure 38 is a plan view illustrating an example in which the container containing the target object to be processed by the end effector is changed from a first container to a second container, and the position and orientation of the first container are approximately the same as those of the second container.FIG. 39 is a plan view showing an example in which the position of the first container differs from the position of the second container and / or the orientation of the first container differs from the orientation of the second container when a container containing a target object on which the end effector is to perform a predetermined process is changed from a first container to a second container. FIG. 40 is a plan view showing an example in which a container containing a target object on which the end effector is to perform a predetermined process moves. FIG. 41 is a flowchart showing a robot control process including a process for changing a designated area set for the first container to a designated area for the second container. FIG. 42 shows an output device displaying the designated area before and after the change. FIG. 43 is a plan view showing an example in which the position of the second container differs from the position of the third container and / or the orientation of the second container differs from the orientation of the third container when a container containing a target object on which the end effector is to perform a predetermined process is changed from a second container to a third container. FIG. 44 is a plan view showing an example in which the dimensions of the first container are substantially the same as the dimensions of the second container when the container housing the target object on which the end effector is to perform a predetermined process is changed from the first container to the second container. FIG. 45 is a plan view showing an example in which the dimensions of the first container are different from the dimensions of the second container when the container housing the target object on which the end effector is to perform a predetermined process is changed from the first container to the second container. FIG. 46 is a plan view showing an example in which the dimensions of the first container are different from the dimensions of the second container when the container housing the target object on which the end effector is to perform a predetermined process is changed from the second container to a third container. FIG. 47 is a plan view showing an example of a designated area. FIG. 48 is a table showing an example of specifications of an imaging system. FIG. 49 is a block diagram showing the configuration of a robot system including a measurement system.

[0027] Next, embodiments of a control device, a control system, a robot system, a control method, and a computer program will be described. Hereinafter, embodiments of a control device, a control system, a robot system, a control method, and a computer program will be described using a robot system SYS.

[0028] (1) Configuration of the Robot System SYS First, the configuration of the robot system SYS will be described.

[0029] (1-1) Overall Configuration of Robot System SYS First, the overall configuration of the robot system SYS will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the overall configuration of the robot system SYS.

[0030] 1, the robot system SYS includes a robot 1, an imaging system 2, a control device 3, and an end effector 4. The imaging system 2 may also be referred to as an imaging unit.

[0031] The robot 1 is a device capable of performing a predetermined process on a target object OBJ. An example of the robot 1 is shown in FIG. 2. FIG. 2 is a side view showing the appearance of the robot 1. As shown in FIG. 2, the robot 1 includes, for example, a base 11, a robot arm 12, and a robot control device 13.

[0032] The base 11 is a component that forms the base of the robot 1. The base 11 is placed on a support surface S, such as a floor surface. The base 11 may be fixed to the support surface S. Alternatively, the base 11 may be movable relative to the support surface S. As an example, the base 11 may be self-propelled on the support surface S. In this case, the base 11 may be installed on an automatic guided vehicle (AGV). When the base 11 is installed on an automatic guided vehicle, the automatic guided vehicle may be considered to be part of the robot 1. Alternatively, the automatic guided vehicle (AGV) may be used as the base 11. Note that FIG. 2 shows an example in which the base 11 is fixed to the support surface S.

[0033] The robot arm 12 is attached to the base 11. The robot arm 12 is a device in which a plurality of links 121 are connected via joints 122. An actuator is built into the joint 122. The link 121 may be rotatable around an axis defined by the joint 122 by the actuator built into the joint 122. At least one link 121 may be extendable and contractible along the direction in which the link 121 extends. A device including the device in which a plurality of links 121 are connected via joints 122 and the base 11 may be referred to as the robot arm 12.

[0034] An end effector 4 is attached to the robot arm 12. That is, the end effector 4 is attached to the robot 1. In the example shown in FIG. 2 , the end effector 4 is attached to the tip of the robot arm 12. The end effector 4 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the end effector 4. That is, the robot 1 moves the end effector 4.

[0035] The end effector 4 is a device that performs a predetermined process (in other words, a predetermined operation) on the target object OBJ. The end effector 4 that performs a predetermined process on the target object OBJ may also be called a processing device.

[0036] For example, the end effector 4 may perform a holding process for holding a target object OBJ as an example of the predetermined process. In this case, the end effector 4 may be considered to be performing the holding process on the target object OBJ that is to be held by the end effector 4. An end effector 4 capable of performing the holding process may be referred to as a holding device.

[0037] For example, holding the target object OBJ may include grasping the target object OBJ. For example, holding the target object OBJ may include gripping the target object OBJ using a hand gripper (described later), which is an example of the end effector 4. Holding the target object OBJ may include attracting the target object OBJ. For example, holding the target object OBJ may include attracting (vacuum suction) the target object OBJ using a vacuum gripper (described later), which is an example of the end effector 4. For example, holding the target object OBJ may include attracting the target object OBJ using a magnetic attraction gripper (described later), which is an example of the end effector 4.

[0038] For example, as an example of the predetermined process, the end effector 4 may perform a release process (in other words, a release operation) to release (i.e., let go of) the target object OBJ that it is holding. In this case, the end effector 4 may be considered to be performing the release process on the target object OBJ that it is holding. An end effector 4 that is capable of performing the release process may be referred to as a release device.

[0039] A hand gripper is an example of an end effector 4 capable of holding and releasing an object OBJ. A hand gripper is an end effector 4 that can hold a target object OBJ by physically pinching the target object OBJ using multiple (e.g., two, three, or four) finger or claw members. Another example of an end effector 4 capable of holding and releasing an object OBJ is a vacuum gripper. A vacuum gripper is an end effector 4 that can hold a target object OBJ by vacuum-suctioning the target object OBJ. Another example of an end effector 4 capable of holding and releasing an object OBJ is a magnetically attractive gripper. FIG. 2 illustrates an example in which the end effector 4 is a hand gripper. However, the end effector 4 capable of holding and releasing an object OBJ may be any other existing end effector.

[0040] The robot 1 may perform, as an example of a predetermined process, a placement process (in other words, a placement operation) for placing a target object OBJ at a desired position using the end effector 4 capable of performing a holding process and a release process. For example, the robot 1 may hold a first target object OBJ using the end effector 4, and then perform a placement process for placing the first target object OBJ held by the end effector 4 at a desired position of a second target object OBJ different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process for the second target object OBJ on which the end effector 4 is to place the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process for the first target object OBJ to be released by the end effector 4.

[0041] The robot 1 may use the end effector 4 capable of performing a holding process and a release process to perform a fitting process (in other words, a fitting operation) for fitting a first target object OBJ into a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation). For example, the robot 1 may hold the first target object OBJ using the end effector 4, and then perform a fitting process for fitting the first target object OBJ held by the end effector 4 into a second target object OBJ different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process on the second target object OBJ into which the end effector 4 should fit the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process on the first target object OBJ to be released by the end effector 4. The fitting process may also be referred to as a processing process.

[0042] As an example, the fitting process may include a process for fitting (i.e., inserting) the first target object OBJ into a hole formed in the second target object OBJ. For example, the robot 1 may hold the first target object OBJ using the end effector 4, and then perform a fitting process for fitting (i.e., inserting) the first target object OBJ held by the end effector 4 into a hole formed in the second target object OBJ. In this case, the end effector 4 may be considered to be performing a release process for the second target object OBJ, which has a hole formed therein into which the first target object OBJ should be fitted (i.e., inserted). Similarly, the end effector 4 may be considered to be performing a release process for the first target object OBJ that the end effector 4 should release.

[0043] The robot 1 may use the end effector 4 capable of performing a holding process and a release process to perform a pasting process (in other words, a pasting operation) for pasting a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation). For example, the robot 1 may hold the first target object OBJ using the end effector 4, and then perform a pasting process for pasting the first target object OBJ held by the end effector 4 to a second target object OBJ different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process on the second target object OBJ to which the end effector 4 is to paste the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process on the first target object OBJ to be released by the end effector 4. The pasting process may also be referred to as a processing process.

[0044] The robot 1 may use the end effector 4 capable of performing a holding process and a release process to perform a bonding process (i.e., a bonding operation) for bonding a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (i.e., a placement operation). For example, the robot 1 may hold the first target object OBJ using the end effector 4, and then perform a bonding process for bonding the first target object OBJ held by the end effector 4 to a second target object OBJ different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process on the second target object OBJ to which the end effector 4 is to bond the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process on the first target object OBJ to which the end effector 4 is to release. The bonding process may also be referred to as a processing process.

[0045] The robot 1 may use the end effector 4 capable of holding and releasing operations to perform a welding process (i.e., a welding operation) for welding a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (i.e., a placement operation). For example, the robot 1 may hold the first target object OBJ using the end effector 4, and then perform a welding process for welding the first target object OBJ held by the end effector 4 to a second target object OBJ different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process on the second target object OBJ to which the end effector 4 is to weld the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process on the first target object OBJ to which the end effector 4 is to release. The welding process may also be referred to as a processing process.

[0046] The robot 1 may use the end effector 4 capable of holding and releasing operations to perform a screw tightening process (i.e., a screw tightening operation) for tightening a first target object OBJ capable of functioning as a screw into a screw hole formed in a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (i.e., a placement operation). For example, the robot 1 may hold the first target object OBJ using the end effector 4, and then perform a screw tightening process for tightening the first target object OBJ held by the end effector 4 into a second target object OBJ different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process for the second target object OBJ into which the end effector 4 is to tighten the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process for the first target object OBJ to be released by the end effector 4. The screw tightening process may also be referred to as a processing process.

[0047] For example, the end effector 4 may perform a predetermined process on each of multiple target objects OBJ. That is, the end effector 4 may perform the predetermined process on the multiple target objects OBJ in sequence. In this case, the robot 1 may move the end effector 4 to a first position where the end effector 4 can perform the predetermined process on a first target object OBJ, and after the end effector 4 has moved to the first position, the end effector 4 may perform the predetermined process on the first target object OBJ. Thereafter, the robot 1 may move the end effector 4 to a second position where the end effector 4 can perform the predetermined process on a second target object OBJ different from the first target object OBJ, and after the end effector 4 has moved to the second position, the end effector 4 may perform the predetermined process on the second target object OBJ.

[0048] For example, the end effector 4 may perform a predetermined process on each of multiple portions of a single target object OBJ. That is, the end effector 4 may sequentially perform a predetermined process on multiple portions of a single target object OBJ. In this case, the robot 1 may move the end effector 4 to a third position where the end effector 4 can perform a predetermined process on a first portion of the target object OBJ, and after the end effector 4 has moved to the third position, the end effector 4 may perform the predetermined process on the first portion of the target object OBJ. Thereafter, the robot 1 may move the end effector 4 to a fourth position where the end effector 4 can perform a predetermined process on a second portion of the target object OBJ, and after the end effector 4 has moved to the fourth position, the end effector 4 may perform the predetermined process on the second portion of the target object OBJ.

[0049] As an example of an operation in which the robot 1 performs a predetermined process on each of multiple portions of a single target object OBJ, the robot 1 may repeatedly perform a fitting process on a second target object OBJ having multiple holes formed therein, fitting the first target object OBJ into each hole. Note that a portion of the target object OBJ on which the end effector 4 performs a predetermined process may be referred to as a target portion or a target member. For example, each portion (each member) of the second target object OBJ in which each hole is formed may be referred to as a target portion or a target member. Note that when the target object OBJ is composed of multiple parts, each part may be referred to as a target portion or a target member. When the target object OBJ is composed of multiple members, each member may be considered as a target portion or a target member.

[0050] As shown in FIG. 2 , the target object OBJ on which the end effector 4 performs a predetermined process may include a workpiece W. The workpiece W may include, for example, a part or member used to manufacture a desired product. The workpiece W may include, for example, a part or member to be processed to manufacture the desired product. The workpiece W may include, for example, a part or member to be transported to manufacture the desired product. The workpiece W may include, for example, a part or member that is moving due to transportation to manufacture the desired product. The workpiece W may include, for example, a part or member that is moving to manufacture the desired product.

[0051] The target object OBJ on which the end effector 4 performs a predetermined process may include a placement device T on which a workpiece W is placed, as shown in FIG. 2 . An example of the placement device T is a container (storage box) CB. The container CB may be a placement device T having a bottom wall BS and a side wall SS protruding upward from the bottom wall BS, in a storage space SP enclosed by the bottom wall BS and the side wall SS, in which the workpiece W is placed on the bottom wall BS. The container CB may be a placement device T having a bottom wall BS and a side wall SS protruding upward from the bottom wall BS, in which the workpiece W can be stored in the storage space SP enclosed by the bottom wall BS and the side wall SS. Note that the storage space SP may include at least a portion of the space above an opening AP of the container CB that connects the space enclosed by the bottom wall BS and the side wall SS to a space outside the container CB, in addition to or instead of at least a portion of the space enclosed by the bottom wall BS and the side wall SS. However, the container CB does not have to have a side wall SS. A container CB that does not have a side wall SS may be called a pallet. The placement device T may be a pallet. Furthermore, the placement device T is not limited to a container CB or a pallet, but may be an existing object on which the workpiece W can be placed. The placement device T may be called a placement member. The placement device T may be disposed on a support surface S. The placement device T may be fixed to the support surface S. Alternatively, at least a portion of the placement device T may be movable relative to the support surface S.

[0052] A first example of a case in which at least a portion of the placement device T is movable relative to the support surface S is an example in which the placement device T is supported by a transport device that can move (in other words, transport) the placement device T. A first example of the transport device is a transport device that is self-propelled on the support surface S. In this case, the placement device T may be placed on the self-propelled transport device or may be held by the self-propelled transport device. Note that the self-propelled transport device may also be referred to as an automatic guided vehicle (AGV). A second example of the transport device is a belt conveyor. In this case, the placement device T may be placed on a transport belt that is part of the belt conveyor and is movable relative to the support surface S. A third example of the transport device is a flying device that can fly above the support surface S. In this case, the placement device T may be placed on the flying transport device or may be held by the flying transport device. The flying transport device may be referred to as an unmanned aerial vehicle or a drone. A fourth example of the transport device is another robot arm that is different from the robot arm 12 and that is equipped with an end effector that can hold the mounting device T. In this case, the mounting device T may be held by the end effector attached to the other robot arm.

[0053] A second example of a placement device T in which at least a portion thereof is movable relative to the support surface S is where the placement device T itself is movable relative to the support surface S. A first example of a movable placement device T is a placement device T that is self-propelled on the support surface S. The self-propelled placement device T may be referred to as an automatic guided vehicle (AGV). A second example of a movable placement device T is a belt conveyor. In this case, a transport belt that is part of the belt conveyor and on which the workpiece W is placed may move relative to the support surface S. A third example of a movable placement device T is a placement device T that can fly above the support surface S. In this case, the placement device T may be movable relative to the support surface S by flying above the support surface S. The flyable placement device T may be referred to as an unmanned aerial vehicle or a drone. A fourth example of the movable mounting device T is another robot arm that is different from the robot arm 12 and is equipped with an end effector that can hold the workpiece W. In this case, the end effector attached to the other robot arm may move relative to the support surface S.

[0054] When the transport device moves the placement device T, the workpiece W placed on the placement device T also moves relative to the support surface S. For this reason, a transport device that can move the placement device T may be considered equivalent to a transport device that can transport the workpiece W. Similarly, when the placement device T moves, the workpiece W placed on the placement device T also moves relative to the support surface S. For this reason, the movable placement device T may be considered equivalent to a moving device that can move the workpiece W (in other words, a transport device that can transport the workpiece W). Note that FIG. 2 shows an example in which the placement device T is self-propelled on the support surface S.

[0055] In addition, when at least a part of the placement device T is movable relative to the support surface S, the first placement device T on which the workpiece W is placed may be placed on a second placement device T that is movable relative to the support surface S. In this case, the device including the first placement device T and the second placement device T may be referred to as the placement device T.

[0056] However, the target object OBJ does not have to include the placement device T. Furthermore, the workpiece W does not have to be placed on the placement device T, and the placement device T may not exist. For example, the workpiece W may be placed on a support surface S.

[0057] When the target object OBJ includes a workpiece W and a mounting device T, the holding process described above may include a process of holding a workpiece W placed on a stationary or moving mounting device T. The holding process described above may include a process of holding a workpiece W placed on a support surface S. The release process described above may include a process of releasing a workpiece W held by the end effector 4 in order to place the workpiece W held by the end effector 4 at a desired position on a stationary or moving mounting device T. The release process described above may include a process of releasing a workpiece W held by the end effector 4 in order to place the workpiece W held by the end effector 4 at a desired position on the support surface S. The release process described above may include a process of releasing a first workpiece W held by the end effector 4 in order to fit the first workpiece W held by the end effector 4 into a second workpiece W placed on a stationary or moving mounting device T. The above-described release process may include a process of releasing a first workpiece W held by the end effector 4 in order to fit the first workpiece W held by the end effector 4 into a second workpiece W placed on the support surface S. The above-described release process may include a process of releasing the first workpiece W held by the end effector 4 in order to fit (i.e., insert) the first workpiece W held by the end effector 4 into a hole formed in a second workpiece W placed on a stationary or moving mounting device T. The above-described release process may include a process of releasing the first workpiece W held by the end effector 4 in order to fit (i.e., insert) the first workpiece W held by the end effector 4 into a hole formed in the second workpiece W placed on the support surface S. Note that the target object OBJ may include the workpiece W or the mounting device T.

[0058] The robot control device 13 controls the operation of the robot 1 .

[0059] Specifically, the robot control device 13 may control the movement of the robot arm 12. For example, the robot control device 13 may control the movement of the robot arm 12 so that a desired link 121 rotates around an axis defined by a desired joint 122. For example, the robot control device 13 may control the movement of the robot arm 12 so that the end effector 4 attached to the robot arm 12 is positioned at a desired position. For example, the robot control device 13 may control the movement of the robot arm 12 so that the end effector 4 attached to the robot arm 12 moves to a desired position.

[0060] In addition to controlling the operation of the robot 1, the robot control device 13 may also control the operation of the end effector 4 attached to the robot 1. For example, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 holds the target object OBJ at a desired timing. That is, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 performs a holding process at a desired timing. For example, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 releases the held target object OBJ at a desired timing. That is, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 performs a release process at a desired timing. If the end effector 4 is a hand gripper, the robot control device 13 may control the timing of opening and closing the hand gripper. If the end effector 4 is a vacuum gripper, the robot control device 13 may control the timing of turning on and off a vacuum device of the vacuum gripper. If the end effector 4 is a magnetic attraction gripper, the robot control device 13 may control the timing of turning on and off the magnetic attraction device of the magnetic attraction gripper.

[0061] Note that FIG. 2 shows an example in which the robot 1 is a robot arm 12 (i.e., a vertical articulated robot). However, the robot 1 may be a robot different from a vertical articulated robot. For example, the robot 1 may be a SCARA robot (i.e., a horizontal articulated robot). For example, the robot 1 may be a parallel link robot. For example, the robot 1 may be a dual-arm robot having two robot arms 12. For example, the robot 1 may be a Cartesian coordinate robot. For example, the robot 1 may be a cylindrical coordinate robot. The robot 1 may be referred to as a movable device. The movable device may include, in addition to the robot 1, at least one of an automatic guided vehicle and an unmanned aerial vehicle. For example, the robot 1 may be installed in at least one of an automatic guided vehicle and an unmanned aerial vehicle.

[0062] 1 again, the imaging system 2 captures an image of a target object OBJ. To capture an image of the target object OBJ, the imaging system 2 includes an imaging device 21, an imaging device 22, and a projection device 23. The imaging system 2 may also be referred to as an imaging unit.

[0063] The imaging device 21 is a camera capable of capturing an image of the target object OBJ. In this embodiment, the imaging device 21 is a monocular camera. The imaging device 21 captures an image of the target object OBJ to generate image data IMG_2D. That is, the imaging device 21 generates image data IMG_2D, which is the result of capturing the image of the target object OBJ. The image data IMG_2D generated by the imaging device 21 is output from the imaging device 21 to the control device 3. As a result, the control device 3 acquires the image data IMG_2D acquired by the imaging device 21 capturing the image of the target object OBJ. In this embodiment, the imaging device 21 is a monocular camera. Specifically, the imaging device 21 is capable of capturing an image of the target object OBJ using a monocular camera (in other words, an image sensor). Specifically, because the imaging device 21 is a monocular camera, the imaging device 21 generates one piece of image data generated by the monocular camera as the image data IMG_2D. In this case, the image data IMG_2D corresponding to one image data (i.e., showing one image) may be referred to as monocular image data. The image capturing device 21 is not limited to a monocular camera. The image capturing device 21 may be a stereo camera capable of capturing an image of the target object OBJ using two monocular cameras, or may include three or more monocular cameras. The image capturing device 21 may be at least one of a light field camera, a plenoptic camera, and a multispectral camera.

[0064] Like the imaging device 21, the imaging device 22 is a camera capable of capturing an image of the target object OBJ. In this embodiment, the imaging device 22 is a stereo camera. Specifically, the imaging device 22 is a stereo camera capable of capturing an image of the target object OBJ using two monocular cameras (in other words, two image sensors). The imaging device 22 captures an image of the target object OBJ to generate image data IMG_3D. That is, the imaging device 22 generates image data IMG_3D that is the result of capturing the image of the target object OBJ. Specifically, because the imaging device 22 is a stereo camera, the imaging device 22 generates image data IMG_3D that includes two pieces of image data generated by the two monocular cameras. In this case, the image data IMG_3D that includes the two pieces of image data (i.e., shows two images) may be referred to as stereo image data. The image data IMG_3D generated by the imaging device 22 is output from the imaging device 22 to the control device 3. As a result, the control device 3 acquires image data IMG_3D acquired by capturing an image of the target object OBJ using the imaging device 22. Note that the imaging device 22 is not limited to a stereo camera. The imaging device 22 may be a monocular camera or may include three or more monocular cameras. Note that the imaging device 22 may be at least one of a light field camera, a plenoptic camera, and a multispectral camera.

[0065] The projection device 23 is a device capable of irradiating projection light onto the target object OBJ. In particular, the projection device 23 is a device capable of projecting a desired projection pattern onto the target object OBJ by irradiating the target object OBJ with projection light. The projection pattern may also be referred to as the intensity distribution of light projected onto the target object OBJ. If the intensity distribution of the projected light changes, the projection pattern also changes. The desired projection pattern may include, for example, a random pattern. The random pattern may be a projection pattern having a different pattern for each unit irradiation area. The random pattern may include a random dot pattern. The desired projection pattern may include, for example, a one-dimensional or two-dimensional grid pattern. The desired projection pattern may include, for example, a line pattern. The desired projection pattern may include, for example, a stripe pattern. The desired projection pattern may include other projection patterns. The imaging device 22 captures an image of the target object OBJ onto which the projection pattern from the projection device 23 is projected. In this case, the image represented by the image data IMG_3D includes a target object OBJ onto which the projection pattern is projected. On the other hand, the imaging device 21 does not need to capture an image of the target object OBJ onto which the projection pattern is projected. The imaging device 21 may capture an image of a target object OBJ onto which no projection pattern is projected. In this case, the image represented by the image data IMG_2D does not need to include a target object OBJ onto which the projection pattern is projected. The image represented by the image data IMG_2D may include a target object OBJ onto which no projection pattern is projected. The projection light for projecting a desired projection pattern onto the target object OBJ may be referred to as pattern light or structured light. In this case, the projection light may include pattern light or structured light. The projection light may be light of a uniform projection pattern (i.e., a uniform intensity distribution).

[0066] Note that the projection device 23 may be considered to illuminate the target object OBJ with the projection light by irradiating the target object OBJ with the projection light. In this case, the projection device 23 may function as an illumination device that illuminates the target object OBJ. When the projection device 23 functions as an illumination device, the projection light may be referred to as illumination light. The projection light (illumination light) used as an illumination device may be light with a uniform intensity distribution. When the projection device 23 functions as an illumination device, the projection light does not need to be light that can project a desired projection pattern onto the target object OBJ. When the projection device 23 functions as an illumination device, the projection light (i.e., illumination light) may be any light that can illuminate the target object OBJ.

[0067] The imaging device 21 may image the entire target object OBJ. Alternatively, the imaging device 21 may image a portion of the target object OBJ. That is, the imaging device 21 may image a portion of the target object OBJ while not imaging another portion of the target object OBJ. Similarly, the imaging device 22 may image the entire target object OBJ. Alternatively, the imaging device 22 may image a portion of the target object OBJ. That is, the imaging device 22 may image a portion of the target object OBJ while not imaging another portion of the target object OBJ.

[0068] The imaging device 21 may capture an image of a single target object OBJ. In other words, a single target object OBJ may appear in the image represented by the image data IMG_2D. Alternatively, the imaging device 21 may capture an image of multiple target objects OBJ. In other words, a plurality of target objects OBJ may appear in the image represented by the image data IMG_2D. In this case, as will be described in detail later, the control device 3 may determine (in other words, select) one of the multiple target objects OBJ captured by the imaging device 21 as the target object OBJ on which the end effector 4 will actually perform a predetermined process. Note that one of the multiple target objects OBJ captured by the imaging device 21 on which the end effector 4 will actually perform a predetermined process may be referred to as a process execution object.

[0069] The imaging device 22 may capture an image of a single target object OBJ. In other words, a single target object OBJ may appear in the image represented by the image data IMG_3D. Alternatively, the imaging device 22 may capture an image of multiple target objects OBJ. In other words, a plurality of target objects OBJ may appear in the image represented by the image data IMG_3D. In this case, as will be described in detail later, the control device 3 may determine (in other words, select) one of the multiple target objects OBJ captured by the imaging device 22 as the target object OBJ on which the end effector 4 will actually perform a predetermined process. Note that one of the multiple target objects OBJ captured by the imaging device 21 on which the end effector 4 will actually perform a predetermined process may be referred to as a process execution object.

[0070] When each of the imaging devices 21 and 22 captures an image of a plurality of target objects OBJ, the plurality of target objects OBJ captured by the imaging device 21 may be the same as the plurality of target objects OBJ captured by the imaging device 22. That is, the plurality of target objects OBJ captured in the image represented by the image data IMG_2D may be the same as the plurality of target objects OBJ captured in the image represented by the image data IMG_3D. Alternatively, at least one of the plurality of target objects OBJ captured by the imaging device 21 may be different from at least one of the plurality of target objects OBJ captured by the imaging device 22. That is, at least one of the plurality of target objects OBJ captured in the image represented by the image data IMG_2D may be the same as at least one of the plurality of target objects OBJ captured in the image represented by the image data IMG_3D.

[0071] The multiple target objects OBJ captured by at least one of the imaging devices 21 and 22 may be arranged such that at least two of the multiple target objects OBJ at least partially overlap. As an example, if the target objects OBJ are workpieces W corresponding to parts used to manufacture a desired product, the multiple workpieces W (i.e., multiple parts) may be arranged such that at least two of the multiple workpieces W at least partially overlap. In this case, the robot 1 may perform bulk picking, in which the robot 1 picks workpieces W one by one from the multiple workpieces W that are arranged in a disorderly manner.

[0072] The imaging system 2 is attached to the robot arm 12, similar to the end effector 4. That is, the imaging devices 21 and 22 and the projection device 23 are attached to the robot arm 12. For example, as shown in FIG. 2 , the imaging devices 21 and 22 and the projection device 23 may be attached to the tip of the robot arm 12, similar to the end effector 4. In this case, the imaging devices 21 and 22 and the projection device 23 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the imaging devices 21 and 22 and the projection device 23.

[0073] However, the imaging system 2 does not have to be attached to the robot arm 12. The imaging system 2 may be provided at any position where it can irradiate the target object OBJ with projection light and capture an image of the target object OBJ. In this case, for example, the imaging system 2 may be attached to a structure such as a pillar so that it can irradiate the target object OBJ with projection light and capture an image of the target object OBJ. Note that at least one of the imaging devices 21, 22, and projection device 23 may be attached to the robot arm 12, and at least one of the imaging devices 21, 22, and projection device 23 may be attached to a location different from the robot arm 12 (for example, a structure such as a pillar). When at least one of the imaging devices 21 and 22 is attached to a location different from the robot arm 12, at least one of the imaging devices 21 and 22 may be attached to a structure such as a pillar arranged so that it can capture an image of the target object OBJ. Furthermore, when the projection device 23 is attached to a location other than the robot arm 12, the projection device 23 may be attached to a structure such as a pillar arranged so that it can irradiate the projection light onto the target object OBJ.

[0074] The imaging device 21 may capture an image of the target object OBJ while the imaging device 21 and the target object OBJ are displaced relative to each other. Note that the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may refer to a state in which the relative positional relationship between the imaging device 21 and the target object OBJ is changing. The state in which the imaging device 21 and the target object OBJ are displaced relative to each other may refer to a state in which the imaging device 21 and the target object OBJ are moving relative to each other. For example, the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the target object OBJ is moving relative to the imaging device 21. For example, the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the imaging device 21 is moving relative to the target object OBJ. In this case, the imaging device 21 does not need to be stationary to capture an image of the target object OBJ, and the robot system SYS can efficiently perform a predetermined process on the target object OBJ using the end effector 4.

[0075] Alternatively, the imaging device 21 may capture an image of the target object OBJ during a period in which there is no relative displacement between the imaging device 21 and the target object OBJ. Note that the state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the relative positional relationship between the imaging device 21 and the target object OBJ is not changing. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which there is no relative movement between the imaging device 21 and the target object OBJ. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the imaging device 21 and the target object OBJ are stationary. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the imaging device 21 and the target object OBJ are moving at the same moving speed in the same moving direction.

[0076] The imaging device 22 may capture an image of the target object OBJ while the imaging device 22 and the target object OBJ are displaced relative to each other. Note that the state in which the imaging device 22 and the target object OBJ are displaced relative to each other may refer to a state in which the relative positional relationship between the imaging device 22 and the target object OBJ is changing. The state in which the imaging device 22 and the target object OBJ are displaced relative to each other may refer to a state in which the imaging device 22 and the target object OBJ are moving relative to each other. For example, the state in which the imaging device 22 and the target object OBJ are displaced relative to each other may include a state in which the target object OBJ is moving relative to the imaging device 22. For example, the state in which the imaging device 22 and the target object OBJ are displaced relative to each other may include a state in which the imaging device 22 is moving relative to the target object OBJ. In this case, the imaging device 22 does not need to be stationary to capture an image of the target object OBJ, and the robot system SYS can efficiently perform a predetermined process on the target object OBJ using the end effector 4.

[0077] Alternatively, the imaging device 22 may capture an image of the target object OBJ during a period in which there is no relative displacement between the imaging device 22 and the target object OBJ. Note that the state in which there is no relative displacement between the imaging device 22 and the target object OBJ may refer to a state in which the relative positional relationship between the imaging device 22 and the target object OBJ is not changing. The state in which there is no relative displacement between the imaging device 22 and the target object OBJ may refer to a state in which there is no relative movement between the imaging device 22 and the target object OBJ. The state in which there is no relative displacement between the imaging device 22 and the target object OBJ may refer to a state in which the imaging device 22 and the target object OBJ are stationary. The state in which there is no relative displacement between the imaging device 22 and the target object OBJ may refer to a state in which the imaging device 22 and the target object OBJ are moving at the same moving speed in the same moving direction.

[0078] The imaging devices 21 and 22 may capture images of the target object OBJ in synchronization with each other. For example, the imaging devices 21 and 22 may capture images of the target object OBJ simultaneously. That is, the imaging devices 21 and 22 may capture images of the target object OBJ such that the 2D imaging time at which the imaging device 21 captures the target object OBJ and the 3D imaging time at which the imaging device 22 captures the target object OBJ are the same. The imaging devices 21 and 22 may capture images of the target object OBJ such that the 2D imaging time at which the imaging device 21 captures the target object OBJ to generate image data IMG_2D and the 3D imaging time at which the imaging device 22 captures the target object OBJ to generate image data IMG_3D are the same.

[0079] The imaging devices 21 and 22 may capture images of the target object OBJ under the control of the control device 3. In this case, the time (in other words, the timing) at which each of the imaging devices 21 and 22 captures an image of the target object OBJ may be controlled by the control device 3. For example, the control device 3 may control the imaging devices 21 and 22 so that the imaging devices 21 and 22 capture images of the target object OBJ in synchronization with each other. For example, the control device 3 may control the imaging devices 21 and 22 so that the imaging devices 21 and 22 capture images of the target object OBJ simultaneously. In other words, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same time.

[0080] Here, the state in which "the 2D imaging time and the 3D imaging time are the same" may include a state in which "the 2D imaging time and the 3D imaging time are literally exactly the same." The state in which "the 2D imaging time and the 3D imaging time are the same" may include a state in which "the 2D imaging time and the 3D imaging time are not exactly the same, but the time difference between the 2D imaging time and the 3D imaging time is smaller than a first allowable upper limit, so the 2D imaging time and the 3D imaging time can be considered to be substantially the same." Here, the first allowable upper limit may be a first allowable upper limit based on a control error of the robot arm 12. For example, a time difference between the 2D imaging time and the 3D imaging time may cause an error in the calculation result of at least one of the position and orientation of the target object OBJ (described later) (i.e., a case in which the accuracy of at least one of the calculated position and orientation of the target object OBJ is reduced). In this case, an error in the calculation result of at least one of the position and orientation of the target object OBJ may cause a control error of the robot arm 12. A control error of the robot arm 12 may result in a movement error of the end effector 4, which may prevent the end effector 4 from properly performing a predetermined process on the target object OBJ. The first allowable upper limit may be set to an appropriate value that can avoid a situation in which the end effector 4 is unable to properly perform a predetermined process on the target object OBJ due to such a control error of the robot arm 12. The first allowable upper limit may be considered equivalent to the allowable upper limit of the movement error of the end effector 4 by the robot arm 12. Furthermore, even if there is a time difference between the 2D imaging time and the 3D imaging time due to a synchronization error in the imaging processes of the imaging devices 21 and 22, the 2D imaging time and the 3D imaging time may be considered to be substantially the same time. The synchronization error in the imaging processes of the imaging devices 21 and 22 may be a synchronization control error of the imaging processes of the imaging devices 21 and 22 by the control device 3.

[0081] However, the imaging devices 21 and 22 do not have to image the target object OBJ at the same time. In other words, the imaging devices 21 and 22 may image the target object OBJ such that the 2D imaging time at which the imaging device 21 images the target object OBJ and the 3D imaging time at which the imaging device 22 images the target object OBJ are different times. Note that the state in which "the 2D imaging time and the 3D imaging time are different times" may include a state in which "the 2D imaging time and the 3D imaging time cannot be considered to be substantially the same time because the time difference between the 2D imaging time and the 3D imaging time is larger than a first allowable upper limit value."

[0082] In the present embodiment, when the imaging devices 21 and 22 capture images of the target object OBJ during a period in which the imaging devices 21 and 22 are displaced relative to the target object OBJ, the imaging devices 21 and 22 may capture images of the target object OBJ such that the 2D imaging time and the 3D imaging time are the same. In other words, when the imaging devices 21 and 22 capture images of the target object OBJ during a period in which the imaging devices 21 and 22 are displaced relative to the target object OBJ, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same.

[0083] On the other hand, when the imaging devices 21 and 22 capture images of the target object OBJ during a period in which there is no relative displacement between each of the imaging devices 21 and 22 and the target object OBJ, the imaging devices 21 and 22 do not need to capture images of the target object OBJ so that the 2D imaging time and the 3D imaging time are the same. That is, the control device 3 does not need to control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same. For example, the imaging devices 21 and 22 may capture images of the target object OBJ so that the 2D imaging time and the 3D imaging time are different times. That is, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are different times. However, when the imaging devices 21 and 22 capture images of the target object OBJ during a period when there is no relative displacement between the imaging devices 21 and 22 and the target object OBJ, the imaging devices 21 and 22 may capture images of the target object OBJ so that the 2D imaging time and the 3D imaging time are the same. In other words, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same.

[0084] The control device 3 performs robot control processing. The robot control processing includes processing for generating a robot control signal for controlling the robot 1. Specifically, the control device 3 generates the robot control signal based on at least one of the image data IMG_2D and IMG_3D output from the imaging system 2. In this embodiment, the control device 3 calculates at least one of the position and orientation of the target object OBJ in the global coordinate system of the robot system SYS based on at least one of the image data IMG_2D and IMG_3D, and generates the robot control signal based on the calculated at least one of the position and orientation of the target object OBJ.

[0085] The global coordinate system is a coordinate system that serves as the reference for the robot system SYS. For example, the global coordinate system may be a coordinate system that serves as the reference for the robot 1. The global coordinate system can also be said to be a coordinate system used to control the robot 1. For example, the global coordinate system may be a world coordinate system that is determined based on the support surface S on which the robot 1 is placed. In other words, the global coordinate system may be a world coordinate system that is fixed with respect to the support surface S on which the robot 1 is placed.

[0086] However, the control device 3 may calculate at least one of the position and orientation of the target object OBJ in a coordinate system different from the global coordinate system based on at least one of the image data IMG_2D and IMG_3D.

[0087] As a first example, the control device 3 may calculate at least one of the position and orientation of the target object OBJ in the robot coordinate system based on at least one of the image data IMG_2D and IMG_3D. The robot coordinate system may be a coordinate system defined based on the robot 1. In other words, the robot coordinate system may be a coordinate system fixed with respect to the robot 1 (for example, fixed with respect to the base 11 of the robot 1). Since the robot coordinate system is a coordinate system defined based on the robot 1, it may be referred to as a global coordinate system. The robot coordinate system can also be said to be a coordinate system used to control the robot 1.

[0088] As a second example, the control device 3 may calculate at least one of the position and orientation of the target object OBJ in a 2D imaging coordinate system based on at least one of the image data IMG_2D and IMG_3D. The 2D imaging coordinate system may be a coordinate system defined based on the imaging device 21. In other words, the 2D imaging coordinate system may be a coordinate system fixed with respect to the imaging device 21. An example of the 2D imaging coordinate system is a coordinate system defined based on the optical axis AX21 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 21. An example of the 2D imaging coordinate system is a coordinate system in which one of the three coordinate axes constituting the 2D imaging coordinate system is an axis aligned with the optical axis AX21 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 21.

[0089] As a third example, the control device 3 may calculate at least one of the position and orientation of the target object OBJ in a 3D imaging coordinate system based on at least one of the image data IMG_2D and IMG_3D. The 3D imaging coordinate system may be a coordinate system defined based on the imaging device 22. In other words, the 3D imaging coordinate system may be a coordinate system fixed with respect to the imaging device 22. An example of the 3D imaging coordinate system is a coordinate system defined based on the optical axis AX22 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 22. An example of the 3D imaging coordinate system is a coordinate system in which one of the three coordinate axes constituting the 3D imaging coordinate system is an axis aligned with the optical axis AX22 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 22.

[0090] The control device 3 may perform end effector control processing in addition to performing robot control processing. The end effector control processing may include processing for generating an end effector control signal for controlling the end effector 4. Specifically, the control device 3 may generate the end effector control signal based on at least one of the calculated position and orientation of the target object OBJ.

[0091] The end effector control processing may or may not be included in the robot control processing. In other words, the end effector control signal generated by the control device 3 may or may not be included in the robot control signal. In the following description, for convenience of explanation, an example will be described in which the end effector control processing is included in the robot control processing (in other words, the end effector control signal is included in the robot control signal). Therefore, in the following description, the robot control processing may mean processing for generating at least one of a robot control signal and an end effector control signal. In addition, in the following description, the robot control signal may mean at least one of a signal for controlling the robot 1 and a signal for controlling the end effector 4. The robot control signal may also be simply referred to as a control signal.

[0092] In this way, the control device 3 and the imaging system 2 are used to control the robot 1. Therefore, a system including the control device 3 and the imaging system 2 may be referred to as a robot control system or a control system.

[0093] The robot control signal generated by the control device 3 is output to the robot control device 13 of the robot 1. The robot control device 13 controls the operation of the robot 1 based on the robot control signal generated by the control device 3. For this reason, the robot control signal may include a signal for controlling the operation of the robot 1.

[0094] As described above, when the robot control signal includes a signal for controlling the robot arm 12, the robot control device 13 may control the robot arm 12 based on the robot control signal. For example, the robot control device 13 may control the operation of the actuator built into the joint 122 based on the robot control signal, thereby controlling the operation of the robot arm 12.

[0095] For example, as described above, the robot arm 12 moves the end effector 4. In this case, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at a desired position. The robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves to a desired position. The robot control signal may include a signal for controlling the robot arm 12 so that the positional relationship between the end effector 4 and the target object OBJ is a desired positional relationship. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 is located at a desired position. The robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves to a desired position. The robot control device 13 may control the robot arm 12 based on the robot control signal so that the positional relationship between the end effector 4 and the target object OBJ is a desired positional relationship.

[0096] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves toward (i.e., approaches) a holding position where the end effector 4 can hold the target object OBJ. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at the holding position. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves toward (i.e., approaches) the holding position. That is, the robot control signal may control the robot arm 12 so that the end effector 4 is located at the holding position.

[0097] As another example, when performing a release process to release the target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves toward (i.e., approaches) a release position where the target object OBJ held by the end effector 4 should be released. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at the release position. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves toward (i.e., approaches) the release position. That is, the robot control signal may control the robot arm 12 so that the end effector 4 is located at the release position.

[0098] As described above, when the robot control signal includes a signal for controlling the end effector 4, the robot control device 13 may control the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the end effector 4 by controlling the operation of an actuator that moves a hand gripper that constitutes the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the vacuum device of a vacuum gripper that constitutes the end effector 4 based on the robot control signal.

[0099] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the end effector 4 so that the end effector 4 located at the holding position holds the target object OBJ. In this case, the robot control device 13 may control the end effector 4 based on the robot control signal so that the end effector 4 located at the holding position holds the target object OBJ.

[0100] As another example, when performing a release process to release the target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the end effector 4 to release the target object OBJ held by the end effector 4 located at the release position described above. In this case, the robot control device 13 may control the end effector 4, based on the robot control signal, to release the target object OBJ held by the end effector 4 located at the release position.

[0101] The robot control signal may include a signal that can be used directly by the robot control device 13 to control the operation of the robot 1. The robot control signal may include a signal that can be used directly as a robot drive signal used by the robot control device 13 to control the operation of the robot 1. In this case, the robot control device 13 may use the robot control signal as is to control the operation of the robot 1. For example, the control device 3 may generate a drive signal for an actuator built into the joint 122 of the robot arm 12 as the robot control signal, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the actuator built into the joint 122 of the robot arm 12. The robot control signal may include a signal that can be used directly by the robot control device 13 to control the operation of the end effector 4. The robot control signal may include a signal that can be used directly as an end effector drive signal used by the robot control device 13 to control the operation of the end effector 4. In this case, for example, the control device 3 may generate, as a robot control signal, a drive signal (end effector drive signal) for an actuator that moves a hand gripper that constitutes the end effector 4, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the actuator of the end effector 4. For example, the control device 3 may generate, as a robot control signal, a drive signal (end effector drive signal) for driving a vacuum device of a vacuum gripper that constitutes the end effector 4, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the vacuum device of the end effector 4.

[0102] As described above, if the robot control signal includes a signal that can be used directly by the robot control device 13 to control the operation of at least one of the robot 1 and the end effector 4, the robot 1 does not need to be equipped with the robot control device 13.

[0103] Alternatively, the robot control signal may include a signal that can be used by the robot control device 13 to generate a robot drive signal for controlling the operation of the robot 1. In this case, the robot control device 13 may generate a robot drive signal for controlling the operation of the robot 1 based on the robot control signal, and control the operation of the robot 1 based on the generated robot drive signal. For example, the robot control device 13 may generate a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 based on the robot control signal, and control the actuator built into the joint 122 of the robot arm 12 based on the generated robot drive signal.

[0104] The robot control signal may include a signal usable by the robot control device 13 to generate an end effector drive signal for controlling the operation of the end effector 4. In this case, the robot control device 13 may generate an end effector drive signal for controlling the operation of the end effector 4 based on the end effector control signal, and control the operation of the end effector 4 based on the generated end effector drive signal. For example, if the end effector 4 is a hand gripper, the robot control device 13 may generate an end effector drive signal for driving an actuator of the hand gripper based on the robot control signal, and control the actuator of the hand gripper based on the generated end effector drive signal. For example, if the end effector 4 is a magnetically attractive gripper, the robot control device 13 may generate an end effector drive signal for driving a magnetically attractive device of the magnetically attractive gripper based on the robot control signal, and control the magnetically attractive device based on the generated end effector drive signal.

[0105] The signals that the robot control device 13 can use to generate the robot drive signal may include signals that represent at least one of the position and orientation of the target object OBJ in the global coordinate system. In this case, for example, the robot control device 13 may generate, based on the robot control signal, a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 so that the end effector 4 approaches the target object OBJ, whose position and / or orientation in the global coordinate system has been calculated by the robot control signal (i.e., so that the positional relationship between the robot 1 (end effector 4) and the target object OBJ becomes a desired positional relationship), and control the operation of the robot arm 12 based on the generated robot drive signal.

[0106] The signals that the robot controller 13 can use to generate the robot drive signal may include a signal that represents a desired positional relationship between the robot 1 and the target object OBJ in a global coordinate system. In this case, for example, the robot controller 13 may generate a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 based on the robot control signal so that the positional relationship between the robot 1 (end effector 4) and the target object OBJ becomes the desired positional relationship represented by the robot control signal, and control the operation of the robot arm 12 based on the generated robot drive signal.

[0107] The signals that the robot control device 13 can use to generate the robot drive signal may include a signal representing a desired position of the end effector 4 in a global coordinate system. In this case, for example, the robot control device 13 may generate a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 based on the robot control signal so that the end effector 4 is positioned at the desired position represented by the robot control signal (i.e., so that the positional relationship between the robot 1 (end effector 4) and the target object OBJ is the desired positional relationship), and control the operation of the robot arm 12 based on the generated robot drive signal. An example of the desired position is a processing position where the end effector 4 should process the target object OBJ. A specific example of the desired position is a holding position where the end effector 4 should hold the target object OBJ. In this case, the robot control device 13 may generate, based on the robot control signal, a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 so that the end effector 4 moves to the holding position represented by the robot control signal, and may control the operation of the robot arm 12 based on the generated robot drive signal. Another example of a desired position is a release position where the end effector 4 should release the target object OBJ. In this case, the robot control device 13 may generate, based on the robot control signal, a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 so that the end effector 4 is positioned at the release position represented by the robot control signal, and may control the operation of the robot arm 12 based on the generated robot drive signal.

[0108] The signals that the robot controller 13 can use to generate the robot drive signal may include, for example, a signal that represents a desired position of the tip of the robot arm 12 (e.g., a tool center point) in the global coordinate system, or a signal that represents a desired position of the imaging system 2 in the global coordinate system.The signals that the robot controller 13 can use to generate the robot drive signal may also be signals that represent the amount and direction of movement from the current position of the end effector 4 to a desired position of the end effector 4.

[0109] The coordinate system that serves as the reference for the robot control signal may be a coordinate system other than the global coordinate system (for example, a robot coordinate system, a 2D imaging coordinate system, or a 3D imaging coordinate system).

[0110] (1-2) Configuration of the Control Device 3 Next, the configuration of the control device 3 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the control device 3.

[0111] 3 , the control device 3 includes a calculation device 31, a storage device 32, and a communication device 33. The control device 3 may further include an input device 34 and an output device 35. However, the control device 3 does not necessarily have to include at least one of the input device 34 and the output device 35. The calculation device 31, the storage device 32, the communication device 33, the input device 34, and the output device 35 may be connected via a data bus 36.

[0112] The arithmetic device 31 includes, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a field programmable gate array (FPGA). The arithmetic device 31 reads a computer program. For example, the arithmetic device 31 may read a computer program stored in the storage device 32. For example, the arithmetic device 31 may read a computer program stored in a computer-readable, non-transitory storage medium using a storage medium reading device (not shown) included in the control device 3. The arithmetic device 31 may acquire (i.e., download or read) the computer program from a device (not shown) located outside the control device 3 via the communication device 33 (or another communication device). The arithmetic device 31 executes the read computer program. As a result, a logical function block for executing the processing to be performed by the control device 3 (for example, the robot control processing described above) is realized within the arithmetic device 31. In other words, the arithmetic device 31 can function as a controller for realizing the logical function block for executing the processing to be performed by the control device 3.

[0113] A computational model that can be constructed by machine learning may be implemented in the computational device 31 by the computational device executing a computer program. An example of a computational model that can be constructed by machine learning is a computational model including a neural network (so-called artificial intelligence (AI)). In this case, learning of the computational model may include learning of parameters of the neural network (e.g., at least one of a weight and a bias). The computational device 31 may execute a robot control process using the computational model. In other words, the operation of executing the robot control process may include the operation of executing the robot control process using the computational model. Note that a computational model that has been constructed by offline machine learning using teacher data may be implemented in the computational device 31. Furthermore, the computational model implemented in the computational device 31 may be updated by online machine learning on the computational device 31. Alternatively, the calculation device 31 may perform robot control processing using a calculation model implemented in a device external to the calculation device 31 (i.e., a device provided outside the control device 3) in addition to or instead of the calculation model implemented in the calculation device 31.

[0114] The recording medium for recording the computer program executed by the arithmetic unit 31 may be at least one of the following: a CD-ROM, CD-R, CD-RW, a flexible disk, an MO, a DVD-ROM, a DVD-RAM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, and an optical disk such as Blu-ray (registered trademark), a magnetic medium such as a magnetic tape, a magneto-optical disk, a semiconductor memory such as a USB memory, and any other medium capable of storing a program. The recording medium may also include a device capable of recording a computer program (for example, a general-purpose device or a dedicated device in which a computer program is implemented in an executable state in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program may be realized by a logical processing block realized within the arithmetic device 31 (i.e., a computer) when the arithmetic device 31 executes the computer program, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) included in the arithmetic device 31, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.

[0115] 3 shows an example of logical functional blocks realized in the arithmetic device 31 for executing robot control processing. As shown in FIG. 3, a three-dimensional position data generation unit 311, a position and orientation calculation unit 312, and a signal generation unit 313 are realized in the arithmetic device 31. Note that the processes performed by the three-dimensional position data generation unit 311, the position and orientation calculation unit 312, and the signal generation unit 313 will be described in detail later with reference to FIG. 4 etc., and therefore will not be described here. Note that the arithmetic device 31 may also be referred to as a calculation unit.

[0116] The storage device 32 can store desired data. For example, the storage device 32 may temporarily store a computer program executed by the arithmetic device 31. The storage device 32 may temporarily store data that the arithmetic device 31 temporarily uses when the arithmetic device 31 is executing a computer program. The storage device 32 may also store data that the control device 3 stores long-term. The storage device 32 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 32 may include a non-temporary recording medium.

[0117] The communication device 33 is capable of communicating with both the robot 1 and the imaging system 2 via a communication network (not shown). Alternatively, the communication device 33 may be capable of communicating with another device different from the robot 1 and the imaging system 2, in addition to or instead of at least one of the robot 1 and the imaging system 2, via a communication network (not shown). In this embodiment, the communication device 33 may receive (i.e., acquire) at least one of the image data IMG_2D and IMG_3D from the imaging system 2. Furthermore, the communication device 33 may transmit (i.e., output) a robot control signal to the robot 1. The communication device 33 that outputs the robot control signal to the robot 1 may be referred to as an output unit.

[0118] The input device 34 is a device that accepts information input to the control device 3 from outside the control device 3. For example, the input device 34 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by a user of the control device 3. For example, the input device 34 may include a recording medium reading device that can read information recorded as data on a recording medium that can be externally attached to the control device 3.

[0119] It should be noted that information can be input as data to the control device 3 from a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an input device that accepts information input to the control device 3 from outside the control device 3.

[0120] The output device 35 is a device that outputs information to the outside of the control device 3. For example, the output device 35 may output information as an image. That is, the output device 35 may include a display device (a so-called display) that can display an image. For example, the output device 35 may output information as sound. That is, the output device 35 may include an audio device (a so-called speaker) that can output sound. For example, the output device 35 may output information on paper. That is, the output device 35 may include a printing device (a so-called printer) that can print desired information on paper. For example, the output device 35 may output information as data to a recording medium that can be externally attached to the control device 3.

[0121] The control device 3 can output information as data to a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an output device that outputs information to a device external to the control device 3.

[0122] (2) Robot Control Processing Next, the robot control processing performed by the control device 3 will be described.

[0123] (2-1) Flow of Robot Control Processing First, the flow of the robot control processing will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of the robot control processing.

[0124] As shown in FIG. 4 , the control device 3 acquires image data IMG_3D from the imaging device 22 using the communication device 33 (step S1). Specifically, the imaging device 22 captures an image of the target object OBJ at a predetermined 3D imaging rate. For example, the imaging device 22 may capture an image of the target object OBJ at a 3D imaging rate of tens to hundreds (e.g., 500 times) per second. As a result, the imaging device 22 generates image data IMG_3D at a period corresponding to the predetermined 3D imaging rate. For example, the imaging device 22 may generate tens to hundreds (e.g., 500) pieces of image data IMG_3D per second. The control device 3 acquires the image data IMG_3D each time the imaging device 22 generates image data IMG_3D. In other words, the control device 3 may acquire tens to hundreds (e.g., 500) pieces of image data IMG_3D per second.

[0125] However, the imaging device 22 does not have to periodically capture images of the target object OBJ at a desired 3D imaging rate. For example, the imaging device 22 may capture an image of the target object OBJ when receiving a control signal from the control device 3 to control the imaging device 22 to capture an image of the target object OBJ.

[0126] The imaging device 22 may capture images of objects other than the target object OBJ, in addition to the target object OBJ on which the robot 1 performs a predetermined process. Because the other objects are not targets for the robot 1 to perform a predetermined process, the other objects other than the target object OBJ are referred to as non-target objects in the following description. Examples of non-target objects include at least a portion of the end effector 4, at least a portion of the robot arm 12, and at least one of peripheral objects located around the robot 1. Examples of peripheral objects include at least one of the placement device T and the container CB. For example, if both the target object OBJ and the non-target object are included in the imaging range (field of view) of the imaging device 22, the imaging device 22 may capture images of both the target object OBJ and the non-target object. As a result, the imaging device 22 may generate image data IMG_3D representing an image in which both the target object OBJ and the non-target object are captured. However, the imaging device 22 may capture images of the target object OBJ but not the non-target object. That is, the image capturing device 22 may generate image data IMG_3D representing an image in which the target object OBJ is captured but no non-target objects are captured. In either case, the image capturing device 22 generates image data IMG_3D representing an image in which at least the target object OBJ is captured. That is, the image capturing device 22 generates image data IMG_3D including image data of at least the target object OBJ.

[0127] As described above, the imaging device 22 may capture images of multiple target objects OBJ on which the end effector 4 sequentially performs a predetermined process. In this case, the end effector 4 may perform the predetermined process on a first target object OBJ among the multiple target objects OBJ during a first period, and may perform the predetermined process on a second target object OBJ different from the first target object OBJ among the multiple target objects OBJ during a second period following the first period. In this case, among the multiple target objects OBJ, target objects OBJ other than the first target object OBJ on which the end effector 4 performs the predetermined process may be considered to be target objects OBJ that are not processing objects on which the robot 1 performs the predetermined process during the first period (or non-target objects). Similarly, among the multiple target objects OBJ, target objects OBJ other than the second target object OBJ on which the end effector 4 performs the predetermined process during the second period may be considered to be target objects OBJ that are not processing objects on which the robot 1 performs the predetermined process during the second period (or non-target objects). The first target object OBJ and the second target object OBJ may be different target objects OBJ from among a plurality of target objects OBJ.

[0128] The control device 3 may further use the communication device 33 to acquire image data IMG_2D from the imaging device 21 in addition to or instead of acquiring image data IMG_3D from the imaging device 22 (step S1). Specifically, the imaging device 21 captures an image of the target object OBJ at a predetermined 2D imaging rate. The 2D imaging rate is the same as the 3D imaging rate. However, the 2D imaging rate may differ from the 3D imaging rate. For example, the imaging device 21 may capture an image of the target object OBJ at a 2D imaging rate that captures an image of the target object OBJ tens to hundreds of times (e.g., 500 times) per second. As a result, the imaging device 21 generates image data IMG_2D at a period corresponding to the predetermined 2D imaging rate. For example, the imaging device 21 may generate tens to hundreds of pieces of image data IMG_2D per second (e.g., 500 pieces). The control device 3 acquires the image data IMG_2D every time the imaging device 21 generates the image data IMG_2D. That is, the control device 3 may acquire several tens to several hundreds (for example, 500) of image data IMG_2D per second.

[0129] However, the imaging device 21 does not have to periodically capture images of the target object OBJ at a desired 2D imaging rate. For example, the imaging device 21 may capture an image of the target object OBJ when receiving a control signal from the control device 3 to control the imaging device 21 to capture an image of the target object OBJ.

[0130] The image data IMG_2D may be referred to as two-dimensional position data indicating the two-dimensional position of the target object OBJ reflected in the image data IMG_2D. The image data IMG_2D (two-dimensional position data) may be data indicating the two-dimensional position of at least a portion of the surface of the target object OBJ. In particular, the image data IMG_2D (two-dimensional position data) is data indicating the two-dimensional position of each of multiple points on the surface of the target object OBJ. For example, the image data IMG_2D (two-dimensional position data) may be data indicating the two-dimensional position of each of multiple points on the surface of the target object OBJ. For example, the image data IMG_2D (two-dimensional position data) may be data indicating the two-dimensional position of each of multiple points corresponding to multiple portions on the surface of the target object OBJ.

[0131] The imaging device 21 may capture an image of a non-target object different from the target object OBJ, in addition to the target object OBJ on which the robot 1 performs a predetermined process. For example, if both the target object OBJ and the non-target object are included in the imaging range (field of view) of the imaging device 21, the imaging device 21 may capture images of both the target object OBJ and the non-target object. As a result, the imaging device 21 may generate image data IMG_2D representing an image in which both the target object OBJ and the non-target object are captured. However, the imaging device 21 may capture an image of the target object OBJ but not the non-target object. In other words, the imaging device 21 may generate image data IMG_2D representing an image in which the target object OBJ is captured but the non-target object is not captured. In either case, the imaging device 21 generates image data IMG_2D representing an image in which at least the target object OBJ is captured. In other words, the imaging device 21 generates image data IMG_2D including image data of at least the target object OBJ.

[0132] As described above, the imaging device 21 may capture images of multiple target objects OBJ on which the end effector 4 sequentially performs a predetermined process. In this case, the end effector 4 may perform a predetermined process on a third target object OBJ among the multiple target objects OBJ during a third period, and then perform a predetermined process on a fourth target object OBJ different from the third target object OBJ among the multiple target objects OBJ during a fourth period following the third period. In this case, during the third period, the target objects OBJ other than the third target object OBJ on which the end effector 4 performs a predetermined process may be considered to be target objects OBJ that are not processing objects on which the robot 1 performs a predetermined process during the third period (or non-target objects). Similarly, during the fourth period, the target objects OBJ other than the fourth target object OBJ on which the end effector 4 performs a predetermined process may be considered to be target objects OBJ that are not processing objects on which the robot 1 performs a predetermined process during the fourth period (or non-target objects). The third target object OBJ and the fourth target object OBJ may be different target objects OBJ from among a plurality of target objects OBJ.

[0133] When the control device 3 acquires image data IMG_3D, the three-dimensional position data generator 311 generates three-dimensional position data WSD based on the acquired image data IMG_3D each time the control device 3 acquires image data IMG_3D (step S2). Furthermore, the three-dimensional position data generator 311 outputs the generated three-dimensional position data WSD to the position and orientation calculation unit 312. However, if the control device 3 has not acquired image data IMG_3D (for example, if the control device 3 has acquired image data IMG_2D), the three-dimensional position data generator 311 does not need to generate the three-dimensional position data WSD. In this case, the control device 3 does not need to perform the operation of step S2.

[0134] The control device 3 may generate the three-dimensional position data WSD based on each of some image data IMG_3D among the plurality of image data IMG_3D generated by the imaging device 22. The control device 3 may generate the three-dimensional position data WSD without using other some image data IMG_3D among the plurality of image data IMG_3D generated by the imaging device 22.

[0135] The three-dimensional position data WSD is data indicating the three-dimensional position of the target object OBJ captured in the image data IMG_3D. For example, the three-dimensional position data WSD may be data indicating the three-dimensional position of at least a portion of the surface of the target object OBJ. In particular, the three-dimensional position data WSD is data indicating the three-dimensional position of each of multiple points on the target object OBJ. In other words, the three-dimensional position data WSD is data indicating the three-dimensional position of each of multiple points on the target object OBJ captured by the imaging device 22. For example, the three-dimensional position data WSD may be data indicating the three-dimensional position of each of multiple points on the surface of the target object OBJ. For example, the three-dimensional position data WSD may be data indicating the three-dimensional position of each of multiple points corresponding to multiple portions on the surface of the target object OBJ. In the following explanation, unless otherwise specified, the three-dimensional position of the target object OBJ may mean at least one of the three-dimensional position of at least a portion of the surface of the target object OBJ, the three-dimensional position of each of multiple points on the target object OBJ, the three-dimensional position of each of multiple points on the surface of the target object OBJ, and the three-dimensional position of each of multiple points corresponding to multiple parts of the surface of the target object OBJ.

[0136] Specifically, as described above, the image represented by the image data IMG_3D reflects the target object OBJ onto which the projection pattern is projected. In this case, the projection pattern reflected in the image represented by the image data IMG_3D reflects the three-dimensional shape of at least a portion of the surface of the target object OBJ onto which the projection pattern is projected. The shape of the projection pattern reflected in the image represented by the image data IMG_3D reflects the three-dimensional shape of at least a portion of the surface of the target object OBJ onto which the projection pattern is projected. Therefore, the three-dimensional position data generation unit 311 can calculate the three-dimensional shape of at least a portion of the surface of the target object OBJ based on the projection pattern reflected in the image represented by the image data IMG_3D. The three-dimensional shape of at least a portion of the surface of the target object OBJ essentially indicates the three-dimensional position of each of multiple points on the target object OBJ. This is because each of the multiple points on the target object OBJ is contained within the surface of the target object OBJ. Therefore, the process of calculating the three-dimensional shape of at least a part of the surface of the target object OBJ may be considered to be substantially equivalent to the process of calculating the three-dimensional position of each of multiple points on the target object OBJ. Therefore, the three-dimensional position data generator 311 can generate the three-dimensional position data WSD based on the image data IMG_3D.

[0137] Furthermore, considering that each of the multiple points of the target object OBJ is contained within the surface of the target object OBJ, the three-dimensional position data WSD indicating the three-dimensional position of each of the multiple points of the target object OBJ may be considered to be essentially equivalent to three-dimensional shape data indicating the three-dimensional shape of at least a portion of the target object OBJ (in particular, the three-dimensional shape of at least a portion of the surface of the target object OBJ).

[0138] To generate the three-dimensional position data WSD, the three-dimensional position data generation unit 311 may calculate the parallax by associating each portion (e.g., each pixel) of the images represented by each of the two image data included in the image data IMG_3D. Specifically, in this association, the three-dimensional position data generation unit 311 may calculate the parallax by associating each portion of the projection pattern reflected in the images represented by each of the two image data (i.e., each portion of the projection pattern reflected in each image). The three-dimensional position data generation unit 311 may calculate the three-dimensional position of each of multiple points on the target object OBJ using a well-known method based on the principle of triangulation using the calculated parallax. As a result, three-dimensional position data WSD indicating the three-dimensional position of each of the multiple points on the target object OBJ is generated. In this case, the accuracy of the parallax calculation is higher when associating each portion of the images in which the projection pattern is reflected (i.e., each portion of the reflected projection pattern) rather than when associating each portion of the images in which the projection pattern is not reflected. Therefore, the accuracy of the generated three-dimensional position data WSD (that is, the accuracy of calculating the three-dimensional positions of each of the multiple points of the target object OBJ) is improved.

[0139] The three-dimensional position data WSD may be any data as long as it can indicate the three-dimensional position of each of the multiple points of the target object OBJ. In other words, the three-dimensional position data WSD may be any data that directly or indirectly indicates the three-dimensional position of each of the multiple points of the target object OBJ. For example, the three-dimensional position data WSD may include coordinate information indicating the three-dimensional position of each of the multiple points of the target object OBJ. For example, the three-dimensional position data WSD may include information indicating the three-dimensional shape of at least a portion of the target object OBJ.

[0140] An example of three-dimensional position data WSD is depth image data. Depth image data is image data in which depth information is associated with each pixel of a depth image represented by the depth image data, in addition to or instead of brightness information. Depth information is information indicating the distance (i.e., depth) between each part of the target object OBJ reflected in each pixel and the imaging device 22. The distance (i.e., depth) between each part of the target object OBJ reflected in each pixel and the imaging device 22 can be calculated from the above-mentioned parallax. Note that the depth image data may be image data in which the brightness information of each pixel indicates the depth of each part of the target object OBJ (i.e., the distance between each part of the target object OBJ and the imaging device 22). The three-dimensional position data generation unit 311 may calculate the distance between each part of the target object OBJ reflected in the image indicated by the image data IMG_3D and the imaging device 22 based on the projection pattern reflected in the image indicated by the image data IMG_3D, and generate a depth image by associating the calculated distance with each pixel of the image indicated by the image data IMG_3D as depth information.

[0141] Another example of the three-dimensional position data WSD is point cloud data. Point cloud data is data that indicates a collection of points in three-dimensional space that correspond to each part of the target object OBJ that appears in the image indicated by the image data IMG_3D. The three-dimensional position data generator 311 may generate the point cloud data based on the depth image data and the camera parameters of the imaging device 22. Note that the following description will discuss an example in which point cloud data is used as the three-dimensional position data WSD.

[0142] As described above, when the imaging range (field of view) of the imaging device 22 includes a non-target object different from the target object OBJ in addition to the target object OBJ on which the robot 1 performs a predetermined process, the image data IMG_3D represents an image in which both the target object OBJ and the non-target object are captured. In this case, the three-dimensional position data WSD may be data indicating the three-dimensional position of the target object OBJ captured in the image data IMG_3D as well as the three-dimensional position of the non-target object captured in the image data IMG_3D. In other words, the three-dimensional position data WSD may include data indicating the three-dimensional positions of each of multiple points of the non-target object captured by the imaging device 22. Even in this case, as long as the three-dimensional position data WSD includes data indicating the three-dimensional position of the target object OBJ, the three-dimensional position data WSD may be considered to be data indicating the three-dimensional position of each of the multiple points of the target object OBJ. This is because even if data indicating the three-dimensional position of a non-target object is included in the three-dimensional position data WSD, the fact remains that data indicating the three-dimensional position of the target object OBJ is included in the three-dimensional position data WSD.

[0143] Thereafter, the position and orientation calculation unit 312 calculates at least one of the position and orientation of the target object OBJ based on at least one of the image data IMG_2D acquired in step S1 and the three-dimensional position data WSD generated in step S2 (step S3). As a result, the position and orientation calculation unit 312 generates position and orientation data POI indicating at least one of the position and orientation of the target object OBJ (step S3). The position and orientation data POI may include position data indicating the position of the target object OBJ. The position and orientation data POI may include orientation data indicating the orientation of the target object OBJ in addition to or instead of the position data indicating the position of the target object OBJ. In other words, the position and orientation data POI may include at least one of position data and orientation data.

[0144] In step S3, the position and orientation calculation unit 312 calculates at least one of the position and orientation of the target object OBJ in the global coordinate system. That is, the position and orientation calculation unit 312 generates position and orientation data POI indicating at least one of the position and orientation of the target object OBJ in the global coordinate system. For example, the robot control device 13 may control the robot arm 12 so that the end effector 4 is positioned at a desired position in the global coordinate system. The global coordinate system is a coordinate system defined by an X-axis (GL), a Y-axis (GL), and a Z-axis (GL) that are orthogonal to each other. The X-axis (GL) may be an axis along a horizontal plane. The Y-axis (GL) may be an axis along a horizontal plane. The Z-axis (GL) may be an axis perpendicular to the horizontal plane. The Z-axis (GL) may be an axis extending in the direction of gravity. The X-axis, Y-axis, and Z-axis shown in FIG. 2 may be the X-axis (GL), the Y-axis (GL), and the Z-axis (GL), respectively. The origin of the global coordinate system does not have to be the origin of the X-axis (GL), Y-axis (GL), and Z-axis (GL) shown in Fig. 2. For example, the origin of the global coordinate system may be set at any position on the base 11 in Fig. 2. For example, the origin of the global coordinate system may be set at any position on the support surface S. For example, the origin of the global coordinate system may be set at any position on the contact surface of the base 11 with the support surface S (for example, the center or center of gravity of the contact surface).

[0145] The position and orientation calculation unit 312 may calculate, as the position of the target object OBJ in the global coordinate system, at least one of a position Tx(GL) of the target object OBJ in an X-axis direction (GL) parallel to the X-axis (GL), a position Ty(GL) of the target object OBJ in a Y-axis direction (GL) parallel to the Y-axis (GL), and a position Tz(GL) of the target object OBJ in a Z-axis direction (GL) parallel to the Z-axis (GL). The position and orientation calculation unit 312 may calculate, as the orientation of the target object OBJ in the global coordinate system, at least one of a rotation amount Rx(GL) of the target object OBJ around the X-axis (GL), a rotation amount Ry(GL) of the target object OBJ around the Y-axis (GL), and a rotation amount Rz(GL) of the target object OBJ around the Z-axis (GL). This is because the rotation amount Rx(GL) of the target object OBJ around the X-axis (GL), the rotation amount Ry(GL) of the target object OBJ around the Y-axis (GL), and the rotation amount Rz(GL) of the target object OBJ around the Z-axis (GL) are equivalent to the parameters representing the orientation of the target object OBJ around the X-axis (GL), the parameters representing the orientation of the target object OBJ around the Y-axis (GL), and the parameters representing the orientation of the target object OBJ around the Z-axis (GL), respectively. For this reason, in the following description, the rotation amount Rx(GL) of the target object OBJ around the X-axis (GL), the rotation amount Ry(GL) of the target object OBJ around the Y-axis (GL), and the rotation amount Rz(GL) of the target object OBJ around the Z-axis (GL) will be referred to as the orientation Rx(GL) of the target object OBJ around the X-axis (GL), the orientation Ry(GL) of the target object OBJ around the Y-axis (GL), and the orientation Rz(GL) of the target object OBJ around the Z-axis (GL), respectively.

[0146] The orientation Rx(GL) of the target object OBJ around the X-axis (GL), the orientation Ry(GL) of the target object OBJ around the Y-axis (GL), and the orientation Rz(GL) of the target object OBJ around the Z-axis (GL) may be considered to indicate the position of the target object OBJ in the rotation direction around the X-axis (GL), the position of the target object OBJ in the rotation direction around the Y-axis (GL), and the position of the target object OBJ in the rotation direction around the Z-axis (GL), respectively. In other words, the orientation Rx(GL) of the target object OBJ around the X-axis (GL), the orientation Ry(GL) of the target object OBJ around the Y-axis (GL), and the orientation Rz(GL) of the target object OBJ around the Z-axis (GL) may all be considered to be parameters representing the position of the target object OBJ.

[0147] In this way, in step S3 of FIG. 4, the position and orientation calculation unit 312 may calculate at least one of the position Tx(GL), position Ty(GL), position Tz(GL), orientation Rx(GL), orientation Ry(GL), and orientation Rz(GL) as at least one of the position and orientation of the target object OBJ in the global coordinate system.

[0148] 4, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ by performing a matching process using at least one of the image data IMG_2D and the three-dimensional position data WSD. For convenience of explanation, the following description will mainly focus on an example in which the position and orientation calculation unit 312 calculates at least one of the position and orientation of the target object OBJ by performing a matching process using the three-dimensional position data WSD. Note that in the following description, the matching process using the three-dimensional position data WSD will be referred to as a 3D matching process.

[0149] Specifically, the position and orientation calculation unit 312 may perform 3D matching processing using the three-dimensional position data WSD and three-dimensional model data WMD indicating a three-dimensional model that serves as a reference for the target object OBJ. In this case, the 3D matching processing may be considered to be, for example, a matching processing using a point cloud indicated by the three-dimensional position data WSD and a three-dimensional model indicated by the three-dimensional model data. Note that the 3D matching processing itself may be the same as an existing matching processing. For example, the position and orientation calculation unit 312 may perform the 3D matching processing using a well-known method including at least one of RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), ICP (Iterative Closest Point), and DSO (Direct Sparse Odometry).

[0150] The three-dimensional model based on the three-dimensional position data WSD may include a point cloud model, which is a three-dimensional model that indicates the three-dimensional shape of the target object OBJ using the point cloud indicated by the three-dimensional position data WSD. The three-dimensional model based on the three-dimensional position data WSD may include a polygon model or a mesh model that is generated by connecting multiple points that make up the point cloud indicated by the three-dimensional position data WSD with surfaces (particularly, polygonal surfaces with each point as a vertex). The three-dimensional model based on the three-dimensional position data WSD may include a solid model that is generated by connecting multiple points that make up the point cloud indicated by the three-dimensional position data WSD with smooth surfaces.

[0151] The three-dimensional model data WMD is data representing a template model TM3, which is a three-dimensional model of the target object OBJ. In other words, the three-dimensional model data WMD is data representing the template model TM3, which is a three-dimensional model having a three-dimensional shape that serves as a reference for the target object OBJ. The template model may be a CAD model of the target object OBJ. The template model may be a three-dimensional model having a shape similar to the three-dimensional shape of the target object OBJ obtained by measuring the three-dimensional shape of the actual target object OBJ in advance. The template model may be a polygon model, a mesh model, or a solid model. The three-dimensional model data WMD may be point cloud data representing the template model TM3 of the target object OBJ. The three-dimensional model data WMD may be polygon model data, mesh model data, or solid model data generated from the point cloud data. Note that the actual target object OBJ measured in advance to generate the three-dimensional model data WMD may be a reference or non-defective target object OBJ.

[0152] In this case, the position and orientation calculation unit 312 may perform 3D matching processing (in other words, template matching processing) on ​​the three-dimensional position data WSD using the template model TM3 indicated by the three-dimensional model data WMD as a template, as shown in Fig. 5. Specifically, the position and orientation calculation unit 312 may translate, enlarge, reduce, and / or rotate the template model TM3 within the 3D imaging coordinate system so that characteristic locations (e.g., at least one of feature points and edges) of the template model TM3 indicated by the three-dimensional model data WMD approach (e.g., match) characteristic locations of the target object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD (e.g., a point cloud corresponding to the target object OBJ indicated by the three-dimensional position data WSD, or a three-dimensional model (e.g., a point cloud model) reflecting the actual three-dimensional position of the target object OBJ). In other words, the position and orientation calculation unit 312 may change the positional relationship between the coordinate system of the three-dimensional model data WMD (e.g., the coordinate system of a CAD model) and the 3D imaging coordinate system so that characteristic locations of the target object OBJ, whose three-dimensional position is indicated by the three-dimensional position data WSD (e.g., a point cloud corresponding to the target object OBJ indicated by the three-dimensional position data WSD, or a three-dimensional model (e.g., a point cloud model) reflecting the actual three-dimensional position of the target object OBJ) approach (e.g., coincide) with characteristic locations of the template model TM3. As a result, the position and orientation calculation unit 312 can specify the positional relationship between the coordinate system of the three-dimensional model data WMD and the 3D imaging coordinate system. Thereafter, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ in the 3D imaging coordinate system from at least one of the position and orientation of the template model TM3 in the coordinate system of the three-dimensional model data WMD based on the positional relationship between the coordinate system of the three-dimensional model data WMD and the 3D imaging coordinate system.

[0153] Then, the position and orientation calculation unit 312 converts at least one of the position and orientation of the target object OBJ in the 3D imaging coordinate system into at least one of the position and orientation of the target object OBJ in the robot coordinate system based on a transformation matrix for converting three-dimensional coordinates in either the robot coordinate system or the 3D imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the 3D imaging coordinate system. When the robot coordinate system is used as the global coordinate system, at least one of the position and orientation of the target object OBJ in the robot coordinate system corresponds to at least one of the position and orientation of the target object OBJ in the global coordinate system. However, when a coordinate system other than the robot coordinate system is used as the global coordinate system, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ in the global coordinate system from at least one of the position and orientation of the target object OBJ in the robot coordinate system using a transformation matrix for converting three-dimensional coordinates in either the robot coordinate system or the global coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the global coordinate system.

[0154] When the position and orientation calculation unit 312 brings characteristic locations of the three-dimensional model indicated by the three-dimensional model data WMD closer to characteristic locations of the target object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD, the position and orientation calculation unit 312 may calculate a matching similarity, which is the similarity between the three-dimensional model of the target object OBJ indicated by the three-dimensional model data WMD and the target object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD (e.g., a point cloud of the target object OBJ). In the following description, the matching similarity, which is the similarity between the three-dimensional model of the target object OBJ indicated by the three-dimensional model data WMD and the target object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD (e.g., a point cloud of the target object OBJ), may be simply referred to as the matching similarity of the target object OBJ. The position and orientation calculation unit 312 may translate, enlarge, reduce, and / or rotate the three-dimensional model indicated by the three-dimensional model data WMD to maximize the matching similarity. The matching similarity may be considered to be equivalent to the correlation between the three-dimensional model indicated by the three-dimensional model data WMD and the target object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD. The correlation may also be considered an index showing the correlation between the three-dimensional model indicated by the three-dimensional model data WMD and the target object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD. The matching similarity may also be called a matching score.

[0155] When a target object OBJ whose calculated matching similarity exceeds a predetermined matching determination threshold is detected by the 3D matching process, the position and orientation calculation unit 312 may select the target object OBJ as one target object OBJ on which the end effector 4 should perform a predetermined process (i.e., a processing execution object). On the other hand, when a target object OBJ whose calculated matching similarity is below the matching determination threshold is detected by the 3D matching process, the position and orientation calculation unit 312 may not select the target object OBJ as one target object OBJ on which the end effector 4 should perform a predetermined process (i.e., a processing execution object).

[0156] Depending on the three-dimensional position data WSD, the three-dimensional position data WSD may indicate the respective positions of multiple target objects OBJ. For example, if multiple workpieces W are placed on the mounting device T in a disorderly or orderly manner, the imaging device 22 may capture images of the multiple workpieces W and generate image data IMG_3D in which the multiple workpieces W are captured as multiple target objects OBJ. For example, if multiple workpieces W are placed in a disorderly or orderly manner in a container (storage box) CB, which is an example of the mounting device T, the imaging device 22 may capture images of the multiple workpieces W and generate image data IMG_3D in which the multiple workpieces W are captured as multiple target objects OBJ. As a result, the position and orientation calculation unit 312 may generate three-dimensional position data WSD indicating the respective positions of the multiple target objects OBJ. In this case, the position and orientation calculation unit 312 may perform the above-described 3D matching process on each of the multiple target objects OBJ. For example, the position and orientation calculation unit 312 may sequentially perform the 3D matching process on the multiple target objects OBJ. Alternatively, for example, the position and orientation calculation unit 312 may simultaneously perform multiple 3D matching processes for multiple target objects OBJ, respectively. Thereafter, the position and orientation calculation unit 312 may select, from the multiple target objects OBJ, one target object OBJ whose matching similarity exceeds the matching determination threshold and has the largest matching similarity, as a processing execution object on which the end effector 4 should perform a predetermined process. Alternatively, the position and orientation calculation unit 312 may select, from the multiple target objects OBJ, one target object OBJ whose matching similarity exceeds the matching determination threshold and has the Nth largest matching similarity (where N is a constant representing an integer greater than or equal to 2), as a processing execution object on which the end effector 4 should perform a predetermined process. Alternatively, the position and orientation calculation unit 312 may select, from the multiple target objects OBJ, one target object OBJ whose matching similarity exceeds a predetermined matching determination threshold, as a processing execution object. Alternatively, the position and orientation calculation unit 312 may select, as the processing execution object, one target object OBJ from among the plurality of target objects OBJ, which corresponds to a matching similarity that exceeds the matching determination threshold and is closest to the end effector 4.Alternatively, for example, when a plurality of target objects OBJ are contained in the container CB (mounting device T) (e.g., when a plurality of objects are stacked in bulk), the position and orientation calculation unit 312 may select, as the processing execution object, one target object OBJ among the plurality of target objects OBJ that corresponds to a matching similarity that exceeds the matching determination threshold and has the largest Z coordinate along the Z axis (located at the highest position). Note that the Z coordinate of each of the plurality of target objects OBJ may be a Z position along the Z axis of the global coordinate system of the robot 1, calculated by the above-described 3D matching process. Alternatively, the position and orientation calculation unit 312 may select, as the processing execution object, one target object OBJ among the plurality of target objects OBJ that corresponds to a matching similarity that exceeds the matching determination threshold and on which the end effector 4 can perform a predetermined process.

[0157] However, if the end effector 4 is capable of simultaneously performing a predetermined process on two or more target objects OBJ, the position and orientation calculation unit 312 may select at least two of the multiple target objects OBJ as the process execution objects. In other words, the position and orientation calculation unit 312 may select at least two process execution objects on which the end effector 4 should simultaneously perform a predetermined process. For convenience of explanation, the following description will be given of an example in which the position and orientation calculation unit 312 selects one target object OBJ of the multiple target objects OBJ as the process execution object.

[0158] The position and orientation calculation unit 312 may calculate the matching similarity of multiple target objects OBJ, then select one target object OBJ on which the end effector 4 is to perform a predetermined process based on the matching similarity of the multiple target objects OBJ, and then calculate at least one of the position and orientation of the selected target object OBJ. In this case, the position and orientation calculation unit 312 does not need to calculate the position and orientation of other target objects OBJ that are not selected as the one target object OBJ on which the end effector 4 is to perform a predetermined process. Alternatively, because at least one of the position and orientation of the target object OBJ is calculated as a result of the 3D matching process as described above, the position and orientation calculation unit 312 may be considered to have calculated the above-mentioned matching similarity together with at least one of the position and orientation of the target object OBJ. In this case, the position and orientation calculation unit 312 may calculate the matching similarity of the multiple target objects OBJ together with at least one of the positions and orientations of each of the multiple target objects OBJ, and then, based on the matching similarity of the multiple target objects OBJ, select one target object OBJ on which the end effector 4 should perform a predetermined process. Alternatively, the position and orientation calculation unit 312 may calculate at least one of the positions and orientations of the multiple target objects OBJ, then calculate the matching similarity of the multiple target objects OBJ, and then, based on the calculated matching similarity, select one target object OBJ on which the end effector 4 should perform a predetermined process.

[0159] 4, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ by performing a matching process using the image data IMG_2D in addition to or instead of performing the 3D matching process using the three-dimensional position data WSD. In the following description, the matching process using the image data IMG_2D will be referred to as a 2D matching process.

[0160] Specifically, the position and orientation calculation unit 312 may perform a 2D matching process using the image data IMG_2D (two-dimensional position data) and two-dimensional model data indicating a two-dimensional model serving as a reference for the target object OBJ. Specifically, as the 2D matching process, the position and orientation calculation unit 312 may perform an object detection process in which, by using a two-dimensional model indicated by the two-dimensional model data as a template image within the image indicated by the image data IMG_2D, the target object OBJ indicated by the template image is detected. In other words, as the matching process, the position and orientation calculation unit 312 may perform an object detection process in which, by detecting a similar image portion within the image indicated by the image data IMG_2D that is similar to the template image, the target object OBJ is detected within the image indicated by the image data IMG_2D. Note that the 2D matching process (in this case, the object detection process) itself may be the same as an existing matching process. For example, the position and orientation calculation unit 312 may perform 2D matching processing using a well-known method such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed-Up Robust Feature).

[0161] The two-dimensional model data may be data based on three-dimensional model data of the target object OBJ. Here, the two-dimensional model data may be generated from CAD (Computer Aided Design) model data of the target object OBJ, or may be generated from model data (data representing a polygon model or mesh model) of the target object OBJ generated based on prior measurement results using a well-known three-dimensional shape measurement device or the imaging device 22. For example, the two-dimensional model data may be two-dimensional image data representing at least a portion of a two-dimensional model of the target object OBJ generated by virtually projecting at least a portion of a three-dimensional model represented by the three-dimensional model data of the target object OBJ onto a virtual plane. For example, the two-dimensional model data may be two-dimensional image data representing at least a portion of an edge of the two-dimensional model of the target object OBJ generated by virtually projecting at least a portion of a three-dimensional model represented by the three-dimensional model data of the target object OBJ onto a virtual plane. Note that the two-dimensional model data does not have to be data based on the three-dimensional model data of the target object OBJ. For example, the two-dimensional model data may be image data IMG_2D generated in advance by capturing an image of the target object OBJ using the imaging device 21. For example, the two-dimensional model data may be a characteristic location on an image represented by the image data IMG_2D generated in advance by capturing an image of the target object OBJ using the imaging device 21. For example, the characteristic location on the image represented by the image data IMG_2D may be at least one of a feature point and an edge of the target object OBJ that appears in the image represented by the image data IMG_2D. Note that if the characteristic location on the image represented by the image data IMG_2D includes an edge of the target object OBJ, the two-dimensional model may be considered to be a model of at least a portion of the edge of the target object OBJ (edge ​​model). The characteristic location on the image represented by the image data IMG_2D may be at least one of a plurality of feature points of the target object OBJ that appear in the image represented by the image data IMG_2D and at least a portion of the edges of the target object OBJ. Note that the characteristic portions on the image indicated by the image data IMG_2D may be detected by well-known image processing.

[0162] The position and orientation calculation unit 312 may translate, enlarge, reduce, and / or rotate the two-dimensional model of the target object OBJ reflected in the template image so that characteristic features (e.g., at least one of feature points and edges) of the two-dimensional model of the target object OBJ reflected in the template image approach (e.g., match) characteristic features of the target object OBJ reflected in the image represented by the image data IMG_2D. As a result, the position and orientation calculation unit 312 can identify the positional relationship between the coordinate system of the two-dimensional model data and the 2D imaging coordinate system. Thereafter, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ in the 2D imaging coordinate system from at least one of the position and orientation of the target object OBJ in the coordinate system of the two-dimensional model data based on the positional relationship between the coordinate system of the two-dimensional model data and the 2D imaging coordinate system.

[0163] Then, the position and orientation calculation unit 312 converts at least one of the position and orientation of the target object OBJ in the 2D imaging coordinate system into at least one of the position and orientation of the target object OBJ in the robot coordinate system based on a transformation matrix for converting three-dimensional coordinates in either the robot coordinate system or the 2D imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the 2D imaging coordinate system. When the robot coordinate system is used as the global coordinate system, at least one of the position and orientation of the target object OBJ in the robot coordinate system corresponds to at least one of the position and orientation of the target object OBJ in the global coordinate system. However, when a coordinate system other than the robot coordinate system is used as the global coordinate system, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ in the global coordinate system from at least one of the position and orientation of the target object OBJ in the robot coordinate system using a transformation matrix for converting three-dimensional coordinates in either the robot coordinate system or the global coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the global coordinate system.

[0164] When performing 2D matching processing, the position and orientation calculation unit 312 may calculate matching similarity, as in the case of 3D matching processing. The matching similarity calculated in the 2D matching processing is the similarity between the template image (i.e., the two-dimensional model of the target object OBJ) and the image indicated by the image data IMG_2D (particularly, the image portion to which the target object OBJ of the template image is fitted). In the following description, the matching similarity, which is the similarity between the template image (i.e., the two-dimensional model of the target object OBJ) and the image indicated by the image data IMG_2D (particularly, the image portion to which the target object OBJ of the template image is fitted), may simply be referred to as the matching similarity of the target object OBJ. Note that the use of the matching similarity calculated in the 2D matching processing may be the same as the use of the matching similarity calculated in the above-mentioned 3D matching processing, and therefore a detailed description thereof will be omitted.

[0165] When performing 2D matching processing, as in the case of 3D matching processing, the position and orientation calculation unit 312 may perform the 2D matching processing sequentially or simultaneously on multiple target objects OBJ captured in the image represented by the image data IMG_2D. Then, as in the case of 3D matching processing, the position and orientation calculation unit 312 may select at least one of the multiple target objects OBJ as the processing execution object. In this case, the position and orientation calculation unit 312 may select at least one of the multiple target objects OBJ as the processing execution object based on the matching similarity. Note that in step S3 of FIG. 4 , if the position and orientation calculation unit 312 calculates at least one of the position and orientation of the target object OBJ by performing matching processing using the image data IMG_2D instead of performing 3D matching processing using the three-dimensional position data WSD, the robot control processing shown in FIG. 4 does not need to include step S2 of FIG. 4 .

[0166] Alternatively, in step S3 of FIG. 4, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ by performing a matching process using both the three-dimensional position data WSD and the image data IMG_2D, in addition to or instead of performing at least one of a 3D matching process using the three-dimensional position data WSD and a 2D matching process using the image data IMG_2D.

[0167] For example, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ in the global coordinate system from a portion of the position and orientation of the target object OBJ in the 2D imaging coordinate system calculated based on the image data IMG_2D and a portion of the position and orientation of the target object OBJ in the 3D imaging coordinate system calculated based on the image data IMG_3D. For example, the position and orientation calculation unit 312 may set at least one of the initial position and initial orientation of a three-dimensional model (template model TM3) used when performing the 3D matching process based on at least one of the position and orientation of the target object OBJ in the 2D imaging coordinate system calculated based on the image data IMG_2D, calculate at least one of the position and orientation of the target object OBJ in the 3D imaging coordinate system by performing the 3D matching process using the three-dimensional model arranged at the set initial position and / or the set initial orientation, and calculate at least one of the position and orientation of the target object OBJ in the global coordinate system from at least one of the position and orientation of the target object OBJ in the 3D imaging coordinate system.

[0168] When calculating at least one of the position and orientation of the target object OBJ using both the image data IMG_2D and the three-dimensional position data WSD, the position and orientation calculation unit 312 may select, as the target object OBJ on which the end effector 4 should perform a predetermined process, a target object OBJ that satisfies a condition that the sum or product of the matching similarity calculated by the 2D matching process (hereinafter referred to as 2D matching similarity) and the matching similarity calculated by the 3D matching process (hereinafter referred to as 3D matching similarity) is maximized. The position and orientation calculation unit 312 may select, as the target object OBJ on which the end effector 4 should perform a predetermined process, a target object OBJ that satisfies a condition that the 2D matching similarity exceeds a matching determination threshold and is maximized. The position and orientation calculation unit 312 may select, as the target object OBJ on which the end effector 4 should perform a predetermined process, a target object OBJ that satisfies a condition that the 3D matching similarity exceeds a matching determination threshold and is maximized. The position and orientation calculation unit 312 may select a target object OBJ that satisfies the condition that both the 2D matching similarity and the 3D matching similarity exceed the matching determination threshold as a target object OBJ on which the end effector 4 should perform a predetermined process. The position and orientation calculation unit 312 may select a target object OBJ that satisfies the condition that both the 2D matching similarity and the 3D matching similarity exceed the matching determination threshold and is closest to the end effector 4 as a target object OBJ on which the end effector 4 should perform a predetermined process. The position and orientation calculation unit 312 may select a target object OBJ that satisfies the condition that both the 2D matching similarity and the 3D matching similarity exceed the matching determination threshold and has the largest Z coordinate along the Z axis (is located at the highest position) as a target object OBJ on which the end effector 4 should perform a predetermined process.

[0169] Note that if the position and orientation calculation unit 312 performs matching processing using the three-dimensional position data WSD but does not perform matching processing using the image data IMG_2D, the control device 3 does not need to acquire the image data IMG_2D in step S1. If the control device 3 does not acquire the image data IMG_2D, the imaging device 21 does not need to capture an image of the target object OBJ. If the control device 3 does not acquire the image data IMG_2D, the imaging system 2 does not need to include the imaging device 21.

[0170] If the position and orientation calculation unit 312 performs matching processing using the image data IMG_2D but does not perform matching processing using the three-dimensional position data WSD, the control device 3 does not need to acquire the image data IMG_3D in step S1. If the control device 3 does not acquire the image data IMG_3D, the imaging device 22 does not need to capture an image of the target object OBJ. If the control device 3 does not acquire the image data IMG_3D, the imaging system 2 does not need to include the imaging device 22. If the control device 3 does not acquire the image data IMG_3D, the control device 3 does not need to include the three-dimensional position data generation unit 311.

[0171] For simplicity of explanation, the following description will mainly focus on an example in which the position and orientation calculation unit 312 performs 3D matching processing in step S3 of FIG.

[0172] Thereafter, the signal generator 313 generates a robot control signal using the position and orientation data POI generated in step S3 (step S4). For example, the signal generator 313 may generate a robot control signal so that the end effector 4 can perform a predetermined process on the target object OBJ. For example, the signal generator 313 may generate a robot control signal so that the end effector 4 approaches the target object OBJ in order for the end effector 4 to perform the predetermined process on the target object OBJ. For example, the signal generator 313 may generate a robot control signal so that the positional relationship between the end effector 4 and the target object OBJ becomes a desired positional relationship. For example, the signal generator 313 may generate a robot control signal to control the operation of the robot arm 12 so that the positional relationship between the end effector 4 and the target object OBJ becomes a desired positional relationship. For example, the signal generator 313 may generate a robot control signal so that the end effector 4 performs a predetermined process on the target object OBJ when the positional relationship between the end effector 4 and the target object OBJ becomes the desired positional relationship. For example, the signal generating unit 313 may generate a robot control signal for controlling the operation of the end effector 4 so that a predetermined process is performed on the target object OBJ when the positional relationship between the end effector 4 and the target object OBJ becomes a desired positional relationship. As already described above, the robot control signal for controlling the operation of the end effector 4 may also be referred to as an end effector control signal.

[0173] The signal generator 313 may use the position and orientation data POI generated in step S3 to generate a robot control signal to perform feedback control of the robot 1. For example, the signal generator 313 may use the position and orientation data POI to generate a robot control signal to perform feedback control including P (Proportional) control. For example, the signal generator 313 may use the position and orientation data POI to generate a robot control signal to perform feedback control including PI (Proportional-Integral) control. For example, the signal generator 313 may use the position and orientation data POI to generate a robot control signal to perform feedback control including PID (Proportional-Integral-Differential) control.

[0174] As an example, each of FIGS. 6( a) to 6(d) is a side view showing the positional relationship between the robot 1 and the workpiece W at a certain point during a holding process for holding the workpiece W (i.e., an example of the target object OBJ) placed on a placement device T#1, such as an AGV (Automatic Guided Vehicle), moving on a support surface S. In this case, the control device 3 may generate position and orientation data POI indicating at least one of the position and orientation of the workpiece W (i.e., an example of the target object OBJ in the holding process) that is the target of the holding process, and generate a robot control signal using the generated position and orientation data POI. For example, as shown in FIG. 6(a), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 moves toward the space directly above the moving workpiece W. As an example, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 is positioned (in other words, approaches) in a desired orientation at the sequentially updated movement target point while sequentially updating the movement target point of the end effector 4 in accordance with the moving workpiece W based on at least one of the position and orientation of the workpiece W sequentially calculated by the position and orientation calculation unit 312. As shown in FIG. 6( b), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 located directly above the moving workpiece W approaches the workpiece W until it can hold the workpiece W while remaining positioned directly above the workpiece W. As shown in FIG. 6( c), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4 located in a position capable of holding the moving workpiece W holds the workpiece W while following the moving workpiece W. As shown in Figure 6 (d), the signal generating unit 313 may generate a robot control signal to control the operation of the robot arm 12 and the end effector 4 so that the end effector 4 holding the workpiece W moves away from the moving mounting device T#1 while holding the workpiece W.

[0175] When the robot 1 holds the workpiece W placed on the placement device T#1 using the end effector 4, the robot 1 may perform a release process to release the held workpiece W so that the held workpiece W is placed on a placement device T#2 (i.e., an example of the target object OBJ in the release process) different from the placement device T#1. That is, the robot 1 may perform a placement process to place the workpiece W on the placement device T#2 by successively performing the holding process and the release process. In this case, the control device 3 may generate position and orientation data POI indicating at least one of the position and orientation of the placement device T#2 that is the target of the release process, and generate a robot control signal using the generated position and orientation data POI. For example, each of FIGS. 7( a ) to 7 ( d ) is a side view showing the positional relationship between the robot 1 and the workpiece W at a certain point during a release process to place the workpiece W on the placement device T#2 moving on the support surface S. 7A, the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 holding the workpiece W moves to the space directly above the moving mounting device T#2 while still holding the workpiece W. As an example, the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 is positioned in a desired orientation (in other words, moves closer to) the sequentially updated movement target point while sequentially updating the movement target point in accordance with the moving mounting device T#2 based on at least one of the position and orientation of the mounting device T#2 sequentially calculated by the position and orientation calculating unit 312. As shown in Figure 7 (b), the signal generating unit 313 may generate a robot control signal to control the operation of the robot arm 12 so that the end effector 4, which is located directly above the moving mounting device T#2 and holding the workpiece W, approaches the mounting device T#2 while remaining located directly above the mounting device T#2 and holding the workpiece W, until the workpiece W can be placed on the mounting device T#2.7(c), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4, which is located in a position where it can place the workpiece W on the placement device T#2, places the workpiece W on the placement device T#2 while following the moving placement device T#2 (i.e., releases the workpiece W that it was holding). As shown in FIG. 7(d), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4, which has finished placing the workpiece W on the placement device T#2, moves away from the placement device T#2.

[0176] When a release process is performed (particularly when the release process is performed to perform a placement process), the control device 3 may further generate position and orientation data POI indicating at least one of the position and orientation of the workpiece W that is a second target of the release process (i.e., an example of the target object OBJ in the release process), in addition to position and orientation data POI indicating at least one of the position and orientation of the placement device T#2 that is a first target of the release process (i.e., an example of the target object OBJ in the release process). That is, the control device 3 may generate position and orientation data POI indicating at least one of the position and orientation of the workpiece W that is not yet held by the end effector 4 before the end effector 4 holds the workpiece W, and may also generate position and orientation data POI indicating at least one of the position and orientation of the workpiece W that is held by the end effector 4 after the end effector 4 holds the workpiece W.

[0177] When a release process is performed, before releasing the workpiece W held by the end effector 4 through the release process, the control device 3 may generate a robot control signal using, in addition to the position and orientation data POI regarding the mounting device T#2, the position and orientation data POI indicating at least one of the position and orientation of the workpiece W held by the end effector 4. In other words, the control device 3 may generate a robot control signal using the position and orientation data POI indicating at least one of the position and orientation of the workpiece W held by the end effector 4 during at least a portion of the period during which the end effector 4 holds the workpiece W.

[0178] For example, the signal generating unit 313 may generate a robot control signal for controlling the robot arm 12 to move the workpiece W held by the end effector 4 to a desired position (e.g., a position where the workpiece W should be released). In this case, compared to when the position and orientation data POI related to the workpiece W is not used, the robot 1 can appropriately place the workpiece W held by the end effector 4 at a desired position on the mounting device T (the robot 1 can execute the placement process). This is because the position of the workpiece W held by the end effector 4 becomes known information for the control device 3. If the position of the workpiece W is not known information, a technical problem may arise in that the workpiece W collides with the mounting device T. Alternatively, if the position of the workpiece W is not known information, the above-described placement process may result in a technical problem in that the workpiece W is placed at an unintended position on the mounting device T (or another object). Alternatively, if the position of the workpiece W is not known information, the above-described fitting process may result in a technical problem in that the workpiece W is fitted into an unintended position on another object. Alternatively, if the position of the workpiece W is not known information, the above-described pasting process may result in a technical problem in which the workpiece W is pasted in an unintended position on another object. Alternatively, if the position of the workpiece W is not known information, the above-described bonding process may result in a technical problem in which the workpiece W is adhered in an unintended position on another object. Alternatively, if the position of the workpiece W is not known information, the above-described welding process may result in a technical problem in which the workpiece W is welded to an unintended position on another object. However, in this embodiment, because the position of the workpiece W is known information, the possibility of such technical problems occurring is low or none.

[0179] Furthermore, for example, the signal generating unit 313 may generate a robot control signal for controlling the end effector 4 to change the posture of the workpiece W held by the end effector 4 to a desired posture. In this case, compared to when the position and posture data POI related to the workpiece W is not used, the robot 1 can use the end effector 4 to place the workpiece W in a desired posture on the mounting device T (the mounting process can be executed). This is because the posture of the workpiece W held by the end effector 4 becomes known information to the control device 3. If the posture of the workpiece W is not the desired posture, a technical problem may occur in that the workpiece W collides with the mounting device T. Alternatively, if the posture of the workpiece W is not the desired posture, a technical problem may occur in that the workpiece W is placed on the mounting device T (or another object) in an unintended posture as a result of the above-described mounting process. Alternatively, if the posture of the workpiece W is not the desired posture, a technical problem may occur in that the workpiece W is fitted into another object in an unintended posture as a result of the above-described fitting process. Alternatively, if the orientation of the workpiece W is not the desired orientation, the above-described attaching process may result in a technical problem in which the workpiece W is attached to another object in an unintended orientation. Alternatively, if the orientation of the workpiece W is not the desired orientation, the above-described bonding process may result in a technical problem in which the workpiece W is attached to another object in an unintended orientation. Alternatively, if the orientation of the workpiece W is not the desired orientation, the above-described welding process may result in a technical problem in which the workpiece W is welded to another object in an unintended orientation. However, in this embodiment, because the robot 1 can be controlled so that the orientation of the workpiece W is the desired orientation, the possibility of such technical problems occurring is low or none.

[0180] In addition, not only when a release process is performed, but also in any situation where the end effector 4 is holding the workpiece W, the control device 3 may generate a robot control signal using position and orientation data POI that indicates at least one of the position and orientation of the workpiece W held by the end effector 4.

[0181] At least one of the placement device T#1 shown in FIGS. 6( a) to 6(d) and the placement device T#2 shown in FIGS. 7( a) to 7(d) does not necessarily have to move on the support surface S. For example, FIGS. 8( a) and 8(b) are side views showing the positional relationship between the robot 1 and the workpiece W at a certain point during a holding process for holding the workpiece W placed on the placement device T#1 that is stationary on the support surface S. In this case, as shown in FIG. 8( a), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 approaches the stationary workpiece W until it can hold the stationary workpiece W. As an example, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 based on at least one of the position and orientation of the workpiece W calculated by the position and orientation calculation unit 312 so that the end effector 4 is positioned in a desired orientation (in other words, approaches) at a position where the end effector 4 can hold the workpiece W, at least one of the position and orientation of which has been calculated. As shown in FIG. 8( b), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4, which is positioned to hold the stationary workpiece W, holds the workpiece W. Furthermore, each of FIGS. 8( c) to 8( e) is a side view showing the positional relationship between the robot 1 and the workpiece W at a certain point during a period in which a release process is being performed to place the workpiece W on the placement device T#2 that is stationary on the support surface S. In this case, as shown in FIG. 8( c), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4, which is holding the workpiece W, approaches the placement device T#2 while still holding the workpiece W until it is able to place the workpiece W on the placement device T#2.8(d), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4, which is positioned at a position where it can place the workpiece W on the placement device T#2, places the workpiece W on the stationary placement device T#2 (i.e., releases the workpiece W that it was holding). As shown in FIG. 8(e), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4, which has finished placing the workpiece W on the placement device T#2, moves away from the placement device T#2.

[0182] A plurality of workpieces W may be placed on the placement device T#1. For example, the plurality of workpieces W may be placed on the placement device T#1 so that they are arranged according to a certain standard on the placement device T#1. For example, the plurality of workpieces W may be placed on the placement device T#1 so that they are randomly stacked on the placement device T#1. In this case, the robot 1 may perform a holding process to selectively hold a desired workpiece W from the plurality of workpieces W placed on the placement device T#1. In particular, the robot 1 may perform a holding process to hold the plurality of workpieces W placed on the placement device T#1 one by one in order.

[0183] Furthermore, the robot 1 may perform a release process to place multiple workpieces W on the placement device T#2. That is, the robot 1 may perform a placement process to place multiple workpieces W placed on the placement device T#1 one by one on the placement device T#2 (or another placement device) by performing a holding process and a release process. In this case, the robot 1 may perform a release process to place multiple workpieces W one by one on the placement device T#2 so that the multiple workpieces W are lined up according to a certain standard on the placement device T#2. The robot 1 may perform a release process to place multiple workpieces W one by one on the placement device T#2 so that the multiple workpieces W are randomly stacked on the placement device T#2.

[0184] 9(a) to 9(e) and 10(a) to 10(e) are side views showing the positional relationship between the robot 1 and the workpiece W at a certain point during a period in which a holding process for holding two workpieces W#1 and W#2 placed on the placement device T#1 in order one by one and a release process for placing the two workpieces W#1 and W#2 in order one by one on the placement device T#2 are being performed. In this case, as shown in FIG. 9(a), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 approaches workpiece W#2 until it is able to hold either one of workpieces W#1 or W#2 (workpiece W#2 in the example shown in FIG. 9(a)). As an example, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 is positioned (in other words, approaches) in a desired orientation at a position where the end effector 4 can hold the workpiece W#2, whose position and orientation have been calculated, based on at least one of the position and orientation of the workpiece W#2 calculated by the position and orientation calculation unit 312. Thereafter, as shown in FIG. 9B, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4, positioned at a position where it can hold the workpiece W#2, holds the workpiece W#2. Thereafter, as shown in FIG. 9C, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 holding the workpiece W#2 approaches the placement device T#2 until it can place the workpiece W#2 on the placement device T#2, while still holding the workpiece W#2. As an example, the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 based on at least one of the position and posture of the mounting device T#2 calculated by the position and posture calculating unit 312 so that the end effector 4 is positioned in the desired posture (in other words, approaches) at a position where the end effector 4 can place the workpiece W#2 on the mounting device T#2, at least one of the position and posture of which has been calculated.9(d), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4, which is positioned to place the workpiece W#2 on the placement device T#2, places the workpiece W#2 on the placement device T#2 (i.e., releases the workpiece W#2 that it was holding). Then, as shown in FIG. 9(e), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4, which has placed the workpiece W#2 on the placement device T#2, moves away from the placement device T#2. Then, as shown in FIG. 10(a), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 approaches the remaining one of the workpieces W#1 and W#2 (workpiece W#1 in the example shown in FIG. 10(a)). As an example, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 is positioned (in other words, approaches) in a desired orientation at a position where the end effector 4 can hold the workpiece W#1, whose position and orientation have been calculated, based on at least one of the position and orientation of the workpiece W#1 calculated by the position and orientation calculation unit 312. Thereafter, as shown in FIG. 10( b), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4, positioned at a position where it can hold the workpiece W#1, holds the workpiece W#1. Thereafter, as shown in FIG. 10( c), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4, holding the workpiece W#1, approaches the placement device T#2 until it can place the workpiece W#1 on the placement device T#2, while still holding the workpiece W#1.As an example, the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4 is positioned in a desired orientation at a position where the end effector 4 can place the workpiece W#12 on the placement device T#2, at least one of the position and orientation of which has been calculated, based on at least one of the position and orientation of the placement device T#2 calculated by the position and orientation calculation unit 312. Thereafter, as shown in FIG. 10( d ), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 4 so that the end effector 4, which is positioned at a position where the workpiece W#1 can be placed on the placement device T#2, places the workpiece W#1 on the placement device T#2 (i.e., releases the workpiece W#1 that it was holding). Thereafter, as shown in FIG. 10( e ), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 4, which has placed the workpiece W#1 on the placement device T#2, moves away from the placement device T#2.

[0185] The robot 1 may perform a holding process to hold, one by one, a plurality of workpieces W placed on a placement device T#1 that moves on the support surface S. Alternatively, the robot 1 may perform a holding process to hold, one by one, a plurality of workpieces W placed on a placement device T#1 that is stationary on the support surface S. The robot 1 may perform a release process to place, one by one, a plurality of workpieces W on a placement device T#2 that moves on the support surface S. The robot 1 may perform a release process to place, one by one, a plurality of workpieces W on a placement device T#2 that is stationary on the support surface S.

[0186] 4, the signal generator 313 outputs the robot control signal generated in step S4 to the robot 1 (particularly, the robot control device 13) using the communication device 33. As a result, the robot control device 13 controls at least one of the operation of the robot 1 (for example, the operation of the robot arm 12) and the operation of the end effector 4 based on the robot control signal.

[0187] Thereafter, the control device 3 repeats the series of processes from step S1 to step S4 until it is determined to end the robot control process (step S5). That is, the control device 3 continues to acquire at least one of the image data IMG_2D and IMG_3D from the imaging devices 21 and 22, even during the period when the operation of at least one of the robot arm 12 and the end effector 4 is being controlled based on the robot control signal. For example, when the end effector 4 performs the holding process as shown in FIGS. 6( a) to 6(d), the control device 3 may repeat the series of processes from step S1 to step S4 until the end effector 4 holds the workpiece W (and further until the end effector 4 holding the workpiece W is separated from the placement device T#1). For example, when the end effector 4 performs a release process as shown in Figures 7(a) to 7(d), the control device 3 may repeat the series of processes from step S1 to step S4 until the end effector 4 places the workpiece W on the placement device T#2 (and further, until the end effector 4, having finished placing the workpiece W on the placement device T#2, leaves the placement device T#2).

[0188] In this case, because the operation of at least one of the robot arm 12 and the end effector 4 is controlled based on the robot control signal as described above, each of the imaging devices 21 and 22 may capture an image of the target object OBJ during relative movement between the target object OBJ and the imaging devices 21 and 22. For example, each of the imaging devices 21 and 22 may capture an image of the target object OBJ while the target object OBJ is stationary and the imaging devices 21 and 22 are moving. For example, each of the imaging devices 21 and 22 may capture an image of the target object OBJ while the target object OBJ is moving and the imaging devices 21 and 22 are stationary. For example, each of the imaging devices 21 and 22 may capture an image of the target object OBJ while the target object OBJ is moving and the imaging devices 21 and 22 are moving. 4 (or may repeat the robot control process shown in FIG. 4 ) during the relative movement between the target object OBJ and the image capturing devices 21 and 22 (i.e., while at least one of the image capturing devices 21 and 22 and the target object OBJ is moving). As a result, even during the period in which the operation of the robot 1 is being controlled based on the robot control signal, the control device 3 can newly generate (i.e., update) position and orientation data POI indicating at least one of the position and orientation of the target object OBJ based on newly acquired image data IMG_2D and IMG_3D.

[0189] However, each of the imaging devices 21 and 22 may capture an image of the target object OBJ while the target object OBJ is stationary and the imaging devices 21 and 22 are stationary. The control device 3 may perform the robot control process shown in Fig. 4 while the imaging devices 21 and 22 and the target object OBJ are stationary. Note that the control device 3 may repeat the robot control process shown in Fig. 4 while the imaging devices 21 and 22 and the target object OBJ are stationary.

[0190] As described above, the position and orientation calculation unit 312 does not have to calculate at least one of the position and orientation of the target object OBJ in the global coordinate system in step S3. In this case, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the target object OBJ in a coordinate system different from the global coordinate system (e.g., a robot coordinate system, a 2D imaging coordinate system, or a 3D imaging coordinate system) in step S3 of FIG. 4 . That is, the position and orientation calculation unit 312 may generate position and orientation data POI indicating at least one of the position and orientation of the target object OBJ in a coordinate system different from the global coordinate system (e.g., a robot coordinate system, a 2D imaging coordinate system, or a 3D imaging coordinate system). In this case, the signal generation unit 313 may generate a robot control signal in step S4 using the position and orientation data POI indicating at least one of the position and orientation of the target object OBJ in the coordinate system different from the global coordinate system calculated in step S3.

[0191] (4) Modifications Next, modifications of the robot system SYS will be described. Note that each modification described below can be combined with other modifications. Alternatively, detailed descriptions of components that have already been described will be omitted in the following description. Furthermore, detailed descriptions of processes that have already been described will be omitted in the following description. In other words, the following description will particularly describe components that are different from components that have already been described and processes that are different from processes that have already been described. Therefore, unless otherwise specified, the robot system SYS may also be provided with components that have already been described, and the robot system SYS may perform processes that have already been described, even in the modifications described below.

[0192] (4-1) First Modification As described above, the position and orientation calculation unit 312 performs 3D matching processing (in other words, template matching processing) on ​​the three-dimensional position data WSD using the three-dimensional model data WMD to generate the position and orientation data POI. Specifically, the position and orientation calculation unit 312 performs 3D matching processing (in other words, template matching processing) on ​​the three-dimensional position data WSD using the template model TM3 (three-dimensional model) indicated by the three-dimensional model data WMD as a template to generate the position and orientation data POI. In the first modification, in addition to or instead of performing 3D matching processing using the entire template model TM3 (three-dimensional model) indicated by the three-dimensional model data WMD as a template, the position and orientation calculation unit 312 may perform 3D matching processing using a part of the template model TM3 (three-dimensional model) indicated by the three-dimensional model data WMD as a template. That is, the position and orientation calculation unit 312 may perform 3D matching processing using partial model data, which is three-dimensional model data indicating a portion of the template model TM3 (three-dimensional model). In the following description, a template model corresponding to the entire template model TM3 indicated by the three-dimensional model data WMD will be referred to as an entire model TM3_W. That is, in the following description, the template model TM3 (three-dimensional model) itself will be referred to as an entire model TM3_W. Meanwhile, in the following description, a template model corresponding to a portion of the template model TM3 indicated by the three-dimensional model data WMD will be referred to as a partial model TM3_P. In this case, the partial model TM3_P may be considered to be a portion of the entire model TM3_W. A template model corresponding to a portion of the template model TM3 may be referred to as a three-dimensional partial model.

[0193] The control device 3 (particularly, the arithmetic device 31) may generate the partial model TM3_P. For example, the control device 3 (particularly, the arithmetic device 31) may extract at least a portion of the overall model TM3_W from the overall model TM3_W, thereby generating a partial model TM3_P corresponding to at least a portion of the extracted overall model TM3_W. However, a device different from the control device 3 (for example, a arithmetic device different from the arithmetic device 31) may generate the partial model TM3_P.

[0194] As an example, the control device 3 may generate the partial model TM3_P based on an instruction from a user. For example, the control device 3 may generate the partial model TM3_P based on an instruction from a user to extract at least a part of the overall model TM3_W as the partial model TM3_P. In other words, the control device 3 may generate the partial model TM3_P based on an instruction from a user to generate the partial model TM3_P.

[0195] The user may use the input device 34 to input model generation information related to the user's instructions for generating the partial model TM3_P to the control device 3. For example, the control device 3 may display a model generation screen 351, which the user can operate to generate the partial model TM3_P, on the output device 35, which can function as a display device. An example of the model generation screen 351 is shown in FIGS. 11( a) and 11(b). As shown in FIGS. 11(a) and 11(b), the model generation screen 351 may display the overall model TM3_W. The user may input instructions (model generation information) on the model generation screen 351 to the control device 3 for generating the partial model TM3_P using the overall model TM3_W displayed on the model generation screen 351.

[0196] As shown in FIG. 11( b), for example, the user may input an instruction to the control device 3 to select at least a portion of the overall model TM3_W displayed on the model generation screen 351 as a selected model TM3_S. Note that in FIG. 11( b), the selected model TM3_S selected by the user from the overall model TM3_W is indicated by a solid line, and model portions of the overall model TM3_W not selected by the user are indicated by dashed lines. Furthermore, the user may input an instruction to the control device 3 to designate the selected model TM3_S selected by the user as a model portion to be included in the partial model TM3_P. In this case, the control device 3 may extract the model portion selected by the user from the overall model TM3_W from the overall model TM3_W, thereby generating a partial model TM3_P including at least a portion of the extracted overall model TM3_W. Alternatively, the user may input an instruction to the control device 3 to designate the selected model TM3_S selected by the user as a model portion that should not be included in the partial model TM3_P. In this case, the control device 3 may extract from the overall model TM3_W a model portion of the overall model TM3_W that has not been selected by the user, and generate a partial model TM3_P that includes at least a portion of the extracted overall model TM3_W.

[0197] As another example, the control device 3 may generate the partial model TM3_P without using instructions from the user. For example, the control device 3 may generate the partial model TM3_P based on simulation image data. The simulation image data may be generated by assuming that the imaging system 2 captures an image of the target object OBJ. For example, image data that is expected to be generated by the imaging system 2 when the imaging system 2 captures an image of the target object OBJ may be generated as the simulation image data. The simulation image data used to generate the partial model TM3_P used in the 3D matching process may be, for example, image data in the same format as the image data IMG_3D (e.g., stereo image data). Then, the control device 3 may generate the partial model TM3_P by selecting from the whole model TM3_W based on the simulation image data (e.g., using simulation point cloud data calculated from the simulation image data) a partial model TM3_P that satisfies the condition that "the target object OBJ is accurately detected by the matching process using the partial model TM3_P (as a result, at least one of the position and orientation of the target object OBJ is accurately calculated)."

[0198] After generating the partial model TM3_P, the control device 3 may perform a 3D matching process using the three-dimensional position data WSD and the partial model TM3_P in addition to or instead of performing a 3D matching process using the three-dimensional position data WSD and the overall model TM3_W.

[0199] As a first example, the control device 3 (particularly, the position and orientation calculation unit 312) may first perform 3D matching processing using the entire model TM3_W. Then, the control device 3 may determine whether a calculation result condition is satisfied regarding the result of the position and orientation calculation (position and orientation calculation processing) including the 3D matching processing using the entire model TM3_W. If it is determined that the calculation result condition is satisfied, the control device 3 may generate a robot control signal based on the result of the 3D matching processing using the entire model TM3_W (i.e., the position and orientation data POI). On the other hand, if it is determined that the calculation result condition is not satisfied, the control device 3 (particularly, the position and orientation calculation unit 312) may further perform 3D matching processing using the partial model TM3_P. Then, the control device 3 may determine whether a calculation result condition is satisfied regarding the result of the position and orientation calculation (position and orientation calculation processing) including the 3D matching processing using the partial model TM3_P. If it is determined that the calculation result condition is met, the control device 3 may generate a robot control signal based on the result of the 3D matching process using the partial model TM3_P (i.e., the position and orientation data POI).

[0200] As a second example, the control device 3 (particularly, the position and orientation calculation unit 312) may first perform 3D matching processing using the partial model TM3_P. The control device 3 may then determine whether a calculation result condition related to the result of the position and orientation calculation, including the 3D matching processing using the partial model TM3_P, is satisfied. If it is determined that the calculation result condition is satisfied, the control device 3 may generate a robot control signal based on the result of the 3D matching processing using the partial model TM3_P (i.e., the position and orientation data POI). On the other hand, if it is determined that the calculation result condition is not satisfied, the control device 3 (particularly, the position and orientation calculation unit 312) may further perform 3D matching processing using the entire model TM3_W. The control device 3 may then determine whether a calculation result condition related to the result of the position and orientation calculation, including the 3D matching processing using the entire model TM3_W, is satisfied. If it is determined that the calculation result condition is met, the control device 3 may generate a robot control signal based on the result of the 3D matching process using the overall model TM3_W (i.e., the position and orientation data POI).

[0201] As a third example, the control device 3 (particularly the position and orientation calculation unit 312) may perform both a 3D matching process using the partial model TM3_P and a 3D matching process using the entire model TM3_W. The control device 3 may then generate position and orientation data POI by comparing the matching similarity calculated in the 3D matching process using the partial model TM3_P with a matching judgment threshold, and by comparing the matching similarity calculated in the 3D matching process using the entire model TM3_W with the matching judgment threshold. For example, the control device 3 may select, as the processing execution object, a target object OBJ whose matching similarity calculated in the 3D matching process using the partial model TM3_P exceeds the matching judgment threshold and whose matching similarity calculated in the 3D matching process using the entire model TM3_W exceeds the matching judgment threshold, and generate position and orientation data POI for the target object OBJ. The control device 3 may then generate a robot control signal based on the generated position and orientation data POI.

[0202] In at least one of the first to third examples, the matching judgment threshold used in the position and orientation calculation including the 3D matching process using the entire model TM3_W may be the same as the matching judgment threshold used in the position and orientation calculation including the 3D matching process using the partial model TM3_P. Alternatively, the matching judgment threshold used in the position and orientation calculation including the 3D matching process using the entire model TM3_W may be different from the matching judgment threshold used in the position and orientation calculation including the 3D matching process using the partial model TM3_P. For example, the matching judgment threshold used in the position and orientation calculation including the 3D matching process using the entire model TM3_W may be larger than the matching judgment threshold used in the position and orientation calculation including the 3D matching process using the partial model TM3_P. For example, the matching judgment threshold used in the position and orientation calculation including the 3D matching process using the entire model TM3_W may be smaller than the matching judgment threshold used in the position and orientation calculation including the 3D matching process using the partial model TM3_P.

[0203] As a fourth example, the control device 3 (particularly, the position and orientation calculation unit 312) may perform a 3D matching process using a first partial model TM3_P representing a first model portion of the overall model TM3_W (three-dimensional model) before performing a 3D matching process using the overall model TM3_W, after performing a 3D matching process using the overall model TM3_W, or without performing a 3D matching process using the overall model TM3_W. The control device 3 may then determine whether a calculation result condition related to the result of the position and orientation calculation including the 3D matching process using the first partial model TM3_P is satisfied. If it is determined that the calculation result condition is satisfied, the control device 3 may generate a robot control signal based on the result of the 3D matching process using the first partial model TM3_P (i.e., the position and orientation data POI). On the other hand, if it is determined that the calculation result condition is satisfied, the control device 3 (particularly, the position and orientation calculation unit 312) may further perform a 3D matching process using a second partial model TM3_P different from the first partial model TM3_P. The second partial model TM3_P differs from the first partial model TM3_P, which indicates a first model portion of the overall model TM3_W, in that the second partial model TM3_P indicates a second model portion of the overall model TM3_W that is at least partially different from the first model portion. The control device 3 may then determine whether a calculation result condition regarding the result of the position and orientation calculation, including the 3D matching process using the second partial model TM3_P, is satisfied. If it is determined that the calculation result condition is satisfied, the control device 3 may generate a robot control signal based on the result of the 3D matching process using the second partial model TM3_P (i.e., the position and orientation data POI). On the other hand, if it is determined that the calculation result condition is not satisfied, the control device 3 (particularly the position and orientation calculation unit 312) may further perform a 3D matching process using a third partial model TM3_P that is different from the first and second partial models TM3_P. The third partial model TM3_P differs from the first and second model portions in that the third partial model TM3_P indicates a third model portion of the overall model TM3_W that is at least partially different from each of the first and second model portions. Thereafter, the same operation may be repeated until it is determined that the calculation result condition is met.

[0204] As described above, in the fourth example, a plurality of partial models TM3_P are used. Therefore, the control device 3 may generate a plurality of partial models TM3_P that are different from one another. In other words, the control device 3 may generate a plurality of partial models TM3_P that are a plurality of three-dimensional models that are at least partially different from one another. An example of a plurality of partial models TM3_P is shown in FIG. 12. In the example shown in FIG. 12, the control device 3 generates three partial models TM3_P (specifically, partial model TM3_P#1, partial model TM3_P#2, and partial model TM3_P#3). Note that the control device 3 may generate two partial models TM3_P that are different from one another, or four or more partial models TM3_P that are different from one another.

[0205] For example, as in the partial model TM3_P#1 shown in FIG. 12 , the control device 3 may generate a partial model TM3_P by excluding at least one curved portion (curved portion) from the overall model TM3_W. In this case, as described above, the control device 3 may generate the partial model TM3_P based on instructions from the user. As another example, the control device 3 may generate the partial model TM3_P by distinguishing between the curved portion (curved portion) and other portions (e.g., flat portions) of the overall model TM3_W without using instructions from the user and excluding the curved portion (curved portion) from the overall model TM3_W. Furthermore, without being limited to the partial model TM3_P shown in FIG. 12 , the control device 3 may generate as the partial model TM3_P a model in which at least a portion of a model of the overall model TM3_W has been thinned. For example, the control device 3 may generate as the partial model TM3_P a model in which at least a portion of a model of the overall model TM3_W has been thinned. A model in which at least a portion of the overall model TM3_W has been thinned can be said to be a partial model TM3_P because it is a model of at least a portion of the overall model TM3_W. In this case, the control device 3 may generate the partial model TM3_P based on instructions from the user, or the control device 3 may generate the partial model TM3_P without using instructions from the user.

[0206] Note that the position and orientation calculation in this embodiment is a calculation for generating position and orientation data POI. That is, the position and orientation calculation in this embodiment is a calculation for calculating at least one of the position and orientation of the target object OBJ. When the position and orientation data POI is generated by performing 3D matching processing as described above, the position and orientation calculation is a calculation including the 3D matching processing. When the position and orientation data POI is generated by performing 2D matching processing as described above, the position and orientation calculation is a calculation including the 2D matching processing.

[0207] In this embodiment, an example will be shown in which a condition (good calculation result condition) that the result of the position and orientation calculation (position and orientation calculation processing) is good is used as the calculation result condition. In this case, the control device 3 may determine whether the calculation result condition (good calculation result condition) is satisfied by determining whether the result of the position and orientation calculation is good. The signal generation unit 313 may determine that the calculation result condition (good calculation result condition) is satisfied when the result of the position and orientation calculation is good. The signal generation unit 313 may determine that the calculation result condition (good calculation result condition) is not satisfied when the result of the position and orientation calculation is not good (i.e., the result of the position and orientation calculation is poor).

[0208] The control device 3 may determine whether the result of the position and orientation calculation is satisfactory by determining whether a target object OBJ exists for which the end effector 4 can perform a predetermined process using the result of the position and orientation calculation. In addition to or instead of determining whether a target object OBJ exists for which the end effector 4 can perform a predetermined process using the result of the position and orientation calculation, the signal generation unit 313 may determine whether the result of the position and orientation calculation is satisfactory by determining whether a target object OBJ exists for which the end effector 4 cannot perform a predetermined process using the result of the position and orientation calculation. The reason for this will be explained below.

[0209] As described above, since the robot control signal is generated using the result of the position and orientation calculation (i.e., information including the result of the matching process, for example, information including the position and orientation data POI), if the result of the position and orientation calculation is not good, it is assumed that the control device 3 is unable to generate an appropriate robot control signal. As a result, it is assumed that the end effector 4 may not be able to perform a predetermined process on the target object OBJ because an appropriate robot control signal cannot be generated. In other words, it is assumed that there may be a target object OBJ for which the end effector 4 cannot perform a predetermined process using the result of the position and orientation calculation. Therefore, if the end effector 4 is unable to perform a predetermined process on the target object OBJ, it is likely that the result of the position and orientation calculation is not good. Therefore, the existence of a target object OBJ for which the end effector 4 cannot perform a predetermined process using the result of the position and orientation calculation may be used as one of the reasons for determining that the result of the position and orientation calculation is not good. On the other hand, if the result of the position and orientation calculation is good, it is assumed that the control device 3 is able to generate an appropriate robot control signal. As a result, it is assumed that there is a high possibility that the end effector 4 will be able to perform a predetermined process on the target object OBJ because an appropriate robot control signal can be generated. In other words, it is assumed that there is a high possibility that a target object OBJ exists on which the end effector 4 can perform a predetermined process using the result of the position and orientation calculation. Therefore, if the end effector 4 can perform a predetermined process on the target object OBJ, there is a high possibility that the result of the position and orientation calculation is good. Therefore, the existence of a target object OBJ on which the end effector 4 can perform a predetermined process using the result of the position and orientation calculation may be used as one of the reasons for determining that the result of the position and orientation calculation is good.

[0210] As a first example, the control device 3 may determine that the result of the position and orientation calculation is good when the result of the position and orientation calculation indicates that there is at least one target object OBJ on which the end effector 4 can perform a predetermined process. On the other hand, the control device 3 may determine that the result of the position and orientation calculation is bad when the result of the position and orientation calculation indicates that there is not a single target object OBJ on which the end effector 4 can perform a predetermined process.

[0211] As a second example, the control device 3 may determine that the result of the position and orientation calculation is not good when the result of the position and orientation calculation indicates that there is at least one target object OBJ for which the end effector 4 cannot perform a predetermined process. More specifically, the control device 3 may determine that the result of the position and orientation calculation is not good when the result of the position and orientation calculation indicates that the number of target objects OBJ for which the end effector 4 cannot perform a predetermined process is equal to or greater than a predetermined detection threshold. In other words, the control device 3 may determine that the result of the position and orientation calculation is not good when the result of the position and orientation calculation indicates that the number of target objects OBJ for which the end effector 4 can perform a predetermined process is below a predetermined detection threshold. On the other hand, the control device 3 may determine that the result of the position and orientation calculation is good when the result of the position and orientation calculation indicates that the number of target objects OBJ for which the end effector 4 cannot perform a predetermined process is below a predetermined detection threshold. In other words, the control device 3 may determine that the result of the position and orientation calculation is good when the result of the position and orientation calculation indicates that the number of target objects OBJ for which the end effector 4 can perform a predetermined process is equal to or greater than a predetermined detection threshold.

[0212] The predetermined detection threshold used in the second example may be set by a user, or may be automatically set by the control device 3. The first example described above may be considered equivalent to the second example in which the predetermined detection threshold is set to "1."

[0213] A first example of a target object OBJ for which the end effector 4 can perform a predetermined process is a target object OBJ for which position and orientation data POI have been calculated by position and orientation calculation. On the other hand, a first example of a target object OBJ for which the end effector 4 cannot perform a predetermined process is a target object OBJ for which position and orientation data POI have not been calculated by position and orientation calculation. In this case, the control device 3 may determine whether the calculation result condition is met by determining whether the position and orientation data POI of the target object OBJ has been calculated by position and orientation calculation.

[0214] A second example of a target object OBJ for which the end effector 4 can perform a predetermined process is a target object OBJ for which the matching similarity calculated by the position and orientation calculation exceeds the matching determination threshold. On the other hand, a second example of a target object OBJ for which the end effector 4 cannot perform a predetermined process is a target object OBJ for which the matching similarity calculated by the position and orientation calculation is below the matching determination threshold. In this case, the control device 3 may determine whether the calculation result condition is met by determining whether the position and orientation calculation has detected a target object OBJ for which the matching similarity exceeds the matching determination threshold. The control device 3 may determine whether the calculation result condition is met by determining whether the position and orientation calculation has calculated position and orientation data POI of the target object OBJ for which the matching similarity exceeds the matching determination threshold.

[0215] It is assumed that the higher the matching similarity of the target object OBJ, the higher the accuracy of the position and orientation data POI of the target object OBJ (i.e., the calculation accuracy of at least one of the position and orientation). Therefore, the position and orientation data POI of the target object OBJ whose matching similarity exceeds the matching determination threshold may be considered equivalent to position and orientation data POI whose accuracy is relatively high (specifically, above a predetermined accuracy threshold). On the other hand, the position and orientation data POI of the target object OBJ whose matching similarity is below the matching determination threshold may be considered equivalent to position and orientation data POI whose accuracy is relatively low (specifically, below a predetermined accuracy threshold). In this case, the matching judgment threshold may be set to a desired value that can distinguish, from the matching similarity, between a state in which the accuracy (in other words, reliability) of the position and orientation data POI of the target object OBJ is high enough that the end effector 4 will be able to successfully perform a specified processing on the target object OBJ, and a state in which the accuracy (in other words, reliability) of the position and orientation data POI of the target object OBJ is low enough that the end effector 4 may fail to perform a specified processing on the target object OBJ.

[0216] Note that if the control device 3 (particularly the position and orientation calculation unit 312) is unable to perform the position and orientation calculation normally in the first place, the result of the position and orientation calculation is unlikely to be good. Therefore, the control device 3 may determine whether the result of the position and orientation calculation is good by determining whether the position and orientation calculation unit 312 was able to perform the position and orientation calculation normally. Specifically, the control device 3 may determine that the result of the position and orientation calculation is good if the position and orientation calculation unit 312 was able to perform the position and orientation calculation normally. On the other hand, the control device 3 may determine that the result of the position and orientation calculation is not good if the position and orientation calculation unit 312 was unable to perform the position and orientation calculation normally. The state in which the position and orientation calculation unit 312 is unable to perform the position and orientation calculation normally may include a state in which the position and orientation calculation unit 312 is unable to start the position and orientation calculation normally. The state in which the position and orientation calculation unit 312 is unable to perform the position and orientation calculation normally may include a state in which the position and orientation calculation unit 312 is unable to end the position and orientation calculation normally. The state in which the position and orientation calculation unit 312 cannot normally perform the position and orientation calculation may include a state in which the position and orientation calculation started by the position and orientation calculation unit 312 ends abnormally.

[0217] If it is determined that the calculation result condition is not satisfied, the end effector 4 does not perform a predetermined process on the target object OBJ. Therefore, determining whether or not the calculation result condition is satisfied may include determining whether or not the end effector 4 performs a predetermined process on the target object OBJ. Determining that the calculation result condition is not satisfied may include determining that the end effector 4 does not perform a predetermined process on the target object OBJ.

[0218] In this embodiment, particularly when the three-dimensional position data WSD indicates the three-dimensional positions of multiple target objects OBJ (especially multiple target objects OBJ that at least partially overlap), the control device 3 may perform 3D matching processing using the partial model TM3_P. In this case, compared to when only the 3D matching processing using the entire model TM3_W is performed, the control device 3 is more likely to be able to properly detect (recognize) one target object OBJ on which the end effector 4 is to perform a predetermined process as a processing execution object. In other words, when multiple target objects OBJ (especially multiple target objects OBJ that at least partially overlap) are captured in the image indicated by the image data IMG_3D, the control device 3 is more likely to be able to properly detect (recognize) one target object OBJ on which the end effector 4 is to perform a predetermined process as a processing execution object.

[0219] As an example, FIG. 13( a) shows a U-shaped part, which is an example of a target object OBJ. In the example shown in FIG. 13( a), the target object OBJ includes a plate-shaped first portion OB1 and a plate-shaped second portion OB2 that faces the first portion OB1 and is larger than the first portion OB1. In this case, if an entire model TM3_W that shows both the three-dimensional shape of the first portion OB1 and the three-dimensional shape of the second portion OB2 is used, as shown in FIG. 13( b), the following technical problem may arise. Specifically, as shown in FIG. 13( c), an example will be described in which a portion of the second portion OB2 of a first target object OBJ#1 is hidden by a second target object OBJ#2 that is partially located below the first target object OBJ#1. Note that when the target object OBJ has a U-shape, the first target object OBJ#1 and the second target object OBJ#2 may fit together, causing a portion of the first target object OBJ#1 to be hidden by the second target object OBJ#2, a portion of which is located below the first target object OBJ#1. In this case, the position and orientation calculation unit 312 may detect an edge E_OB1 (see the thick line in FIG. 13C ) of the first portion OB1 of the first target object OBJ#1 that is located above the second target object OBJ#2 (particularly, the first portion OB1), rather than the edge of the second portion OB2 of the first target object OBJ#1 that is partially hidden by the second target object OBJ#2 (particularly, the first portion OB1). However, due to the low similarity between the detected edge E_OB1 and the edge E_TM3 of the entire model TM3_W (see the thick line in FIG. 13( b) ), the position and orientation calculation unit 312 may not be able to properly detect the first target object OBJ#1 (that is, may not be able to properly calculate at least one of the position and orientation of the first target object OBJ#1). In this case, as shown in FIG. 13( d) , when a model portion of the template model TM3 corresponding to the first portion OB1 is used as the partial model TM3_P, due to the high similarity between the detected edge E_OB1 and the edge E_TM3′ of the partial model TM3_P (see the thick line in FIG. 13( d) ), the position and orientation calculation unit 312 may properly detect the first target object OBJ#1 (that is, may properly calculate at least one of the position and orientation of the first target object OBJ#1).

[0220] Therefore, in the first modified example, the position and orientation calculation unit 312 is more likely to be able to appropriately detect the target object OBJ in the 3D matching process.

[0221] In this case, the possibility of unintentionally selecting a second target object OBJ#2, which is different from the first target object OBJ#1 that should have been selected as the object to be processed, is reduced. If the second target object OBJ#2 that should not have been selected as the object to be processed is unintentionally selected as the object to be processed, as the end effector 4 approaches the second target object OBJ#2 that should not have been selected as the object to be processed, the end effector 4 may collide with the first target object OBJ#1 that should have been selected as the object to be processed (e.g., the first target object OBJ#1 located above the second target object OBJ#2), possibly damaging the first target object OBJ#1. Alternatively, after the end effector 4 holds the second target object OBJ#2 that should not have been selected as the object to be processed, the first target object OBJ#1 located above the second target object OBJ#2 may be dragged and moved by the second target object OBJ#2. As a result, the first target object OBJ#1 that is not held by the end effector 4 may fall, and the fallen first target object OBJ#1 may damage the surrounding target objects OBJ. However, the first modified example is advantageous in that it reduces the possibility of such a technical problem occurring.

[0222] 12 , when multiple partial models TM3_P are generated, the control device 3 may select, from the multiple partial models TM3_P, one partial model TM3_P that is likely to be able to properly detect (recognize) one target object OBJ on which the end effector 4 is to perform a predetermined process, as a process execution object, taking into account the tendency of overlapping among the multiple target objects OBJ, and perform 3D matching processing using the selected one partial model TM3_P. As a result, compared to when a single partial model TM3_P is used, the control device 3 is more likely to be able to properly detect (recognize) one target object OBJ on which the end effector 4 is to perform a predetermined process.

[0223] Furthermore, because the partial model TM3_P is a part of the overall model TM3_W, the data size of the partial model TM3_P is smaller than the data size of the overall model TM3_W. Therefore, in the first modified example, by performing the 3D matching process using the partial model TM3_P, the control device 3 can reduce the time required to perform the 3D matching process compared to when the 3D matching process is not performed using the partial model TM3_P (i.e., when 3D matching is performed using only the overall model TM3_W).

[0224] Incidentally, even when performing 2D matching processing using a template image as a template, as in the case of 3D matching processing, the position and orientation calculation unit 312 may perform 2D matching processing using a partial model corresponding to a portion of the template image as a template, in addition to or instead of performing 2D matching processing using an overall model corresponding to the entire template image as a template. Incidentally, when the partial model used in the 2D matching processing is generated based on the above-described simulation image data, the simulation image data may be, for example, image data in the same format as the image data IMG_2D (e.g., monocular image data). Incidentally, the method for generating the partial model used in the 2D matching processing may be the same as the method for generating the partial model TM3_P used in the above-described 3D matching processing, and therefore a detailed description thereof will be omitted. Furthermore, the 2D matching processing using the partial model may be the same as the 3D matching processing using the above-described partial model TM3_P, and therefore a detailed description thereof will be omitted. Incidentally, the partial model corresponding to a portion of the template image may be referred to as a 2D partial model.

[0225] When a two-dimensional partial model is used in the 2D matching process, multiple two-dimensional partial models may be used, as in the case where a partial model TM3_P is used in the 3D matching process. In this case, the control device 3 may generate a two-dimensional partial model by excluding image portions corresponding to curved portions (curved portions) of an object captured in a template image corresponding to the overall model. Alternatively, the control device 3 may distinguish between the curved portions (curved portions) and other portions (e.g., planar portions) in the overall model TM3_W, generate a partial model TM3_P by excluding the curved portions (curved portions) from the overall model TM3_W, and generate a two-dimensional partial model by projecting the partial model TM3_P from which the curved portions have been removed onto a two-dimensional plane. In this case, as described above, the control device 3 may generate a two-dimensional partial model based on instructions from the user. As another example, the control device 3 may generate a two-dimensional partial model by distinguishing between curved portions (curved portions) and other portions (e.g., flat portions) of an object reflected in a template image corresponding to the overall model without using instructions from the user, and excluding image portions corresponding to the curved portions (curved portions) from the template image corresponding to the overall model. Furthermore, the control device 3 may generate a model in which the thickness of at least a portion of the object reflected in the template image corresponding to the overall model is thinned as a partial model TM3_P, and generate a two-dimensional partial model obtained by projecting the partial model TM3_P corresponding to the thinned model onto a two-dimensional plane.

[0226] (4-2) Second Modification Next, a robot system SYS in a second modification will be described. In the following description, the robot system SYS in the second modification will be referred to as a "robot system SYSb."

[0227] (4-2-1) Configuration of Robot System SYSb of Second Modification The robot system SYSb of the second modification differs from the robot system SYS described above in that it includes a control device 3b instead of the control device 3. Other features of the robot system SYSb may be the same as other features of the robot system SYS. Therefore, hereinafter, the control device 3b of the second modification will be described with reference to FIG. 14. FIG. 14 is a block diagram showing the configuration of the control device 3b of the second modification.

[0228] 14 , the control device 3b in the second modification is different from the above-described control device 3 in that the calculation device 31 includes a movement control unit 314b as a logical processing block. Other features of the control device 3b may be the same as other features of the control device 3.

[0229] The movement control unit 314b may determine whether the calculation result condition described in the first modified example is met (i.e., whether the result of the position and orientation calculation including the matching process is good). The position and orientation calculation is performed using at least one of the image data IMG_2D and IMG_3D acquired in step S1 of FIG. 4. Therefore, determining whether the calculation result condition is met may be considered equivalent to determining whether the result of the position and orientation calculation performed based on at least one of the image data IMG_2D and IMG_3D acquired in step S1 of FIG. 4 is good.

[0230] Furthermore, when it is determined that the calculation result condition is not satisfied, the movement control unit 314b moves the imaging system 2 so as to change the positional relationship between the imaging system 2 and the target object OBJ. Note that in the second modified example, the positional relationship between the imaging system 2 and the target object OBJ may refer to the positional relationship between the target object OBJ and at least one of the imaging devices 21 and 22. In the second modified example, moving the imaging system 2 may refer to moving at least one of the imaging devices 21 and 22.

[0231] Specifically, when it is determined that the end effector 4 cannot perform a predetermined process on the target object OBJ in a situation where the positional relationship between the imaging system 2 and the target object OBJ is the first positional relationship, the movement control unit 314b moves the imaging system 2 so that the positional relationship between the imaging system 2 and the target object OBJ changes from the first positional relationship to a second positional relationship that is different from the first positional relationship. In other words, when it is determined that the end effector 4 cannot perform a predetermined process on the target object OBJ based on at least one of the image data IMG_2D and IMG_3D acquired in a situation where the positional relationship between the imaging system 2 and the target object OBJ is the first positional relationship, the movement control unit 314b moves the imaging system 2 so that the positional relationship between the imaging system 2 and the target object OBJ changes from the first positional relationship to the second positional relationship. In other words, when it is determined that the end effector 4 cannot perform a specified processing on the target object OBJ based on the results of a matching process performed using at least one of the image data IMG_2D and IMG_3D acquired under the condition that the positional relationship between the imaging system 2 and the target object OBJ is a first positional relationship, the movement control unit 314b moves the imaging system 2 so that the positional relationship between the imaging system 2 and the target object OBJ changes from the first positional relationship to the second positional relationship.

[0232] The positional relationship between the imaging system 2 and the target object OBJ may include the relationship between the position of the imaging system 2 and the position of the target object OBJ. In the second modified example, the position of the imaging system 2 may refer to the position of at least one of the imaging devices 21 and 22. Therefore, in the second modified example, the positional relationship between the imaging system 2 and the target object OBJ may include the relationship between the position of at least one of the imaging devices 21 and 22 and the position of the target object OBJ. In this case, the movement control unit 314b may change the positional relationship between the imaging system 2 and the target object OBJ from the first positional relationship to the second positional relationship by moving the imaging system 2 so that the position of the imaging system 2 changes. Specifically, the movement control unit 314b may change the positional relationship between the imaging system 2 and the target object OBJ from the first positional relationship to the second positional relationship by moving the imaging system 2 so that the position of the imaging system 2 changes from a first position corresponding to the first positional relationship to a second position corresponding to the second positional relationship.

[0233] The positional relationship between the imaging system 2 and the target object OBJ may include the relationship between the orientation of the imaging system 2 and the orientation of the target object OBJ. Note that in the second modified example, the orientation of the imaging system 2 may refer to the orientation of at least one of the imaging devices 21 and 22. Therefore, in the second modified example, the positional relationship between the imaging system 2 and the target object OBJ may include the relationship between the orientation of at least one of the imaging devices 21 and 22 and the orientation of the target object OBJ. In this case, the movement control unit 314b may change the positional relationship between the imaging system 2 and the target object OBJ from the first positional relationship to the second positional relationship by moving the imaging system 2 so as to change the orientation of the imaging system 2. Specifically, the movement control unit 314b may change the positional relationship between the imaging system 2 and the target object OBJ from the first positional relationship to the second positional relationship by moving the imaging system 2 so as to change the orientation of the imaging system 2 from a first orientation corresponding to the first positional relationship to a second orientation corresponding to the second positional relationship. Note that, when the positional relationship between the imaging system 2 and the target object OBJ indicates the relationship between the orientation of at least one of the imaging devices 21 and 22 and the orientation of the target object OBJ, the positional relationship between the imaging system 2 and the target object OBJ may be referred to as the orientation relationship between the imaging system 2 and the target object OBJ. In this case, the movement control unit 314b may change the orientation relationship between the imaging system 2 and the target object OBJ from the first orientation relationship to the second orientation relationship by moving the imaging system 2 so that the orientation of the imaging system 2 changes from a first orientation corresponding to the first orientation relationship to a second orientation corresponding to the second orientation relationship.

[0234] It can also be said that the processing performed by the movement control unit 314b includes an operation of moving the imaging system 2 so as to change at least one of the position and orientation of the imaging system 2 with respect to one target object OBJ. It can also be said that the processing performed by the movement control unit 314b includes an operation of moving the imaging system 2 so as to change at least one of the position and orientation of the imaging system 2 around one target object OBJ.

[0235] In order to move the imaging system 2, the movement control unit 314b generates a robot control signal for controlling the robot arm 12 to move the imaging system 2 so that the positional relationship between the imaging system 2 and the target object OBJ changes by moving the imaging system 2. In other words, the movement control unit 314b generates a robot control signal for controlling the robot arm 12 so that the positional relationship between the imaging system 2 and the target object OBJ changes from a first positional relationship to a second positional relationship by moving the imaging system 2.

[0236] Thereafter, after the robot arm 12 moves the imaging system 2 based on the robot control signal generated by the movement control unit 314b, the control device 3b again performs position and orientation calculation to generate position and orientation data POI. In other words, after the robot arm 12 moves the imaging system 2, the imaging system 2 again captures an image of the target object OBJ, thereby again generating at least one of the image data IMG_2D and IMG_3D. The control device 3b generates the position and orientation data POI using at least one of the image data IMG_2D and IMG_3D again generated by the imaging system 2. In other words, the control device 3b generates the position and orientation data POI using at least one of the image data IMG_2D and IMG_3D again generated by the imaging system 2 under conditions in which the positional relationship between the imaging system 2 and the target object OBJ is the second positional relationship.

[0237] Here, because the imaging system 2 moves, the imaging system 2 captures an image of the target object OBJ from a position different from the position of the imaging system 2 before the imaging system 2 moved. Alternatively, the imaging system 2 captures an image of the target object OBJ in a posture different from the posture of the imaging system 2 before the imaging system 2 moved. Therefore, the way the target object OBJ appears in the images represented by the image data IMG_2D and IMG_3D changes. As a result, even if it is determined that the end effector 4 cannot perform a predetermined process on the target object OBJ based on at least one of the image data IMG_2D and IMG_3D acquired before the imaging system 2 moved, there is a possibility that the end effector 4 can perform the predetermined process on the target object OBJ based on at least one of the image data IMG_2D and IMG_3D acquired after the imaging system 2 moved.

[0238] As an example, as described above, if the result of the position and orientation calculation is poor (i.e., not good), it may be determined that the end effector 4 cannot perform a predetermined process on the target object OBJ. In this case, because the appearance of the target object OBJ in the images represented by the image data IMG_2D and IMG_3D changes, even if the result of the position and orientation calculation performed before the imaging system 2 moves is poor, the result of the position and orientation calculation performed after the imaging system 2 moves may no longer be poor (i.e., may be good). Therefore, if the imaging system 2 moves due to a poor result of the position and orientation calculation, the result of the position and orientation calculation is more likely to be good compared to when the imaging system 2 does not move, even if the result of the position and orientation calculation is poor. As a result, the position and orientation calculation unit 312 can appropriately calculate at least one of the position and orientation of the target object OBJ. As a result, it is more likely that the calculation result condition is satisfied.

[0239] As another example, as described above, it may be determined that the calculation result condition is not satisfied when the position and orientation calculation unit 312 is unable to perform the position and orientation calculation normally. In this case, even if the position and orientation calculation unit 312 was unable to perform the position and orientation calculation normally before the image capture system 2 moved because the appearance of the target object OBJ changes in the images represented by the image data IMG_2D and IMG_3D, there is a possibility that the position and orientation calculation unit 312 will be able to perform the position and orientation calculation normally after the image capture system 2 has moved. Therefore, when the image capture system 2 moves because the position and orientation calculation unit 312 was unable to perform the position and orientation calculation normally, the position and orientation calculation unit 312 is more likely to perform the position and orientation calculation normally compared to when the image capture system 2 does not move, even if the position and orientation calculation unit 312 was unable to perform the position and orientation calculation normally. In other words, the position and orientation calculation result is more likely to be satisfactory. As a result, the position and orientation calculation unit 312 can appropriately calculate at least one of the position and orientation of the target object OBJ. As a result, it is more likely that the calculation result condition is satisfied.

[0240] Thereafter, if at least one of the position and orientation of the target object OBJ has been properly calculated (i.e., the position and orientation data POI has been properly generated), in step S4 of FIG. 4, the control device 3b (particularly, the signal generation unit 313) generates a robot control signal based on the generated position and orientation data POI so that the robot 1 (end effector 4) can perform a predetermined process on the target object OBJ.

[0241] 4 (i.e., the robot control signal used mainly to cause the robot 1 (end effector 4) to perform a predetermined process) generated by the signal generating unit 313 in step S4 in Fig. 4, in that the robot control signal generated by the movement control unit 314b is a signal used mainly to move the imaging system 2. In this case, the robot control signal generated by the movement control unit 314b may be referred to as an imaging device movement signal.

[0242] However, the robot control signal generated by the movement control unit 314b may be considered to be the same as the robot control signal generated by the signal generation unit 313 in step S4 of FIG. 4 in that it is a signal for controlling the robot 1. In other words, the movement control unit 314b and the signal generation unit 313 may have in common the fact that they both generate robot control signals. In this case, the signal generation unit 313 may perform the operations performed by the movement control unit 314b. In other words, the signal generation unit 313 may function as the movement control unit 314b. In this case, the control device 3b may not be equipped with the movement control unit 314b.

[0243] Note that because both the imaging devices 21 and 22 are attached to the robot arm 12, the robot arm 12, for example, moves both the imaging devices 21 and 22. For this reason, the movement control unit 314b generates a robot control signal for controlling the robot arm 12 that can move both the imaging devices 21 and 22, for example, so that moving both the imaging devices 21 and 22 changes the positional relationship between both the imaging devices 21 and 22 and the target object OBJ. In the following description, an example will be described in which the movement control unit 314b generates a robot control signal for controlling the robot arm 12 so that moving both the imaging devices 21 and 22 changes the positional relationship between both the imaging devices 21 and 22 and the target object OBJ. However, depending on the arrangement of the imaging devices 21 and 22, the movement control unit 314b may generate a robot control signal for controlling the robot arm 12 that can move either one of the imaging devices 21 and 22 so that moving either one of the imaging devices 21 and 22 changes the positional relationship between either one of the imaging devices 21 and 22 and the target object OBJ.

[0244] (4-2-2) Flow of Robot Control Processing in Second Modification Next, the robot control processing performed by the control device 3b in the second modification will be described with reference to Fig. 15. Fig. 15 is a flowchart showing the flow of the robot control processing performed by the control device 3b in the second modification.

[0245] 15 , in the second modification, the control device 3b also acquires at least one of image data IMG_2D and IMG_3D from the imaging system 2 using the communication device 33 (step S1). Furthermore, each time the control device 3b acquires image data IMG_3D, the three-dimensional position data generator 311 of the control device 3b generates three-dimensional position data WSD based on the acquired image data IMG_3D (step S2). Thereafter, the position and orientation calculator 312 of the control device 3b performs a position and orientation calculation (i.e., a calculation including a matching process) to calculate at least one of the position and orientation of the target object OBJ based on at least one of the image data IMG_2D acquired in step S1 and the three-dimensional position data WSD generated in step S2 (step S3). That is, the position and orientation calculation unit 312 performs position and orientation calculation to generate position and orientation data POI indicating at least one of the position and orientation of the target object OBJ based on at least one of the image data IMG_2D acquired in step S1 and the three-dimensional position data WSD generated in step S2 (step S3). Note that, in the second modified example, as in the above description using FIG. 4 , for simplicity of explanation, an example will be described in which the position and orientation calculation unit 312 performs 3D matching processing in step S3 of FIG. 15 (i.e., an example in which the control device 3b acquires image data IMG_3D in step S1 and the control device 3b performs 3D matching processing in step S3). However, in the second modified example, the position and orientation calculation unit 312 may also perform 2D matching processing, or may perform both 2D matching processing and 3D matching processing in step S3 of FIG.

[0246] Thereafter, in the second modified example, the movement control unit 314b of the control device 3b determines whether the calculation result condition is met (step S61b). Specifically, the movement control unit 314b determines whether the result of the position and orientation calculation performed based on at least one of the image data IMG_2D and IMG_3D acquired in step S1 is satisfactory, thereby determining whether the calculation result condition is met. In other words, the movement control unit 314b determines whether the result of the position and orientation calculation performed in step S3 of FIG. 4 is satisfactory, thereby determining whether the calculation result condition is met.

[0247] The details of the "computation result condition" have already been explained in the first modified example, and therefore will not be explained in detail in the second modified example.

[0248] If it is determined in step S61b that the calculation result condition is satisfied (step S61b: Yes), the signal generator 313 generates a robot control signal using the position and orientation data POI generated in step S3 (step S4). That is, the signal generator 313 uses the position and orientation data POI generated in step S3 to generate a robot control signal for controlling the robot 1 to bring the end effector 4 closer to the target object OBJ (step S4).

[0249] On the other hand, if it is determined in step S61b that the calculation result condition is not satisfied (step S61b: Yes), the signal generation unit 313 does not need to generate a robot control signal using the position and orientation data POI generated in step S3. Alternatively, if the position and orientation calculation unit 312 was unable to successfully perform the position and orientation calculation in step S3 in the first place, it is possible that the position and orientation data POI was not generated, and therefore the signal generation unit 313 cannot generate a robot control signal. In this case, the movement control unit 314b moves the imaging system 2 so as to change the positional relationship between the imaging system 2 and the target object OBJ (steps S62b to S64b).

[0250] Specifically, the movement control unit 314b acquires image data IMG_2D from the imaging system 2 (step S62b). However, as will be described later, in step S62b, the movement control unit 314b may acquire image data IMG_3D from the imaging system 2 in addition to or instead of the image data IMG_2D.

[0251] For example, the imaging system 2 may generate the image data IMG_2D acquired by the movement control unit 314b in step S62b by capturing an image of the target object OBJ after it is determined in step S61b that the calculation result condition is not satisfied. Note that the imaging system 2 may generate the image data IMG_2D acquired by the movement control unit 314b in step S62b by capturing a new image of the target object OBJ from the same direction as the direction in which the target object OBJ was captured to generate the image data IMG_3D acquired in step S1. In this case, the movement control unit 314b may acquire the image data IMG_2D generated by the imaging system 2 after it is determined that the calculation result condition is not satisfied.

[0252] For example, the imaging system 2 may generate the image data IMG_2D acquired by the movement control unit 314b in step S62b by capturing an image of the target object OBJ before it is determined in step S61b that the calculation result condition is not satisfied. Note that the imaging system 2 may generate the image data IMG_2D acquired by the movement control unit 314b in step S62b by capturing a new image of the target object OBJ from the same direction as the direction in which the target object OBJ was captured to generate the image data IMG_3D acquired in step S1. In this case, the movement control unit 314b may acquire the image data IMG_2D generated by the imaging system 2 before it is determined that the calculation result condition is not satisfied. As one example, the imaging system 2 may generate image data IMG_2D to be acquired by the movement control unit 314b in step S62b by capturing an image of the target object OBJ after the imaging system 2 generates at least one of image data IMG_2D and IMG_3D for position and orientation calculation by capturing an image of the target object OBJ in step S1 and before it is determined in step S61b that the calculation result condition is not satisfied. As another example, the imaging system 2 may generate image data IMG_3D for position and orientation calculation in step S1 and also generate image data IMG_2D to be acquired by the movement control unit 314b in step S62b. In other words, the imaging system 2 may generate the image data IMG_3D for position and orientation calculation and the image data IMG_2D to be acquired by the movement control unit 314b in step S62b at the same time.

[0253] As described above, the second modified example describes an example in which the position and orientation calculation unit 312 performs 3D matching processing in step S3 of FIG. 15 . However, if the position and orientation calculation unit 312 performs 2D matching processing in step S3 of FIG. 15 , the imaging system 2 generates image data IMG_2D for position and orientation calculation in step S1. In this case, the movement control unit 314b may acquire the image data IMG_2D generated in step S1 in step S62b. That is, the image data IMG_2D for the position and orientation calculation in step S3 may be used as the image data IMG_2D acquired by the movement control unit 314b in step S62b. Alternatively, in addition to or instead of using the image data IMG_2D, the position and orientation data POI generated by the position and orientation calculation in step S3 of FIG. 15 may be used by the movement control unit 314b to set the movement mode of the imaging system 2 in step S63b of FIG. 15 .

[0254] In either case, the image data IMG_2D acquired by the movement control unit 314b in step S62b is image data IMG_2D generated by the imaging system 2 capturing an image of the target object OBJ before the movement control unit 314b changes the positional relationship between the imaging system 2 and the target object OBJ. Specifically, as described above, when it is determined that the calculation result condition is not satisfied in a situation where the positional relationship between the imaging system 2 and the target object OBJ is the first positional relationship, the movement control unit 314b moves the imaging system 2 so that the positional relationship between the imaging system 2 and the target object OBJ changes from the first positional relationship to the second positional relationship. In this case, the image data IMG_2D acquired by the movement control unit 314b in step S62b may be image data IMG_2D generated by the imaging system 2 capturing an image of the target object OBJ in a situation where the positional relationship between the imaging system 2 and the target object OBJ is the first positional relationship. In other words, the image data IMG_2D acquired by the movement control unit 314b in step S62b may be image data IMG_2D generated by the imaging system 2 capturing an image of the target object OBJ under a situation in which the positional relationship between the imaging system 2 and the target object OBJ is different from the second positional relationship. However, the image data IMG_2D acquired by the movement control unit 314b in step S62b may be image data IMG_2D generated by the imaging system 2 capturing an image of the target object OBJ under a situation in which the positional relationship between the imaging system 2 and the target object OBJ is different from the first positional relationship.

[0255] Thereafter, the movement control unit 314b sets a movement mode of the imaging system 2 relative to one target object (e.g., a peering target object OBJ_g described below) based on the image data IMG_2D acquired in step S62b (step S63b). The movement mode of the imaging system 2 may refer to a movement pattern or movement method of the imaging system 2 relative to one target object (e.g., a peering target object OBJ_g described below). In other words, the movement mode of the imaging system 2 may refer to a movement pattern or movement method indicating how the imaging system 2 is moved relative to one target object (e.g., a peering target object OBJ_g described below). The movement mode can also be said to refer to a movement pattern or movement method of the imaging system 2 based on one target object (e.g., a peering target object OBJ_g described below). The movement mode can also be said to refer to a movement pattern or movement method of the imaging system 2 around one target object (e.g., a peering target object OBJ_g described below).

[0256] In particular, the movement control unit 314b may automatically set the movement mode of the imaging system 2 based on the image data IMG_2D acquired in step S62b (step S63b). For example, the movement control unit 314b may calculate at least one of the position and orientation of the target object OBJ by performing 2D matching processing using the image data IMG_2D. That is, the movement control unit 314b may generate position and orientation data POI indicating at least one of the position and orientation of the target object OBJ by performing 2D matching processing using the image data IMG_2D. In particular, the position and orientation calculation unit 312 may generate position and orientation data POI indicating at least one of the position and orientation of one target object OBJ that was captured in the latest image data IMG_3D (or image data IMG_2D) acquired in step S1 of FIG. 15 but for which it was determined by the position and orientation calculation that the end effector 4 cannot perform the predetermined processing. As described below, this includes an operation of moving the imaging system 2 so as to change at least one of the position and orientation of the imaging system 2 relative to one target object OBJ for which the position and orientation data POI was calculated in step S63b. Therefore, in the following description, one target object OBJ for which the position and orientation data POI was calculated in step S63b will be referred to as the "looking target object OBJ_g." The 2D matching process in step S63b may be the same as the 2D matching process in step S3 described above, and therefore a detailed description thereof will be omitted. Thereafter, the movement control unit 314b may set the movement mode of the imaging system 2 based on the generated position and orientation data POI.

[0257] Note that, if the image data IMG_3D is acquired in step S62b as described above, the movement control unit 314b may set the movement mode of the imaging system 2 based on the image data IMG_3D in step S63b. For example, the movement control unit 314b may generate position and orientation data POI indicating at least one of the position and orientation of the target object OBJ by performing 3D matching processing using the three-dimensional position data WSD generated from the image data IMG_3D. The 3D matching processing in step S63b may be the same as the 3D matching processing in step S3 described above, and therefore a detailed description thereof will be omitted. Thereafter, the movement control unit 314b may set the movement mode of the imaging system 2 based on the generated position and orientation data POI. Alternatively, the movement control unit 314b may set the movement mode of the imaging system 2 based on both the image data IMG_2D and IMG_3D in step S63b. For example, the movement control unit 314b may generate position and orientation data POI indicating at least one of the position and orientation of the target object OBJ by performing both a 2D matching process using the image data IMG_2D and a 3D matching process using three-dimensional position data WSD generated from the image data IMG_3D. The process of performing both the 2D matching process and the 3D matching process in step S63b may be the same as the process of performing both the 2D matching process and the 3D matching process in step S3 described above, and therefore a detailed description thereof will be omitted. The movement control unit 314b may then set the movement mode of the imaging system 2 based on the generated position and orientation data POI. For simplicity's sake, the following description will be given of an example in which the movement control unit 314b sets the movement mode of the imaging system 2 based on the image data IMG_2D in step S63b.

[0258] The movement control unit 314b may set the movement mode of the imaging system 2 so that it is more likely that the calculation result condition will be satisfied (i.e., the result of the position and orientation calculation will be good) after the imaging system 2 has moved with respect to the peering target object OBJ_g in accordance with the movement mode set in step S63b. Specifically, the movement control unit 314b may set the movement mode of the imaging system 2 based on the position and orientation data POI indicating at least one of the position and orientation of the peering target object OBJ_g so that the positional relationship between the imaging system 2 and the peering target object OBJ_g satisfies the condition that "it is more likely that the calculation result condition related to the result of the position and orientation calculation performed again after the imaging system 2 has moved in accordance with the movement mode set in step S63b will be satisfied." The movement control unit 314b may set the movement mode of the imaging system 2 based on the position and orientation data POI indicating at least one of the position and orientation of the object OBJ_g to be looked at, so that the positional relationship between the imaging system 2 and the object OBJ_g to be looked at satisfies the condition that "the possibility that the calculation result condition regarding the result of the position and orientation calculation performed again after the imaging system 2 has moved according to the movement mode set in step S63b will be determined to be satisfied is higher than the possibility that the calculation result condition regarding the result of the position and orientation calculation performed before the imaging system 2 has moved according to the movement mode set in step S63b will be determined to be satisfied."

[0259] When the calculation result condition is met (i.e., the result of the position and orientation calculation is good), it is expected that the end effector 4 will be able to perform the predetermined processing on the look-in target object OBJ_g. For this reason, the movement control unit 314b may set the movement mode of the imaging system 2 so that it is more likely that it will be determined that the end effector 4 is able to perform the predetermined processing on the look-in target object OBJ_g after the imaging system 2 has moved relative to the look-in target object OBJ_g in accordance with the movement mode set in step S63b. Specifically, the movement control unit 314b may set the movement mode of the imaging system 2 based on the position and orientation data POI indicating at least one of the position and orientation of the look-in target object OBJ_g, so that the positional relationship between the imaging system 2 and the look-in target object OBJ_g satisfies the condition that "it is more likely that it will be determined that the end effector 4 is able to perform the predetermined processing on the look-in target object OBJ_g after the imaging system 2 has moved relative to the look-in target object OBJ_g in accordance with the movement mode set in step S63b." The movement control unit 314b may set the movement mode of the imaging system 2 based on the position and orientation data POI indicating at least one of the position and orientation of the object OBJ_g to be looked at, so that the positional relationship between the imaging system 2 and the object OBJ_g to be looked at satisfies the condition that "the possibility that it will be determined that the end effector 4 can perform a specified processing on the object OBJ_g to be looked at after the imaging system 2 has moved in accordance with the movement mode set in step S63b is higher than the possibility that it will be determined that the end effector 4 can perform a specified processing on the object OBJ_g to be looked at before the imaging system 2 moves relative to the object OBJ_g to be looked at in accordance with the movement mode set in step S63b."

[0260] As an example, when a position and orientation calculation including a 3D matching process is performed to generate position and orientation data POI in step S3 of FIG. 4 , the greater the number of points of the target object OBJ included in the point cloud indicated by the three-dimensional position data WSD used to perform the 3D matching process, the greater the amount of information available through the 3D matching process. Therefore, the greater the number of points of the target object OBJ included in the point cloud indicated by the three-dimensional position data WSD, the greater the likelihood that the target object OBJ will be detected through the 3D matching process. Furthermore, the greater the number of points of the target object OBJ included in the point cloud indicated by the three-dimensional position data WSD, the greater the accuracy (reliability) of the position and orientation data POI generated through the position and orientation calculation including the 3D matching process. Therefore, the movement control unit 314b may set the movement mode of the imaging system 2 so that the number of points of the peering target object OBJ_g included in the point cloud indicated by the three-dimensional position data WSD increases. Specifically, the movement control unit 314b may set the movement mode of the imaging system 2 so that the number of points of the peering target object OBJ_g included in the point cloud indicated by the three-dimensional position data WSD generated again in step S2 after the imaging system 2 has moved is greater than the number of points of the peering target object OBJ_g included in the point cloud indicated by the three-dimensional position data WSD generated in step S2 before the imaging system 2 moved. As a result, the position and orientation calculation including the 3D matching process is more likely to produce good results.

[0261] The number of points of the target object OBJ included in the point cloud indicated by the three-dimensional position data WSD is likely to increase as the proportion of the target object OBJ occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 22) increases. Therefore, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 22) increases. Specifically, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 22) after the imaging system 2 has moved is greater than the proportion of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 22) before the imaging system 2 moved. In other words, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of the peering target object OBJ_g in the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 22) for generating the image data IMG_3D after the imaging system 2 has moved is larger than the proportion of the peering target object OBJ_g in the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 22) for generating the image data IMG_3D before the imaging system 2 moved. As a result, the position and orientation calculation including the 3D matching process is more likely to produce good results.

[0262] Note that, when the proportion of the portion of the target object OBJ occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 22) increases, there is a possibility that the proportion of the target object OBJ occupying the imaging field of view of the imaging system 2 will increase. For this reason, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of the portion of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 22) increases. Specifically, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of the portion of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 22) after the imaging system 2 has moved is larger than the proportion of the portion of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 22) before the imaging system 2 has moved.

[0263] Furthermore, when the proportion of the contour of the target object OBJ in the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 22) increases, there is a possibility that the proportion of the target object OBJ in the imaging field of view of the imaging system 2 increases. For this reason, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of the contour of the peering target object OBJ_g in the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 22) increases. Specifically, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of the contour of the peering target object OBJ_g in the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 22) after the imaging system 2 has moved is greater than the proportion of the contour of the peering target object OBJ_g in the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 22) before the imaging system 2 moved. Here, when the entire contour of the target object OBJ falls within the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 22), the proportion of the contour of the target object OBJ occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 22) becomes maximum. For this reason, the movement control unit 314b may set the movement mode of the imaging system 2 so that the entire contour of the peering target object OBJ_g falls within the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 22).

[0264] Furthermore, the movement control unit 314b may set the movement mode of the imaging system 2 so that both the proportion of a portion of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 22) and the proportion of the outline of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 22) are maximized (or become appropriate proportions). Alternatively, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of a portion of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 22) and the proportion of the outline of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 22) become appropriate ratios. In other words, the movement control unit 314b may set the movement mode of the imaging system 2 so as to balance the proportion of a portion of the object OBJ_g to be looked at that occupies the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 22) with the proportion of the contour of the object OBJ_g to be looked at that occupies the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 22).

[0265] For example, Fig. 16(a) shows the imaging device 22 before it moves under the control of the movement control unit 314b and the imaging device 22 after it moves under the control of the movement control unit 314b. In the example shown in Fig. 16(a), the imaging device 22 before it moves under the control of the movement control unit 314b is located at a first position P1. On the other hand, in the example shown in Fig. 16(a), the imaging device 22 after it moves under the control of the movement control unit 314b is located at a second position P2 that is different from the first position P1. In this case, the movement control unit 314b may set the movement mode of the imaging system 2 relative to the object OBJ_g to be looked at so that the number of points of the object OBJ_g to be looked at included in the point cloud indicated by the three-dimensional position data WSD generated based on the image data IMG_3D generated by the imaging device 22 located at the second position P2 is greater than the number of points of the object OBJ_g to be looked at included in the point cloud indicated by the three-dimensional position data WSD generated based on the image data IMG_3D generated by the imaging device 22 located at the first position P1. The movement control unit 314b may set the movement mode of the imaging system 2 based on the object OBJ_g to be looked at so that the number of points of the object OBJ_g to be looked at included in the point cloud indicated by the three-dimensional position data WSD generated based on image data IMG_3D generated by the imaging device 22 located at the second position P2 is greater than the number of points of the object OBJ_g to be looked at included in the point cloud indicated by the three-dimensional position data WSD generated based on image data IMG_3D generated by the imaging device 22 located at the first position P1. The movement control unit 314b may set the movement pattern of the imaging system 2 around the object OBJ_g to be looked at so that the number of points of the object OBJ_g to be looked at included in the point cloud indicated by the three-dimensional position data WSD generated based on image data IMG_3D generated by the imaging device 22 located at the second position P2 is greater than the number of points of the object OBJ_g to be looked at included in the point cloud indicated by the three-dimensional position data WSD generated based on image data IMG_3D generated by the imaging device 22 located at the first position P1.The movement control unit 314b may set a movement mode of the imaging system 2 relative to the peering target object OBJ_g so that the proportion of the peering target object OBJ_g shown in the imaging field of view of the imaging device 22 located at the second position P2 is larger than the proportion of the peering target object OBJ_g shown in the imaging field of view of the imaging device 22 located at the first position P1. The movement control unit 314b may set a movement mode of the imaging system 2 based on the peering target object OBJ_g so that the proportion of the peering target object OBJ_g shown in the imaging field of view of the imaging device 22 located at the second position P2 is larger than the proportion of the peering target object OBJ_g shown in the imaging field of view of the imaging device 22 located at the first position P1. The movement control unit 314b may set the movement pattern of the imaging system 2 around the object to be looked at OBJ_g so that the proportion of the object to be looked at shown in the imaging field of view of the imaging device 22 located at the second position P2 is greater than the proportion of the object to be looked at OBJ_g shown in the imaging field of view of the imaging device 22 located at the first position P1. For example, the movement control unit 314b may set a second position P2 at which the imaging device 22 should be located after movement so that the number of points of the object OBJ_g to be looked at included in the point cloud indicated by the three-dimensional position data WSD generated based on image data IMG_3D generated by the imaging device 22 located at the second position P2 is greater than the number of points of the object OBJ_g to be looked at included in the point cloud indicated by the three-dimensional position data WSD generated based on image data IMG_3D generated by the imaging device 22 located at the first position P1, and may set the movement mode of the imaging system 2 relative to the object OBJ_g to be looked at so that the imaging device 22 moves from the first position P1 to the second position P2.For example, the movement control unit 314b may set a second position P2 at which the imaging device 22 should be located after movement so that the number of points of the object OBJ_g to be looked at included in the point cloud indicated by the three-dimensional position data WSD generated based on image data IMG_3D generated by the imaging device 22 located at the second position P2 is greater than the number of points of the object OBJ_g to be looked at included in the point cloud indicated by the three-dimensional position data WSD generated based on image data IMG_3D generated by the imaging device 22 located at the first position P1, and may set the movement mode of the imaging system 2 based on the object OBJ_g to be looked at so that the imaging device 22 moves from the first position P1 to the second position P2. For example, the movement control unit 314b may set a second position P2 at which the imaging device 22 should be located after movement so that the number of points of the object OBJ_g to be looked at included in the point cloud indicated by the three-dimensional position data WSD generated based on image data IMG_3D generated by the imaging device 22 located at the second position P2 is greater than the number of points of the object OBJ_g to be looked at included in the point cloud indicated by the three-dimensional position data WSD generated based on image data IMG_3D generated by the imaging device 22 located at the first position P1, and may set the movement pattern of the imaging system 2 around the object OBJ_g to be looked at so that the imaging device 22 moves from the first position P1 to the second position P2. For example, the movement control unit 314b may set a second position P2 at which the imaging device 22 should be located after movement so that the proportion of the object to be looked at OBJ_g shown in the imaging field of view of the imaging device 22 located at the second position P2 is greater than the proportion of the object to be looked at OBJ_g shown in the imaging field of view of the imaging device 22 located at the first position P1, and may set the movement mode of the imaging system 2 relative to the object to be looked at OBJ_g so that the imaging device 22 moves from the first position P1 to the second position P2.For example, the movement control unit 314b may set a second position P2 at which the imaging device 22 should be located after movement so that the proportion of the object to be looked at OBJ_g shown in the imaging field of view of the imaging device 22 located at the second position P2 is greater than the proportion of the object to be looked at OBJ_g shown in the imaging field of view of the imaging device 22 located at the first position P1, and may set the movement mode of the imaging system 2 based on the object to be looked at OBJ_g so that the imaging device 22 moves from the first position P1 to the second position P2. For example, the movement control unit 314b may set a second position P2 at which the image capture device 22 should be located after movement so that the proportion of the peering target object OBJ_g shown in the imaging field of view of the image capture device 22 located at the second position P2 is larger than the proportion of the peering target object OBJ_g shown in the imaging field of view of the image capture device 22 located at the first position P1, and may set the movement mode of the image capture system 2 around the peering target object OBJ_g so that the image capture device 22 moves from the first position P1 to the second position P2. Note that setting the movement mode of the image capture system 2 relative to the peering target object OBJ_g may be considered equivalent to setting the movement mode of the image capture system 2 based on the peering target object OBJ_g and setting the movement mode of the image capture system 2 around the peering target object OBJ_g.

[0266] 16( a), the image capture device 22 located at the first position P1 faces the first surface S1 of the object OBJ_g being looked at, which is narrower than the second surface S2 of the object OBJ_g being looked at. On the other hand, the image capture device 22 located at the second position P2 faces the second surface S2 of the object OBJ_g being looked at, which is wider than the first surface S1 of the object OBJ_g being looked at. Therefore, in the example shown in FIG. 16( a), the number of points of the object OBJ_g being looked at, which are included in the point cloud indicated by the three-dimensional position data WSD generated based on the image data IMG_3D generated by the image capture device 22 located at the second position P2, is greater than the number of points of the object OBJ_g being looked at, which are included in the point cloud indicated by the three-dimensional position data WSD generated based on the image data IMG_3D generated by the image capture device 22 located at the first position P1. Furthermore, the proportion of the object OBJ_g to be looked at shown in the imaging field of view of the imaging device 22 located at the second position P2 is greater than the proportion of the object OBJ_g to be looked at shown in the imaging field of view of the imaging device 22 located at the first position P1.

[0267] As described above, the second modified example describes an example in which the position and orientation calculation unit 312 performs 3D matching processing in step S3 of FIG. 15 . However, if a position and orientation calculation including 2D matching processing is performed to generate position and orientation data POI in step S3 of FIG. 15 , the greater the number of pixels in which at least a portion of the target object OBJ is captured in the image represented by the image data IMG_2D used to perform the 2D matching processing, the greater the amount of information available through the 2D matching processing. Therefore, the greater the number of pixels in which at least a portion of the target object OBJ is captured in the image represented by the image data IMG_2D, the greater the likelihood that the target object OBJ will be detected through the 2D matching processing. Furthermore, the greater the number of pixels in which at least a portion of the target object OBJ is captured in the image represented by the image data IMG_2D, the greater the accuracy (reliability) of the position and orientation data POI generated by the position and orientation calculation including the 2D matching processing. For this reason, the movement control unit 314b may set the movement mode of the imaging system 2 so that the number of pixels in which at least a portion of the peering target object OBJ_g is reflected is increased in the image represented by the image data IMG_2D. Specifically, the movement control unit 314b may set the movement mode of the imaging system 2 so that the number of pixels in which at least a portion of the peering target object OBJ_g is reflected in the image represented by the image data IMG_2D acquired again in step S1 after the imaging system 2 has moved is greater than the number of pixels in which at least a portion of the peering target object OBJ_g is reflected in the image represented by the image data IMG_2D already acquired in step S1 before the imaging system 2 moved. As a result, the position and orientation calculation including the 2D matching process is more likely to result in a good result.

[0268] The number of pixels in which at least a portion of the target object OBJ is captured in the image indicated by the image data IMG_2D is likely to increase as the proportion of the target object OBJ in the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 21) increases. For this reason, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of the peering target object OBJ_g in the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 21) increases. Specifically, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of the peering target object OBJ_g in the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 21) after the imaging system 2 has moved is greater than the proportion of the peering target object OBJ_g in the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 21) before the imaging system 2 moved. In other words, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of the peering target object OBJ_g in the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 21) for generating the image data IMG_2D after the imaging system 2 has moved is larger than the proportion of the peering target object OBJ_g in the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 21) for generating the image data IMG_2D before the imaging system 2 has moved. As a result, the position and orientation calculation including the 2D matching process is more likely to produce good results.

[0269] Note that, when the proportion of the portion of the target object OBJ occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 21) increases, there is a possibility that the proportion of the target object OBJ occupying the imaging field of view of the imaging system 2 will increase. For this reason, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of the portion of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 21) increases. Specifically, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of the portion of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 21) after the imaging system 2 has moved is larger than the proportion of the portion of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 21) before the imaging system 2 moved.

[0270] Furthermore, when the proportion of the contour of the target object OBJ occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 21) increases, there is a possibility that the proportion of the target object OBJ occupying the imaging field of view of the imaging system 2 will increase. For this reason, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of the contour of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 21) increases. Specifically, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of the contour of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 21) after the imaging system 2 has moved is greater than the proportion of the contour of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 21) before the imaging system 2 has moved. Here, when the entire contour of the target object OBJ falls within the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 21), the proportion of the contour of the target object OBJ occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 21) becomes maximum. For this reason, the movement control unit 314b may set the movement mode of the imaging system 2 so that the entire contour of the peering target object OBJ_g falls within the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 21).

[0271] Furthermore, the movement control unit 314b may set the movement mode of the imaging system 2 so that both the proportion of a portion of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 21) and the proportion of the outline of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 21) are maximized (or become appropriate proportions). Alternatively, the movement control unit 314b may set the movement mode of the imaging system 2 so that the proportion of a portion of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 21) and the proportion of the outline of the peering target object OBJ_g occupying the imaging field of view of the imaging system 2 (particularly, the imaging field of the imaging device 21) become appropriate proportions. In other words, the movement control unit 314b may set the movement mode of the imaging system 2 so as to balance the proportion of a portion of the object OBJ_g to be looked at that occupies the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 21) with the proportion of the contour of the object OBJ_g to be looked at that occupies the imaging field of view of the imaging system 2 (particularly, the imaging field of view of the imaging device 21).

[0272] For example, FIG. 16( b) shows the imaging device 21 before it moves under the control of the movement control unit 314b and the imaging device 21 after it moves under the control of the movement control unit 314b. In the example shown in FIG. 16( b), the imaging device 21 before it moves under the control of the movement control unit 314b is located at a third position P3. On the other hand, in the example shown in FIG. 16( b), the imaging device 21 after it moves under the control of the movement control unit 314b is located at a fourth position P4 that is different from the third position P3. The movement control unit 314b may set the movement mode of the imaging system 2 with respect to the peering target object OBJ_g so that the number of pixels in which at least a portion of the peering target object OBJ_g is reflected in the image represented by the image data IMG_2D generated by the imaging device 21 located at the fourth position P4 is greater than the number of pixels in which at least a portion of the peering target object OBJ_g is reflected in the image represented by the image data IMG_2D generated by the imaging device 21 located at the third position P3. The movement control unit 314b may set a movement mode of the imaging system 2 relative to the peering target object OBJ_g so that the proportion of the peering target object OBJ_g shown in the imaging field of view of the imaging device 21 located at the fourth position P4 is greater than the proportion of the peering target object OBJ_g shown in the imaging field of view of the imaging device 21 located at the third position P3. For example, the movement control unit 314b may set a fourth position P4 at which the imaging device 21 should be located after movement so that the number of pixels in which at least a portion of the peering target object OBJ_g is reflected in the image represented by the image data IMG_2D generated by the imaging device 21 located at the fourth position P4 is greater than the number of pixels in which at least a portion of the peering target object OBJ_g is reflected in the image represented by the image data IMG_2D generated by the imaging device 21 located at the third position P3, and may set a movement mode of the imaging system 2 relative to the peering target object OBJ_g so that the imaging device 21 moves from the third position P3 to the fourth position P4.For example, the movement control unit 314b may set a fourth position P4 at which the imaging device 21 should be located after movement so that the proportion of the object OBJ_g to be looked at shown in the imaging field of view of the imaging device 21 located at the fourth position P4 is greater than the proportion of the object OBJ_g to be looked at shown in the imaging field of view of the imaging device 21 located at the third position P3, and may set the movement mode of the imaging system 2 relative to the object OBJ_g to be looked at so that the imaging device 21 moves from the third position P3 to the fourth position P4.

[0273] As described above, setting the movement mode of the imaging system 2 relative to the peering target object OBJ_g may be considered equivalent to setting the movement mode of the imaging system 2 with respect to the peering target object OBJ_g as a reference and setting the movement mode of the imaging system 2 around the peering target object OBJ_g. Therefore, the signal generator 313 may set the movement mode of the imaging system 2 with respect to the peering target object OBJ_g as a reference so as to satisfy the conditions described in this paragraph. The signal generator 313 may set the movement mode of the imaging system 2 around the peering target object OBJ_g so as to satisfy the conditions described in this paragraph.

[0274] 16(b), the image capture device 21 located at the third position P3 faces the first surface S1 of the object OBJ_g being looked at, which is narrower than the second surface S2 of the object OBJ_g being looked at. On the other hand, the image capture device 21 located at the fourth position P4 faces the second surface S2 of the object OBJ_g being looked at, which is wider than the first surface S1 of the object OBJ_g being looked at. Therefore, in the example shown in FIG. 16(b), the number of pixels in which at least a portion of the object OBJ_g being looked at is reflected in the image represented by the image data IMG_2D generated by the image capture device 21 located at the fourth position P4 is greater than the number of pixels in which at least a portion of the object OBJ_g being looked at is reflected in the image represented by the image data IMG_2D generated by the image capture device 21 located at the third position P3. Furthermore, the proportion of the object OBJ_g to be looked at shown in the imaging field of view of the imaging device 22 located at the fourth position P4 is greater than the proportion of the object OBJ_g to be looked at shown in the imaging field of view of the imaging device 22 located at the third position P3.

[0275] Note that the number of pixels in which at least a portion of the target object OBJ is reflected in the image represented by the image data IMG_2D increases as the area of ​​the portion of the target object OBJ reflected in the image represented by the image data IMG_2D increases. Therefore, the movement control unit 314b may set the movement mode of the imaging system 2 so that the area of ​​the portion of the peering target object OBJ_g reflected in the image represented by the image data IMG_2D acquired again in step S1 after the imaging system 2 has moved is larger than the area of ​​the portion of the peering target object OBJ_g reflected in the image represented by the image data IMG_2D already acquired in step S1 before the imaging system 2 moved. As a result, the position and orientation calculation including the 2D matching process is more likely to result in a good result.

[0276] Furthermore, the number of pixels in which at least a portion of the target object OBJ is reflected in the image indicated by the image data IMG_2D may be considered to be equivalent to the number of points of the target object OBJ reflected in the image indicated by the image data IMG_2D (i.e., the points of the target object OBJ imaged by the imaging device 21). For this reason, the movement control unit 314b may set the movement mode of the imaging system 2 so that the number of points of the peering target object OBJ_g reflected in the image indicated by the image data IMG_2D newly acquired in step S1 after the imaging system 2 has moved (i.e., the points of the peering target object OBJ_g imaged by the imaging device 21) is greater than the number of points of the peering target object OBJ_g reflected in the image indicated by the image data IMG_2D already acquired in step S1 befor...

Claims

1. A control device provided with a processing device and an imaging system for processing a target object, and which generates a control signal for controlling a robot which moves the processing device and the imaging system, the control device comprising: an arithmetic device which generates the control signal; and a communication device which outputs the control signal generated by the arithmetic device, wherein the arithmetic device, when it is determined that there is no target object which can be processed by the processing device, based on first image data generated by the imaging system imaging the target object when the positional relationship between the target object and the imaging system is a first positional relationship, generates a first control signal as the control signal, based on second image data generated by the imaging system imaging the target object, for controlling the robot so that the positional relationship changes to a second positional relationship different from the first positional relationship, and generates the second control signal as the control signal, based on position data and orientation data indicating the position and orientation of the target object which are generated based on third image data generated by the imaging system imaging the target object when the positional relationship is the second positional relationship.

2. The control device according to claim 1, wherein the first control signal is a signal for controlling the robot so as to change the position and orientation of the imaging system relative to the target object.

3. A control device according to claim 1 or 2, wherein the first control signal is a signal for controlling the robot so that the imaging direction of the imaging system relative to the target object changes from a first direction to a second direction.

4. The control device according to claim 3, wherein at least one of the first image data and the second image data is generated by the imaging system capturing an image of the target object from the first direction.

5. A control device as described in any one of claims 1 to 4, wherein, when the position data and orientation data indicating the position and orientation of the target object calculated based on the third image data are second position data and second orientation data, the calculation device generates first position data and first orientation data indicating the position and orientation of the target object based on the second image data, and generates the first control signal based on the first position data and first orientation data and predetermined position and orientation information indicating the relative position and orientation of the target object and the imaging system.

6. The control device according to claim 2, wherein the position and orientation information is set in advance by a user through input.

7. A control device described in any one of claims 1 to 6, wherein the proportion of the target object occupying the imaging field of view of the imaging system in the second positional relationship is greater than the proportion of the target object occupying the imaging field of view of the imaging system in the first positional relationship.

8. The control device described in claim 7, wherein the first control signal is a signal for controlling the robot so that the proportion of the target object occupying the imaging field of view in the second positional relationship is greater than the proportion of the target object occupying the imaging field of view in the first positional relationship.

9. A control device as described in any one of claims 1 to 8, wherein the calculation device determines that there is no target object that can be processed by the processing device based on first three-dimensional position data generated from the first image data and indicating the three-dimensional position of each of the multiple points of the target object, calculates the position data and the attitude data of the target object based on second three-dimensional position data generated from the third image data and indicating the three-dimensional position of each of the multiple points of the target object, and the number of points of the target object whose three-dimensional positions are indicated by the second three-dimensional position data is greater than the number of points of the target object whose three-dimensional positions are indicated by the first three-dimensional position data.

10. The control device according to claim 9, wherein the first control signal is a signal for controlling the robot so that the number of points of the target object whose three-dimensional positions are indicated by the second three-dimensional position data is greater than the number of points of the target object whose three-dimensional positions are indicated by the first three-dimensional position data.

11. A control device according to any one of claims 1 to 10, wherein the first control signal is a signal for controlling the robot to rotate the imaging system around a predetermined rotation axis.

12. A control device as claimed in any one of claims 1 to 10, wherein the first control signal is a signal for controlling the robot to rotate and move the imaging system so that an optical axis direction extending from the imaging system along the optical axis of the imaging system changes relative to the target object.

13. A control device as claimed in any one of claims 1 to 12, wherein the calculation device generates the position data and the attitude data of the target object based on partial model data indicating a part of a model of the target object and the third image data, and generates the second control signal based on the generated position data and attitude data.

14. A control device as described in any one of claims 1 to 13, wherein when the position data and orientation data indicating the position and orientation of the target object generated based on the third image data are second position data and second orientation data, the calculation device generates first position data and first orientation data indicating the position and orientation of the target object based on model data indicating a model of the target object and the second image data, and generates the first control signal based on the generated first position data and first orientation data.

15. A control device as described in any one of claims 1 to 12, wherein the calculation device generates first position data and first orientation data indicating the position and orientation of the target object based on model data indicating a model of the target object and the second image data, generates the first control signal based on the first position data and the first orientation data, generates the second position data and the second orientation data as the position data and orientation data of the target object based on partial model data indicating a part of the model of the target object and the third image data, and generates the second control signal based on the generated second position data and second orientation data.

16. A control device as described in any one of claims 13 to 15, wherein the part of the model represented by the partial model data is a part corresponding to an area where the density of points representing the three-dimensional position of the target object represented by the three-dimensional position data generated from the third image data is equal to or greater than a threshold value.

17. A control device as described in any one of claims 13 to 16, wherein the portion of the model indicated by the partial model data is a portion of the model excluding a portion corresponding to an area where the density of points representing the three-dimensional position of the target object indicated by the three-dimensional position data generated from the third image data is less than a threshold value.

18. A control device as described in any one of claims 13 to 17, wherein the partial model data is generated based on simulation image data generated by assuming that the imaging system captures the target object when the target object and the imaging system are in a predetermined positional relationship.

19. The control device according to claim 18, wherein the predetermined positional relationship is a positional relationship based on position and orientation information that indicates a relative position and orientation between the target object and the imaging system that is set in advance.

20. The control device according to claim 18 or 19, wherein the partial model data is generated based on the contrast of an image represented by the simulation image data.

21. The control device according to any one of claims 18 to 20, wherein the arithmetic unit generates the partial model data based on the simulation image data.

22. A control device according to any one of claims 13 to 17, wherein the calculation device generates the partial model data based on at least one of information regarding an area to be excluded from the model and information regarding an area to be left in the model, input in advance by a user.

23. A control device as described in any one of claims 13 to 22, wherein the calculation device calculates the position data and the attitude data of the target object based on the three-dimensional position data generated from the third image data and indicating the three-dimensional positions of each of multiple points of the target object, and the partial model data.

24. A control device according to any one of claims 1 to 23, wherein the second control signal is a signal for controlling the robot so that the processing device approaches the target object in order to perform the processing on the target object.

25. A control device according to any one of claims 1 to 24, wherein the second image data is generated by the imaging system capturing an image of the target object when the target object is in the first positional relationship.

26. A control device according to any one of claims 1 to 25, wherein the second image data is generated by the imaging system capturing an image of the target object when the positional relationship is different from the second positional relationship.

27. A control device according to any one of claims 1 to 26, wherein the imaging system generates the second image data by imaging the target object after determining, based on the first image data, that there is no target object that can be processed by the processing device.

28. A control device according to any one of claims 1 to 26, wherein the imaging system generates the second image data by imaging the target object based on the first image data before determining that there is no target object that can be processed by the processing device.

29. The control device according to claim 28, wherein the imaging system generates the second image data together with the first image data by imaging the target object when the imaging system is in the first positional relationship.

30. A control device according to any one of claims 1 to 29, wherein the calculation device determines that there is no target object that can be processed by the processing device includes the calculation device determining that the position and orientation calculation processing of the target object based on the first image data is defective.

31. A control device as described in any one of claims 1 to 30, wherein the imaging system includes a first imaging device and a second imaging device different from the first imaging device, the first imaging device generates the first image data by imaging the target object when in the first positional relationship, the second imaging device generates the second image data by imaging the target object, and the first imaging device generates the third image data by imaging the target object when in the second positional relationship.

32. The control device according to claim 31, wherein the first imaging device is one of a stereo camera having two monocular cameras and a monocular camera different from the two monocular cameras, and the second imaging device is the other of the stereo camera and the monocular camera different from the two monocular cameras.

33. The control device according to claim 32, wherein the first imaging device is the stereo camera, and the second imaging device is a monocular camera different from the two monocular cameras.

34. The control device according to any one of claims 1 to 30, wherein the imaging system is a stereo camera having two monocular cameras, or a monocular camera.

35. The control device described in claim 34, wherein the imaging system includes the stereo camera, and the two monocular cameras of the stereo camera generate the first image data including two pieces of image data generated from each of the two monocular cameras by imaging the target object when in the first positional relationship, and the two monocular cameras of the stereo camera generate the second image data including at least one of the two pieces of image data generated from each of the two monocular cameras by imaging the target object.

36. The control device described in claim 35, wherein the two monocular cameras of the stereo camera generate the second image data including two image data generated from each of the two monocular cameras by capturing an image of the target object when in the first positional relationship, and the second image data is the first image data.

37. The control device described in claim 34, wherein the imaging system includes the monocular camera, the monocular camera generates the first image data by imaging the target object when in the first positional relationship, and the monocular camera generates the second image data by imaging the target object.

38. The control device according to claim 37, wherein the monocular camera generates the second image data by capturing an image of the target object when in the first positional relationship, and the second image data is the first image data.

39. A control device as described in any one of claims 1 to 4, wherein, when the position data and orientation data indicating the position and orientation of the target object calculated based on the third image data are second position data and second orientation data, the calculation device generates first position data and first orientation data indicating the position and orientation of the target object based on the second image data, and generates the first control signal based on the first position data and the first orientation data and predetermined position and orientation information indicating the relative position and orientation of the target object and the imaging system.

40. A control system comprising: a control device according to any one of claims 1 to 39; and the imaging system.

41. A robot system comprising: a control device according to any one of claims 1 to 39; the imaging system; and the robot.

42. A control method for generating a control signal for controlling a robot provided with a processing device for processing a target object and an imaging system and for moving the processing device and the imaging system, the control method including: when it is determined that there is no target object that can be processed by the processing device based on first image data generated by the imaging system imaging the target object when the positional relationship between the target object and the imaging system is in a first positional relationship, generating a first control signal as the control signal based on second image data generated by the imaging system imaging the target object, for controlling the robot to change the positional relationship to a second positional relationship different from the first positional relationship; and generating a second control signal as the control signal based on position data and orientation data indicating the position and orientation of the target object, which are generated based on third image data generated by the imaging system imaging the target object when in the second positional relationship.

43. A computer program for causing a computer to execute the control method according to claim 42.

44. A control device provided with a processing device that processes a target object and an imaging system, and that generates a control signal for controlling a robot that moves the processing device and the imaging system, comprising: an arithmetic device that generates the control signal; and a communication device that outputs the control signal generated by the arithmetic device, wherein the arithmetic device, when it is determined based on first image data generated by the imaging system imaging a first target object group including a plurality of the target objects when the imaging system is in a first position and a first orientation that there is no target object that can be processed by the processing device among the first target object group, generates a first control signal as the control signal based on second image data generated by the imaging system imaging a second target object group including the plurality of the target objects, for controlling the robot so that the imaging system changes to a second position and a second orientation different from the first position and the first orientation.

45. The control device according to claim 44, wherein the first control signal is a signal for controlling the robot so as to change the position and orientation of the imaging system relative to one of the second group of target objects.

46. ​​A control device as described in claim 44 or 45, wherein the first control signal is a signal for controlling the robot so that the imaging direction of the imaging system for one of the second target objects is changed from a first direction to a second direction.

47. The control device according to claim 46, wherein the first group of target objects includes the one target object, and the first image data is generated by the imaging system capturing an image of the one target object from the first direction.

48. The control device according to claim 46 or 47, wherein the second image data is generated by the imaging system capturing an image of the one target object from the first direction.

49. A control device as described in any one of claims 44 to 48, wherein the first control signal is a signal for controlling the robot so that a proportion of one of the second group of target objects that is included in the imaging field of view of the imaging system after the imaging system is moved by control of the robot based on the first control signal is greater than a proportion of the one target object that is included in the imaging field of view before the imaging system is moved by control of the robot based on the first control signal.

50. The control device described in claim 44 or 45, wherein the first control signal is a signal for controlling the robot to move to a position and orientation where the imaging system can image one of the second group of target objects from a first direction, and the computing device generates as the control signal a second control signal for controlling the robot to move to a position and orientation where the imaging system can image the one target object from the first direction after controlling the robot based on the first control signal and moving the robot to a position and orientation where the imaging system can image the one target object from the first direction.

51. The control device according to any one of claims 44 to 50, wherein at least one of the target objects in the second group of target objects is the same as at least one of the target objects in the first group of target objects.

52. A control device as described in any one of claims 44 to 51, wherein the calculation device generates the first control signal based on third position data and third orientation data, which are generated based on the second image data and indicate the position and orientation of one of the second target object group, and based on predetermined relative position and orientation information, which indicates the relative position and orientation between the target object and the imaging system.

53. The control device according to claim 52, wherein the relative position and orientation information is set in advance by a user through input.

54. A control device as described in any one of claims 44 to 53, wherein when the computing device determines, based on third image data generated by the imaging system imaging a third group of targets including a plurality of the target objects when the computing device is at the second position and the second orientation, that there is no target object among the third group of targets that can be processed by the processing device, the computing device generates, as the control signal, a second control signal for controlling the robot so that the imaging system changes to a fourth position and a fourth orientation different from the second position and the second orientation.

55. The control device according to claim 54, wherein the arithmetic unit generates the second control signal without using new image data as an imaging result of the imaging system.

56. A control device as described in claim 54 or 55, wherein the calculation device generates the first control signal based on third position data and third orientation data indicating the position and orientation of one of the second target object group, which are generated based on the second image data, and based on predetermined relative position and orientation information indicating the relative position and orientation between the target object and the imaging system; and generates the second control signal based on predetermined change information indicating at least one of the direction and amount of change in at least one of the relative position and orientation between the target object and the imaging system.

57. The control device according to claim 56, wherein the change information is set by a user inputting it in advance.

58. A control device as described in any one of claims 54 to 57, wherein the calculation device generates a third control signal as the control signal based on fifth position data and fifth orientation data indicating the position and orientation of one of the target objects in the fourth target object group, the fifth control signal being generated based on fourth image data generated by the imaging system imaging the fourth target object group including the plurality of target objects when the imaging system is at the fourth position and the fourth orientation.

59. The control device according to claim 58, wherein the third control signal is a signal for controlling the robot so that the processing device approaches the one target object of the fourth target object group in order to perform the processing on the target object.

60. A control device as described in claim 58 or 59, wherein the calculation device generates the fifth position data and the fifth orientation data of the one target object in the fourth target object group based on partial model data indicating a part of a model of the target object and the fourth image data, and generates the third control signal based on the generated fifth position data and fifth orientation data.

61. A control device as described in claim 60, wherein the part of the model represented by the partial model data is a part corresponding to an area where the density of points representing the three-dimensional position of one of the fourth target object group represented by the three-dimensional position data generated from the fourth image data is equal to or greater than a threshold value.

62. A control device as described in claim 60 or 61, wherein the part of the model represented by the partial model data is a part of the model excluding a part corresponding to an area where the density of points representing the three-dimensional position of one of the target objects in the fourth target object group represented by the three-dimensional position data generated from the fourth image data is less than a threshold value.

63. The control device according to any one of claims 58 to 62, wherein the arithmetic unit generates the first control signal based on model data indicative of the model of the target object and the second image data.

64. A control device as described in any one of claims 58 to 63, wherein the calculation device determines whether or not at least one target object in the first group of target objects can be processed by the processing device based on model data indicating the model of the target object and the first image data, and determines that there is no target object that can be processed by the processing device.

65. A control device as described in any one of claims 58 to 64, wherein the calculation device determines whether at least one of the target objects in the third target object group can be processed by the processing device based on first partial model data indicating a part of the model of the target object and the third image data, and as a result, determines that there is no target object in the third target object group that can be processed by the processing device; generates the fifth position data and the fifth orientation data of the one target object in the fourth target object group based on second partial model data indicating a part of the model and the fourth image data; and generates the third control signal based on the generated fifth position data and fifth orientation data.

66. The control device according to claim 62, wherein the first part model data and the second part model data are the same data representing the same part of the model.

67. The control device according to claim 65, wherein the second partial model data is data representing a part of the model that overlaps with or is different from a part of the part of the model represented by the first partial model data.

68. A control device as described in any one of claims 65 to 67, wherein when the third target object group includes a second target object as the target object, and the one target object in the fourth target object group is the fourth target object, the calculation device determines, based on the first partial model data and the third image data, whether the second target object, which is one of the target objects in the third target object group, can be processed by the processing device, and determines that there is no second target object in the third target object group that can be processed by the processing device; and generates the third control signal based on the fifth position data and the fifth attitude data of the fourth target object generated based on the second partial model data and the fourth image data.

69. The control device according to claim 68, wherein the fourth object is the second object.

70. The control device according to claim 68, wherein the fourth object is a different object than the second object.

71. A control device as described in any one of claims 68 to 70, wherein the calculation device generates the first control signal based on third position data and third orientation data indicating the position and orientation of a first target object, which is one of the second target object group, generated based on the second image data, and generates the second control signal for controlling the robot so as to change the position and orientation of the imaging system relative to a third target object, which is one of the third target object group.

72. The control device according to claim 71, wherein the fourth target object is the same target object as the first target object, the second target object, and the third target object.

73. The control device according to claim 71, wherein the fourth target object is different from at least one of the first target object, the second target object, and the third target object.

74. A control device as described in any one of claims 65 to 73, wherein the second partial model data is data indicating a part of the model that overlaps with or is different from a part of the part of the model indicated by the first partial model data, the first partial model data is generated based on first simulation image data generated by assuming that the imaging system captures the target object when the target object and the imaging system are in a first positional relationship, and the second partial model data is generated based on second simulation image data generated by assuming that the imaging system captures the target object when the target object and the imaging system are in a second positional relationship different from the first positional relationship.

75. The control device according to claim 74, wherein the arithmetic device generates the first partial model data based on the first simulation image data, and generates the second partial model data based on the second simulation image data.

76. A control device as described in claim 74 or 75, wherein each of the first positional relationship and the second positional relationship is a positional relationship based on predetermined relative position and orientation information indicating the relative position and orientation between the target object and the imaging system.

77. The control device described in claim 76, wherein the first positional relationship is a positional relationship based on the relative position and orientation information, and the second positional relationship is a positional relationship based on the relative position and orientation information and predetermined change information indicating at least one of a direction and an amount of change in at least one of the relative position and orientation between the target object and the imaging system.

78. A control device as described in any one of claims 65 to 73, wherein the second partial model data is data indicating a part of the model that overlaps with or is different from a part of the part of the model indicated by the first partial model data, and the calculation device generates the one-part model data based on at least one of first exclusion information regarding an area to be excluded from the model and first remaining information regarding an area to be left of the model, both input in advance by a user, and generates the two-part model data based on at least one of second exclusion information and second remaining information, both input in advance by a user.

79. The control device described in any one of claims 54 to 57, wherein the imaging system includes a first imaging device and a second imaging device different from the first imaging device, the first imaging device generates the first image data and the third image data, the second imaging device generates the second image data, the first imaging device is one of a stereo camera having two monocular cameras and a monocular camera different from the two monocular cameras, and the second imaging device is the other of the stereo camera and the monocular camera different from the two monocular cameras.

80. The control device described in any one of claims 58 to 78, wherein the imaging system includes a first imaging device and a second imaging device different from the first imaging device, the first imaging device generates the first image data, the third image data, and the fourth image data, the second imaging device generates the second image data, the first imaging device is one of a stereo camera having two monocular cameras and a monocular camera different from the two monocular cameras, and the second imaging device is the other of the stereo camera and the monocular camera different from the two monocular cameras.

81. A control device according to claim 79 or 80, wherein the first imaging device is the stereo camera, and the second imaging device is a monocular camera different from the two monocular cameras.

82. A control device as described in any one of claims 54 to 56, wherein when the computing device determines, based on fourth image data generated by the imaging system imaging a fourth group of targets including a plurality of the target objects when the imaging system is at the fourth position and fourth orientation, that there is no target object among the fourth group of targets that can be processed by the processing device, the computing device generates, as the control signal, a fourth control signal for controlling the robot so that the imaging system changes from the fourth position and fourth orientation to a fifth position and fifth orientation indicating a predetermined position and orientation for re-imaging.

83. The control device according to claim 82, wherein the arithmetic unit generates the fourth control signal without using new image data as an imaging result of the imaging system.

84. The control device described in claim 83, wherein the fourth control signal is a control signal for controlling the robot so that at least half of the target objects of the fifth target object group included in the imaging field of view of the imaging system at the fifth position and the fifth orientation are different target objects from the target objects of the fourth target object group included in the imaging field of view at the fourth position and the fourth orientation.

85. The control device described in claim 84, wherein the fourth control signal is a control signal for controlling the robot so that all of the target objects of the fifth group of target objects included in the imaging field of view at the fifth position and the fifth orientation are different target objects from the target objects of the fourth group of target objects included in the imaging field of view at the fourth position and the fourth orientation.

86. A control device according to any one of claims 82 to 85, wherein the calculation device sets the fifth position and the fifth orientation based on re-imaging position and orientation information indicating the position and orientation at which re-imaging is performed by the imaging system, the re-imaging position and orientation information having been input in advance by the user.

87. A control device according to any one of claims 82 to 85, wherein the calculation device sets the fifth position and the fifth attitude based on an imaging field of view of the imaging system.

88. The control device described in claim 87, wherein at least the plurality of target objects of the fourth target object group and the plurality of target objects of the fifth target object group are contained in a container, and the computing device sets the fifth position and the fifth attitude based on the imaging field of view and dimensions of the container.

89. A control device as described in any one of claims 82 to 88, wherein the calculation device generates a fifth control signal as the control signal based on sixth position data and sixth orientation data indicating the position and orientation of one of the target objects in the fifth target object group, the sixth position data and sixth orientation data being generated based on fifth image data generated by the imaging system imaging a fifth target object group including the plurality of target objects when the imaging system is at the fifth position and the fifth orientation.

90. The control device according to claim 89, wherein the fifth control signal is a signal for controlling the robot so that the processing device approaches the one target object of the fifth target object group in order to perform the processing on the target object.

91. The control device described in claim 89 or 90, wherein the imaging system includes a first imaging device and a second imaging device different from the first imaging device, the first imaging device generates the first image data, the third image data, the fourth image data, and the fifth image data, the second imaging device generates the second image data, the first imaging device is one of a stereo camera having two monocular cameras and a monocular camera different from the two monocular cameras, and the second imaging device is the other of the stereo camera and the monocular camera different from the two monocular cameras.

92. A control device according to any one of claims 44 to 91, wherein the imaging system is a stereo camera having two monocular cameras, or a monocular camera.

93. A control system comprising: a control device according to any one of claims 44 to 92; and the imaging system.

94. A robot system comprising: a control device according to any one of claims 44 to 92; the imaging system; and the robot.

95. A control method for generating a control signal for controlling a robot provided with a processing device for processing a target object and an imaging system, and for moving the processing device and the imaging system, the control method including: when it is determined based on first image data generated by the imaging system imaging a first target object group including the plurality of target objects when the imaging system is in a first position and a first attitude that there is no target object that can be processed by the processing device among the first target object group, the control method including generating a first control signal as the control signal for controlling the robot so that the imaging system changes to a second position and a second attitude different from the first position and the first attitude, based on second image data generated by the imaging system imaging a second target object group including the plurality of target objects.

96. A computer program for causing a computer to execute the control method according to claim 95.

97. A control device provided with a processing device for processing a target object and an imaging system, and for generating a control signal for controlling a robot which moves the processing device and the imaging system, the control device comprising: a calculation device which generates the control signal based on first three-dimensional position data indicating the three-dimensional position of each of a plurality of points of the first object group, which are generated from an imaging result of a first object group including a plurality of the target objects by the imaging system, and second three-dimensional position data which has less three-dimensional position data than the first three-dimensional position data and indicates the three-dimensional position of each of the plurality of points of the first object group; and a communication device which outputs the control signal generated by the calculation device, wherein the calculation device generates a plurality of first position data and first orientation data indicating the position and orientation of each of the target objects of a second object group consisting of a plurality of the target objects of the first object group, based on the second three-dimensional position data; generates second position data and second orientation data indicating the position and orientation of one of the target objects of the second object group, based on the generated plurality of first position data and first orientation data and the first three-dimensional position data; and generates the control signal based on the generated second position data and second orientation data.

98. The control device described in claim 97, wherein the calculation device generates the first position data and the first attitude data for each of the target objects in the second object group based on partial three-dimensional model data showing a portion of a three-dimensional model of the target object and the second three-dimensional position data.

99. The control device described in claim 98, wherein the calculation device generates the first position data and the first attitude data of each of the target objects in the second object group based on a plurality of partial three-dimensional model data each showing a different portion of the three-dimensional model and the second three-dimensional position data.

100. A control device according to claim 99, wherein each of the plurality of partial three-dimensional model data is assigned a priority used for generating the first position data and the first orientation data.

101. The control device according to claim 99 or 100, wherein the plurality of partial three-dimensional model data include first partial three-dimensional model data and second partial three-dimensional model data, and the calculation device performs a first position and orientation calculation process of a plurality of target objects of the first target object group based on the first partial three-dimensional model data and the second three-dimensional position data, and then a second position and orientation calculation process of at least one of the target objects of the first object group based on the second partial three-dimensional model data and the second three-dimensional position data.

102. The control device according to claim 100, wherein the plurality of partial three-dimensional model data include first partial three-dimensional model data assigned a first priority as the priority, and second partial three-dimensional model data assigned a second priority lower than the first priority, and the calculation device performs a second position and orientation calculation process of at least one of the target objects in the first object group based on the second partial three-dimensional model data assigned a second priority lower than the first priority and the second three-dimensional position data when the number of the target objects for which the first position and orientation calculation process based on the first partial three-dimensional model data assigned the first priority and the second three-dimensional position data is good is less than a threshold value.

103. The control device described in claim 102, wherein the target object for which the first position and orientation calculation processing is favorable includes a target object for which the similarity between a part of the three-dimensional model represented by the first partial three-dimensional model data, calculated based on the first partial three-dimensional model data and the second three-dimensional position data, and multiple points indicating the three-dimensional positions of each of the target objects in the first object group represented by the second three-dimensional position data, is equal to or greater than a threshold value.

104. A control device as described in any one of claims 101 to 103, wherein the first position data and the first attitude data of each of the target objects in the second object group include the first position data and the first attitude data of multiple target objects in the second object group generated by the first position and attitude calculation process, and the first position data and the first attitude data of at least one target object in the second object group generated by the second position and attitude calculation process.

105. A control device as described in any one of claims 101 to 104, wherein the calculation device generates the second position data and the second orientation data based on the first position data and the first orientation data of multiple target objects in the second object group generated by the first position and orientation calculation process, the first position data and the first orientation data of at least one target object in the second object group generated by the second position and orientation calculation process, and the first three-dimensional position data.

106. A control device as described in any one of claims 101 to 105, wherein the calculation device generates the second position data and the second orientation data based on the first position data and the first orientation data of multiple target objects in the second object group generated by the first position and orientation calculation process, the first position data and the first orientation data of at least one target object in the second object group generated by the second position and orientation calculation process, the first three-dimensional position data, the first partial three-dimensional model data, and the second partial three-dimensional model data.

107. A control device as described in any one of claims 101 to 106, wherein the calculation device selects the one target object from the second object group based on the first position data and the first orientation data of multiple target objects in the second object group generated by the first position and orientation calculation process, and the first position data and the first orientation data of at least one target object in the second object group generated by the second position and orientation calculation process, and the first three-dimensional position data.

108. A control device as described in any one of claims 101 to 107, wherein the calculation device calculates, based on the first partial three-dimensional model data and the second three-dimensional position data, a first similarity between a portion of the three-dimensional model represented by the first partial three-dimensional model data and a plurality of points indicating the three-dimensional position of each of the target objects in the first object group indicated by the second three-dimensional position data; calculates, based on the second partial three-dimensional model data and the second three-dimensional position data, a second similarity between a portion of the three-dimensional model represented by the second partial three-dimensional model data and a plurality of points indicating the three-dimensional position of each of the target objects in the first object group indicated by the second three-dimensional position data; the first similarity corresponding to the target object for which the first position data and the first orientation data are calculated by the first position and orientation calculation process is equal to or greater than a threshold; and the second similarity corresponding to the target object for which the first position data and the first orientation data are calculated by the second position and orientation calculation process is equal to or greater than a threshold.

109. A control device as described in any one of claims 101 to 108, wherein the calculation device selects respective portions of the first three-dimensional position data as multiple first focus data including data corresponding to at least a portion of a plurality of points of each of multiple target objects in the second object group based on the multiple first position data and the first orientation data generated by the first position and orientation calculation process, selects at least a portion of the first three-dimensional position data as at least one second focus data including data corresponding to at least a portion of a plurality of points of at least one target object in the second object group based on at least one of the first position data and the first orientation data generated by the second position and orientation calculation process, and selects the one target object from the second object group based on the first partial three-dimensional model data, the second partial three-dimensional model data, the multiple first focus data, and the at least one second focus data.

110. The control device described in claim 109, wherein the calculation device selects the one target object from the second object group based on a similarity between a plurality of points indicating the three-dimensional positions of each of the plurality of target objects in the second object group, indicated by each of the plurality of first attention data, and a portion of the three-dimensional model indicated by the first partial three-dimensional model data, and a similarity between a plurality of points indicating the three-dimensional position of at least one target object in the second object group, indicated by the at least one second attention data, and a portion of the three-dimensional model indicated by the second partial three-dimensional model data.

111. A control device as described in claim 109 or 110, wherein the calculation device generates the second position data and the second attitude data based on one of the plurality of first attention data corresponding to the one selected target object and the first partial three-dimensional model data.

112. A control device as described in claim 109 or 110, wherein the calculation device generates the second position data and the second attitude data based on one of the at least one second attention data corresponding to the one selected target object and the second partial three-dimensional model data.

113. A control device as described in any one of claims 97 to 104, wherein the calculation device generates the second position data and the second orientation data based on partial three-dimensional model data indicating a portion of a three-dimensional model of the target object, the generated plurality of first position data and first orientation data, and the first three-dimensional position data.

114. The control device according to claim 113, wherein the calculation device generates the second position data and the second orientation data based on a plurality of partial three-dimensional model data each showing a different portion of the three-dimensional model, the generated plurality of first position data and first orientation data, and the first three-dimensional position data.

115. The control device described in claim 113 or 114, wherein the calculation device selects the one target object from the second object group based on a plurality of partial three-dimensional model data each indicating a different portion of the three-dimensional model, the generated plurality of first position data and first orientation data, and the first three-dimensional position data, and generates the second position data and the second orientation data based on one of the plurality of first position data and first orientation data corresponding to the one target object selected, and one of the plurality of partial three-dimensional model data.

116. The control device described in claim 97, wherein the calculation device selects the one target object from the second object group based on the generated plurality of first position data and first orientation data and the first three-dimensional position data, and generates the second position data and the second orientation data based on the first position data and the first orientation data of the selected one target object and the first three-dimensional position data.

117. The control device described in claim 116, wherein the calculation device selects respective portions of the first three-dimensional position data as a plurality of focus data including data corresponding to at least a portion of a plurality of points of each of the target objects in the second object group imaged by the imaging system based on the generated plurality of first position data and the first attitude data, and selects the one target object from the second object group based on model data indicating a three-dimensional model of the target object and the selected plurality of focus data.

118. The control device according to claim 117, wherein the calculation device selects the one target object from the second object group based on the similarity between the three-dimensional model and multiple points indicating the three-dimensional positions of each of the target objects in the second object group, as indicated by each of the multiple pieces of attention data.

119. A control device according to claim 117 or 118, wherein the arithmetic unit generates the second position data and the second attitude data based on the model data and the attention data corresponding to the one target object.

120. A control device as described in any one of claims 97 and 116 to 119, wherein the calculation device selects respective portions of the first three-dimensional position data as a plurality of focus data including data corresponding to at least a portion of a plurality of points of each of the target objects in the second object group imaged by the imaging system based on the generated plurality of first position data and first orientation data, and generates second position data and second orientation data indicating the position and orientation of the one target object in the second object group based on the plurality of focus data.

121. A control device as described in any one of claims 97 to 120, wherein the imaging system includes a stereo camera, and the computing device generates the first three-dimensional position data and the second three-dimensional position data based on stereo image data generated as the imaging result by the stereo camera.

122. The control device described in claim 121, wherein the calculation device generates depth image data in which distance information is associated with each pixel based on the stereo image data, and generates the first three-dimensional position data and the second three-dimensional position data based on the depth image data.

123. The control device described in claim 121, wherein the calculation device generates first depth image data based on stereo image data, in which distance information is associated with each pixel; generates second depth image data by reducing some of the pixel data of the generated first depth image data; generates the first three-dimensional position data based on the first depth image data; and generates the second three-dimensional position data based on the second depth image data.

124. The control device according to claim 121, wherein the calculation device generates the second three-dimensional position data by reducing data indicating a three-dimensional position of a portion of the first three-dimensional position data.

125. The control device described in claim 121, wherein first stereo image data and second stereo image data having less pixel data than the first stereo image data are generated as the imaging result by the stereo camera, and the calculation device generates the first three-dimensional position data based on the first stereo image data, and generates the second three-dimensional position data based on the second stereo image data.

126. A control system comprising: a control device according to any one of claims 97 to 125; and said imaging system.

127. A robot system comprising: a control device according to any one of claims 97 to 125; the imaging system; and the robot.

128. A control method for generating a control signal for controlling a robot provided with a processing device for processing a target object and an imaging system and for moving the processing device and the imaging system, the control signal being generated based on first three-dimensional position data indicating the three-dimensional position of each of a plurality of points of the first object group, which are generated from an imaging result of a first object group including a plurality of the target objects by the imaging system, and second three-dimensional position data having less three-dimensional position data than the first three-dimensional position data and indicating the three-dimensional position of each of the plurality of points of the first object group, wherein generating the control signal includes: generating a plurality of first position data and first orientation data indicating the position and orientation of each of the target objects of a second object group consisting of a plurality of the target objects of the first object group, based on the second three-dimensional position data; generating second position data and second orientation data indicating the position and orientation of one of the target objects of the second object group, based on the generated plurality of first position data and first orientation data and the first three-dimensional position data; and generating the control signal based on the generated second position data and second orientation data.

129. A computer program product for causing a computer to execute the control method recited in claim 128.

130. A control device provided with a processing device and an imaging system for processing target objects contained in a container, and which generates a control signal for controlling a robot which moves the processing device and the imaging system, the control device comprising: an arithmetic device which generates the control signal; and a communication device which outputs the control signal generated by the arithmetic device, wherein the arithmetic device generates position and orientation data indicating at least one of the position and orientation of the container based on first image data generated by the imaging system imaging at least a portion of the container; generates three-dimensional position data indicating the three-dimensional positions of each of a plurality of points of the group of objects based on second image data generated by the imaging system imaging a group of objects including a group of target objects including at least a portion of the plurality of target objects contained in the container; selects a portion of the three-dimensional position data as selection data based on the position and orientation data; and generates the control signal for controlling the robot to cause the processing device to approach one of the group of target objects in order to process the one target object by the processing device, based on the selection data.

131. The control device of claim 130, wherein the group of objects includes at least the group of target objects and at least a portion of the container.

132. A control device as claimed in claim 130 or 131, wherein the selected selection data does not include data indicating the three-dimensional position of each of a plurality of points of at least a portion of the container.

133. A control device according to any one of claims 130 to 132, wherein the selected selection data includes data indicating the three-dimensional positions of each of a plurality of points of at least a portion of the group of target objects.

134. A control device as described in any one of claims 130 to 133, wherein the calculation device sets a designated area including at least a portion of a space within the container capable of accommodating multiple target objects based on the position and orientation data, and selects a portion of the three-dimensional position data as the selection data corresponding to the designated area.

135. The control device described in claim 134, wherein the group of objects includes at least the group of target objects and at least a portion of the container, and the set designated area does not include multiple points of at least a portion of the container indicated by the three-dimensional position data.

136. A control device as described in claim 134 or 135, wherein the set designated area includes a plurality of points of at least a part of the group of target objects contained in the container, as indicated by the three-dimensional position data.

137. A control device as described in any one of claims 134 to 136, wherein, when the container is a first container and the designated area is a first designated area, the calculation device changes a second designated area, which includes at least a portion of a space capable of accommodating a plurality of the target objects, in a second container different from the first container, to the first designated area based on the position and orientation data, and selects a portion of the three-dimensional position data as the selection data corresponding to the first designated area.

138. A control device according to claim 137, wherein the calculation device changes at least one of the position and orientation of the second designated area based on the position and orientation data, thereby changing the second designated area into the first designated area.

139. A control device as described in claim 137 or 138, wherein the group of objects includes at least the group of target objects and at least a portion of the first container, and the calculation device changes the second designated area to the first designated area based on the position and orientation data so that multiple points of at least a portion of the first container indicated by the three-dimensional position data are not included in the first designated area.

140. A control device as described in any one of claims 137 to 139, wherein the calculation device changes the second designated area to the first designated area based on the position and orientation data so that at least a portion of the target object group contained in the first container, as indicated by the three-dimensional position data, is included in the first designated area.

141. A control device as claimed in any one of claims 137 to 140, wherein at least one of a position and an orientation of the first container is different from at least one of a position and an orientation of the second container.

142. A control device as described in any one of claims 134 to 135, wherein, when the container is a first container and the designated area is a first designated area, the calculation device sets at least one of the position and orientation of a second designated area, which includes at least a portion of a space capable of accommodating a plurality of the target objects, in a second container different from the first container, as at least one of the position and orientation of the first designated area based on the position and orientation data, and selects a portion of the three-dimensional position data as the selection data corresponding to the first designated area.

143. The control device of claim 142, wherein the position and orientation of the first container is substantially the same as the position and orientation of the second container.

144. A control device according to any one of claims 137 to 143, wherein the second designated area is a preset initial designated area.

145. The control device according to claim 144, wherein the second designated area is designated in advance by a user.

146. A control device as described in claim 144 or 145, wherein the second container is a container used to set the second designated area.

147. A control device according to any one of claims 144 to 146, wherein the calculation device changes the second designated area to the first designated area by changing the second specified area from an initial position and orientation indicating at least one of an initial position and initial orientation of the second designated area to a first position and orientation indicating at least one of a first position and first orientation based on the position and orientation data.

148. A control device as described in any one of claims 137 to 143, wherein a plurality of the target objects contained in the second container are treated in the treatment device before the target objects contained in the first container are treated in the treatment device.

149. The control device described in claim 148, wherein the first container containing the plurality of target objects is replaced with the second container after the plurality of target objects contained in the second container are processed in the processing device.

150. A control device as described in claim 148 or 149, wherein, when the position and orientation data is first position and orientation data, before the target object contained in the first container is processed by the processing device, the second specified area is set based on second position and orientation data indicating at least one of the position and orientation of the second container generated based on third image data generated by the imaging system imaging at least a portion of the second container.

151. The control device of claim 150, wherein the target object group is a first target object group, the object group is a first object group, the three-dimensional position data is first three-dimensional position data, the selection data is first selection data, and the control signal is a first control signal, and before the target objects contained in the first container are processed by the processing device, the computing device selects as the second selection data corresponding to the second designated area a portion of second three-dimensional position data indicating the three-dimensional positions of each of a plurality of points of the second object group, the second three-dimensional position data being generated based on fourth image data generated by the imaging system imaging a second object group including a second object group including at least a portion of the plurality of target objects contained in the second container, and generates a second control signal for controlling the robot to approach one of the target objects of the second target object group in order to process the one target object by the processing device based on the second selection data.

152. A control device as described in claim 150 or 151, wherein the calculation device changes the second designated area to the first designated area by changing the second specified area from a second position and orientation indicating at least one of a second position and second orientation of the second designated area to a third position and orientation indicating at least one of a third position and third orientation based on the one position and orientation data.

153. A control device as described in any one of claims 134 to 136, 142 and 143, wherein, when the container is a first container and the designated area is a first designated area, the calculation device sets, based on the position and orientation data, a second designated area in a second container different from the first container, the second designated area including at least a portion of a space capable of accommodating a plurality of the target objects, as the first designated area, and selects a portion of the three-dimensional position data as the selection data corresponding to the first designated area.

154. A control device as claimed in any one of claims 137 to 153, wherein the dimensions of the second container are approximately the same as the dimensions of the first container.

155. A control device as claimed in any one of claims 130 to 136, wherein the computing device selects a portion of the three-dimensional position data as the selected data based on dimensional information relating to dimensions of the container.

156. The control device described in claim 155, wherein the calculation device sets a designated area including at least a portion of a space within the container capable of accommodating a plurality of the target objects based on the dimensional information, and selects a portion of the three-dimensional position data as the selection data corresponding to the designated area.

157. A control device as described in any one of claims 137 to 152, wherein the calculation device changes a second designated area, which includes at least a portion of a space within the second container capable of accommodating a plurality of the target objects, to the first designated area based on dimensional difference information regarding the difference between the dimensions of the first container and the dimensions of a second container different from the first container and the position and orientation data, and selects a portion of the three-dimensional position data as the selection data corresponding to the first designated area.

158. A control device as described in claim 157, wherein the calculation device changes the second designated area into the first designated area by changing the dimensions of the second designated area and at least one of the position and orientation of the second designated area based on the dimensional difference information and the position and orientation data.

159. A control device as described in claim 157 or 158, wherein the group of objects includes at least the group of target objects and at least a portion of the first container, and the calculation device changes the second designated area to the first designated area based on the dimensional difference information and the position and orientation data so that multiple points of at least a portion of the first container indicated by the three-dimensional position data are not included in the first designated area.

160. A control device as described in any one of claims 157 to 159, wherein the calculation device changes the second designated area to the first designated area based on the dimensional difference information and the position and orientation data so that at least a portion of the points of the group of target objects contained in the first container, as indicated by the three-dimensional position data, are included in the first designated area.

161. A control device as described in any one of claims 157 to 160, wherein the calculation device changes the second designated area from an initial dimension to a first dimension based on the dimensional difference information, and changes the second designated area to the first designated area by changing the second designated area from an initial position and orientation indicating at least one of an initial position and initial orientation of the second designated area to a first position and orientation indicating at least one of a first position and first orientation based on the position and orientation data.

162. A control device as described in any one of claims 157 to 161, wherein the calculation device changes the second designated area from the second dimension to a third dimension based on the dimensional difference information, and changes the second designated area from a second position and orientation indicating at least one of the second position and second orientation of the second designated area to a third position and orientation indicating at least one of the third position and third orientation based on the position and orientation data, thereby changing the second designated area to the first designated area.

163. A control device as claimed in any one of claims 157 to 162, wherein the dimensions of the second container are different from the dimensions of the first container.

164. A control device as described in any one of claims 137 to 154 and 157 to 163, wherein the position and orientation data is set as first position and orientation data, and after the processing device processes the multiple target objects contained in the first container, the calculation device generates third position and orientation data indicating at least one of the position and orientation of the third container based on fifth image data generated by the imaging system capturing an image of at least a portion of a third container that is different from the first container and the second container and contains the multiple target objects, and changes the second designated area to a third designated area including at least a portion of a space in the third container that can accommodate the multiple target objects based on the third position and orientation data.

165. A control device according to any one of claims 137 to 154 and 157 to 163, wherein the position and orientation data is set as first position and orientation data, and after the processing device processes the multiple target objects contained in the first container, the calculation device generates third position and orientation data indicating at least one of the position and orientation of the third container based on fifth image data generated by the imaging system capturing an image of at least a portion of a third container that is different from the first container and the second container and contains the multiple target objects, and changes the first designated area to a third designated area including at least a portion of a space in the third container that can accommodate the multiple target objects based on the third position and orientation data.

166. The control device according to claim 164 or 165, wherein when the target object group is a first target object group, the object group is the first object group, the three-dimensional position data is first three-dimensional position data, and the selection data is first selection data, the calculation device selects, as third selection data corresponding to the third designated area, a portion of third three-dimensional position data indicating the three-dimensional positions of each of a plurality of points of the third object group, which is generated based on sixth image data generated by the imaging system imaging a third object group including a third object group including at least a portion of the plurality of target objects contained in the third container, and generates, based on the third selection data, a third control signal for controlling the robot to cause the processing device to approach one of the target objects of the third target object group in order to process the one target object by the processing device.

167. The control device described in claim 166, wherein the third object group includes at least the third target object group and at least a portion of the third container, and the third designated area does not include multiple points of at least a portion of the third container indicated by the third three-dimensional position data.

168. A control device as described in claim 166 or 167, wherein the third specified area includes a plurality of points of at least a portion of the third target object group contained in the third container, as indicated by the third three-dimensional position data.

169. A control device as described in any one of claims 134 to 154 and 157 to 168, wherein the specified area includes a plurality of partial areas, and at least a portion of one of the plurality of partial areas is different from other partial areas of the plurality of partial areas.

170. A control device according to any one of claims 130 to 169, wherein the calculation device selects a portion of the three-dimensional position data as the selected data based on robot position and orientation data indicating at least one of the position and orientation of a reference point of the processing device and the position and orientation data.

171. A control device as described in any one of claims 134 to 154 and 157 to 169, wherein, when the designated area is a first designated area, the calculation device sets a fourth designated area including at least a portion of the processing device within the space in which the robot is placed based on robot position and orientation data indicating at least one of the position and orientation of a reference point of the processing device, and selects a portion of the three-dimensional position data as the selected data that does not correspond to the fourth designated area and corresponds to the first designated area.

172. A control device as described in claim 170 or 171, wherein the selection data does not include data indicating the three-dimensional position of each of at least a portion of a plurality of points of the processing device and data indicating the three-dimensional position of each of at least a portion of a plurality of points of the container.

173. A control device according to any one of claims 170 to 174, wherein the selection data includes data indicating the three-dimensional positions of each of a plurality of points of at least a portion of the group of target objects.

174. A control device as claimed in any one of claims 170 to 173, wherein the reference point of the processing device is a tool centre point of the processing device.

175. A control device according to any one of claims 130 to 174, wherein the processing device is a holding device capable of holding the target object.

176. A control system comprising: a control device according to any one of claims 130 to 175; and said imaging system.

177. A robot system comprising: a control device according to any one of claims 130 to 175; the imaging system; and the robot.

178. A control method for generating a control signal for controlling a robot provided with a processing device for processing a target object and an imaging system, and for moving the processing device and the imaging system, the control method comprising: generating position and orientation data indicating at least one of the position and orientation of the container based on first image data generated by the imaging system imaging at least a part of the container; generating three-dimensional position data indicating three-dimensional positions of each of a plurality of points of the object group based on second image data generated by the imaging system imaging a group of objects including a group of target objects including at least a part of the plurality of target objects contained in the container; selecting a part of the three-dimensional position data as selection data based on the position and orientation data; and generating the control signal for controlling the robot to approach one of the target objects in order to process the one target object by the processing device, based on the selection data.

179. A computer program product for causing a computer to execute the control method recited in claim 178.

180. A control device provided with a processing device for processing a target object and an imaging system, and generating a control signal for controlling a robot which moves the processing device and the imaging system, comprising: an arithmetic device which generates the control signal; and a communication device which outputs the control signal generated by the arithmetic device, wherein the arithmetic device generates the control signal for controlling the robot so that at least one of the position and orientation of the imaging system with respect to one of the target objects in the target object group is changed when the arithmetic device determines, based on an imaging result by the imaging system of a target object group including a plurality of the target objects, that at least one of the target objects in the target object group cannot be processed by the processing device.

Images