Control device, control system, robot system, control method, and computer program

Abstract

This control device generates a control signal for controlling a holding device, which is capable of holding a target object, and a robot, which is provided with the holding device and moves the holding device. The control device: selects, as a selection model, a model indicating a part of the target object, on the basis of a first imaging result obtained by imaging the target object, using an imaging system, before the holding device holds the target object; and generates a control signal for controlling the robot and the holding device in order to process the target object held by the holding device, such generating being executed on the basis of the selection model and a second imaging result obtained by imaging, using the imaging system, the target object which is held by the holding device through the controlling of the robot and the holding device.

Description

Control device, control system, robot system, control method, and computer program

[0001] The present invention relates to the technical fields of a control device, a control system, a robot system, a control method, and a computer program that can generate a control signal for controlling a robot, for example.

[0002] An example of a control device for controlling a robot provided with a holding device capable of holding an object is described in Patent Document 1. Such a control device is required to generate an appropriate control signal for processing the object held by the holding device.

[0003] US Patent Application Publication No. 2013 / 0230235

[0004] According to a first aspect, there is provided a robot provided with a holding device capable of holding an object and moving the holding device, and a control device generating a control signal for controlling at least one of the holding devices, the control device including: a calculation device generating the control signal; and a communication device outputting the control signal generated by the calculation device, wherein the calculation device selects a model representing a part of the object as a selected model based on a first imaging result obtained by imaging the object before it is held by the holding device with an imaging system, and generates the control signal for controlling at least one of the robot and the holding device to process the object held by the holding device based on a second imaging result obtained by imaging the object held by the imaging system with the robot and the selected model. According to a second aspect, there is provided a control system including the control device provided by the first aspect and the imaging system. According to a third aspect, there is provided a robot system including the control device provided by the first aspect, the imaging system, and the robot.

[0005] According to a fourth aspect, there is provided a control method for generating a control signal for controlling a robot provided with a holding device capable of holding an object and moving the holding device, and at least one of the holding device, the control method including: selecting a model representing a part of the object as a selected model based on a first imaging result obtained by imaging the object before being held by the holding device with an imaging system; and generating the control signal for controlling the robot and at least one of the holding device to process the object held by the holding device based on a second imaging result obtained by imaging the object held by the holding device with the imaging system by controlling at least one of the robot and the holding device. According to a fifth aspect, there is provided a computer program for causing a computer to execute the control method provided by the fourth aspect.

[0006] According to a sixth aspect, there is provided a robot provided with a holding device capable of holding an object and moving the holding device, and a control device generating a control signal for controlling at least one of the holding devices, the control device including: a calculation device generating the control signal; and a communication device outputting the control signal generated by the calculation device, wherein the calculation device controls at least one of the robot and the holding device to select a model representing a part of the object as a selected model based on a seventh imaging result of the object held in the holding device captured by an imaging system, and generates the control signal for controlling at least one of the robot and the holding device to process the object held in the holding device based on a second imaging result of the object held in the holding device captured by the imaging system and the selected model. According to a seventh aspect, there is provided a control system including the control device provided by the sixth aspect and the imaging system. According to an eighth aspect, there is provided a robot system including the control device provided by the sixth aspect, the imaging system, and the robot.

[0007] According to a ninth aspect, there is provided a control method for generating a control signal for controlling a robot provided with a holding device capable of holding an object and moving the holding device, and at least one of the holding devices, the control method including: selecting a model representing a part of the object as a selected model based on seventh imaging results obtained by imaging the object held in the holding device with an imaging system by controlling at least one of the robot and the holding device; and generating the control signal for controlling at least one of the robot and the holding device to process the object held in the holding device based on second imaging results obtained by imaging the object held in the holding device with the imaging system and the selected model. According to a tenth aspect, there is provided a computer program causing a computer to execute the control method provided by the ninth aspect. The operation and other advantages of the present invention will become apparent from the following detailed description of the embodiments.

[0008] FIG. 1 is a block diagram showing the configuration of a robot system according to this embodiment. FIG. 2 is a side view showing the appearance of a robot according to this embodiment. FIG. 3 is a block diagram showing the configuration of a control device according to this embodiment. FIG. 4 is a block diagram showing the configuration of a robot control device according to this embodiment. FIGS. 5( a) and 5(b) are side views showing the positional relationship between a robot and a workpiece at a certain point in time during a holding process for holding a workpiece placed on a mounting device, and FIGS. 5(c) to 5(e) are side views showing the positional relationship between a robot and a workpiece at a certain point in time during a release process for placing a workpiece on the mounting device. FIG. 6 is a flowchart showing the overall flow of robot control processing. FIG. 7 conceptually shows holding position and orientation information. FIG. 8 shows an example of a holding position and orientation registration screen. FIGS. 9(a) to 9(d) are cross-sectional views showing the holding position and orientation indicated by the holding position and orientation information using an end effector and a first target object. FIG. 10 conceptually shows release position and orientation information. FIG. 11 shows an example of a release position and orientation registration screen. FIGS. 12(a) to 12(d) are cross-sectional views showing the release position and posture indicated by the release position and posture information using the end effector, the first target object, and the second target object. FIGS. 13(a) to 13(c) are template models. FIG. 14 is a flowchart showing the process of generating a template model. FIGS. 15(a) and 15(b) are an example of a model generation screen on which a user's instructions for generating a template model are input. FIG. 16 is an example of an image represented by image data generated by an imaging system provided in the robot 1 capturing an image of the first target object held by the end effector. FIG. 17 is an association table for associating multiple different template models with multiple different holding position and posture information. FIGS. 18(a) and 18(b) are an example of a model generation screen on which a user's instructions for generating a contour model are input. FIG. 19 is an example of a reference model divided into multiple division models. FIG. 20 is an example of a reference model divided into multiple division models.FIG. 21 conceptually illustrates an inference device that outputs a model that can be used as a template model TM when image data is input. FIG. 22 is a flowchart illustrating the process flow of controlling at least one of the robot, end effector, and robot movable device to hold a first target object in step S2 of FIG. 6. FIG. 23 is a flowchart illustrating the process flow of controlling at least one of the robot, end effector, and robot movable device to release the first target object onto a second target object in step S3 of FIG. 6. FIG. 24 conceptually illustrates multiple pieces of holding position and orientation information and multiple template models that are associated with each other. FIG. 25 conceptually illustrates multiple pieces of holding position and orientation information and multiple template models that are associated with each other. FIG. 26 conceptually illustrates multiple pieces of holding position and orientation information and multiple template models that are associated with each other. FIG. 27 conceptually illustrates multiple pieces of holding position and orientation information and multiple template models that are associated with each other. FIG. 28 conceptually illustrates multiple template models associated with one piece of holding position and orientation information. FIG. 29 conceptually illustrates multiple template models associated with one piece of holding position and orientation information. FIG. 30 conceptually shows multiple template models associated with one piece of holding position and orientation information. FIG. 31 conceptually shows multiple template models associated with one piece of holding position and orientation information. FIG. 32 conceptually shows multiple template models associated with one piece of holding position and orientation information. FIG. 33 is a flowchart showing the process of controlling at least one of the robot, the end effector, and the robot movable device so as to release the first target object onto the second target object in step S3 of FIG. 6. FIG. 34(a) is a cross-sectional view showing the first target object that is not vibrating, and FIG. 34(b) is a cross-sectional view showing the first target object that is vibrating. FIG. 35 is a block diagram showing the configuration of a robot system including a measurement system.

[0009] Next, embodiments of a control device, a control system, a robot system, a control method, and a computer program will be described with reference to the drawings. Below, embodiments of a control device, a control system, a robot system, a control method, and a computer program will be described using a robot system SYS. (1) Configuration of the robot system SYS First, the configuration of the robot system SYS will be described. (1-1) Overall configuration of the robot system SYS First, the overall configuration of the robot system SYS will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the overall configuration of the robot system SYS.

[0010] 1, the robot system SYS includes a robot 1, an imaging system 2, a control device 3, and an end effector 4. The imaging system 2 may also be referred to as an imaging unit.

[0011] The robot 1 is a device capable of performing a predetermined process on a target object OBJ. An example of the robot 1 is shown in FIG. 2. FIG. 2 is a side view showing the appearance of the robot 1. As shown in FIG. 2, the robot 1 includes, for example, a base 11, a robot arm 12, and a robot control device 13.

[0012] The base 11 is a component that forms the base of the robot 1. The base 11 is placed on a support surface S such as a floor surface. The base 11 may be fixed to the support surface S. Alternatively, the base 11 may be movable relative to the support surface S. Note that FIG. 2 shows an example in which the base 11 is fixed to the support surface S.

[0013] The robot arm 12 is attached to the base 11. The robot arm 12 is a device in which a plurality of links 121 are connected via joints 122. An actuator is built into the joint 122. The link 121 may be rotatable around an axis defined by the joint 122 by the actuator built into the joint 122. At least one link 121 may be extendable and contractible along the direction in which the link 121 extends. A device including the device in which a plurality of links 121 are connected via joints 122 and the base 11 may be referred to as the robot arm 12.

[0014] An end effector 4 is attached to the robot arm 12. That is, the end effector 4 is attached to the robot 1. In the example shown in FIG. 2 , the end effector 4 is attached to the tip of the robot arm 12. The end effector 4 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the end effector 4. That is, the robot 1 moves the end effector 4.

[0015] The end effector 4 is a device that performs a predetermined process (in other words, a predetermined operation) on the target object OBJ. The end effector 4 that performs a predetermined process on the target object OBJ may also be called a processing device.

[0016] For example, the end effector 4 may perform a holding process to hold the target object OBJ as an example of the predetermined process. In this case, the end effector 4 may be considered to be performing the holding process on the target object OBJ that the end effector 4 is to hold. An end effector 4 capable of performing the holding process may be referred to as a holding device. Note that holding the target object OBJ may be considered to be equivalent to picking up the target object OBJ. The holding process of holding the target object OBJ may be considered to be equivalent to a pick-up process of picking up the target object OBJ.

[0017] For example, holding the target object OBJ may include grasping the target object OBJ. For example, holding the target object OBJ may include gripping the target object OBJ using a hand gripper, which is an example of the end effector 4. Holding the target object OBJ may include attracting the target object OBJ. For example, holding the target object OBJ may include attracting (vacuum suction) the target object OBJ using a vacuum gripper, which is an example of the end effector 4. For example, holding the target object OBJ may include attracting (vacuum suction) the target object OBJ using a magnetic gripper (e.g., a magnetically attractive gripper), which is an example of the end effector 4.

[0018] For example, as an example of the predetermined process, the end effector 4 may perform a release process (in other words, a release operation) to release (i.e., let go of) the target object OBJ that it is holding. In this case, the end effector 4 may be considered to be performing the release process on the target object OBJ that it is holding. An end effector 4 that is capable of performing the release process may be called a release device. The release process may also be called a placement process.

[0019] A hand gripper is an example of an end effector 4 capable of holding and releasing an object OBJ. A hand gripper is an end effector 4 that can hold (e.g., grasp) a target object OBJ by physically pinching the target object OBJ using multiple (e.g., two, three, or four) finger or claw members. Another example of an end effector 4 capable of holding and releasing an object OBJ is a vacuum gripper (e.g., a vacuum suction gripper). A vacuum gripper is an end effector 4 that can hold (e.g., attract) a target object OBJ by vacuum suction. Another example of an end effector 4 capable of holding and releasing an object OBJ is a magnetic gripper (e.g., a magnetic suction gripper). FIG. 2 illustrates an example in which the end effector 4 is a hand gripper. However, the end effector 4 capable of holding and releasing an object OBJ is not limited to the above example, and may be any other existing end effector capable of holding and releasing an object OBJ. For example, the end effector 4 may be a Bernoulli chuck capable of holding the target object OBJ in a non-contact manner.

[0020] The robot 1 may perform a placement process (in other words, a placement operation) for placing a target object OBJ at a desired position using the end effector 4 capable of holding and releasing the target object OBJ, as an example of a predetermined process. For example, the robot 1 may use the end effector 4 to hold a first target object OBJ that is a first example of the target object OBJ, and then perform a placement process for placing the first target object OBJ held by the end effector 4 at a desired position of a second target object OBJ that is a second example of the target object OBJ and different from the first target object OBJ. In this case, the end effector 4 may be considered to be performing a release process for the second target object OBJ on which the end effector 4 is to place the first target object OBJ. Similarly, the end effector 4 may be considered to be performing a release process for the first target object OBJ that the end effector 4 is to release. Note that a specific example of the placement process described in this paragraph is shown in FIGS. 5( a) to 5(e) described later. The target object OBJ may be a first target object OBJ or a second target object OBJ.

[0021] The robot 1 may use the end effector 4 capable of holding and releasing operations to perform a fitting process (i.e., a fitting operation) for fitting a first target object OBJ into a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (i.e., a placement operation). The fitting process may include, for example, a process for fitting (i.e., inserting) the first target object OBJ (e.g., a convex portion of the first target object OBJ) into a concave portion (e.g., a hole) formed in the second target object OBJ. The fitting process may include, for example, a process for fitting the first target object OBJ (e.g., a concave portion of the first target object OBJ) into a convex portion (e.g., a rod) formed in the second target object OBJ. In this case, the second target object OBJ may be an object (workpiece) into which the first target object OBJ is to be fitted.

[0022] The robot 1 may use the end effector 4 capable of performing a holding process and a release process to perform a placing process (in other words, a placing operation) for placing a first target object OBJ on a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placing operation). In this case, the second target object OBJ may be an object (workpiece) onto which the first target object OBJ is to be placed. The robot 1 may use the end effector 4 capable of performing a holding process and a release process to perform a pasting process (in other words, a pasting operation) for pasting the first target object OBJ on a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placing operation). In this case, the second target object OBJ may be an object (workpiece) onto which the first target object OBJ is to be pasted. The robot 1 may use the end effector 4 capable of holding and releasing processes to perform a bonding process (in other words, a bonding operation) for bonding a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation). In this case, the second target object OBJ may be an object (workpiece) to which the first target object OBJ is to be bonded. Alternatively, another end effector capable of dispensing adhesive for the bonding process may be provided on the robot 1 or another robot. The robot 1 may use the end effector 4 capable of holding and releasing processes to perform a welding process (in other words, a welding operation) for welding the first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation). In this case, the second target object OBJ may be an object (workpiece) to which the first target object OBJ is to be welded. Furthermore, another end effector (for example, a processing device for welding with an energy beam) for welding the first target object OBJ and the second target object OBJ may be provided on the robot 1 or another robot.The robot 1 may use the end effector 4 capable of holding and releasing processes to perform a screw tightening process (i.e., a screw tightening operation) for tightening a first target object OBJ capable of functioning as a screw into a screw hole formed in a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (i.e., a placement operation). In this case, the first target object OBJ may be a screw member capable of functioning as a screw, such as a bolt or a nut. In this case, the second target object OBJ may be an object (workpiece) to which the first target object OBJ is to be tightened. Furthermore, the end effector 4 may be a tool such as a screwdriver capable of tightening screws. At least one of the adhering process, bonding process, welding process, and screw tightening process may be referred to as a processing process.

[0023] The robot 1 may use an end effector 4 capable of holding and releasing processes to perform a process for discarding the target object OBJ (in other words, a discarding operation) as a specific example of a placement process (in other words, a placement operation).

[0024] For example, the end effector 4 may perform a predetermined process on each of multiple target objects OBJ. That is, the end effector 4 may perform the predetermined process on the multiple target objects OBJ in sequence. In this case, the robot 1 may move the end effector 4 to a first position where the end effector 4 can perform the predetermined process on a first target object OBJ, and after the end effector 4 has moved to the first position, the end effector 4 may perform the predetermined process on the first target object OBJ. Thereafter, the robot 1 may move the end effector 4 to a second position where the end effector 4 can perform the predetermined process on a second target object OBJ different from the first target object OBJ, and after the end effector 4 has moved to the second position, the end effector 4 may perform the predetermined process on the second target object OBJ.

[0025] For example, the end effector 4 may perform a predetermined process on each of multiple portions of a single target object OBJ. That is, the end effector 4 may sequentially perform a predetermined process on multiple portions of a single target object OBJ. In this case, the robot 1 may move the end effector 4 to a third position where the end effector 4 can perform a predetermined process on a first portion of the target object OBJ, and after the end effector 4 has moved to the third position, the end effector 4 may perform the predetermined process on the first portion of the target object OBJ. Thereafter, the robot 1 may move the end effector 4 to a fourth position where the end effector 4 can perform a predetermined process on a second portion of the target object OBJ, and after the end effector 4 has moved to the fourth position, the end effector 4 may perform the predetermined process on the second portion of the target object OBJ.

[0026] As shown in FIG. 2 , the target object OBJ on which the end effector 4 performs a predetermined process may include a workpiece W. The workpiece W may include, for example, a part or member used to manufacture a desired product. The workpiece W may include, for example, a part or member to be processed to manufacture the desired product. The workpiece W may include, for example, a part or member to be transported to manufacture the desired product. The workpiece W may include, for example, a part or member that is moving due to transportation to manufacture the desired product. The workpiece W may include, for example, a part or member that is moving to manufacture the desired product.

[0027] As shown in FIG. 2 , the target object OBJ on which the end effector 4 performs a predetermined process may include a placement device T on which a workpiece W is placed. An example of the placement device T is a container (storage box) CB. The container CB may have a bottom wall BS and a side wall SS protruding upward from the bottom wall BS. The placement device T may be a placement device T in which the workpiece W is placed on the bottom wall BS in a storage space SP enclosed by the bottom wall BS and the side wall SS. The container CB may also have a bottom wall BS and a side wall SS protruding upward from the bottom wall BS. The placement device T may be a placement device T capable of accommodating the workpiece W in the storage space SP enclosed by the bottom wall BS and the side wall SS. However, the container CB does not necessarily have a side wall SS. A container CB without a side wall SS may be referred to as a pallet. Furthermore, the placement device T is not limited to a container CB or a pallet, but may also be an existing object on which the workpiece W can be placed. The mounting device T may also be referred to as a mounting member.

[0028] The placement device T may be disposed on a support surface S. The placement device T may be fixed to the support surface S. Alternatively, at least a portion of the placement device T may be movable relative to the support surface S. A first example in which at least a portion of the placement device T is movable relative to the support surface S is when the placement device T is supported by a transport device that can move (in other words, transport) the placement device T. In this case, the transport device may be, for example, a belt conveyor. A second example in which at least a portion of the placement device T is movable relative to the support surface S is when the placement device T is supported by a movable device (movable placement device) that can move the placement device T. An example of the movable placement device is at least one of an automatic guided vehicle (AGV), an autonomous mobile robot, an unmanned aerial vehicle (e.g., a drone), and a submarine. A third example in which at least a part of the placement device T is movable relative to the support surface S is where the placement device T functions as a movable placement device. That is, a third example in which at least a part of the placement device T is movable relative to the support surface S is where a movable placement device is used as the placement device T. Note that Fig. 2 shows an example in which the placement device T is self-propelled on the support surface S. The robot control device 13 controls the operation of the robot 1.

[0029] Specifically, the robot control device 13 may control the movement of the robot arm 12. For example, the robot control device 13 may control the movement of the robot arm 12 so that a desired link 121 rotates around an axis defined by a desired joint 122. For example, the robot control device 13 may control the movement of the robot arm 12 so that the end effector 4 attached to the robot arm 12 is positioned at a desired position. For example, the robot control device 13 may control the movement of the robot arm 12 so that the end effector 4 attached to the robot arm 12 moves to a desired position.

[0030] Specifically, the robot control device 13 may control the movement of the robot arm 12. For example, the robot control device 13 may control the movement of the robot arm 12 so that a desired link 121 rotates around an axis defined by a desired joint 122. For example, the robot control device 13 may control the movement of the robot arm 12 so that the end effector 4 attached to the robot arm 12 is positioned at a desired position. For example, the robot control device 13 may control the movement of the robot arm 12 so that the end effector 4 attached to the robot arm 12 moves to a desired position.

[0031] In addition to controlling the operation of the robot 1, the robot control device 13 may also control the operation of the end effector 4 attached to the robot 1 (processing performed by the end effector 4). For example, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 holds the target object OBJ at a desired timing. That is, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 performs a holding process at a desired timing. For example, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 releases the held target object OBJ at a desired timing. That is, the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 performs a release process at a desired timing. In order for the end effector 4 to hold or release the target object OBJ, if the end effector 4 is a hand gripper, the robot control device 13 may control the timing of opening and closing the hand gripper. If the end effector 4 is a vacuum gripper, the robot control device 13 may control the timing of turning on / off the vacuum device of the vacuum gripper (or the vacuum suction force).If the end effector 4 is a magnetic gripper, the robot control device 13 may control the timing of turning on / off the magnetic suction device of the magnetic gripper (or the magnetic force).

[0032] Note that FIG. 2 shows an example in which the robot 1 is a robot arm 12 (i.e., a vertical articulated robot). However, the robot 1 may be a robot different from a vertical articulated robot. For example, the robot 1 may be a SCARA robot (i.e., a horizontal articulated robot). For example, the robot 1 may be a parallel link robot. For example, the robot 1 may be a dual-arm robot having two robot arms 12. For example, the robot 1 may be a Cartesian coordinate robot. For example, the robot 1 may be a cylindrical coordinate robot.

[0033] The robot 1 may be installed on a movable device (robot movable device) that can move the robot 1. Examples of the robot movable device include at least one of an automatic guided vehicle (AGV), an autonomous mobile robot, an unmanned aerial vehicle (e.g., a drone), and a submarine. When the robot 1 is installed on a robot movable device different from the robot 1, a device including the robot 1 and the robot movable device on which the robot 1 is installed may be referred to as a movable device (robot movable device). When the robot 1 is installed on a robot movable device, the robot control device 13 may control the operation of the robot movable device on which the robot 1 is installed in addition to controlling the operation of the robot 1. Note that since the robot 1 moves the robot arm 12, the robot 1 itself may be considered to be a movable device.

[0034] 1, the imaging system 2 captures an image of a target object OBJ. To capture an image of the target object OBJ, the imaging system 2 includes an imaging device 21 and an illumination device 23.

[0035] The imaging device 21 is a camera capable of capturing an image of the target object OBJ. For example, the imaging device 21 may capture an image of the target object OBJ under the control of the control device 3. The imaging device 21 generates image data IMG by capturing an image of the target object OBJ. In other words, the imaging device 21 generates image data IMG that is the result of capturing the image of the target object OBJ. The image data IMG generated by the imaging device 21 is output from the imaging device 21 to the control device 3. As a result, the control device 3 acquires the image data IMG acquired by the imaging device 21 capturing an image of the target object OBJ.

[0036] The imaging system 2 may include an imaging device 21 including a monocular camera. In this case, the imaging device 21 may generate image data IMG including one image data item generated by the monocular camera. Alternatively, the imaging system 2 may include an imaging device 21 including a stereo camera including two monocular cameras. In this case, the imaging device 21 may generate image data IMG including two image data items generated by the two monocular cameras. Alternatively, the imaging system 2 may include an imaging device 21 including a first monocular camera and a stereo camera including two second monocular cameras different from the first monocular camera. In this case, the imaging device 21 may generate image data IMG including one image data item generated by the first monocular camera and image data IMG including two image data items generated by the two second monocular cameras.

[0037] The imaging system 2 may include an imaging device 21 that includes a camera other than a monocular camera and a stereo camera. For example, the imaging system 2 may include an imaging device 21 that includes three or more monocular cameras. For example, the imaging system 2 may include an imaging device 21 that includes at least one of a light field camera, a plenoptic camera, and a multispectral camera.

[0038] The imaging device 21 may capture an image of the entire target object OBJ. Alternatively, the imaging device 21 may capture an image of a portion of the target object OBJ. In other words, the imaging device 21 may capture an image of a portion of the target object OBJ while not capturing an image of another portion of the target object OBJ.

[0039] The imaging device 21 may capture an image of a single target object OBJ. In other words, a single target object OBJ may appear in the image represented by the image data IMG. Alternatively, the imaging device 21 may capture an image of multiple target objects OBJ. In other words, a plurality of target objects OBJ may appear in the image represented by the image data IMG. In this case, as will be described in detail later, the control device 3 may determine (in other words, select) one of the multiple target objects OBJ captured by the imaging device 21 as the target object OBJ on which the end effector 4 will actually perform a predetermined process. Note that one of the multiple target objects OBJ captured by the imaging device 21 on which the end effector 4 will actually perform a predetermined process may be referred to as a process execution object.

[0040] When the imaging device 21 captures images of multiple target objects OBJ, the multiple target objects OBJ captured by the imaging device 21 may be arranged such that at least two of the multiple target objects OBJ at least partially overlap. As an example, when the target objects OBJ are workpieces W corresponding to components used to manufacture a desired product, the multiple workpieces W (i.e., multiple components) may be arranged such that at least two of the multiple workpieces W at least partially overlap. In this case, the end effector 4 may perform the above-described holding process of holding at least one workpiece W among the multiple workpieces W that are randomly arranged. In other words, the robot 1 may perform bulk picking, in which workpieces W are picked one by one from the multiple workpieces W that are randomly arranged. Note that the multiple workpieces W that are randomly arranged may also be referred to as multiple workpieces W that are irregularly arranged, multiple workpieces W that are casually arranged, or multiple workpieces W that are randomly arranged.

[0041] Alternatively, the multiple target objects OBJ captured by the imaging device 21 may be arranged regularly. As an example, if the target objects OBJ are workpieces W corresponding to parts used to manufacture a desired product, the multiple workpieces W (i.e., multiple parts) may be arranged in a matrix. In this case, the end effector 4 may perform the above-mentioned holding process of holding at least one workpiece W among the multiple regularly arranged workpieces W. In other words, the robot 1 may pick up a workpiece W one by one from the multiple regularly arranged workpieces W. Note that the multiple regularly arranged workpieces W can be rephrased as multiple workpieces W that are orderly arranged, or multiple workpieces W that are arranged according to a certain arrangement rule.

[0042] Note that the placement process for placing the target object OBJ held by the end effector 4 may include a placement process for placing the target object OBJ randomly (in other words, haphazardly), as with the holding process, or a placement process for placing the target object OBJ regularly (in other words, orderly). For example, the robot 1 may use the end effector 4 to hold one of a plurality of workpieces W that are regularly or randomly placed in the first container CB, and then place the one workpiece W held by the end effector 4 regularly or randomly in the second container CB.

[0043] The imaging device 21 may capture images of not only the target object OBJ on which the end effector 4 performs a predetermined process, but also other objects different from the target object OBJ. Note that, because the other objects are not targets on which the robot 1 performs a predetermined process, the other objects different from the target object OBJ are referred to as non-target objects in the following description. An example of a non-target object is at least a portion of the end effector 4. An example of a non-target object is at least a portion of the robot arm 12. An example of a non-target object is at least a portion of a peripheral object, which is an object located around the robot 1. An example of a peripheral object is the mounting device T. For example, if both the target object OBJ and the non-target object are included in the imaging range (field of view) of the imaging device 21, the imaging device 21 may capture images of both the target object OBJ and the non-target object. As a result, the imaging device 21 may generate image data IMG representing an image in which both the target object OBJ and the non-target object are captured. However, the imaging device 21 may capture an image of the target object OBJ without capturing an image of a non-target object. That is, the imaging device 21 may generate image data IMG representing an image in which the target object OBJ is captured but the non-target object is not captured. In either case, the imaging device 21 generates image data IMG representing an image in which at least the target object OBJ is captured. That is, the imaging device 21 generates image data IMG including image data of at least the target object OBJ.

[0044] The illumination device 23 is a device capable of irradiating illumination light onto the target object OBJ (for example, at least one target object OBJ when multiple target objects OBJ exist). For example, the illumination device 23 may irradiate the target object OBJ with illumination light under the control of the control device 3. In particular, the illumination device 23 is a device capable of irradiating the target object OBJ with illumination light, thereby illuminating the target object OBJ with illumination light. In this case, the imaging device 21 may capture an image of the target object OBJ illuminated with illumination light. However, the illumination device 23 does not need to irradiate the target object OBJ with illumination light. In this case, the imaging system 2 (robot system SYS) does not need to be equipped with the illumination device 23.

[0045] When the imaging device 21 includes a stereo camera, the illumination device 23 may be a device capable of projecting a desired projection pattern onto the target object OBJ by irradiating the target object OBJ with illumination light. The desired projection pattern may include, for example, a random pattern. The random pattern may include a random dot pattern. The desired projection pattern may include, for example, a one-dimensional or two-dimensional grid pattern. The desired projection pattern may include, for example, a line pattern. The desired projection pattern may include, for example, a stripe pattern. The desired projection pattern may include other projection patterns. However, even when the imaging device 21 does not include a stereo camera (i.e., the imaging device 21 includes a monocular camera), the illumination device 23 may project the desired projection pattern onto the target object OBJ by irradiating the target object OBJ with illumination light. In this case, the illumination device 23 may irradiate the target object OBJ with illumination light having a uniform intensity distribution (e.g., a uniform intensity distribution). The desired projection pattern may also be referred to as light having a desired intensity distribution.

[0046] The imaging system 2 is attached to the robot arm 12, similar to the end effector 4. That is, the imaging device 21 and the lighting device 23 are attached to the robot arm 12. For example, as shown in FIG. 2 , the imaging device 21 and the lighting device 23 may be attached to the tip of the robot arm 12, similar to the end effector 4. In this case, the imaging device 21 and the lighting device 23 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the imaging device 21 and the lighting device 23.

[0047] However, the imaging system 2 does not have to be attached to the robot arm 12. For example, the imaging system 2 may be placed at any position where it can capture an image of the target object OBJ. Furthermore, for example, the imaging system 2 may be placed at any position where it can irradiate the target object OBJ with illumination light. For example, the imaging system 2 may be attached to a structure such as a pillar. When the robot 1 is installed in a robot movable device as described above, the imaging system 2 may be installed in the robot movable device on which the robot 1 is installed.

[0048] In addition, either the imaging device 21 or the lighting device 23 may be attached to the robot arm 12, and the other either the imaging device 21 or the lighting device 23 may be attached to a location different from the robot arm 12 (for example, a structure such as a pillar or a robot movable device).

[0049] The imaging device 21 may capture an image of the target object OBJ (e.g., at least one target object OBJ when multiple target objects OBJ are present) during a period in which the imaging device 21 and the target object OBJ are displaced relative to each other. Note that the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may refer to a state in which the relative positional relationship between the imaging device 21 and the target object OBJ is changing. The state in which the imaging device 21 and the target object OBJ are displaced relative to each other may refer to a state in which the imaging device 21 and the target object OBJ are moving relative to each other. For example, the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the target object OBJ is moving relative to the imaging device 21. For example, the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the imaging device 21 is moving relative to the target object OBJ. In this case, the imaging device 21 does not need to be stationary to capture an image of the target object OBJ, and the robot system SYS can efficiently perform predetermined processing on the target object OBJ using the end effector 4.

[0050] Alternatively, the imaging device 21 may capture an image of the target object OBJ during a period in which there is no relative displacement between the imaging device 21 and the target object OBJ. Note that the state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the relative positional relationship between the imaging device 21 and the target object OBJ is not changing. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which there is no relative movement between the imaging device 21 and the target object OBJ. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the imaging device 21 and the target object OBJ are stationary. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the imaging device 21 and the target object OBJ are moving at the same moving speed in the same moving direction.

[0051] The control device 3 performs robot control processing. The robot control processing includes processing for generating a robot control signal for controlling the robot 1. Specifically, the control device 3 generates the robot control signal based on image data IMG output from the imaging system 2. In this embodiment, the control device 3 calculates at least one of the position and orientation of the target object OBJ in the global coordinate system of the robot system SYS based on the image data IMG, and generates the robot control signal based on the calculated at least one of the position and orientation of the target object OBJ.

[0052] The global coordinate system is a coordinate system that serves as the reference for the robot system SYS. For example, the global coordinate system may be a coordinate system that serves as the reference for the robot 1. The global coordinate system can also be said to be a coordinate system used to control the robot 1. For example, a world coordinate system that is defined based on the support surface S on which the robot 1 is placed may be used as the global coordinate system. In other words, a world coordinate system that is fixed with respect to the support surface S on which the robot 1 is placed may be used as the global coordinate system. In the following description, unless otherwise specified, the X-axis, Y-axis, and Z-axis may refer to the X-axis, Y-axis, and Z-axis in the global coordinate system, respectively.

[0053] However, the control device 3 may calculate at least one of the position and orientation of the target object OBJ in a coordinate system different from the global coordinate system based on the image data IMG. The coordinate system different from the global coordinate system may include at least one of the robot coordinate system and the imaging coordinate system. The robot coordinate system may be a coordinate system defined based on the robot 1. That is, the robot coordinate system may be a coordinate system fixed with respect to the robot 1 (for example, fixed with respect to the base 11 of the robot 1). The imaging coordinate system may be a coordinate system defined based on the imaging device 21. That is, the imaging coordinate system may be a coordinate system fixed with respect to the imaging device 21. An example of the imaging coordinate system is a coordinate system defined based on the optical axis AX21 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 21. An example of the imaging coordinate system is a coordinate system in which one of the three coordinate axes constituting the imaging coordinate system is an axis along the optical axis AX21 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) included in the imaging device 21.

[0054] The control device 3 may perform end effector control processing in addition to performing robot control processing. The end effector control processing may include processing for generating an end effector control signal for controlling the end effector 4. Specifically, the control device 3 may generate the end effector control signal based on at least one of the calculated position and orientation of the target object OBJ.

[0055] The end effector control processing may or may not be included in the robot control processing. In other words, the end effector control signal generated by the control device 3 may or may not be included in the robot control signal. In the following description, for convenience of explanation, an example will be described in which the end effector control processing is included in the robot control processing (in other words, the end effector control signal is included in the robot control signal). Therefore, in the following description, the robot control processing may mean processing for generating at least one of a robot control signal and an end effector control signal. In addition, in the following description, the robot control signal may mean at least one of a signal for controlling the robot 1 and a signal for controlling the end effector 4. The robot control signal may also be simply referred to as a control signal.

[0056] As described above, when the robot 1 is installed on a robot movable device (for example, at least one of an automatic guided vehicle, an autonomous transport robot, an unmanned aerial vehicle, and a submarine), the control device 3 may perform movable device control processing in addition to performing robot control processing. The movable device control processing may include processing for generating a movable device control signal for controlling the robot movable device. Specifically, the control device 3 may generate the movable device control signal based on at least one of the calculated position and orientation of the target object OBJ.

[0057] The movable device control process may or may not be included in the robot control process. In other words, the movable device control signal generated by the control device 3 may or may not be included in the robot control signal. In the following description, for convenience of explanation, an example will be described in which the movable device control process is included in the robot control process (i.e., the movable device control signal is included in the robot control signal). Therefore, in the following description, the robot control process may mean a process for generating at least one of a robot control signal, an end effector control signal, and a movable device control signal. In addition, in the following description, the robot control signal may mean at least one of a signal for controlling the robot 1, a signal for controlling the end effector 4, and a signal for controlling the robot movable device.

[0058] In this way, the control device 3 and the imaging system 2 are used to control the robot 1. Therefore, a system including the control device 3 and the imaging system 2 may be referred to as a robot control system or a control system.

[0059] The robot control signal generated by the control device 3 is output to the robot control device 13 of the robot 1. The robot control device 13 controls the operation of the robot 1 based on the robot control signal generated by the control device 3. For this reason, the robot control signal may include a signal for controlling the operation of the robot 1.

[0060] As described above, when the robot control signal includes a signal for controlling the robot arm 12, the robot control device 13 may control the robot arm 12 based on the robot control signal. For example, the robot control device 13 may control the operation of the actuator built into the joint 122 based on the robot control signal, thereby controlling the operation of the robot arm 12.

[0061] For example, as described above, the robot arm 12 moves the end effector 4. In this case, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at a desired position. The robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves to a desired position. The robot control signal may include a signal for controlling the robot arm 12 so that the positional relationship between the end effector 4 and the target object OBJ is a desired positional relationship. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 is located at a desired position. The robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves to a desired position. The robot control device 13 may control the robot arm 12 based on the robot control signal so that the positional relationship between the end effector 4 and the target object OBJ is a desired positional relationship.

[0062] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves toward (i.e., approaches) a first desired position where the end effector 4 can hold the target object OBJ. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at the first desired position. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves toward (i.e., approaches) the first desired position. That is, the robot control signal may control the robot arm 12 so that the end effector 4 is located at the first desired position. Note that, because the end effector 4 located at the first desired position holds the target object OBJ, controlling the robot arm 12 so that the end effector 4 approaches the first desired position may be considered equivalent to controlling the robot arm 12 so that the end effector 4 approaches the target object OBJ that is to be held at the first desired position.

[0063] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the robot arm 12 so that the posture of the end effector 4 becomes a first desired posture that enables the end effector 4 to hold the target object OBJ. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the posture of the end effector 4 becomes the first desired posture.

[0064] As another example, when performing a release process to release the target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 moves toward (i.e., approaches) a second desired position where the target object OBJ held by the end effector 4 should be released. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 4 is located at the second desired position. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the end effector 4 moves toward (i.e., approaches) the second desired position. That is, the robot control signal may control the robot arm 12 so that the end effector 4 is located at the second desired position. Furthermore, since the end effector 4 located at the second desired position releases the first target object OBJ that it is holding to the second target object OBJ, controlling the robot arm 12 so that the end effector 4 approaches the second desired position may be considered equivalent to controlling the robot arm 12 so that at least one of the end effector 4 and the first target object approaches the second target object OBJ from which the first target object OBJ is released at the second desired position.

[0065] As another example, when performing a release process to release the target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the robot arm 12 so that the posture of the end effector 4 becomes a second desired posture that allows the end effector 4 to release the target object OBJ. In this case, the robot control device 13 may control the robot arm 12 based on the robot control signal so that the posture of the end effector 4 becomes the second desired posture.

[0066] As described above, when the robot control signal includes a signal for controlling the end effector 4, the robot control device 13 may control the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the end effector 4 by controlling the operation of an actuator that moves a hand gripper that constitutes the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the end effector 4 by controlling the operation of a vacuum device of a vacuum gripper that constitutes the end effector 4 based on the robot control signal. For example, the robot control device 13 may control the operation of the end effector 4 by controlling the operation of a magnetic attraction device of a magnetic gripper that constitutes the end effector 4 based on the robot control signal.

[0067] As an example, when the end effector 4 performs a holding process to hold the target object OBJ, the robot control signal may include a signal for controlling the end effector 4 so that the end effector 4 located at the above-mentioned first desired position and / or in the above-mentioned holding target posture holds the target object OBJ. In this case, the robot control device 13 may control the end effector 4 based on the robot control signal so that the end effector 4 located at the first desired position and / or in the holding target posture holds the target object OBJ.

[0068] As another example, when performing a release process to release the target object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the end effector 4 to release the target object OBJ held by the end effector 4 located at the above-mentioned release target position and / or in the above-mentioned release target posture. In this case, the robot control device 13 may control the end effector 4 based on the robot control signal to release the target object OBJ held by the end effector 4 located at the release target position and / or in the release target posture.

[0069] As described above, when the robot control signal includes a signal for controlling a robotic movable device (e.g., at least one of an automatic guided vehicle, an autonomously traveling transport robot, an unmanned aerial vehicle, and a submarine) on which the robot 1 is installed, the robot controller 13 may control the robotic movable device based on the robot control signal. For example, the robot controller 13 may control a power source (e.g., a motor or an engine) of the robotic movable device based on the robot control signal so that the robot 1 moves to a target position indicated directly or indirectly by the robot control signal.

[0070] The robot control signal may include a signal that can be used as is by the robot control device 13 to control the operation of the robot 1. The robot control signal may include a signal that can be used as is as a robot drive signal that the robot control device 13 uses to control the operation of the robot 1. In this case, the robot control device 13 may use the robot control signal as is to control the operation of the robot 1. For example, the control device 3 may generate a drive signal for an actuator built into the joint 122 of the robot arm 12 as the robot control signal, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the actuator built into the joint 122 of the robot arm 12.

[0071] The robot control signal may include a signal that can be used directly by the robot control device 13 to control the operation of the end effector 4. The robot control signal may include a signal that can be used directly as an end effector drive signal used by the robot control device 13 to control the operation of the end effector 4. In this case, the robot control device 13 may use the robot control signal directly to control the operation of the end effector 4. For example, the control device 3 may generate, as the robot control signal, a drive signal (end effector drive signal) for an actuator that moves a hand gripper that constitutes the end effector 4, and the robot control device 13 may use the robot control signal generated by the control device 3 directly to control the actuator of the end effector 4. For example, the control device 3 may generate, as the robot control signal, a drive signal (end effector drive signal) for driving a vacuum device of a vacuum gripper that constitutes the end effector 4, and the robot control device 13 may use the robot control signal generated by the control device 3 directly to control the vacuum device of the end effector 4. For example, the control device 3 may generate a drive signal (end effector drive signal) for driving the magnetic adsorption device of the magnetic gripper that constitutes the end effector 4 as a robot control signal, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the magnetic adsorption device of the end effector 4.

[0072] The robot control signal may include a signal that can be used directly by the robot controller 13 to control the operation of a robot movable device on which the robot 1 is installed. The robot control signal may include a signal that can be used directly as a movable device drive signal that the robot controller 13 uses to control the operation of the robot movable device. In this case, the robot controller 13 may use the robot control signal directly to control the operation of the robot movable device. For example, the controller 3 may generate a power source drive signal (movable device drive signal) for driving a power source (e.g., a motor or engine) of the robot movable device as the robot control signal, and the robot controller 13 may use the robot control signal generated by the controller 3 directly to control the power source of the robot movable device.

[0073] As described above, if the robot control signal includes a signal that the robot control device 13 can use to control the operation of at least one of the robot 1, the end effector 4, and the robot movable device, the robot 1 does not need to be equipped with the robot control device 13. In this case, the control device 3 may use the robot control signal to control an actuator built into the joint 122 of the robot arm 12. For example, the control device 3 may use the robot control signal (end effector drive signal) to control an actuator that moves a hand gripper that constitutes the end effector 4. For example, the control device 3 may use the robot control signal (end effector drive signal) to control a vacuum device of a vacuum gripper that constitutes the end effector 4. For example, the control device 3 may use the robot control signal (end effector drive signal) to control a magnetic attraction device of a magnetic gripper that constitutes the end effector 4. For example, the control device 3 may use the robot control signal (end effector drive signal) to control a robot movable device installed on the robot 1.

[0074] Alternatively, the robot control signal may include a signal that can be used by the robot control device 13 to generate a robot drive signal for controlling the operation of the robot 1. In this case, the robot control device 13 may generate a robot drive signal for controlling the operation of the robot 1 based on the robot control signal, and control the operation of the robot 1 based on the generated robot drive signal. For example, the robot control device 13 may generate a robot drive signal for driving an actuator built into the joint 122 of the robot arm 12 based on the robot control signal, and control the actuator built into the joint 122 of the robot arm 12 based on the generated robot drive signal.

[0075] The robot control signal may include a signal that the robot control device 13 can use to generate an end effector drive signal for controlling the operation of the end effector 4. In this case, the robot control device 13 may generate an end effector drive signal for controlling the operation of the end effector 4 based on the robot control signal and control the operation of the end effector 4 based on the generated end effector drive signal. For example, if the end effector 4 is a hand gripper, the robot control device 13 may generate an end effector drive signal for driving an actuator of the hand gripper based on the robot control signal and control the actuator of the hand gripper based on the generated end effector drive signal. For example, if the end effector 4 is a magnetic gripper, the robot control device 13 may generate an end effector drive signal for driving a magnetic attraction device of the magnetic gripper based on the robot control signal and control the magnetic attraction device based on the generated end effector drive signal.

[0076] The robot control signal may include a signal usable by the robot controller 13 to generate a movable unit drive signal for controlling the operation of a robot movable unit on which the robot 1 is installed. In this case, the robot controller 13 may generate a movable unit drive signal for controlling the operation of the robot movable unit based on the robot control signal, and control the operation of the robot movable unit based on the generated movable unit drive signal. For example, the robot controller 13 may generate a power source drive signal (movable unit drive signal) for driving a power source (e.g., a motor or an engine) of the robot movable unit based on the robot control signal, and control the power source of the robot movable unit based on the generated movable unit drive signal.

[0077] The signals available to the robot controller 13 for generating the robot drive signals may include signals representing at least one of the position and orientation of the target object OBJ in a global coordinate system. The signals available to the robot controller 13 for generating the robot drive signals may include signals representing a desired positional relationship between the robot 1 and the target object OBJ in a global coordinate system.

[0078] The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value (target position) of the position of the end effector 4 in the global coordinate system. An example of a target position is a processing position where the end effector 4 should process the target object OBJ. For example, the target position may include a position where the end effector 4 should hold the target object OBJ. For example, the target position may include a position where the end effector 4 should release the target object OBJ. The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value of the position of the tip of the robot arm 12 (e.g., a tool center point) in the global coordinate system. The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value of the position of the imaging system 2 in the global coordinate system.

[0079] The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value (target posture) of the posture of the end effector 4 in the global coordinate system. An example of a target posture is a posture (processing posture) that the end effector 4 should take when processing the target object OBJ. For example, the target posture may include a posture that the end effector 4 should take when holding the target object OBJ. For example, the target posture may include a posture that the end effector 4 should take when releasing the target object OBJ. The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value of the posture of the tip of the robot arm 12 (e.g., a tool center point) in the global coordinate system. The signals available to the robot controller 13 for generating the robot drive signal may include a signal representing a target value of the posture of the imaging system 2 in the global coordinate system.

[0080] The signal that the robot control device 13 can use to generate the robot drive signal may be a signal that indicates the amount and direction of movement from the current position of the end effector 4 to the target position of the end effector 4. (1-2) Configuration of the Control Device 3 Next, the configuration of the control device 3 will be described with reference to Figure 3. Figure 3 is a block diagram showing the configuration of the control device 3.

[0081] 3 , the control device 3 includes a calculation device 31, a storage device 32, and a communication device 33. The control device 3 may further include an input device 34 and an output device 35. However, the control device 3 does not necessarily have to include at least one of the input device 34 and the output device 35. The calculation device 31, the storage device 32, the communication device 33, the input device 34, and the output device 35 may be connected via a data bus 36.

[0082] The arithmetic device 31 is hardware that includes at least a circuit (for example, at least one of an electronic circuit and an electric circuit). For this reason, the arithmetic device 31 may be referred to as a group of circuits.

[0083] The arithmetic device 31 includes at least one processor (i.e., one processor or multiple processors) as hardware. The processor may include, for example, a processor conforming to a von Neumann computer architecture. The processor conforming to the von Neumann computer architecture may include at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor may include, for example, a processor conforming to a non-von Neumann computer architecture. The processor conforming to the non-von Neumann computer architecture may include at least one of an FPGA (Field Programmable Gate Array) and an ASIC (Application Specific Circuit). The processor may be realized by a group of circuits (e.g., at least one of an electronic circuit and an electric circuit).

[0084] The arithmetic device 31 reads a computer program 321 including at least one of computer program code and computer program instructions. For example, the arithmetic device 31 may read the computer program 321 stored in the storage device 32. For example, the arithmetic device 31 may read the computer program 321 stored in a computer-readable, non-transitory storage medium using a storage medium reading device (not shown) included in the control device 3. The computer program 321 read from the storage medium may be stored in the storage device 32. The arithmetic device 31 may acquire (i.e., download or read) the computer program 321 from a device (not shown) located outside the control device 3 via the communication device 33 (or another communication device). The downloaded computer program 321 may be stored in the storage device 32.

[0085] The arithmetic device 31 executes the loaded computer program 321. As a result, logical functional blocks for executing the processing to be performed by the control device 3 (for example, the robot control processing described above) are realized within the arithmetic device 31. In other words, the arithmetic device 31, together with the storage device 32 or the like in which the computer program 321 is recorded (in other words, together with the storage device 32 and the computer program 321 recorded in the storage device 32 or the like), can function as a controller or computer for realizing the logical functional blocks for executing the processing to be performed by the control device 3. In other words, the at least one processor included in the arithmetic device 31, the memory (recording medium) included in the storage device 32 or the like, and the computer program 321 are configured so that the control device 3 performs the processing to be performed by the control device 3 (for example, the robot control processing described above).

[0086] The arithmetic device 31 may include a single processor. In this case, the arithmetic device 31 may use the single processor to perform the processing to be performed by the control device 3 (e.g., the robot control processing described above). For example, if the arithmetic device 31 performs a first operation (e.g., a first process that is part of the robot control processing) and a second operation (e.g., a second process that is another part of the robot control processing), the arithmetic device 31 may use a single processor to perform both the first and second operations. Alternatively, the arithmetic device 31 may include multiple processors. In this case, the arithmetic device 31 may use any one of the multiple processors to perform the processing to be performed by the control device 3 (e.g., the robot control processing described above). For example, if the arithmetic device 31 includes first and second processors and performs the first and second operations, the arithmetic device 31 may use any one of the first and second processors to perform each of the first and second operations. For example, the computing device 31 may perform a first operation using a first processor, may perform a second operation using the first processor, may perform the first operation using a second processor, or may perform the second operation using the second processor.

[0087] A computational model that can be constructed by machine learning may be implemented in the computational device 31 by the computational device executing the computer program 321. An example of a computational model that can be constructed by machine learning is a computational model including a neural network (so-called artificial intelligence (AI)). In this case, learning of the computational model may include learning of parameters of the neural network (e.g., at least one of a weight and a bias). The computational device 31 may execute a robot control process using the computational model. In other words, the operation of executing the robot control process may include the operation of executing the robot control process using the computational model. Note that a computational model that has been constructed by offline machine learning using teacher data may be implemented in the computational device 31. Furthermore, the computational model implemented in the computational device 31 may be updated by online machine learning on the computational device 31. Alternatively, the calculation device 31 may perform robot control processing using a calculation model implemented in a device external to the calculation device 31 (i.e., a device provided outside the control device 3) in addition to or instead of the calculation model implemented in the calculation device 31.

[0088] The recording medium for recording the computer program 321 executed by the arithmetic device 31 may be at least one of the following: a CD-ROM, CD-R, CD-RW, a flexible disk, an MO, a DVD-ROM, a DVD-RAM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, or an optical disk such as Blu-ray (registered trademark), a magnetic medium such as a magnetic tape, a magneto-optical disk, a semiconductor memory such as a USB memory, or any other medium capable of storing a program. The recording medium may include a device capable of recording the computer program 321 (for example, a general-purpose device or a dedicated device in which the computer program 321 is implemented in a state in which it can be executed in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program 321 may be realized by a logical processing block realized within the arithmetic device 31 (i.e., processor) when the arithmetic device 31 executes the computer program 321, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) provided in the arithmetic device 31, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.

[0089] 3 shows an example of logical functional blocks implemented in the arithmetic device 31 for executing robot control processing. As shown in Fig. 3, a position and orientation calculation unit 311 and a signal generation unit 312 are implemented in the arithmetic device 31. Note that the processing performed by the position and orientation calculation unit 311 and the signal generation unit 312 will be described in detail later with reference to Fig. 6 etc., and therefore will not be described here.

[0090] The storage device 32 includes at least one memory capable of storing desired data. In other words, the storage device 32 includes at least one memory containing desired data. The memory may be realized by a group of circuits (e.g., at least one of electronic circuits and electric circuits). For example, the storage device 32 may store a computer program 321 executed by the arithmetic device 31. In this case, the storage device 32 (memory) may be used as the above-mentioned recording medium for recording the computer program 321 executed by the arithmetic device 31. The storage device 32 may temporarily store data used by the arithmetic device 31 when the arithmetic device 31 is executing the computer program 321. The storage device 32 may also store data to be stored long-term by the control device 3. The storage device 32 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. That is, the storage device 32 may include a non-transitory recording medium.

[0091] The communication device 33 is capable of communicating with both the robot 1 and the imaging system 2 via a communication network (not shown). Alternatively, the communication device 33 may be capable of communicating with another device different from the robot 1 and the imaging system 2, in addition to or instead of at least one of the robot 1 and the imaging system 2, via a communication network (not shown). In this embodiment, the communication device 33 may receive (i.e., acquire) image data IMG from the imaging system 2. Furthermore, the communication device 33 may transmit (i.e., output) a robot control signal to the robot 1. Note that the communication device 33 that outputs the robot control signal to the robot 1 may be referred to as an output unit or an output device.

[0092] The input device 34 is a device that accepts information input to the control device 3 from outside the control device 3. For example, the input device 34 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by a user of the control device 3. For example, the input device 34 may include a recording medium reading device that can read information recorded as data on a recording medium that can be externally attached to the control device 3.

[0093] It should be noted that information can be input as data to the control device 3 from a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an input device that accepts information input to the control device 3 from outside the control device 3.

[0094] The output device 35 is a device that outputs information to the outside of the control device 3. For example, the output device 35 may output information as an image. That is, the output device 35 may include a display device (so-called display) 37 that can display an image. In this case, the calculation device 31 may generate a display signal for displaying the image on the display device 37 and output the generated display signal to the display device 37 via the data bus 36. The display device 37 may display the image based on the display signal generated by the calculation device 31.

[0095] 1 , the robot system SYS may include a display device 8 separate from the control device 3. That is, the robot system SYS may include a display device 8 external to the control device 3 in addition to or instead of the display device 37 included in the control device 3. In this case, the arithmetic device 31 may output the generated display signal to the display device 8 external to the control device 3 via the communication device 33. The display device 8 may display an image based on the display signal generated by the arithmetic device 31.

[0096] The output device 35 may include an output device different from the display device 37. For example, the output device 35 may output information as sound. That is, the output device 35 may include an audio device (a so-called speaker) capable of outputting sound. For example, the output device 35 may output information on paper. That is, the output device 35 may include a printing device (a so-called printer) capable of printing desired information on paper. For example, the output device 35 may output information as data to a recording medium that can be externally attached to the control device 3.

[0097] The control device 3 can output information as data to a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an output device that outputs information to a device external to the control device 3.

[0098] The robot control device 13 provided in the robot 1 described above may also have a configuration similar to that of the control device 3. That is, as shown in Fig. 4, which is a block diagram showing the configuration of the robot control device 13, the robot control device 13 includes an arithmetic device 131, a storage device 132, and a communication device 133. The robot control device 13 may further include an input device 134 and an output device 135. However, the robot control device 13 does not have to include at least one of the input device 134 and the output device 135. The arithmetic device 131, the storage device 132, the communication device 133, the input device 134, and the output device 135 may be connected via a data bus 136.

[0099] The arithmetic device 131 includes at least one processor (i.e., one processor or multiple processors) as hardware. The processor may include, for example, a processor conforming to a von Neumann computer architecture. The processor conforming to the von Neumann computer architecture may include at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor may include, for example, a processor conforming to a non-von Neumann computer architecture. The processor conforming to the non-von Neumann computer architecture may include at least one of an FPGA (Field Programmable Gate Array) and an ASIC (Application Specific Circuit).

[0100] The arithmetic device 131 reads a computer program 1321 including at least one of computer program code and computer program instructions. For example, the arithmetic device 131 may read the computer program 1321 stored in the storage device 132. For example, the arithmetic device 131 may read the computer program 1321 stored in a computer-readable, non-transitory storage medium using a storage medium reading device (not shown) included in the robot control device 13. The computer program 1321 read from the storage medium may be stored in the storage device 132. The arithmetic device 131 may acquire (i.e., download or read) the computer program 1321 from a device (not shown) located outside the robot control device 13 via the communication device 133 (or another communication device). The downloaded computer program 1321 may be stored in the storage device 132.

[0101] The arithmetic device 131 executes the loaded computer program 1321. As a result, logical functional blocks for executing processing to be performed by the robot control device 13 (e.g., processing to control at least one of the robot 1, the end effector 4, and the robot movable device) are realized within the arithmetic device 131. In other words, the arithmetic device 131, together with the storage device 132 etc. in which the computer program 1321 is recorded (in other words, together with the storage device 132 and the computer program 1321 recorded in the storage device 132 etc.), can function as a controller or computer for realizing the logical functional blocks for executing processing to be performed by the robot control device 13. In other words, the at least one processor included in the arithmetic device 131, the memory (recording medium) included in the storage device 132 etc., and the computer program 1321 are configured so that the robot control device 13 performs the processing to be performed by the robot control device 13.

[0102] The arithmetic device 131 may include a single processor. In this case, the arithmetic device 131 may use the single processor to perform the processing to be performed by the robot control device 13 (e.g., processing to control at least one of the robot 1, the end effector 4, and the robot movable device). For example, if the arithmetic device 131 performs a first operation (e.g., a first processing for controlling at least one of the robot 1, the end effector 4, and the robot movable device) and a second operation (e.g., a second processing for controlling at least one of the robot 1, the end effector 4, and the robot movable device), the arithmetic device 131 may use a single processor to perform both the first and second operations. Alternatively, the arithmetic device 131 may include multiple processors. In this case, the arithmetic device 131 may use any one of the multiple processors to perform the following operations. For example, if the arithmetic device 131 includes first and second processors and performs the first and second operations, the arithmetic device 131 may use any one of the first and second processors to perform each of the first and second operations. For example, the computing device 131 may perform a first operation using a first processor, may perform a second operation using the first processor, may perform the first operation using a second processor, or may perform the second operation using the second processor.

[0103] A computational model that can be constructed by machine learning may be implemented in the computational device 131 by the computational device executing the computer program 1321. An example of a computational model that can be constructed by machine learning is a computational model including a neural network (so-called artificial intelligence (AI)). In this case, learning of the computational model may include learning of parameters of the neural network (e.g., at least one of a weight and a bias). The computational device 131 may control at least one of the robot 1 and the end effector 4 using the computational model. In other words, the operation of controlling at least one of the robot 1 and the end effector 4 may include the operation of controlling at least one of the robot 1 and the end effector 4 using the computational model. The computational device 131 may be implemented with a computational model that has been constructed by offline machine learning using training data. Furthermore, the computational model implemented in the computational device 131 may be updated by online machine learning on the computational device 131. Alternatively, the computing device 131 may execute the robot control process using a computational model implemented in a device external to the computing device 131 (i.e., a device provided outside the robot control device 13) in addition to or instead of the computational model implemented in the computing device 131.

[0104] The recording medium for recording the computer program 1321 executed by the arithmetic device 131 may be at least one of a CD-ROM, CD-R, CD-RW, flexible disk, MO, DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW, and Blu-ray (registered trademark) optical disk, magnetic medium such as magnetic tape, magneto-optical disk, semiconductor memory such as USB memory, and any other medium capable of storing a program. The recording medium may include a device capable of recording the computer program 1321 (for example, a general-purpose device or dedicated device in which the computer program 1321 is implemented in a state in which it can be executed in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program 1321 may be realized by a logical processing block realized within the arithmetic device 131 when the arithmetic device 131 (i.e., processor) executes the computer program 1321, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) provided in the arithmetic device 131, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.

[0105] The storage device 132 includes at least one memory capable of storing desired data. In other words, the storage device 132 includes at least one memory containing desired data. For example, the storage device 132 may store a computer program 1321 executed by the arithmetic device 131. In this case, the storage device 132 (memory) may be used as the above-mentioned recording medium for recording the computer program 1321 executed by the arithmetic device 131. The storage device 132 may temporarily store data used by the arithmetic device 131 when the arithmetic device 131 is executing the computer program 1321. The storage device 132 may also store data to be stored long-term by the robot control device 13. The storage device 132 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. That is, the storage device 132 may include a non-transitory recording medium.

[0106] The communication device 133 is capable of communicating with at least one of the imaging system 2 and the control device 3 via a communication network (not shown). Alternatively, the communication device 133 may be capable of communicating with another device different from the imaging system 2 and the control device 3 via a communication network (not shown), in addition to or instead of at least one of the imaging system 2 and the control device 3. In this embodiment, the communication device 133 may receive (i.e., acquire) a robot control signal from the control device 3.

[0107] The input device 134 is a device that accepts information input to the robot control device 13 from outside the robot control device 13. For example, the input device 134 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by a user of the robot control device 13. For example, the input device 134 may include a recording medium reader that can read information recorded as data on a recording medium that can be externally attached to the robot control device 13.

[0108] It should be noted that information can be input as data to the robot control device 13 from a device external to the robot control device 13 via the communication device 133. In this case, the communication device 133 may function as an input device that accepts information input to the robot control device 13 from outside the robot control device 13.

[0109] The output device 135 is a device that outputs information to the outside of the robot control device 13. For example, the output device 135 may output information as an image. That is, the output device 135 may include a display device (a so-called display) capable of displaying an image. For example, the output device 135 may output information as sound. That is, the output device 135 may include an audio device (a so-called speaker) capable of outputting sound. For example, the output device 135 may output information on paper. That is, the output device 135 may include a printing device (a so-called printer) capable of printing desired information on paper. For example, the output device 135 may output information as data to a recording medium that can be attached externally to the robot control device 13.

[0110] The robot control device 13 can output information as data to a device external to the robot control device 13 via the communication device 133. In this case, the communication device 133 may function as an output device that outputs information to a device external to the robot control device 13. (2) Robot Control Processing

[0111] Next, the robot control process performed by the control device 3 will be described. For convenience of explanation, the following description will discuss the robot control process for controlling at least one of the robot 1, the end effector 4, and the robot movable device so as to perform the processes shown in FIGS. 5( a) to 5(e). That is, for convenience of explanation, the following description will discuss an example in which the control device 3 controls at least one of the robot 1, the end effector 4, and the robot movable device so as to perform the processes shown in FIGS. 5( a) to 5(e). Specifically, as shown in FIGS. 5( a) and 5(b), the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so as to perform a holding process in which the end effector 4 holds a first target object OBJ#1 (workpiece W in the example shown in FIGS. 5( a) to 5(e)) placed on the first placement device T#1. In this case, as shown in FIG. 5(a), the control device 3 may control at least one of the robot 1 and the robot movable device so as to move the end effector 4 closer to the first target object OBJ#1. That is, the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves toward the first target object OBJ#1. Then, as shown in FIG. 5( b), the control device 3 may control at least the end effector 4 so that the end effector 4 holds the first target object OBJ#1. Then, as shown in FIGS. 5( c) to 5( e), the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holding the first target object OBJ#1 performs a release process to release the first target object OBJ#1 held by the end effector 4 onto the second target object OBJ#2 (the second placement device T#2 in the example shown in FIGS. 5( a) to 5( e)). In this case, as shown in FIG. 5( c), the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 holding the first target object OBJ#1 approaches the second target object OBJ#2. In other words, the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 holding the first target object OBJ#1 moves toward the second target object OBJ#2.Thereafter, as shown in Fig. 5(d), the control device 3 may control at least the end effector 4 so that the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. Thereafter, as shown in Fig. 5(e), the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4, which has released the first target object OBJ#1, moves away from the second target object OBJ#2.

[0112] However, after the end effector 4 holds the first target object OBJ#1, another end effector provided on a robot arm other than the robot arm 12 may hold the first target object OBJ#1 held by the end effector 4. In this case, the control device 3 may control the end effector 4 so that the end effector 4 releases (i.e., releases) the first target object OBJ#1 without moving the end effector 4 holding the first target object OBJ#1. Thereafter, the other end effector holding the first target object OBJ#2 may release the first target object OBJ#2 onto the second target object OBJ#2. (2-1) Overall Flow of Robot Control Processing

[0113] First, the overall flow of the robot control process performed by the control device 3 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the overall flow of the robot control process.

[0114] 6, the control device 3 (particularly the arithmetic device 31) performs a pre-setting process to control at least one of the robot 1, the end effector 4, and the robot movable device (step S1). The pre-setting process is an initial setting process that must be performed in advance to control at least one of the robot 1, the end effector 4, and the robot movable device. The pre-setting process will be described in more detail later with reference to FIG. 7, etc.

[0115] 6, the control device 3 (particularly, the arithmetic device 31) controls at least one of the robot 1, the end effector 4, and the robot movable device to hold the first target object OBJ#1 (step S2). As a result, the end effector 4 approaches the first target object OBJ#1, and then the end effector 4 holds the first target object OBJ#1.

[0116] 6, the control device 3 (particularly, the arithmetic device 31) controls at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holding the first target object OBJ#1 releases the first target object OBJ#1 to the second target object OBJ#2 (step S3). As a result, the end effector 4 holding the first target object OBJ#1 approaches the second target object OBJ#2, and then the end effector 4 releases the first target object OBJ#1 to the second target object OBJ#2. (2-2) Presetting Process Next, the presetting process performed by the control device 3 in step S1 of FIG. 6 will be further described. Note that the presetting process may be performed by a device other than the control device 3. For example, as will be described in detail later, in the case where the pre-setting process includes at least two of a process of registering the holding position and orientation information 71, a process of registering the release position and orientation information 72, and a process of generating a template model TM, the control device 3 may perform at least one of the process of registering the holding position and orientation information 71, the process of registering the release position and orientation information 72, and the process of generating a template model TM, and a device (control device) different from the control device 3 may perform at least another of the process of registering the holding position and orientation information 71, the process of registering the release position and orientation information 72, and the process of generating a template model TM. (2-2-1) Registration of Holding Position and Orientation Information 71

[0117] The presetting process may include a process of registering holding position and orientation information 71. The holding position and orientation information 71 includes information about the holding position and orientation. In particular, the holding position and orientation information 71 includes information about candidates for the holding position and orientation (in other words, candidates that can be used as the target holding position and orientation). In this case, the control device 3 (arithmetic device 31) may register the holding position and orientation information 71. That is, the control device 3 (arithmetic device 31) may register the holding position and orientation. Note that the holding position and orientation information 71 registered by the presetting process may be referred to as registered position and orientation information or registered position and orientation information for holding. The holding position and orientation registered by the presetting process may be referred to as registered position and orientation or registered position and orientation for holding.

[0118] As shown in FIG. 7 conceptually illustrating an example of the holding position and orientation information 71, the holding position and orientation information 71 may include holding position information 711. The holding position information 711 is information regarding the holding position of the first target object OBJ#1 by the end effector 4. In particular, the holding position information 711 includes information regarding candidate holding positions (in other words, candidates that can be used as target holding positions). More specifically, the holding position information 711 is information regarding positions that have been registered in advance as candidate holding positions for the first target object OBJ#1 by the end effector 4. In this case, the holding position and orientation indicated by the holding position and orientation information 71 may be considered to include the holding position of the first target object OBJ#1 by the end effector 4. Note that the holding position information 711 registered by the pre-setting process may also be referred to as registered position information or registered holding position information. The holding position registered by the pre-setting process may also be referred to as a registered position or a registered holding position.

[0119] The holding position information 711 may indicate the position on the first target object OBJ#1 held by the end effector 4, as an example of the holding position of the first target object OBJ#1 by the end effector 4. The holding position on the first target object OBJ#1 by the end effector 4 may refer to the position on the first target object OBJ#1 that the end effector 4 contacts in order to hold the first target object OBJ#1. For example, if the end effector 4 is a hand gripper, the holding position information 711 may indicate the position on the first target object OBJ#1 that the finger members or claw members of the hand gripper contact. For example, if the end effector 4 is a vacuum gripper, the holding position information 711 may indicate the position on the first target object OBJ#1 that the suction ports of the vacuum device of the vacuum gripper contact. For example, if the end effector 4 is a magnetic gripper, the holding position information 711 may indicate the position on the first target object OBJ#1 that the magnetic attraction device of the magnetic gripper comes into contact with.

[0120] In this case, the control device 3 may register the position on the first target object OBJ#1 where the end effector 4 will be held by performing a pre-setting process in step S1 of FIG. 6 . Then, in step S2 of FIG. 6 , the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves to a position on the first target object OBJ#1 indicated by the holding position information 711 where the end effector 4 can hold it. That is, the signal generator 312 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves until the end effector 4 can hold the first target object OBJ#1 at the holding position and orientation indicated by the holding position and orientation information 71. Then, in step S2 of FIG. 6 , the control device 3 may control the end effector 4 to hold the first target object OBJ#1 after the end effector 4 moves to a position on the first target object OBJ#1 indicated by the holding position information 711 where the end effector 4 can hold it.

[0121] The holding position information 711 may indicate the positional relationship between the first target object OBJ#1 and the end effector 4 for holding the first target object OBJ#1, as an example of the holding position of the first target object OBJ#1 by the end effector 4. In other words, the holding position information 711 may indicate the positional relationship between the first target object OBJ#1 and the end effector 4 at the time when the end effector 4 holds the first target object OBJ#1. Note that the positional relationship between the first target object OBJ#1 and the end effector 4 may refer to the relationship between the position of the first target object OBJ#1 and the position of the end effector 4. Furthermore, the first target object OBJ#1 and the end effector 4 in the positional relationship indicated by the holding position information 711 may be in contact with each other or may be separated from each other. In this case, the holding position may be considered to be set on the first target object OBJ#1, or may be considered to be set at a position separated from the first target object OBJ#1. Furthermore, "the time when the end effector 4 holds the first target object OBJ#1" may mean "the time when the end effector 4 starts to hold the first target object OBJ#1." For example, if the end effector 4 is a hand gripper that physically grips the first target object OBJ#1 using multiple finger members or claw members, "the time when the end effector 4 holds the first target object OBJ#1" may mean "the time when the state of the multiple finger members or claw members is switched from an open state to a closed state." For example, if the end effector 4 is a vacuum gripper that vacuum-sucks the first target object OBJ#1, "the time when the end effector 4 holds the first target object OBJ#1" may mean "the time when the vacuum device of the vacuum gripper is switched from off to on." For example, if the end effector 4 is a magnetic gripper that magnetically attracts the first target object OBJ#1, "the point at which the end effector 4 holds the first target object OBJ#1" may mean "the point at which the magnetic attraction device of the magnetic gripper is switched from off to on."

[0122] In this case, the control device 3 may register the positional relationship between the first target object OBJ#1 and the end effector 4 by performing a pre-setting process in step S1 of Fig. 6. Thereafter, in step S2 of Fig. 6, the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves until the positional relationship between the first target object OBJ#1 and the end effector 4 becomes the positional relationship indicated by the holding position information 711. In other words, the signal generation unit 312 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves until the end effector 4 can hold the first target object OBJ#1 at the holding position and orientation indicated by the holding position and orientation information 71 (particularly, indicated by the holding position information 711). Then, in step S2 of Figure 6, the control device 3 may control the end effector 4 to hold the first target object OBJ#1 after the positional relationship between the first target object OBJ#1 and the end effector 4 becomes the positional relationship indicated by the holding position information 711.

[0123] The holding position information 711 may indicate the position of the end effector 4 for holding the first target object OBJ#1, as an example of the holding position of the first target object OBJ#1 by the end effector 4. In particular, the holding position information 711 may indicate the position of the end effector 4 for holding the first target object OBJ#1 relative to the first target object OBJ#1. In other words, the holding position information 711 may indicate the position of the end effector 4 for holding the first target object OBJ#1 in a coordinate system based on the first target object OBJ#1. More specifically, the holding position information 711 may indicate the position of the end effector 4 at the time when the end effector 4 holds the first target object OBJ#1. In particular, the holding position information 711 may indicate the position of the end effector 4 relative to the first target object OBJ#1 at the time when the end effector 4 holds the first target object OBJ#1. In other words, the holding position information 711 may indicate the position of the end effector 4 in a coordinate system based on the first target object OBJ#1 at the time when the end effector 4 holds the first target object OBJ#1. The end effector 4 located at the position indicated by the holding position information 711 may be in contact with the first target object OBJ#1 or may be separated from the first target object OBJ#1. The position indicated by the holding position information 711 may be a position on the first target object OBJ#1 or a position separated from the first target object OBJ#1. In this case, the holding position may be considered to be set on the first target object OBJ#1 or may be considered to be set at a position separated from the first target object OBJ#1. The holding position information 711 indicating the position of the end effector 4 for holding the first target object OBJ#1 may be referred to as first holding position information.

[0124] In this case, the control device 3 may register the position of the end effector 4 relative to the first target object OBJ#1 by performing a pre-setting process in step S1 of FIG. 6 . Thereafter, in step S2 of FIG. 6 , the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves relative to the first target object OBJ#1 until the position of the end effector 4 relative to the first target object OBJ#1 becomes the position indicated by the holding position information 711. In other words, the signal generator 312 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves until the end effector 4 can hold the first target object OBJ#1 at the holding position and orientation indicated by the holding position and orientation information 71 (particularly, indicated by the holding position information 711). Thereafter, in step S2 of FIG. 6 , the control device 3 may control the end effector 4 so that it holds the first target object OBJ#1 after the position of the end effector 4 relative to the first target object OBJ#1 becomes the position indicated by the holding position information 711.

[0125] The position of the end effector 4 may refer to the position of a reference point of the end effector 4. The tool center point of the end effector 4 may be used as the reference point of the end effector 4. However, any point determined based on the end effector 4, different from the tool center point, may be used as the reference point of the end effector 4. The reference point of the end effector 4 may be set on the end effector 4 or may be set at a position away from the end effector 4. As an example, the center of gravity of the end effector 4 may be used as the reference point of the end effector 4.

[0126] The holding position information 711 may indicate the position of the first target object OBJ#1 to be held by the end effector 4, as an example of the holding position of the first target object OBJ#1 by the end effector 4. In particular, the holding position information 711 may indicate the position of the first target object OBJ#1 to be held by the end effector 4 relative to the end effector 4. In other words, the holding position information 711 may indicate the position of the first target object OBJ#1 to be held by the end effector 4 in a coordinate system based on the end effector 4. More specifically, the holding position information 711 may indicate the position of the first target object OBJ#1 at the time the end effector 4 holds the first target object OBJ#1. In particular, the holding position information 711 may indicate the position of the first target object OBJ#1 relative to the end effector 4 at the time the end effector 4 holds the first target object OBJ#1. In other words, the holding position information 711 may indicate the position of the first target object OBJ#1 in a coordinate system based on the end effector 4 at the time when the end effector 4 holds the first target object OBJ#1. The first target object OBJ#1 located at the position indicated by the holding position information 711 may be in contact with the end effector 4 or may be separated from the end effector 4. The position indicated by the holding position information 711 may be a position on the end effector 4 or a position separated from the end effector 4. In this case, the holding position may be considered to be set on the end effector 4 or may be considered to be set at a position separated from the end effector 4. The holding position information 711 indicating the position of the first target object OBJ#1 to be held by the end effector 4 may be referred to as second holding position information.

[0127] In this case, the control device 3 may register the position of the first target object OBJ#1 relative to the end effector 4 by performing a pre-setting process in step S1 of FIG. 6 . Thereafter, in step S2 of FIG. 6 , the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves until the position of the first target object OBJ#1 relative to the end effector 4 becomes the position indicated by the holding position information 711. In other words, the signal generator 312 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves until the end effector 4 can hold the first target object OBJ#1 at the holding position and orientation indicated by the holding position and orientation information 71 (particularly, indicated by the holding position information 711). Thereafter, in step S2 of FIG. 6 , the control device 3 may control the end effector 4 to hold the first target object OBJ#1 after the first target object OBJ#1 relative to the end effector 4 becomes the position indicated by the holding position information 711.

[0128] As shown in FIG. 7 conceptually illustrating an example of the holding position and orientation information 71, the holding position and orientation information 71 may include first holding orientation information 712 in addition to or instead of the holding position information 711. The first holding orientation information 712 is information regarding the orientation (holding orientation) of the end effector 4 holding the first target object OBJ#1. In particular, the first holding orientation information 712 includes information regarding candidates for the orientation (holding orientation) of the end effector 4 holding the first target object OBJ#1 (in other words, candidates that can be used as a target holding orientation). More specifically, the first holding orientation information 712 is information regarding orientations that are registered in advance as candidates for the orientation (holding orientation) of the end effector 4 holding the first target object OBJ#1. In this case, the holding position and orientation indicated by the holding position and orientation information 71 may be considered to include the orientation (holding orientation) of the end effector 4 holding the first target object OBJ#1. Note that the holding orientation information registered by the pre-setting process may also be referred to as registered orientation information or registered orientation information for holding. The holding posture registered by the presetting process may be referred to as a registered posture or a registered posture for holding.

[0129] The first holding orientation information 712 may indicate the orientation of the end effector 4 for holding the first target object OBJ#1. In particular, the first holding orientation information 712 may indicate the orientation of the end effector 4 for holding the first target object OBJ#1 with respect to the first target object OBJ#1. In other words, the first holding orientation information 712 may indicate the orientation of the end effector 4 for holding the first target object OBJ#1 in a coordinate system based on the first target object OBJ#1. More specifically, the first holding orientation information 712 may indicate the orientation of the end effector 4 at the time when the end effector 4 holds the first target object OBJ#1. In particular, the first holding orientation information 712 may indicate the orientation of the end effector 4 with respect to the first target object OBJ#1 at the time when the end effector 4 holds the first target object OBJ#1. In other words, the first holding posture information 712 may indicate the posture of the end effector 4 in a coordinate system based on the first target object OBJ#1 at the time the end effector 4 holds the first target object OBJ#1.

[0130] In this case, the control device 3 may register the orientation of the end effector 4 with respect to the first target object OBJ#1 by performing a presetting process in step S1 of Fig. 6. Thereafter, in step S2 of Fig. 6, the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves until the orientation of the end effector 4 with respect to the first target object OBJ#1 becomes the orientation indicated by the first holding orientation information 712. In other words, the signal generator 312 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves until the end effector 4 can hold the first target object OBJ#1 at the holding position and orientation indicated by the holding position and orientation information 71 (particularly, indicated by the first holding orientation information 712). Then, in step S2 of Figure 6, the control device 3 may control the end effector 4 to hold the first target object OBJ#1 after the orientation of the end effector 4 relative to the first target object OBJ#1 becomes the orientation indicated by the first holding orientation information 712.

[0131] The first holding posture information 712 may indicate the posture relationship between the first target object OBJ#1 for holding the first target object OBJ#1 and the end effector 4, as an example of the posture of the end effector 4 holding the first target object OBJ#1. In other words, the first holding posture information 712 may indicate the posture relationship between the first target object OBJ#1 and the end effector 4 at the time when the end effector 4 holds the first target object OBJ#1. Note that the posture relationship between the first target object OBJ#1 and the end effector 4 may refer to the relationship between the posture of the first target object OBJ#1 and the posture of the end effector 4.

[0132] In this case, the control device 3 may register the posture relationship between the first target object OBJ#1 and the end effector 4 by performing a pre-setting process in step S1 of Fig. 6. Thereafter, in step S2 of Fig. 6, the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves until the posture relationship between the first target object OBJ#1 and the end effector 4 becomes the posture relationship indicated by the first holding posture information 712. In other words, the signal generation unit 312 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves until the end effector 4 can hold the first target object OBJ#1 in the holding position and posture indicated by the holding position and posture information 71 (particularly, indicated by the first holding posture information 712). Then, in step S2 of FIG. 6, the control device 3 may control the end effector 4 to hold the first target object OBJ#1 after the posture relationship between the first target object OBJ#1 and the end effector 4 becomes the posture relationship indicated by the first holding posture information 712.

[0133] As shown in FIG. 7 conceptually illustrating an example of the holding position and orientation information 71, the holding position and orientation information 71 may include second holding orientation information 713 in addition to or instead of at least one of the holding position information 711 and the first holding orientation information 712. The second holding orientation information 713 is information regarding the orientation (holding orientation) of the first target object OBJ#1 held by the end effector 4. In particular, the second holding orientation information 713 includes information regarding candidates for the orientation (holding orientation) of the first target object OBJ#1 held by the end effector 4 (in other words, candidates that can be used as a target holding orientation). More specifically, the second holding orientation information 713 is information regarding orientations that are registered in advance as candidates for the orientation (holding orientation) of the first target object OBJ#1 held by the end effector 4. In this case, the holding position and orientation indicated by the holding position and orientation information 71 may be considered to include the orientation (holding orientation) of the first target object OBJ#1 held by the end effector 4.

[0134] The second holding orientation information 713 may indicate the orientation of the first target object OBJ#1 held by the end effector 4. In particular, the second holding orientation information 713 may indicate the orientation of the first target object OBJ#1 held by the end effector 4 with respect to the end effector 4. In other words, the second holding orientation information 713 may indicate the orientation of the first target object OBJ#1 held by the end effector 4 in a coordinate system with the end effector 4 as the reference. More specifically, the second holding orientation information 713 may indicate the orientation of the first target object OBJ#1 at the time when the end effector 4 holds the first target object OBJ#1. In particular, the second holding orientation information 713 may indicate the orientation of the first target object OBJ#1 with respect to the end effector 4 with respect to the end effector 4 at the time when the end effector 4 holds the first target object OBJ#1. In other words, the second holding posture information 713 may indicate the posture of the first target object OBJ#1 in a coordinate system based on the end effector 4 at the time when the end effector 4 holds the first target object OBJ#1.

[0135] In this case, the control device 3 may register the orientation of the first target object OBJ#1 relative to the end effector 4 by performing a presetting process in step S1 of Fig. 6. Thereafter, in step S2 of Fig. 6, the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves until the orientation of the first target object OBJ#1 relative to the end effector 4 becomes the orientation indicated by the second holding orientation information 713. In other words, the signal generation unit 312 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves until the end effector 4 can hold the first target object OBJ#1 at the holding position and orientation indicated by the holding position and orientation information 71 (particularly, indicated by the second holding orientation information 713). Then, in step S2 of Figure 6, the control device 3 may control the end effector 4 to hold the first target object OBJ#1 after the orientation of the first target object OBJ#1 relative to the end effector 4 becomes the orientation indicated by the second holding orientation information 713.

[0136] The second holding attitude information 713 may indicate, as an example of the attitude of the first target object OBJ#1 held by the end effector 4, the attitude relationship between the first target object OBJ#1 held by the end effector 4 and the end effector 4. In other words, the second holding attitude information 713 may indicate the attitude relationship between the first target object OBJ#1 and the end effector 4 at the time when the end effector 4 holds the first target object OBJ#1.

[0137] In this case, the control device 3 may register the posture relationship between the first target object OBJ#1 and the end effector 4 by performing a pre-setting process in step S1 of Fig. 6. Thereafter, in step S2 of Fig. 6, the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves until the posture relationship between the first target object OBJ#1 and the end effector 4 becomes the posture relationship indicated by the second holding posture information 713. In other words, the signal generation unit 312 may control at least one of the robot 1 and the robot movable device so that the end effector 4 moves until the end effector 4 can hold the first target object OBJ#1 in the holding position and posture indicated by the holding position and posture information 71 (particularly, indicated by the second holding posture information 713). Then, in step S2 of FIG. 6, the control device 3 may control the end effector 4 to hold the first target object OBJ#1 after the posture relationship between the first target object OBJ#1 and the end effector 4 becomes the posture relationship indicated by the second holding posture information 713.

[0138] The control device 3 may register the holding position and orientation information 71 based on an instruction from a user (operator) of the robot system SYS. In this case, the control device 3 may display a holding position and orientation registration screen 61 that can be used by the user to input an instruction to register the holding position and orientation information 71. Specifically, the arithmetic device 31 of the control device 3 may generate a display signal for displaying the holding position and orientation registration screen 61 and output the generated display signal to the display device 37 of the control device 3. The display device 37 may display the holding position and orientation registration screen 61 based on the display signal generated by the arithmetic device 31.

[0139] An example of the holding position and orientation registration screen 61 is shown in FIG. 8 . As shown in FIG. 8 , a virtual three-dimensional space 61SP is constructed within the holding position and orientation registration screen 61. An object model OM#1, which is a three-dimensional model of the first target object OBJ#1, may be placed in the three-dimensional space 61SP. In this case, the user may use the input device 34 to specify, on the object model OM#1, a position on the first target object OBJ#1 where the end effector 4 will hold the first target object OBJ#1. The control device 3 may then register information indicating the position specified by the user as holding position information 711 indicating the position on the first target object OBJ#1 where the end effector 4 will hold the first target object OBJ#1.

[0140] In addition to the object model OM#1, an end effector model EM, which is a three-dimensional model of the end effector 4, may be placed in the three-dimensional space 61SP. In this case, the user may use the input device 34 to move at least one of the object model OM#1 and the end effector model EM within the three-dimensional space 61SP. The control device 3 may register the holding position and orientation information 71 based on at least one of the position and orientation of the object model OM#1 and the position and orientation of the end effector model EM after the user moves at least one of the object model OM#1 and the end effector model EM. For example, the control device 3 may register information indicating the positional relationship between the object model OM#1 and the end effector model EM as holding position information 711 indicating the positional relationship between the first target object OBJ#1 and the end effector 4 at the time when the end effector 4 holds the first target object OBJ#1. For example, the control device 3 may register information indicating the position of the end effector model EM relative to the object model OM#1 as holding position information 711 indicating the position of the end effector 4 at the time when the end effector 4 holds the first target object OBJ#1. For example, the control device 3 may register information indicating the position of the object model OM#1 relative to the end effector model EM as holding position information 711 indicating the position of the first target object OBJ#1 at the time when the end effector 4 holds the first target object OBJ#1. For example, the control device 3 may register information indicating the orientation relationship between the object model OM#1 and the end effector model EM as first holding orientation information 712 and second holding orientation information 713 indicating the orientation relationship between the first target object OBJ#1 and the end effector 4 at the time when the end effector 4 holds the first target object OBJ#1. For example, the control device 3 may register information indicating the attitude of the end effector model EM relative to the object model OM#1 as first holding attitude information 712 indicating the attitude of the end effector 4 relative to the first target object OBJ#1 at the time the end effector 4 holds the first target object OBJ#1.For example, the control device 3 may register information indicating the attitude of the object model OM#1 relative to the end effector model EM as second holding attitude information 713 indicating the attitude of the first target object OBJ#1 relative to the end effector 4 at the time the end effector 4 holds the first target object OBJ#1.

[0141] The control device 3 may register a single piece of holding position and orientation information 71. In this case, in step S2 of Fig. 6, the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holds the first target object OBJ#1 in the holding position and orientation indicated by the single piece of holding position and orientation information 71.

[0142] Alternatively, the control device 3 may register a plurality of different holding position and orientation information 71. Each of the plurality of different holding position and orientation information 71 indicates a plurality of different holding positions and orientations. For example, at least two of the plurality of different holding position and orientation information 71 may indicate at least two different holding positions. That is, each of at least two of the plurality of different holding position and orientation information 71 may include at least two different holding position information 711. For example, at least two of the plurality of different holding position and orientation information 71 may indicate at least two different holding orientations. That is, each of at least two of the plurality of different holding position and orientation information 71 may include at least two different first holding orientation information 712. Each of at least two of the plurality of different holding position and orientation information 71 may include at least two different second holding orientation information 713.

[0143] Examples of different holding positions and orientations are conceptually shown in Fig. 9(a) to Fig. 9(d). Fig. 9(a) shows a first holding position and orientation indicated by first holding position and orientation information 71, Fig. 9(b) shows a second holding position and orientation indicated by second holding position and orientation information 71, Fig. 9(c) shows a third holding position and orientation indicated by third holding position and orientation information 71, and Fig. 9(d) shows a fourth holding position and orientation indicated by fourth holding position and orientation information 71.

[0144] 6 , the control device 3 may select (in other words, determine) one piece of holding position and orientation information 71 from the plurality of pieces of holding position and orientation information 71, and control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 holds the first target object OBJ#1 at the holding position and orientation indicated by the selected (in other words, determined) piece of holding position and orientation information 71. In this case, compared to when only a single piece of holding position and orientation information 71 is registered, even if at least one of the position and orientation of the first target object OBJ#1 placed on the placement device T#1 varies (in other words, is not consistent), the selection of appropriate holding position and orientation information 71 corresponding to at least one of the position and orientation of the first target object OBJ#1 placed on the placement device T#1 increases the likelihood that the end effector 4 will be able to properly hold the first target object OBJ#1. As an example, even if a plurality of first target objects OBJ#1 are piled up in bulk on the placement device T#1 (for example, in the container CB), the end effector 4 is more likely to be able to hold the plurality of first target objects OBJ#1 in an appropriate order. (2-2-2) Registration of Release Position and Orientation Information 72

[0145] The pre-setting process may include a process of registering release position and orientation information 72 in addition to or instead of the process of registering the holding position and orientation information 71 described above. The release position and orientation information 72 includes information about the release position and orientation. In particular, the release position and orientation information 72 includes information about candidates for the release position and orientation (in other words, candidates that can be used as the target release position and orientation). In this case, the control device 3 (arithmetic device 31) may register the release position and orientation information 72. That is, the control device 3 (arithmetic device 31) may register the release position and orientation. Note that the release position and orientation information 72 registered by the pre-setting process may also be referred to as registered position and orientation information or release registered position and orientation information. The release position and orientation registered by the pre-setting process may also be referred to as registered position and orientation or release registered position and orientation.

[0146] As shown in FIG. 10 conceptually illustrating an example of the release position and orientation information 72, the release position and orientation information 72 may include release position information 721. The release position information 721 is information regarding the release position of the first target object OBJ#1 by the end effector 4. In particular, the release position information 721 includes information regarding candidate release positions (in other words, candidates that can be used as target release positions). More specifically, the release position information 721 is information regarding positions that have been registered in advance as candidate release positions for the first target object OBJ#1 by the end effector 4. In this case, the release position and orientation indicated by the release position and orientation information 72 may be considered to include the release position of the first target object OBJ#1 by the end effector 4. The release position information 721 registered by the pre-setting process may also be referred to as registered position information or registered release position information. The release position registered by the pre-setting process may also be referred to as registered position or registered release position.

[0147] The release position information 721 may indicate a position on the second target object OBJ#2 where the end effector 4 releases the first target object OBJ#1. For example, the release position information 721 may indicate a position on the second target object OBJ#2 where the first target object OBJ#1 released by the end effector 4 is to be placed.

[0148] In this case, the control device 3 may register a position on the second target object OBJ#2 where the end effector 4 releases the first target object OBJ#1 by performing a pre-setting process in step S1 of Fig. 6 . Thereafter, in step S3 of Fig. 6 , the control device 3 may control at least one of the robot 1 and the robot movable device so that at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 moves to a position where the end effector 4 can release the first target object OBJ#1 to the position indicated by the release position information 721. In other words, the signal generator 312 may control at least one of the robot 1 and the robot movable device so that at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 moves until the end effector 4 can release the first target object OBJ#1 onto the second target object OBJ#2 at the release position and orientation indicated by the release position and orientation information 72 (particularly, indicated by the release position information 721). Then, in step S3 of Figure 6, the control device 3 may control the end effector 4 to release the first target object OBJ#1 onto the second target object OBJ#2 after at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 has moved to a position where the end effector 4 can release the first target object OBJ#1 to the position indicated by the release position information 721.

[0149] The release position information 721 may indicate the positional relationship between at least one of the first target object OBJ#1 and the end effector 4 and the second target object OBJ#2 for releasing the first target object OBJ#1 onto the second target object OBJ#2. In other words, the release position information 721 may indicate the positional relationship between at least one of the first target object OBJ#1 and the end effector 4 and the second target object OBJ#2 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. Note that the positional relationship between at least one of the first target object OBJ#1 and the end effector 4 and the second target object OBJ#2 may refer to the relationship between the position of at least one of the first target object OBJ#1 and the end effector 4 and the position of the second target object OBJ#2. Furthermore, at least one of the first target object OBJ#1 and the end effector 4, which are in the positional relationship indicated by the release position information 721, may be in contact with or separated from the second target object OBJ#2. In this case, the release position may be considered to be set on the second target object OBJ#2, or may be considered to be set at a position separated from the second target object OBJ#2. Furthermore, "the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2" may mean "the time when the end effector 4 starts to release the first target object OBJ#1 onto the second target object OBJ#2." For example, if the end effector 4 is a hand gripper that physically grips the first target object OBJ#1 using multiple finger members or claw members, "the point in time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2" may mean "the point in time when the state of the multiple finger members or claw members is switched from a closed state to an open state." For example, if the end effector 4 is a vacuum gripper that vacuum-sucks the first target object OBJ#1, "the point in time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2" may mean "the point in time when the vacuum device of the vacuum gripper is switched from on to off."For example, if the end effector 4 is a magnetic gripper that magnetically attracts the first target object OBJ#1, "the point at which the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2" may mean "the point at which the magnetic attraction device of the magnetic gripper is switched from on to off."

[0150] In this case, the control device 3 may register the positional relationship between the second target object OBJ#2 and at least one of the first target object OBJ#1 and the end effector 4 by performing a pre-setting process in step S1 of Fig. 6. Thereafter, in step S3 of Fig. 6, the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 and at least one of the first target object OBJ#1 held by the end effector 4 move until the positional relationship between the second target object OBJ#2 and at least one of the first target object OBJ#1 and the end effector 4 becomes the positional relationship indicated by the release position information 721. That is, the signal generating unit 312 may control at least one of the robot 1 and the robot movable device so that at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 moves until the end effector 4 can release the first target object OBJ#1 onto the second target object OBJ#2 at the release position and posture indicated by the release position and posture information 72 (particularly, indicated by the release position information 721). Thereafter, in step S3 of Fig. 6, the control device 3 may control the end effector 4 to release the first target object OBJ#1 onto the second target object OBJ#2 after the positional relationship between at least one of the first target object OBJ#1 and the end effector 4 and the second target object OBJ#2 becomes the positional relationship indicated by the release position information 721.

[0151] The release position information 721 may indicate the position of at least one of the first target object OBJ#1 and the end effector 4 for releasing the first target object OBJ#1. In particular, the release position information 721 may indicate the position of at least one of the first target object OBJ#1 and the end effector 4 for releasing the first target object OBJ#1 relative to the second target object OBJ#2. In other words, the release position information 721 may indicate the position of at least one of the first target object OBJ#1 and the end effector 4 for releasing the first target object OBJ#1 in a coordinate system based on the second target object OBJ#2. More specifically, the release position information 721 may indicate the position of at least one of the first target object OBJ#1 and the end effector 4 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. In particular, the release position information 721 may indicate the position of at least one of the first target object OBJ#1 and the end effector 4 relative to the second target object OBJ#2 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. In other words, the release position information 721 may indicate the position of at least one of the first target object OBJ#1 and the end effector 4 in a coordinate system based on the second target object OBJ#2 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. Note that at least one of the first target object OBJ#1 and the end effector 4 located at the position indicated by the release position information 721 may be in contact with the second target object OBJ#2 or may be separated from the second target object OBJ#2. The position indicated by the release position information 721 may be a position on the second target object OBJ#2, or a position away from the second target object OBJ#2. In this case, the release position may be considered to be set on the second target object OBJ#2, or may be considered to be set at a position away from the second target object OBJ#2. Note that the release position information 721 indicating the positions of at least one of the first target object OBJ#1 and the end effector 4 for releasing the first target object OBJ#1 may be referred to as first release position information.

[0152] In this case, the control device 3 may register the position of at least one of the first target object OBJ#1 and the end effector 4 relative to the second target object OBJ#2 by performing a pre-setting process in step S1 of Fig. 6. Thereafter, in step S3 of Fig. 6, the control device 3 may control at least one of the robot 1 and the robot movable device so that at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 moves until the position of at least one of the first target object OBJ#1 and the end effector 4 relative to the second target object OBJ#2 becomes the position indicated by the release position information 721. That is, the signal generating unit 312 may control at least one of the robot 1 and the robot movable device so that at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 moves until the end effector 4 can release the first target object OBJ#1 onto the second target object OBJ#2 at the release position and posture indicated by the release position and posture information 72 (particularly, indicated by the release position information 721). Thereafter, in step S3 of Fig. 6, the control device 3 may control the end effector 4 so as to release the first target object OBJ#1 onto the second target object OBJ#2 after the position of at least one of the first target object OBJ#1 and the end effector 4 relative to the second target object OBJ#2 becomes the position indicated by the release position information 721.

[0153] The release position information 721 may indicate the position of the second target object OBJ#2 at which the end effector 4 releases the first target object OBJ#1. In particular, the release position information 721 may indicate the position of the second target object OBJ#2 at which the end effector 4 releases the first target object OBJ#1, relative to at least one of the first target object OBJ#1 and the end effector 4. In other words, the release position information 721 may indicate the position of the second target object OBJ#2 at which the end effector 4 releases the first target object OBJ#1, in a coordinate system based on at least one of the first target object OBJ#1 and the end effector 4. More specifically, the release position information 721 may indicate the position of the second target object OBJ#2 at the time the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. In particular, the release position information 721 may indicate the position of the second target object OBJ#2 relative to at least one of the first target object OBJ#1 and the end effector 4 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. In other words, the release position information 721 may indicate the position of the second target object OBJ#2 in a coordinate system based on at least one of the first target object OBJ#1 and the end effector 4 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. Note that the second target object OBJ#2 located at the position indicated by the release position information 721 may be in contact with at least one of the first target object OBJ#1 and the end effector 4, or may be separated from at least one of the first target object OBJ#1 and the end effector 4. The position indicated by the release position information 721 may be a position above at least one of the first target object OBJ#1 and the end effector 4, or may be a position away from at least one of the first target object OBJ#1 and the end effector 4. In this case, the release position may be considered to be set above at least one of the first target object OBJ#1 and the end effector 4, or may be considered to be set at a position away from at least one of the first target object OBJ#1 and the end effector 4.The release position information 721 indicating the position of the second target object OBJ#2 at which the end effector 4 releases the first target object OBJ#1 may be referred to as second release position information.

[0154] 6, the control device 3 may perform a pre-setting process to register the position of the second target object OBJ#2 relative to at least one of the first target object OBJ#1 and the end effector 4. Thereafter, in step S3 of FIG. 6, the control device 3 may control at least one of the robot 1 and the robot movable device so that at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 moves until the position of the second target object OBJ#2 relative to at least one of the first target object OBJ#1 and the end effector 4 becomes the position indicated by the release position information 721. That is, the signal generating unit 312 may control at least one of the robot 1 and the robot movable device so that at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 moves until the end effector 4 can release the first target object OBJ#1 onto the second target object OBJ#2 at the release position and posture indicated by the release position and posture information 72 (particularly, indicated by the release position information 721). Thereafter, in step S3 of Fig. 6, the control device 3 may control the end effector 4 so as to release the first target object OBJ#1 onto the second target object OBJ#2 after the second target object OBJ#2 relative to at least one of the first target object OBJ#1 and the end effector 4 reaches the position indicated by the release position information 721.

[0155] As shown in FIG. 10 conceptually illustrating an example of the release position and orientation information 72, the release position and orientation information 72 may include first release orientation information 722 in addition to or instead of the release position information 721. The first release orientation information 722 is information regarding the orientation (release orientation) of the end effector 4 releasing the first target object OBJ#1. In particular, the first release orientation information 722 is information regarding candidates for the orientation (release orientation) of the end effector 4 releasing the first target object OBJ#1 (in other words, candidates that can be used as a target release orientation). More specifically, the first release orientation information 722 is information regarding orientations that are registered in advance as candidates for the orientation (release orientation) of the end effector 4 releasing the first target object OBJ#1. In this case, the release position and orientation indicated by the release position and orientation information 72 may be considered to include the orientation (release orientation) of the end effector 4 releasing the first target object OBJ#1. The release orientation information registered by the presetting process may be referred to as registered orientation information or registered orientation information for release. The release orientation registered by the presetting process may be referred to as registered orientation or registered orientation for release.

[0156] The first release orientation information 722 may indicate the orientation of the end effector 4 for releasing the first target object OBJ#1. In particular, the first release orientation information 722 may indicate the orientation of the end effector 4 for releasing the first target object OBJ#1 with respect to the second target object OBJ#2. In other words, the first release orientation information 722 may indicate the orientation of the end effector 4 for releasing the first target object OBJ#1 in a coordinate system based on the second target object OBJ#2. More specifically, the first release orientation information 722 may indicate the orientation of the end effector 4 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. In particular, the first release orientation information 722 may represent the orientation of the end effector 4 with respect to the second target object OBJ#2 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. In other words, the first release orientation information 722 may represent the orientation of the end effector 4 in a coordinate system based on the second target object OBJ#2 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2.

[0157] In this case, the control device 3 may register the orientation of the end effector 4 with respect to the second target object OBJ#2 by performing a presetting process in step S1 of Fig. 6 . Thereafter, in step S3 of Fig. 6 , the control device 3 may control at least one of the robot 1 and the robot movable device so that at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 moves until the orientation of the end effector 4 with respect to the second target object OBJ#2 becomes the orientation indicated by the first release orientation information 722. In other words, the signal generator 312 may control at least one of the robot 1 and the robot movable device so that at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 moves until the end effector 4 is able to release the first target object OBJ#1 onto the second target object OBJ#2 at the release position and orientation indicated by the release position and orientation information 72 (particularly, indicated by the first release orientation information 722). Then, in step S3 of Figure 6, the control device 3 may control the end effector 4 to release the first target object OBJ#1 onto the second target object OBJ#2 after the orientation of the end effector 4 with respect to the second target object OBJ#2 becomes the orientation indicated by the first release orientation information 722.

[0158] The first release orientation information 722 may indicate the orientation relationship between the end effector 4 for releasing the first target object OBJ#1 and the second target object OBJ#2. In other words, the first release orientation information 722 may indicate the orientation relationship between the second target object OBJ#2 and the end effector 4 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. Note that the orientation relationship between the second target object OBJ#2 and the end effector 4 may refer to the relationship between the orientation of the second target object OBJ#2 and the orientation of the end effector 4.

[0159] 6, the control device 3 may register the posture relationship between the second target object OBJ#2 and the end effector 4. Then, in step S3 of FIG. 6, the control device 3 may control at least one of the robot 1 and the robot movable device so that at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 moves until the posture relationship between the second target object OBJ#2 and the end effector 4 becomes the posture relationship indicated by the first release posture information 722. In other words, the signal generating unit 312 may control the end effector 4 to move at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 until the end effector 4 is able to release the first target object OBJ#1 onto the second target object OBJ#2 at the release position and posture indicated by the release position and posture information 72 (particularly, indicated by the first release posture information 722), and then, in step S3 of Figure 6, the control device 3 may control the end effector 4 to release the first target object OBJ#1 onto the second target object OBJ#2 after the posture relationship between the second target object OBJ#2 and the end effector 4 becomes the posture relationship indicated by the first release posture information 722.

[0160] As shown in FIG. 10 conceptually illustrating an example of the release position and orientation information 72, the release position and orientation information 72 may include second release orientation information 723 in addition to or instead of at least one of the release position information 721 and the first release orientation information 722. The second release orientation information 723 is information regarding the orientation (release orientation) of the first target object OBJ#1 when released to the second target object OBJ#2. In particular, the second release orientation information 723 is information regarding candidates for the orientation (release orientation) of the first target object OBJ#1 when released to the second target object OBJ#2 (in other words, candidates that can be used as a target release orientation). More specifically, the second release orientation information 723 is information regarding orientations that are registered in advance as candidates for the orientation (release orientation) of the first target object OBJ#1 when released to the second target object OBJ#2. In this case, the release position and orientation indicated by the release position and orientation information 72 may be considered to include the orientation (release orientation) of the first target object OBJ#1 released onto the second target object OBJ#2.

[0161] The second release orientation information 723 may indicate the orientation of the first target object OBJ#1 when released onto the second target object OBJ#2. In particular, the second release orientation information 723 may indicate the orientation of the first target object OBJ#1 when released onto the second target object OBJ#2, relative to the second target object OBJ#2. In other words, the second release orientation information 723 may indicate the orientation of the first target object OBJ#1 when released onto the second target object OBJ#2, in a coordinate system based on the second target object OBJ#2. More specifically, the second release orientation information 723 may indicate the orientation of the first target object OBJ#1 at the time the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. In particular, the second release orientation information 723 may represent the orientation of the first target object OBJ#1 relative to the second target object OBJ#2 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. In other words, the second release orientation information 723 may represent the orientation of the first target object OBJ#1 in a coordinate system based on the second target object OBJ#2 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2.

[0162] In this case, the control device 3 may register the orientation of the first target object OBJ#1 relative to the second target object OBJ#2 by performing a pre-setting process in step S1 of Fig. 6. Thereafter, in step S3 of Fig. 6, the control device 3 may control at least one of the robot 1 and the robot movable device so that at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 moves until the orientation of the first target object OBJ#1 relative to the second target object OBJ#2 becomes the orientation indicated by the second release orientation information 723. In other words, the signal generating unit 312 may control the end effector 4 and at least one of the first target object OBJ#1 held by the end effector 4 to move until the end effector 4 is able to release the first target object OBJ#1 onto the second target object OBJ#2 at the release position and posture indicated by the release position and posture information 72 (particularly, indicated by the second release posture information 723), and then, in step S3 of Figure 6, the control device 3 may control the end effector 4 to release the first target object OBJ#1 onto the second target object OBJ#2 after the posture of the first target object OBJ#1 relative to the second target object OBJ#2 becomes the posture indicated by the second release posture information 723.

[0163] The second release orientation information 723 may indicate the orientation relationship between the first target object OBJ#1 and the second target object OBJ#2 when they are released onto the second target object OBJ#2. In other words, the second release orientation information 723 may indicate the orientation relationship between the second target object OBJ#2 and the first target object OBJ#1 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. Note that the orientation relationship between the second target object OBJ#2 and the first target object OBJ#1 may refer to the relationship between the orientation of the second target object OBJ#2 and the orientation of the first target object OBJ#1.

[0164] In this case, the control device 3 may register the posture relationship between the second target object OBJ#2 and the first target object OBJ#1 by performing a pre-setting process in step S1 of Fig. 6. Thereafter, in step S3 of Fig. 6, the control device 3 may control at least one of the robot 1 and the robot movable device so that at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 moves until the posture relationship between the second target object OBJ#2 and the first target object OBJ#1 becomes the posture relationship indicated by the second release posture information 723. In other words, the signal generating unit 312 may control the end effector 4 to move at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 until the end effector 4 is able to release the first target object OBJ#1 onto the second target object OBJ#2 at the release position and posture indicated by the release position and posture information 72 (particularly, indicated by the second release posture information 723), and then, in step S3 of Figure 6, the control device 3 may control the end effector 4 to release the first target object OBJ#1 onto the second target object OBJ#2 after the posture relationship between the second target object OBJ#2 and the first target object OBJ#1 becomes the posture relationship indicated by the second release posture information 723.

[0165] 10 conceptually illustrates an example of the release position and orientation information 72. The release position and orientation information 72 may include third release orientation information 724 in addition to or instead of the release position information 721, first release orientation information 722, and second release orientation information 723. The third release orientation information 724 is information regarding the orientation (release orientation) of the second target object OBJ#2 when the end effector 4 releases the first target object OBJ#1. In particular, the third release orientation information 724 is information regarding candidates for the orientation (release orientation) of the second target object OBJ#2 when the end effector 4 releases the first target object OBJ#1 (in other words, candidates that can be used as a target release orientation). More specifically, the third release orientation information 724 is information regarding orientations that have been registered in advance as candidates for the orientation (release orientation) of the second target object OBJ#2 when the end effector 4 releases the first target object OBJ#1. In this case, the release position and posture indicated by the release position and posture information 72 may be considered to include the posture (release posture) of the second target object OBJ#2 when the end effector 4 releases the first target object OBJ#1.

[0166] The third release orientation information 724 may indicate the orientation of the second target object OBJ#2 when the end effector 4 releases the first target object OBJ#1. In particular, the third release orientation information 724 may indicate the orientation of the second target object OBJ#2 when the end effector 4 releases the first target object OBJ#1, relative to at least one of the first target object OBJ#1 and the end effector 4. In other words, the third release orientation information 724 may indicate the orientation of the second target object OBJ#2 when the end effector 4 releases the first target object OBJ#1, in a coordinate system based on at least one of the first target object OBJ#1 and the end effector 4. More specifically, the third release orientation information 724 may indicate the orientation of the second target object OBJ#2 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. In particular, the third release orientation information 724 may represent the orientation of the second target object OBJ#2 relative to at least one of the first target object OBJ#1 and the end effector 4 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. In other words, the third release orientation information 724 may represent the orientation of the second target object OBJ#2 in a coordinate system based on at least one of the first target object OBJ#1 and the end effector 4 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2.

[0167] 6, the control device 3 may perform a pre-setting process to register the orientation of the second target object OBJ#2 relative to at least one of the first target object OBJ#1 and the end effector 4. Thereafter, in step S3 of FIG. 6, the control device 3 may control at least one of the robot 1 and the robot movable device so that at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 moves until the orientation of the second target object OBJ#2 relative to at least one of the first target object OBJ#1 and the end effector 4 becomes the orientation indicated by the third release orientation information 724. In other words, the signal generating unit 312 may control the end effector 4 to move at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 until the end effector 4 is able to release the first target object OBJ#1 onto the second target object OBJ#2 at the release position and posture indicated by the release position and posture information 72 (particularly, indicated by the third release posture information 724), and then, in step S3 of Figure 6, the control device 3 may control the end effector 4 to release the first target object OBJ#1 onto the second target object OBJ#2 after the posture of the first target object OBJ#1 relative to the end effector 4 becomes the posture indicated by the third release posture information 724.

[0168] The third release orientation information 724 may indicate the orientation relationship between the second target object OBJ#2 when the end effector 4 releases the first target object OBJ#1, and at least one of the first target object OBJ#1 and the end effector 4. In other words, the third release orientation information 724 may indicate the orientation relationship between the second target object OBJ#2 and at least one of the first target object OBJ#1 and the end effector 4 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2.

[0169] In this case, the control device 3 may register the posture relationship between the second target object OBJ#2 and at least one of the first target object OBJ#1 and the end effector 4 by performing a pre-setting process in step S1 of Fig. 6. Thereafter, in step S3 of Fig. 6, the control device 3 may control at least one of the robot 1 and the robot movable device so that the end effector 4 and at least one of the first target object OBJ#1 held by the end effector 4 move until the posture relationship between the second target object OBJ#2 and at least one of the first target object OBJ#1 and the end effector 4 becomes the posture relationship indicated by the third release posture information 724. In other words, the signal generating unit 312 may control the end effector 4 to move at least one of the end effector 4 and the first target object OBJ#1 held by the end effector 4 until the end effector 4 is able to release the first target object OBJ#1 onto the second target object OBJ#2 at the release position and posture indicated by the release position and posture information 72 (particularly, indicated by the third release posture information 724), and then, in step S3 of Figure 6, the control device 3 may control the end effector 4 to release the first target object OBJ#1 onto the second target object OBJ#2 after the posture relationship between at least one of the first target object OBJ#1 and the end effector 4 and the second target object OBJ#2 becomes the posture relationship indicated by the third release posture information 724.

[0170] The control device 3 may register the release position and posture information 72 based on an instruction from a user (operator) of the robot system SYS. In this case, the control device 3 may display a release position and posture registration screen 62 that can be used by the user to input an instruction to register the release position and posture information 72. Specifically, the arithmetic device 31 of the control device 3 may generate a display signal for displaying the release position and posture registration screen 62 and output the generated display signal to the display device 37 of the control device 3. The display device 37 may display the release position and posture registration screen 62 based on the display signal generated by the arithmetic device 31.

[0171] An example of the release position and orientation registration screen 62 is shown in FIG. 11 . As shown in FIG. 11 , a virtual three-dimensional space 62SP is constructed within the release position and orientation registration screen 62. An object model OM#2, which is a three-dimensional model of the second target object OBJ#2, may be placed in the three-dimensional space 62SP. In this case, the user may use the input device 34 to specify on the object model OM#2 the position on the second target object OBJ#2 at which the end effector 4 releases the first target object OBJ#1. The control device 3 may then register information indicating the user-specified position as release position information 721 indicating the position on the second target object OBJ#2 at which the end effector 4 releases the first target object OBJ#1.

[0172] In addition to the object model OM#2, at least one of an end effector model EM which is a three-dimensional model of the end effector 4 and an object model OM#1 which is a three-dimensional model of the first target object OBJ#1 may be arranged in the three-dimensional space 62SP. In this case, the user may move at least one of the object model OM#1, the object model OM#2, and the end effector model EM within the three-dimensional space 62SP using the input device 34. The control device 3 may register the release position and orientation information 72 based on at least one of the position and orientation of the object model OM#1, the position and orientation of the object model OM#2, and the position and orientation of the end effector model EM after the user moves at least one of the object model OM#1, the object model OM#2, and the end effector model EM. For example, the control device 3 may register information indicating the positional relationship between the object model OM#2 and at least one of the object model OM#1 and the end effector model EM as release position information 721 indicating the positional relationship between the second object OBJ#2 and at least one of the first target object OBJ#1 and the end effector 4 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. For example, the control device 3 may register information indicating the position of at least one of the object model OM#1 and the end effector model EM relative to the object model OM#2 as release position information 721 indicating the position of at least one of the first target object OBJ#1 and the end effector 4 relative to the second target object OBJ#2 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. For example, the control device 3 may register information indicating the position of the object model OM#2 relative to at least one of the object model OM#1 and the end effector model EM as release position information 721 indicating the position of the second target object OBJ#2 relative to at least one of the first target object OBJ#1 and the end effector 4 at the time the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2.For example, the control device 3 may register information indicating the orientation relationship between the object model OM#1 and the end effector model EM and the object model OM#2 as at least one of first release orientation information 722 to third release orientation information 724, which indicate the orientation relationship between at least one of the first target object OBJ#1 and the end effector 4 and the second target object OBJ#2 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. For example, the control device 3 may register information indicating the orientation of at least one of the object model OM#1 and the end effector model EM relative to the object model OM#2 as first release orientation information 722 or second release orientation information 723, which indicate the orientation of at least one of the first target object OBJ#1 and the end effector 4 relative to the second target object OBJ#2 at the time when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. For example, the control device 3 may register information indicating the attitude of the object model OM#2 relative to the object model OM#1 and the end effector model EM as third release attitude information 724 indicating the attitude of the second target object OBJ#2 relative to at least one of the first target object OBJ#1 and the end effector 4 at the time the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2.

[0173] The control device 3 may register a single piece of release position and orientation information 72. In this case, in step S3 of Fig. 6, the control device 3 may control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2 at the release position and orientation indicated by the single piece of release position and orientation information 72.

[0174] Alternatively, the control device 3 may register a plurality of different release position and orientation information 72. Each of the plurality of different release position and orientation information 72 indicates a plurality of different release positions and orientations. For example, at least two of the plurality of different release position and orientation information 72 may indicate at least two different release positions. That is, at least two of the plurality of different release position and orientation information 72 may include at least two different release position information 721. For example, at least two of the plurality of different release position and orientation information 72 may indicate at least two different release orientations. That is, at least two of the plurality of different release position and orientation information 72 may include at least two different first release orientation information 722. At least two of the plurality of different release position and orientation information 72 may include at least two different second release orientation information 723. At least two of the plurality of different release position and orientation information 72 may include at least two different third release orientation information 724.

[0175] Examples of different release positions and orientations are conceptually shown in Figures 12(a) to 12(d). Figure 12(a) shows a first release position and orientation indicated by the first release position and orientation information 72, Figure 12(b) shows a second release position and orientation indicated by the second release position and orientation information 72, Figure 12(c) shows a third release position and orientation indicated by the third release position and orientation information 72, and Figure 12(d) shows a fourth release position and orientation indicated by the fourth release position and orientation information 72.

[0176] 6 , the control device 3 may select (in other words, determine) one piece of release position and orientation information 72 from the plurality of pieces of release position and orientation information 72, and control at least one of the robot 1, the end effector 4, and the robot movable device so that the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2 at the release position and orientation indicated by the selected (in other words, determined) piece of release position and orientation information 72. In this case, compared to when only a single piece of release position and orientation information 72 is registered, even if at least one of the position and orientation of the first target object OBJ#1 held by the end effector 4 varies (in other words, is not consistent), the selection of appropriate release position and orientation information 72 corresponding to at least one of the position and orientation of the first target object OBJ#1 held by the end effector 4 increases the likelihood that the end effector 4 will be able to appropriately release the first target object OBJ#1 onto the second target object OBJ#2. Furthermore, compared to a case where only a single release position and orientation information 72 is registered, even if at least one of the position and orientation of the second target object OBJ#2 onto which the first target object OBJ#1 is released varies (in other words, is not uniform), by selecting appropriate release position and orientation information 72 according to at least one of the position and orientation of the second target object OBJ#2 onto which the first target object OBJ#1 is released, the end effector 4 is more likely to be able to appropriately release the first target object OBJ#1 onto the second target object OBJ#2. (2-2-3) Generation of Template Model TM

[0177] The pre-setting process may include a process of generating a template model TM in addition to or instead of at least one of the processes of registering the holding position and orientation information 71 and registering the release position and orientation information 72. The template model TM is used to calculate at least one of the position and orientation of a target object OBJ (at least one of the first target object OBJ#1 and the second target object OBJ#2 in the examples shown in FIGS. 5A to 5E ) on which the end effector 4 performs a predetermined process. Specifically, in this embodiment, to calculate at least one of the position and orientation of the target object OBJ, the control device 3 performs a matching process based on the template model TM representing at least a portion of the target object OBJ and image data IMG generated by the imaging system 2 by capturing an image of the target object OBJ, as will be described in detail later. The pre-setting process may include a process of generating the template model TM used in this matching process. The template model TM may also be referred to as a model representing the shape of at least a portion of the target object OBJ. In the following description, for the sake of simplicity, the term "template model TM indicating the target object OBJ" refers to a "template model TM indicating at least a part of the target object OBJ." It can also be said that the term "template model TM indicating the shape of at least a part of the target object OBJ" refers to a "template model TM indicating the shape of at least a part of the target object OBJ."

[0178] When the end effector 4 holds the first target object OBJ#1 as described above, the control device 3 calculates at least one of the position and orientation of the first target object OBJ#1 by performing a matching process based on a template model TM that indicates the shape of the first target object OBJ#1, as will be described in detail later. Furthermore, when the first target object OBJ#1 held by the end effector 4 is released onto the second target object OBJ#2 as described above, the control device 3 may also calculate at least one of the position and orientation of the first target object OBJ#1 by performing a matching process based on the template model TM that indicates the shape of the first target object OBJ#1, as will be described in detail later. In this case, the control device 3 may generate a template model TM that indicates the shape of the first target object OBJ#1. Furthermore, when the first target object OBJ#1 held by the end effector 4 is released onto the second target object OBJ#2 as described above, the control device 3 may calculate at least one of the position and orientation of the second target object OBJ#2 by performing a matching process based on a template model TM that indicates the shape of the second target object OBJ#2, as will be described in detail later. In this case, the control device 3 may generate the template model TM that indicates the shape of the second target object OBJ#2.

[0179] When the shape of the first target object OBJ#1 and the shape of the second target object OBJ#2 are different, the control device 3 may separately generate a template model TM that indicates the shape of the first target object OBJ#1 and a template model TM that indicates the shape of the second target object OBJ#2. When the shape of the first target object OBJ#1 and the shape of the second target object OBJ#2 are the same, the control device 3 may generate a common template model TM that indicates the shape of the first target object OBJ#1 and the shape of the second target object OBJ#2.

[0180] Below, as a premise for explaining the process of generating the template model TM, the template model TM and the matching process based on the template model TM will be explained first. After that, the process of generating the template model TM will be explained. (2-2-3-1) Template Model TM

[0181] The template model TM is a model that represents at least a part of the target object OBJ. The template model TM is a model that represents the shape of at least a part of the target object OBJ. The template model TM may be a model that represents the two-dimensional shape of at least a part of the target object OBJ. The template model TM may be a model that represents the three-dimensional shape of at least a part of the target object OBJ. Examples of the template model TM are shown in FIGS. 13( a) to 13(c).

[0182] As shown in FIG. 13A, the template model TM may include a contour model TM_edge. The contour model TM_edge may represent at least a portion of the contour OL of the target object OBJ. In other words, the contour model TM_edge may not represent the portion SL of the target object OBJ that is surrounded by the contour OL. In other words, the contour model TM_edge may be a model that represents at least a portion of the target object OBJ, using the contour OL of the target object OBJ but not using the portion SL of the target object OBJ that is surrounded by the contour OL. Note that the contour OL of the target object OBJ may be referred to as the edge of the target object OBJ. In this case, the contour model TM_edge may be referred to as an edge model.

[0183] As shown in FIGS. 13( b) and 13(c), the template model TM may include a multidimensional model TM_M. The multidimensional model TM_M may represent both at least a portion of the contour OL of the target object OBJ and at least a portion of the region SL of the target object OBJ that is surrounded by the contour OL. In other words, the multidimensional model TM_M may be a model that represents at least a portion of the target object OBJ using both the contour OL of the target object OBJ and the region SL of the target object OBJ that is surrounded by the contour OL. As shown in FIG. 13(b), the multidimensional model TM_M may include a two-dimensional model TM_2D that is a two-dimensional model that represents the two-dimensional shape of at least a portion of the target object OBJ. As shown in FIG. 13(c), the multidimensional model TM_M may include a three-dimensional model TM_3D that is a three-dimensional model that represents the three-dimensional shape of at least a portion of the target object OBJ. A CAD (Computer Aided Design) model TM_CAD may be used as the multidimensional model. (2-2-3-2) Matching Process Based on Template Model TM

[0184] The control device 3 (particularly, the position and orientation calculation unit 311 included in the calculation device 31) may calculate at least one of the position and orientation of the target object OBJ by performing a matching process based on the template model TM. In particular, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a matching process based on the template model TM and image data IMG generated by the imaging system 2 capturing an image of the target object OBJ.

[0185] As a first example, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a contour matching process, which is an example of a matching process. Specifically, to perform the contour matching process, the position and orientation calculation unit 311 may use image data IMG generated by the imaging device 21 including a monocular camera (i.e., image data IMG including one image generated by the monocular camera, which may hereinafter be referred to as 2D image data IMG_2D as necessary). In this case, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a contour matching process using the 2D image data IMG_2D (i.e., an image indicated by the 2D image data IMG_2D) and a contour model TM_edge, which is an example of a template model TM. In this case, the position and orientation calculation unit 311 may perform an object detection process as a contour matching process, in which the target object OBJ (in other words, the contour of the target object OBJ) is detected by detecting a contour (edge) corresponding to the contour model TM_edge within the image indicated by the 2D image data IMG_2D.

[0186] In this case, the position and orientation calculation unit 311 may translate, enlarge, reduce, and / or rotate the contour model TM_edge in an imaging coordinate system based on the imaging device 21 so that the contour model TM_edge approaches (e.g., coincides with) the contour of the target object OBJ appearing in the image represented by the 2D image data IMG_2D. As a result, the position and orientation calculation unit 311 can identify the positional relationship between the coordinate system of the contour model TM_edge and the imaging coordinate system. Thereafter, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ in the imaging coordinate system from at least one of the position and orientation of the target object OBJ in the coordinate system of the contour model TM_edge, based on the positional relationship between the coordinate system of the contour model TM_edge and the imaging coordinate system. Thereafter, the position and orientation calculation unit 311 may convert at least one of the position and orientation of the target object OBJ in the imaging coordinate system into at least one of the position and orientation of the target object OBJ in the global coordinate system (or the robot coordinate system) based on a transformation matrix for converting three-dimensional coordinates in the imaging coordinate system into three-dimensional coordinates in the global coordinate system (or the robot coordinate system).

[0187] As a second example, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a 2D matching process, which is an example of a matching process. Specifically, the position and orientation calculation unit 311 may use the 2D image data IMG_2D to perform the 2D matching process. In this case, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a 2D matching process using the 2D image data IMG_2D (i.e., an image indicated by the 2D image data IMG_2D) and a two-dimensional model TM_2D. In this case, the position and orientation calculation unit 311 may perform, as the 2D matching process, an object detection process that detects the target object OBJ indicated by the two-dimensional model TM_2D (in other words, an image portion corresponding to the two-dimensional model TM_2D) within the image indicated by the 2D image data IMG_2D. Note that the 2D matching process (in this case, the object detection process) itself may be the same as an existing 2D matching process. For example, the position and orientation calculation unit 311 may perform 2D matching processing using a well-known method such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed-Up Robust Feature).

[0188] In this case, the position and orientation calculation unit 311 may translate, enlarge, reduce, and / or rotate the two-dimensional model TM_2D in an imaging coordinate system based on the imaging device 21 so that characteristic locations (e.g., at least one of feature points and edges) of the two-dimensional model TM_2D approach (e.g., coincide with) characteristic locations of the target object OBJ captured in the image represented by the 2D image data IMG_2D. As a result, the position and orientation calculation unit 311 can identify the positional relationship between the coordinate system of the two-dimensional model TM_2D and the imaging coordinate system. Thereafter, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ in the imaging coordinate system from at least one of the position and orientation of the target object OBJ in the coordinate system of the two-dimensional model TM_2D based on the positional relationship between the coordinate system of the two-dimensional model TM_2D and the imaging coordinate system. Thereafter, the position and orientation calculation unit 311 may convert at least one of the position and orientation of the target object OBJ in the imaging coordinate system into at least one of the position and orientation of the target object OBJ in the global coordinate system (or the robot coordinate system) based on a transformation matrix for converting three-dimensional coordinates in the imaging coordinate system into three-dimensional coordinates in the global coordinate system (or the robot coordinate system).

[0189] As a third example, if the imaging device 21 includes a stereo camera, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a 3D matching process, which is an example of a matching process. Specifically, to perform the 3D matching process, the position and orientation calculation unit 311 may use image data IMG generated by the imaging device 21 including a stereo camera (i.e., image data IMG including two images generated by the stereo camera, which may hereinafter be referred to as 3D image data IMG_3D as necessary). In this case, the position and orientation calculation unit 311 may generate three-dimensional position data indicating the three-dimensional position of at least a portion of the target object OBJ based on the 3D image data IMG_3D (i.e., the two images indicated by the 3D image data IMG_3D). In this case, the position and orientation calculation unit 311 may calculate parallax based on the 3D image data IMG_3D (i.e., the two image data indicated by the 3D image data IMG_3D) and generate the three-dimensional position data using the calculated parallax using a well-known method based on the principle of triangulation. The three-dimensional position data may be data indicating the three-dimensional position of each of multiple points of the target object OBJ. The three-dimensional position data may be data indicating the three-dimensional position of each of multiple points on the surface of the target object OBJ. The three-dimensional position data may be data indicating the three-dimensional position of each of multiple points corresponding to multiple portions on the surface of the target object OBJ. In the following description, an example will be described in which point cloud data indicating a point cloud is used as the three-dimensional position data. However, three-dimensional position data other than point cloud data indicating a point cloud (e.g., depth image data indicating a depth image) may also be used. Thereafter, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing a 3D matching process using the three-dimensional position data (e.g., point cloud data) and the three-dimensional model TM_3D. In this case, the position and orientation calculation unit 311 may perform, as the 3D matching process, an object detection process to detect the target object OBJ indicated by the three-dimensional model TM_3D (in other words, a set of points corresponding to the three-dimensional model TM_3D) within the point cloud indicated by the three-dimensional position data. The 3D matching process (in this case, the object detection process) itself may be the same as an existing 3D matching process.For example, the position and orientation calculation unit 311 may perform the 3D matching process using a well-known method including at least one of RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), ICP (Iterative Closest Point), and DSO (Direct Sparse Odometry).

[0190] Note that even if the imaging device 21 is an imaging device 21 that includes a monocular camera (in other words, an imaging device 21 that does not include a stereo camera), when the imaging device 21 captures an image of a target object OBJ onto which a projection pattern is projected by the lighting device 23, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing 3D matching processing. This is because the position and orientation calculation unit 311 can calculate a parallax that can be used to generate three-dimensional position data based on the projection pattern (specifically, the projection pattern deformed according to the shape of the target object OBJ) captured in the image indicated by the 2D image data IMG_2D.

[0191] When 3D matching processing is performed, the position and orientation calculation unit 311 may translate, enlarge, reduce, and / or rotate the three-dimensional model TM_3D in the imaging coordinate system so that characteristic locations (e.g., at least one of a feature point and an edge) of the three-dimensional model TM_3D approach (e.g., coincide with) characteristic locations of the target object OBJ whose three-dimensional position data indicates (e.g., a point cloud corresponding to the target object OBJ indicated by the three-dimensional position data). As a result, the position and orientation calculation unit 311 can identify the positional relationship between the coordinate system of the three-dimensional model TM_3D and the imaging coordinate system. Thereafter, even when 3D matching processing is performed, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ in the global coordinate system (or the robot coordinate system) by performing processing similar to that when 2D matching processing is performed.

[0192] The position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by performing at least two of a contour matching process, a 2D matching process, and a 3D matching process. For example, the position and orientation calculation unit 311 may calculate at least a portion of the position and orientation of the target object OBJ by performing at least one of the contour matching process, the 2D matching process, and the 3D matching process, and calculate at least another portion of the position and orientation of the target object OBJ by performing at least another of the contour matching process, the 2D matching process, and the 3D matching process. For example, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by merging at least two of the calculation results of at least one of the position and orientation of the target object OBJ by the contour matching process, the calculation results of at least one of the position and orientation of the target object OBJ by the 2D matching process, and the calculation results of at least one of the position and orientation of the target object OBJ by the 3D matching process.

[0193] Here, in the 2D matching process or the 3D matching process, the target object OBJ is detected taking into consideration the portion SL of the target object OBJ surrounded by the contour OL, while in the contour matching process, the target object OBJ is detected without considering the portion SL of the target object OBJ surrounded by the contour OL. Therefore, the accuracy of at least one of the position and orientation of the target object OBJ calculated by the 2D matching process or the 3D matching process is higher than the accuracy of at least one of the position and orientation of the target object OBJ calculated by the contour matching process. Furthermore, the amount of information in the three-dimensional position data (e.g., point cloud data) used in the 3D matching process is greater than the amount of information in the image data IMG used in the 2D matching process. Therefore, the accuracy of at least one of the position and orientation of the target object OBJ calculated by the 3D matching process is higher than the accuracy of at least one of the position and orientation of the target object OBJ calculated by the 2D matching process. On the other hand, the higher the accuracy of at least one of the position and orientation of the target object OBJ calculated by the matching process, the longer the time required to perform the matching process. For this reason, the following relationship is established among the contour matching process, the 2D matching process, and the 3D matching process: (i) the accuracy of the 3D matching process is higher than the accuracy of the 2D matching process, (ii) the accuracy of the 2D matching process is higher than the accuracy of the contour matching process, (iii) the processing time of the 3D matching process is longer than the processing time of the 2D matching process, and (iv) the processing time of the 2D matching process is longer than the processing time of the contour matching process. For this reason, the control device 3 may select at least one of the contour matching process, the 2D matching process, and the 3D matching process as the matching process for calculating at least one of the position and orientation of the target object OBJ, taking the above-mentioned relationship into consideration. For example, if improving the accuracy of the matching process is to be given top priority, the control device 3 may select (in other words, determine) the 3D matching process as the matching process for calculating at least one of the position and orientation of the target object OBJ.For example, when it is desired to give top priority to shortening the processing time of the matching process, the control device 3 may select (in other words, determine) the contour matching process as the matching process for calculating at least one of the position and orientation of the target object OBJ. For example, when it is desired to improve the accuracy of the matching process while also shortening the processing time of the matching process, the control device 3 may select (in other words, determine) the 2D matching process as the matching process for calculating at least one of the position and orientation of the target object OBJ.

[0194] By performing the matching process, the position and orientation calculation unit 311 may calculate, as the position of the target object OBJ in the global coordinate system (or the robot coordinate system), at least one of a position Tx of the target object OBJ in an X-axis direction parallel to the X-axis of the global coordinate system (or the robot coordinate system), a position Ty of the target object OBJ in a Y-axis direction parallel to the Y-axis of the global coordinate system (or the robot coordinate system), and a position Tz of the target object OBJ in a Z-axis direction parallel to the Z-axis of the global coordinate system (or the robot coordinate system). By performing the matching process, the position and orientation calculation unit 311 may calculate, as the orientation of the target object OBJ in the global coordinate system (or the robot coordinate system), at least one of a rotation amount Rx of the target object OBJ about the X-axis of the global coordinate system, a rotation amount Ry of the target object OBJ about the Y-axis of the global coordinate system (or the robot coordinate system), and a rotation amount Rz of the target object OBJ about the Z-axis of the global coordinate system (or the robot coordinate system). This is because the amount of rotation Rx of the target object OBJ around the X-axis, the amount of rotation Ry of the target object OBJ around the Y-axis, and the amount of rotation Rz of the target object OBJ around the Z-axis are equivalent to the parameters representing the orientation of the target object OBJ around the X-axis, the parameters representing the orientation of the target object OBJ around the Y-axis, and the parameters representing the orientation of the target object OBJ around the Z-axis, respectively. For this reason, in the following description, the amount of rotation Rx of the target object OBJ around the X-axis, the amount of rotation Ry of the target object OBJ around the Y-axis, and the amount of rotation Rz of the target object OBJ around the Z-axis will be referred to as the orientation Rx of the target object OBJ around the X-axis, the orientation Ry of the target object OBJ around the Y-axis, and the orientation Rz of the target object OBJ around the Z-axis, respectively.

[0195] The orientation Rx of the target object OBJ around the X-axis, the orientation Ry of the target object OBJ around the Y-axis, and the orientation Rz of the target object OBJ around the Z-axis may be considered to indicate the position of the target object OBJ in the rotation direction around the X-axis, the position of the target object OBJ in the rotation direction around the Y-axis, and the position of the target object OBJ in the rotation direction around the Z-axis, respectively. In other words, the orientation Rx of the target object OBJ around the X-axis, the orientation Ry of the target object OBJ around the Y-axis, and the orientation Rz of the target object OBJ around the Z-axis may all be considered to be parameters representing the position of the target object OBJ.

[0196] When performing the matching process, the position and orientation calculation unit 311 may calculate a matching similarity, which is the similarity between the template model TM and the detected target object OBJ. When a contour matching process is performed, the matching similarity may refer to the similarity between the contour model TM_edge and the contour OL of the target object OBJ reflected in the image represented by the image data IMG. When a 2D matching process is performed, the matching similarity may refer to the similarity between the two-dimensional model TM_2D and the target object OBJ reflected in the image represented by the image data IMG. When a 3D matching process is performed, the matching similarity may refer to the similarity between the three-dimensional model TM_3D and the target object OBJ whose three-dimensional position is indicated by three-dimensional position data generated from the image data IMG. When a target object OBJ whose calculated matching similarity exceeds a predetermined matching determination threshold used in the matching process is detected by the matching process, the position and orientation calculation unit 311 may select (in other words, determine) the target object OBJ as a processing execution object on which the end effector 4 should actually perform a predetermined process. On the other hand, when a target object OBJ whose calculated matching similarity is below the matching judgment threshold is detected by the matching process, the position and orientation calculation unit 311 does not have to select the target object OBJ as a processing execution object on which the end effector 4 should actually perform a specified processing.

[0197] For example, when the end effector 4 holds a first target object OBJ#1, if the matching process detects a first target object OBJ#1 whose matching similarity exceeds a matching determination threshold, the position and orientation calculation unit 311 may select the first target object OBJ#1 as the processing execution object to be actually held by the end effector 4. For example, when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2, if the matching process detects a first target object OBJ#1 whose matching similarity exceeds a matching determination threshold, the position and orientation calculation unit 311 may select the first target object OBJ#1 as the processing execution object to be actually released onto the second target object OBJ#2 by the end effector 4. For example, when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2, if the second target object OBJ#2 whose matching similarity exceeds the matching judgment threshold is detected by the matching process, the position and orientation calculation unit 311 may select the second target object OBJ#2 as the processing execution object onto which the end effector 4 should actually release the first target object OBJ#1.

[0198] Here, in some cases, the matching process may not detect an object (e.g., target object OBJ) whose matching similarity exceeds the matching determination threshold. Furthermore, the matching process may not detect any object (e.g., target object OBJ). In this case, the position and orientation calculation unit 311 may determine that the matching process (specifically, the arithmetic process for calculating at least one of the position and orientation of the target object OBJ, including the matching process) is defective.

[0199] Note that even when an object whose matching similarity is below the matching determination threshold is detected as the target object OBJ, the position and orientation calculation unit 311 can calculate at least one of the position and orientation of the target object OBJ. However, the accuracy of the calculation result of at least one of the position and orientation of the target object OBJ whose matching similarity is below the matching determination threshold is lower than the accuracy of the calculation result of at least one of the position and orientation of the target object OBJ whose matching similarity is above the matching determination threshold. For this reason, determining that the matching process is poor may be considered equivalent to determining that the accuracy of the calculation result of at least one of the position and orientation of the target object OBJ is low.

[0200] Furthermore, as will be described in detail later, the signal generation unit 312 included in the control device 3 generates a robot control signal to cause the end effector 4 to perform a predetermined process based on the results of the matching process (i.e., the calculation results of at least one of the position and orientation of the target object OBJ). However, if the matching process is unsuccessful, the signal generation unit 312 cannot generate a robot control signal. Therefore, if the matching process is unsuccessful, the end effector 4 cannot perform a predetermined process. Alternatively, even if the matching process is unsuccessful, at least one of the position and orientation of the target object OBJ is calculated, albeit with low accuracy, and therefore the signal generation unit 312 can generate a robot control signal. However, even if at least one of the robot 1, the end effector 4, and the robot movable device is controlled based on such a robot control signal, the end effector 4 may not be able to perform a predetermined process. Therefore, determining that the matching process is poor may be considered equivalent to at least one of determining that a robot control signal cannot be generated, determining that the end effector 4 cannot perform the specified processing because a robot control signal cannot be generated, determining that the end effector 4 cannot perform the specified processing even if a robot control signal is generated, and determining that the end effector 4 will not perform the specified processing using the robot control signal even if a robot control signal is generated.

[0201] The matching determination threshold is a threshold used for detecting the target object OBJ from the image represented by the image data IMG by the matching process. Specifically, the matching determination threshold is a threshold used for distinguishing, from the matching similarity of the object detected by the matching process, a state in which the object detected by the matching process is highly likely to be the target object OBJ and a state in which the object detected by the matching process is low likely to be the target object OBJ. The matching determination threshold can also be said to be a threshold used for distinguishing, from the matching similarity, a state in which the accuracy of at least one of the position and orientation of the target object OBJ calculated by the matching process is high and a state in which the accuracy of at least one of the position and orientation of the target object OBJ calculated by the matching process is low. (2-2-3-3) Process for Generating a Template Model TM

[0202] Next, a process for generating a template model TM will be described with reference to Fig. 14. Fig. 14 is a flowchart showing the flow of the process for generating a template model TM.

[0203] 14, the imaging system 2 captures an image of the target object OBJ (step S11). As a result, the imaging system 2 generates image data IMG representing an image in which the target object OBJ is captured (step S11). The arithmetic unit 31 acquires the image data IMG generated by the imaging system 2 from the imaging system 2 (step S11).

[0204] In step S11, the imaging system 2 may capture an image of an object that is different from the target object OBJ but has the same shape as the target object OBJ as the target object OBJ. The object having the same shape as the target object OBJ may include an object that satisfies a first condition that the shape of the target object OBJ and the shape of the object are literally identical. The object having the same shape as the target object OBJ may include an object that does not have the same shape as the target object OBJ but satisfies a second condition that the difference between the shape of the target object OBJ and the shape of the object is small enough that the shape of the target object OBJ and the shape of the object can be considered to be substantially the same. An object that satisfies at least one of the first and second conditions may be considered to have approximately the same shape as the target object OBJ. For example, "a difference between the shape of the target object OBJ and the shape of the object that is small enough that the shape of the target object OBJ and the shape of the object can be considered to be substantially the same" may mean a difference that is equal to or smaller than a manufacturing error that occurs when manufacturing the target object OBJ.

[0205] Furthermore, in step S11, an imaging system other than the imaging system 2 may capture an image of the target object OBJ or an object having the same shape as the target object OBJ. In this case, the calculation device 31 may acquire, from the other imaging system, image data generated by the other imaging system as image data IMG.

[0206] Thereafter, the arithmetic device 31 acquires a user instruction for generating a template model TM (step S12). In this case, the user may use the input device 34 to input the instruction for generating the template model TM to the control device 3. Thereafter, the arithmetic device 31 generates the template model TM based on the user instruction (step S13).

[0207] As a first example, when generating a contour model TM_edge as a template model TM, the calculation device 31 may acquire a user instruction specifying at least a portion of the contour OL of the target object OBJ extracted within the image indicated by the image data IMG as a user instruction for generating the template model TM.

[0208] In this case, the calculation device 31 may first extract edges as a contour OL of the target object OBJ in the image represented by the image data IMG. For example, the contour OL of the target object OBJ may be extracted using a filter for extracting edges in the image. Examples of the filter for extracting edges include at least one of a Gaussian filter, a Sobel filter, and a Laplacian filter.

[0209] In addition to using the filter, the calculation device 31 may detect the target object OBJ by performing a matching process based on the generated multidimensional model TM_M and the image data IMG, and then extract the contour OL of the target object OBJ based on the detection result of the target object OBJ obtained by the matching process and the edge extraction result obtained by the filter. Specifically, from the edges extracted using the filter, edges that match the contour OL of the target object OBJ detected by the matching process may be further extracted, and the extracted edges may be extracted as the contour OL of the target object OBJ. In this case, the user's effort in generating the contour model TM_edge is reduced compared to when the contour OL of the target object OBJ is extracted without using the detection result of the target object OBJ obtained by the matching process. This is because, first, the edges extracted using the filter may include edges that are not the contour OL of the target object OBJ in addition to the edges that are the contour OL of the target object OBJ. Therefore, in order to generate a contour model TM_edge that appropriately represents the target object OBJ from edges extracted using a filter, the user needs to specify edges that are not the contour OL of the target object OBJ as contour portions OL_ex that should not be included in the contour model TM_edge, as described below. On the other hand, because the target object OBJ is detected by the matching process, the detection result of the target object OBJ by the matching process includes information about the edges that are the contour OL of the target object OBJ. Conversely, the detection result of the target object OBJ by the matching process does not include information about edges that are different from the contour OL of the target object OBJ. Therefore, further extracting edges that match the contour OL of the target object OBJ detected by the matching process from the edge extraction result using a filter is essentially equivalent to removing edges that are not the contour OL of the target object OBJ from the edge extraction result using a filter. Therefore, in order to generate a contour model TM_edge that properly represents the target object OBJ from the contour OL extracted based on the results of the matching process, the user does not necessarily have to specify an edge that is not the contour OL of the target object OBJ as a contour portion OL_ex that should not be included in the contour model TM_edge.This reduces the user's effort in generating the contour model TM_edge. However, in order to generate a contour model TM_edge that appropriately represents the target object OBJ from the contour OL extracted based on the results of the matching process, the user may specify a portion of the contour OL of the target object OBJ as a contour portion OL_ex or OL_in, which will be described later. The multidimensional model TM_M used to accurately extract the contour OL of the target object OBJ may represent a portion of the target object OBJ. In this case, the multidimensional model TM_M used to accurately extract the contour OL of the target object OBJ may represent a portion of the target object OBJ that includes the contour OL to be extracted as part of the target object OBJ. Even in this case, the detection result of the target object OBJ by the matching process includes information about the contour OL of the target object OBJ (particularly the extracted contour OL).

[0210] Furthermore, in order to accurately extract the contour OL of the target object OBJ, the calculation device 31 may perform a matching process based on a generated contour model TM_edge that represents all or most of the contour of the target object OBJ, in addition to or instead of a matching process based on the generated multidimensional model TM_M. The contour model TM_edge used to accurately extract the contour OL of the target object OBJ (i.e., the contour model TM_edge used in the presetting process) may be different from the contour model TM_edge used in the matching process performed to actually control the robot 1 after the presetting process. Even in this case, since the target object OBJ is detected by the matching process, the detection result of the target object OBJ by the matching process includes information about all or most of the contour OL of the target object OBJ. Therefore, even in this case, the user's effort in generating the contour model TM_edge is reduced, similar to when a matching process is performed based on the multidimensional model TM_M. Furthermore, since the processing time for the contour matching process based on the contour model TM_edge is shorter than the processing time for the 3D matching process and the 2D matching process based on the multidimensional model TM_M as described above, when the contour OL is extracted by performing a matching process on the generated multidimensional model TM_M, the time required to extract the contour model OL is reduced when the contour OL is extracted by performing a matching process on the generated contour model TM_edge and the target object OBJ. However, the contour OL indicated by the contour model TM_edge that is newly generated using the results of the matching process based on the generated contour model TM_edge that represents all or most of the contour of the target object OBJ will be smaller than the contour OL indicated by the generated contour model TM_edge.

[0211] 15( a), the arithmetic device 31 may control the display device 37 to display a model generation screen 63 showing the outline OL of the detected target object OBJ. Specifically, the arithmetic device 31 may generate a display signal for displaying the model generation screen 63 and output the generated display signal to the display device 37. The display device 37 may display the model generation screen 63 based on the display signal generated by the arithmetic device 31.

[0212] The model generation screen 63 can function as an input screen for inputting user instructions. In this case, the user may input user instructions for generating a template model TM via the model generation screen 63.

[0213] For example, the user may input an instruction to the control device 3 on the model generation screen 63 to specify a portion of the contour OL displayed on the display device 37 as a user instruction for generating a template model TM. As an example, as shown in FIG. 15B , the user may input an instruction to the control device 3 to specify a contour portion OL_in of the contour OL displayed on the display device 37 that should be included in the contour model TM_edge. The instruction to specify the contour portion OL_in may include an instruction to specify the contour portion OL_in (the portion indicated by the dashed dotted line in FIG. 15B ) or an instruction to specify an area that includes the contour portion OL_in. In this case, the calculation device 31 may generate a contour model TM_edge that includes the contour portion OL_in specified by the user. As another example, as shown in FIG. 15B , the user may input an instruction to the control device 3 to specify a contour portion OL_ex (the portion indicated by the dotted line in FIG. 15B ) of the contour OL displayed on the display device 37 that should not be included in the contour model TM_edge. The instruction to specify the contour portion OL_ex may include an instruction to specify the contour portion OL_ex, or may include an instruction to specify an area including the contour portion OL_ex, in which case the calculation device 31 may generate a contour model TM_edge that does not include the contour portion OL_ex specified by the user.

[0214] Note that the model generation screen 63 may display an image indicated by the image data IMG in addition to or instead of the contour OL of the target object OBJ extracted by the arithmetic device 31. In this case, the user may input an instruction to specify at least a portion of the contour OL of the target object OBJ on the image displayed on the model generation screen 63. For example, the user may input an instruction to trace at least a portion of the contour OL of the target object OBJ on the image displayed on the model generation screen 63. For example, the user may input an instruction to specify an area including at least a portion of the contour OL of the target object OBJ on the image displayed on the model generation screen 63. In this case, the arithmetic device 31 may generate the contour model TM_edge based on a position (area) specified by the user on the image displayed on the model generation screen 63.

[0215] As an example, the arithmetic device 31 may extract the contour OL of the target object OBJ by extracting edges based on a position (area) specified by the user on the image displayed on the model generation screen 63. For example, the arithmetic device 31 may extract edges as the contour OL of the target object OBJ at a position (area) specified by the user, while not extracting edges as the contour OL of the target object OBJ at a position (area) not specified by the user. As a result, the arithmetic device 31 can accurately extract the contour OL of the target object OBJ. Thereafter, the arithmetic device 31 may generate a contour model TM_edge including the contour OL of the target object OBJ extracted by the arithmetic device 31 based on a user instruction. Alternatively, the arithmetic device 31 may display the model generation screen 63 displaying the contour OL of the target object OBJ extracted by the arithmetic device 31 based on a user instruction, and further acquire a user instruction. However, the detected contour OL of the target object OBJ may be used as the contour model TM_edge as is. In this case, the user may input a user instruction for generating the template model TM.

[0216] As a second example, when generating a two-dimensional model TM_2D as a template model TM, the calculation device 31 may acquire a user instruction specifying a portion of the image indicated by the image data IMG as a user instruction for generating the template model TM. In this case, the calculation device 31 may control the display device 37 to display, instead of the model generation screen 63, a model generation screen on which the image indicated by the image data IMG is displayed and on which the user's instruction is input. The user may then input, on the model generation screen, an instruction to the control device 3 specifying a portion of the image displayed on the display device 37 that should be included in the two-dimensional model TM_2D. As an example, the user may input, to the control device 3, an instruction to specify a portion of the image displayed on the display device 37 that should not be included in the two-dimensional model TM_2D. In this case, the calculation device 31 may generate a two-dimensional model TM_2D that includes the image portion specified by the user. As another example, the user may input, to the control device 3, an instruction to specify a portion of the image displayed on the display device 37 that should not be included in the two-dimensional model TM_2D. In this case, the calculation device 31 may generate a two-dimensional model TM_2D that does not include the image portion specified by the user.

[0217] As a third example, when generating a three-dimensional model TM_3D as a template model TM, the calculation device 31 may acquire, as a user instruction for generating the template model TM, a user instruction specifying a portion of a three-dimensional model indicated by three-dimensional position data generated from image data IMG (e.g., a point cloud model indicated by point cloud data generated from image data IMG). In this case, the calculation device 31 may first generate three-dimensional position data based on the image data IMG. Then, the calculation device 31 may control the display device 37 to display, instead of the model generation screen 63, a model generation screen that displays a three-dimensional model indicated by the generated three-dimensional position data and into which user instructions are input. Then, the user may input, on the model generation screen, an instruction to the control device 3 specifying a portion of the three-dimensional model displayed on the display device 37. As an example, the user may input, to the control device 3, an instruction to specify a model portion of the three-dimensional model displayed on the display device 37 that should be included in the three-dimensional model TM_3D. In this case, the calculation device 31 may generate a three-dimensional model TM_3D that includes the model portion specified by the user. As another example, the user may input an instruction to the control device 3 to specify a model portion of the three-dimensional model displayed on the display device 37 that should not be included in the three-dimensional model TM_3D. In this case, the calculation device 31 may generate a three-dimensional model TM_3D that does not include the model portion specified by the user.

[0218] Here, the accuracy of the matching process based on a template model TM indicating a portion (partial shape) of the target object OBJ is likely to be lower than the accuracy of the matching process based on a template model TM indicating the entire target object OBJ (entire shape). Furthermore, the accuracy of the matching process based on a template model TM indicating a first portion (shape of the first portion) of the target object OBJ is likely to be lower than the accuracy of the matching process based on a template model TM indicating a second portion (shape of the second portion) of the target object OBJ, which is wider than the first portion. Therefore, the wider the range of the target object OBJ indicated by the template model TM, the higher the accuracy of the matching process based on the template model TM. In other words, the greater the proportion of the range of the target object OBJ indicated by the template model TM to the entire target object OBJ, the higher the accuracy of the matching process based on the template model TM. In other words, the greater the proportion of the range of the target object OBJ indicated by the template model TM to the entire target object OBJ, the higher the accuracy of the matching process based on the template model TM. In other words, the greater the proportion of the shape of the target object OBJ indicated by the template model TM relative to the overall shape of the target object OBJ, the higher the accuracy of the matching process.

[0219] On the other hand, the processing time of the matching process based on a template model TM indicating a portion (partial shape) of the target object OBJ is likely to be shorter than the processing time of the matching process based on a template model TM indicating the entire target object OBJ (entire shape). Furthermore, the processing time of the matching process based on a template model TM indicating a first portion (shape of the first portion) of the target object OBJ is likely to be shorter than the processing time of the matching process based on a template model TM indicating a second portion (shape of the second portion) of the target object OBJ, which is wider than the first portion. Therefore, the narrower the range of the target object OBJ indicated by the template model TM, the shorter the processing time of the matching process based on the template model TM. In other words, the smaller the proportion of the range of the target object OBJ indicated by the template model TM to the entire target object OBJ, the shorter the processing time of the matching process based on the template model TM. In other words, the smaller the proportion of the range of the target object OBJ indicated by the template model TM to the entire target object OBJ, the shorter the processing time of the matching process based on the template model TM. In other words, the smaller the proportion of the shape indicated by the template model TM relative to the overall shape of the target object OBJ, the shorter the processing time for the matching process.

[0220] The user may input an instruction to generate a template model TM while taking into consideration the relationship between at least one of the accuracy and processing time of the matching process and the width of the range of the target object OBJ indicated by the template model TM. For example, the user may input an instruction to generate a template model TM such that the range of the target object OBJ indicated by the template model TM used when prioritizing improvement of the accuracy of the matching process over reduction of the processing time of the matching process is wider than the range of the target object OBJ indicated by the template model TM used when prioritizing reduction of the processing time of the matching process over improvement of the accuracy of the matching process. For example, the user may input an instruction to generate a template model TM such that the range of the target object OBJ indicated by the template model TM used when prioritizing reduction of the processing time of the matching process over improvement of the accuracy of the matching process is narrower than the range of the target object OBJ indicated by the template model TM used when prioritizing improvement of the accuracy of the matching process over reduction of the processing time of the matching process.

[0221] As described above, when the end effector 4 holds the first target object OBJ#1, the control device 3 calculates at least one of the position and orientation of the first target object OBJ#1 before it is held by the end effector 4 by performing a matching process based on a template model TM that indicates the shape of the first target object OBJ#1. In this case, the user may input an instruction to generate a first template model TM that can be used in the matching process for calculating at least one of the position and orientation of the first target object OBJ#1 before it is held by the end effector 4 when the end effector 4 holds the first target object OBJ#1. The control device 3 may generate a first template model TM that can be used in the matching process for calculating at least one of the position and orientation of the first target object OBJ#1 before it is held by the end effector 4 when the end effector 4 holds the first target object OBJ#1. The control device 3 may generate an entire model that indicates the entire first target object OBJ#1 (overall shape) as the first template model TM. In this case, the control device 3 may calculate at least one of the position and orientation of the first target object OBJ#1 before it is held by the end effector 4 by performing a matching process based on the first template model TM, which is an entire model. The control device 3 may generate a partial model representing a part (partial shape) of the first target object OBJ#1 as the first template model TM in addition to or instead of the first template model TM representing the entire first target object OBJ#1 (the entire shape). In this case, the control device 3 may calculate at least one of the position and orientation of the first target object OBJ#1 before it is held by the end effector 4 by performing a matching process based on the first template model TM, which is a partial model. When a matching process based on a partial model is performed in this manner, even if at least one of the position and orientation of the first target object OBJ#1 before it is held by the end effector 4 varies (in other words, is not consistent), the control device 3 is more likely to be able to calculate at least one of the position and orientation of the first target object OBJ#1 before it is held by the end effector 4 by using an appropriate partial model.

[0222] As described above, when the end effector 4 releases the first target object OBJ#1 held by the end effector 4 onto the second target object OBJ#2, the control device 3 calculates at least one of the position and orientation of the second target object OBJ#2 by performing a matching process based on a template model TM that indicates the shape of the second target object OBJ#2. In this case, the user may input an instruction to generate a second template model TM that can be used in the matching process for calculating at least one of the position and orientation of the second target object OBJ#2 when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. The control device 3 may generate a second template model TM that can be used in the matching process for calculating at least one of the position and orientation of the second target object OBJ#2 when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. The control device 3 may generate an entire model that indicates the entire second target object OBJ#2 (overall shape) as the second template model TM. In this case, the control device 3 may calculate at least one of the position and orientation of the second target object OBJ#2 by performing a matching process based on the second template model TM, which is an entire model. The control device 3 may generate a partial model representing a portion (partial shape) of the second target object OBJ#2 as the second template model TM in addition to or instead of the second template model TM representing the entire second target object OBJ#2 (the entire shape). In this case, the control device 3 may calculate at least one of the position and orientation of the second target object OBJ#2 by performing a matching process based on the second template model TM, which is a partial model. When a matching process based on a partial model is performed in this manner, even if at least one of the position and orientation of the second target object OBJ#2 before being held by the end effector 4 varies (in other words, is not consistent), the control device 3 is more likely to be able to calculate at least one of the position and orientation of the second target object OBJ#2 before being held by the end effector 4 by using an appropriate partial model.

[0223] As described above, when the first target object OBJ#1 held by the end effector 4 is released onto the second target object OBJ#2, the control device 3 calculates at least one of the position and orientation of the first target object OBJ#1 held by the end effector 4 by performing a matching process based on the template model TM that indicates the shape of the first target object OBJ#1. In this case, the user may input an instruction to generate a third template model TM that can be used in the matching process for calculating at least one of the position and orientation of the first target object OBJ#1 held by the end effector 4 when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. The control device 3 may generate the third template model TM that can be used in the matching process for calculating at least one of the position and orientation of the first target object OBJ#1 held by the end effector 4 when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2. The control device 3 may generate an overall model representing the entire first target object OBJ#1 (overall shape) as the third template model TM. In this case, the control device 3 may calculate at least one of the position and the orientation of the first target object OBJ#1 held by the end effector 4 by performing a matching process based on the third template model TM, which is the overall model. The control device 3 may generate a partial model representing a part (partial shape) of the first target object OBJ#1 as the third template model TM in addition to or instead of the first template model TM representing the entire first target object OBJ#1 (overall shape). In this case, the control device 3 may calculate at least one of the position and the orientation of the first target object OBJ#1 held by the end effector 4 by performing a matching process based on the third template model TM, which is the partial model.When a matching process based on a partial model is performed in this manner, even if at least one of the position and orientation of the first target object OBJ#1 held by the end effector 4 varies (in other words, is not uniform), the control device 3 is more likely to be able to calculate at least one of the position and orientation of the first target object OBJ#1 held by the end effector 4 by using an appropriate partial model.

[0224] Here, when the end effector 4 holds the first target object OBJ#1, the control device 3 calculates at least one of the position and orientation of the first target object OBJ#1 that is not held by the end effector 4 by performing a first matching process based on the image data IMG of the first target object OBJ#1 that is not held by the end effector 4 and the first template model TM. On the other hand, when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2, the control device 3 calculates at least one of the position and orientation of the first target object OBJ#1 that is held by the end effector 4 by performing a second matching process based on the image data IMG of the first target object OBJ#1 that is held by the end effector 4 and the third template model TM. Therefore, there is a difference between the first matching process and the second matching process in that when the first matching process is performed, the end effector 4 is not holding the first target object OBJ#1, whereas when the second matching process is performed, the end effector 4 is holding the first target object OBJ#1.

[0225] For this reason, the user may input an instruction to generate a template model TM different from the first template model TM used in the first matching process as the third template model TM used in the second matching process. The user may input an instruction to generate a template model TM different from the third template model TM used in the second matching process as the first template model TM used in the first matching process. The control device 3 may generate a template model TM different from the first template model TM used in the first matching process as the third template model TM used in the second matching process. The control device 3 may generate a template model TM different from the third template model TM used in the second matching process as the first template model TM used in the first matching process.

[0226] In this case, the control device 3 may perform the first matching process using the first template model TM used in the first matching process. The control device 3 may perform the second matching process using the third template model TM used in the second matching process. However, the control device 3 may perform the first matching process using the third template model TM used in the second matching process. The control device 3 may perform the second matching process using the first template model TM used in the first matching process.

[0227] As an example, in order to generate a third template model TM for calculating at least one of the position and orientation of the first target object OBJ#1 held by the end effector 4, in step S11 of FIG. 14 , the imaging system 2 (or another imaging system different from the imaging system 2; the same applies hereinafter in this paragraph) may capture an image of the first target object OBJ#1 (or another object having the same shape as the first target object OBJ#1; the same applies hereinafter in this paragraph) held by the end effector 4. Here, FIG. 16 shows an example of an image represented by image data IMG generated by the imaging system 2 provided on the robot 1 capturing an image of the first target object OBJ#1 held by the end effector 4. As shown in FIG. 16 , when the imaging system 2 provided on the robot 1 captures an image of the first target object OBJ#1 held by the end effector 4, there is a possibility that a part of the first target object OBJ#1 will be hidden by the end effector 4 in the image represented by the image data IMG. Note that in FIG. 16 , a portion of the first target object OBJ#1 that is hidden by the end effector 4 is indicated by a dotted line. In other words, a portion of the first target object OBJ#1 may be blocked by the end effector 4. As a result, the imaging system 2 may not be able to capture an image of a hidden portion HP1, which is a portion of the first target object OBJ#1 that is hidden (or blocked) by the end effector 4. As a result, as shown in FIG. 16 , the hidden portion HP1 of the first target object OBJ#1 may not appear in the image represented by the image data IMG. In this case, the imaging system 2 will capture an image of a non-hidden portion HP2 of the first target object OBJ#1 that is different from the hidden portion HP1. The non-hidden portion HP2 may include at least a portion of the first target object OBJ#1 that is not blocked by the end effector 4. The non-hidden portion HP2 may include at least a portion of the first target object OBJ#1 that is not occluded by the end effector 4.Incidentally, even when an imaging system 2 not provided on the robot 1 captures an image of the first target object OBJ#1 held by the end effector 4, there is a possibility that the non-hidden portion HP2 will appear in the image represented by the image data IMG while the hidden portion HP1 will not appear. For example, even when an imaging system 2 provided on the above-described robot movable device captures an image of the first target object OBJ#1 held by the end effector 4, there is a possibility that the non-hidden portion HP2 will appear in the image represented by the image data IMG while the hidden portion HP1 will not appear. For example, even when an imaging system 2 provided on a structure such as a pillar captures an image of the first target object OBJ#1 held by the end effector 4, there is a possibility that the non-hidden portion HP2 will appear in the image represented by the image data IMG while the hidden portion HP1 will not appear. In this case, the user may input an instruction to generate, as the third template model TM, a template model TM that does not include at least a part of a model portion corresponding to the hidden portion HP1 of the first target object OBJ#1 that is hidden by the end effector 4 (specifically, in the first target object OBJ#1 that cannot be imaged by the imaging system 2 because it is blocked by the end effector 4), based on image data IMG that is the imaging result of the first target object OBJ#1 held by the end effector 4. The user may input an instruction to generate, as the third template model TM, a template model TM that includes at least a part of a model portion corresponding to the non-hidden portion HP2 of the first target object OBJ#1 that is not hidden by the end effector 4 (specifically, in the first target object OBJ#1 that can be imaged by the imaging system 2 without being blocked by the end effector 4), based on the image data IMG.The calculation device 31 may generate, as the third template model TM, a template model TM that does not include at least a part of a model portion corresponding to a hidden portion HP1 of the first target object OBJ#1 that is hidden by the end effector 4 (specifically, a part of the first target object OBJ#1 that cannot be imaged by the imaging system 2 because it is blocked by the end effector 4), based on image data IMG that is the imaging result of the first target object OBJ#1 held by the end effector 4. The calculation device 31 may generate, as the third template model TM, a template model TM that does not include at least a part of a model portion corresponding to a non-hidden portion HP2 of the first target object OBJ#1 that is not hidden by the end effector 4 (specifically, a part of the first target object OBJ#1 that can be imaged by the imaging system 2 without being blocked by the end effector 4), based on the image data IMG. In this case, under conditions in which the hidden portion HP1 of the first target object OBJ#1 is hidden by the end effector 4 holding the first target object OBJ#1 (specifically, in the imaging field of view of the imaging system 2, a part of the first target object OBJ#1 is blocked by the end effector 4), a matching process is performed based on a template model TM that does not include at least a part of a model portion corresponding to the hidden portion HP1 of the first target object OBJ#1 that is hidden by the end effector 4. Also, under conditions in which the non-hidden portion HP2 of the first target object OBJ#1 is not hidden by the end effector 4 holding the first target object OBJ#1 (specifically, in the imaging field of view of the imaging system 2, a part of the first target object OBJ#1 is not blocked by the end effector 4), a matching process is performed based on a template model TM that includes at least a part of a model portion corresponding to the non-hidden portion HP2 of the first target object OBJ#1 that is not hidden by the end effector 4.Therefore, compared to when the matching process is performed based on a template model TM that includes a model portion corresponding to the hidden portion HP1 of the first target object OBJ#1 that is hidden by the end effector 4 and / or when the matching process is performed based on a template model TM that does not include a model portion corresponding to the non-hidden portion HP2 of the first target object OBJ#1 that is not hidden by the end effector 4, the likelihood of a higher matching similarity being calculated in the matching process (the likelihood of a more accurate execution of the matching process) is increased. This reduces the likelihood of a situation in which the first target object OBJ#1 held by the end effector 4 cannot be detected by the matching process. As a result, this reduces the likelihood of a situation in which at least one of the position and orientation of the first target object OBJ#1 held by the end effector 4 cannot be calculated by the matching process. Furthermore, due to the increased likelihood of a higher matching similarity being calculated in the matching process, the arithmetic unit 31 can accurately calculate at least one of the position and orientation of the first target object OBJ#1 held by the end effector 4.

[0228] In step S11, the imaging system 2 (or another imaging system different from the imaging system 2; hereinafter the same in this paragraph) may image the first target object OBJ#1 held by an end effector different from the end effector 4, in addition to or instead of the first target object OBJ#1 (or another object having the same shape as the first target object OBJ#1; hereinafter the same in this paragraph) held by the end effector 4. In this case, the other end effector may hold the first target object OBJ#1 such that a hidden portion HP1 of the first target object OBJ#1 that is hidden by the end effector 4 when the end effector 4 holds the first target object OBJ#1 is hidden by the other end effector. As a result, similar to the case where the first target object OBJ#1 held by the end effector 4 is imaged, the calculation device 31 may generate a template model TM that does not include at least a part of a model portion corresponding to the hidden portion HP1 of the first target object OBJ#1 that is hidden by the other end effector. The computing device 31 may generate a template model TM that includes at least a part of a model portion that corresponds to the non-hidden portion HP2 of the first target object OBJ#1 that is not hidden by another end effector. As a result, it is possible to obtain the same effect as when the template model TM is generated from image data IMG that is the imaging result of the first target object OBJ#1 held by the end effector 4.

[0229] 14 , in order to generate a first template model TM for calculating at least one of the position and orientation of the first target object OBJ#1 that is not held by the end effector 4, the imaging system 2 (or another imaging system different from the imaging system 2; the same applies hereinafter in this paragraph) may capture an image of the first target object OBJ#1 (or another object having the same shape as the first target object OBJ#1; the same applies hereinafter in this paragraph) that is not held by the end effector 4 (or another end effector different from the end effector 4; the same applies hereinafter in this paragraph). In this case, the user may input an instruction to generate, as the first template model TM, a template model TM that indicates the shape of the first target object OBJ#1 that is not hidden by the end effector 4, based on image data IMG that is the imaging result of the first target object OBJ#1 that is not held by the end effector 4. The computing device 31 may generate, as the first template model TM, a template model TM that indicates the shape of the first target object OBJ#1 that is not hidden by the end effector 4, based on image data IMG that is the imaging result of the first target object OBJ#1 that is not held by the end effector 4. In this case, under a situation where a portion of the first target object OBJ#1 is not hidden by the end effector 4 (specifically, a portion of the first target object OBJ#1 is not blocked by the end effector 4 in the imaging field of view of the imaging system 2), the matching process is performed based on the template model TM that indicates the shape of the first target object OBJ#1 that is not hidden by the end effector 4. Therefore, compared to a case where the matching process is performed based on a template model TM that does not include a model portion corresponding to the portion of the first target object OBJ#1 that is hidden by the end effector 4, the matching similarity calculated in the matching process is more likely to be higher (the matching process is more likely to be performed accurately). This reduces the possibility of a situation occurring in which the first target object OBJ#1, which the end effector 4 has not yet held but is about to hold, cannot be detected by the matching process.As a result, there is a reduced possibility that the matching process will fail to calculate at least one of the position and orientation of the first target object OBJ#1, which the end effector 4 has not yet held but is about to hold. Furthermore, due to the increased possibility that the matching similarity calculated in the matching process will be high, the arithmetic unit 31 can calculate with high accuracy at least one of the position and orientation of the first target object OBJ#1, which the end effector 4 has not yet held but is about to hold.

[0230] When the end effector 4 is not holding the first target object OBJ#1, there is little possibility that a portion of the first target object OBJ#1 will be hidden (occluded) by the end effector 4. Therefore, when generating a first template model TM for calculating at least one of the position and orientation of the first target object OBJ#1 that is not held by the end effector 4, the calculation device 31 may generate the first template model TM without considering at least one of the hidden portion HP1 and the non-hidden portion HP2, unlike the case of generating the third template model TM for calculating at least one of the position and orientation of the first target object OBJ#1 that is held by the end effector 4 described above. However, in a situation where an end effector 4 that is not holding the first target object OBJ#1 is approaching the first target object OBJ#1 in order to hold it, there is a possibility that a portion of the first target object OBJ#1 will be hidden (occluded) by the end effector 4 that has approached the first target object OBJ#1. That is, even when the first target object OBJ#1 is not held by the end effector 4, there is a possibility that a part of the first target object OBJ#1 becomes the hidden portion HP1 that is hidden by the end effector 4, and another part of the first target object OBJ#1 becomes the non-hidden portion HP2 that is not hidden by the end effector 4. In this case, when generating a first template model TM for calculating at least one of the position and orientation of the first target object OBJ#1 that is not held by the end effector 4, the calculation device 31 may generate the first template model TM based on at least one of the hidden portion HP1 and the non-hidden portion HP2, as in the case of generating the third template model TM for calculating at least one of the position and orientation of the first target object OBJ#1 that is held by the end effector 4. For example, the calculation device 31 may generate a first template model TM that does not include a model portion corresponding to the hidden portion HP1 and / or that includes a model portion corresponding to the non-hidden portion HP2.Furthermore, in a situation where at least one of the position and orientation of a first target object OBJ#1 that is not held by the end effector 4 is calculated, if multiple first target objects OBJ#1 (multiple workpieces W) are bulk-stacked on the first placement device T#1 (e.g., a container), a portion of one first target object OBJ#1 may be hidden by another first target object OBJ#1 that is different from the first target object OBJ#1. The user may input an instruction to generate a template model TM taking into account the possibility that a portion of the first target object OBJ#1 may be hidden by such bulk stacking. For example, in order to generate a template model TM for calculating at least one of the position and orientation of a first target object OBJ#1 in a situation where a first portion of the first target object OBJ#1 is hidden by another first target object OBJ#1, the user may input an instruction to designate a model portion corresponding to the first portion of the first target object OBJ#1 as a model portion that should not be included in the template model TM. As a result, when multiple first target objects OBJ#1 are stacked in bulk on the first mounting device T#1, by selecting (in other words, determining) an appropriate template model TM according to the bulk stacking situation, the possibility of a situation occurring in which at least one of the position and orientation of the first target object OBJ#1 cannot be calculated by the matching process is reduced.

[0231] In the above description, the control device 3 generates the template model TM based on the image data IMG generated by the imaging system 2 (or another imaging system different from the imaging system 2) capturing an image of the target object OBJ. However, the control device 3 may generate the template model TM without using the image data IMG generated by the imaging system 2. For example, the control device 3 may extract an edge (e.g., the above-mentioned contour OL; the same applies hereinafter in this paragraph) of a two-dimensional model (e.g., a two-dimensional image) indicating the designed two-dimensional shape of the target object OBJ from the two-dimensional model, acquire a user instruction specifying a part of the edge on the model generation screen 63 displaying the extracted edge, and generate the contour model TM_edge based on the acquired user instruction. For example, the control device 3 may generate a two-dimensional model by projecting a three-dimensional model (e.g., a CAD model of the target object OBJ) indicating the designed three-dimensional shape of the target object OBJ onto a virtual two-dimensional plane, extract edges of the two-dimensional model, acquire a user instruction specifying a portion of the edge on a model generation screen 63 on which the extracted edges are displayed, and generate a contour model TM_edge based on the acquired user instruction. For example, the control device 3 may acquire a user instruction specifying a portion of the two-dimensional model on a model generation screen on which a two-dimensional model indicating the designed two-dimensional shape of the target object OBJ is displayed, and generate a two-dimensional model TM_2D based on the acquired user instruction. For example, the control device 3 may generate a two-dimensional model by projecting a three-dimensional model indicating the designed three-dimensional shape of the target object OBJ onto a virtual two-dimensional plane, acquire a user instruction specifying a portion of the two-dimensional model on a model generation screen on which the two-dimensional model is displayed, and generate a two-dimensional model TM_2D based on the acquired user instruction. For example, the control device 3 may acquire a user instruction specifying a part of a three-dimensional model (e.g., a CAD model of the target object OBJ) on a model generation screen that displays the three-dimensional model indicating the designed three-dimensional shape of the target object OBJ, and generate a three-dimensional model TM_3D based on the acquired user instruction. (2-2-3-4) Generation of Multiple Template Models TM

[0232] The control device 3 may generate a plurality of different template models TM. For example, the control device 3 may generate a plurality of different contour models TM_edge. For example, the control device 3 may generate a plurality of different two-dimensional models TM_2D. For example, the control device 3 may generate a plurality of different three-dimensional models TM_3D.

[0233] The multiple template models TM may each represent multiple different shapes. In other words, the multiple template models TM may each have multiple different shapes. For example, the multiple template models TM may include a first template model TM representing a first shape and a second template model TM representing a second shape different from the first shape. The multiple template models TM may include a first template model TM representing the shape of a first portion of the target object OBJ and a second template model TM representing the shape of a second portion of the target object OBJ that is partially different from the first portion. Note that the first portion of the target object OBJ and the second portion of the target object OBJ may or may not partially overlap. The first portion of the target object OBJ and the second portion of the target object OBJ may be the same size or different sizes. The first portion of the target object OBJ may correspond to the entire target object OBJ, and the second portion of the target object OBJ may correspond to a portion of the target object OBJ. Both the first portion of the target object OBJ and the second portion of the target object OBJ may correspond to a part of the target object OBJ.

[0234] As a first example, the multiple contour models TM_edge may each represent multiple contours OL having shapes different from one another. In other words, the multiple contour models TM_edge may each have multiple contours OL having shapes different from one another. For example, the multiple contour models TM_edge may include a first contour model TM_edge representing at least a portion of the contour OL of a target object OBJ having a first shape, and a second contour model TM_edge representing at least a portion of the contour OL of a target object OBJ having a second shape different from the first shape. The multiple contour models TM_edge may include a first contour model TM_edge representing the shape of the contour OL of a first portion of the target object OBJ, and a second contour model TM_edge representing the shape of the contour OL of a second portion of the target object OBJ that is partially different from the first portion.

[0235] As a second example, the multiple two-dimensional models TM_2D may each represent a plurality of two-dimensional shapes that are different from one another. In other words, the multiple two-dimensional models TM_2D may each have a plurality of two-dimensional shapes that are different from one another. For example, the multiple two-dimensional models TM_2D may include a first two-dimensional model TM_2D that represents a first two-dimensional shape and a second two-dimensional model TM_2D that represents a second two-dimensional shape that is different from the first two-dimensional shape. The multiple two-dimensional models TM_2D may include a first two-dimensional model TM_2D that represents a two-dimensional shape of a first portion of the target object OBJ and a second two-dimensional model TM_2D that represents a two-dimensional shape of a second portion of the target object OBJ that is partially different from the first portion.

[0236] As a third example, the multiple three-dimensional models TM_3D may each represent a multiple three-dimensional shape that is different from one another. In other words, the multiple three-dimensional models TM_3D may each have a multiple three-dimensional shape that is different from one another. For example, the multiple three-dimensional models TM_3D may include a first three-dimensional model TM_3D representing a first three-dimensional shape and a second three-dimensional model TM_3D representing a second three-dimensional shape that is different from the first three-dimensional shape. The multiple three-dimensional models TM_3D may include a first three-dimensional model TM_3D representing the three-dimensional shape of a first portion of the target object OBJ and a second three-dimensional model TM_3D representing the three-dimensional shape of a second portion of the target object OBJ that is partially different from the first portion.

[0237] If a plurality of different template models TM are generated, the control device 3 may select (in other words, determine) one template model TM that satisfies predetermined selection criteria (in other words, determination criteria) from among the plurality of different template models TM as the template model TM actually used in the matching process when performing the matching process. In this case, by generating a plurality of different template models TM, the control device 3 may consider the plurality of different template models TM to have been registered in advance as candidates for the template model TM actually used in the matching process. In this case, the generated (registered) template model TM may be referred to as a registered model, and the template model TM selected as the template model TM actually used in the matching process may be referred to as a selected model.

[0238] As an example, the control device 3 may assign priorities to multiple different template models TM in advance. The priorities may refer to the priorities at which the template models TM are selected as template models TM for performing the matching process (i.e., the computational process including the matching process). In this case, when performing the matching process, the control device 3 may select, from among the multiple different template models TM, one template model TM that satisfies a selection criterion based on the priorities as the one template model TM actually used for performing the matching process. For example, the control device 3 may select the one template model TM with the highest priority as the one template model TM actually used for performing the matching process. In other words, the control device 3 may select the template models TM in descending order of priority. The priorities may also be referred to as a priority order.

[0239] The control device 3 may assign priorities to multiple different template models TM in advance based on a user instruction. In this case, the user may input an instruction to assign priorities on a model generation screen where user instructions are input. In this case, as shown in FIGS. 18( a) and 18(b), priority information 634 indicating the priorities assigned to each template model TM may be displayed on the model generation screen (model generation screen 63 in the example shown in FIGS. 18(a) and 18(b)). In the example shown in FIGS. 18(a) and 18(b), the priority of the contour model TM_edge having a model ID of "#1" is set to priority 1, and the priority of the contour model TM_edge having a model ID of "#2" is set to priority 2, which is lower than priority 1.

[0240] As described above, when the end effector 4 holds the first target object OBJ#1, the control device 3 performs a matching process based on a template model TM that indicates the shape of the first target object OBJ#1, thereby calculating at least one of the position and orientation of the first target object OBJ#1 before it is held by the end effector 4. In this case, the control device 3 may generate a plurality of different template models TM that can each be used in the matching process for calculating at least one of the position and orientation of the first target object OBJ#1 when the end effector 4 holds the first target object OBJ#1.

[0241] As described above, when the first target object OBJ#1 held by the end effector 4 is released onto the second target object OBJ#2, the control device 3 calculates at least one of the position and orientation of the second target object OBJ#2 by performing a matching process based on a template model TM that indicates the shape of the second target object OBJ#2. In this case, the control device 3 may generate multiple different template models TM that can each be used in the matching process for calculating at least one of the position and orientation of the second target object OBJ#2 when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2.

[0242] As described above, when the first target object OBJ#1 held by the end effector 4 is released onto the second target object OBJ#2, the control device 3 calculates at least one of the position and orientation of the first target object OBJ#1 by performing a matching process based on a template model TM that indicates the shape of the first target object OBJ#1. In this case, the control device 3 may generate multiple different template models TM that can each be used in the matching process for calculating at least one of the position and orientation of the first target object OBJ#1 when the end effector 4 releases the first target object OBJ#1 onto the second target object OBJ#2.

[0243] As described above, when the end effector 4 holds the first target object OBJ#1, the end effector 4 is controlled to hold the first target object OBJ#1 in the holding position and orientation indicated by the holding position and orientation information 71. In this case, the hidden portion HP1 of the first target object OBJ#1 that is hidden by the end effector 4 (specifically, the hidden portion HP1 of the first target object OBJ#1 that cannot be imaged by the imaging system 2 because it is occluded by the end effector 4) may change depending on the holding position and orientation information 71 selected for holding the first target object OBJ#1. Furthermore, the non-hidden portion HP2 of the first target object OBJ#1 that is not hidden by the end effector 4 (specifically, the non-hidden portion HP2 of the first target object OBJ#1 that can be imaged by the imaging system 2 because it is not occluded by the end effector 4) may change depending on the holding position and orientation information 71 selected for holding the first target object OBJ#1. For example, a hidden portion HP1 in a situation where the end effector 4 is holding the first target object OBJ#1 at the holding position and posture indicated by one piece of holding position and posture information 71 may be at least partially different from a hidden portion HP1 in a situation where the end effector 4 is holding the first target object OBJ#1 at the holding position and posture indicated by another piece of holding position and posture information 71. For example, a non-hidden portion HP2 in a situation where the end effector 4 is holding the first target object OBJ#1 at the holding position and posture indicated by one piece of holding position and posture information 71 may be at least partially different from a non-hidden portion HP2 in a situation where the end effector 4 is holding the first target object OBJ#1 at the holding position and posture indicated by another piece of holding position and posture information 71.In this case, when a template model TM that does not include at least a portion of a model portion corresponding to the hidden portion HP1 and / or includes at least a portion of a model portion corresponding to the non-hidden portion HP2 in a situation where the end effector 4 holds the first target object OBJ#1 in the holding position and posture indicated by one piece of holding position and posture information 71 is used to calculate at least one of the position and posture of the first target object OBJ#1 held by the end effector 4 in the holding position and posture indicated by another piece of holding position and posture information 71, the template model TM may not include at least a portion of a model portion corresponding to the hidden portion HP1 and / or not include at least a portion of a model portion corresponding to the non-hidden portion HP2 in a situation where the end effector 4 holds the first target object OBJ#1 in the holding position and posture indicated by the other piece of holding position and posture information 71. As a result, there is a possibility that the matching similarity calculated in the matching process will be low (the accuracy of the matching process may be reduced). For this reason, the user may input an instruction to generate, for each piece of holding position and orientation information 71, a template model TM for calculating at least one of the position and orientation of the first target object OBJ#1 held by the end effector 4, taking into consideration that at least one of the hidden portion HP1 of the first target object OBJ#1 that is hidden by the end effector 4 and the non-hidden portion HP2 of the first target object OBJ#1 that is not hidden by the end effector 4 changes depending on the holding position and orientation information 71. That is, the user (control device 3) may generate multiple template models TM corresponding to multiple pieces of holding position and orientation information 71, respectively. In other words, the user may generate multiple template models TM associated with multiple pieces of holding position and orientation information 71, respectively. In other words, the user (control device 3) may generate multiple template models TM associated with multiple pieces of holding position and orientation information 71, respectively.

[0244] One template model TM associated with one piece of holding position and orientation information 71 may satisfy a first model condition that at least a part of a model portion corresponding to a non-hidden portion HP2 of the first target object OBJ#1 that is not hidden by the end effector 4 (specifically, the non-hidden portion HP2 of the first target object OBJ#1 that can be imaged by the imaging system 2 without being blocked by the end effector 4) is included in the one template model TM when the end effector 4 holds the first target object OBJ#1 in the holding position and orientation indicated by the one piece of holding position and orientation information 71. In this case, for example, in a situation assuming that the end effector 4 holds the first target object OBJ#1 in the holding position and orientation indicated by the one piece of holding position and orientation information 71, the user may input an instruction to generate, as the template model TM associated with the one piece of holding position and orientation information 71, a template model TM that includes at least a part of a model portion corresponding to the non-hidden portion HP2 of the first target object OBJ#1 that is not hidden by the end effector 4. In a situation where it is assumed that the end effector 4 holds the first target object OBJ#1 in the holding position and posture indicated by the one holding position and posture information 71, the calculation device 31 may generate a template model TM that includes at least a part of a model portion that corresponds to the non-hidden portion HP2 of the first target object OBJ#1 that is not hidden by the end effector 4, as a template model TM associated with the one holding position and posture information 71.

[0245] One template model TM associated with one piece of holding position and orientation information 71 may satisfy a second model condition that at least a part of a model portion corresponding to a hidden portion HP1 of the first target object OBJ#1 that is hidden by the end effector 4 (specifically, the hidden portion HP1 of the first target object OBJ#1 that cannot be imaged by the imaging system 2 because it is blocked by the end effector 4) is not included in the one template model TM when the end effector 4 holds the first target object OBJ#1 in the holding position and orientation indicated by the one piece of holding position and orientation information 71. In this case, for example, in a situation assuming that the end effector 4 holds the first target object OBJ#1 in the holding position and orientation indicated by the one piece of holding position and orientation information 71, the user may input an instruction to generate, as the template model TM associated with the one piece of holding position and orientation information 71, a template model TM that does not include at least a part of a model portion corresponding to the hidden portion HP1 of the first target object OBJ#1 that is hidden by the end effector 4. In a situation where it is assumed that the end effector 4 holds the first target object OBJ#1 in the holding position and orientation indicated by the holding position and orientation information 71, the calculation device 31 may generate a template model TM that does not include at least a portion of the model portion corresponding to the hidden portion HP1 of the first target object OBJ#1 that is hidden by the end effector 4, as a template model TM associated with the holding position and orientation information 71.

[0246] When the matching process is performed based on a template model TM that satisfies at least one of the first and second model conditions, the matching similarity calculated in the matching process is more likely to be high (the matching process is more likely to be executed accurately) than when the matching process is performed based on a template model TM that does not satisfy the first and second model conditions. This reduces the likelihood of a situation in which the first target object OBJ#1 held by the end effector 4 cannot be detected by the matching process. As a result, this reduces the likelihood of a situation in which at least one of the position and orientation of the first target object OBJ#1 held by the end effector 4 cannot be calculated by the matching process. Furthermore, due to the increased likelihood of a high matching similarity calculated in the matching process, the arithmetic unit 31 can calculate at least one of the position and orientation of the first target object OBJ#1 held by the end effector 4 with high accuracy.

[0247] When a plurality of template models TM respectively associated with a plurality of pieces of holding position and orientation information 71 are generated, the control device 3 may have information for associating the plurality of template models TM with the plurality of different pieces of holding position and orientation information 71. For example, as shown in Fig. 17 , the control device 3 may have, in the storage device 32, an association table 73 for associating the plurality of different template models TM with the plurality of different pieces of holding position and orientation information 71. The association table 73 shown in Fig. 17 associates the template models TM with the holding position and orientation information 71 by associating a model ID for identifying the template model TM with a holding position and orientation ID for identifying the holding position and orientation information 71.

[0248] In this embodiment, an example will be described in which one template model TM is associated with one piece of holding position and orientation information 71. However, as will be described later in a first modified example, a plurality of template models TM may be associated with one piece of holding position and orientation information 71. Even in this case, when holding position and orientation information 71 associated with a plurality of template models TM is generated, it may be considered that a plurality of template models TM are associated with a plurality of pieces of holding position and orientation information 71, respectively.

[0249] 18( a) and 18(b) show an example of a model generation screen 63 on which a user's instruction to generate a contour model TM_edge, which is an example of a template model TM, is input, the model generation screen 63 may include a screen 631 on which a contour model TM_edge generated on the basis of a user's instruction is displayed, and a screen 632 on which a piece of holding position and orientation information 71 associated with the contour model TM_edge is displayed. 18( a) and 18(b) show an example in which a contour OL of the first target object OBJ#1 reflecting the user's instruction (e.g., a contour OL in which a contour portion OL_in that should be included in the contour model TM_edge is distinguished from a contour portion OL_ex that should not be included in the contour model TM_edge) is displayed on the screen 631, thereby substantially displaying a contour model TM_edge generated based on the user's instruction. Also, FIGS. 18(a) and 18(b) show an example in which the holding position and orientation information 71 is displayed on the screen 632 by arranging the end effector 4 (end effector model EM) and the first target object OBJ#1 (object model OM#1) at the holding position and orientation indicated by the holding position and orientation information 71. However, the holding position and orientation information 71 may be displayed on the screen 632 by displaying any information indicating the holding position and orientation information 71. The holding position and orientation information 71 may be displayed by displaying numerical values ​​(for example, coordinate values) indicating the holding position and orientation indicated by the holding position and orientation information 71 on the screen 632 .

[0250] The model generation screen may further display information indicating the association between one template model TM and one holding position and orientation information 71. As shown in FIGS. 18A and 18B , an example of the information indicating the association between one template model TM and one holding position and orientation information 71 is ID information 633 indicating a combination of a model ID for identifying the template model TM and a holding position and orientation ID for identifying the holding position and orientation information 71. In the example shown in FIG. 18A , ID information 633 indicating that a contour model TM_edge having a model ID of “#1” is associated with the holding position and orientation information 71 having a holding position and orientation ID of “#1” is displayed. In the example shown in FIG. 18B , ID information 633 indicating that a contour model TM_edge having a model ID of “#2” is associated with the holding position and orientation information 71 having a holding position and orientation ID of “#2” is displayed.

[0251] The user may input an instruction to change the combination of the template model TM and the holding position and orientation information 71 displayed on the model generation screen. For example, the user may input an instruction to select (in other words, determine) one of the plurality of holding position and orientation information 71 registered in advance as an instruction to change the combination of the template model TM and the holding position and orientation information 71 displayed on the model generation screen. In this case, the calculation device 31 may control the display device 37 to display a model generation screen on which the one holding position and orientation information 71 selected by the user and the one template model TM associated with the one holding position and orientation information 71 are displayed.

[0252] 18(a) and 18(b) show an example in which a single template model TM and a single piece of holding position and orientation information 71 associated with the single template model TM are displayed on the model generation screen. However, the model generation screen may display at least two template models TM and at least two pieces of holding position and orientation information 71 associated with the at least two template models TM, respectively. In this case, the user may generate at least two template models TM in parallel. (2-2-3-5) Automatic Generation of Template Model TM

[0253] In the above description, the control device 3 (particularly, the arithmetic device 31) generates the template model TM based on a user's instruction. In addition to or instead of generating the template model TM based on a user's instruction, the arithmetic device 31 may automatically generate the template model TM without using a user's instruction. In this case, the user's effort to input an instruction to generate the template model TM is reduced. Furthermore, since the arithmetic device 31 does not need to wait for a user's instruction to generate the template model TM, the arithmetic device 31 can efficiently generate the template model TM. This effect is particularly effective in a situation where the arithmetic device 31 generates multiple template models TM.

[0254] As a first example, the calculation device 31 may divide a reference model BM, which is used as a reference for automatically generating a template model TM, into a plurality of division models DM, and generate each division model DM as a template model TM. For example, as shown in Figure 19 , which shows a top view of a plurality of division models DM divided from the reference model BM, the calculation device 31 may divide the reference model BM into a plurality of division models DM so that at least one division line serves as a boundary line between the plurality of division models DM, and generate each division model DM as a template model TM. In the example shown in Figure 19 , the calculation device 31 divides the reference model BM into four division models DM.

[0255] The division lines for dividing the reference model BM (e.g., at least one of the position and number of the division lines) may be automatically set by the calculation device 31. However, the division lines for dividing the reference model BM may also be set by a user. Note that, because the reference model BM is divided into divided models DM in a number corresponding to the division lines, setting the division lines may be considered equivalent to setting the number of divided models DM (the number of divisions) generated by dividing the reference model BM. The calculation device 31 may set the division lines based on the holding position and orientation information 71. For example, the calculation device 31 may calculate (estimate) the positional relationship between the end effector 4 holding the first target object OBJ#1 and the first target object OBJ#1 held by the end effector 4 based on the holding position and orientation indicated by the holding position and orientation information 71 (i.e., the holding position and orientation for the end effector 4 to hold the first target object OBJ#1). Thereafter, the calculation device 31 may identify (estimate) a hidden portion HP1 of the first target object OBJ#1 that is hidden by the end effector 4 holding the first target object OBJ#1 and a non-hidden portion HP2 of the first target object OBJ#1 that is not hidden by the end effector 4 holding the first target object OBJ#1, based on the calculation result of the positional relationship between the end effector 4 holding the first target object OBJ#1 and the first target object OBJ#1 held by the end effector 4. Thereafter, the calculation device 31 may set the division line so that at least a portion of the model portion corresponding to the hidden portion HP1 is not included in the division model DM and / or so that at least a portion of the model portion corresponding to the non-hidden portion HP2 is included in the division model DM. As a result, even when an automatically generated template model TM is used, the matching similarity calculated in the matching process is more likely to be high (the matching process is more likely to be executed more accurately).

[0256] When generating the contour model TM_edge as the template model TM, the generated contour model TM_edge may be used as the reference model BM. Alternatively, a model representing at least a portion of an edge extracted in an image represented by image data IMG may be used as the reference model BM. Alternatively, a model representing at least a portion of an edge OL extracted by a matching process based on the generated multidimensional model TM_M and image data IMG may be used as the reference model BM. Note that the process of extracting the contour OL by a matching process based on the generated multidimensional model TM_M and image data IMG has already been described as part of the description of step S11 in FIG. 14. Alternatively, a model representing at least a portion of an edge of a two-dimensional model extracted from a two-dimensional model (e.g., a two-dimensional image) representing the designed two-dimensional shape of the target object OBJ may be used as the reference model BM. Alternatively, the reference model BM may be a model that shows at least a portion of the edge of a two-dimensional model extracted from a two-dimensional model generated by projecting a three-dimensional model showing the designed three-dimensional shape of the target object OBJ onto a virtual two-dimensional plane.

[0257] When generating a two-dimensional model TM_2D as the template model TM, a generated two-dimensional model TM_2D may be used as the reference model BM. Alternatively, a model representing at least a portion of the image represented by the image data IMG may be used as the reference model BM. Alternatively, at least a portion of a two-dimensional model representing the designed two-dimensional shape of the target object OBJ may be used as the reference model BM. Alternatively, at least a portion of a two-dimensional model generated by projecting a three-dimensional model (e.g., a CAD model) representing the designed three-dimensional shape of the target object OBJ onto a virtual two-dimensional plane may be used as the reference model BM.

[0258] When generating a three-dimensional model TM_3D as a template model TM, a generated three-dimensional model TM_3D may be used as the reference model BM. Alternatively, at least a part of a three-dimensional model in which three-dimensional position data is generated from the image data IMG may be used as the reference model BM. Alternatively, at least a part of a three-dimensional model (e.g., a CAD model) indicating the designed three-dimensional shape of the target object OBJ may be used as the reference model BM.

[0259] As a second example, as shown in Fig. 20 , the calculation device 31 may rotate the reference model BM by a predetermined angle around a predetermined rotation axis, and then divide the reference model BM into multiple division models DM, thereby generating each division model DM as a template model TM. In this case, multiple template models TM (see Fig. 20 ) having shapes different from the multiple template models TM (see Fig. 19 ) generated by dividing the reference model BM before rotation are generated by dividing the reference model BM after rotation. In this case, the calculation device 31 can generate a large number of template models TM relatively easily by alternately repeating the rotation of the reference model BM and the division of the reference model BM.

[0260] The rotation axis about which the reference model BM is rotated may be automatically set by the calculation device 31. However, the rotation axis about which the reference model BM is rotated may also be set by a user. The rotation angle about which the reference model BM is rotated may also be automatically set by the calculation device 31. However, the rotation angle about which the reference model BM is rotated may also be set by a user. The calculation device 31 may set at least one of the rotation axis and the rotation angle about which the reference model BM is rotated based on the holding position and orientation information 71. For example, the calculation device 31 may calculate (estimate) the positional relationship between the end effector 4 holding the first target object OBJ#1 and the first target object OBJ#1 held by the end effector 4 based on the holding position and orientation indicated by the holding position and orientation information 71 (i.e., the holding position and orientation for the end effector 4 to hold the first target object OBJ#1). Then, based on the calculation result of the positional relationship between the end effector 4 holding the first target object OBJ#1 and the first target object OBJ#1 held by the end effector 4, the calculation device 31 may identify (estimate) a hidden portion HP1 of the first target object OBJ#1 that is hidden by the end effector 4 holding the first target object OBJ#1 and a non-hidden portion HP2 of the first target object OBJ#1 that is not hidden by the end effector 4 holding the first target object OBJ#1. Then, the calculation device 31 may set at least one of the rotation axis and the rotation angle so that at least a portion of the model portion corresponding to the hidden portion HP1 is not included in the division model DM and / or so that at least a portion of the model portion corresponding to the non-hidden portion HP2 is included in the division model DM. As a result, even when an automatically generated template model TM is used, the matching similarity calculated in the matching process is more likely to be high (the matching process is more likely to be performed accurately). As a third example, the computing device 31 may generate the template model TM using an inference device 9 constructed by machine learning.

[0261] As shown in FIG. 21 , which conceptually illustrates the inference unit 9, the inference unit 9 may be constructed by machine learning so as to output a model usable as a template model TM when image data IMG is input. For example, the inference unit 9 may be constructed by machine learning using a teacher data set including a plurality of teacher data sets including the template model TM and the image data IMG, and a template model TM generated based on the image data IMG and a user instruction. In this case, parameters defining the operation of the inference unit 9 may be adjusted by machine learning so as to reduce the difference between the template model TM output by the inference unit 9 to which the image data IMG included in the teacher data is input and the template model TM included in the teacher data. When generating a first template model TM for calculating at least one of the position and orientation of a first target object OBJ#1 that is not held by the end effector 4, the inference unit 9 may be input with image data IMG generated by capturing an image of the first target object OBJ#1 (e.g., the first target object OBJ#1 that is not held by the end effector 4). The inference unit 9 may receive input of image data IMG generated by capturing an image of a first target object OBJ#1 (e.g., a first target object OBJ#1 that is not being held by the end effector 4) and an end effector 4 (e.g., an end effector 4 that is not holding the first target object OBJ#1 but is attempting to hold the first target object OBJ#1). When generating a second template model TM for calculating at least one of the position and orientation of a second target object OBJ#2 from which the first target object OBJ#1 is released, the inference unit 9 may receive input of image data IMG generated by capturing an image of the second target object OBJ#2. The inference unit 9 may receive input of image data IMG generated by capturing an image of the second target object OBJ#2 and at least one of the first target object OBJ#1 to be released by the second target object OBJ#1 and the end effector 4 holding the first target object OBJ#1.When generating a second template model TM for calculating at least one of the position and orientation of the first target object OBJ#1 held by the end effector 4 (i.e., the first target object OBJ#1 to be released to the second target object OBJ#2), the inference unit 9 may receive image data IMG generated by capturing an image of the first target object OBJ#1 (e.g., the first target object OBJ#1 held by the end effector 4). The inference unit 9 may receive image data IMG generated by capturing an image of the first target object OBJ#1 (e.g., the first target object OBJ#1 that is not being held by the end effector 4) and at least one of the second target object OBJ#2 and the end effector 4 (e.g., the end effector 4 holding the first target object OBJ#1).

[0262] The inference unit 9 may be constructed by machine learning so as to output a single model that can be used as the template model TM when image data IMG is input. Alternatively, the inference unit 9 may be constructed by machine learning so as to output multiple models that can each be used as the template model TM when image data IMG is input. When the inference unit 9 outputs multiple models, the calculation device 31 can generate multiple template models TM more efficiently than when the inference unit 9 outputs a single model.

[0263] The training data for constructing the inferencer 9 that outputs the contour model TM_edge may include image data IMG or virtual image data. The training data for constructing the inferencer 9 that outputs the contour model TM_edge may include, in addition to the image data IMG or virtual image data, a portion of the contour (e.g., a wireframe) of the target object OBJ detected by a matching process based on the ima...

Claims

1. A robot provided with a holding device capable of holding a target object and moving the holding device, and a control device that generates a control signal for controlling at least one of the holding devices, the control device including an arithmetic unit that generates the control signal, and a communication device that outputs the control signal generated by the arithmetic unit, wherein the arithmetic unit selects, as a selection model, a model indicating a part of the target object based on a first imaging result obtained by imaging the target object with an imaging system before the target object is held by the holding device, and generates the control signal for controlling at least one of the robot and the holding device for processing the target object held by the holding device based on a second imaging result obtained by imaging the target object held by the holding device with the imaging system when at least one of the robot and the holding device is controlled and the selection model.

2. The control device according to claim 1, wherein the arithmetic unit selects the model as the selection model based on at least one of information on a holding position of the target object by the holding device predicted based on the first imaging result before the target object is held by the holding device, information on an attitude of the holding device holding the target object, and information on an attitude of the target object held by the holding device.

3. At least one of information on a holding position of the target object by the holding device, information on an attitude of the holding device holding the target object, and information on an attitude of the target object held by the holding device is registered in advance as registered position and attitude information, and the arithmetic unit selects the model as the selection model based on the registered position and attitude information selected based on the first imaging result. The control device according to claim 2.

4. The arithmetic unit selects the model as the selection model from a plurality of different registered position and attitude information registered in advance as at least one candidate of information on a holding position of the target object by the holding device, information on an attitude of the holding device holding the target object, and information on an attitude of the target object held by the holding device based on the registered position and attitude information selected based on the first imaging result. The control device according to claim 3.

5. The arithmetic unit selects the registered model as the selected model from a plurality of different registered models registered in advance as candidates for the model showing a part of the target object based on the selected registered position and orientation information. The control device according to claim 4.

6. The plurality of different registered models include a first registered model and a second registered model having a different shape from the first registered model. The arithmetic unit selects the first registered model or the second registered model as the selected model based on a third imaging result obtained by imaging the target object held by the holding device with the imaging system and the selected registered position and orientation information. The control device according to claim 5.

7. The arithmetic unit selects the first registered model or the second registered model as the selected model based on the result of the matching process based on the third imaging result and the first registered model and the result of the matching process based on the third imaging result and the second registered model. The control device according to claim 6.

8. If the arithmetic unit determines that the process cannot be performed even when the control signal is generated based on the third imaging result and the first registered model, the arithmetic unit selects the second registered model as the selected model. The control device according to claim 6.

9. The first registered model and the second registered model are associated in advance with one of the plurality of different registered position and orientation information. When the selected registered position and orientation information is the one registered position and orientation information, the arithmetic unit selects the first registered model or the second registered model as the selected model based on information regarding the association between the first registered model, the second registered model, and the one registered position and orientation information, the one registered position and orientation information, and the third imaging result. The control device according to any one of claims 6 to 8.

10. The difference in shape between the first registered model and the second registered model is smaller than the difference in shape between the first registered model and a third registered model among the plurality of different registered models associated in advance with registered position and orientation information different from the selected one of the plurality of different registered position and orientation information, and the difference in shape between the third registered model and the second registered model. The control device according to claim 9.

11. The plurality of different registration models are pre-associated with the plurality of different registration position and orientation information. The arithmetic unit selects one of the plurality of different registration models as the selected model based on information regarding the relationship between the associated plurality of different registration models and the plurality of different registration positions and orientations, and the selected registration position and orientation. The control device according to any one of claims 5 to 10.

12. The arithmetic unit selects, as the selected model, a first registration model and a second registration model having a different shape from the first registration model from among a plurality of different registration models pre-registered as candidates for the model indicating a part of the target object, based on the selected registration position and orientation information. When it is determined that the target object held by the holding device cannot be processed by the control signal generated based on the third imaging result of the target object imaged by the imaging system and the first registration model, at least one of the robot and the holding device is controlled by the control signal generated based on the second imaging result and the second registration model. The control device according to claim 4.

13. Using the control signal for controlling at least one of the robot and the holding device for processing the target object held by the holding device as a second control signal, the arithmetic unit generates a first control signal for controlling at least one of the robot and the holding device so that the holding device holds the target object with the selected registration position and orientation information. The second control signal includes a signal for controlling at least one of the robot and the holding device for processing the target object held by the holding device, which is generated based on the second imaging result of the target object held by the holding device imaged by the imaging system and the selected model when at least one of the robot and the holding device is controlled based on the first control signal. The control device according to any one of claims 3 to 12.

14. The arithmetic unit selects the model as the selected model based on the object position and orientation including at least one of the position and orientation of the target object before being held by the holding device, which is calculated based on the first imaging result. The control device according to claim 1.

15. The arithmetic unit selects the registered model as the selected model from a plurality of different registered models registered in advance as candidates for the model indicating a part of the target object based on the calculated object position and orientation. The control device according to claim 14.

16. The plurality of different registered models include a first registered model and a second registered model having a different shape from the first registered model. The arithmetic unit is based on a third imaging result obtained by imaging the target object held by the holding device with the imaging system and the calculated object position and orientation, and selects the first registered model or the second registered model as the selected model. The control device according to claim 15.

17. The arithmetic unit selects the first registered model or the second registered model as the selected model based on the result of the matching process based on the third imaging result and the first registered model and the result of the matching process based on the third imaging result and the second registered model. The control device according to claim 16.

18. When the arithmetic unit determines that the process cannot be performed even if the control signal is generated based on the third imaging result and the first registered model, the arithmetic unit selects the second registered model as the selected model. The control device according to claim 16.

19. The first registered model and the second registered model are associated in advance with one of the plurality of different object positions and orientations. When the calculated object position and orientation is the one object position and orientation, the arithmetic unit is based on information regarding the relationship between the associated first registered model and second registered model and the one object position and orientation, and the one object position and orientation and the third imaging result, and selects the first registered model or the second registered model as the selected model. The control device according to any one of claims 15 to 17.

20. The difference in shape between the first registered model and the second registered model is smaller than the difference in shape between the first registered model and a third registered model among the plurality of different registered models associated in advance with an object position and orientation different from the calculated one object position and orientation among the plurality of different object positions and orientations, and the difference in shape between the third registered model and the second registered model. The control device according to claim 19.

21. The plurality of different registration models are pre-associated with the plurality of different object position postures, and the arithmetic unit selects one of the plurality of different registration models as the selection model based on information regarding the relationship between the plurality of different associated registration models and the plurality of different object position postures and the calculated object position posture. The control device according to any one of claims 15 to 20.

22. The arithmetic unit selects, as the selection model, a first registration model and a second registration model having a different shape from the first registration model from among a plurality of different registration models pre-registered as candidates for the model indicating a part of the target object based on the calculated object position posture. When it is determined that the target object held by the holding device cannot be processed with the control signal generated based on the third imaging result of the target object imaged by the imaging system and the first registration model, at least one of the robot and the holding device is controlled by the control signal generated based on the second imaging result and the second registration model. The control device according to claim 15.

23. Using the control signal for controlling at least one of the robot and the holding device for processing the target object held by the holding device as a second control signal, the arithmetic unit generates a first control signal for controlling at least one of the robot and the holding device so that the holding device holds the target object based on the calculated object position posture. The second control signal includes a signal for controlling at least one of the robot and the holding device for processing the target object held by the holding device, which is generated based on the second imaging result of the target object imaged by the imaging system when at least one of the robot and the holding device is controlled based on the first control signal and the selection model. The control device according to any one of claims 14 to 22.

24. The first registration model and the second registration model are each given a priority in advance for performing arithmetic processing by the arithmetic unit for selection as the selection model, and the priority given to the first registration model is higher than the priority given to the second registration model. The control device according to any one of claims 6 to 10 and 16 to 20.

25. The third imaging result is the same imaging result as the second imaging result. The control device according to any one of claims 6 to 10, 12, 16 to 20, and 22.

26. The arithmetic unit generates the model. The control device according to claim 1 or 2.

27. The model includes a contour model showing a part of the contour of the target object, and the arithmetic unit generates the contour model based on an imaging result obtained by imaging the target object or an object having substantially the same shape as the target object with the imaging system or an imaging system different from the imaging system. The control device according to claim 26.

28. The arithmetic unit generates the contour model based on the imaging result used for generating the contour model and a CAD model showing at least a part of the target object. The control device according to claim 27.

29. The arithmetic unit generates the contour model based on a part of the contour of the target object or the object having substantially the same shape, which is extracted from an image as the imaging result used for generating the contour model, based on the imaging result used for generating the contour model and the CAD model. The control device according to claim 28.

30. The arithmetic unit generates the contour model using an inference unit that outputs a model that can be used as the contour model when image data as the imaging result used for generating the contour model is input. The control device according to any one of claims 27 to 29.

31. The arithmetic unit generates each of at least some of the plurality of different registration models. The control device according to any one of claims 5 to 12 and 15 to 22.

32. Each of at least a part of the registration models includes a contour model showing a part of the contour of the target object, and the arithmetic unit generates a plurality of different contour models to be used as each of at least a part of the registration models based on an imaging result obtained by imaging the target object or an object having substantially the same shape as the target object with the imaging system or an imaging system different from the imaging system. The control device according to claim 31.

33. The arithmetic unit uses an inference device that outputs a plurality of models that can be used as the plurality of different contour models when image data as the imaging result used for generating the contour model is input, to generate the plurality of different contour models. The control device according to claim 32.

34. The imaging result used for generating the contour model includes a fourth imaging result obtained by imaging the target object or the object having substantially the same shape held by the holding device or a holding device different from the holding device with the imaging system or the different imaging system. The control device according to any one of claims 27 to 30, 32, and 33.

35. The model showing a part of the target object is a model showing a part of the contour of the target object. The control device according to any one of claims 1 to 34.

36. After generating the control signal, the arithmetic unit updates the control signal based on a fifth imaging result obtained by imaging the target object held by the holding device moving based on the control signal with the imaging system and the selection model. The control device according to any one of claims 1 to 35.

37. The arithmetic unit repeatedly updates the control signal based on each imaging result obtained by repeatedly imaging the target object held by the holding device with the imaging system and the selection model until the target object held by the holding device is located at at least one of the positions and postures where the processing is possible. The control device according to claim 36.

38. The robot can calculate at least one of the position and posture of the reference point of the holding device, and when the arithmetic unit determines not to update the control signal based on the imaging result and the selection model, the arithmetic unit updates the control signal based on at least one of the calculated position and posture of the reference point. The control device according to claim 37.

39. When the arithmetic unit determines not to update the control signal based on the imaging result and the selection model in the first period, based on at least one of the position and orientation of the target object held by the holding device in a second period prior to the first period, which was imaged by the imaging system, and calculated based on the imaging result and the selection model, the control signal is updated in the first period. The control device according to claim 37 or 38.

40. When at least one of the following occurs: the arithmetic processing for calculating at least one of the position and orientation of the target object held by the holding device based on the imaging result and the selection model is defective, and at least one of the position and orientation of the target object held by the holding device calculated by the arithmetic processing is defective, the arithmetic unit determines not to update the control signal based on the imaging result and the selection model. The control device according to claim 38 or 39.

41. After the target object is held by the holding device by controlling at least one of the robot and the holding device, the arithmetic unit repeatedly generates the control signal based on each imaging result including the second imaging result obtained by repeatedly imaging the target object held by the holding device with the imaging system and the selection model. The control device according to any one of claims 1 to 40.

42. The robot can calculate at least one of the position and orientation of the reference point of the holding device. When the arithmetic unit determines not to generate the control signal based on the imaging result and the selection model, the arithmetic unit generates the control signal based on at least one of the calculated position and orientation of the reference point. The control device according to claim 41.

43. When the arithmetic unit determines not to generate the control signal based on the imaging result and the selection model in the first period, based on at least one of the position and orientation of the target object held by the holding device in a second period prior to the first period, which was imaged by the imaging system, and calculated based on the imaging result and the selection model, the control signal is generated in the first period. The control device according to claim 41 or 42.

44. The arithmetic unit determines that the control signal is not generated based on the imaging result and the selection model when at least one of the following occurs: the arithmetic processing for calculating at least one of the position and orientation of the target object held by the holding device based on the imaging result and the selection model is defective, and at least one of the position and orientation of the target object held by the holding device calculated by the arithmetic processing is defective. The control device according to claim 42 or 43.

45. Taking the target object as the first target object, the processing of the first target object includes the processing of the first target object held by the holding device with respect to a second target object different from the first target object. The control signal is generated based on the sixth imaging result of the second target object imaged by the imaging system, the second imaging result, and the selection model. The control signal includes a signal for controlling at least one of the robot and the holding device so that the first target object held by the holding device approaches the second target object for the processing. The control device according to any one of claims 1 to 44.

46. The sixth imaging result is the same as the second imaging result, and the second imaging result is the result of imaging the first target object and the second target object by the imaging system. The control device according to claim 45.

47. After generating the control signal, the arithmetic unit updates the control signal based on the imaging result of imaging the first target object held by the moving holding device based on the control signal and the second target object by the imaging system and the selection model. The control device according to claim 45 or 46.

48. The arithmetic unit repeatedly updates the control signal based on each imaging result of repeatedly imaging the first target object held by the holding device and the second target object by the imaging system until at least one of the positional relationship and the orientation relationship between the first target object and the second target object that enables the processing is achieved, and the selection model. The control device according to claim 47.

49. The processing includes processing of the first target object held by the holding device with respect to the second target object during movement, and the sixth imaging result is a result of imaging the second target object during movement by the imaging system. The control device according to any one of claims 45 to 48.

50. Regarding the target object as the first target object, the processing of the first target object includes processing of the first target object held by the holding device with respect to a second target object different from the first target object. The second imaging result is a result of imaging the first target object held by the holding device and the second target object by the imaging system. After the first target object is held by the holding device by controlling at least one of the robot and the holding device, the arithmetic unit repeatedly generates the control signal based on each imaging result including the second imaging result, which is a result of repeatedly imaging the first target object held by the holding device and the second target object by the imaging system, and the selection model. The control device according to any one of claims 1 to 49.

51. The processing includes processing of the first target object held by the holding device with respect to the second target object during movement, and the imaging result is a result of imaging the first target object held by the holding device during movement and the second target object during movement by the imaging system based on the control signal. The control device according to claim 50.

52. The third imaging result is the same imaging result as the second imaging result. The control device according to any one of claims 1 to 51.

53. The plurality of different registration models are pre-associated with the plurality of different registration position and orientation information. The arithmetic unit generates a display signal for displaying at least one registration model among the plurality of different registration models associated with each other and at least one registration position and orientation information among the plurality of different registration position and orientation information on a display device, and the communication device outputs the display signal to the display device. The control device according to claim 5.

54. The display signal includes a signal for displaying, on the display device, one piece of registered position and orientation information among the plurality of different pieces of registered position and orientation information, and at least one of the plurality of different registered models associated with the one piece of registered position and orientation information. The control device according to claim 53.

55. The display signal includes a signal for displaying, on the display device, information indicating the association between the one piece of registered position and orientation information and the at least one registered model, which are associated with each other. The control device according to claim 54.

56. The display signal includes a signal for displaying, on the display device, the at least one registered model associated with the one piece of registered position and orientation information and the priority for performing arithmetic processing by the arithmetic device for selecting the at least one registered model as the selected model, which is assigned to the at least one registered model. The control device according to claim 54 or 55.

57. The arithmetic device generates at least one model among the plurality of different registered models based on an instruction from the user. The control device according to claim 5.

58. At least some of the plurality of different models are contour models each showing a part of the contour of the target object. The arithmetic device generates a display signal for displaying, on the display device, an input screen on which an instruction from the user is input. The communication device outputs the display signal to the display device. The user makes the designation as the instruction of at least one of a part of the contour of the target object to be used as at least a part of the contour model and a part of the contour of the target object not to be used as at least a part of the contour model on the input screen displayed on the display device based on the display signal. The arithmetic device generates the contour model based on the designation. The control device according to claim 57.

59. The input screen includes information regarding at least one of a part of the contour of the target object to be used as at least a part of the contour model and a part of the contour of the target object not to be used as at least a part of the contour model. The control device according to claim 58.

60. A plurality of the target objects are placed on a placement device, and the first imaging result is a result of imaging, by the imaging system, a target object group including at least a part of the plurality of target objects placed on the placement device. The control device according to any one of claims 1 to 59.

61. Each of the plurality of target objects is irregularly placed on the placement device. The control device according to claim 60.

62. The imaging system is provided on the robot, and the robot moves the holding device and the imaging system. The control device according to any one of claims 1 to 61.

63. The control signal includes a signal for controlling at least one of the robot and the holding device to move the holding device for the processing of the target object held by the holding device, the control signal being generated based on the second imaging result and the selection model. The control device according to any one of claims 1 to 62.

64. A control system including the control device according to any one of claims 1 to 63 and the imaging system.

65. A robot system including the control device according to any one of claims 1 to 63, the imaging system, and the robot.

66. A control method for generating a control signal for controlling at least one of a robot that can hold a target object and the holding device to move the holding device, the control method including: selecting, as a selection model, a model indicating a part of the target object based on a first imaging result of imaging, by an imaging system, the target object before being held by the holding device; and generating the control signal for controlling at least one of the robot and the holding device for the processing of the target object held by the holding device based on a second imaging result of imaging, by the imaging system, the target object held by the holding device after at least one of the robot and the holding device is controlled and the selection model.

67. A computer program for causing a computer to execute the control method according to claim 66.

68. A robot provided with a holding device capable of holding a target object and moving the holding device, and a control device that generates a control signal for controlling at least one of the holding devices, the control device including an arithmetic unit that generates the control signal, and a communication unit that outputs the control signal generated by the arithmetic unit, wherein the arithmetic unit selects, as a selection model, a model showing a part of the target object based on a seventh imaging result obtained by imaging the target object held by the holding device with an imaging system when at least one of the robot and the holding device is controlled, and generates the control signal for controlling at least one of the robot and the holding device for processing the target object held by the holding device based on a second imaging result obtained by imaging the target object held by the holding device with the imaging system and the selection model.

69. The control device according to claim 68, wherein the model selected as the selection model is a second model, and the arithmetic unit selects the second model as the selection model based on a first model showing at least a part of the target object and the seventh imaging result.

70. The control device according to claim 68, wherein the arithmetic unit selects the model as the selection model based on a holding position and orientation including at least one of the position and orientation of the target object held by the holding device, which is calculated based on the seventh imaging result.

71. The control device according to claim 70, wherein the model selected as the selection model is a second model, and the holding position and orientation are generated based on a first model showing at least a part of the target object and the seventh imaging result.

72. The control device according to claim 69 or 71, wherein a range of a part of the target object shown by the second model is narrower than a range of at least a part of the target object shown by the first model.

73. The control device according to claim 72, wherein the second model is a model showing at least a part of the contour of the target object, and the first model is a model showing at least a part of the contour of the target object and at least a part of a portion surrounded by the contour in the target object.

74. The control device according to claim 72 or 73, wherein the first model is at least a part of a CAD model showing the target object.

75. A plurality of different registered models registered in advance as candidates for the model are associated in advance with a plurality of different holding position postures. The arithmetic unit selects, as the selected model, a first registered model among the plurality of different registered models associated with the calculated holding position posture, a second registered model different from the first registered model, based on an eighth imaging result obtained by imaging the object held by the holding device with an imaging system. The control device according to claim 70 or 71.

76. The arithmetic unit selects, as the selected model, the first registered model or the second registered model based on a result of matching processing based on the eighth imaging result and the first registered model and a result of matching processing based on the calculated eighth imaging result and the second registered model. The control device according to claim 75.

77. If the arithmetic unit determines that the processing cannot be performed even when the control signal is generated based on the eighth imaging result and the first registered model, the arithmetic unit selects the second registered model as the selected model. The control device according to claim 75.

78. Priorities for performing arithmetic processing by the arithmetic unit for selection as the selected model are assigned in advance to the first registered model and the second registered model. The priority assigned to the first registered model is higher than the priority assigned to the second registered model. The control device according to claim 75 or 77.

79. The arithmetic unit selects, as the selected model, a first registered model and a second registered model having a different shape from the first registered model from among a plurality of different registered models registered in advance as candidates for the model based on the calculated holding position posture. When it is determined that the object cannot be processed with the control signal generated based on the eighth imaging result obtained by imaging the object held by the holding device with the imaging system and the first registered model, at least one of the robot and the holding device is controlled with the control signal generated based on the second imaging result and the second registered model. The control device according to claim 70 or 71.

80. The difference in shape between the first registered model and the second registered model is smaller than the difference in shape between the first registered model and a third registered model among the plurality of different registered models that are pre-associated with a holding position and orientation different from the calculated holding position and orientation among the plurality of different holding positions and orientations. The control device according to any one of claims 75 to 79.

81. The eighth imaging result is the same imaging result as the second imaging result. The control device according to any one of claims 75 to 80.

82. The eighth imaging result is the same imaging result as the seventh imaging result. The control device according to any one of claims 75 to 81.

83. The plurality of different registered models pre-registered as candidates for the model are pre-associated with a plurality of different holding positions and orientations, and the arithmetic unit selects one of the plurality of different registered models as the selected model based on information regarding the relationship between the plurality of different registered models associated therewith and the plurality of different holding positions and orientations and the calculated holding position and orientation. The control device according to claim 70 or 71.

84. The model showing a part of the object is a model showing a part of the contour of the object. The control device according to any one of claims 68 to 83.

85. The seventh imaging result is the same imaging result as the second imaging result. The control device according to any one of claims 68 to 84.

86. The control signal includes a signal for controlling at least one of the robot and the holding device to move the holding device for the processing of the object held by the holding device, which is generated based on the second imaging result and the selected model. The control device according to any one of claims 68 to 85.

87. Using the control signal for controlling at least one of the robot and the holding device for processing the target object held by the holding device as a second control signal, the arithmetic unit generates a first control signal for controlling at least one of the robot and the holding device so that the holding device holds the target object based on a first imaging result obtained by imaging the target object before being held by the holding device with the imaging system. The second control signal is generated based on the first control signal, the second imaging result obtained by imaging the target object held by the holding device with the imaging system when at least one of the robot and the holding device is controlled, and the selection model, and includes a signal for controlling at least one of the robot and the holding device for processing the target object held by the holding device. The control device according to any one of claims 68 to 86.

88. The arithmetic unit selects the model as the selection model based on a holding position and orientation including at least one of the position and orientation of the target object held by the holding device, which is calculated based on the seventh imaging result. The first control signal includes information regarding the holding position of the target object by the holding device, information regarding the orientation of the holding device holding the target object, and information regarding the orientation of the target object held by the holding device, based on the first imaging result, and is a signal for controlling at least one of the robot and the holding device so that the holding device holds the target object at at least one of the position and orientation indicated by the registered position and orientation information. At least one of the holding position of the target object held by the holding device, the orientation of the holding device holding the target object, and the orientation of the target object held by the holding device indicated by the calculated holding position and orientation is different from at least one of the holding position of the target object by the holding device, the information regarding the orientation of the holding device holding the target object, and the information regarding the orientation of the target object held by the holding device indicated by the registered position and orientation information. The control device according to claim 87.

89. The arithmetic unit generates the model. The control device according to claim 68.

90. The model includes a contour model showing a part of the contour of the target object, and the arithmetic unit generates the contour model based on an imaging result obtained by imaging the target object or an object having substantially the same shape as the target object with the imaging system or an imaging system different from the imaging system. The control device according to claim 89.

91. The arithmetic unit generates the contour model based on the imaging result used for generating the contour model and a CAD model showing at least a part of the target object. The control device according to claim 90.

92. The arithmetic unit generates the contour model based on a part of the contour of the target object or the object having substantially the same shape, which is extracted from an image as the imaging result used for generating the contour model, based on the imaging result used for generating the contour model and the CAD model. The control device according to claim 91.

93. The arithmetic unit generates the contour model using an inference unit that outputs a model that can be used as the contour model when image data as the imaging result used for generating the contour model is input. The control device according to any one of claims 90 to 92.

94. The arithmetic unit generates each of at least some of the plurality of different registered models. The control device according to any one of claims 75 to 83.

95. Each of the at least some of the registered models includes a contour model showing a part of the contour of the target object, and the arithmetic unit generates a plurality of different contour models used as each of the at least some of the registered models based on an imaging result obtained by imaging the target object or an object having substantially the same shape as the target object with the imaging system or an imaging system different from the imaging system. The control device according to claim 94.

96. The arithmetic unit generates the plurality of different contour models using an inference unit that outputs a plurality of models that can be used as the plurality of different contour models when image data as the imaging result used for generating the contour model is input. The control device according to claim 95.

97. The imaging result used for generating the contour model includes the imaging result of the target object held by the holding device or a holding device different from the holding device, or an object having substantially the same shape, imaged by the imaging system or a different imaging system. The control device according to any one of claims 89 to 93, 95, and 96.

98. The ninth imaging result is the same imaging result as the seventh imaging result. The control device according to claim 97.

99. After generating the control signal, the arithmetic unit updates the control signal based on the tenth imaging result of the target object held by the holding device moving based on the control signal and imaged by the imaging system and the selection model. The control device according to any one of claims 68 to 98.

100. The arithmetic unit repeatedly updates the control signal based on each imaging result of repeatedly imaging the target object held by the holding device with the imaging system until the target object held by the holding device is located at at least one of the positions and postures where the processing is possible, and the selection model. The control device according to claim 99.

101. The robot can calculate at least one of the position and posture of the reference point of the holding device. When the arithmetic unit determines not to update the control signal based on the imaging result and the selection model, the arithmetic unit updates the control signal based on at least one of the calculated position and posture of the reference point. The control device according to claim 100.

102. When the arithmetic unit determines not to update the control signal based on the imaging result and the selection model in the first period, the arithmetic unit updates the control signal in the first period based on at least one of the position and posture of the target object held by the holding device, which is calculated based on the imaging result of imaging the target object held by the holding device with the imaging system in the second period prior to the first period and the selection model. The control device according to claim 100 or 101.

103. The arithmetic device determines not to update the control signal based on the imaging result and the selected model when at least one of the following occurs: the arithmetic processing for calculating at least one of the position and orientation of the target object held by the holding device based on the imaging result and the selected model is defective; and at least one of the position and orientation of the target object held by the holding device calculated by the arithmetic processing is defective. The control device according to claim 101 or 102.

104. After at least one of the robot and the holding device is controlled to hold the target object by the holding device, the arithmetic device repeatedly generates the control signal based on each imaging result including the second imaging result obtained by repeatedly imaging the target object held by the holding device with the imaging system and the selected model. The control device according to any one of claims 68 to 103.

105. The robot is capable of calculating at least one of the position and orientation of the reference point of the holding device. When the arithmetic device determines not to generate the control signal based on the imaging result and the selected model, the arithmetic device generates the control signal based on at least one of the calculated position and orientation of the reference point. The control device according to claim 104.

106. When the arithmetic device determines not to generate the control signal based on the imaging result and the selected model in a first period, the arithmetic device generates the control signal in the first period based on at least one of the position and orientation of the target object held by the holding device, which is calculated based on the imaging result of imaging the target object held by the holding device with the imaging system in a second period prior to the first period and the selected model. The control device according to claim 104 or 105.

107. The arithmetic device determines not to generate the control signal based on the imaging result and the selected model when at least one of the following occurs: the arithmetic processing for calculating at least one of the position and orientation of the target object held by the holding device based on the imaging result and the selected model is defective; and at least one of the position and orientation of the target object held by the holding device calculated by the arithmetic processing is defective. The control device according to claim 105 or 106.

108. Taking the target object as the first target object, the processing of the first target object includes processing of the first target object held by the holding device with respect to a second target object different from the first target object, generating the control signal based on the 11th imaging result of imaging the second target object by the imaging system, the second imaging result, and the selection model, and the control signal includes a signal for controlling at least one of the robot and the holding device so that the first target object held by the holding device approaches the second target object for the processing. The control device according to any one of claims 68 to 107.

109. The 11th imaging result is the same imaging result as the second imaging result, and the second imaging result is the result of imaging the first target object and the second target object by the imaging system. The control device according to claim 108.

110. After generating the control signal, the arithmetic unit updates the control signal based on the imaging result of imaging the first target object held by the holding device during movement and the second target object by the imaging system and the selection model based on the control signal. The control device according to claim 108 or 109.

111. The arithmetic unit repeatedly updates the control signal based on each imaging result of repeatedly imaging the first target object held by the holding device and the second target object by the imaging system until at least one of the positional relationship and the attitude relationship between the first target object and the second target object that enables the processing is reached, and the selection model. The control device according to claim 110.

112. The processing includes processing of the first target object held by the holding device with respect to the second target object during movement, and the 11th imaging result is the result of imaging the second target object during movement by the imaging system. The control device according to any one of claims 108 to 111.

113. Taking the target object as the first target object, the processing of the first target object includes the processing of the first target object held by the holding device for a second target object different from the first target object. The second imaging result is the result of imaging the first target object held by the holding device and the second target object with the imaging system. After the arithmetic unit holds the target object with the holding device by controlling at least one of the robot and the holding device, the arithmetic unit repeatedly generates the control signal based on each imaging result including the second imaging result obtained by repeatedly imaging the first target object held by the holding device and the second target object with the imaging system, and the selection model. The control device according to any one of claims 68 to 112.

114. The processing includes the processing of the first target object held by the holding device for the second target object during movement. The imaging result is the result of imaging the first target object held by the holding device during movement and the second target object during movement with the imaging system based on the control signal. The control device according to claim 113.

115. A plurality of different registered models registered in advance as candidates for the model are associated in advance with a plurality of different holding position postures. The arithmetic unit generates a display signal for displaying at least one registered model among the plurality of different registered models associated with each other and at least one holding position posture among the plurality of different holding position postures on a display device. The communication device outputs the display signal to the display device. The control device according to claim 70.

116. The display signal includes a signal for displaying on the display device at least one of the registered models among the plurality of different registered models associated with one holding position posture among the plurality of different holding position postures and the one holding position posture. The control device according to claim 115.

117. The display signal includes a signal for displaying on the display device information indicating the association between the one holding position posture and the at least one registered model associated with each other. The control device according to claim 116.

118. The display signal includes a signal for displaying on the display device at least one registered model associated with the one holding position and posture, and a priority for performing arithmetic processing by the arithmetic device for selecting the selected model given to the at least one registered model. The control device according to claim 116 or 117.

119. The arithmetic device generates at least one of the first registered model and the second registered model based on a user instruction. The control device according to claim 75.

120. At least one of the first registered model and the second registered model is a contour model showing a part of the contour of the target object. The arithmetic device generates a display signal for displaying an input screen on which the user instruction is input on the display device. The communication device outputs the display signal to the display device. The user makes the instruction by designating at least one of a part of the contour of the target object to be used as at least a part of the contour model and a part of the contour of the target object not to be used as at least a part of the contour model on the input screen displayed on the display device based on the display signal. The arithmetic device generates the contour model based on the designation. The control device according to claim 119.

121. The input screen includes information regarding at least one of a part of the contour of the target object to be used as at least a part of the contour model and a part of the contour of the target object not to be used as at least a part of the contour model. The control device according to claim 120.

122. A plurality of the target objects are placed on the placement device. The first imaging result is a result of imaging, by the imaging system, a group of target objects including at least a part of the plurality of target objects placed on the placement device. The control device according to claim 87 or 88.

123. Each of the plurality of target objects is irregularly placed on the placement device. The control device according to claim 122.

124. The imaging system is provided on the robot. The robot moves the holding device and the imaging system. The control device according to any one of claims 68 to 123. A control system comprising the control device according to any one of claims 68 to 123 and the imaging system. A robot system comprising the control device according to any one of claims 68 to 123, the imaging system, and the robot. A robot capable of holding a target object and provided with a holding device for moving the holding device, and a control method for generating a control signal for controlling at least one of the holding devices, wherein at least one of the robot and the holding device is controlled to select, as a selection model, a model showing a part of the target object based on a seventh imaging result obtained by imaging the target object held by the holding device with an imaging system; and generating the control signal for controlling at least one of the robot and the holding device for processing the target object held by the holding device based on a second imaging result obtained by imaging the target object held by the holding device with the imaging system and the selection model. A computer program for causing a computer to execute the control method according to claim 127.

Images