Control device, control system, robot system, control method, and computer program

Abstract

The control device generates a first control signal for controlling the robot so that the holding device approaches a first object, which is one object in the first object group, to hold the first object, on the basis of an imaging result obtained by imaging the first object group accommodated in the container from a first imaging height by the imaging system, and controls the robot so that the holding device holds the first object on the basis of the imaging result. The control device causes the holding device to approach the first object on the basis of the first control signal, and after the first object held by the holding device is released outside the container, the control device generates a second control signal on the basis of an imaging result obtained by imaging a second object group accommodated in the container from a second imaging height lower than the first imaging height by the imaging system. And a second control signal for controlling the robot so that the holding device approaches a second object, which is one object in the second object group, and holds the second object.

Description

Technical Field

[0001] This invention relates, for example, to the technical field of control devices, control systems, robot systems, control methods, and computer programs capable of generating control signals for controlling robots. Background Technology

[0002] Patent Document 1 describes an example of a control device that controls a robot equipped with a processing device capable of handling the object based on the results of an image capture system photographing an object. In such a control device, the image capture system is required to properly capture the object.

[0003] Patent Document 1: U.S. Patent Application Publication No. 2013 / 0230235 Summary of the Invention

[0004] According to a first method, a control device is provided that generates a control signal for controlling a robot. The robot is equipped with a holding device for holding an object and a shooting system for photographing the object, and the holding device and the shooting system are moved. The control device includes: a computing unit for generating the control signal; and a communication device for outputting the control signal generated by the computing unit. The computing unit generates a first control signal based on a photographing result obtained by the shooting system from a first shooting height of a first group of object objects, including at least a portion of a plurality of object objects contained in a container. The first control signal is used to control the robot, causing the holding device to approach one of the object objects in the first object group, i.e., the first object object, to hold the first object object. Based on the first control signal causing the holding device to approach the first object object, the first object object held by the holding device is removed from the container. After being released outside the container by the holding device, a second control signal is generated based on the imaging result obtained by the imaging system from a second imaging height lower than the first imaging height, which includes at least a portion of the first object group contained in the container, of the second object group. This second control signal is used to control the robot, causing the holding device to approach one of the object objects in the second object group, i.e., the second object object, to hold the second object object.

[0005] According to the second method, a control system is provided, including the control device provided by the first method and the shooting system.

[0006] According to the third method, a robot system is provided, including a control device provided by the first method, the shooting system, and the robot.

[0007] According to the fourth method, a control method is provided, which is a control method for generating control signals for controlling a robot. The robot is equipped with a holding device for holding an object and a shooting system for shooting the object, and moves the holding device and the shooting system. The control method includes: generating a first control signal as the control signal based on the shooting result obtained by the shooting system shooting at a first object group, including at least a portion of a plurality of object objects stored in a container, from a first shooting height; the first control signal is used to control the robot, causing the holding device to approach one of the object objects in the first object group, i.e., the first object, to hold the first object; and based on the first control signal, causing the holding device to approach the first object, the first object held by the holding device is removed from the container and released by the holding device outside the container; and generating a second control signal based on the shooting result obtained by the shooting system shooting at a second object group, including at least a portion of the first object group stored in the container, from a second shooting height lower than the first shooting height. The control signal is used to control the robot so that the holding device approaches one of the object objects in the second object group, i.e., the second object object, in order to hold the second object object.

[0008] According to the fifth method, a computer program is provided that causes a computer to execute the control method provided by the fourth method.

[0009] According to the sixth method, a control device is provided that generates a control signal for controlling a robot. The robot is equipped with a processing device capable of processing object objects and a photographing system capable of photographing the object objects, and moves the processing device and the photographing system. The control device includes: a computing device for generating the control signal; and a communication device for outputting the control signal generated by the computing device. The computing device generates a first control signal as the control signal based on the photographing result of the photographing system photographing a first group of object objects, including at least a portion of the object objects contained in a container. The first control signal is used to control the robot, causing the photographing system to move to a photographing height for photographing the object objects. Based on the photographing result obtained by the photographing system photographing a second group of object objects, including at least a portion of the object objects contained in the first group of object objects, at the photographing height, the second control signal is generated as the control signal. The second control signal is used to control the robot, causing the processing device to approach one of the object objects in the second group of object objects, i.e., the first object object, to perform the processing on the first object object.

[0010] According to the seventh method, a control system is provided, including the control device provided by the sixth method and the shooting system.

[0011] According to the eighth method, a robot system is provided, including a control device provided by the sixth method, the shooting system, and the robot.

[0012] According to the ninth method, a control method is provided, which is a control signal generation method for controlling a robot. The robot is equipped with a processing device capable of processing object objects and a shooting system capable of shooting the object objects, and moves the processing device and the shooting system. The control method includes: generating a first control signal as the control signal based on the shooting system's shooting result of a first object group including at least a portion of a plurality of object objects stored in a container, the first control signal being used to control the robot, causing the shooting system to move to a shooting height for shooting the object objects; and generating a second control signal as the control signal based on the shooting result obtained by the shooting system shooting a second object group including at least a portion of the object objects stored in the first object group at the shooting height, the second control signal being used to control the robot, causing the processing device to approach one of the object objects in the second object group, i.e., the first object object, to perform the processing on the first object object.

[0013] According to the 10th method, a computer program is provided that causes a computer to execute the control method provided by the 9th method.

[0014] According to the 11th method, a control device is provided that generates a control signal for controlling a robot. The robot is equipped with a holding device for holding an object and a shooting system for photographing the object, and moves the holding device and the shooting system. The control device includes: a computing device for generating the control signal; and a communication device for outputting the control signal generated by the computing device. One of the object objects, i.e., a first object object, is held by the holding device among a plurality of object objects contained in a container. The robot is controlled such that the holding device holding the first object object moves from the container to a position outside the container where the first object object is released. The computing device generates a first control signal based on the photographing result obtained by the shooting system photographing a group of first object objects, including at least a portion of the plurality of object objects contained in the container, along the movement path of the shooting system generated as the holding device moves from the container to the position outside the container where the first object object is released. The first control signal is used to control the robot from the position after the holding device releases the first object object.

[0015] According to the 12th method, a control system is provided, including the control device provided by the 11th method and the shooting system.

[0016] According to the 13th method, a robot system is provided, including a control device provided by the 11th method, the shooting system, and the robot.

[0017] According to the 14th method, a control method is provided, which is a control signal for controlling a robot. The robot is provided with a holding device for holding an object and a shooting system for photographing the object, and the holding device and the shooting system are moved. The control method includes: one of the object objects, i.e., a first object object, stored in a container is held by the holding device; the robot is controlled to move the holding device holding the first object object from the container to a position outside the container where the first object object is released; the control method includes: generating a first control signal as the control signal based on the shooting system photographing a group of first object objects, including at least a portion of the multiple object objects stored in the container, along the movement path of the shooting system generated as the holding device moves from the container to the position outside the container where the first object object is released, and the first control signal being used to control the robot from the position after the holding device releases the first object object.

[0018] According to the 15th method, a computer program is provided that causes a computer to execute the control method provided by the 14th method.

[0019] According to the 16th method, a control device is provided that generates a control signal for controlling a robot. The robot is equipped with a holding device for holding an object and a shooting system for photographing the object, and moves the holding device and the shooting system. The control device includes: a computing device for generating the control signal; and a communication device for outputting the control signal generated by the computing device. During the period from when one of the multiple object objects contained in a container, namely a first object object, is held by the holding device until the first object object held by the holding device is removed from the container and released by the holding device outside the container, the computing device generates a first control signal as the control signal based on the photographing result obtained by the shooting system photographing the interior of the container. The first control signal is used to control the robot after the holding device releases the first object object.

[0020] According to the 17th method, a control system is provided, including the control device provided by the 16th method and the shooting system.

[0021] According to the 18th method, a robot system is provided, including a control device provided by the 16th method, the shooting system, and the robot.

[0022] According to the 19th method, a control method is provided to generate a control signal for controlling a robot, the robot having a holding device for holding an object and a shooting system for shooting the object, and moving the holding device and the shooting system. The control method includes: during the period from when one of the object objects (i.e., a first object object) in a plurality of object objects stored in a container is held by the holding device until the first object object held by the holding device is removed from the container and released by the holding device outside the container, generating a first control signal as the control signal based on the shooting result obtained by the shooting system shooting the interior of the container, the first control signal being used to control the robot after the holding device releases the first object object.

[0023] According to the 20th method, a computer program is provided that causes a computer to execute the control method provided by the 19th method.

[0024] According to the 21st method, a control device is provided that generates a control signal for controlling a robot. The robot is equipped with a processing device capable of processing object objects and a photographing system capable of photographing the object objects, and moves the processing device and the photographing system. The control device includes: a computing device for generating the control signal; and a communication device for outputting the control signal generated by the computing device. The computing device generates a first control signal as the control signal based on at least one of the height of a first object group including at least a portion of a plurality of object objects contained in a container and at least one component of the first object group contained in the container. The first control signal is used to control the robot so that the photographing system moves to a photographing height for photographing the object objects. Based on the photographing result obtained by the photographing system photographing a second object group including at least a portion of the object objects contained in the first object group at the photographing height, a second control signal is generated as the control signal. The second control signal is used to control the robot so that the processing device approaches one of the object objects in the second object group to perform the processing on the object object.

[0025] According to the 22nd method, a control system is provided, including the control device provided by the 21st method and the shooting system.

[0026] According to the 23rd method, a robot system is provided, including a control device provided by the 21st method, the shooting system, and the robot.

[0027] According to the 24th method, a control method is provided to generate a control signal for controlling a robot. The robot is equipped with a holding device for holding an object and a shooting system for photographing the object. The method moves the holding device and the shooting system. The control method includes generating a first control signal as the control signal based on the photographing result obtained by the shooting system photographing the interior of the container during the period from when one of the object objects (i.e., a first object object) in a plurality of object objects stored in a container is held by the holding device until the first object object held by the holding device is removed from the container and released by the holding device outside the container. The first control signal is used to control the robot after the holding device releases the first object object.

[0028] According to the 25th method, a computer program is provided that causes a computer to perform the control method provided by the 24th method.

[0029] The effects and other benefits of the present invention will become clear from the embodiments described below. Attached Figure Description

[0030] Figure 1 This is a block diagram illustrating the structure of the robot system in this embodiment. Figure 2 This is a side view showing the appearance of the robot in this embodiment. Figure 3 This is a block diagram illustrating the structure of the control device in this embodiment. Figure 4 It is a flowchart representing the robot control and processing flow. Figure 5 (a) to Figure 5 (d) are side views showing the positional relationship between the robot and the workpiece at a certain moment during the holding process of the workpiece held by the mounting device for holding the workpiece moving on the support surface. Figure 6 (a) to Figure 6 (d) are side views showing the positional relationship between the robot and the workpiece at a certain moment during the release process for positioning the workpiece onto a mounting device that moves on a support surface. Figure 7 (a) to Figure 7 (b) are side views showing the positional relationship between the robot and the workpiece at a certain moment during the holding process of the workpiece held by the mounting device for holding it stationary on the support surface. Figure 8 (c) to Figure 8(e) are side views showing the positional relationship between the robot and the workpiece at a certain moment during the release process for positioning the workpiece on a mounting device that is stationary on the support surface. Figure 8 (a) to Figure 8 (e) are side views showing the positional relationship between the robot and the workpieces at a certain moment during a holding process for holding multiple workpieces placed on the mounting device one by one and a release process for placing multiple workpieces one by one on the mounting device. Figure 9 (a) to Figure 9 (e) are side views showing the positional relationship between the robot and the workpieces at a certain moment during a holding process for holding multiple workpieces placed on the mounting device one by one and a release process for placing multiple workpieces one by one on the mounting device. Figure 10 It is a cross-sectional view representing multiple objects contained in the container and the imaging system. Figure 11 This is a cross-sectional view representing a first example of the permissible shooting range set based on the field of view of the shooting system. Figure 12 This is a cross-sectional view representing a second example of the permissible shooting range set based on the field of view of the shooting system. Figure 13 (a) shows the field of view of the left camera and the relationship between part of the field of view and the image generated by the left camera. Figure 13 (b) shows the field of view of the right camera and the relationship between part of the field of view and the image generated by the right camera. Figure 14 (a) represents a photographing device that photographs an object included within a photographing permissible range determined based on a search area, and an image generated by the photographing device. Figure 14 (b) refers to a photographing device that photographs an object that is not included in the photographing range determined based on the search area, and the image generated by the photographing device. Figure 15 It is a cross-sectional view representing the height of a group of objects, including multiple objects contained within the container. Figure 16 (a) and Figure 16 (b) are cross-sectional views of an example of a method for calculating the height of a group of objects. Figure 17 It is a cross-sectional view representing multiple objects contained in the container and the imaging system. Figure 18 (a) and Figure 18(b) are cross-sectional views of the multiple objects contained in the container and the shooting system, respectively. Figure 19 It is a cross-sectional view representing multiple objects contained in the container and the imaging system. Figure 20 (a) and Figure 20 (b) are cross-sectional views of the multiple objects contained in the container and the shooting system, respectively. Figure 21 (a) and Figure 21 (b) are cross-sectional views of the multiple objects contained in the container and the shooting system, respectively. Figure 22 (a) and Figure 22 (b) are cross-sectional views of the multiple objects contained in the container and the shooting system, respectively. Figure 23 It is a cross-sectional view representing multiple objects contained in the container and the imaging system. Figure 24 This is a flowchart illustrating the robot control process in the first variation. Figure 25 (a) to Figure 25 (c) are cross-sectional views of the imaging system 2, which takes pictures of the objects stored in the container after the end effector 4 holds the object. Figure 26 (a) and Figure 26 (b) are cross-sectional views of the multiple objects contained in the container and the shooting system, respectively. Figure 27 (a) and Figure 27 (b) are cross-sectional views of the multiple objects contained in the container and the shooting system, respectively. Figure 28 (a) and Figure 28 (b) are cross-sectional views of the multiple objects contained in the container and the shooting system, respectively. Figure 29 It is a cross-sectional view representing multiple objects contained in the container and the imaging system. Figure 30 It is a cross-sectional view of a camera system that takes pictures of an object. Figure 31 This is a cross-sectional view of a weight sensor used to measure the weight of an object. Figure 32 This is a cross-sectional view showing a container equipped with an object detection sensor for detecting objects. Figure 33 This is a cross-sectional view showing a container with labels configured. Figure 34 (a) and Figure 34 (b) are cross-sectional views that conceptually represent the process of adjusting the shooting horizontal position of the shooting system. Figure 35 (a) and Figure 35 (b) are cross-sectional views that conceptually represent the process of adjusting the shooting posture of the shooting system. Figure 36 This is a side view showing the camera system located in a different part from the robot. Figure 37 (a) and Figure 37 (b) are cross-sectional views of the multiple objects contained in the container and the shooting system, respectively. Figure 38 This is a block diagram representing a shooting system equipped with a focusing and adjusting optical system. Figure 39 It is a block diagram representing the structure of a robot system with a measurement system. Detailed Implementation

[0031] Next, the implementation methods of the control device, control system, robot system, control method, and computer program will be described. Hereinafter, the implementation methods of the control device, control system, robot system, control method, and computer program will be described using the robot system SYS.

[0032] (1) Structure of the robot system SYS First, the structure of the robot system SYS will be explained.

[0033] (1-1) Overall structure of robot system SYS First, refer to Figure 1 The overall structure of the robot system SYS is explained. Figure 1 This is a block diagram representing the overall structure of the robot system SYS.

[0034] like Figure 1 As shown, the robot system SYS includes a robot 1, a camera system 2, a control device 3, and an end effector 4. Furthermore, the camera system 2 can also be referred to as a camera unit.

[0035] Robot 1 is a device capable of performing prescribed processing on object OBJ. Figure 2 This represents an example of robot 1. Figure 2 This is a side view showing the appearance of robot 1. (e.g.) Figure 2 As shown, robot 1 includes, for example, a base 11, a robotic arm 12, and a robot control device 13.

[0036] The base 11 is a component that forms the foundation of the robot 1. The base 11 is disposed on a supporting surface S, such as the ground. The base 11 may be fixed to the supporting surface S. Alternatively, the base 11 may be movable relative to the supporting surface S. For example, the base 11 may also be able to move independently on the supporting surface S. In this case, the base 11 may also be mounted on an automated guided vehicle (AGV). When the base 11 is mounted on an AGV, the AGV can also be considered part of the robot 1. Alternatively, the AGV can also be used as the base 11. Furthermore, Figure 2 This illustrates an example where the base 11 is fixed to the support surface S.

[0037] Robotic arm 12 is mounted on base 11. Robotic arm 12 is a device that connects multiple links 121 via joints 122. An actuator is built into the joint 122. The links 121 can rotate about an axis defined by the joint 122 via the actuator built into the joint 122. Additionally, at least one link 121 can extend or retract along the direction in which the link 121 extends. Alternatively, the device comprising multiple links 121 connected via joints 122 and the base 11 can also be referred to as robotic arm 12.

[0038] An end effector 4 is mounted on the robotic arm 12. That is, an end effector 4 is mounted on the robot 1. Figure 2 In the example shown, the end effector 4 is mounted on the front end of the robotic arm 12. The end effector 4 is movable by the movement of the robotic arm 12. That is, the robotic arm 12 moves the end effector 4. In other words, the robot 1 moves the end effector 4.

[0039] The end effector 4 is a device that performs prescribed processing (in other words, prescribed actions) on the object OBJ. The end effector 4 that performs prescribed processing on the object OBJ can also be called a processing device.

[0040] For example, as an example of a specified process, the end effector 4 can perform a holding process for holding the object OBJ. In this case, the end effector 4 can be considered as performing a holding process on the object OBJ that the end effector 4 should hold. The end effector 4 capable of performing the holding process can also be called a holding device.

[0041] For example, holding the object OBJ may include clamping the object OBJ. Holding the object OBJ may include using a mechanical gripper (described later as an example of end effector 4) to clamp the object OBJ. Holding the object OBJ may include adsorbing the object OBJ. Holding the object OBJ may include using a vacuum gripper (described later as an example of end effector 4) to adsorb (vacuum adsorption) the object OBJ. Holding the object OBJ may include using a magnetic gripper (described later as an example of end effector 4) to adsorb the object OBJ.

[0042] For example, as an example of a specified process, the end effector 4 can perform a release process (in other words, a release action) on the object OBJ that it holds (i.e., detach). In this case, the end effector 4 can be regarded as performing a release process on the object OBJ held by the end effector 4. The end effector 4 capable of performing the release process can also be called a release device.

[0043] As an example of an end effector 4 capable of performing holding and releasing operations, a mechanical gripper can be cited. A mechanical gripper is an end effector 4 capable of physically clamping an object OBJ using multiple (e.g., 2, 3, or 4) finger or claw components to hold the object OBJ. As another example of an end effector 4 capable of performing holding and releasing operations, a vacuum gripper can be cited. A vacuum gripper is an end effector 4 capable of holding an object OBJ by vacuum suction. As another example of an end effector 4 capable of performing holding and releasing operations, a magnetic gripper can be cited. Figure 2 This example shows the end effector 4 as a mechanical gripper. However, the end effector 4, capable of performing holding and releasing operations, can also be other existing end effectors.

[0044] As an example of the specified process, robot 1 can use end effector 4, which is capable of holding and releasing, to perform a configuration process (in other words, a configuration action) for positioning object OBJ in a desired position. For example, robot 1 can use end effector 4 to hold a first object OBJ, and then perform a configuration process for positioning the first object OBJ held by end effector 4 in a desired position of a second object OBJ, which is different from the first object OBJ. In this case, end effector 4 can be considered as performing a release process on the second object OBJ that end effector 4 is configuring the first object OBJ. Similarly, end effector 4 can be considered as performing a release process on the first object OBJ that end effector 4 should release.

[0045] As a concrete example of configuration processing (in other words, configuration action), robot 1 can use an end effector 4 capable of holding and releasing to perform an embedding process (in other words, embedding action) for embedding a first object OBJ into a second object OBJ that is different from the first object OBJ. For example, robot 1 can use the end effector 4 to hold the first object OBJ and then perform an embedding process for embedding the first object OBJ held by the end effector 4 into a second object OBJ that is different from the first object OBJ. In this case, the end effector 4 can be regarded as releasing the second object OBJ into which the end effector 4 intends to embed the first object OBJ. Similarly, the end effector 4 can be regarded as releasing the first object OBJ that the end effector 4 should release. In addition, the embedding process can also be referred to as a processing process.

[0046] As an example, the embedding process can include a process for embedding (i.e., inserting) a first object OBJ into a hole formed in a second object OBJ. For example, robot 1 can use an end effector 4 to hold the first object OBJ, and then perform an embedding process for embedding (i.e., inserting) the first object OBJ held by the end effector 4 into a hole formed in the second object OBJ. In this case, the end effector 4 can be considered as performing a release process on the second object OBJ with the hole in which the end effector 4 is to embed (i.e. insert) the first object OBJ. Similarly, the end effector 4 can be considered as performing a release process on the first object OBJ that the end effector 4 should release.

[0047] As a concrete example of configuration processing (in other words, configuration action), robot 1 can use an end effector 4 capable of holding and releasing to perform a pasting process (in other words, pasting action) for pasting a first object OBJ to a second object OBJ that is different from the first object OBJ. For example, robot 1 can use the end effector 4 to hold the first object OBJ and then perform a pasting process for pasting the first object OBJ held by the end effector 4 to the second object OBJ that is different from the first object OBJ. In this case, the end effector 4 can be regarded as releasing the second object OBJ to which the end effector 4 is to paste the first object OBJ. Similarly, the end effector 4 can be regarded as releasing the first object OBJ that the end effector 4 should release. In addition, pasting processing can also be called processing.

[0048] As a specific example of configuration processing (in other words, configuration action), robot 1 can use an end effector 4 capable of holding and releasing to perform a bonding process (in other words, bonding action) for attaching a first object OBJ to a second object OBJ that is different from the first object OBJ. For example, robot 1 can use the end effector 4 to hold the first object OBJ and then perform a bonding process for attaching the first object OBJ held by the end effector 4 to the second object OBJ that is different from the first object OBJ. In this case, the end effector 4 can be regarded as releasing the second object OBJ to which the end effector 4 is to attach the first object OBJ. Similarly, the end effector 4 can be regarded as releasing the first object OBJ that the end effector 4 should release. Furthermore, the bonding process can be referred to as a machining process.

[0049] As a specific example of configuration processing (in other words, configuration action), robot 1 can use an end effector 4 capable of holding and releasing to perform a welding process (in other words, welding action) for welding a first object OBJ to a second object OBJ that is different from the first object OBJ. For example, robot 1 can use the end effector 4 to hold the first object OBJ and then perform a welding process for welding the first object OBJ held by the end effector 4 to the second object OBJ that is different from the first object OBJ. In this case, the end effector 4 can be regarded as releasing the second object OBJ to which the end effector 4 is to weld the first object OBJ. Similarly, the end effector 4 can be regarded as releasing the first object OBJ that the end effector 4 should release. Furthermore, the welding process can be referred to as a machining process.

[0050] As a specific example of configuration processing (in other words, configuration action), robot 1 can use an end effector 4 capable of holding and releasing to perform a screw-tightening process (in other words, screw-tightening action) for screwing a first object OBJ, which functions as a screw, into a screw hole formed in a second object OBJ that is different from the first object OBJ. For example, robot 1 can use the end effector 4 to hold the first object OBJ and then perform a screw-tightening process for screwing the first object OBJ held by the end effector 4 into the second object OBJ that is different from the first object OBJ. In this case, the end effector 4 can be regarded as releasing the second object OBJ that the end effector 4 is tightening on the first object OBJ. Similarly, the end effector 4 can be regarded as releasing the first object OBJ that the end effector 4 should release. In addition, the screw-tightening process can also be referred to as a machining process.

[0051] As a specific example of configuration processing (in other words, configuration action), robot 1 can use end effector 4, which is capable of holding and releasing, to perform processing for discarding object OBJ (in other words, discarding action). For example, robot 1 can use end effector 4 to hold object OBJ, and then perform discarding processing to release (i.e., configure) the object OBJ held by end effector 4 to a discarding location for discarding object OBJ. In this case, end effector 4 can be regarded as performing a release processing on object OBJ to be discarded by end effector 4.

[0052] For example, the end effector 4 can perform predefined processing on each of multiple object OBJs. That is, the end effector 4 can perform predefined processing on multiple object OBJs sequentially. In this case, the robot 1 can move the end effector 4 to a first position where it can perform predefined processing on the first object OBJ. After moving to the first position, the end effector 4 performs predefined processing on the first object OBJ. Then, the robot 1 can move the end effector 4 to a second position where it can perform predefined processing on a second object OBJ that is different from the first object OBJ. After moving to the second position, the end effector 4 performs predefined processing on the second object OBJ.

[0053] For example, the end effector 4 can perform predefined processing on each of multiple parts of a single object OBJ. That is, the end effector 4 can sequentially perform predefined processing on multiple parts of a single object OBJ. In this case, the robot 1 can move the end effector 4 to a third position where the end effector 4 can perform predefined processing on the first part of the object OBJ. After moving to the third position, the end effector 4 performs predefined processing on the first part of the object OBJ. Then, the robot 1 can move the end effector 4 to a fourth position where the end effector 4 can perform predefined processing on the second part of the object OBJ. After moving to the fourth position, the end effector 4 performs predefined processing on the second part of the object OBJ.

[0054] As an example of an action that performs a prescribed process on each of multiple parts of a single object OBJ, robot 1 can repeatedly perform an embedding process on a second object OBJ having multiple holes, embedding the first object OBJ into each hole. Furthermore, a portion of the object OBJ that is subjected to prescribed processing by the end effector 4 can be referred to as an object part or object component. For example, each part (component) in the second object OBJ having each hole can be referred to as an object part or object component. Furthermore, when the object OBJ is composed of multiple parts, each part can be referred to as an object part or object component. When the object OBJ is composed of multiple components, each component can be considered an object part or object component.

[0055] like Figure 2 As shown, the object OBJ subjected to prescribed processing by the end effector 4 may contain a workpiece W. The workpiece W may include, for example, components or parts used to manufacture the desired product. The workpiece W may include, for example, components or parts processed for manufacturing the desired product. The workpiece W may include, for example, components or parts transported for manufacturing the desired product. The workpiece W may contain, for example, components or parts that move by transport for manufacturing the desired product. The workpiece W may contain, for example, components or parts that move for manufacturing the desired product.

[0056] like Figure 2 As shown, the object OBJ subjected to specified processing by the end effector 4 may include a mounting device T for holding the workpiece W. As an example of the mounting device T, a container (storage box) CB can be listed. The container CB may be a mounting device T comprising a bottom wall BS and a side wall SS protruding upward from the bottom wall BS, in which the workpiece W is placed on the bottom wall BS within a storage space SP surrounded by the bottom wall BS and the side wall SS. The container CB may also be a mounting device T comprising a bottom wall BS and a side wall SS protruding upward from the bottom wall BS, capable of storing the workpiece W within a storage space SP surrounded by the bottom wall BS and the side wall SS. However, the container CB may also lack the side wall SS. A container CB without the side wall SS may be called a tray. Furthermore, the mounting device T may be a tray. Furthermore, the mounting device T is not limited to the container CB or the tray; it may also be any existing object capable of holding the workpiece W. Additionally, the mounting device T may also be called a mounting member. The mounting device T may also be disposed on a support surface S. The mounting device T may also be fixed to the support surface S. Alternatively, at least a portion of the mounting device T may also be movable relative to the support surface S.

[0057] As a first example, where at least a part of the loading device T can move relative to the support surface S, an example can be given where the loading device T is supported by a conveyor that allows the loading device T to move (in other words, is transferable). As a first example of a conveyor, a conveyor capable of moving independently on the support surface S can be given. In this case, the loading device T can be loaded onto or held by the self-moving conveyor. Furthermore, the self-moving conveyor can be referred to as an Automatic Guided Vehicle (AGV). As a second example of a conveyor, a belt conveyor can be given. In this case, the loading device T can be loaded onto a conveyor belt that is part of the belt conveyor and is movable relative to the support surface S. As a third example of a conveyor, a flying device capable of flying on the support surface S can be given. In this case, the loading device T can be loaded onto or held by the flying conveyor. Furthermore, the flying conveyor can be referred to as an unmanned aerial vehicle (UAV) or a drone. As a fourth example of a conveying device, other robotic arms, different from robotic arm 12, may be considered, which are equipped with an end effector capable of holding the loading device T. In this case, the loading device T can be held by an end effector mounted on the other robotic arm.

[0058] As a second example of a movable loading device T, at least a portion thereof can move relative to the support surface S. An example where the loading device T itself can move relative to the support surface S can be given. As a first example of a movable loading device T, a loading device T capable of moving autonomously on the support surface S can be given. Furthermore, a autonomously moving loading device T can be referred to as an Automatic Guided Vehicle (AGV). As a second example of a movable loading device T, a belt conveyor can be given. In this case, the conveyor belt, which is part of the belt conveyor and carries the workpiece W, can move relative to the support surface S. As a third example of a movable loading device T, a loading device T capable of flying on the support surface S can be given. In this case, the loading device T can move relative to the support surface S by flying on it. Furthermore, a flying loading device T can be referred to as an unmanned aerial vehicle (UAV) or drone. As a fourth example of a movable loading device T, another robotic arm, different from robotic arm 12, can be given, equipped with an end effector capable of holding the workpiece W. In this case, the end effector mounted on another robotic arm can move relative to the support surface S.

[0059] When the conveying device moves the mounting device T, the workpiece W mounted on the mounting device T also moves relative to the supporting surface S. Therefore, a conveying device capable of moving the mounting device T can be considered equivalent to a conveying device capable of conveying the workpiece W. Similarly, when the mounting device T moves, the workpiece W mounted on the mounting device T also moves relative to the supporting surface S. Therefore, a movable mounting device T can be considered equivalent to a moving device capable of moving the workpiece W (in other words, a conveying device capable of conveying the workpiece W). Furthermore, Figure 2 An example is shown where the mounting device T can move independently on the support surface S.

[0060] Furthermore, if at least a portion of the mounting device T is movable relative to the support surface S, the first mounting device T on which the workpiece W is mounted can be mounted on a second mounting device T that is movable relative to the support surface S. In this case, the device comprising the first mounting device T and the second mounting device T can be referred to as the mounting device T.

[0061] However, the object OBJ may not include the mounting device T. Furthermore, the workpiece W may not be mounted on the mounting device T, or may not have a mounting device T at all. For example, the workpiece W may be mounted on the support surface S.

[0062] When the object OBJ includes a workpiece W and a mounting device T, the above-described holding process may include holding the workpiece W mounted on the mounting device T, whether it is stationary or moving. The above-described holding process may also include holding the workpiece W mounted on the support surface S. The above-described release process may include releasing the workpiece W held by the end effector 4 to position the workpiece W held by the end effector 4 at a desired position on the mounting device T, whether it is stationary or moving. The above-described release process may include releasing the workpiece W held by the end effector 4 to position the workpiece W held by the end effector 4 at a desired position on the support surface S. The above-described release process may include releasing a first workpiece W held by the end effector 4 to embed the first workpiece W held by the end effector 4 into a second workpiece W mounted on the mounting device T, whether it is stationary or moving. The above-described release process may include releasing a first workpiece W held by the end effector 4 to embed the first workpiece W held by the end effector 4 into a second workpiece W mounted on the support surface S. The aforementioned release process may include releasing the first workpiece W held by the end effector 4 to insert (i.e., insert) the first workpiece W held by the end effector 4 into a hole formed in the second workpiece W placed on the mounting device T, which is either stationary or moving. The aforementioned release process may also include releasing the first workpiece W held by the end effector 4 to insert (i.e., insert) the first workpiece W held by the end effector 4 into a hole formed in the second workpiece W placed on the support surface S. Furthermore, the object OBJ may include either the workpiece W or the mounting device T.

[0063] The robot control device 13 controls the actions of the robot 1.

[0064] Specifically, the robot control device 13 can control the movements of the robotic arm 12. For example, the robot control device 13 can control the movements of the robotic arm 12 to cause the desired link 121 to rotate about an axis specified by the desired joint 122. For example, the robot control device 13 can control the movements of the robotic arm 12 to cause the end effector 4 mounted on the robotic arm 12 to be in a desired position. For example, the robot control device 13 can control the movements of the robotic arm 12 to cause the end effector 4 mounted on the robotic arm 12 to move to a desired position.

[0065] In addition to controlling the actions of the robot 1, the robot control device 13 can also control the actions of the end effector 4 mounted on the robot 1. For example, the robot control device 13 can control the actions of the end effector 4 to hold the object OBJ at a desired timing. That is, the robot control device 13 can control the actions of the end effector 4 to perform holding operations at a desired timing. For example, the robot control device 13 can control the actions of the end effector 4 to release the held object OBJ at a desired timing. That is, the robot control device 13 can control the actions of the end effector 4 to perform release operations at a desired timing. When the end effector 4 is a mechanical gripper, the robot control device 13 can also control the timing of the opening and closing of the mechanical gripper. When the end effector 4 is a vacuum gripper, the robot control device 13 can also control the timing of turning the vacuum device of the vacuum gripper on / off. When the end effector 4 is a magnetic gripper, the robot control unit 13 can also control the timing of turning the magnetic device of the magnetic gripper on / off.

[0066] in addition, Figure 2 This example illustrates that robot 1 is a robotic arm 12 (i.e., a vertically articulated robot). However, robot 1 can also be a robot different from a vertically articulated robot. For example, robot 1 can also be a SCARA robot (i.e., a horizontally articulated robot). For example, robot 1 can also be a parallel linkage robot. For example, robot 1 can also be a dual-arm robot with two robotic arms 12. For example, robot 1 can also be an orthogonal coordinate robot. For example, robot 1 can also be a cylindrical coordinate robot. Robot 1 can also be referred to as a movable device. In addition to robot 1, the movable device can also include at least one of an automated guided vehicle (AGV) and an unmanned aerial vehicle (UAV). For example, robot 1 can also be disposed in at least one of an AAV and an UAV.

[0067] Back Figure 1 The imaging system 2 captures images of the object OBJ. To capture the object OBJ, the imaging system 2 includes an imaging device 21, an imaging device 22, and a projection device 23. Alternatively, the imaging system 2 can also be referred to as an imaging unit.

[0068] The imaging device 21 is a camera capable of capturing images of the object OBJ. In this embodiment, the imaging device 21 is a monocular camera. The imaging device 21 generates image data IMG_2D by capturing images of the object OBJ. That is, the imaging device 21 generates the image data IMG_2D of the object OBJ. The image data IMG_2D generated by the imaging device 21 is output from the imaging device 21 to the control device 3. As a result, the control device 3 acquires the image data IMG_2D obtained by the imaging device 21 capturing images of the object OBJ. In this embodiment, the imaging device 21 is a monocular camera. Specifically, the imaging device 21 can use a monocular camera (in other words, an imaging element) to capture images of the object OBJ. Specifically, since the imaging device 21 is a monocular camera, the imaging device 21 generates one image data generated by the monocular camera as image data IMG_2D. In this case, the image data IMG_2D, which is equivalent to one image data (i.e., representing an image), can be called monocular image data. In addition, the imaging device 21 is not limited to a monocular camera. The imaging device 21 can be a stereo camera capable of capturing images of the object OBJ using two monocular cameras, or it can include three or more monocular cameras. Alternatively, the imaging device 21 can be at least one of a light field camera, a pre-positioning camera, and a multispectral camera.

[0069] Similar to the imaging device 21, the imaging device 22 is a camera capable of capturing images of the object OBJ. In this embodiment, the imaging device 22 is a stereo camera. Specifically, the imaging device 22 is a stereo camera capable of capturing images of the object OBJ using two monocular cameras (in other words, two imaging elements). The imaging device 22 generates image data IMG_3D by capturing images of the object OBJ. That is, the imaging device 22 generates the image data IMG_3D of the object OBJ. Specifically, since the imaging device 22 is a stereo camera, the imaging device 22 generates image data IMG_3D that includes two image data generated by the two monocular cameras respectively. In this case, the image data IMG_3D containing two image data (i.e., representing two images) can be referred to as stereo image data. The image data IMG_3D generated by the imaging device 22 is output from the imaging device 22 to the control device 3. As a result, the control device 3 acquires the image data IMG_3D obtained by the imaging device 22 capturing images of the object OBJ. In addition, the imaging device 22 is not limited to a stereo camera. The imaging device 22 can be a monocular camera, or it can include three or more monocular cameras. Alternatively, the imaging device 22 can also be at least one of a light field camera, a pre-positioning camera, and a multispectral camera.

[0070] Projection device 23 is a device capable of irradiating projection light onto object OBJ. Specifically, projection device 23 is a device capable of projecting a desired projection pattern onto object OBJ by irradiating projection light onto object OBJ. Furthermore, the projection pattern can also be referred to as the intensity distribution of the light projected onto object OBJ. If the intensity distribution of the projected light changes, the projection pattern will also change. The desired projection pattern can include, for example, a random pattern. A random pattern can be a projection pattern where each unit illumination area has a different pattern. A random pattern can also include a random dot pattern. The desired projection pattern can include, for example, a one-dimensional or two-dimensional grid pattern. The desired projection pattern can include, for example, a linear pattern. The desired projection pattern can include, for example, a striped pattern. The desired projection pattern can include other projection patterns. Imaging device 22 captures an image of object OBJ onto which the projection pattern from projection device 23 has been projected. In this case, the image showing the image data IMG_3D captures an image of object OBJ onto which the projection pattern has been projected. On the other hand, imaging device 21 may not capture an image of object OBJ onto which the projection pattern has been projected. The imaging device 21 can also capture images of object OBJ that does not have a projected pattern. In this case, the object OBJ with the projected pattern may not be captured in the image shown in image data IMG_2D. Alternatively, the object OBJ without a projected pattern may be captured in the image shown in image data IMG_2D. Furthermore, the projection light used to project the desired projection pattern onto the object OBJ can be called pattern light or structured light. In this case, the projection light may include pattern light or structured light. Additionally, the projection light may be light with a uniform projection pattern (i.e., a uniform intensity distribution).

[0071] Alternatively, the projection device 23 can also be considered as illuminating the object OBJ by projecting projection light onto it. In this case, the projection device 23 can function as an illumination device for illuminating the object OBJ. When the projection device 23 functions as an illumination device, the projection light can also be called illumination light. The projection light (illumination light) used as an illumination device can also be light with a uniform intensity distribution. When the projection device 23 functions as an illumination device, the projection light may not be light capable of projecting the desired projection pattern onto the object OBJ. When the projection device 23 functions as an illumination device, the projection light (i.e., illumination light) can be any light capable of illuminating the object OBJ.

[0072] The imaging device 21 can photograph the entire object OBJ. Alternatively, the imaging device 21 can photograph a portion of the object OBJ. That is, the imaging device 21 can photograph a portion of the object OBJ without photographing the other parts of the object OBJ. Similarly, the imaging device 22 can photograph the entire object OBJ. Alternatively, the imaging device 22 can photograph a portion of the object OBJ. That is, the imaging device 22 can photograph a portion of the object OBJ without photographing the other parts of the object OBJ.

[0073] The imaging device 21 can capture a single object OBJ. That is, a single object OBJ can be captured in the image shown in the image data IMG_2D. Alternatively, the imaging device 21 can capture multiple object OBJs. That is, multiple object OBJs can be captured in the image shown in the image data IMG_2D. In this case, as detailed later, the control device 3 can decide (in other words, select) one of the multiple object OBJs captured by the imaging device 21 as the object OBJ that the end effector 4 actually performs the prescribed processing on. Furthermore, the object OBJ that is actually performed on the prescribed processing by the end effector 4 among the multiple object OBJs captured by the imaging device 21 can be referred to as the processing execution object.

[0074] The imaging device 22 can capture images of a single object OBJ. That is, a single object OBJ can be captured in the image shown in the image data IMG_3D. Alternatively, the imaging device 22 can capture images of multiple object OBJs. That is, multiple object OBJs can be captured in the image shown in the image data IMG_3D. In this case, as detailed later, the control device 3 can decide (in other words, select) one of the multiple object OBJs captured by the imaging device 22 as the object OBJ that the end effector 4 will actually perform the prescribed processing on. Furthermore, the object OBJ that is actually performed on the prescribed processing by the end effector 4 among the multiple object OBJs captured by the imaging device 22 can be referred to as the processing execution object.

[0075] When imaging devices 21 and 22 respectively capture multiple object OBJs, the multiple object OBJs captured by imaging device 21 can be the same as the multiple object OBJs captured by imaging device 22. That is, the multiple object OBJs captured in the image shown in image data IMG_2D can be the same as the multiple object OBJs captured in the image shown in image data IMG_3D. Alternatively, at least one of the multiple object OBJs captured by imaging device 21 can be different from at least one of the multiple object OBJs captured by imaging device 22. That is, at least one of the multiple object OBJs captured in the image shown in image data IMG_2D can be the same as at least one of the multiple object OBJs captured in the image shown in image data IMG_3D.

[0076] Multiple object objects OBJ captured by at least one of the imaging devices 21 and 22 can be configured such that at least two of the multiple object objects OBJ at least partially overlap. As an example, in the case where the object objects OBJ are workpieces W equivalent to components used to manufacture a desired product, multiple workpieces W (i.e., multiple components) can be configured such that at least two of the multiple workpieces W at least partially overlap. In this case, robot 1 can perform bulk picking, selecting workpieces W one by one from a plurality of randomly placed workpieces W.

[0077] The imaging system 2 is mounted on the robotic arm 12 in the same manner as the end effector 4. That is, the imaging devices 21 and 22 and the projection device 23 are mounted on the robotic arm 12. For example, as... Figure 2 As shown, the imaging devices 21 and 22 and the projection device 23 can also be mounted on the front end of the robotic arm 12 in the same manner as the end effector 4. In this case, the imaging devices 21 and 22 and the projection device 23 can be moved by the movement of the robotic arm 12. That is, the robotic arm 12 moves the imaging devices 21 and 22 and the projection device 23.

[0078] However, as detailed later in the sixth variation, the imaging system 2 may not be mounted on the robotic arm 12. The imaging system 2 can be mounted at any location capable of projecting light onto the object OBJ and capturing images of the object OBJ. In this case, for example, the imaging system 2 can be mounted on a structure such as a column, so that it can project light onto the object OBJ and capture images of the object OBJ. Alternatively, at least one of the imaging devices 21, 22, and 23 may be mounted on the robotic arm 12, while at least one of the other two imaging devices 21, 22, and 23 may be mounted in a different location from the robotic arm 12 (e.g., a structure such as a column). If at least one of the imaging devices 21 and 22 is located in a different location from the robotic arm 12, at least one of the imaging devices 21 and 22 may also be mounted on a structure such as a column configured to capture images of the object OBJ. Furthermore, if the projection device 23 is installed in a different location than the robotic arm 12, the projection device 23 may also be installed on a structure such as a column that is configured to project projection light onto the object OBJ.

[0079] The imaging device 21 can capture images of the object OBJ during the period of relative displacement between the imaging device 21 and the object OBJ. Furthermore, the state of relative displacement between the imaging device 21 and the object OBJ can represent the state of change in their relative positional relationship. The state of relative displacement between the imaging device 21 and the object OBJ can also represent the state of relative movement between them. For example, the state of phase displacement between the imaging device 21 and the object OBJ can include the state of movement of the object OBJ relative to the imaging device 21. In this case, the imaging device 21 does not need to remain stationary to capture images of the object OBJ; therefore, the robot system SYS can efficiently perform prescribed processing on the object OBJ using the end effector 4.

[0080] Alternatively, the imaging device 21 can capture images of the object OBJ during a period when there is no relative displacement between the imaging device 21 and the object OBJ. Furthermore, the state of no relative displacement between the imaging device 21 and the object OBJ can represent a state where the relative positional relationship between the imaging device 21 and the object OBJ remains unchanged. The state of no relative displacement between the imaging device 21 and the object OBJ can also represent a state where there is no relative movement between the imaging device 21 and the object OBJ. The state of no relative displacement between the imaging device 21 and the object OBJ can also represent a state where the imaging device 21 and the object OBJ are stationary. The state of no relative displacement between the imaging device 21 and the object OBJ can also represent a state where the imaging device 21 and the object OBJ are moving at the same speed in the same direction.

[0081] The imaging device 22 can capture images of the object OBJ during the period of relative displacement between the imaging device 22 and the object OBJ. Furthermore, the state of relative displacement between the imaging device 22 and the object OBJ can represent the change in their relative positional relationship. The state of relative displacement between the imaging device 22 and the object OBJ can also represent the state of relative movement between them. For example, the state of phase displacement between the imaging device 22 and the object OBJ can include the state of movement of the object OBJ relative to the imaging device 22. For example, the state of phase displacement between the imaging device 22 and the object OBJ can include the state of movement of the imaging device 22 relative to the object OBJ. In this case, the imaging device 22 does not need to remain stationary to capture images of the object OBJ; therefore, the robot system SYS can efficiently perform prescribed processing on the object OBJ using the end effector 4.

[0082] Alternatively, the imaging device 22 can capture images of the object OBJ during a period when there is no relative displacement between the imaging device 22 and the object OBJ. Furthermore, the state where there is no relative displacement between the imaging device 22 and the object OBJ can mean that the relative positional relationship between the imaging device 22 and the object OBJ remains unchanged. The state where there is no relative displacement between the imaging device 22 and the object OBJ can also mean that there is no relative movement between the imaging device 22 and the object OBJ. The state where there is no relative displacement between the imaging device 22 and the object OBJ can also mean that the imaging device 22 and the object OBJ are stationary. The state where there is no relative displacement between the imaging device 22 and the object OBJ can also mean that the imaging device 22 and the object OBJ are moving at the same speed in the same direction.

[0083] The imaging devices 21 and 22 can also photograph the object OBJ synchronously. For example, imaging devices 21 and 22 can photograph the object OBJ simultaneously. That is, imaging devices 21 and 22 can photograph the object OBJ such that the 2D shooting time of imaging device 21 photographing the object OBJ is the same as the 3D shooting time of imaging device 22 photographing the object OBJ. Imaging devices 21 and 22 can photograph the object OBJ such that the 2D shooting time of imaging device 21 photographing the object OBJ to generate image data IMG_2D is the same as the 3D shooting time of imaging device 22 photographing the object OBJ to generate image data IMG_3D.

[0084] The shooting devices 21 and 22 can also capture images of the object OBJ under the control of the control device 3. In this case, the control device 3 can control the timing (in other words, the timing) at which the shooting devices 21 and 22 capture images of the object OBJ. For example, the control device 3 can control the shooting devices 21 and 22 so that they capture images of the object OBJ synchronously. For example, the control device 3 can control the shooting devices 21 and 22 so that they capture images of the object OBJ simultaneously. That is, the control device 3 can control the shooting devices 21 and 22 so that the 2D shooting time and the 3D shooting time are the same.

[0085] Here, the state of "the 2D shooting time and the 3D shooting time are the same" can include the state of "the 2D shooting time and the 3D shooting time are literally exactly the same". The state of "the 2D shooting time and the 3D shooting time are the same" can also include the state of "although the 2D shooting time and the 3D shooting time are not exactly the same, because the time deviation between the 2D shooting time and the 3D shooting time is smaller than the first allowable upper limit value, they are considered to be substantially the same". Here, the first allowable upper limit value can also be a first allowable upper limit value based on the control error of the robotic arm 12. For example, there are cases where, due to the time deviation between the 2D shooting time and the 3D shooting time, an error occurs in the calculation result of at least one of the position and orientation of the object OBJ (described later) (i.e., the accuracy of at least one of the calculated position and orientation of the object OBJ is reduced). In this case, due to the error generated in the calculation result of at least one of the position and orientation of the object OBJ, a control error of the robotic arm 12 may sometimes occur. The control error of the robotic arm 12 becomes the movement error of the end effector 4, which sometimes prevents the end effector 4 from properly processing the object OBJ. The first permissible upper limit can be set to an appropriate value that avoids a situation where the end effector 4 cannot properly process the object OBJ due to this control error of the robotic arm 12. Furthermore, the first permissible upper limit can also be considered equivalent to the permissible upper limit of the movement error of the end effector 4 caused by the robotic arm 12. Moreover, for example, even if there is a time difference between the 2D and 3D shooting times due to the synchronization error of the shooting processes of the shooting devices 21 and 22, it can be considered that the 2D and 3D shooting times are essentially the same. Additionally, the synchronization error of the shooting processes of the shooting devices 21 and 22 can also be the synchronization control error of the shooting processes of the shooting devices 21 and 22 performed by the control device 3.

[0086] However, the imaging devices 21 and 22 may not simultaneously image the object OBJ. That is, the imaging devices 21 and 22 can image the object OBJ such that the 2D imaging time of the imaging device 21 and the 3D imaging time of the imaging device 22 are different times. Furthermore, the state where "the 2D imaging time and the 3D imaging time are different times" can also include the state where "the time difference between the 2D imaging time and the 3D imaging time is greater than the first allowable upper limit value, therefore they cannot be considered to be substantially the same time."

[0087] In this embodiment, when the imaging devices 21 and 22 are not relatively displaced from the object OBJ, they can still capture images of the object OBJ, making the 2D and 3D shooting times the same. In other words, when the imaging devices 21 and 22 are capturing images of the object OBJ during periods of relative displacement, the control device 3 can control the imaging devices 21 and 22 to make the 2D and 3D shooting times the same.

[0088] On the other hand, when the shooting devices 21 and 22 are shooting the object OBJ during a period when there is no relative displacement between them and the object OBJ, the shooting devices 21 and 22 may not shoot the object OBJ in a way that makes the 2D shooting time and the 3D shooting time the same. That is, the control device 3 may not control the shooting devices 21 and 22 to make the 2D shooting time and the 3D shooting time the same. For example, the shooting devices 21 and 22 may shoot the object OBJ so that the 2D shooting time and the 3D shooting time are different. In other words, the control device 3 may control the shooting devices 21 and 22 to make the 2D shooting time and the 3D shooting time different. However, when the shooting devices 21 and 22 are shooting the object OBJ during a period when there is no relative displacement between them and the object OBJ, the shooting devices 21 and 22 may also shoot the object OBJ so that the 2D shooting time and the 3D shooting time are the same. That is, the control device 3 can control the shooting devices 21 and 22 so that the 2D shooting time and the 3D shooting time are the same.

[0089] The control device 3 performs robot control processing. This robot control processing may include generating robot control signals for controlling the robot 1. Specifically, the control device 3 generates robot control signals based on at least one of the image data IMG_2D and IMG_3D output from the imaging system 2. In this embodiment, the control device 3 calculates at least one of the position and orientation of the object OBJ within the global coordinate system of the robot system SYS based on at least one of the image data IMG_2D and IMG_3D, and generates robot control signals based on at least one of the calculated position and orientation of the object OBJ.

[0090] The global coordinate system is the coordinate system that serves as the reference for the robot system SYS. For example, the global coordinate system can also be the coordinate system that serves as the reference for robot 1. Furthermore, it can be said that the global coordinate system is the coordinate system used for controlling robot 1. As a global coordinate system, for example, a world coordinate system defined with the support surface S on which robot 1 is configured can be used. That is, as a global coordinate system, a world coordinate system fixed relative to the support surface S on which robot 1 is configured can be used.

[0091] However, the control device 3 can calculate at least one of the position and orientation of the object OBJ in a coordinate system different from the global coordinate system based on at least one of the image data IMG_2D and IMG_3D.

[0092] As a first example, the control device 3 can calculate at least one of the position and orientation of the object OBJ in the robot coordinate system based on at least one of the image data IMG_2D and IMG_3D. The robot coordinate system can be a coordinate system determined with respect to the robot 1. That is, the robot coordinate system can be a coordinate system fixed relative to the robot 1 (e.g., fixed relative to the base 11 of the robot 1). Furthermore, since it is a coordinate system determined with respect to the robot 1, the robot coordinate system can be referred to as the global coordinate system. Moreover, it can be said that the robot coordinate system is a coordinate system used for controlling the robot 1.

[0093] As a second example, the control device 3 can calculate at least one of the position and orientation of the object OBJ in the 2D shooting coordinate system based on at least one of the image data IMG_2D and IMG_3D. The 2D shooting coordinate system can be a coordinate system determined with respect to the shooting device 21. That is, the 2D shooting coordinate system can be a coordinate system fixed relative to the shooting device 21. As an example of a 2D shooting coordinate system, an example based on the optical axis AX21 (refer to...) of the optical system (especially terminal optical elements such as objective lenses) of the shooting device 21 can be given. Figure 2 The coordinate system is determined by this. As an example of a 2D shooting coordinate system, any one of the three coordinate axes constituting the 2D shooting coordinate system can be taken as the optical axis AX21 along the optical system (especially terminal optical elements such as objective lenses) provided by the shooting device 21 (refer to...). Figure 2 The coordinate system of the axes.

[0094] As a third example, the control device 3 can calculate at least one of the position and orientation of the object OBJ in the 3D shooting coordinate system based on at least one of the image data IMG_2D and IMG_3D. The 3D shooting coordinate system can be a coordinate system determined with respect to the shooting device 22. That is, the 3D shooting coordinate system can be a coordinate system fixed relative to the shooting device 22. As an example of a 3D shooting coordinate system, an example based on the optical axis AX22 (refer to...) of the optical system (especially terminal optical elements such as objective lenses) of the shooting device 22 can be given. Figure 2 The coordinate system is determined by this. As an example of a 3D shooting coordinate system, any one of the three coordinate axes constituting the 3D shooting coordinate system can be taken as the optical axis AX22 along the optical system (especially terminal optical elements such as objective lenses) provided by the shooting device 22 (refer to...). Figure 2 The coordinate system of the axes.

[0095] In addition to performing robot control processing, the control device 3 can also perform end effector control processing. End effector control processing may include generating end effector control signals for controlling the end effector 4. Specifically, the control device 3 can generate the end effector control signals based on at least one of the calculated position and orientation of the object OBJ.

[0096] Furthermore, the end effector control processing may or may not be included in the robot control processing. That is, the end effector control signal generated by the control device 3 may or may not be included in the robot control signal. In the following description, for ease of explanation, an example in which the end effector control processing is included in the robot control processing (i.e., the end effector control signal is included in the robot control signal) will be described. Therefore, in the following description, robot control processing may also mean the processing of generating at least one of robot control signal and end effector control signal. In addition, in the following description, robot control signal may also mean at least one of signal for controlling robot 1 and signal for controlling end effector 4. In addition, robot control signal may also be simply referred to as control signal.

[0097] Therefore, the control device 3 and the imaging system 2 can be used to control the robot 1. Thus, the system including the control device 3 and the imaging system 2 can be referred to as a robot control system or a control system.

[0098] The robot control signal generated by control device 3 is output to robot control device 13 of robot 1. Robot control device 13 controls the actions of robot 1 based on the robot control signal generated by control device 3. Therefore, the robot control signal may also include signals for controlling the actions of robot 1.

[0099] As described above, when the robot control signal includes a signal for controlling the robotic arm 12, the robot control device 13 can also control the robotic arm 12 based on the robot control signal. For example, the robot control device 13 can control the movement of the robotic arm 12 by controlling the movement of the actuator built into the joint 122 based on the robot control signal.

[0100] For example, as described above, the robotic arm 12 moves the end effector 4. In this case, the robot control signal may include a signal for controlling the robotic arm 12 to position the end effector 4 in a desired location. The robot control signal may include a signal for controlling the robotic arm 12 to move the end effector 4 to the desired location. The robot control signal may include a signal for controlling the robotic arm 12 to achieve the desired positional relationship between the end effector 4 and the object OBJ. In this case, the robot control device 13 may control the robotic arm 12 based on the robot control signal to position the end effector 4 in the desired location. The robot control device 13 may control the robotic arm 12 based on the robot control signal to move the end effector 4 to the desired location. The robot control device 13 may control the robotic arm 12 based on the robot control signal to achieve the desired positional relationship between the end effector 4 and the object OBJ.

[0101] As an example, when the end effector 4 is performing a holding process to hold the object OBJ, the robot control signal may include a signal for controlling the robotic arm 12 to move (i.e., approach) the end effector 4 toward the holding position where the end effector 4 can hold the object OBJ. That is, the robot control signal may include a signal for controlling the robotic arm 12 to position the end effector 4 in the holding position. In this case, the robot control device 13 may control the robotic arm 12 based on the robot control signal to move (i.e., approach) the end effector 4 toward the holding position. In other words, the robot control signal may control the robotic arm 12 to position the end effector 4 in the holding position.

[0102] As another example, in the case of performing a release process for releasing the object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the robotic arm 12 to move (i.e., approach) the end effector 4 toward the release position where the object OBJ held by the end effector 4 is to be released. That is, the robot control signal may include a signal for controlling the robotic arm 12 to position the end effector 4 in the release position. In this case, the robot control device 13 may control the robotic arm 12 based on the robot control signal to move (i.e., approach) the end effector 4 toward the release position. In other words, the robot control signal may control the robotic arm 12 to position the end effector 4 in the release position.

[0103] As described above, when the robot control signal includes a signal for controlling the end effector 4, the robot control device 13 can also control the end effector 4 based on the robot control signal. For example, the robot control device 13 can also control the operation of the end effector 4 by controlling the operation of the actuator for moving the mechanical gripper constituting the end effector 4 based on the robot control signal. For example, the robot control device 13 can also control the operation of the vacuum device of the vacuum gripper constituting the end effector 4 based on the robot control signal, thereby controlling the operation of the end effector 4.

[0104] As an example, when the end effector 4 performs a holding process to hold the object OBJ, the robot control signal may include a signal for controlling the end effector 4 so that the end effector 4, located in the holding position, holds the object OBJ. In this case, the robot control device 13 can control the end effector 4 based on the robot control signal so that the end effector 4, located in the holding position, holds the object OBJ.

[0105] As another example, in the case of performing a release process for releasing the object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the end effector 4 to release the object OBJ held by the end effector 4 from the release position. In this case, the robot control device 13 can control the end effector 4 based on the robot control signal to release the object OBJ held by the end effector 4 from the release position.

[0106] The robot control signal may also include signals that can be directly used by the robot control device 13 to control the movement of the robot 1. The robot control signal may also include robot drive signals that can be directly used by the robot control device 13 to control the movement of the robot 1. In this case, the robot control device 13 can also directly use the robot control signal to control the movement of the robot 1. For example, the control device 3 can generate drive signals for actuators built into the joints 122 of the robotic arm 12 as robot control signals, and the robot control device 13 can directly use the robot control signals generated by the control device 3 to control the actuators built into the joints 122 of the robotic arm 12.

[0107] The robot control signal may also include signals that can be directly used by the robot control device 13 to control the movement of the end effector 4. The robot control signal may also include end effector drive signals that can be directly used by the robot control device 13 to control the movement of the end effector 4. In this case, for example, the control device 3 may generate a drive signal (end effector drive signal) for the actuator that moves the mechanical gripper constituting the end effector 4 as the robot control signal, and the robot control device 13 may directly use the robot control signal generated by the control device 3 to control the actuator of the end effector 4. For example, the control device 3 may generate a drive signal (end effector drive signal) for driving the vacuum device of the vacuum gripper constituting the end effector 4 as the robot control signal, and the robot control device 13 may directly use the robot control signal generated by the control device 3 to control the vacuum device of the end effector 4.

[0108] In addition, as described above, if the robot control signal includes a signal that can be directly used by the robot control device 13 to control the action of at least one of the robot 1 and the end effector 4, the robot 1 may not have the robot control device 13.

[0109] Alternatively, the robot control signal may also include a signal that enables the robot control device 13 to generate robot drive signals for controlling the movement of the robot 1. In this case, the robot control device 13 may also generate robot drive signals for controlling the movement of the robot 1 based on the robot control signal, and control the movement of the robot 1 based on the generated robot drive signals. For example, the robot control device 13 may generate robot drive signals for driving actuators built into the joints 122 of the robotic arm 12 based on the robot control signal, and control the actuators built into the joints 122 of the robotic arm 12 based on the generated robot drive signals.

[0110] The robot control signal may also include a signal that enables the robot control device 13 to generate an end effector drive signal for controlling the movement of the end effector 4. In this case, the robot control device 13 may also generate an end effector drive signal for controlling the movement of the end effector 4 based on the robot control signal, and control the movement of the end effector 4 based on the generated end effector drive signal. For example, if the end effector 4 is a mechanical gripper, the robot control device 13 may generate an end effector drive signal for driving the actuator of the mechanical gripper based on the robot control signal, and control the actuator of the mechanical gripper based on the generated end effector drive signal. For example, if the end effector 4 is a magnetic gripper, the robot control device 13 may generate an end effector drive signal for driving the magnetic device of the magnetic gripper based on the robot control signal, and control the magnetic device based on the generated end effector drive signal.

[0111] Furthermore, the signals that can be used by the robot control device 13 to generate robot drive signals may include at least one of the signals representing the position and orientation of the object OBJ in the global coordinate system. In this case, for example, the robot control device 13 may generate robot drive signals based on the robot control signals to drive the actuators built into the joints 122 of the robotic arm 12, so that the end effector 4 approaches the object OBJ whose position and orientation in the global coordinate system are calculated by the robot control signals (i.e., the positional relationship between the robot 1 (end effector 4) and the object OBJ becomes the desired positional relationship), and control the movement of the robotic arm 12 based on the generated robot drive signals.

[0112] The signals that can be used by the robot control device 13 to generate robot drive signals may include signals representing the desired positional relationship between the robot 1 and the object OBJ in the global coordinate system. In this case, for example, the robot control device 13 may generate robot drive signals for driving the actuators built into the joints 122 of the robotic arm 12 based on the robot control signals, so that the positional relationship between the robot 1 (end effector 4) and the object OBJ becomes the desired positional relationship represented by the robot control signals, and control the movement of the robotic arm 12 based on the generated robot drive signals.

[0113] Signals that can be used by the robot control device 13 to generate robot drive signals may include signals representing the desired position of the end effector 4 in the global coordinate system. In this case, for example, the robot control device 13 may generate robot drive signals based on the robot control signals to drive the actuators built into the joints 122 of the robotic arm 12, such that the end effector 4 is located at the desired position represented by the robot control signals (i.e., the positional relationship between the robot 1 (end effector 4) and the object OBJ becomes the desired positional relationship), and control the movement of the robotic arm 12 based on the generated robot drive signals. As an example of the desired position, a processing position where the end effector 4 should handle the object OBJ can be given. As a specific example of the desired position, a holding position where the end effector 4 should hold the object OBJ can be given. In this case, the robot control device 13 may generate robot drive signals based on the robot control signals to drive the actuators built into the joints 122 of the robotic arm 12, such that the end effector 4 moves to the holding position represented by the robot control signals, and control the movement of the robotic arm 12 based on the generated robot drive signals. As another example of the desired position, the release position where the end effector 4 should release the object OBJ can be cited. In this case, the robot control unit 13 can generate robot drive signals based on the robot control signals to drive the actuators built into the joints 122 of the robotic arm 12, so that the end effector 4 is in the release position indicated by the robot control signals, and control the movement of the robotic arm 12 based on the generated robot drive signals.

[0114] Furthermore, the signals that can be used by the robot control device 13 to generate robot drive signals may include, for example, signals indicating the desired position of the front end of the robotic arm 12 (e.g., the tool center point) in the global coordinate system, and signals indicating the desired position of the imaging system 2 in the global coordinate system. Additionally, the signals that can be used by the robot control device 13 to generate robot drive signals may be signals indicating the amount and direction of movement from the current position of the end effector 4 to the desired position of the end effector 4.

[0115] In addition, the coordinate system used as a reference in robot control signals can also be a coordinate system other than the global coordinate system (e.g., robot coordinate system, 2D shooting coordinate system or 3D shooting coordinate system).

[0116] (1-2) Structure of control device 3 Next, refer to Figure 3 Explain the structure of control device 3. Figure 3 This is a block diagram showing the structure of the control device 3.

[0117] like Figure 3As shown, the control device 3 includes a processing unit 31, a storage unit 32, and a communication unit 33. Furthermore, the control device 3 may include an input unit 34 and an output unit 35. Alternatively, the control device 3 may not include at least one of the input unit 34 and the output unit 35. The processing unit 31, storage unit 32, communication unit 33, input unit 34, and output unit 35 can be connected via a data bus 36.

[0118] The computing device 31 includes at least one of, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a FPGA (Field Programmable Gate Array). The computing device 31 reads a computer program. For example, the computing device 31 may also read a computer program stored in the storage device 32. For example, the computing device 31 may also use a recording medium reading device (not shown) provided with the control device 3 to read a computer program stored on a non-temporary recording medium that can be read by a computer. The computing device 31 can obtain (i.e., download or read) a computer program from a device (not shown) disposed outside the control device 3 via a communication device 33 (or other communication device). The computing device 31 executes the read computer program. As a result, a logical function module for performing the processing to be performed by the control device 3 (as an example, the robot control processing described above) is implemented within the computing device 31. That is, the computing device 31 can function as a controller for implementing the logical function module for performing the processing to be performed by the control device 3.

[0119] A computational model can be installed within the computing device 31, which can be constructed through machine learning when the computing device executes a computer program. As an example of a computational model that can be constructed through machine learning, a computational model including a neural network (so-called Artificial Intelligence (AI)) can be listed. In this case, the learning of the computational model can include the learning of the parameters of the neural network (e.g., at least one of weights and biases). The computing device 31 can also use the computational model to perform robot control processing. That is, the actions of performing robot control processing can also include actions performed using the computational model. Additionally, a computational model constructed using offline machine learning with teacher data can also be installed in the computing device 31. Furthermore, the computational model installed on the computing device 31 can also be updated on the computing device 31 through online machine learning. Alternatively, in addition to or as an alternative to the computational model installed on the computing device 31, the computing device 31 can also use a computational model installed on a device external to the computing device 31 (i.e., a device located external to the control device 3) to perform robot control processing.

[0120] Furthermore, the recording medium for the computer program executed by the arithmetic device 31 may be at least one of the following: CD-ROM, CD-R, CD-RW or floppy disk, MO, DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW and Blu-ray (registered trademark), magnetic media such as magnetic tape, optical disk, semiconductor memory such as USB memory, and any other medium capable of storing programs. The recording medium may include a device capable of recording a computer program (as an example, a general-purpose or special-purpose device that installs the computer program in a state capable of executing in at least one manner, such as software and firmware). In addition, the various processes or functions included in the computer program may be implemented by a logic processing module implemented within the arithmetic device 31 by executing the computer program, or by hardware such as a specified gate array (FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit)) provided by the arithmetic device 31, or by a combination of logic processing modules and partial hardware modules that implement some elements of the hardware.

[0121] Figure 3 An example of a logic function module implemented within the computing device 31 to perform robot control processing is shown. For example... Figure 3As shown, a three-dimensional position data generation unit 311, a position and attitude calculation unit 312, and a signal generation unit 313 are implemented within the computing device 31. Furthermore, the processing performed by each of the three-dimensional position data generation unit 311, the position and attitude calculation unit 312, and the signal generation unit 313 will be described later. Figure 4 Detailed explanations will be provided later, so they are omitted here. Additionally, the arithmetic unit 31 can also be referred to as the arithmetic section.

[0122] Storage device 32 is capable of storing desired data. For example, storage device 32 can also temporarily store computer programs executed by computing device 31. Storage device 32 can also temporarily store data temporarily used by computing device 31 while computing device 31 is executing computer programs. Storage device 32 can also store data that control device 3 will store permanently. In addition, storage device 32 may include at least one of RAM (Random Access Memory), ROM (Read Only Memory), hard disk drive, magneto-optical disk drive, SSD (Solid State Drive), and disk array drive. That is, storage device 32 may also include non-temporary recording media.

[0123] The communication device 33 can communicate with both the robot 1 and the imaging system 2 via a communication network (not shown). Alternatively, the communication device 33 can also communicate with other devices different from the robot 1 and the imaging system 2 via a communication network (not shown), in addition to or replacing at least one of the robot 1 and the imaging system 2. In this embodiment, the communication device 33 can also receive (i.e., acquire) at least one of image data IMG_2D and IMG_3D from the imaging system 2. Furthermore, the communication device 33 can also send (i.e., output) robot control signals to the robot 1. Moreover, the communication device 33 that outputs robot control signals to the robot 1 can be referred to as an output unit.

[0124] Input device 34 is a device that accepts input of information for control device 3 from outside the control device 3. For example, input device 34 may include user-operable operating devices of control device 3 (e.g., at least one of keyboard, mouse, and touch panel). For example, input device 34 may include a recording medium reading device capable of reading information recorded as data on a recording medium externally connected to control device 3.

[0125] Furthermore, information can be input as data from an external device to the control device 3 via the communication device 33. In this case, the communication device 33 can also function as an input device that receives information from an external source for the control device 3.

[0126] Output device 35 is a device for externally outputting information to control device 3. For example, output device 35 can output information as an image. That is, output device 35 may include a display device (so-called a monitor) capable of displaying images. For example, output device 35 can output information as sound. That is, output device 35 may include a sound device (so-called a speaker) capable of outputting sound. For example, output device 35 can output information onto paper. That is, output device 35 may include a printing device (so-called a printer) capable of printing desired information onto paper. For example, output device 35 can output information as data to a recording medium that can be externally connected to control device 3.

[0127] Furthermore, information can be output as data from the control device 3 to an external device of the control device 3 via the communication device 33. In this case, the communication device 33 can function as an output device for outputting information to the external device of the control device 3.

[0128] (2) Robot control processing Next, the robot control processing performed by control device 3 will be explained.

[0129] (2-1) Robot control and processing flow First, refer to Figure 4 This will illustrate the robot control and processing flow. Figure 4 It is a flowchart representing the robot control and processing flow.

[0130] like Figure 4As shown, the control device 3 acquires image data IMG_3D from the imaging device 22 using the communication device 33 (step S1). Specifically, the imaging device 22 captures the object OBJ at a predetermined 3D imaging rate. For example, the imaging device 22 can capture the object OBJ at a 3D imaging rate of tens to hundreds (for example, 500 times) per second. Furthermore, when the object OBJ is housed in the container CB, capturing the object OBJ can be considered equivalent to capturing the interior of the container CB (i.e., the aforementioned storage space SP). The image data resulting from capturing the object OBJ can be considered equivalent to the image data resulting from capturing the interior of the container CB (i.e., the aforementioned storage space SP). As a result, the imaging device 22 generates image data IMG_3D at a period corresponding to the predetermined 3D imaging rate. For example, the imaging device 22 can generate tens to hundreds (for example, 500 times) of image data IMG_3D per second. Whenever the imaging device 22 generates image data IMG_3D, the control device 3 acquires the image data IMG_3D. That is, the control device 3 can acquire dozens to hundreds (for example, 500) of image data IMG_3D per second.

[0131] However, the imaging device 22 may also periodically photograph the object OBJ without adhering to the prescribed 3D imaging rate. For example, the imaging device 22 may photograph the object OBJ even after receiving a control signal from the control device 3 to control the imaging device 22 to photograph the object OBJ.

[0132] In addition to the object OBJ, which is processed by robot 1, the imaging device 22 can also capture images of other objects different from the object OBJ. Furthermore, since these other objects are not the objects processed by robot 1, they will be referred to as non-object objects in the following description. Examples of non-object objects include at least a portion of the end effector 4, at least a portion of the robotic arm 12, and at least one object located around robot 1, i.e., a peripheral object. Examples of peripheral objects include at least one of the loading device T and the container CB. For example, when both the object OBJ and the non-object object are included within the field of view of the imaging device 22, the imaging device 22 can capture images of both the object OBJ and the non-object object. As a result, the imaging device 22 can generate image data IMG_3D representing images of both the object OBJ and the non-object object. However, the imaging device 22 can also capture images of the object OBJ without capturing images of the non-object object. That is, the imaging device 22 can generate image data IMG_3D representing images of the object OBJ but not of the non-object object. In either case, the imaging device 22 generates image data IMG_3D representing an image of at least the object OBJ. That is, the imaging device 22 generates image data IMG_3D that at least contains image data of the object OBJ.

[0133] As described above, the imaging device 22 sometimes images multiple object OBJs that are sequentially processed by the end effector 4. In this case, the end effector 4 may perform prescribed processing on the first object OBJ among the multiple object OBJs in a first period, and perform prescribed processing on the second object OBJ among the multiple object OBJs that is different from the first object OBJ in a second period after the first period. In this case, during the first period, the other object OBJs among the multiple object OBJs besides the first object OBJ that is processed by the end effector 4 can be regarded as object OBJs (or non-object objects) that are not processed by the robot 1 in the first period. Similarly, during the second period, the other object OBJs among the multiple object OBJs besides the second object OBJ that is processed by the end effector 4 can be regarded as object OBJs (or non-object objects) that are not processed by the robot 1 in the second period. Furthermore, the first object OBJ and the second object OBJ can each be a different object OBJ among the multiple object OBJs captured by the imaging device 22.

[0134] In addition to acquiring image data IMG_3D from the imaging device 22, or as an alternative, the control device 3 can also acquire image data IMG_2D from the imaging device 21 using the communication device 33 (step S1). Specifically, the imaging device 21 captures the object OBJ at a predetermined 2D imaging rate. The 2D imaging rate is the same as the 3D imaging rate. However, the 2D imaging rate can also be different from the 3D imaging rate. For example, the imaging device 21 can capture the object OBJ at a 2D imaging rate of tens to hundreds (for example, 500 times) per second. As a result, the imaging device 21 generates image data IMG_2D at a period corresponding to the predetermined 2D imaging rate. For example, the imaging device 21 can generate tens to hundreds (for example, 500 times) of image data IMG_2D per second. Whenever the imaging device 21 generates image data IMG_2D, the control device 3 acquires the image data IMG_2D. That is, the control device 3 can acquire tens to hundreds (500 as an example) of image data IMG_2D per second.

[0135] However, the imaging device 21 may also periodically photograph the object OBJ without the desired 2D imaging rate. For example, the imaging device 21 may photograph the object OBJ even after receiving a control signal from the control device 3 to control the imaging device 21 to photograph the object OBJ.

[0136] Additionally, image data IMG_2D can also be referred to as two-dimensional position data representing the two-dimensional position of the object OBJ captured in the image data IMG_2D. Image data IMG_2D (two-dimensional position data) can be data representing the two-dimensional position of at least a portion of the surface of the object OBJ. Specifically, image data IMG_2D (two-dimensional position data) is data representing the individual two-dimensional positions of multiple points on the object OBJ. For example, image data IMG_2D (two-dimensional position data) can be data representing the individual two-dimensional positions of multiple points on the surface of the object OBJ. For example, image data IMG_2D (two-dimensional position data) can be data representing the individual two-dimensional positions of multiple points respectively corresponding to multiple locations on the surface of the object OBJ.

[0137] In addition to the object OBJ, which is processed by robot 1, the imaging device 21 can also capture images of non-object objects different from the object OBJ. For example, if both the object OBJ and the non-object objects are included within the imaging range (field of view) of the imaging device 21, the imaging device 21 can capture images of both the object OBJ and the non-object objects. As a result, the imaging device 21 can generate image data IMG_2D representing images of both the object OBJ and the non-object objects. However, the imaging device 21 can also capture the object OBJ without capturing non-object objects. That is, the imaging device 21 can generate image data IMG_2D representing images of the object OBJ but not of non-object objects. In either case, the imaging device 21 generates image data IMG_2D representing images of at least the object OBJ. That is, the imaging device 21 generates image data IMG_2D that includes at least image data of the object OBJ.

[0138] As described above, the imaging device 21 sometimes images multiple object objects OBJ that are sequentially processed by the end effector 4. In this case, the end effector 4 may perform specified processing on the third object object OBJ among the multiple object objects OBJ in the third period, and perform specified processing on the fourth object object OBJ that is different from the third object object OBJ among the multiple object objects OBJ in the fourth period after the third period. In this case, during the third period, the other object objects OBJ among the multiple object objects OBJ besides the third object object OBJ that is processed by the end effector 4 can be regarded as object objects OBJ (or non-object objects) that are not processed by the robot 1 in the third period. Similarly, during the fourth period, the other object objects OBJ among the multiple object objects OBJ besides the fourth object object OBJ that is processed by the end effector 4 can be regarded as object objects OBJ (or non-object objects) that are not processed by the robot 1 in the fourth period. Furthermore, the third object object OBJ and the fourth object object OBJ can each be a different object object OBJ among the multiple object objects OBJ captured by the imaging device 21.

[0139] When the control device 3 acquires image data IMG_3D, the 3D position data generation unit 311 generates 3D position data WSD based on the acquired image data IMG_3D whenever the control device 3 acquires image data IMG_3D (step S2). Furthermore, the 3D position data generation unit 311 outputs the generated 3D position data WSD to the position and attitude calculation unit 312. However, if the control device 3 does not acquire image data IMG_3D (for example, if the control device 3 acquires image data IMG_2D), the 3D position data generation unit 311 may not generate 3D position data WSD. In this case, the control device 3 may not perform step S2.

[0140] Furthermore, the control device 3 can generate three-dimensional position data WSD based on each of the multiple image data IMG_3D generated by the imaging device 22. Alternatively, the control device 3 can generate three-dimensional position data WSD without using another portion of the multiple image data IMG_3D generated by the imaging device 22.

[0141] The three-dimensional position data (WSD) represents the three-dimensional position of the object OBJ captured in the image data IMG_3D. For example, the three-dimensional position data WSD may represent the three-dimensional position of at least a portion of the surface of the object OBJ. In particular, the three-dimensional position data WSD represents the three-dimensional position of each of multiple points of the object OBJ. That is, the three-dimensional position data WSD represents the three-dimensional position of each of multiple points of the object OBJ captured by the imaging device 22. For example, the three-dimensional position data WSD may represent the three-dimensional position of each of multiple points on the surface of the object OBJ. For example, the three-dimensional position data WSD may also represent the three-dimensional position of multiple points respectively corresponding to multiple parts of the surface of the object OBJ. Furthermore, in the following description, unless otherwise stated, the three-dimensional position of the object OBJ may represent at least one of the following: the three-dimensional position of at least a portion of the surface of the object OBJ, the three-dimensional position of each of multiple points of the object OBJ, the three-dimensional position of each of multiple points on the surface of the object OBJ, and the three-dimensional position of each of multiple points corresponding to multiple parts of the surface of the object OBJ.

[0142] Specifically, as described above, an object OBJ with a projected pattern is captured in the image shown in image data IMG_3D. In this case, the projected pattern captured in the image shown in image data IMG_3D reflects the three-dimensional shape of at least a portion of the surface of the object OBJ with the projected pattern. The shape of the projected pattern captured in the image shown in image data IMG_3D reflects the three-dimensional shape of at least a portion of the surface of the object OBJ with the projected pattern. Therefore, the three-dimensional position data generation unit 311 can calculate the three-dimensional shape of at least a portion of the surface of the object OBJ based on the projected pattern captured in the image shown in image data IMG_3D. The three-dimensional shape of at least a portion of the surface of the object OBJ substantially represents the three-dimensional position of each of a plurality of points of the object OBJ. This is because each of the plurality of points of the object OBJ is contained within the surface of the object OBJ. Therefore, the process of calculating the three-dimensional shape of at least a portion of the surface of the object OBJ can be considered substantially equivalent to the process of calculating the three-dimensional position of each of the plurality of points of the object OBJ. Therefore, the three-dimensional position data generation unit 311 can generate three-dimensional position data WSD based on image data IMG_3D.

[0143] Considering that each of the multiple points of object OBJ is contained on the surface of object OBJ, the three-dimensional position data WSD representing the three-dimensional position of each of the multiple points of object OBJ can be regarded as equivalent to the three-dimensional shape data representing the three-dimensional shape of at least a part of object OBJ (in particular, the three-dimensional shape of at least a part of the surface of object OBJ).

[0144] To generate 3D position data WSD, the 3D position data generation unit 311 can calculate disparity by mapping the portions (e.g., pixels) of the images shown in the two image data contained in the image data IMG_3D to each other. Specifically, in this mapping, the 3D position data generation unit 311 can calculate disparity by mapping the portions of the projected patterns captured in the images shown in the two image data (i.e., the portions of the projected patterns captured in each image to each other). The 3D position data generation unit 311 can calculate the 3D position of each of the multiple points of the object OBJ using a known method based on the triangulation principle using the calculated disparity. As a result, 3D position data WSD representing the 3D position of each of the multiple points of the object OBJ is generated. In this case, the accuracy of disparity calculation is higher when mapping the portions of the images captured with projected patterns (i.e., the portions of the projected patterns captured to each other) compared to mapping the portions of the images that were not captured with projected patterns. Therefore, the accuracy of the generated 3D position data WSD (i.e., the accuracy of the calculation of the 3D position of each point of the object OBJ) becomes higher.

[0145] The 3D position data (WSD) can be any data as long as it can represent the individual 3D positions of multiple points on the object OBJ. That is, the 3D position data (WSD) can be any data that directly or indirectly shows the individual 3D positions of multiple points on the object OBJ. For example, the 3D position data (WSD) can contain coordinate information representing the individual 3D positions of multiple points on the object OBJ. For example, the 3D position data (WSD) can contain information representing the 3D shape of at least a portion of the object OBJ.

[0146] As an example of three-dimensional position data (WSD), depth image data can be cited. Depth image data is image data that associates brightness information with each pixel of a depth image represented by depth image data, or alternatively, also associates depth information. Depth information represents the distance (i.e., depth) between each part of the object OBJ captured in each pixel and the imaging device 22. The distance (i.e., depth) between each part of the object OBJ captured in each pixel and the imaging device 22 can be calculated based on the parallax mentioned above. Alternatively, depth image data can also be image data where the brightness information of each pixel represents the depth of each part of the object OBJ (i.e., the distance between each part of the object OBJ and the imaging device 22). The three-dimensional position data generation unit 311 can also calculate the distance between each part of the object OBJ captured in the image shown in the image data IMG_3D and the imaging device 22 based on the projection pattern captured in the image shown in the image data IMG_3D, and generate a depth image by associating the calculated distance as depth information with each pixel of the image shown in the image data IMG_3D.

[0147] As another example of 3D position data WSD, point cloud data can be cited. Point cloud data is data representing the set of points in three-dimensional space corresponding to each part of an object OBJ captured in the image shown in image data IMG_3D. The 3D position data generation unit 311 can generate point cloud data based on depth image data and camera parameters of the imaging device 22. Furthermore, the following description will illustrate an example of using point cloud data as 3D position data WSD.

[0148] As described above, when the shooting range (field of view) of the shooting device 22 includes other objects different from the object OBJ besides the object being processed by the robot 1, the image data IMG_3D represents images of both the object OBJ and the other objects. In this case, the three-dimensional position data WSD can be data representing the three-dimensional position of the other objects captured in the image data IMG_3D, in addition to the three-dimensional position of the object OBJ captured in the image data IMG_3D. That is, the three-dimensional position data WSD can contain data representing the three-dimensional positions of multiple points of the other objects captured by the shooting device 22. Even in this case, as long as the three-dimensional position data WSD contains data representing the three-dimensional position of the object OBJ, the three-dimensional position data WSD can be regarded as data representing the three-dimensional positions of multiple points of the object OBJ. This is because even if the data representing the three-dimensional position of the other objects is included in the three-dimensional position data WSD, the data representing the three-dimensional position of the object OBJ is still included in the three-dimensional position data WSD.

[0149] Then, the position and pose calculation unit 312 calculates at least one of the position and pose of the object OBJ based on at least one of the image data IMG_2D acquired in step S1 and the three-dimensional position data WSD generated in step S2 (step S3). As a result, the position and pose calculation unit 312 generates position and pose data POI representing at least one of the position and pose of the object OBJ (step S3). The position and pose data POI may contain position data representing the position of the object OBJ. In addition to or as a substitute for position data representing the position of the object OBJ, the position and pose data POI may also contain pose data representing the pose of the object OBJ. That is, the position and pose data POI may contain at least one of position data and pose data.

[0150] Furthermore, the image data IMG_2D used to generate position and pose data POI is equivalent to the image capture result of the imaging device 21. Similarly, the three-dimensional position data WSD used to generate position and pose data POI is generated from the image data IMG_3D, which is equivalent to the image capture result of the imaging device 22. Therefore, the position and pose calculation unit 312 can be considered to calculate the position and pose data POI based on at least one of the image capture results of the imaging device 21 and the image capture results of the imaging device 22. In other words, the position and pose calculation unit 312 can be considered to calculate the position and pose data POI based on the image capture results of the imaging system 2. In addition, in the following description, the processing (action) based on image data IMG_2D can also be considered equivalent to the processing (action) based on the image capture results of the imaging device 21 (i.e., the image capture results of the imaging system 2). The processing (action) based on image data IMG_3D can also be considered equivalent to the processing (action) based on the image capture results of the imaging device 22 (i.e., the image capture results of the imaging system 2). The processing (action) based on the three-dimensional position data WSD generated from the image data IMG_3D can also be regarded as equivalent to the processing (action) based on the shooting results of the shooting device 22 (i.e., the shooting results of the shooting system 2).

[0151] In step S3, the position and attitude calculation unit 312 calculates at least one of the position and attitude of the object OBJ in the global coordinate system. That is, the position and attitude calculation unit 312 generates position and attitude data POI representing at least one of the position and attitude of the object OBJ in the global coordinate system. For example, the robot control device 13 can control the robotic arm 12 so that the end effector 4 is located at a desired position in the global coordinate system. The global coordinate system is defined by the mutually orthogonal X-axis (GL), Y-axis (GL), and Z-axis (GL). The X-axis (GL) can be an axis along the horizontal plane. The Y-axis (GL) can be an axis along the horizontal plane. The Z-axis (GL) can be an axis orthogonal to the horizontal plane. The Z-axis (GL) can also be an axis extending along the direction of gravity. Furthermore, Figure 2 The X-axis, Y-axis, and Z-axis shown can be the X-axis (GL), Y-axis (GL), and Z-axis (GL), respectively. Furthermore, the origin of the global coordinate system does not have to be... Figure 2 The origins of the X-axis (GL), Y-axis (GL), and Z-axis (GL) are shown. For example, the origin of the global coordinate system can be set at... Figure 2 The origin of the global coordinate system can be set at any position on the support surface S. For example, the origin of the global coordinate system can be set at any position on the contact surface between the base 11 and the support surface S (e.g., the center or centroid of the contact surface).

[0152] The position and attitude calculation unit 312 can calculate at least one of the following: the position Tx(GL) of the object OBJ in the X-axis direction (GL) parallel to the X-axis (GL); the position Ty(GL) of the object OBJ in the Y-axis direction (GL) parallel to the Y-axis (GL); and the position Tz(GL) of the object OBJ in the Z-axis direction (GL) parallel to the Z-axis (GL), to determine the position of the object OBJ in the global coordinate system. The position and attitude calculation unit 312 can also calculate at least one of the following: the rotation amount Rx(GL) of the object OBJ around the X-axis (GL); the rotation amount Rey(GL) of the object OBJ around the Y-axis (GL); and the rotation amount Rz(GL) of the object OBJ around the Z-axis (GL), to determine the attitude of the object OBJ in the global coordinate system. This is because the rotations of object OBJ around the X-axis (GL) Rx(GL), around the Y-axis (GL) Ry(GL), and around the Z-axis (GL) Rz(GL) are equivalent to the parameters representing the orientation of object OBJ around the X-axis (GL), around the Y-axis (GL), and around the Z-axis (GL), respectively. Therefore, in the following description, the rotations of object OBJ around the X-axis (GL) Rx(GL), around the Y-axis (GL) Ry(GL), and around the Z-axis (GL) Rz(GL) are referred to as the orientations of object OBJ around the X-axis (GL) Rx(GL), around the Y-axis (GL) Ry(GL), and around the Z-axis (GL) Rz(GL), respectively.

[0153] Furthermore, the poses Rx(GL) of object OBJ around the X-axis (GL), Ry(GL) of object OBJ around the Y-axis (GL), and Rz(GL) of object OBJ around the Z-axis (GL) can be considered as representing the positions of object OBJ in the rotational direction around the X-axis (GL), the rotational direction around the Y-axis (GL), and the rotational direction around the Z-axis (GL), respectively. That is, the poses Rx(GL) of object OBJ around the X-axis (GL), Ry(GL) of object OBJ around the Y-axis (GL), and Rz(GL) of object OBJ around the Z-axis (GL) can all be considered as parameters representing the position of object OBJ.

[0154] Therefore, in Figure 4 In step S3, the position and attitude calculation unit 312 can calculate at least one of position Tx (GL), position Ty (GL), position Tz (GL), attitude Rx (GL), attitude Ry (GL), and attitude Rz (GL) as at least one of the position and attitude of object OBJ in the global coordinate system.

[0155] Figure 4 In step S3, the position and pose calculation unit 312 can calculate at least one of the position and pose of the object OBJ by performing a matching process using at least one of the image data IMG_2D and the three-dimensional position data WSD. For ease of explanation, the following example will be used primarily: the position and pose calculation unit 312 calculates at least one of the position and pose of the object OBJ by performing a matching process using the three-dimensional position data WSD. Furthermore, in the following explanation, the matching process using the three-dimensional position data WSD will be referred to as 3D matching process.

[0156] Specifically, the position and attitude calculation unit 312 can perform 3D matching processing, which uses 3D position data WSD and 3D model data WMD representing a 3D model that serves as the reference for the object OBJ. In this case, the 3D matching processing can be viewed, for example, as a matching process using a point cloud represented by the 3D position data WSD and a 3D model represented by the 3D model data. Alternatively, the 3D matching processing itself can be the same as existing matching processes. For example, the position and attitude calculation unit 312 can also use a known method including at least one of RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), ICP (Iterative Closest Point), and DSO (Direct Sparse Odometry) to perform 3D matching processing.

[0157] A 3D model based on WSD (Web Surface Deposition Data) can include a point cloud model, which is a 3D model that uses the point cloud shown in the WSD to represent the 3D shape of an object OBJ. A 3D model based on WSD can include a polygonal model or a mesh model generated by combining multiple points constituting the point cloud shown in the WSD on faces (especially faces of polygons with each point as a vertex). A 3D model based on WSD can also include a solid model generated by combining multiple points constituting the point cloud shown in the WSD on a smooth surface.

[0158] The 3D model data WMD is data representing a template model TM3 that serves as a 3D model of the object OBJ. Specifically, the 3D model data WMD represents the template model TM3, which is a 3D model with a 3D shape that serves as a reference for the object OBJ. The template model can be a CAD model of the object OBJ. The template model can be a 3D model with the same shape as the actual object OBJ obtained by pre-measuring its 3D shape. The template model can be a polygonal model, a mesh model, or a solid model. The 3D model data WMD can be point cloud data representing the template model TM3 of the object OBJ. The 3D model data WMD can be polygonal model data, mesh model data, or solid model data generated from point cloud data. Furthermore, the actual object OBJ measured beforehand to generate the 3D model data WMD can be a reference or a prototype object OBJ.

[0159] In this case, the position and attitude calculation unit 312 can perform 3D matching processing (in other words, template matching processing) on ​​the three-dimensional position data WSD, which uses the template model TM3 shown in the three-dimensional model data WMD as a template. Specifically, the position and attitude calculation unit 312 can translate, enlarge, reduce and / or rotate the template model TM3 in the 3D shooting coordinate system so that the feature parts (e.g., at least one feature point and edge) of the template model TM3 shown in the three-dimensional model data WMD are close to (e.g., matched) the feature parts of the object OBJ (e.g., the point cloud corresponding to the object OBJ shown in the three-dimensional position data WSD, or the three-dimensional model (e.g., the point cloud model) reflecting the actual three-dimensional position of the object OBJ) represented by the three-dimensional position data WSD. That is, the position and attitude calculation unit 312 can change the positional relationship between the coordinate system of the 3D model data WMD (e.g., the coordinate system of the CAD model) and the 3D shooting coordinate system, so that the feature parts of the object OBJ represented by the 3D position data WSD (e.g., the point cloud corresponding to the object OBJ shown in the 3D position data WSD, or the 3D model reflecting the actual 3D position of the object OBJ (e.g., the point cloud model)) approach (e.g., match) the feature parts of the template model TM3. As a result, the position and attitude calculation unit 312 can determine the positional relationship between the coordinate system of the 3D model data WMD and the 3D shooting coordinate system. Then, based on the positional relationship between the coordinate system of the 3D model data WMD and the 3D shooting coordinate system, the position and attitude calculation unit 312 can calculate at least one of the position and attitude of the object OBJ in the 3D shooting coordinate system according to at least one of the position and attitude of the template model TM3 in the coordinate system of the 3D model data WMD.

[0160] Subsequently, the position and pose calculation unit 312, based on a transformation matrix used to convert the three-dimensional coordinates in either the robot coordinate system or the 3D imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system and the 3D imaging coordinate system, transforms at least one of the position and pose of the object OBJ in the 3D imaging coordinate system into at least one of the position and pose of the object OBJ in the robot coordinate system. When the robot coordinate system is used as the global coordinate system, at least one of the position and pose of the object OBJ in the robot coordinate system is equivalent to at least one of the position and pose of the object OBJ in the global coordinate system. However, when a coordinate system different from the robot coordinate system is used as the global coordinate system, the position and pose calculation unit 312 can use a transformation matrix used to convert the three-dimensional coordinates in either the robot coordinate system or the global coordinate system into three-dimensional coordinates in the other of the robot coordinate system and the global coordinate system, and calculate at least one of the position and pose of the object OBJ in the global coordinate system based on at least one of the position and pose of the object OBJ in the robot coordinate system.

[0161] When a feature part of the 3D model shown in the 3D model data WMD is brought close to a feature part of the object OBJ whose 3D position is represented by the 3D position data WSD, the position and pose calculation unit 312 can calculate a matching similarity. This matching similarity is the similarity between the 3D model of the object OBJ shown in the 3D model data WMD and the object OBJ whose 3D position is represented by the 3D position data WSD (e.g., a point cloud of the object OBJ). Furthermore, in the following description, the similarity between the 3D model of the object OBJ shown in the 3D model data WMD and the object OBJ whose 3D position is represented by the 3D position data WSD (e.g., a point cloud of the object OBJ), i.e., the matching similarity, can be simply referred to as the matching similarity of the object OBJ. The position and pose calculation unit 312 can translate, enlarge, reduce, and / or rotate the 3D model shown in the 3D model data WMD to maximize the matching similarity. Furthermore, the matching similarity can also be considered equivalent to the correlation degree representing the correlation between the 3D model shown in the 3D model data WMD and the object OBJ whose 3D position is represented by the 3D position data WSD. Furthermore, relevance can also be described as an indicator of the correlation between the 3D model represented by the 3D model data WMD and the object OBJ represented by the 3D position data WSD. Additionally, matching similarity can also be referred to as matching score.

[0162] If an object OBJ is detected by 3D matching processing and its calculated matching similarity exceeds a predetermined matching threshold, the position and attitude calculation unit 312 may select this object OBJ as an object OBJ for which the end effector 4 should perform predetermined processing (i.e., the processing execution object). On the other hand, if an object OBJ is detected by 3D matching processing and its calculated matching similarity is lower than the matching threshold, the position and attitude calculation unit 312 may not select this object OBJ as an object OBJ for which the end effector 4 should perform predetermined processing (i.e., the processing execution object).

[0163] Based on the three-dimensional position data WSD, the three-dimensional position data WSD may represent the positions of multiple object objects OBJ. For example, when multiple workpieces W are randomly or neatly placed on the mounting device T, the imaging device 22 may capture images of the multiple workpieces W and generate image data IMG_3D that captures the multiple workpieces W as multiple object objects OBJ. For example, when multiple workpieces W are randomly or neatly stored in a container (storage box) CB, which is an example of the mounting device T, the imaging device 22 may capture images of the multiple workpieces W and generate image data IMG_3D that captures the multiple workpieces W as multiple object objects OBJ. As a result, the position and attitude calculation unit 312 may generate three-dimensional position data WSD representing the positions of the multiple object objects OBJ. In this case, the position and attitude calculation unit 312 can perform the above-mentioned 3D matching processing on each of the multiple object objects OBJ. For example, the position and attitude calculation unit 312 can perform 3D matching processing sequentially on the multiple object objects OBJ. Alternatively, for example, the position and attitude calculation unit 312 can simultaneously perform multiple 3D matching processes, with multiple object objects OBJ as objects. Then, the position and attitude calculation unit 312 can select the object object OBJ among the multiple object objects whose matching similarity exceeds the matching determination threshold and has the highest matching similarity, as the processing execution object for which the end effector 4 should perform the prescribed processing. Alternatively, the position and attitude calculation unit 312 can select the object object OBJ among the multiple object objects whose matching similarity exceeds the matching determination threshold and has the Nth largest matching similarity (N is a constant representing an integer greater than 2), as the processing execution object for which the end effector 4 should perform the prescribed processing. Alternatively, the position and attitude calculation unit 312 can select the object object OBJ among the multiple object objects whose matching similarity exceeds the predetermined matching determination threshold, as the processing execution object. Alternatively, the position and attitude calculation unit 312 can select the object object OBJ among the multiple object objects whose matching similarity exceeds the matching determination threshold and is closest to the end effector 4, as the processing execution object. Alternatively, for example, when multiple object objects OBJ are stored in the container CB (loading device T) (e.g., when multiple objects are stacked separately), the position and orientation calculation unit 312 can select the object object OBJ among the multiple object objects OBJ that corresponds to a matching similarity exceeding the matching determination threshold and has the largest Z-coordinate along the Z-axis (located at the highest position) as the processing execution object. Furthermore, the Z-coordinate of each of the multiple object objects OBJ can also be the Z-position along the Z-axis of the robot 1's global coordinate system calculated through the aforementioned 3D matching process. Alternatively, the position and orientation calculation unit 312 can select the object object OBJ among the multiple object objects OBJ that corresponds to a matching similarity exceeding the matching determination threshold and for which the end effector 4 can perform the specified processing as the processing execution object.

[0164] However, when the end effector 4 can perform specified processing on two or more object objects OBJ simultaneously, the position and attitude calculation unit 312 can select at least two of the multiple object objects OBJ as processing execution objects. That is, the position and attitude calculation unit 312 can select at least two processing execution objects that the end effector 4 should perform specified processing on simultaneously. Furthermore, in the following description, for ease of explanation, an example will be given of the position and attitude calculation unit 312 selecting one object object OBJ from the multiple object objects OBJ as the processing execution object.

[0165] Alternatively, the position and attitude calculation unit 312 can calculate the matching similarity of multiple object OBJs, and then, based on the matching similarity of the multiple object OBJs, select one object OBJ for which the end effector 4 should perform the prescribed processing. Then, it calculates at least one of the position and attitude of the selected object OBJ. In this case, the position and attitude calculation unit 312 may not calculate the position and attitude of other object OBJs that were not selected as the object OBJ for which the end effector 4 should perform the prescribed processing. Alternatively, as described above, since at least one of the position and attitude of the object OBJs is calculated as the result of the 3D matching process, it can also be considered that the position and attitude calculation unit 312 calculates the matching similarity together with at least one of the position and attitude of the object OBJs. In this case, the position and attitude calculation unit 312 can calculate the matching similarity of the multiple object OBJs together with at least one of the position and attitude of each of the multiple object OBJs, and then select one object OBJ for which the end effector 4 should perform the prescribed processing based on the matching similarity of the multiple object OBJs. Alternatively, the position and attitude calculation unit 312 can calculate at least one of the positions and attitudes of multiple object objects OBJ, then calculate the matching similarity of the multiple object objects OBJ, and then select an object object OBJ that the end effector 4 should perform specified processing based on the calculated matching similarity.

[0166] However, in Figure 4 In step S3, in addition to performing 3D matching processing using three-dimensional position data WSD, or as an alternative, the position and pose calculation unit 312 can also calculate at least one of the position and pose of the object OBJ by performing matching processing using image data IMG_2D. Furthermore, in the following description, the matching processing using image data IMG_2D will be referred to as 2D matching processing.

[0167] Specifically, the position and pose calculation unit 312 can perform 2D matching processing, which uses image data IMG_2D (two-dimensional position data) and two-dimensional model data representing a two-dimensional model that serves as a reference for the object OBJ. Specifically, as 2D matching processing, the position and pose calculation unit 312 can use the two-dimensional model shown in the two-dimensional model data as a template image within the image shown in the image data IMG_2D, thereby performing object detection processing to detect the object OBJ shown in the template image. In other words, as matching processing, the position and pose calculation unit 312 can also perform object detection processing, which detects the object OBJ within the image shown in the image data IMG_2D by detecting similar image portions similar to the template image within the image shown in the image data IMG_2D. Furthermore, the 2D matching processing (in this case, object detection processing) itself can be the same as existing matching processing. For example, the position and pose calculation unit 312 can use known methods such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed-Upped RobustFeature) to perform 2D matching processing.

[0168] The two-dimensional model data can be based on the three-dimensional model data of the object OBJ. Here, the two-dimensional model data can be generated from the CAD (Computer Aided Design) model data of the object OBJ, or from the model data (representing a polygonal model or mesh model) of the object OBJ generated based on prior measurements using a known three-dimensional shape measuring device or imaging device 22. For example, the two-dimensional model data can be two-dimensional image data, representing at least a portion of the two-dimensional model of the object OBJ generated by virtually projecting at least a portion of the three-dimensional model shown in the three-dimensional model data of the object OBJ onto a virtual plane. For example, the two-dimensional model data can be two-dimensional image data, representing at least a portion of the edge of the two-dimensional model of the object OBJ generated by virtually projecting at least a portion of the three-dimensional model shown in the three-dimensional model data of the object OBJ onto a virtual plane. Alternatively, the two-dimensional model data may not be based on the three-dimensional model data of the object OBJ. For example, the two-dimensional model data can be image data IMG_2D generated by prior imaging of the object OBJ using imaging device 21. For example, the two-dimensional model data can be feature regions on an image shown in image data IMG_2D generated by pre-captured object OBJ by imaging device 21. For example, the feature regions on the image shown in image data IMG_2D can be at least one of feature points and edges of object OBJ captured in the image shown in image data IMG_2D. Furthermore, if the feature regions on the image shown in image data IMG_2D include edges of object OBJ, the two-dimensional model can also be considered as a model of the edges of at least a portion of object OBJ (edge ​​model). The feature regions on the image shown in image data IMG_2D can be at least one of at least a plurality of feature points of object OBJ captured in the image shown in image data IMG_2D, and at least one portion of the edges of object OBJ. Additionally, the feature regions on the image shown in image data IMG_2D can also be detected using known image processing techniques.

[0169] The position and pose calculation unit 312 can translate, enlarge, reduce, and / or rotate the two-dimensional model of the object OBJ captured in the template image, so that the feature parts (e.g., at least one feature point and edge) of the two-dimensional model of the object OBJ captured in the template image are close to (e.g., matched) the feature parts of the object OBJ captured in the image data IMG_2D. As a result, the position and pose calculation unit 312 can determine the positional relationship between the coordinate system of the two-dimensional model data and the 2D shooting coordinate system. Then, based on the positional relationship between the coordinate system of the two-dimensional model data and the 2D shooting coordinate system, the position and pose calculation unit 312 can calculate at least one of the position and pose of the object OBJ in the 2D shooting coordinate system, based on at least one of the position and pose of the object OBJ in the coordinate system of the two-dimensional model data.

[0170] Subsequently, the position and attitude calculation unit 312, based on a transformation matrix used to convert the three-dimensional coordinates in either the robot coordinate system or the 2D camera coordinate system into three-dimensional coordinates in the other, converts at least one of the position and attitude of the object OBJ in the 2D camera coordinate system into at least one of the position and attitude of the object OBJ in the robot coordinate system. When the robot coordinate system is used as the global coordinate system, at least one of the position and attitude of the object OBJ in the robot coordinate system is equivalent to at least one of the position and attitude of the object OBJ in the global coordinate system. However, when a coordinate system different from the robot coordinate system is used as the global coordinate system, the position and attitude calculation unit 312 can use a transformation matrix used to convert the three-dimensional coordinates in either the robot coordinate system or the global coordinate system into three-dimensional coordinates in the other, and calculate at least one of the position and attitude of the object OBJ in the global coordinate system based on at least one of the position and attitude of the object OBJ in the robot coordinate system.

[0171] In the case of 2D matching processing, the position and pose calculation unit 312 can also calculate the matching similarity, similar to the case of 3D matching processing. The matching similarity calculated in 2D matching processing is the similarity between the template image (i.e., the two-dimensional model of the object OBJ) and the image shown in the image data IMG_2D (particularly the image portion of the template image that incorporates the object OBJ). Furthermore, in the following description, the similarity between the template image (i.e., the two-dimensional model of the object OBJ) and the image shown in the image data IMG_2D (particularly the image portion of the template image that incorporates the object OBJ), i.e., the matching similarity, can be simply referred to as the matching similarity of the object OBJ. Moreover, the purpose of the matching similarity calculated in 2D matching processing is the same as that of the matching similarity calculated in the 3D matching processing described above, therefore, its detailed explanation is omitted.

[0172] In the case of 2D matching processing, similarly to the case of 3D matching processing, the position and pose calculation unit 312 can also perform 2D matching processing sequentially or simultaneously in parallel on multiple object objects OBJ captured in the image shown in the image data IMG_2D. Then, similarly to the case of 3D matching processing, the position and pose calculation unit 312 can select at least one object object OBJ from the multiple object objects OBJ as the processing execution object. In this case, the position and pose calculation unit 312 can select at least one object object OBJ from the multiple object objects OBJ as the processing execution object based on matching similarity. Furthermore, Figure 4 In step S3, the position and pose calculation unit 312 performs matching processing using image data IMG_2D instead of 3D matching processing using 3D position data WSD, thereby calculating at least one of the position and pose of the object OBJ. Figure 4 The robot control process shown may not include Figure 4 Step S2.

[0173] Or, in Figure 4 In step S3, in addition to performing at least one of the 3D matching processing using three-dimensional position data WSD and 2D matching processing using image data IMG_2D, or as an alternative, the position and pose calculation unit 312 may perform matching processing using both three-dimensional position data WSD and image data IMG_2D to calculate at least one of the position and pose of the object OJB.

[0174] For example, the position and pose calculation unit 312 can calculate at least one of the position and pose of the object OBJ in the global coordinate system based on a portion of the position and pose of the object OBJ in the 2D shooting coordinate system calculated based on image data IMG_2D, and a portion of the position and pose of the object OBJ in the 3D shooting coordinate system calculated based on image data IMG_3D. For example, the position and pose calculation unit 312 can set at least one of the initial position and initial pose of the three-dimensional model (template model TM3) used for 3D matching processing based on at least one of the position and pose of the object OBJ in the 2D shooting coordinate system calculated based on image data IMG_2D, and perform 3D matching processing using the three-dimensional model configured at the set initial position and / or with the set initial pose, thereby calculating at least one of the position and pose of the object OBJ in the 3D shooting coordinate system, and calculate at least one of the position and pose of the object OBJ in the global coordinate system based on at least one of the position and pose of the object OBJ in the 3D shooting coordinate system.

[0175] When calculating the position and orientation of an object OBJ using both image data IMG_2D and three-dimensional position data WSD, the position and orientation calculation unit 312 can select an object OBJ that satisfies the condition that the sum or product of the matching similarity calculated through 2D matching processing (hereinafter referred to as 2D matching similarity) and the matching similarity calculated through 3D matching processing (hereinafter referred to as 3D matching similarity) is the largest, as an object OBJ for which the end effector 4 should perform prescribed processing. The position and orientation calculation unit 312 can also select an object OBJ that satisfies the condition that the 2D matching similarity exceeds the matching determination threshold and is the largest, as an object OBJ for which the end effector 4 should perform prescribed processing. Similarly, the position and orientation calculation unit 312 can select an object OBJ that satisfies the condition that the 3D matching similarity exceeds the matching determination threshold and is the largest, as an object OBJ for which the end effector 4 should perform prescribed processing. The position and attitude calculation unit 312 may also select an object OBJ that satisfies both the 2D matching similarity and 3D matching similarity exceeding the matching determination threshold, as an object OBJ for which the end effector 4 should perform prescribed processing. Alternatively, the position and attitude calculation unit 312 may select an object OBJ that satisfies both the 2D matching similarity and 3D matching similarity exceeding the matching determination threshold and is closest to the end effector 4, as an object OBJ for which the end effector 4 should perform prescribed processing. Furthermore, the position and attitude calculation unit 312 may select an object OBJ that satisfies both the 2D matching similarity and 3D matching similarity exceeding the matching determination threshold and has the largest Z-coordinate along the Z-axis (located at the highest position), as an object OBJ for which the end effector 4 should perform prescribed processing.

[0176] Furthermore, if the position and attitude calculation unit 312 performs matching processing using three-dimensional position data WSD but not matching processing using image data IMG_2D, the control device 3 may not acquire image data IMG_2D in step S1. If the control device 3 does not acquire image data IMG_2D, the imaging device 21 may not capture the object OBJ. If the control device 3 does not acquire image data IMG_2D, the imaging system 2 may not include the imaging device 21.

[0177] If the position and attitude calculation unit 312 performs matching processing using image data IMG_2D but not matching processing using three-dimensional position data WSD, the control device 3 may not acquire image data IMG_3D in step S1. If the control device 3 does not acquire image data IMG_3D, the imaging device 22 may not capture the object OBJ. If the control device 3 does not acquire image data IMG_3D, the imaging system 2 may not include the imaging device 22. If the control device 3 does not acquire image data IMG_3D, the control device 3 may not include the three-dimensional position data generation unit 311.

[0178] Furthermore, in the following explanations, for the sake of simplicity, the main focus is on... Figure 4 The following explanation will be based on the 3D matching process performed by the position and attitude calculation unit 312 in step S3.

[0179] Subsequently, the signal generation unit 313 uses the position and orientation data POI (i.e., the calculation result of at least one of the position and orientation of the processing execution object) generated in step S3 to generate a robot control signal (step S4). For example, the signal generation unit 313 can generate a robot control signal so that the end effector 4 can perform prescribed processing on the object OBJ (especially the processing execution object). For example, the signal generation unit 313 can generate a robot control signal so that the end effector 4 approaches the object OBJ (especially the processing execution object) so that the end effector 4 performs prescribed processing on the object OBJ (especially the processing execution object). For example, the signal generation unit 313 can generate a robot control signal so that the positional relationship between the end effector 4 and the object OBJ (especially the processing execution object) becomes a desired positional relationship. For example, the signal generation unit 313 can also generate a robot control signal for controlling the movement of the robotic arm 12 so that the positional relationship between the end effector 4 and the object OBJ (especially the processing execution object) becomes a desired positional relationship. For example, the signal generation unit 313 can generate a robot control signal so that the end effector 4 performs prescribed processing on the object OBJ (especially the processing object) at a time when the positional relationship between the end effector 4 and the object OBJ (especially the processing object) is the desired positional relationship. For example, the signal generation unit 313 can generate a robot control signal for controlling the action of the end effector 4 so that prescribed processing is performed on the object OBJ (especially the processing object) at a time when the positional relationship between the end effector 4 and the object OBJ (especially the processing object) is the desired positional relationship. Furthermore, as described above, the robot control signal for controlling the action of the end effector 4 can be referred to as an end effector control signal.

[0180] Furthermore, the signal generation unit 313 can use the position and orientation data POI generated in step S3 to generate a robot control signal for feedback control of robot 1. For example, the signal generation unit 313 can use the position and orientation data POI to generate a robot control signal for feedback control including P (Proportional) control. For example, the signal generation unit 313 can use the position and orientation data POI to generate a robot control signal for feedback control including PI (Proportional-Integral) control. For example, the signal generation unit 313 can use the position and orientation data POI to generate a robot control signal for feedback control including PID (Proportional-Integral-Differential) control.

[0181] As an example, Figure 5 (a) to Figure 5 (d) are side views showing the positional relationship between robot 1 and workpiece W at a certain moment during a holding process for holding a workpiece W (i.e., an example of an object OBJ) placed by a loading device T#1 such as an AGV (Automatic Guided Vehicle) moving on support surface S. In this case, control device 3 can generate position and orientation data POI representing at least one of the position and orientation of workpiece W (i.e., an example of an object OBJ in the holding process), which is the object of the holding process, and use the generated position and orientation data POI to generate robot control signals. For example, as Figure 5 As shown in (a), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4 moves toward the space directly above the moving workpiece W. As an example, the signal generation unit 313 can generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4 is positioned (in other words, close to) the successively updated target location of the end effector 4 in a desired posture, based on at least one of the position and orientation of the workpiece W calculated successively by the position and orientation calculation unit 312. Figure 5 As shown in (b), the signal generation unit 313 can also generate robot control signals to control the movement of the robotic arm 12 so that the end effector 4, located directly above the moving workpiece W, continues to approach the workpiece W from directly above it until it can hold the workpiece W. Figure 5As shown in (c), the signal generation unit 313 can also generate robot control signals for controlling the movements of the robotic arm 12 and the end effector 4, so that the end effector 4, located at a position capable of holding the moving workpiece W, follows and holds the moving workpiece W. Figure 5 As shown in (d), the signal generation unit 313 can also generate robot control signals for controlling the movements of the robotic arm 12 and the end effector 4 so that the end effector 4 holding the workpiece W leaves the moving mounting device T#1 while holding the workpiece W.

[0182] When robot 1 uses end effector 4 to hold workpiece W placed on loading device T#1, robot 1 can perform a release process to release the held workpiece W, thereby placing the held workpiece W onto loading device T#2 (i.e., an example of the object OBJ in the release process) which is different from loading device T#1. That is, robot 1 can also perform a configuration process to place workpiece W on loading device T#2 by successively performing holding and release processes. In this case, control device 3 can generate position and orientation data POI representing at least one of the position and orientation of loading device T#2, which is the object of the release process, and use the generated position and orientation data POI to generate robot control signals. For example, Figure 6 (a) to Figure 6 (d) are side views showing the positional relationship between the robot 1 and the workpiece W at a certain moment during the release process of positioning the workpiece W onto the mounting device T#2, which moves on the support surface S. In this case, as... Figure 6 As shown in (a), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4 holding the workpiece W moves to the space directly above the moving mounting device T#2 while holding the workpiece W. As an example, the signal generation unit 313 can generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4 is positioned (in other words, close to) the successively updated moving target location based on the moving mounting device T#2, based on at least one of the position and orientation of the mounting device T#2 calculated successively by the position and orientation calculation unit 312. Figure 6 As shown in (b), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4, which is located directly above the moving mounting device T#2 and holding the workpiece W, continues to be located directly above the mounting device T#2 and continues to hold the workpiece W as it approaches the mounting device T#2 until the workpiece W can be positioned on the mounting device T#2. For example... Figure 6As shown in (c), the signal generation unit 313 can also generate robot control signals for controlling the actions of the robotic arm 12 and the end effector 4 so that the end effector 4, located at a position capable of placing the workpiece W on the loading device T#2, follows the moving loading device T#2 while loading the workpiece W onto the loading device T#2 (i.e., releasing the held workpiece W). Figure 6 As shown in (d), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4, after placing the workpiece W on the loading device T#2, leaves the loading device T#2.

[0183] In the case of performing a release process (especially a release process performed for the purpose of configuration processing), in addition to position and attitude data POI representing at least one of the position and attitude of the loading device T#2 (i.e., an example of the object OBJ in the release process) which is the first object of the release process, the control device 3 may further generate position and attitude data POI representing at least one of the position and attitude of the workpiece W (i.e., an example of the object OBJ in the release process) which is the second object of the release process. That is, in addition to generating position and attitude data POI representing at least one of the position and attitude of the workpiece W which is not yet held by the end effector 4 before the end effector 4 holds the workpiece W, the control device 3 may also generate position and attitude data POI representing at least one of the position and attitude of the workpiece W held by the end effector 4 after the end effector 4 holds the workpiece W.

[0184] In the case of release processing, before releasing the workpiece W held by the end effector 4 through the release processing, in addition to the position and orientation data POI regarding the loading device T#2, the control device 3 may also use position and orientation data POI representing at least one of the position and orientation of the workpiece W held by the end effector 4 to generate a robot control signal. That is, the control device 3 may use position and orientation data POI representing at least one of the position and orientation of the workpiece W held by the end effector 4 to generate a robot control signal during at least a portion of the period during which the end effector 4 holds the workpiece W.

[0185] For example, the signal generation unit 313 can also generate robot control signals to control the robotic arm 12 to move the workpiece W held by the end effector 4 to a desired position (e.g., the position where the workpiece W should be released). In this case, compared to not using the position and orientation data (POI) of the workpiece W, the robot 1 can properly position the workpiece W held by the end effector 4 to the desired position on the mounting device T (and can perform the configuration process). This is because the position of the workpiece W held by the end effector 4 is known information to the control device 3. If the position of the workpiece W is not known, a technical problem may arise where the workpiece W collides with the mounting device T. Or, if the position of the workpiece W is not known, the configuration process described above may result in the workpiece W being positioned in an unintended position on the mounting device T (or other object). Or, if the position of the workpiece W is not known, the embedding process described above may result in the workpiece W being embedded in an unintended position on another object. Or, if the position of the workpiece W is not known, the pasting process described above may result in the workpiece W being pasted to an unintended position on another object. Alternatively, if the position of workpiece W is not known, the above-described bonding process may result in the workpiece W being bonded to an unintended position on another object. Alternatively, if the position of workpiece W is not known, the above-described welding process may result in the workpiece W being welded to an unintended position on another object. However, in this embodiment, since the position of workpiece W is known, the possibility of such a technical problem is nonexistent or low.

[0186] Furthermore, for example, the signal generation unit 313 can also generate robot control signals for controlling the end effector 4 to change the posture of the workpiece W held by the end effector 4 to the desired posture. In this case, compared to the case where the position and orientation information (POI) related to the workpiece W is not used, the robot 1 can use the end effector 4 to place the workpiece W in the desired posture on the mounting device T (and can perform the configuration process). This is because the posture of the workpiece W held by the end effector 4 is known information to the control device 3. Suppose that if the posture of the workpiece W is not the desired posture, a technical problem may occur where the workpiece W collides with the mounting device T. Or, if the posture of the workpiece W is not the desired posture, the above configuration process may result in a technical problem where the workpiece W is placed on the mounting device T (or other object) in an unexpected posture. Or, if the posture of the workpiece W is not the desired posture, the above embedding process may result in a technical problem where the workpiece W is embedded into other objects in an unexpected posture. Alternatively, if the orientation of workpiece W is not the desired orientation, the above-described adhesive process may result in the workpiece W being adhered to another object in an unexpected orientation. Similarly, if the orientation of workpiece W is not the desired orientation, the above-described adhesive process may result in the workpiece W being adhered to another object in an unexpected orientation. Or, if the orientation of workpiece W is not the desired orientation, the above-described welding process may result in the workpiece W being welded to another object in an unexpected orientation. However, in this embodiment, since the robot 1 can be controlled to make the orientation of workpiece W the desired orientation, the possibility of such technical problems occurring is non-existent or low.

[0187] Furthermore, not limited to the case of release processing, in any scenario where the end effector 4 holds the workpiece W, the control device 3 can use position and orientation data POI, which represents at least one of the position and orientation of the workpiece W held by the end effector 4, to generate robot control signals.

[0188] Figure 5 (a) to Figure 5 (d) shows the mounting device T#1 and Figure 6 (a) to Figure 6 (d) It is not necessarily required that at least one of the mounting devices T#2 needs to move on the support surface S. For example, Figure 7 (a) to Figure 7 (b) are side views showing the positional relationship between the robot 1 and the workpiece W at a certain moment during the holding process for holding the workpiece W placed on the mounting device T#1, which is stationary on the support surface S. In this case, as Figure 7As shown in (a), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robotic arm 12 to bring the end effector 4 close to the workpiece W until the workpiece W can be kept stationary. As an example, the signal generation unit 313 can generate robot control signals for controlling the movement of the robotic arm 12 to bring the end effector 4 to a position (in other words, close to) a position where the end effector 4 can hold the workpiece W, whose position and orientation have been calculated, based on at least one of the positions and orientations of the workpiece W calculated by the position and orientation calculation unit 312. Figure 7 As shown in (b), the signal generation unit 313 can generate robot control signals for controlling the movements of the robotic arm 12 and the end effector 4 so that the end effector 4, located at a position capable of holding the workpiece W stationary, maintains the workpiece W. Furthermore, Figure 7 (c) to Figure 7 (e) are side views showing the positional relationship between the robot 1 and the workpiece W at a certain moment during the release process of positioning the workpiece W onto the mounting device T#2, which is stationary on the support surface S. In this case, as Figure 7 As shown in (c), the signal generation unit 313 can also generate robot control signals to control the movement of the robotic arm 12 so that the end effector 4 holding the workpiece W approaches the mounting device T#2 while holding the workpiece W until the workpiece W can be positioned on the stationary mounting device T#2. Figure 7 As shown in (d), the signal generation unit 313 can also generate robot control signals for controlling the actions of the robotic arm 12 and the end effector 4 so that the end effector 4, located at a position capable of placing the workpiece W in the mounting device T#2, places the workpiece W in the stationary mounting device T#2 (i.e., releases the held workpiece W). Figure 7 As shown in (e), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4, after placing the workpiece W on the loading device T#2, leaves the loading device T#2.

[0189] Multiple workpieces W can be placed on the loading device T#1. For example, multiple workpieces W can be placed on the loading device T#1 in a manner in which multiple workpieces W are arranged on the loading device T#1 according to a certain reference. For example, multiple workpieces W can be placed on the loading device T#1 in a manner in which multiple workpieces W are randomly stacked on the loading device T#1. In this case, the robot 1 can perform a holding process for selectively holding one desired workpiece W among the multiple workpieces W placed on the loading device T#1. In particular, the robot 1 can perform a holding process for holding the multiple workpieces W placed on the loading device T#1 one by one in sequence.

[0190] Furthermore, robot 1 can perform a release process for placing multiple workpieces W on the mounting device T#2. That is, robot 1 can perform a holding process and a release process to sequentially place multiple workpieces W, currently mounted on mounting device T#1, onto mounting device T#2 (or further onto other mounting devices). In this case, robot 1 can perform a release process where multiple workpieces W are arranged sequentially on mounting device T#2 according to a certain reference. Alternatively, robot 1 can perform a release process where multiple workpieces W are randomly stacked on mounting device T#2.

[0191] As an example, Figure 8 (a) to Figure 8 (e) and Figure 9 (a) to Figure 9 (e) are side views showing the positional relationship between the robot 1 and the workpiece W at a certain moment during the holding process for sequentially holding two workpieces W#1 and W#2 placed on the mounting device T#1 and the release process for sequentially placing the two workpieces W#1 and W#2 on the mounting device T#2. In this case, as Figure 8 As shown in (a), the signal generation unit 313 can generate signals to control the movement of the robotic arm 12 so that the end effector 4 approaches either workpiece W#1 or W#2. Figure 8 In the example shown in (a), the robot control signal for workpiece W#2 continues until workpiece W#2 can be held. As an example, the signal generation unit 313 can generate a robot control signal for controlling the movement of the robotic arm 12 so that the end effector 4 is positioned (in other words, close to) a position where the end effector 4 can hold the workpiece W#2, which has at least one calculated position and orientation, based on at least one of the position and orientation of workpiece W#2 calculated by the position and orientation calculation unit 312. Then, as Figure 8 As shown in (b), the signal generation unit 313 can generate robot control signals for controlling the movements of the robotic arm 12 and the end effector 4 so that the end effector 4, located in a position capable of holding the workpiece W#2, holds the workpiece W#2. Then, as... Figure 8As shown in (c), the signal generation unit 313 can generate a robot control signal for controlling the movement of the robotic arm 12 so that the end effector 4 holding the workpiece W#2 approaches the loading device T#2 while holding the workpiece W#2 until the workpiece W#2 can be placed on the loading device T#2. As an example, the signal generation unit 313 can generate a robot control signal for controlling the movement of the robotic arm 12 so that the end effector 4 is positioned (in other words, approaching) in a desired posture at a position on the loading device T#2 where the end effector 4 can place the workpiece W2 on the loading device T#2 for which at least one of the calculated positions and postures has been calculated, based on at least one of the positions and postures of the loading device T#2 calculated by the position and posture calculation unit 312. Then, as... Figure 8 As shown in (d), the signal generation unit 313 can also generate robot control signals for controlling the actions of the robotic arm 12 and the end effector 4 so that the end effector 4, located at a position capable of placing the workpiece W#2 on the loading device T#2, loads the workpiece W#2 onto the loading device T#2 (i.e., releases the held workpiece W#2). Then, as... Figure 8 As shown in (e), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robotic arm 12 to cause the end effector 4, which has positioned the workpiece W#2 onto the mounting device T#2, to leave the mounting device T#2. Then, as... Figure 9 As shown in (a), the signal generation unit 313 can generate actions to control the robotic arm 12 so that the end effector 4 approaches the remaining one of workpieces W#1 and W#2. Figure 9 In the example shown in (a), the robot control signal for workpiece W#1 continues until workpiece W#1 can be held. As an example, the signal generation unit 313 can generate a robot control signal for controlling the movement of the robotic arm 12 so that the end effector 4 is positioned (in other words, close to) a position where the end effector 4 can hold the workpiece W#1, which has at least one calculated position and orientation, based on at least one of the position and orientation of workpiece W#1 calculated by the position and orientation calculation unit 312. Then, as Figure 9 As shown in (b), the signal generation unit 313 can generate robot control signals for controlling the movements of the robotic arm 12 and the end effector 4 so that the end effector 4, located in a position capable of holding the workpiece W#1, holds the workpiece W#1. Then, as... Figure 9As shown in (c), the signal generation unit 313 can generate a robot control signal for controlling the movement of the robotic arm 12 so that the end effector 4 holding the workpiece W#1 approaches the loading device T#2 while holding the workpiece W#1 until the workpiece W#1 can be placed on the loading device T#2. As an example, the signal generation unit 313 can generate a robot control signal for controlling the movement of the robotic arm 12 so that the end effector 4 is positioned (in other words, approaching) in a desired posture at a position on the loading device T#2 where the end effector 4 can place the workpiece W1 on the loading device T#2, based on at least one of the position and posture calculated by the position and posture calculation unit 312. Then, as... Figure 9 As shown in (d), the signal generation unit 313 can also generate robot control signals for controlling the actions of the robotic arm 12 and the end effector 4 so that the end effector 4, located at a position capable of placing the workpiece W#1 on the loading device T#2, loads the workpiece W#1 onto the loading device T#2 (i.e., releases the held workpiece W#1). Then, as... Figure 9 As shown in (e), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4, after placing the workpiece W#1 on the loading device T#2, leaves the loading device T#2.

[0192] Robot 1 can perform a holding process to hold multiple workpieces W sequentially on a mounting device T#1 that moves on the support surface S. Alternatively, robot 1 can perform a holding process to hold multiple workpieces W sequentially on a mounting device T#1 that is stationary on the support surface S. Robot 1 can perform a release process to sequentially place multiple workpieces W onto a mounting device T#2 that moves on the support surface S. Robot 1 can perform a release process to sequentially place multiple workpieces W onto a mounting device T#2 that is stationary on the support surface S.

[0193] Back Figure 4 The signal generation unit 313 uses the communication device 33 to output the robot control signal generated in step S4 to the robot 1 (specifically the robot control device 13). As a result, the robot control device 13 controls at least one of the actions of the robot 1 (e.g., the actions of the robotic arm 12) and the actions of the end effector 4 based on the robot control signal.

[0194] Afterwards, control device 3 repeats the series of processes from steps S1 to S4 until it is determined that robot 1 has completed the prescribed processing on the object OBJ selected as the processing execution object in step S3 (step S5). For example, as Figure 5 (a) to Figure 5As shown in (d), when the end effector 4 performs holding processing, the control device 3 repeats a series of processes from steps S1 to S4 until it is determined that the end effector 4 is holding an object OBJ selected as the object to be processed in step S3 (step S5). For example, as Figure 6 (a) to Figure 6 As shown in (d), when the end effector 4 performs a release process, the control device 3 repeats a series of processes from step S1 to step S4 until it is determined that the end effector 4 has released an object OBJ held by the end effector 4 (step S5).

[0195] Since the series of processes from steps S1 to S4 are repeated when it is determined that robot 1 has not completed the prescribed processing for an object OBJ (step S5: no), the control device 3 also continuously acquires at least one of image data IMG_2D and IMG_3D from the imaging devices 21 and 22 while controlling the movement of at least one of the robotic arm 12 and the end effector 4 based on the robot control signal. For example, as Figure 5 (a) to Figure 5 As shown in (d), when the end effector 4 is performing a holding process, the control device 3 can repeat a series of processes from steps S1 to S4 until the end effector 4 holds the workpiece W (furthermore, as needed, until the end effector 4 holding the workpiece W leaves the mounting device T#1). For example, as... Figure 6 (a) to Figure 6 As shown in (d), when the end effector 4 performs the release process, the control device 3 can repeat a series of processes from step S1 to step S4 until the end effector 4 places the workpiece W on the loading device T#2 (furthermore, as needed, until the end effector 4 leaves the loading device T#2 after placing the workpiece W on the loading device T#2).

[0196] In this case, as described above, since the movement of at least one of the robotic arm 12 and the end effector 4 is controlled based on robot control signals, the capturing devices 21 and 22 can capture images of the object OBJ during the relative movement of the object OBJ and the capturing devices 21 and 22, respectively. For example, the capturing devices 21 and 22 can capture images of the object OBJ during a period when the object OBJ is stationary and the capturing devices 21 and 22 are moving, respectively. For example, the capturing devices 21 and 22 can capture images of the object OBJ during a period when the object OBJ is moving and the capturing devices 21 and 22 are stationary, respectively. For example, the capturing devices 21 and 22 can capture images of the object OBJ during a period when the object OBJ is moving and the capturing devices 21 and 22 are moving, respectively. That is, the control device 3 can continuously operate during the relative movement of the object OBJ and the capturing devices 21 and 22 (i.e., during the period when the capturing devices 21 and 22 and at least one of the object OBJ are moving). Figure 4 The robot control process shown can also be repeated. Figure 4 (The robot control processing shown). As a result, even during the period when the control device 3 controls the actions of the robot 1 based on the robot control signal, it is able to regenerate (i.e., update) the position and orientation data POI representing at least one of the position and orientation of the object OBJ based on the newly acquired image data IMG_2D and IMG_3D.

[0197] In this situation, particularly when at least one of the positions and orientations of the object OBJ changes due to its movement, the control device 3 can recalculate at least one of the positions and orientations of the object OBJ. That is, the control device 3 can regenerate position and orientation data POI that reflects the change in at least one of the positions and orientations of the object OBJ caused by its movement. Therefore, the possibility of a large deviation between at least one of the positions and orientations shown in the position and orientation data POI and at least one of the actual positions and orientations of the object OBJ is low. Therefore, the control device 3 can appropriately control the movement of the robotic arm 12 so that the end effector 4 can appropriately approach the object OBJ (especially a moving object OBJ).

[0198] The imaging devices 21 and 22 can respectively capture images of the object OBJ while both the object OBJ and the imaging devices 21 and 22 are stationary. The control device 3 can operate during the periods when the imaging devices 21 and 22 and the object OBJ are stationary. Figure 4 The robot control process is shown. Furthermore, control device 3 can repeat the process while the imaging devices 21 and 22 are stationary and while the object OBJ is at rest. Figure 4 The robot control process is shown.

[0199] Alternatively, if the object OBJ is stationary, the likelihood that at least one of the positions and orientations of the object OBJ calculated in step S3 will change over time is low. Therefore, the necessity to recalculate at least one of the positions and orientations of the object OBJ by repeating the series of processes from steps S1 to S3 is low. Therefore, even if it is determined that robot 1 has not completed the prescribed processing for an object OBJ when the object OBJ is stationary, control device 3 may not repeat the series of processes from steps S1 to S3. In this case, control device 3 can continue to control robot 1 based on the robot control signal generated in step S4. That is, control device 3 does not need to generate a new robot control signal in step S4, but can continue to control robot 1 based on the robot control signal generated in step S4. In other words, in order to control robot 1, control device 3 does not need to generate a new robot control signal in step S4, but can continue to use the robot control signal generated in step S4.

[0200] Furthermore, as described above, the position and attitude calculation unit 312 may not calculate at least one of the position and attitude of the object OBJ in the global coordinate system in step S3. In this case, the position and attitude calculation unit 312 may... Figure 4 In step S3, at least one of the position and orientation of the object OBJ in a coordinate system different from the global coordinate system (e.g., the robot coordinate system, the 2D camera coordinate system, or the 3D camera coordinate system) is calculated. That is, the position and orientation calculation unit 312 can generate position and orientation data POI representing at least one of the position and orientation of the object OBJ in a coordinate system different from the global coordinate system (e.g., the robot coordinate system, the 2D camera coordinate system, or the 3D camera coordinate system). In this case, the signal generation unit 313 can use the position and orientation data POI calculated in step S3, representing at least one of the position and orientation of the object OBJ in a coordinate system different from the global coordinate system, to generate a robot control signal in step S4.

[0201] Afterwards, if it is determined that robot 1 has completed the prescribed processing on an object OBJ (step S5: yes), control device 3 determines whether to end. Figure 4 The robot control process shown is step S6. For example, if robot 1 needs to perform a specified process on another object OBJ, the control device 3 can determine that the process should not end. Figure 4 The robot control process is illustrated. For example, if there is still an object OBJ that robot 1 is supposed to process, the control device 3 can determine that the process should not be terminated. Figure 4The robot control process is shown. For example, if robot 1 does not need to perform the specified processing on other objects OBJ, control device 3 can determine that the process is complete. Figure 4 The robot control process is illustrated. For example, if there are no remaining objects OBJ that robot 1 is supposed to process, the control device 3 can determine that the process is complete. Figure 4 The robot control process is shown.

[0202] The determination result in step S6 is that the process is considered complete. Figure 4 In the case of robot control processing shown (step S6: Yes), control device 3 ends. Figure 4 The robot control process is shown.

[0203] On the other hand, the determination result in step S6 is that the process does not end. Figure 4 In the case of robot control processing shown (step S6: No), the control device 3 performs the series of processes from steps S1 to S5 again. That is, the control device 3 performs the series of processes from steps S1 to S5 again so that the robot 1 re-performs the prescribed processing on other object OBJs that are different from the object OBJ that has already undergone the prescribed processing.

[0204] In this embodiment, the control device 3 (particularly the signal generation unit 313) adjusts the height of the imaging system 2 (step S7) after performing the series of processes from steps S1 to S5 again so that the robot 1 performs the prescribed processing on one object OBJ and before performing the prescribed processing on other object OBJs. Specifically, the control device 3 can adjust the height at which the imaging system 2 captures the object OBJ (step S7). Furthermore, in the following description, the height of the imaging system 2 (i.e., the height at which the imaging system 2 captures the object OBJ) is referred to as the "capture height CH". In addition, the capture height CH can also be said to be the height at which the imaging system 2 captures the object OBJ (capture processing). Therefore, the capture height CH can also be called the capture processing height.

[0205] In this embodiment, "height" refers to the position along the Z-axis in the global coordinate system. In this case, the height can be the Z-axis coordinate value in the global coordinate system. However, the height can also refer to the position of the imaging system 2 along the Z-axis in a coordinate system different from the global coordinate system. In this case, the height can be the Z-axis coordinate value in a coordinate system different from the global coordinate system. As an example of a coordinate system different from the global coordinate system, at least one of the aforementioned robot coordinate system, 2D imaging coordinate system, and 3D imaging coordinate system can be listed. Alternatively, the height can refer to the position along the optical axis of the imaging system 2. Furthermore, at least one of the optical axes AX21 of the imaging device 21 and AX22 of the imaging device 22 can be used as the optical axis of the imaging system 2. Furthermore, the optical axis direction of the imaging system 2 can be considered equivalent to the imaging direction of the imaging system 2.

[0206] Furthermore, in this embodiment, "height" can refer to the height from the reference point. As an example of a reference point, the origin of the global coordinate system can be used. As an example of a reference point, the origin of a coordinate system different from the global coordinate system can be used. As an example of a reference point, a reference part of a reference object can be used. As a reference object, the object object OBJ can be used, or other objects different from the object object OBJ can be used (e.g., a mounting device T or a container CB).

[0207] Furthermore, the shooting height CH can refer to the height of the shooting system 2 above the object OBJ. In this case, the shooting height CH can be considered equivalent to the distance between the shooting system 2 and the object OBJ. Specifically, the shooting height CH can be considered equivalent to the distance between the shooting system 2 that is shooting the object OBJ and the object OBJ. Therefore, the shooting height CH can also be referred to as the shooting distance. As mentioned above, the shooting height CH can be considered equivalent to the distance between the shooting system 2 and the object OBJ. In this case, the shooting height CH can be considered equivalent to the distance between the shooting system 2 and the object OBJ in the Z-axis direction of the global coordinate system. For example, the shooting height CH can be considered equivalent to the distance between the shooting system 2 and the object OBJ in the Z-axis direction, which is different from the global coordinate system. For example, the shooting height CH can be considered equivalent to the distance between the shooting system 2 and the object OBJ in the optical axis direction of the shooting system 2.

[0208] Furthermore, in this embodiment, "the height of the shooting system 2" can refer to the height of at least one of the shooting devices 21 and 22. Therefore, in this embodiment, "the shooting height CH of the shooting system 2" can refer to the height of the object OBJ being photographed by at least one of the shooting devices 21 and 22. For example, when image data IMG_2D is used in step S3 to calculate position and pose data POI, "the height of the shooting system 2" can refer to the height of the shooting device 21. That is, "the shooting height CH of the shooting system 2" can refer to the height at which the shooting device 21 photographs the object OBJ. In this case, "the shooting height CH of the shooting system 2" can refer to the height of a point representing the shooting device 21. For example, when image data IMG_3D (specifically, three-dimensional position data WSD generated based on image data IMG_3D) is used in step S3 to calculate position and pose data POI, "the height of the shooting system 2" can refer to the height of the shooting device 22. That is, "the shooting height CH of the shooting system 2" can refer to the height at which the shooting device 22 photographs the object OBJ. In this case, "the shooting height CH of the shooting system 2" can refer to the height of a point representing the shooting device 22. For example, in step S3, when both image data IMG_2D and image data IMG_3D (specifically, the three-dimensional position data WSD generated from image data IMG_3D) are used to calculate the position and pose data POI, "the height of the shooting system 2" can refer to the height of both shooting devices 21 and 22. That is, "the shooting height CH of the shooting system 2" can refer to the height at which both shooting devices 21 and 22 capture the object OBJ. In this case, "the shooting height CH of the shooting system 2" can refer to the height of a point representing both shooting devices 21 and 22.

[0209] After the shooting height CH of the shooting system 2 is adjusted, the control device 3 performs a series of processes from steps S1 to S5 again. Specifically, after adjusting the shooting height CH of the shooting system 2 in step S7, the shooting system 2 re-shoots the object OBJ, thereby generating at least one of image data IMG_2D and IMG_3D again (step S1). Then, the control device 3 can again acquire at least one of the image data IMG_2D and IMG_3D generated by the shooting system 2 again (step S1). Then, the control device 3 can re-generate three-dimensional position data WSD based on the image data IMG_3D acquired again in step S1 (step S2). Then, the control device 3 can re-generate position and pose data POI based on at least one of the image data IMG_2D acquired again in step S1 and the three-dimensional position data WSD generated again in step S2 (step S3). In other words, control device 3 can generate position and orientation data (POI) indicating at least one of the positions and orientations of other object OBJs (i.e., the next processing object) that the end effector 4 should next perform the prescribed processing on (step S3). Then, control device 3 generates robot control signals again based on the position and orientation data (POI) generated in step S3 (step S4). For example, control device 3 can generate robot control signals to cause the end effector 4 to approach the next processing object so that the end effector 4 can perform the prescribed processing on the next processing object. Then, control device 3 repeats the series of processes from steps S1 to S4 until it is determined that robot 1 has completed the prescribed processing on the next processing object (step S5).

[0210] Subsequently, the control device 3 alternately repeats the processes of controlling the robot 1 to perform the prescribed processing on the object (steps S1 to S5) and adjusting the shooting height CH of the shooting system 2 (step S7) until it is determined to be finished. Figure 4 The robot control process is shown up to step S6: Yes. That is, whenever robot 1 completes the prescribed processing on a processing object, control device 3 adjusts the shooting height CH of shooting system 2. However, as described later, control device 3 does not necessarily need to adjust the shooting height CH of shooting system 2 whenever robot 1 completes the prescribed processing on a single processing object. Control device 3 can adjust the shooting height CH of shooting system 2 whenever the robot completes the prescribed processing on two or more processing objects.

[0211] (2-2) Adjusting the shooting height CH of shooting system 2 Next, regarding Figure 4 The process of adjusting the shooting height CH of the shooting system 2 in step S7 will be explained in more detail.

[0212] In order to adjust the shooting height CH of the shooting system 2, the control device 3 can determine the target value of the shooting height CH of the shooting system 2. Furthermore, in the following description, the target value of the shooting height CH of the shooting system 2 will be referred to as "target shooting height CH_target".

[0213] For example, a cross-sectional view representing multiple objects OBJ and camera system 2. Figure 10 As shown, the control device 3 can determine the target shooting height CH_target to satisfy the condition that at least one object OBJ is included within the shooting allowable range CAR of the shooting system 2 located at the target shooting height CH_target. Specifically, the control device 3 can determine the target shooting height CH_target to satisfy the condition that at least one object OBJ housed in the container CB is included within the shooting allowable range CAR of the shooting system 2 located at the target shooting height CH_target. Furthermore, the shooting system 2 located at the target shooting height CH_target can refer to a shooting system 2 whose shooting height CH is the same as the target shooting height CH_target. In particular, the control device 3 can determine the target shooting height CH_target to satisfy the condition that at least one object OBJ, which the robot 1 (end effector 4) is to subsequently perform specified processing, is included within the shooting allowable range CAR of the shooting system 2 located at the target shooting height CH_target.

[0214] Here, as Figure 10As shown, when multiple object OBJs are configured such that at least two of them partially overlap (e.g., the multiple object OBJs are randomly configured), robot 1 is likely to process the multiple object OBJs sequentially from top to bottom. For example, robot 1 may process an object OBJ located above other object OBJs. For example, robot 1 may process the topmost object OBJ. For example, robot 1 may process an object OBJ whose upper portion is not covered by other object OBJs. For example, robot 1 is more likely to process an object OBJ whose upper portion is covered by other object OBJs with a relatively small area. Therefore, control device 3 can determine the target shooting height CH_target to satisfy the condition that the shooting range CAR of shooting system 2 at the target shooting height CH_target includes at least one object OBJ located above other object OBJs. Control device 3 can determine the target shooting height CH_target to satisfy the condition that the shooting range CAR of shooting system 2 at the target shooting height CH_target includes the topmost object OBJ. The control device 3 can determine the target shooting height CH_target to satisfy the condition that the shooting range CAR of the shooting system 2 located at the target shooting height CH_target includes at least one object OBJ whose upper part is not covered by other object OBJs. The control device 3 can determine the target shooting height CH_target to satisfy the condition that the shooting range CAR of the shooting system 2 located at the target shooting height CH_target includes at least one object OBJ whose upper part is covered by a relatively small area of ​​other object OBJs.

[0215] The allowed shooting range (CAR) can be a range determined based on the depth of field of the shooting system 2. For example, the allowed shooting range (CAR) can be the same as the depth of field of the shooting system 2. For example, the allowed shooting range (CAR) can be a range that at least partially overlaps with the depth of field of the shooting system 2. That is, the allowed shooting range (CAR) can be a range that includes a portion or all of the depth of field of the shooting system 2. In this case, the allowed shooting range (CAR) can be a range that is narrower than the depth of field of the shooting system 2, or it can be a range that is wider than the depth of field of the shooting system 2. Furthermore, the depth of field of the shooting system 2 can refer to the depth of field of at least one of the shooting devices 21 and 22.

[0216] Furthermore, the depth of field of the imaging system 2 can be, for example, a range extending laterally along the optical axis of the imaging system 2 (on the object side of the imaging system 2) centered on the optimal focus position BFP of the object-side optical system (e.g., the objective lens) of the imaging system 2. Therefore, the permissible shooting range CAR can be a range determined based on the optimal focus position BFP of the object-side optical system of the imaging system 2. For example, the permissible shooting range CAR can be a range extending laterally along the optical axis of the imaging system 2 centered on the optimal focus position BFP of the object-side optical system of the imaging system 2. Furthermore, the object-side optical system of the imaging system 2 can refer to the object-side optical system of at least one of the imaging devices 21 and 22. Additionally, the optimal focus position BFP can also be referred to as the focal position. Furthermore, the permissible shooting range CAR can be the optimal focus position BFP. That is, the permissible shooting range CAR can coincide with the optimal focus position BFP. In this case, the permissible shooting range CAR can also be referred to as the permissible shooting position.

[0217] The allowed shooting range (CAR) can be a range determined based on the depth of field of the shooting system 2. For example, the allowed shooting range (CAR) can be the same as the depth of field of the shooting system 2. For example, the allowed shooting range (CAR) can be a range that at least partially overlaps with the depth of field of the shooting system 2. That is, the allowed shooting range (CAR) can be a range that includes part or all of the depth of field of the shooting system 2. In this case, the allowed shooting range (CAR) can be a range narrower than the depth of field of the shooting system 2, or it can be a range wider than the depth of field of the shooting system 2. Furthermore, the depth of field of the shooting system 2 can refer to the depth of field of at least one of the shooting devices 21 and 22. The depth of field of the shooting system 2 can also refer to the depth of field on the image plane side of the projection device 23 that projects the projected pattern.

[0218] The allowed shooting range (CAR) can be a range that satisfies the accuracy condition that "the accuracy of at least one of the position and pose data (POI) of the image data IMG_2D and IMG_3D generated by the shooting system 2 by shooting the object OBJ contained within the allowed shooting range (CAR) is above a predetermined accuracy threshold." Specifically, as described above, the control device 3 (particularly the position and pose calculation unit 312) generates the position and pose data (POI) of the object OBJ with a matching similarity exceeding the matching determination threshold by performing position and pose calculations including matching processing. In this case, it can also be envisioned that the higher the matching similarity of the object OBJ, the higher the accuracy of the position and pose data (POI) of the object OBJ (i.e., the calculation accuracy of at least one of the position and pose). Therefore, the allowed shooting range (CAR) can be a range that satisfies the accuracy condition that "the matching similarity of the object OBJ generated by the shooting system 2 through the image data IMG_2D and IMG_3D of the object OBJ contained within the allowed shooting range (CAR) exceeds both the matching determination threshold and a specified accuracy threshold different from the matching determination threshold." Furthermore, as mentioned above, when the allowed shooting range (CAR) is determined based on at least one of the depth of field, optimal focus position (BFP), and depth of focus of the shooting system 2, the allowed shooting range (CAR) can also be determined based on at least one of the depth of field, optimal focus position (BFP), and depth of focus of the shooting system 2 to satisfy the accuracy condition.

[0219] The shooting range (CAR) can be determined based on the field of view (shooting range, which can also be expressed as field of view or angle of view) of the shooting system 2. Furthermore, the field of view of the shooting system 2 can refer to at least one of the field of view (shooting range) of the shooting device 21 and the field of view (shooting range) of the shooting device 22. Specifically, when the shooting system 2 includes the shooting device 22 and... Figure 4 In robot control processing, when position and pose data (POI) is generated based on three-dimensional position data (WSD) generated from image data IMG_3D generated from the imaging device 22, the allowed shooting range (CAR) can be determined based on the field of view (shooting range) of the imaging device 22. Hereinafter, for ease of explanation, two examples of the allowed shooting range (CAR) determined based on the field of view of the imaging device 22, which is a stereo camera (i.e., equipped with two monocular cameras), will be described.

[0220] Figure 11This is a cross-sectional view showing a first example of the permissible shooting range CAR determined based on the field of view of the shooting device 22. The permissible shooting range CAR can be a range determined based on the field of view of the two monocular cameras provided by the shooting device 22. Furthermore, in the following description, the two monocular cameras provided by the shooting device 22 will be referred to as the left camera 22L and the right camera 22R, respectively. Specifically, the permissible shooting range CAR can be a range determined based on the overlapping area 221O of the field of view 221L of the left camera 22L and the field of view 221R of the right camera 22R. For example, the permissible shooting range CAR can be the same range as the overlapping area 221O. For example, the permissible shooting range CAR can be a range that includes a part of the overlapping area 221O. For example, the permissible shooting range CAR can be a range that includes the entire overlapping area 221O. For example, the permissible shooting range CAR can be the range included by the overlapping area 221O.

[0221] When the shooting device 22 is included in such Figure 11 When the object OBJ is captured within the permitted shooting range CAR shown, both the left camera 22L and the right camera 22R can capture images of the object OBJ. As a result, the control device 3 can appropriately calculate the parallax based on the image data IMG_3D representing the images generated by the left camera 22L and the right camera 22R. Therefore, the control device 3 can appropriately generate three-dimensional position data WSD based on the parallax calculation results.

[0222] The larger the field of view of the shooting device 22 (i.e., at least one of the field of view 221L of the left camera 22L and the field of view 221R of the right camera 22R), the larger the size of the allowed shooting range (CAR) can be. Therefore, in order to set the desired size of the allowed shooting range (CAR), the field of view of the shooting device 22 can be enlarged. However, if the field of view of the shooting device 22 is enlarged while the number of pixels of the shooting element provided by the shooting device 22 remains unchanged, the resolution of the shooting device 22 will decrease. As a result, the accuracy of the position and pose data (POI) may deteriorate. Therefore, when the field of view of the shooting device 22 is enlarged, the number of pixels of the shooting element provided by the shooting device 22 can be increased. In other words, the field of view of the shooting device 22 and the number of pixels of the shooting element provided by the shooting device 22 can be adjusted in a coordinated manner so that the resolution of the shooting device 22 meets the desired reference. In addition, when the field of view of the shooting device 22 is enlarged, in addition to increasing the number of pixels of the shooting element provided by the shooting device 22, or as an alternative, the pixel size of the shooting element provided by the shooting device 22 can also be reduced. In other words, the field of view of the shooting device 22 and the pixel size of the shooting element of the shooting device 22 can be adjusted in a coordinated manner so that the resolution of the shooting device 22 meets the desired benchmark.

[0223] Figure 12 This is a cross-sectional view illustrating a second example of the permissible shooting range (CAR) determined based on the field of view of the shooting device 22. In this second example, the permissible shooting range (CAR) may be a range determined based on a portion of the field of view that is part of the field of view of the shooting device 22. Specifically, the permissible shooting range (CAR) may be a range determined based on the overlapping area 222O of the field of view 222L (partial field of view 222L) of the left camera 22L and the overlapping area 222R (partial field of view 221R) of the right camera 22R. For example, the permissible shooting range (CAR) may be the same range as the overlapping area 222O. For example, the permissible shooting range (CAR) may be a range that includes a portion of the overlapping area 222O. For example, the permissible shooting range (CAR) may be a range that includes the entire overlapping area 222O. For example, the permissible shooting range (CAR) may be the range encompassed by the overlapping area 222O.

[0224] When the shooting device 22 is included in such Figure 12 When the object OBJ within the shooting allowable range CAR shown is photographed, the shooting device 22 is used to photograph the object containing the object OBJ. Figure 11 Similarly, in the case where the object OBJ is captured within the permitted shooting range CAR, both the left camera 22L and the right camera 22R can capture images of the object OBJ. As a result, the control device 3 can appropriately calculate the parallax based on the image data IMG_3D representing the images generated by the left camera 22L and the right camera 22R. Therefore, the control device 3 can appropriately generate three-dimensional position data WSD based on the parallax calculation result.

[0225] Figure 12 The shown partial fields of view 222L and 222R can be the fields of view corresponding to the search areas 224L and 224R used in stereo matching when calculating parallax based on image data IMG_3D. In this case, the shooting allowable range CAR can be considered as the range determined based on search areas 224L and 224R. (See below for reference.) Figure 13 (a) and Figure 13 (b) The search regions 224L and 224R used in stereo matching are described. Specifically, Figure 13 (a) illustrates the relationship between the field of view 221L and a portion of the field of view 222L of the left camera 22L and the image 223L generated by the left camera 22L. For example... Figure 13 As shown in (a), the left camera 22L generates an image 223L containing objects captured within the field of view 221L. Furthermore, as... Figure 13As shown in (a), image 223L includes a search area 224L that captures objects contained within a portion of the field of view 221L, namely a portion of the field of view 222L. On the other hand, Figure 13 (b) illustrates the relationship between the field of view 221R and a portion of the field of view 222R of the right camera 22R and the image 223R generated by the right camera 22R. Figure 13 As shown in (b), the right camera 22R generates an image 223R that captures objects contained within the field of view 221R. Furthermore, as... Figure 13 As shown in (b), image 223R includes a search area 224R that captures objects contained within a portion of the field of view 221R, namely, a partial field of view 222R. To calculate disparity based on the image data IMG_3D representing images 223L and 223R, control device 3 performs stereo matching of the same portion BP of the target object OBJ in each of search areas 224L and 224R, and calculates the deviation between the position of the portion BP detected in search area 224L and the position of the same portion BP detected in search area 224R as disparity. Control device 3 can calculate multiple disparities by calculating the deviations of multiple different portions BP between search areas 224L and 224R. That is, control device 3 can calculate the disparity of multiple parts of the target object OBJ (or any object). Any part of the target object OBJ can be used as a portion BP. A characteristic part of the target object OBJ can be used as a portion BP. A projection pattern projected onto the target object OBJ by projection device 23 can be used as a portion BP. Thus, partial fields of view 222L and 222R can be the fields of view corresponding to search areas 224L and 224R, respectively.

[0226] Given that the fields of view 222L and 222R correspond to the search areas 224L and 224R respectively, when the shooting device 22 takes a picture of the object OBJ contained within the shooting allowable range CAR, such as Figure 14 As shown in the upper part of (a), the object OBJ (specifically, part BP of the object OBJ) is contained within both partial fields of view 222L and 222R. The result is as follows: Figure 14 As shown in the lower part of (a), the same portion BP of the object OBJ is captured in both the search area 224L of image 223L and the search area 224R of image 223R. Therefore, the control device 3 can appropriately calculate the disparity based on the image data IMG_3D representing images 223L and 223R. Therefore, the control device 3 can appropriately generate three-dimensional position data WSD based on the disparity calculation result.

[0227] On the other hand, given that partial fields of view 222L and 222R correspond to search areas 224L and 224R respectively, when the shooting device 22 shoots an object OBJ that is not included in the shooting allowable range CAR, such as Figure 14 As shown in the upper part of (b), the object OBJ (partially the BP portion of the object OBJ) may not be included in at least one of the partial fields of view 222L and 222R. Furthermore, Figure 14 (b) shows an example where the object OBJ (specifically, part BP of the object object OBJ) is not included in both partial fields of view 222L and 222R. The result is as follows: Figure 14 As shown in the lower part of (b), in at least one of the search areas 224L and 224R of image 223L, it is possible that the same portion BP of the object OBJ is not captured. In this case, the control device 3 may be unable to properly calculate the disparity based on the image data IMG_3D representing images 223L and 223R. Therefore, the control device 3 may be unable to properly generate the three-dimensional position data WSD based on the disparity calculation results.

[0228] Here, in Figure 14 In the situation shown in (b), by expanding at least one of the search regions 224R and 224R, it is possible to capture the same portion BP of the object OBJ in both the search region 224L of image 223L and the search region 224R of image 223R. However, expanding at least one of the search regions 224R and 224R will increase the time required to search for the portion BP within the search regions 224R and 224R. That is, it will increase the time required to generate the three-dimensional position data WSD.

[0229] However, in this embodiment, since the shooting height CH of the shooting system 2 is adjusted to include the object OBJ within the shooting allowable range CAR determined based on search areas 224R and 224R, the control device 3 can achieve a state where the same portion BP of the object OBJ is captured in both the search area 224L of image 223L and the search area 224R of image 223R. Therefore, the increase in the time required to generate the three-dimensional position data WSD is suppressed. In other words, since the shooting height CH of the shooting system 2 is adjusted to include the object OBJ within the shooting allowable range CAR determined based on search areas 224R and 224R, the time required to generate the three-dimensional position data WSD can be shortened. Figure 14 Compared to the case where the shooting system 2 shown in (b) shoots the object OBJ, it can shorten the time required to generate the three-dimensional position data WSD.

[0230] The permissible shooting range (CAR) can be automatically set by the control device 3. For example, the control device 3 can acquire information related to the specifications of the shooting system 2 and set the permissible shooting range (CAR) based on the acquired information. For example, the control device 3 can acquire information related to the depth of field of the shooting system 2 as information related to the specifications of the shooting system 2, and set the permissible shooting range (CAR) based on the acquired depth-of-field information. For example, the control device 3 can acquire information related to the optimal focus position (BFP) of the object-side optical system of the shooting system 2 as information related to the specifications of the shooting system 2, and set the permissible shooting range (CAR) based on the acquired information related to the optimal focus position (BFP). For example, the control device 3 can acquire information related to the depth of focus of the shooting system 2 as information related to the specifications of the shooting system 2, and set the permissible shooting range (CAR) based on the acquired depth-of-focus information. For example, the control device 3 can acquire information related to the field of view of the shooting system 2 (e.g., the field of view 221L of the left camera 22L and the field of view 221R of the right camera 22R) as information related to the specifications of the shooting system 2, and set the shooting allowable range CAR based on the acquired field of view related information. For example, the control device 3 can acquire information related to a portion of the field of view of the shooting system 2 (e.g., a portion of the field of view 222L of the left camera 22L and a portion of the field of view 222R of the right camera 22R) as information related to the specifications of the shooting system 2, and set the shooting allowable range CAR based on the acquired partial field of view related information. For example, the control device 3 can acquire information related to the search areas 224L and 224R used in stereo matching as information related to the specifications of the shooting system 2, and set the shooting allowable range CAR based on the acquired information related to the search areas 224L and 224R.

[0231] The allowed shooting range (CAR) can also be set based on information input by the user. For example, the user can input information related to the specifications of the shooting system 2 into the control device 3 and set the allowed shooting range (CAR) based on the input information. For example, the user can input information related to the depth of field of the shooting system 2 as information related to the specifications of the shooting system 2, and set the allowed shooting range (CAR) based on the input depth-of-field information. For example, the user can input information related to the optimal focus position (BFP) of the object-side optical system of the shooting system 2 as information related to the specifications of the shooting system 2, and set the allowed shooting range (CAR) based on the input information related to the optimal focus position (BFP). For example, the user can input information related to the depth of focus of the shooting system 2 as information related to the specifications of the shooting system 2, and set the allowed shooting range (CAR) based on the input depth-of-focus information. For example, the user can input information related to the field of view of the shooting system 2 (e.g., the field of view 221L of the left camera 22L and the field of view 221R of the right camera 22R) as information related to the specifications of the shooting system 2, and set the allowed shooting range (CAR) based on the input field of view information. For example, a user can input information related to a portion of the field of view of the shooting system 2 (e.g., a portion of the field of view 222L of the left camera 22L and a portion of the field of view 222R of the right camera 22R) as information related to the specifications of the shooting system 2, and set the allowed shooting range CAR based on the input information related to the portion of the field of view. For example, a user can input information related to the search areas 224L and 224R used in stereo matching as information related to the specifications of the shooting system 2, and set the allowed shooting range CAR based on the input information related to the search areas 224L and 224R. Alternatively, for example, a user can input information for directly or indirectly specifying the allowed shooting range CAR into the control device 3, and set the allowed shooting range CAR based on the input information.

[0232] The control device 3 can use at least one of image data IMG_2D and IMG_3D generated by the shooting system 2 capturing at least one object OBJ contained in the container CB to determine the target shooting height CH_target. That is, the control device 3 can adjust the shooting height CH of the shooting system 2 based on at least one of the image data IMG_2D and IMG_3D. Specifically, the control device 3 can adjust the shooting height CH based on... Figure 4 The target shooting height CH_target is determined by at least one of the image data IMG_2D and IMG_3D obtained in step S1. However, as detailed in the first variation described later, the control device 3 can also obtain the target shooting height CH_target from the image data IMG_2D and IMG_3D obtained in step S1. Figure 4In step S1, at least one of the image data IMG_2D and IMG_3D is obtained, and the target shooting height CH_target is determined based on at least one of the obtained image data IMG_2D and IMG_3D.

[0233] Furthermore, in the control device 3 based on Figure 4 In step S1, if the target shooting height CH_target is determined by at least one of the image data IMG_2D and IMG_3D obtained last, the control device 3 can... Figure 4 In step S1, after acquiring at least one of the image data IMG_2D and IMG_3D, at any given time, the target shooting height CH_target is determined. For example, the control device 3 can... Figure 4 After obtaining at least one of image data IMG_2D and IMG_3D in step S1, and in Figure 4 Before the process of adjusting the shooting height CH in step S7 begins, the target shooting height CH_target is predetermined. Furthermore, as described later, the method used to determine the target shooting height CH_target (specifically, calculating the height OH of the object group OBG) is... Figure 4 In the case of the three-dimensional position data WSD generated in step S2, the control device 3 can... Figure 4 In step S2, after generating the three-dimensional position data WSD, arbitrary timing is used to determine the target shooting height CH_target. For example, control device 3 can... Figure 4 After generating the three-dimensional position data WSD in step S2, and in Figure 4 Before the process of adjusting the shooting height CH in step S7 begins, the target shooting height CH_target is predetermined. Alternatively, the control device 3 can also... Figure 4 After the process of adjusting the shooting height CH in step S7 begins, the target shooting height CH_target is determined.

[0234] The control device 3 first calculates the height OH of the object OBJ contained in the container CB based on at least one of the image data IMG_2D and IMG_3D, and then determines the target shooting height CH_target based on at least one of the image data IMG_2D and IMG_3D. Here, as described above, adjusting the shooting height CH of the shooting system 2 is as follows... Figure 15As shown, robot 1 sequentially processes multiple object objects OBJ stored in container CB according to a prescribed procedure. Therefore, control device 3 can calculate the height OH of the multiple object objects OBJ stored in container CB based on at least one of image data IMG_2D and IMG_3D.

[0235] The height OH of the multiple object objects OBJ contained in container CB can refer to the height OH of the object group OBG that includes all of the multiple object objects OBJ contained in container CB. Alternatively, the height OH of the multiple object objects OBJ contained in container CB can refer to the height OH of the object group OBG that includes a portion of the multiple object objects OBJ contained in container CB. In other words, the height OH of the multiple object objects OBJ contained in container CB can refer to the height OH of the object group OBG that includes at least a portion of the multiple object objects OBJ contained in container CB.

[0236] Furthermore, in this embodiment, the term "object group OBG" is used to treat multiple object objects OBJ as a group. Multiple object objects OBJ included in an object group OBG may include at least two object objects OBJ concentrated in a local area. Multiple object objects OBJ included in an object cluster OBG may include at least two object objects OBJ separated by a predetermined distance. That is, multiple object objects OBJ included in an object group OBG may include at least two object objects OBJ respectively distributed in at least two areas separated by a predetermined distance.

[0237] Considering the case described above where at least one of image data IMG_2D and IMG_3D is used to calculate the height OH of the object group OBG, the height OH of the object group OBG is equivalent to the height OH of the plurality of object objects OBJ captured in the image shown by at least one of the image data IMG_2D and IMG_3D. In other words, the height OH of the object group OBG is equivalent to the height OH of the object group OBG that includes at least a portion of the plurality of object objects OBJ contained in the container CB that is within the field of view of the shooting system 2.

[0238] Similar to the shooting height CH of the aforementioned shooting system 2, the height OH of the object group OBG can refer to the height OH of the object group OBG from the reference point. Furthermore, the height OH of the object group OBG can be the Z-axis coordinate value of the object group OBG in the global coordinate system. Alternatively, the height OH of the object group OBG can be the Z-axis coordinate value of the object group OBG in a coordinate system different from the global coordinate system (e.g., the robot coordinate system).

[0239] Furthermore, the reference point for the height OH of the object OBG can be the same as the reference point for the shooting height CH of the shooting system 2. Alternatively, the reference point for the height OH of the object OBG can be different from the reference point for the shooting height CH of the shooting system 2. For example, the height of the object OBG can refer to the height from a first reference point (e.g., a point on the bottom surface of the bottom wall BS of container CB), and the height of the object group OBG can also refer to the height from a second reference point different from the first reference point (e.g., a point on object OBJ).

[0240] The height OH of an object group OBG can be used as the height of the highest point within that object group. For example, the height OH of the highest object OBJ within the object group OBG can be used as the height of the object group OBG. Alternatively, the height OH of the highest point of the highest object OBJ within the object group OBG can be used as the height of the object group OBG. Finally, the height OH of the height of any point within the highest object OBJ within the object group OBG can be used as the height of the object group OBG.

[0241] The height OH of the object group OBG can be used as the height of the highest point among a portion of the object group OBG. For example, the height OH of the object group OBG can be used as the height of at least one object OBJ selected from among the multiple object OBJs included in the object group OBG according to a desired selection criterion. For example, the height OH of the object group OBG can be used as the height of the highest point among among the multiple object OBJs included in the object group OBG according to a desired selection criterion. For example, the height OH of the object group OBG can be used as the height of any point among among the multiple object OBJs included in the object group OBG according to a desired selection criterion.

[0242] The average height of an object group OBG can be used as the height OH of the object group OBG. The average height of the object group OBG can refer to the average height of multiple parts of the object group OBG. The average height of the object group OBG can also refer to the average height of multiple object objects OBJ included in the object group OBG. In the case of multiple object objects OBJ stacked (i.e., in bulk), the average height of the object group OBG can refer to the average height of a portion of the object objects OBJ included in the object group OBG that are not topped by other object objects OBJ (i.e., the topmost portion of object objects OBJ). In the case of multiple object objects OBJ stacked (i.e., in bulk), the average height of the object group OBG can refer to the average height of a portion of the object objects OBJ included in the object group OBG that have at least a portion of their upper part protruding upwards (i.e., the topmost portion of object objects OBJ).

[0243] The average height of a subset of object groups OBG can be used as the height OH of object group OBG. The average height of a subset of object groups OBG can refer to the average height of at least one object OBJ selected from the multiple object objects OBJ included in object group OBG according to the desired selection criteria.

[0244] As described above, control device 3 uses at least one of image data IMG_2D and IMG_3D to calculate the height OH of such object group OBG. For example, control device 3 can calculate the height OH of object group OBG by analyzing an image shown by at least one of image data IMG_2D and IMG_3D. Alternatively, control device 3 can calculate at least one of the position and pose of object objects OBJ included in object group OBG by performing 2D matching processing using image data IMG_2D, and calculate the height OH of object group OBG based on the calculation result of at least one of the position and pose of object objects OBJ. Alternatively, control device 3 can, based on the calculation result of at least one of the position and pose of object objects OBJ calculated by 2D matching processing, configure a three-dimensional model of object object OBJ in a virtual simulation space simulating a three-dimensional space in which object group OBG is configured, and calculate the height of the three-dimensional model as the height OH of object group OBG. Alternatively, besides analyzing the image shown in at least one of the image data IMG_2D and IMG_3D, or as an alternative, the control device 3 can use the three-dimensional position data WSD generated from the image data IMG_3D to calculate the height OH of the object group OBG. As an example, the control device 3 can calculate at least one of the positions and orientations of the object objects OBJ included in the object group OBG by performing 3D matching processing using the three-dimensional position data WSD, and calculate the height OH of the object group OBG based on the calculation result of at least one of the positions and orientations of the object objects OBJ. As another example, the control device 3 can, based on the calculation result of at least one of the positions and orientations of the object objects OBJ calculated by the 3D matching processing, configure the three-dimensional model of the object object OBJ in a virtual simulation space simulating the three-dimensional space in which the object group OBG is configured, and calculate the height of the three-dimensional model as the height OH of the object group OBG. Furthermore, when using the three-dimensional position data WSD to calculate the height OH of the object group OBG, the control device 3 can continue to use... Figure 4 The three-dimensional position data WSD generated in step S2 is used to calculate the height OH of the object group OBG. The control device 3 can also operate independently of... Figure 4 The three-dimensional position data WSD generated in step S2 is used to calculate the height OH of the object group OBG based on the newly generated three-dimensional position data WSD.

[0245] When calculating the height OH of an object group OBG using 3D position data WSD, the control device 3 can calculate the height of the point cloud shown in the 3D position data WSD (specifically, the height of the point cloud corresponding to the object group OBG) as the height OH of the object group OBG. Specifically, the control device 3 can calculate the height of at least a portion of the multiple points contained in the point cloud shown in the 3D position data WSD as the height OH of the object group OBG. For example, a cross-sectional view showing the point cloud shown in the 3D position data WSD... Figure 16 As shown in (a), the control device 3 can calculate the height of the highest point among the multiple points contained in the point cloud shown in the three-dimensional position data WSD as the height OH of the object group OBG. In this case, the control device 3 can calculate the height of the highest part in the object group OBG as the height OH of the object group OBG. Alternatively, for example, the control device 3 can calculate the height of the point at the Nth (where N is a constant representing an integer greater than 2) high position among the multiple points contained in the point cloud shown in the three-dimensional position data WSD as the height OH of the object group OBG. In this case, the control device 3 can calculate the height of one object OBJ at the Nth high position among the multiple object objects OBJ included in the object group OBG as the height OH of the object group OBG. Alternatively, for example, the control device 3 can calculate the average height of at least a portion of the heights of the multiple points contained in the point cloud shown in the three-dimensional position data WSD as the height OH of the object group OBG. In this case, the control device 3 can calculate the average height of at least a portion of the object group OBG as the height OH of the object group OBG. As an example, control device 3 can calculate the average height of all points contained in the point cloud shown by the three-dimensional position data WSD as the height OH of the object group OBG. As another example, a cross-sectional view representing the point cloud shown by the three-dimensional position data WSD is shown below. Figure 16 As shown in (b), the control device 3 can calculate the average height of a portion of the point cloud contained in the point cloud shown in the three-dimensional position data WSD, which is contained within a specified point cloud space PCSP, as the height OH of the object group OBG. As an example of a specified point cloud space PCSP, such as... Figure 16 As shown in (b), a space can be enumerated that includes the point located at the highest (or Nth highest) position among multiple points contained in the point cloud shown by the three-dimensional position data WSD. For example, the point cloud space PCSP can be a three-dimensional space that includes the following: a virtual surface (e.g., a horizontal plane) containing the point located at the highest (or Nth highest) position (the highest point) among multiple points contained in the point cloud shown by the three-dimensional position data WSD, and a virtual surface (e.g., a horizontal plane) located at a position that decreases by a specified height along the Z-axis from that point (the highest point).

[0246] When calculating the height OH of the object group OBG using image data IMG_2D, the control device 3 can segment the image data IMG_2D to calculate the height OH of the object group OBG as the height of the object group OBG. Alternatively, the control device 3 can perform 2D matching processing using image data IMG_2D to calculate the position and orientation of the object OBJ, and then calculate the calculated position (specifically, the position in the Z-axis direction, i.e., the height) of the object OBJ as the height OH of the object group OBG. Alternatively, the control device 3 can use the position and orientation of the object OBJ calculated through 2D matching processing, and the template model TM3 of the object OBJ, to calculate the height (i.e., the position in the Z-axis direction) of the object group OBG.

[0247] After calculating the height OH of the object group OBG, the control device 3 can determine the target shooting height CH_target based on the calculated height OH. Specifically, the control device 3 can determine the target shooting height CH_target based on the calculated height OH of the object group OBG and the shooting allowable range CAR of the shooting system 2. For example, a cross-sectional view showing the relationship between the height OH of the object group OBG and the shooting allowable range CAR of the shooting system 2 is shown below. Figure 17 As shown, the control device 3 can determine the target shooting height CH_target so that the calculated height OH is included within the shooting allowable range CAR of the shooting system 2 located at the target shooting height CH_target. That is, as... Figure 17 As shown, the control device 3 can determine the target shooting height CH_target, so that the shooting range CAR of the shooting system 2 located at the target shooting height CH_target includes the part of the object group OBG located at the calculated height OH. Figure 17 As shown, the control device 3 can determine the target shooting height CH_target to include at least a portion of the object OBJ contained in the object group OBG and located at the calculated height OH within the shooting allowable range CAR of the shooting system 2 located at the target shooting height CH_target. The result is as follows: Figure 17As shown, the target shooting height CH_target is determined to satisfy the condition that the shooting range CAR of the shooting system 2 located at the target shooting height CH_target includes at least one object OBJ. Furthermore, the control device 3 can determine the target shooting height CH_target such that at least a portion of the object OBJ located at the calculated height OH is located at the optimal focus position BFP of the shooting system 2 at the target shooting height CH_target.

[0248] After determining the target shooting height CH_target, the control device 3 (particularly the signal generation unit 313) generates a robot control signal to control the robot 1 so that the shooting system 2 is positioned at the determined target shooting height CH_target. In other words, the control device 3 (particularly the signal generation unit 313) generates a robot control signal to control the robot 1 so that the shooting height CH of the shooting system 2 is consistent with the determined target shooting height CH_target. As a result, the robot 1 moves the shooting system 2 according to the robot control signal, thereby completing the adjustment of the shooting height CH of the shooting system 2. That is, the shooting system 2 moves according to the robot control signal, thereby completing the adjustment of the shooting height CH of the shooting system 2.

[0249] Whenever robot 1 completes the prescribed processing of the object to be processed, control device 3 performs the above-described adjustment of the shooting height CH of shooting system 2. Here, if the prescribed processing performed by robot 1 includes holding one of the multiple object objects OBJ stored in container CB, and releasing the held object object OBJ to a release position outside container CB after holding it, the number of object objects OBJ stored in container CB decreases whenever robot 1 completes the prescribed processing of the object to be processed. Therefore, as time passes, Figure 4 The height OH of the object group OBG calculated in step S7 is likely to decrease. As a result, over time, Figure 4 In step S7, the target shooting height CH_target calculated based on the height OH of the object group OBG may also become lower.

[0250] For example, Figure 18 (a) shows an example where, at time 1, the height OH of the object group OBG contained in container CB is height OH#1. In this case, Figure 4In step S7, the control device 3 can adjust the shooting height CH of the shooting system 2 so that the shooting height CH of the shooting system 2 becomes the first target shooting height CH_target#1 determined based on the first height OH#1. Figure 18 As shown in (a), the first target shooting height CH_target#1 satisfies the condition that the shooting range CAR of the shooting system 2 located at the first target shooting height CH_target#1 includes the first height OH#1.

[0251] Subsequently, during the period from time 1 to time 2 later than time 1, robot 1 holds at least one object OBJ contained in container CB and releases the held object OBJ to a release position outside container CB. That is, robot 1 removes at least one object OBJ from container CB. The result is as follows: (The last sentence appears to be incomplete and possibly refers to a different context.) Figure 18 As shown in (b), the number of object OBJs included in the object group OBG in container CB at time 2 is less than the number of object OBJs included in the object group OBG in container CB at time 1. As a result, the height OH of the object group OBG in container CB at time 2 is likely to be a second height OH#2, which is lower than the height of the object OBJs included in the object group OBG in container CB at time 1, i.e., the first height OH#1.

[0252] Here, as Figure 18 As shown in (b), assuming that the shooting height CH of shooting system 2 is not adjusted at time 2, the shooting height CH of shooting system 2 remains unchanged at the first target shooting height CH_target#1. The result is as follows: Figure 18 As shown in (b), at time 2, the object group OBG contained in container CB may not be included within the allowed shooting range CAR of shooting system 2 located at the first target shooting height CH_target#1. This is because, during the period from time 1 to time 2, object OBJ contained within the allowed shooting range CAR of shooting system 2 located at the first target shooting height CH_target#1 is moved out of container CB by robot 1. As a result, when the object group OBG contained in container CB is not included within the allowed shooting range CAR of shooting system 2, the accuracy of position and pose data POI generated based on at least one of image data IMG_2D and IMG_3D generated by shooting object group OBG through shooting system 2 may deteriorate. That is, the calculation accuracy of at least one of the position and pose of object OBJ included in object group OBG may deteriorate. The technical reasons are explained below.

[0253] First, the technical reasons for the deterioration in accuracy of Point of Interest (POI) data generated from IMG_3D image data are explained. As mentioned above, in order to generate POI data from IMG_3D image data, as described above... Figure 13 (a) and Figure 13 As shown in (b), the control device 3 first acquires image data IMG_3D representing image 223L captured by the left camera 22L and image 223R captured by the right camera 22R. Then, the control device 3 searches for the same portion BP of the object OBJ in each of the search areas 224L and 224R of image 223L, and calculates the discrepancy between the position of the portion BP detected in search area 224L and the position of the same portion BP detected in search area 224R. Here, if the object OBJ captured by the shooting system 2 is not included in the allowed shooting range CAR, compared to the case where the object OBJ captured by the shooting system 2 is included in the allowed shooting range CAR, the object OBJ may not be clearly captured in each of images 223L and 223R. For example, the object OBJ may be captured in each of images 223L and 223R in a blurred state. Therefore, when the object OBJ captured by imaging system 2 is not included within the allowed shooting range CAR, the accuracy of stereo matching is lower compared to when the object OBJ is included within the allowed shooting range CAR. Lower stereo matching accuracy leads to lower accuracy in disparity calculation based on the portion of the BP detected through stereo matching. Consequently, the 3D position data WSD generated based on the low-accuracy disparity may not accurately represent the 3D position of the object OBJ. Consequently, the position and pose data POI generated based on the low-accuracy 3D position data WSD may not accurately represent at least one of the position and pose of the object OBJ.

[0254] Next, the technical reason for the deterioration in accuracy of the position and pose data (POI) generated based on the image data IMG_2D will be explained. In this case, if the object OBJ captured by the imaging system 2 is not included within the allowed shooting range (CAR), the object OBJ may not be clearly captured in the image shown by the image data IMG_2D compared to when the object OBJ is included within the allowed shooting range (CAR). For example, the object OBJ may be captured in the image shown by the image data IMG_2D in a blurred state. Therefore, when the object OBJ captured by the imaging system 2 is not included within the allowed shooting range (CAR), the accuracy of the 2D matching process for searching for the same image portion as the template image within the image shown by the image data IMG_2D becomes lower compared to when the object OBJ is included within the allowed shooting range (CAR). As a result, the position and pose data (POI) generated as a result of the low-accuracy 2D matching process may not accurately show at least one of the position and pose of the object OBJ.

[0255] However, in this embodiment, such as Figure 19 As shown, the shooting height CH of the shooting system 2 is adjusted according to the change in the height OH of the object group OBG. Specifically, in Figure 4 In step S7, the control device 3 adjusts the shooting height CH of the shooting system 2 so that the shooting height CH of the shooting system 2 changes from the first target shooting height CH_target#1 determined based on the first height OH#1 to the second target shooting height CH_target#2 determined based on the second height OH#2. Figure 19 As shown, the second target shooting height CH_target#2 satisfies the condition that the shooting range CAR of the shooting system 2 located at the second target shooting height CH_target#2 includes the second height OH#2. The result is as follows: Figure 19As shown, at the second moment, at least a portion of the object group OBG contained in the container CB is included within the shooting allowable range CAR of the shooting system 2 located at the second target shooting height CH_target#2. That is, object OBJs not included within the shooting allowable range CAR of the shooting system 2 located at the first target shooting height CH_target#1 are included within the shooting allowable range CAR of the shooting system 2 located at the second target shooting height CH_target#2. As a result, the control device 3 can generate position and attitude data POI that accurately represents at least one of the position and attitude of the object OBJs included in the object group OBG, based on at least one of the image data IMG_2D and IMG_3D generated by the shooting system 2 when the object group OBG contained in the container CB is included within the shooting allowable range CAR of the shooting system 2. For example, when generating position and pose data (POI) based on image data IMG_3D, since the object OBJ captured by the imaging system 2 is included within the allowed shooting range (CAR), the object OBJ is captured more clearly in each of images 223L and 223R compared to the case where the object OBJ is not included in the allowed shooting range (CAR). Therefore, when the object OBJ captured by the imaging system 2 is included within the allowed shooting range (CAR), the accuracy of stereo matching is higher compared to the case where the object OBJ is not included in the allowed shooting range (CAR). If the accuracy of stereo matching is higher, the accuracy of disparity calculation based on the partial BP detected by stereo matching is also higher. As a result, the control device 3 can generate three-dimensional position data (WSD) representing the three-dimensional position of the object OBJ with higher accuracy based on the high-accuracy disparity. As a result, the control device 3 can generate position and pose data (POI) representing at least one of the position and pose of the object OBJ with higher accuracy based on the high-accuracy three-dimensional position data (WSD). As a result, the control device 3 is able to generate position and attitude data POI that represents at least one of the position and attitude of the object OBJ with higher precision, based on the high-precision position and attitude data POI.

[0256] For example, when generating position and pose data (POI) based on image data IMG_2D, since the object OBJ captured by the imaging system 2 is included within the allowed shooting range (CAR), the object OBJ is clearly captured in the image shown by the image data IMG_2D compared to the case where the object OBJ is not included in the allowed shooting range (CAR). Therefore, when the object OBJ captured by the imaging system 2 is included within the allowed shooting range (CAR), the accuracy of the 2D matching process is higher compared to the case where the object OBJ is not included in the allowed shooting range (CAR). As a result, the control device 3 can generate position and pose data (POI) that represents at least one of the position and pose of the object OBJ with higher accuracy, as a result of the high-precision 2D matching process.

[0257] Thus, in this embodiment, since the shooting height CH of the shooting system 2 is adjusted according to the change in the height OH of the object group OBG, the control device 3 can also generate position and attitude data POI that represents at least one of the position and orientation of the object OBJ with higher precision when the height OH of the object group OBG changes. For example, when the height OH of the object group OBG changes because the robot 1 sequentially moves multiple object OBJs stored in the container CB out of the container CB, the control device 3 can also generate position and attitude data POI that represents at least one of the position and orientation of the object OBJ with higher precision.

[0258] Therefore, when generating position and orientation data (POI) representing at least one of the positions and orientations of an object OBJ with higher precision, the control device 3 can control the robot 1 so that the end effector 4 appropriately performs prescribed processing on at least one object OBJ included in the object group OBG. For example, if the height OH of the object group OBG changes because the robot 1 sequentially moves multiple object OBJs contained in the container CB out of the container CB, the control device 3 can control the robot 1 so that the end effector 4 appropriately performs prescribed processing on at least one object OBJ included in the object group OBG. As an example, since at least one of the positions and orientations of the object OBJ is calculated with high precision, the control device 3 can control the robot 1 so that the actual positional relationship between the object OBJ and the end effector 4 is appropriately close to the ideal positional relationship when the end effector 4 performs prescribed processing on the object OBJ. As an example, since at least one of the position and orientation of the object OBJ is calculated with high precision, the control device 3 is able to control the robot 1 such that the deviation between the actual position of the end effector 4 relative to the object OBJ and the ideal position of the end effector 4 relative to the object OBJ when performing the specified processing on the object OBJ is reduced.

[0259] In addition, Figure 4 In the robot control process shown, the control device 3 adjusts the shooting height CH of the shooting system 2 whenever robot 1 completes the prescribed processing on a selected object OBJ. However, if only one object OBJ is removed from container CB (i.e., robot 1 holds one object OBJ and releases it to a release position outside container CB), the height OH of the object group OBG stored in container CB may not change significantly. In this case, the necessity of adjusting the shooting height CH of the shooting system 2 based on the change in the height OH of the object group OBG is not so high. Therefore, the control device 3 does not need to adjust the shooting height CH of the shooting system 2 every time robot 1 completes the prescribed processing on a single object. Specifically, the control device 3 can adjust the shooting height CH of the shooting system 2 every time robot 1 completes the prescribed processing on two or more objects. Even in this case, the aforementioned technical effect can still be enjoyed.

[0260] Furthermore, the user can use input device 34 to input the interval for adjusting the shooting height CH of the shooting system 2 (step S7). The input interval can be, for example, the number of objects processed after the shooting height CH of the shooting system 2 is adjusted, or the elapsed time since the shooting height CH of the shooting system 2 was adjusted. The user can use input device 34 to input the timing for adjusting the shooting height CH of the shooting system 2 (step S7).

[0261] In addition, Figure 4 In the robot control process shown, after the robot control process begins, before the imaging system 2 first captures a group of object objects OBG, including at least a portion of the multiple object objects OBJ contained in the container CB, the imaging height CH of the imaging system 2 is not adjusted. Specifically, the imaging height CH of the imaging system 2 is not adjusted based on the height OH of the object group OBG. In this case, the control device 3 can adjust the imaging height CH of the imaging system 2 so that the imaging height CH of the imaging system 2 becomes the initial imaging height. The initial imaging height can be automatically set by the control device 3 or set by the user. Alternatively, the imaging system 2 can operate independently of... Figure 4 In step S1, the control device 3 acquires at least one of the image data IMG_2D and IMG_3D to capture the object group OBG to generate at least one of the image data IMG_2D and IMG_3D for initial adjustment of the shooting height CH of the shooting system 2. The control device 3 can perform initial adjustment of the shooting height CH of the shooting system 2 based on at least one of the image data IMG_2D and IMG_3D for initial adjustment of the shooting height CH. Furthermore, the process of initial adjustment of the shooting height CH of the shooting system 2 can itself be combined with… Figure 4 The process of adjusting the shooting height CH of the shooting system 2 in step S7 is the same.

[0262] (2-3) Specific examples of robot control processing Next, a specific example of robot control processing will be explained. The following is a specific example of robot control processing performed when multiple object objects OBJ are contained in a container CB, and the robot 1 repeatedly holds one of the multiple object objects OBJ contained in the container CB and releases the held object object OBJ to a release position outside the container CB.

[0263] exist Figure 4 After the robot control process begins as shown, Figure 20 As shown in (a), the imaging system 2 captures images of the first object group OBG#11, which includes at least a portion of the multiple object objects OBJ contained in the container CB. Figure 4Step S1). In this case, such as Figure 20 As shown in (a), the shooting height CH of the shooting system 2 can be set to the initial shooting height, i.e., the first target shooting height CH_target#11. That is, the shooting system 2, located at the first target shooting height CH_target#11, can capture the first object group OBG#11. As a result, the control device 3 acquires image data IMG_3D (referred to as image data IMG_3D#11) from the shooting system 2, representing the image of the first object group OBG#11 captured. Figure 4 Step S1). Furthermore, in the following description, for ease of explanation, the object objects OBJ included in the first object object group OBG#11 will be referred to as the first object object OBJ#11. Then, the position and pose calculation unit 312 generates position and pose data POI (referred to as position and pose data POI#11) of one of the multiple first object objects OBJ#11 selected as the processing execution object, based on the image data IMG_3D#11 (specifically, based on the three-dimensional position data WSD generated from the image data IMG_3D#11). Figure 4 (Steps S2 to S3). Furthermore, in the following description, the first object OBJ#11 selected as the object to be processed is referred to as the first object OBJ#11_target.

[0264] The position and attitude calculation unit 312 is highly likely to select the first object OBJ#11, which is included within the allowed shooting range (CAR) of the shooting system 2 located at the first target shooting height CH_target#11, as the processing execution object, i.e., the first object OBJ#11_target. This is because there is a higher probability that the matching similarity of object OBJs included within the allowed shooting range (CAR) will be higher than that of object OBJs not included within the allowed shooting range (CAR). In this case, the position and attitude calculation unit 312 can generate position and attitude data (POI) that accurately represents at least one of the position and attitude of the first object OBJ#11 included within the allowed shooting range (CAR) of the shooting system 2 located at the first target shooting height CH_target#11. This is because the higher the matching similarity of the object OBJs, the higher the probability that the accuracy of the position and attitude data (POI) of the object OBJs will be. However, the position and attitude calculation unit 312 may also select the first object OBJ#11, which is not included in the shooting range CAR of the shooting system 2 located at the first target shooting height CH_target#11, as the processing execution object, namely the first object OBJ#11_target.

[0265] Subsequently, the signal generation unit 313 generates a robot control signal based on the position and orientation data POI#11 of the first object OBJ#11_target to control the robot 1 so that the end effector 4 approaches the first object OBJ#11_target. Figure 4 (Step S4). After the end effector 4 approaches the first object OBJ#11_target according to the robot control signal generated in step S4, in step S5, the robot 1 performs a prescribed process on the first object OBJ#11_target. That is, as a prescribed process, the robot 1 can perform the following process: hold the first object OBJ#11_target stored in the container CB, remove the held first object OBJ#11_target from the container CB, and release it to a release position outside the container CB.

[0266] After robot 1 performs the specified processing on the first object OBJ#11_target ( Figure 4 Step S5: Yes), control device 3 adjusts the shooting height CH of shooting system 2. Figure 4 (Step S7). Furthermore, as described above, the control device 3 can adjust the shooting height CH of the shooting system 2 after the robot 1 has performed the prescribed processing on a single first object OBJ#11. Alternatively, as described above, the control device 3 can adjust the shooting height CH of the shooting system 2 after the robot 1 has performed the prescribed processing on multiple first object OBJ#11s respectively. Hereinafter, for ease of explanation, an example of the control device 3 adjusting the shooting height CH of the shooting system 2 after the robot 1 has performed the prescribed processing on a single first object OBJ#11 will be described.

[0267] In order to adjust the shooting height CH of the shooting system 2, the control device 3 uses robot control signals to generate robot control signals to control the robot 1 so that the end effector 4 approaches the first object OBJ#11_target. Figure 4 The image data IMG_3D#11 acquired in step S1 (specifically, the 3D position data WSD generated from the image data IMG_3D#11) is used to calculate the height OH of the first object group OBG#11. The result is as follows: Figure 20 As shown in (b), the control device 3 calculates the first height OH#11 as the height OH of the first object group OBG#11.

[0268] In addition, Figure 20In the example shown in (a), at the moment the first object group OBG#11 is captured, the first object OBJ#11_target, which has not yet been removed from container CB by robot 1, is located at the highest position in the first object group OBG#11. Therefore, the control device 3 calculates the height of the first object OBJ#11_target as the height OH of the first object group OBG#11. However, as Figure 20 As shown in (b), at the moment when the shooting height CH of the shooting system 2 is actually adjusted based on the height OH of the first object group OBG#11, the first object OBJ#11_target has been removed from the container CB. Therefore, during the process... Figure 4 In the case of robot control processing shown, Figure 4 The height OH of the first object group OBG#11 calculated in step S7 may contain errors relative to the actual height OH of the object group OBG contained in container CB at the moment when the shooting height CH of the shooting system 2 is actually adjusted based on the height OH of the first object group OBG#11. However, even in this case, the height OH of the object group OBG contained in container CB is still within the range of the calculated height OH. Figure 4 The possibility that the error in the height OH of the first object group OBG#11 calculated in step S7 is large enough to adversely affect the adjustment of the shooting height CH of the shooting system 2 is low. However, even when performing... Figure 4 In the case of robot control processing shown, Figure 4 The height OH of the first object group OBG#11 calculated in step S7 may sometimes be consistent with the height OH of the object group OBG actually stored in container CB at the moment when the shooting height CH of the shooting system 2 is actually adjusted based on the height OH of the first object group OBG#11.

[0269] Furthermore, if at least one of the image data IMG_2D and IMG_3D used to calculate the height OH of the object group OBG contains information related to the object OBJ that has been removed from the container CB at the moment when the shooting height CH of the shooting system 2 is actually adjusted based on the height OH of the object group OBG, the control device 3 can perform a removal process to remove the information related to the object OBJ that has been removed from the container CB. For example, when calculating the height OH of the object group OBG based on the three-dimensional position data WSD (e.g., point cloud data) generated according to the image data IMG_3D, the control device 3 can perform a removal process that removes from the three-dimensional position data WSD (e.g., point cloud data) the three-dimensional position data (e.g., the point cloud data corresponding to the object OBJ that has been removed from the container CB) from the object OBJ that has been removed from the container CB at the moment when the shooting height CH of the shooting system 2 is actually adjusted based on the height OH of the object group OBG. In this case, the three-dimensional position data WSD that has undergone the removal process and the information related to the object OBJ that has been removed from the container CB at the moment when the shooting height CH of the shooting system 2 is actually adjusted based on the height OH of the object group OBG are removed. Figure 24 The 3D position data WSD generated from the image data IMG_3D acquired in step S8a is essentially the same. Afterwards, the control device 3 can calculate the height OH of the object group OBG based on the 3D position data WSD (e.g., point cloud data) that has undergone removal processing. In this case, the error in calculating the height OH of the object group OBG is smaller compared to the case without removal processing. Furthermore, it is not limited to... Figure 20 (a) to Figure 20 In the scenario shown in (b), in any scenario where the height OH of the object group OBG is calculated, the control device 3 can perform the removal process to remove information related to the object OBJ that has been moved out of the container CB.

[0270] Subsequently, based on the calculated first height OH#11 and the shooting allowable range CAR of the shooting system 2, the control device 3 determines the target shooting height CH_target of the shooting system 2. Specifically, as follows... Figure 20 As shown in (b), the control device 3 can determine the target shooting height CH_target as the second target shooting height CH_target#12, satisfying the condition that "the calculated first height OH#11 is included within the shooting allowable range CAR of the shooting system 2 located at the target shooting height CH_target". Then, the control device 3 generates a robot control signal to control the robot 1 so that the shooting height CH of the shooting system 2 becomes the second target shooting height CH_target#12 determined based on the first height OH#11. As a result, the shooting system 2 is located at the second target shooting height CH_target#12.

[0271] After that, as Figure 21 As shown in (a), the shooting system 2, located at the second target shooting height CH_target#12, shoots the second object group OBG#12, which includes at least a portion of the multiple object objects OBJ contained in the container CB. Figure 4 Step S1). As a result, the control device 3 acquires from the imaging system 2 image data IMG_3D (referred to as image data IMG_3D#12) representing an image of the second object group OBG#12 captured in the image. Figure 4 Step S1). Furthermore, in the following description, for ease of explanation, the object OBJ included in the second object group OBG#12 will be referred to as the second object OBJ#12.

[0272] The second object group OBG#12 captured by the shooting system 2 at the second target shooting height CH_target#12 may be different from the first object group OBG#11 captured by the shooting system 2 to determine the second target shooting height CH_target#12. Specifically, at least one of the plurality of first object objects OBJ#11 included in the first object group OBG#11 may not be included in the second object group OBG#12. As an example, at the moment when the shooting system 2 at the second target shooting height CH_target#12 captures the second object group OBG#12, at least one of the plurality of first object objects OBJ#11 included in the first object group OBG#11 should have been removed from the container CB by the robot 1. In this case, at least one first object object OBJ#11 that has been removed from the container CB may not be included in the second object group OBG#12. As another example, due to the adjustment of the shooting height CH of the shooting system 2, the field of view of the shooting system 2 relative to the object group OBG contained in the container CB may change. In this case, the second object group OBG#12 may be different from the first object group OBG#11.

[0273] However, at least one of the multiple first object objects OBJ#11 included in the first object group OBG#11 that has not yet been removed from the container CB can be included in the second object group OBG#12. That is, the first object group OBG#11 and the second object group OBG#12 can partially overlap. In this case, since the second target shooting height CH_target#12 of the shooting system 2 that shoots the second object group OBG#12 is determined based on the first height OH#11 of the first object group OBG#11, therefore... Figure 21 As shown in (a), the shooting range CAR of the shooting system 2 located at the second target shooting height CH_target#12 includes at least a portion of the second object group OBG#12 that partially overlaps with the first object group OBG#11. That is, the shooting range CAR of the shooting system 2 located at the second target shooting height CH_target#12 includes at least one of the plurality of second object objects OBJ#12 included in the second object group OBG#12. Therefore, the shooting system 2 is capable of appropriately shooting at least one of the plurality of second object objects OBJ#12 included in the second object group OBG#12. However, all of the plurality of second object objects OBJ#12 included in the second object group OBG#12 may be different from the plurality of first object objects OBJ#11 included in the first object group OBG#11.

[0274] Furthermore, as described above, since at least one first object OBJ#11 has been removed from container CB when the shooting system 2 shoots the second object group OBG#12 from the second target shooting height CH_target#12, the number of object OBJs stored in container CB when the shooting system 2 shoots the second object group OBG#12 from the second target shooting height CH_target#12 can be less than the number of object OBJs stored in container CB when the shooting system 2 shoots the first object group OBG#11 from the first target shooting height CH_target#11. The number of second object OBJs#12 included in the second object group OBG#12 captured by the shooting system 2 from the second target shooting height CH_target#12 can be less than the number of first object OBJs#11 included in the first object group OBG#11 captured by the shooting system 2 from the first target shooting height CH_target#11. The weight of the container CB containing multiple object objects OBJs at the moment when shooting system 2 shoots the second object group OBG#12 from the second target shooting height CH_target#12 can be lighter than the weight of the container CB containing multiple object objects OBJs at the moment when shooting system 2 shoots the first object group OBG#11 from the first target shooting height CH_target#11. Furthermore, the "weight of the container CB containing multiple object objects OBJs" mentioned here can include both the weight of the container CB itself and the weight of the multiple object objects OBJs contained within the container CB.

[0275] Subsequently, the position and attitude calculation unit 312 generates position and attitude data POI (referred to as position and attitude data POI#12) of one of the multiple second object objects OBJ#12 selected as the processing execution object, based on the image data IMG_3D#12 (specifically, based on the three-dimensional position data WSD generated from the image data IMG_3D#12). Figure 4 (Steps S2 to S3). Furthermore, in the following description, the second object OBJ#12 selected as the object to be processed is referred to as the second object OBJ#12_target.

[0276] The position and attitude calculation unit 312 is highly likely to select the second object OBJ#12, which is included within the shooting range (CAR) of the shooting system 2 located at the second target shooting height CH_target#12, as the processing execution object, i.e., the second object OBJ#12_target. This is because the matching similarity of the position and attitude data (POI) of the object OBJ included within the shooting range (CAR) is more likely to be higher than the matching similarity of the object OBJ not included within the shooting range (CAR). In this case, the position and attitude calculation unit 312 can generate position and attitude data (POI) that accurately represents at least one of the position and attitude of the second object OBJ#12 included within the shooting range (CAR) of the shooting system 2 located at the second target shooting height CH_target#12. This is because the higher the matching similarity of the object OBJ, the higher the likelihood that the accuracy of the position and attitude data (POI) of the object OBJ will be. Conversely, the second target shooting height CH_target#12 can be a shooting height CH that satisfies the condition that "the second object OBJ#12 selected as the processing execution object is included within the shooting allowable range CAR of the shooting system 2 located at the second target shooting height CH_target#12". That is, the control device 3 can determine the second target shooting height CH_target#12 to satisfy the condition that "the second object OBJ#12 selected as the processing execution object is included within the shooting allowable range CAR of the shooting system 2 located at the second target shooting height CH_target#12". However, the position and attitude calculation unit 312 can select a second object OBJ#12 that is not included within the shooting allowable range CAR of the shooting system 2 located at the second target shooting height CH_target#12 as the processing execution object, i.e., the second object OBJ#12_target.

[0277] On the other hand, the second object object OBJ#12_target selected as the object to be processed may not be included within the shooting range CAR of the shooting system 2 before the shooting height CH is adjusted. That is, the second object object OBJ#12_target selected as the object to be processed may not be included within the shooting range CAR of the shooting system 2 located at the shooting height CH_target#11 of the first target. This is because, as shown in the side reference... Figure 18 (a) to Figure 18 (b) and Figure 19 As explained, since the shooting height CH of the shooting system 2 is adjusted according to the height OH of the object group OBG contained in the container CB, even if an object OBJ is not included in the shooting allowable range CAR of the shooting system 2 at the first moment, it will be included in the shooting allowable range CAR of the shooting system 2 at the second moment, which is later than the first moment. As a result, the object OBJ can be selected as the object to be processed. That is to say, even if an object OBJ is not included in the shooting allowable range CAR of the shooting system 2 at the first target shooting height CH_target#11, it will be included in the shooting allowable range CAR of the shooting system 2 at the second target shooting height CH_target#12 as time goes by. As a result, the object OBJ can be selected as the object to be processed.

[0278] Furthermore, the second object object OBJ#12_target selected as the object to be processed can be the first object object OBJ#11 that is not included in the shooting range CAR of the shooting system 2 located at the shooting height CH_target#11 of the first target. Alternatively, the second object object OBJ#12_target selected as the object to be processed can be the first object object OBJ#11 that is included in the shooting range CAR of the shooting system 2 located at the shooting height CH_target#11 of the first target but was not selected as the object to be processed.

[0279] Subsequently, the signal generation unit 313 generates a robot control signal based on the position and orientation data POI#12 of the second object OBJ#12_target to control the robot 1 so that the end effector 4 approaches the second object OBJ#12_target. Figure 4(Step S4). After the end effector 4 approaches the second object OBJ#12_target according to the robot control signal generated in step S4, in step S5, the robot 1 performs the prescribed processing on the second object OBJ#12_target. That is, the robot 1 holds the second object OBJ#12_target stored in the container CB, removes the held second object OBJ#12_target from the container CB, and releases it to a release position outside the container CB.

[0280] After robot 1 performs the specified processing on the second object OBJ#12_target ( Figure 4 Step S5: Yes), control device 3 adjusts the shooting height CH of shooting system 2. Figure 4 (Step S7). Furthermore, as described above, the control device 3 can adjust the shooting height CH of the shooting system 2 after the robot 1 has performed the prescribed processing on a single second object OBJ#12. Alternatively, as described above, the control device 3 can adjust the shooting height CH of the shooting system 2 after the robot 1 has performed the prescribed processing on multiple second object OBJ#12 objects respectively. Hereinafter, for ease of explanation, an example will be described where the control device 3 adjusts the shooting height CH of the shooting system 2 after the robot 1 has performed the prescribed processing on a single second object OBJ#12.

[0281] In order to adjust the shooting height CH of the shooting system 2, the control device 3 uses robot control signals to generate robot control signals to control robot 1 so that the end effector 4 approaches the second object OBJ#12_target. Figure 4 The image data IMG_3D#12 acquired in step S1 (specifically, the 3D position data WSD generated from the image data IMG_3D#12) is used to calculate the height OH of the second object group OBG#12. The result is as follows: Figure 21 As shown in (b), the control device 3 calculates the second height OH#12 as the height OH of the second object group OBG#12.

[0282] Here, as described above, at the moment the second object group OBG#12 is photographed, at least one of the plurality of first object objects OBJ#11 included in the first object group OBG#11 has been removed from the container CB by robot 1. Therefore, at least one of the plurality of first object objects OBJ#11 included in the first object group OBG#11 is not included in the second object group OBG#12. That is, the number of second object objects OBJ#12 included in the second object group OBG#12 becomes less than the number of first object objects OBJ#11 included in the first object group OBG#11. As a result, the second height OH#12 of the second object group OBG#12 generally becomes lower than the first height OH#11 of the first object group OBG#11. However, depending on the circumstances, the second height OH#12 of the second object group OBG#12 may also be the same as the first height OH#11 of the first object group OBG#11.

[0283] In addition, Figure 21 In the example shown in (a), at the moment the second object group OBG#12 is captured, the second object OBJ#12_target, which has not yet been removed from container CB by robot 1, is located at the highest position in the second object group OBG#12. Therefore, control device 3 calculates the height of the second object OBJ#12_target as the height OH of the second object group OBG#12. However, as Figure 21 As shown in (b), at the moment when the shooting height CH of the shooting system 2 is actually adjusted based on the height OH of the second object group OBG#12, the second object OBJ#12_target has been removed from the container CB. Therefore, during the process... Figure 4 In the case of robot control processing shown, Figure 4 The height OH of the second object group OBG#12 calculated in step S7 may contain errors relative to the actual height OH of the object group OBG contained in container CB at the moment when the shooting height CH of the shooting system 2 is actually adjusted based on the height OH of the second object group OBG#12. However, even in this case, the height OH of the object group OBG contained in container CB may contain errors. Figure 4 The error in the height OH of the second object group OBG#12 calculated in step S7 is unlikely to be large enough to adversely affect the adjustment of the shooting height CH of the shooting system 2. However, even when performing... Figure 4 In the case of robot control processing shown, Figure 4The height OH of the second object group OBG#12 calculated in step S7 may sometimes be consistent with the height OH of the object group OBG actually stored in container CB at the moment when the shooting height CH of the shooting system 2 is actually adjusted based on the height OH of the second object group OBG#12.

[0284] Subsequently, based on the calculated second height OH#12 and the permissible shooting range CAR of the shooting system 2, the control device 3 determines the target shooting height CH_target of the shooting system 2. Specifically, as follows... Figure 21 As shown in (b), the control device 3 can determine the target shooting height CH_target as the third target shooting height CH_target#13, satisfying the condition that "the calculated second height OH#12 is included within the shooting allowable range CAR of the shooting system 2 located at the target shooting height CH_target". Then, the control device 3 generates a robot control signal to control the robot 1 so that the shooting height CH of the shooting system 2 becomes the third target shooting height CH_target#13 determined based on the second height OH#12. As a result, the shooting system 2 is located at the third target shooting height CH_target#13.

[0285] Here, as described above, the second height OH#12 of the second object group OBG#12 is typically lower than the first height OH#11 of the first object group OBG#11. In this case, the third target shooting height CH_target#13, determined based on the second height OH#12, can also become lower than the second target shooting height CH_target#12, determined based on the first height OH#11.

[0286] However, depending on the circumstances, the second height OH#12 of the second object group OBG#12 may typically become higher than the first height OH#11 of the first object group OBG#11. For example, when multiple first object objects OBJ#11 included in the first object group OBG#11 are configured to partially overlap (e.g., randomly configured), the stack of the first object group OBG#11 (i.e., the second object group OBG#12) after removing one of the first object objects OBJ#11, which is being held by the end effector 4, may collapse. As a result, the orientation of at least one second object object OBJ#12 included in the second object group OBG#12 may change. As a result, for example, if the orientation of the second object OBJ#12 changes from its length direction along the horizontal direction to its vertical direction, the height OH of the second object OBJ#12 (i.e., the height of the second object group OBG#12) may increase. In this case, see below. Figure 24 As described in the first variant example, after the end effector 4 holds a first object OBJ#11 ( Figure 4 Step S5: Yes), the shooting system 2 can shoot the second object group OBG#12 ( Figure 24 In step S8a), the control device 3 can calculate the second height OH#12 of the second object group OBG#12 based on the shooting results of the second object group OBG#12. Figure 4 Step S7). Furthermore, the same applies not only to calculating the second height OH#12 of the second object group OBG#12, but also to calculating the height OH of any object group OBG.

[0287] After that, as Figure 22 As shown in (a), the shooting system 2, located at the third target shooting height CH_target#13, shoots the third object group OBG#13, which includes at least a portion of the multiple object objects OBJ contained in the container CB. Figure 4 Step S1). As a result, the control device 3 acquires from the imaging system 2 image data IMG_3D (referred to as image data IMG_3D#13) representing an image of the third object group OBG#13 captured in the image. Figure 4 Step S1). Furthermore, in the following description, for ease of explanation, the object OBJ included in the third object group OBG#13 will be referred to as the third object OBJ#13.

[0288] The third object group OBG#13 captured by the shooting system 2 at the third target shooting height CH_target#13 may be different from the second object group OBG#12 captured by the shooting system 2 to determine the third target shooting height CH_target#13. Specifically, at least one of the plurality of second object objects OBJ#12 included in the second object group OBG#12 may not be included in the third object group OBG#13. As an example, at the moment when the shooting system 2 at the third target shooting height CH_target#13 captures the third object group OBG#13, at least one of the plurality of second object objects OBJ#12 included in the second object group OBG#12 should have been removed from the container CB by the robot 1. In this case, at least one second object object OBJ#12 that has been removed from the container CB may not be included in the third object group OBG#13. As another example, due to the adjustment of the shooting height CH of the shooting system 2, the field of view of the shooting system 2 relative to the object group OBG contained in the container CB may change. In this case, the third object group OBG#13 may be different from the second object group OBG#12.

[0289] However, at least one of the multiple second object objects OBJ#12 included in the second object group OBG#12 that has not yet been removed from the container CB can be included in the third object group OBG#13. That is, the second object group OBG#12 and the third object group OBG#13 can partially overlap. In this case, since the third target shooting height CH_target#13 of the shooting system 2 that shoots the third object group OBG#13 is determined based on the second height OH#12 of the second object group OBG#12, therefore, as Figure 22As shown in (a), the shooting range CAR of the shooting system 2 located at the third target shooting height CH_target#13 includes at least a portion of the third object group OBG#13, which partially overlaps with the second object group OBG#12. That is, the shooting range CAR of the shooting system 2 located at the third target shooting height CH_target#13 includes at least one of the plurality of third object objects OBJ#13 contained in the third object group OBG#13. Therefore, the shooting system 2 is capable of appropriately shooting at least one of the plurality of third object objects OBJ#13 contained in the third object group OBG#13. However, all of the plurality of third object objects OBJ#13 contained in the third object group OBG#13 may be different from the plurality of second object objects OBJ#12 contained in the second object group OBG#12.

[0290] Furthermore, as described above, since at least one second object OBJ#12 is removed from container CB when shooting system 2 shoots the third object group OBG#13 from the third target shooting height CH_target#13, the number of object OBJs stored in container CB when shooting system 2 shoots the third object group OBG#13 from the third target shooting height CH_target#13 can be less than the number of object OBJs stored in container CB when shooting system 2 shoots the second object group OBG#12 from the second target shooting height CH_target#12. The number of third object OBJs#13 included in the third object group OBG#13 captured by shooting system 2 from the third target shooting height CH_target#13 can be less than the number of second object OBJs#12 included in the second object group OBG#12 captured by shooting system 2 from the second target shooting height CH_target#12. The weight of the container CB containing multiple object objects OBJs at the moment when shooting system 2 shoots the third object group OBG#13 from the third target shooting height CH_target#13 can be lighter than the weight of the container CB containing multiple object objects OBJs at the moment when shooting system 2 shoots the second object group OBG#12 from the second target shooting height CH_target#12. Furthermore, the "weight of the container CB containing multiple object objects OBJs" mentioned here can include both the weight of the container CB itself and the weight of the multiple object objects OBJs contained within the container CB.

[0291] Subsequently, the position and attitude calculation unit 312 generates position and attitude data POI (referred to as position and attitude data POI#13) of one of the multiple third object objects OBJ#13 selected as the object to be processed, based on the image data IMG_3D#13 (specifically, based on the three-dimensional position data WSD generated from the image data IMG_3D#13). Figure 4 (Steps S2 to S3). Furthermore, in the following description, the third object OBJ#13 selected as the object to be processed is referred to as the third object OBJ#13_target.

[0292] The position and attitude calculation unit 312 is highly likely to select the third object OBJ#13, which is included within the shooting range CAR of the shooting system 2 located at the third target shooting height CH_target#13, as the processing execution object, i.e., the third object OBJ#13_target. This is because the matching similarity of object OBJs included within the shooting range CAR is more likely to be higher than the matching similarity of object OBJs not included within the shooting range CAR. In this case, the position and attitude calculation unit 312 can generate position and attitude data POI, which accurately represents at least one of the position and attitude of the third object OBJ#13 included within the shooting range CAR of the shooting system 2 located at the third target shooting height CH_target#13. This is because the higher the matching similarity of the object OBJs, the higher the likelihood that the accuracy of the position and attitude data POI of the object OBJs will be. Conversely, the third target shooting height CH_target#13 can be a shooting height CH that satisfies the condition that "the third object OBJ#13 selected as the processing execution object is included within the shooting allowable range CAR of the shooting system 2 located at the third target shooting height CH_target#13". That is, the control device 3 can determine the third target shooting height CH_target#13 to satisfy the condition that "the third object OBJ#13 selected as the processing execution object is included within the shooting allowable range CAR of the shooting system 2 located at the third target shooting height CH_target#13". However, the position and attitude calculation unit 312 can select a third object OBJ#13 that is not included within the shooting allowable range CAR of the shooting system 2 located at the third target shooting height CH_target#13 as the processing execution object, i.e., the third object OBJ#13_target.

[0293] On the other hand, the third object object OBJ#13_target selected as the object to be processed may not be included within the shooting range CAR of the shooting system 2 located at the shooting height CH_target#12 of the second target. In other words, the third object object OBJ#13_target selected as the object to be processed may not be included within the shooting range CAR of the shooting system 2 before adjusting the shooting height CH. This is because, as shown by the side reference... Figure 18 (a) to Figure 18 (b) and Figure 19 As explained, since the shooting height CH of the shooting system 2 is adjusted according to the height OH of the object group OBG contained in the container CB, even if an object OBJ is not included in the shooting allowable range CAR of the shooting system 2 at the first moment, it will be included in the shooting allowable range CAR of the shooting system 2 at the second moment, which is later than the first moment. As a result, the object OBJ can be selected as the object to be processed. That is to say, even if an object OBJ is not included in the shooting allowable range CAR of the shooting system 2 at the second target shooting height CH_target#12, it will be included in the shooting allowable range CAR of the shooting system 2 at the third target shooting height CH_target#13 as time goes by. As a result, the object OBJ can be selected as the object to be processed.

[0294] Furthermore, the third object object OBJ#13_target selected as the object to be processed can be the second object object OBJ#12 that is not included in the shooting range CAR of the shooting system 2 located at the shooting height CH_target#12 of the second target. Alternatively, the third object object OBJ#13_target selected as the object to be processed can be the second object object OBJ#12 that is included in the shooting range CAR of the shooting system 2 located at the shooting height CH_target#12 of the second target but was not selected as the object to be processed.

[0295] Subsequently, the signal generation unit 313 generates a robot control signal based on the position and orientation data POI#13 of the third object OBJ#13_target to control the robot 1 so that the end effector 4 approaches the third object OBJ#13_target. Figure 4Step S5). After the end effector 4 approaches the third object OBJ#13_target according to the robot control signal generated in step S4, in step S5, the robot 1 performs the prescribed processing on the third object OBJ#13_target. That is, the robot 1 holds the third object OBJ#13_target stored in the container CB, removes the held third object OBJ#13_target from the container CB, and releases it to a release position outside the container CB.

[0296] After robot 1 performs the specified processing on the third object OBJ#13_target ( Figure 4 Step S5: Yes), control device 3 adjusts the shooting height CH of shooting system 2. Figure 4 (Step S7). Furthermore, as described above, the control device 3 can adjust the shooting height CH of the shooting system 2 after the robot 1 has performed the prescribed processing on a single third object OBJ#13. Alternatively, as described above, the control device 3 can adjust the shooting height CH of the shooting system 2 after the robot 1 has performed the prescribed processing on multiple third object OBJ#13s respectively. Hereinafter, for ease of explanation, an example will be described where the control device 3 adjusts the shooting height CH of the shooting system 2 after the robot 1 has performed the prescribed processing on a single third object OBJ#13.

[0297] In order to adjust the shooting height CH of the shooting system 2, the control device 3 uses robot control signals to generate robot control signals for controlling robot 1 to bring end effector 4 closer to the third object OBJ#13_target. Figure 4 The image data IMG_3D#13 acquired in step S1 (specifically, the 3D position data WSD generated from the image data IMG_3D#13) is used to calculate the height OH of the third object group OBG#13. The result is as follows: Figure 22 As shown in (b), the control device 3 calculates the third height OH#13 as the height OH of the third object group OBG#13.

[0298] Here, as described above, at the moment the third object group OBG#13 is photographed, at least one of the multiple second object objects OBJ#12 included in the second object group OBG#12 has been removed from the container CB by robot 1. Therefore, at least one of the multiple second object objects OBJ#12 included in the second object group OBG#12 is not included in the third object group OBG#13. That is, the number of third object objects OBJ#13 included in the third object group OBG#13 becomes less than the number of second object objects OBJ#12 included in the second object group OBG#12. As a result, the third height OH#13 of the third object group OBG#13 usually becomes lower than the second height OH#12 of the second object group OBG#12. However, depending on the circumstances, the third height OH#13 of the third object group OBG#13 may also be the same as the second height OH#12 of the second object group OBG#12.

[0299] In addition, Figure 22 In the example shown in (a), at the moment the third object group OBG#13 is captured, the third object OBJ#13_target, which has not yet been removed from container CB by robot 3, is located at the highest position in the third object group OBG#13. Therefore, control device 3 calculates the height of the third object OBJ#13_target as the height OH of the third object group OBG#13. However, as Figure 22 As shown in (b), at the moment when the shooting height CH of the shooting system 2 is actually adjusted based on the height OH of the third object group OBG#13, the third object OBJ#13_target has been removed from the container CB. Therefore, during the process... Figure 4 In the case of robot control processing shown, Figure 4 The height OH of the third object group OBG#13 calculated in step S7 may contain errors relative to the actual height OH of the object group OBG contained in container CB at the moment when the shooting height CH of the shooting system 2 is actually adjusted based on the height OH of the third object group OBG#13. However, even in this case, the height OH of the object group OBG contained in container CB may contain errors. Figure 4 The error in the height OH of the third object group OBG#13 calculated in step S7 is unlikely to be large enough to adversely affect the adjustment of the shooting height CH of the shooting system 2. However, even when performing... Figure 4 In the case of robot control processing shown, Figure 4The height OH of the third object group OBG#13 calculated in step S7 may sometimes be consistent with the height OH of the object group OBG actually stored in container CB at the moment when the shooting height CH of shooting system 2 is actually adjusted based on the height OH of the third object group OBG#13.

[0300] Subsequently, based on the calculated third height OH#13 and the permissible shooting range CAR of the shooting system 2, the control device 3 determines the target shooting height CH_target of the shooting system 2. Specifically, as follows... Figure 22 As shown in (b), the control device 3 can determine the target shooting height CH_target as the fourth target shooting height CH_target#14, which satisfies the condition that "the calculated third height OH#13 is included within the shooting allowable range CAR of the shooting system 2 located at the target shooting height CH_target". Then, the control device 3 generates a robot control signal to control the robot 1 so that the shooting height CH of the shooting system 2 becomes the fourth target shooting height CH_target#14 determined based on the third height OH#13. As a result, the shooting system 2 is located at the fourth target shooting height CH_target#14.

[0301] Here, as described above, the third height OH#13 of the third object group OBG#13 is generally lower than the second height OH#12 of the second object group OBG#12. In this case, the fourth target shooting height CH_target#14 determined based on the third height OH#13 can also become lower than the third target shooting height CH_target#13 determined based on the second height OH#12.

[0302] After that, as Figure 23 As shown, the shooting system 2, located at the fourth target shooting height CH_target#14, shoots the fourth object group OBG#14, which includes at least a portion of the multiple object objects OBJ contained in the container CB. Figure 4 Step S1). As a result, the control device 3 acquires from the imaging system 2 image data IMG_3D (referred to as image data IMG_3D#14) representing an im...

Claims

1. A control device that generates control signals for controlling a robot, the robot being provided with a holding device for holding an object and a shooting system for shooting the object, and for moving the holding device and the shooting system. The control device is characterized in that it includes: A computing device that generates the control signal; as well as A communication device that outputs the control signals generated by the computing device. The computing device generates a first control signal based on the imaging results obtained by the imaging system from a first imaging height, which captures images of a first group of object objects, including at least a portion of the plurality of object objects contained in the container. This first control signal controls the robot to bring the holding device close to one of the object objects in the first group of object objects, i.e., the first object object, in order to hold the first object object. Based on the first control signal, the holding device approaches the first object, and the first object held by the holding device is removed from the container. After being released by the holding device outside the container, a second control signal is generated based on the imaging result obtained by the imaging system from a second imaging height lower than the first imaging height, which captures at least a portion of the first object group contained in the container. This second control signal is used to control the robot, causing the holding device to approach one of the object objects in the second object group, i.e., the second object, to hold the second object.

2. The control device as described in claim 1, characterized in that, The height of the second object group is lower than the height of the first object group.

3. The control device as described in claim 1 or 2, characterized in that, When the first object group is photographed from the second shooting height, the number of the object objects contained in the container is less than the number of the object objects contained in the container when the photograph is taken from the first shooting height.

4. The control device according to any one of claims 1 to 3, characterized in that, The number of objects included in the second object group is less than the number of object groups included in the first object group.

5. The control device according to any one of claims 1 to 4, characterized in that, When the second object group is photographed from the second shooting height, the weight of the container holding multiple object objects is less than the weight of the container holding multiple object objects when photographed from the first shooting height.

6. The control device according to any one of claims 1 to 5, characterized in that, The first shooting height is the height of the first object, which is within the shooting range allowed by the shooting system located at the first shooting height. The second shooting height is the height of the second object within the shooting range allowed by the shooting system located at the second shooting height.

7. The control device as described in claim 6, characterized in that, The first shooting height is the height of the second object within the shooting range allowed by the shooting system at the first shooting height.

8. The control device as described in claim 6 or 7, characterized in that, The permitted shooting range is at least one of the depth of focus and depth of field of the shooting system.

9. The control device according to any one of claims 6 to 8, characterized in that, The permitted shooting range is set based on the field of view of the shooting system.

10. The control device according to any one of claims 6 to 9, characterized in that, The allowed shooting range is set by user input.

11. The control device according to any one of claims 1 to 10, characterized in that, Based on the second control signal, the holding device is brought close to the second object, and the second object held by the holding device is removed from the container. After being released by the holding device outside the container, a third control signal is generated based on the imaging result obtained by the imaging system from a third imaging height lower than the second imaging height to photograph the third object group, including at least a portion of the second object group contained in the container. This third control signal is used to control the robot to bring the holding device close to one of the object objects in the third object group, i.e., the third object, in order to hold the third object.

12. The control device as claimed in claim 11, characterized in that, The height of the third object group is lower than the height of the second object group.

13. The control device as described in claim 11 or 12, characterized in that, The second shooting height is the height of the shooting system within the allowed shooting range at the second shooting height, including the second object but excluding the third object. The third shooting height is the height of the third object within the shooting range allowed by the shooting system located at the third shooting height.

14. The control device according to any one of claims 1 to 13, characterized in that, The shooting system includes at least one of a monocular camera and a stereo camera having two monocular cameras that are different from the monocular camera.

15. A control system, characterized in that, include: The control device as described in any one of claims 1 to 14; as well as The shooting system.

16. A robot system, characterized in that, include: The control device as described in any one of claims 1 to 14; The shooting system; as well as The robot.

17. A control method comprising generating control signals for controlling a robot, the robot having a holding device for holding an object and a shooting system for shooting the object, and moving the holding device and the shooting system. The control method is characterized by including: Based on the imaging results obtained by the imaging system from a first imaging height, capturing images of a first group of object objects, including at least a portion of the plurality of object objects contained in the container, a first control signal is generated as the control signal. This first control signal is used to control the robot, causing the holding device to approach one of the object objects in the first group of object objects, i.e., the first object object, to hold the first object object; and Based on the first control signal, the holding device is brought close to the first object, and the first object held by the holding device is removed from the container. After being released by the holding device outside the container, a second control signal is generated based on the imaging result obtained by the imaging system from a second imaging height lower than the first imaging height to photograph a second object group including at least a portion of the first object group contained in the container. This second control signal is used to control the robot to bring the holding device close to one of the object objects in the second object group, i.e., the second object, in order to hold the second object.

18. A computer program, characterized in that, The computer is made to execute the control method as described in claim 17.

19. A control device that generates control signals for controlling a robot, the robot being equipped with a processing device for processing an object and a photographing system for photographing the object, and for moving the processing device and the photographing system. The control device is characterized in that it includes: A computing device that generates the control signal; as well as A communication device that outputs the control signals generated by the computing device. The computing device generates a first control signal based on the imaging results of the imaging system on a first group of object objects, including at least a portion of the plurality of object objects contained in the container. This first control signal is used to control the robot, causing the imaging system to move to an imaging height for imaging the object objects. Based on the shooting results obtained by the shooting system taking pictures of the second object group, including at least a portion of the object objects in the first object group contained in the container, at the shooting height, a second control signal is generated as the control signal. The second control signal is used to control the robot so that the processing device approaches one of the object objects in the second object group, namely the first object object, to perform the processing on the first object object.

20. The control device as claimed in claim 19, characterized in that, The computing device determines the shooting height based on the shooting results of the shooting system on the first group of objects. Generate the first control signal for controlling the robot so that the shooting system moves to the determined shooting height.

21. The control device as described in claim 19 or 20, characterized in that, Based on the captured images, the computing device calculates the height of the first group of objects. The shooting height is determined based on the height of the first object group and the shooting range allowed by the shooting system.

22. The control device according to any one of claims 19 to 21, characterized in that, The shooting height is the height of the first object within the shooting range allowed by the shooting system at the shooting height.

23. The control device according to any one of claims 19 to 22, characterized in that, The processing device is a holding device capable of holding the object. When the shooting height is set to the first shooting height Based on the second control signal, the holding device is brought close to the first object, and the first object held by the holding device is removed from the container. After being released by the holding device outside the container, The computing device generates a third control signal as the control signal based on the imaging results of the imaging system on the third object group, which includes at least a portion of the second object group contained in the container. The third control signal is used to control the robot so that the imaging system moves to a second imaging height that is lower than the first imaging height.

24. The control device as claimed in claim 23, characterized in that, The third object group is the second object group. The shooting result of the shooting system used to generate the third control signal on the third object group is the shooting result of the shooting system used to generate the second control signal on the second object group.

25. The control device as described in claim 23 or 24, characterized in that, The computing device determines the second shooting height based on the shooting results of the shooting system on the third object group. A third control signal is generated to control the robot so that the imaging system moves to the determined second imaging height.

26. The control device according to any one of claims 23 to 25, characterized in that, The computing device generates a fourth control signal based on the imaging results obtained by the imaging system taking pictures of a fourth object group, including at least a portion of the object objects in the third object group contained in the container, at the second imaging height. The fourth control signal is used to control the robot so that the holding device approaches one of the object objects in the fourth object group, namely the second object object, in order to hold the second object object.

27. The control device as claimed in claim 26, characterized in that, The height of the fourth object group is lower than the height of the second object group.

28. The control device as claimed in claim 26 or 27, characterized in that, The first shooting height is the height of the shooting system within the allowed shooting range at the first shooting height, including the first object but excluding the second object. The second shooting height is the height of the second object within the shooting range allowed by the shooting system located at the second shooting height.

29. The control device as described in claim 21, 22 or 28, characterized in that, The permitted shooting range is at least one of the depth of focus and depth of field of the shooting system.

30. The control device as described in claim 21, 22, 28, or 29, characterized in that, The permitted shooting range is set based on the field of view of the shooting system.

31. The control device according to any one of claims 21, 22, 28 to 30, characterized in that, The allowed shooting range is set by user input.

32. The control device according to any one of claims 19 to 31, characterized in that, The computing device generates the first control signal based on the shooting results of the shooting system on the first object group. The first control signal is used to control the robot so that the shooting system moves to at least one of the shooting posture of the shooting system on the object and the shooting horizontal position of the shooting system on the horizontal plane of the object.

33. The control device as described in claim 32, characterized in that, Based on the imaging results of the imaging system on the first group of objects, the computing device determines at least one of the imaging posture and the imaging horizontal position. The first control signal is generated to control the robot so that the shooting system moves to at least one of the determined shooting posture and the shooting horizontal position.

34. The control device according to any one of claims 19 to 33, characterized in that, The shooting system includes at least one of a monocular camera and a stereo camera having two monocular cameras that are different from the monocular camera.

35. A control system, characterized in that, include: The control device as described in any one of claims 19 to 33; as well as The shooting system.

36. A robot system, characterized in that, include: The control device as described in any one of claims 19 to 33; The shooting system; as well as The robot.

37. A control method comprising generating control signals for controlling a robot, the robot having a processing device for processing an object and a photographing system for photographing the object, and moving the processing device and the photographing system. The control method is characterized by including: Based on the imaging results of the imaging system on a first group of object objects, including at least a portion of the plurality of object objects contained in the container, a first control signal is generated as the control signal. This first control signal is used to control the robot, causing the imaging system to move to a shooting height for imaging the object objects; and Based on the shooting results obtained by the shooting system taking pictures of the second object group, including at least a portion of the object objects in the first object group contained in the container, at the shooting height, a second control signal is generated as the control signal. The second control signal is used to control the robot so that the processing device approaches one of the object objects in the second object group, namely the first object object, to perform the processing on the first object object.

38. A computer program, characterized in that, The computer is made to execute the control method as described in claim 37.

39. A control device that generates control signals for controlling a robot, the robot being provided with a holding device for holding an object and a shooting system for shooting the object, and moves the holding device and the shooting system. The control device is characterized in that it includes: A computing device that generates the control signal; as well as A communication device that outputs the control signals generated by the computing device. One of the multiple object objects contained in the container, namely the first object object, is held by the holding device. The robot is controlled to move the holding device that holds the first object from the container to a position outside the container where the first object can be released. The computing device generates a first control signal based on the imaging results obtained by the imaging system photographing a first group of object objects, including at least a portion of the plurality of object objects contained in the container, along the movement path of the imaging system generated by the imaging system on the position of releasing the first object object as the holding device moves from the container to outside the container. The first control signal is used to control the robot from the position after the holding device releases the first object object.

40. The control device as claimed in claim 39, characterized in that, The shooting result of the first object group is obtained by the shooting system moving along the moving path while shooting the first object group contained in the container.

41. The control device as described in claim 39 or 40, characterized in that, Based on the imaging results obtained by the imaging system moving according to the first control signal to photograph the second object group, including at least a portion of the object objects in the first object group contained in the container, a second control signal is generated as the control signal. The second control signal is used to control the robot so that the holding device approaches one of the object objects in the second object group, i.e., the second object object, in order to hold the second object object.

42. The control device according to any one of claims 39 to 41, characterized in that, The first control signal is a signal used to control the robot so that the shooting system moves from the position after releasing the first object from the holding device to the shooting height at which the shooting system can shoot the object contained in the container.

43. The control device as described in claim 42, characterized in that, The first control signal is a signal used to control the robot so that the shooting system moves from the position of the first object released from the holding device to the shooting height.

44. The control device as described in claim 42 or 43, characterized in that, The computing device determines the shooting height based on the shooting results of the shooting system on the first group of objects. Generate the first control signal for controlling the robot so that the shooting system moves to the determined shooting height.

45. The control device according to any one of claims 42 to 44, characterized in that, The computing device calculates the height of the first object group based on the imaging results of the imaging system on the first object group. The shooting height is determined based on the height of the first object group and the shooting range allowed by the shooting system.

46. ​​The control device according to any one of claims 42 to 45, characterized in that, The computing device generates a second control signal based on the imaging result obtained by the imaging system, which moves to the imaging height according to the first control signal, to photograph the second object group, including at least a portion of the object objects in the first object group contained in the container. The second control signal is used to control the robot so that the holding device approaches one of the object objects in the second object group, i.e., the second object object, to hold the second object object.

47. The control device as claimed in claim 46, characterized in that, The shooting height is the height of the second object within the shooting range allowed by the shooting system at the shooting height.

48. The control device as described in claim 46 or 47, characterized in that, When the shooting height is set to the first shooting height The second object is held by the holding device based on the second control signal. The robot is controlled to move the holding device that holds the second object from the container to a position outside the container where the second object can be released. The computing device generates a third control signal based on the imaging results obtained by the imaging system photographing a third object group, including at least a portion of the object group of the second object group contained in the container, along the movement path of the imaging system generated by the imaging system on the position of releasing the second object as the holding device moves from the container to outside the container. The third control signal is used to control the robot so that the imaging system moves from the position after the holding device releases the second object to a second imaging height lower than the first imaging height.

49. The control device as claimed in claim 48, characterized in that, The shooting result of the third object group is obtained by the shooting system moving along the moving path of the shooting system generated by the release of the second object as the holding device moves from the container to the outside of the container, while shooting the third object group contained in the container.

50. The control device as described in claim 48 or 49, characterized in that, The third control signal is a signal used to control the robot so that the shooting system moves from the position where the second object is released from the holding device to the second shooting height.

51. The control device according to any one of claims 48 to 50, characterized in that, The computing device determines the second shooting height based on the shooting results of the shooting system on the third object group. A third control signal is generated to control the robot so that the imaging system moves to the determined second imaging height.

52. The control device according to any one of claims 48 to 51, characterized in that, The computing device generates a fourth control signal based on the imaging results obtained by the imaging system taking pictures of the fourth object group, including at least a portion of the object objects in the third object group contained in the container, at the second imaging height. The fourth control signal is used to control the robot so that the holding device approaches one of the object objects in the fourth object group, namely the third object object, in order to hold the third object object.

53. The control device as described in claim 52, characterized in that, The height of the fourth object group is lower than the height of the second object group.

54. The control device as described in claim 52 or 53, characterized in that, The first shooting height is the height of the shooting system within the allowed shooting range at the first shooting height, including the second object but excluding the third object. The second shooting height is the height of the third object within the shooting range allowed by the shooting system located at the second shooting height.

55. The control device as described in claim 45, 47, or 54, characterized in that, The permitted shooting range is at least one of the depth of focus and depth of field of the shooting system.

56. The control device as described in claim 45, 47, 54 or 55, characterized in that, The permitted shooting range is set based on the field of view of the shooting system.

57. The control device according to any one of claims 45, 47 and 54 to 56, characterized in that, The allowed shooting range is set by user input.

58. The control device according to any one of claims 39 to 57, characterized in that, The first control signal is a signal used to control the robot so that the shooting system moves from the position after releasing the first object from the holding device to the shooting posture of the shooting system to shoot the object contained in the container, i.e., the shooting posture, and the position of the shooting system on the horizontal plane of the object contained in the container to shoot the object, i.e., the shooting horizontal position.

59. The control device as described in claim 58, characterized in that, The computing device determines at least one of the shooting posture and the shooting horizontal position based on the shooting results obtained by the shooting system taking pictures of the first group of object objects contained in the container along the moving path of the shooting system. The first control signal is generated to control the robot so that the shooting system moves to at least one of the determined shooting posture and the shooting horizontal position.

60. The control device according to any one of claims 39 to 59, characterized in that, The shooting system includes at least one of a monocular camera and a stereo camera having two monocular cameras that are different from the monocular camera.

61. A control system, characterized in that, include: The control device as described in any one of claims 39 to 60; as well as The shooting system.

62. A robot system, characterized in that, include: The control device as described in any one of claims 39 to 60; The shooting system; as well as The robot.

63. A control method comprising generating control signals for controlling a robot, the robot having a holding device for holding an object and a shooting system for shooting the object, and moving the holding device and the shooting system. The control method is characterized in that... One of the multiple object objects contained in the container, namely the first object object, is held by the holding device. The robot is controlled to move the holding device that holds the first object from the container to a position outside the container where the first object can be released. The control method includes: Based on the imaging results obtained by the imaging system photographing a first group of object objects, including at least a portion of the plurality of object objects contained in the container, along the movement path of the imaging system generated by the imaging system on the position of releasing the first object object as the holding device moves from the container to outside the container, a first control signal is generated as the control signal, which is used to control the robot from the position after the holding device releases the first object object.

64. A computer program, characterized in that, The computer is made to execute the control method as described in claim 63.

65. A control device that generates control signals for controlling a robot, the robot being provided with a holding device for holding an object and a shooting system for shooting the object, and moves the holding device and the shooting system. The control device is characterized in that it includes: A computing device that generates the control signal; as well as A communication device that outputs the control signals generated by the computing device. During the period from when one of the multiple object objects contained in the container, namely the first object object, is held by the holding device until the first object object held by the holding device is removed from the container and released by the holding device outside the container, the computing device generates a first control signal based on the imaging results obtained by the imaging system from imaging the interior of the container. This first control signal is used to control the robot after the holding device releases the first object object.

66. The control device as claimed in claim 65, characterized in that, The photographic results obtained by photographing the interior of the container include the photographic results obtained by photographing a first group of object objects, including at least a portion of the plurality of object objects contained in the container.

67. The control device as claimed in claim 66, characterized in that, Based on the imaging results obtained by the imaging system moving according to the first control signal to photograph the second object group, including at least a portion of the object objects in the first object group contained in the container, a second control signal is generated as the control signal. The second control signal is used to control the robot so that the holding device approaches one of the object objects in the second object group, i.e., the second object object, in order to hold the second object object.

68. The control device as claimed in claim 67, characterized in that, The first control signal is a signal used to control the robot to move the shooting system to a shooting height for the shooting system to shoot the second object group after the holding device releases the first object.

69. The control device as claimed in claim 68, characterized in that, The computing device determines the shooting height based on the shooting results of the shooting system on the first group of objects. Generate the first control signal for controlling the robot so that the shooting system moves to the determined shooting height.

70. The control device as described in claim 68 or 69, characterized in that, The computing device calculates the height of the first object group based on the imaging results of the imaging system on the first object group. The shooting height is determined based on the height of the first object group and the shooting range allowed by the shooting system.

71. The control device according to any one of claims 68 to 70, characterized in that, The shooting height is the height of the second object within the shooting range allowed by the shooting system at the shooting height.

72. The control device according to any one of claims 67 to 71, characterized in that, During the period from when the second object is held by the holding device according to the second control signal, to when the second object held by the holding device is removed from the container and released by the holding device outside the container, a third control signal is generated based on the imaging results obtained by the imaging system from imaging the interior of the container. This third control signal is used to control the robot after the holding device releases the second object.

73. The control device according to any one of claims 68 to 71, characterized in that, When the shooting height is set to the first shooting height During the period from when the second object is held by the holding device according to the second control signal, until the second object held by the holding device is removed from the container and released by the holding device outside the container, a third control signal is generated based on the imaging results obtained by the imaging system in imaging a third object group including at least a portion of the object groups of the second object group contained in the container. This third control signal is used to control the robot so that after the holding device releases the second object, the imaging system moves to a second imaging height, which is lower than the first imaging height at which the imaging system images the second object group.

74. The control device as claimed in claim 73, characterized in that, The computing device determines the second shooting height based on the shooting results of the shooting system on the third object group. A third control signal is generated to control the robot so that the imaging system moves to the determined second imaging height.

75. The control device as described in claim 73 or 74, characterized in that, The computing device generates a fourth control signal based on the imaging results obtained by the imaging system taking pictures of the fourth object group, including at least a portion of the object objects in the third object group contained in the container, at the second imaging height. The fourth control signal is used to control the robot so that the holding device approaches one of the object objects in the fourth object group, namely the third object object, in order to hold the third object object.

76. The control device as claimed in claim 75, characterized in that, The height of the fourth object group is lower than the height of the second object group.

77. The control device as described in claim 75 or 76, characterized in that, The first shooting height is the height of the shooting system within the allowed shooting range at the first shooting height, including the second object but excluding the third object. The second shooting height is the height of the third object within the shooting range allowed by the shooting system located at the second shooting height.

78. The control device as described in claim 70, 71, or 77, characterized in that, The permitted shooting range is at least one of the depth of focus and depth of field of the shooting system.

79. The control device as described in claim 70, 71, 77 or 78, characterized in that, The permitted shooting range is set based on the field of view of the shooting system.

80. The control device according to any one of claims 70, 71 and 77 to 79, characterized in that, The allowed shooting range is set by user input.

81. The control device according to any one of claims 65 to 80, characterized in that, The first control signal is a signal used to control the robot to move the shooting system to at least one of the shooting posture (i.e., the shooting posture) and the position on the horizontal plane where the shooting system is shooting (i.e., the shooting horizontal position) after the holding device releases the first object.

82. The control device as claimed in claim 81, characterized in that, During the period from when the first object contained in the container is held by the holding device until the first object held by the holding device is removed from the container and released by the holding device outside the container, the computing device determines at least one of the shooting posture and the shooting horizontal position based on the shooting results obtained by the shooting system from shooting the interior of the container. The first control signal is generated to control the robot so that the shooting system moves to at least one of the determined shooting posture and the shooting horizontal position.

83. The control device according to any one of claims 65 to 82, characterized in that, The shooting system includes at least one of a monocular camera and a stereo camera having two monocular cameras that are different from the monocular camera.

84. A control system, characterized in that, include: The control device as described in any one of claims 65 to 83; as well as The shooting system.

85. A robot system, characterized in that, include: The control device as described in any one of claims 65 to 83; The shooting system; as well as The robot.

86. A control method for generating control signals for controlling a robot, the robot having a holding device for holding an object and a shooting system for shooting the object, and for moving the holding device and the shooting system. The control method is characterized by including: During the period from when one of the multiple object objects contained in the container, namely the first object object, is held by the holding device until the first object object held by the holding device is removed from the container and released by the holding device outside the container, a first control signal is generated based on the imaging results obtained by the imaging system from imaging the interior of the container. This first control signal is used to control the robot after the holding device releases the first object object.

87. A computer program, characterized in that, The computer is made to execute the control method as described in claim 86.

88. A control device that generates control signals for controlling a robot, the robot being equipped with a processing device for processing an object and a photographing system for photographing the object, and for moving the processing device and the photographing system. The control device is characterized in that it includes: A computing device that generates the control signal; as well as A communication device that outputs the control signals generated by the computing device. The computing device generates a first control signal based on at least one of the height of a first object group, including at least a portion of the plurality of object objects contained in the container, and at least one of the components of the first object group contained in the container. This first control signal is used to control the robot so that the imaging system moves to a shooting height for imaging the object objects. Based on the shooting results obtained by the shooting system taking pictures of the second object group, including at least a portion of the object objects in the first object group contained in the container, at the shooting height, a second control signal is generated as the control signal. The second control signal is used to control the robot so that the processing device approaches one of the object objects in the second object group to perform the processing on the object object.

89. The control device as claimed in claim 88, characterized in that, The second object group is the same as the first object group.

90. The control device as described in claim 88 or 89, characterized in that, The shooting height is the height of at least one of the object objects in the first object group, within the shooting range allowed by the shooting system at the shooting height.

91. The control device according to any one of claims 88 to 90, characterized in that, The object in the second object group is one of the objects in the first object group that is within the shooting range allowed by the shooting system at the shooting height.

92. The control device according to any one of claims 88 to 91, characterized in that, The computing device determines the shooting height based on at least one of the height of the first object group contained in the container and the components of the first object group contained in the container. Generate the first control signal for controlling the robot so that the shooting system moves to the determined shooting height.

93. The control device according to any one of claims 88 to 92, characterized in that, The computing device calculates the height of the first group of objects stored in the container based on the imaging results of the imaging system. The first control signal is generated based on the calculated height.

94. The control device according to any one of claims 88 to 93, characterized in that, At least one of the height of the first object group contained in the container and the weight of the first object group contained in the container is calculated directly or indirectly based on the measurement results of the measuring device.

95. The control device as described in claim 94, characterized in that, The shooting system is the first shooting system. The measuring device includes at least one of the first imaging system and a second imaging system different from the first imaging system. The computing device calculates at least one of the height of the first object group contained in the container and the component of the first object group contained in the container, based on the shooting results of at least one of the first shooting system and the second shooting system, which are the measurement results.

96. The control device as described in claim 95, characterized in that, The shooting result of at least one of the first shooting system and the second shooting system is the shooting result of the first object group contained in the container.

97. The control device as described in claim 95 or 96, characterized in that, The imaging result of at least one of the first and second imaging systems is the imaging result of at least one mark formed on the surface of the container at a location facing the space where the first object group is located.

98. The control device as claimed in claim 97, characterized in that, The computing device calculates at least one of the height of the first object group contained in the container and the component of the first object group contained in the container, based on the number of the markers detected from the imaging results of the markers.

99. The control device according to any one of claims 94 to 98, characterized in that, The measuring device includes a weight sensor capable of measuring the weight of the first object group contained in the container.

100. The control device according to any one of claims 94 to 99, characterized in that, The measuring device includes at least one sensor capable of detecting at least one of the object objects in the first object group within the container.

101. The control device as claimed in claim 100, characterized in that, The computing device calculates at least one of the height of the first object group contained in the container and the component of the first object group contained in the container, based on the number of sensors that do not detect at least one of the object objects in the first object group.

102. The control device as claimed in claim 100 or 101, characterized in that, The at least one sensor is configured in the container.

103. The control device according to any one of claims 88 to 102, characterized in that, The computing device calculates at least one of the height of the first object group contained in the container and the component of the first object group contained in the container, based on the number of object objects that the processing device has processed.

104. The control device according to any one of claims 88 to 103, characterized in that, The computing device generates a first control signal based on at least one of the height of the first object group contained in the container and the component of the first object group contained in the container. The first control signal is used to control the robot so that the shooting system moves to at least one of the shooting posture of the shooting system for shooting the object and the position of the shooting system on the horizontal plane for shooting the object.

105. The control device as claimed in claim 104, characterized in that, The computing device determines at least one of the shooting posture and the shooting horizontal position based on at least one of the height of the first object group contained in the container and the component of the first object group contained in the container. The first control signal is generated to control the robot so that the shooting system moves to at least one of the determined shooting posture and the shooting horizontal position.

106. The control device according to any one of claims 88 to 105, characterized in that, The shooting system includes at least one of a monocular camera and a stereo camera having two monocular cameras that are different from the monocular camera.

107. A control system, characterized in that, include: The control device as described in any one of claims 88 to 106; and The shooting system.

108. A robot system, characterized in that, include: The control device as described in any one of claims 88 to 106; The shooting system; as well as The robot.

109. A control method for generating control signals for controlling a robot, the robot having a holding device for holding an object and a shooting system for shooting the object, and for moving the holding device and the shooting system. The control method is characterized by including: A first control signal is generated based on at least one of the height of a first object group, including at least a portion of the plurality of object objects contained in the container, and a component of the first object group contained in the container. This first control signal controls the robot to move the imaging system to a shooting height for photographing the object objects. Based on the shooting results obtained by the shooting system taking pictures of the second object group, including at least a portion of the object objects in the first object group contained in the container, at the shooting height, a second control signal is generated as the control signal. The second control signal is used to control the robot so that the processing device approaches one of the object objects in the second object group to perform the processing on the object object.

110. A computer program, characterized in that, The computer is made to execute the control method as described in claim 109.

Images