Control device, control system, robot system, control method, and computer program

Abstract

A control device controls a robot that moves a processing device that processes an object and an imaging system. When the control device determines that there is no target object that can be processed by the processing device on the basis of first image data generated by imaging the target object by the imaging system when the positional relationship between the object and the imaging system is a first positional relationship, the control device determines that there is no target object that can be processed by the processing device on the basis of second image data generated by imaging the target object by the imaging system. Generating a first control signal for controlling the robot so that the positional relationship between the target object and the imaging system changes to a second positional relationship; and generates a second control signal for controlling the robot on the basis of position / attitude data of the target object generated from third image data generated by imaging the target object by the imaging system when the positional relationship between the target object and the imaging system is a second positional relationship.

Description

Technical Field

[0001] This invention relates, for example, to the technical fields of control devices, control systems, robot systems, control methods, and computer programs capable of generating control signals for controlling robots. Background Technology

[0002] Patent Document 1 describes an example of a control device that calculates at least one of the position and orientation of an object to be processed by a robot, and controls the robot based on the calculated position and orientation. In this control device, in order to control the robot to process an object, it is required to calculate at least one of the object's position and orientation.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: U.S. Patent Application Publication No. 2013 / 0230235 Summary of the Invention

[0006] According to a first method, a control device is provided that generates a control signal for controlling a robot. The robot is equipped with a processing device for processing an object and an imaging system, and moves the processing device and the imaging system. The control device includes: a computing device for generating the control signal; and a communication device for outputting the control signal generated by the computing device. If, based on a first positional relationship between the object and the imaging system, a first image data generated by the imaging system of the object indicates that no object exists that can be processed by the processing device, the computing device generates a first control signal for controlling the robot to change the positional relationship to a second positional relationship different from the first positional relationship, based on second image data generated by the imaging system of the object. The second control signal is generated based on position data and attitude data representing the position and attitude of the object, generated from third image data of the object captured by the imaging system in the second positional relationship.

[0007] According to the second method, a control system is provided, including the control device provided by the first method and the shooting system.

[0008] According to the third method, a robot system is provided, including a control device provided by the first method, the shooting system, and the robot.

[0009] According to a fourth method, a control method is provided that generates a control signal for controlling a robot equipped with a processing device for processing an object and an imaging system, and moves the processing device and the imaging system. The control method includes: if, when a first image data generated by the imaging system of the object being photographed when the positional relationship between the object and the imaging system is a first positional relationship determines that there is no object that can be processed by the processing device, a first control signal is generated based on second image data generated by the imaging system of the object being photographed, to control the robot to change the positional relationship to a second positional relationship different from the first positional relationship, and this first control signal is used as the control signal; and a second control signal is generated based on position data and attitude data representing the position and attitude of the object being photographed by the imaging system when the object is photographed in the second positional relationship, which are generated based on third image data representing the position and attitude of the object being photographed by the imaging system.

[0010] According to the fifth method, a computer program is provided that causes a computer to execute the control method provided by the fourth method.

[0011] According to the sixth method, a control device is provided that generates a control signal for controlling a robot equipped with a processing device for processing object objects and an imaging system, and moves the processing device and the imaging system. The control device includes: a computing device for generating the control signal; and a communication device for outputting the control signal generated by the computing device. If, based on first image data generated by the imaging system from a first object group containing multiple object objects captured by the imaging system when the imaging system is in a first position and a first posture, it is determined that there is no object object in the first object group that can be processed by the processing device, the computing device generates a first control signal based on second image data generated by the imaging system from a second object group containing multiple object objects captured by the imaging system, to control the robot to change the imaging system to a second position and a second posture different from the first position and the first posture, as the control signal.

[0012] According to the seventh method, a control system is provided, including the control device provided by the sixth method and the shooting system.

[0013] According to the eighth method, a robot system is provided, including a control device provided by the sixth method, the shooting system, and the robot.

[0014] According to the ninth method, a control method is provided to generate a control signal for controlling a robot equipped with a processing device for processing object objects and an imaging system, and to move the processing device and the imaging system. The control method includes: if, when it is determined that there is no object object in the first object object group that can be processed by the processing device, based on a first image data generated by the imaging system capturing a first object object group containing multiple object objects when the imaging system is in a first position and a first posture, a first control signal is generated for controlling the robot to change the imaging system to a second position and a second posture different from the first position and the first posture, based on a second image data generated by the imaging system capturing a second object object group containing multiple object objects, and the first control signal is used as the control signal.

[0015] According to the 10th method, a computer program is provided that causes a computer to execute the control method provided by the 9th method.

[0016] According to the eleventh method, a control device is provided that generates a control signal for controlling a robot equipped with a processing device for processing object objects and an imaging system, and moves the processing device and the imaging system. The control device includes: a computing unit that generates the control signal based on first three-dimensional position data representing the three-dimensional positions of multiple points in the first object group (containing multiple object objects) generated from imaging results of the imaging system on the first object group, and second three-dimensional position data that is less than the first three-dimensional position data; and outputs the control signal generated by the computing unit. The communication device that generates the control signal, wherein the computing device generates, based on the second three-dimensional position data, multiple first position data and first attitude data representing the position and attitude of each object in the second object group composed of multiple object objects in the first object group, and generates, based on the generated multiple first position data and first attitude data, and the first three-dimensional position data, second position data and second attitude data representing the position and attitude of one object in the second object group, and generates the control signal based on the generated second position data and second attitude data.

[0017] According to the 12th method, a control system is provided, including the control device provided by the 11th method and the shooting system.

[0018] According to the 13th method, a control system is provided, including the control device provided by the 11th method and the shooting system.

[0019] According to the 14th embodiment, a control method is provided to generate a control signal for controlling a robot equipped with a processing device for processing object objects and an imaging system, and to move the processing device and the imaging system. The control method includes: generating the control signal based on first three-dimensional position data representing the three-dimensional positions of multiple points in the first object group, generated from imaging results of the imaging system of the first object group comprising multiple object objects, and second three-dimensional position data, which is less than the first three-dimensional position data. Generating the control signal includes: generating multiple first position data and first posture data representing the positions and orientations of each object object in the second object group composed of multiple object objects in the first object group, based on the second three-dimensional position data; generating second position data and second posture data representing the position and orientation of one object object in the second object group based on the generated multiple first position data and first posture data, and the first three-dimensional position data; and generating the control signal based on the generated second position data and second posture data.

[0020] According to the 15th method, a computer program is provided that causes a computer to execute the control method provided by the 13th method.

[0021] According to the 16th method, a control device is provided that generates control signals for controlling a robot equipped with a processing device and an imaging system for processing object objects contained in a container, and for moving the processing device and the imaging system. The control device includes: a computing device for generating the control signals; and a communication device for outputting the control signals generated by the computing device. The computing device generates position and orientation data representing at least one of the position and orientation of the container based on first image data generated by the imaging system capturing at least a portion of the container, and generates three-dimensional position data representing the three-dimensional positions of multiple points in the object group based on second image data generated by the imaging system capturing a group of object objects. The object group includes at least a portion of a plurality of object objects contained in the container. A portion of the three-dimensional position data is selected as selection data based on the position and orientation data. Based on the selection data, the control signals for controlling the robot to bring the processing device closer to one object object so that the processing device can process one of the object objects in the object group are generated.

[0022] According to the 17th method, a control system is provided, including the control device provided by the 16th method and the shooting system.

[0023] According to the 18th method, a robot system is provided, including a control device provided by the 16th method, the shooting system, and the robot.

[0024] According to the 19th embodiment, a control method is provided, generating control signals for controlling a robot equipped with a processing device for processing object objects and an imaging system, and moving the processing device and the imaging system. The control method includes: generating position and orientation data representing at least one of the position and orientation of the container based on first image data generated by the imaging system capturing at least a portion of the container; generating three-dimensional position data representing the three-dimensional positions of multiple points in the object group based on second image data generated by the imaging system capturing a group of object objects, wherein the object group includes at least a portion of a plurality of object objects contained in the container; selecting a portion of the three-dimensional position data as selection data based on the position and orientation data; and generating the control signals for controlling the robot such that the processing device approaches one of the object objects so that the processing device can process one of the object objects in the object group based on the selection data.

[0025] According to the 20th method, a computer program is provided that causes a computer to execute the control method provided by the 19th method.

[0026] According to the 21st method, a control device is provided that generates a control signal for controlling a robot equipped with a processing device for processing object objects and an imaging system, and moves the processing device and the imaging system. The control device includes: a computing device for generating the control signal; and a communication device for outputting the control signal generated by the computing device. The computing device generates the control signal for the robot to change at least one of the positions and orientations of the imaging system relative to one of the object objects in the object object group, based on the imaging system's imaging results of an object group containing multiple object objects, whereby the processing device cannot process at least one object in the object group.

[0027] The effects and other advantages of the present invention will become clear from the embodiments described below. Attached Figure Description

[0028] Figure 1 This is a block diagram illustrating the structure of the robot system in this embodiment.

[0029] Figure 2 This is a side view showing the appearance of the robot according to this embodiment.

[0030] Figure 3 This is a block diagram showing the structure of the control device in this embodiment.

[0031] Figure 4 This is a flowchart illustrating the robot control process.

[0032] Figure 5 The illustration shows the 3D matching process.

[0033] Figure 6 (a) to Figure 6 (d) are side views showing the positional relationship between the robot and the workpiece at a certain moment during the holding process of the workpiece held by the mounting device for holding the workpiece moving on the support surface.

[0034] Figure 7 (a) to Figure 7 (d) are side views showing the positional relationship between the robot and the workpiece at a certain moment during the release process for positioning the workpiece onto a mounting device that moves on the support surface.

[0035] Figure 8 (a) to Figure 8 (b) are side views showing the positional relationship between the robot and the workpiece at a certain moment during the holding process of the workpiece held by the mounting device for holding it stationary on the support surface. Figure 8 (c) to Figure 8 (e) are side views showing the positional relationship between the robot and the workpiece at a certain moment during the release process of positioning the workpiece on a mounting device that is stationary on the support surface.

[0036] Figure 9 (a) to Figure 9 (e) are side views showing the positional relationship between the robot and the workpieces at a certain moment during a holding process for holding multiple workpieces placed on the mounting device one by one and a release process for placing multiple workpieces one by one on the mounting device.

[0037] Figure 10 (a) to Figure 10 (e) are side views showing the positional relationship between the robot and the workpieces at a certain moment during a holding process for holding multiple workpieces placed on the mounting device one by one and a release process for placing multiple workpieces one by one on the mounting device.

[0038] Figure 11 (a) to Figure 11 (b) shows an example of a model-generated screen.

[0039] Figure 12 Multiple local models are shown.

[0040] Figure 13 (a) and Figure 13 (c) Shows an example of an object, Figure 13 (b) and Figure 13 (d) shows an example of a template model.

[0041] Figure 14 This is a block diagram showing the structure of the control device in Modified Example 2.

[0042] Figure 15 This is a flowchart illustrating the robot control process in Variation Example 2.

[0043] Figure 16 (a) and Figure 16 (b) Show the positional relationship between the object and the shooting device.

[0044] Figure 17 (a) shows the camera device rotating and moving. Figure 17 (b) shows the camera device in translational motion.

[0045] Figure 18 (a) and Figure 18 (b) The moving camera device is shown.

[0046] Figure 19 (a) shows an example of an object. Figure 19 (b) and Figure 19 (c) Showing images taken using the camera system before movement. Figure 19 (a) is an example of a point cloud contained in the point cloud data (3D position data) generated from the image data of the object shown. Figure 19 (d) shows Figure 19 (a) shows the overall model of the object. Figure 19 (e) shows Figure 19 (a) is a partial model of the object shown.

[0047] Figure 20 (a) to Figure 20 (b) Showing a camera system for photographing multiple objects.

[0048] Figure 21 (a) to Figure 21 (b) Showing a camera system for photographing multiple objects.

[0049] Figure 22 Multiple reset positions are shown.

[0050] Figure 23 Multiple reset postures are shown.

[0051] Figure 24 (a) and Figure 24 (b) shows a method for setting multiple reset positions.

[0052] Figure 25 This is a block diagram showing the structure of the control device in Modified Example 3.

[0053] Figure 26 The sizing process is conceptually illustrated.

[0054] Figure 27 This is a flowchart illustrating the robot control process in Variation Example 3.

[0055] Figure 28 (a) and Figure 28 (b) Conceptually present the data of interest separately.

[0056] Figure 29 This illustrates a camera system for capturing images of multiple objects.

[0057] Figure 30 This is a flowchart illustrating the robot control process in Variation Example 3.

[0058] Figure 31 (a) to Figure 31 (c) Several template models are conceptually shown respectively.

[0059] Figure 32 This is a flowchart illustrating the robot control process in Variation Example 3.

[0060] Figure 33 This is a block diagram showing the structure of the control device in Modified Example 3.

[0061] Figure 34 The image shows a captured image of the object being held.

[0062] Figure 35 (a) to Figure 35 (c) The initial settings actions are shown respectively.

[0063] Figure 36 (a) to Figure 36 (c) The selected data are shown conceptually.

[0064] Figure 37 Displays the screen for the specified area.

[0065] Figure 38This is a top view showing the following example: that is, when the container of the object to be processed by the end effector changes from container 1 to container 2, the position and orientation of container 1 are approximately the same as those of container 2.

[0066] Figure 39 This is a top view illustrating the following example: when the container for the object to be processed by the end effector changes from container 1 to container 2, the position of container 1 is different from that of container 2, and / or the orientation of container 1 is different from that of container 2.

[0067] Figure 40 This is a top view showing an example of the movement of a container holding an object to be processed by an end effector.

[0068] Figure 41 It is a flowchart illustrating the process of robot control processing, including the process of changing a specified area set for the first container to a specified area for the second container.

[0069] Figure 42 The output device shows the specified area before and after the change.

[0070] Figure 43 This is a top view illustrating the following example: when the container for the object to be processed by the end effector changes from a second container to a third container, the position of the second container is different from that of the third container, and / or the orientation of the second container is different from that of the third container.

[0071] Figure 44 This is a top view showing the following example: In the case where the container for the object to be processed by the end effector changes from container 1 to container 2, the size of container 1 and container 2 are approximately the same.

[0072] Figure 45 This is a top view showing the following example: In the case where the container for the object to be processed by the end effector changes from container 1 to container 2, the size of container 1 is different from that of container 2.

[0073] Figure 46 This is a top view showing the following example: In the case where the container for the object to be processed by the end effector changes from the second container to the third container, the size of the first container is different from that of the second container.

[0074] Figure 47 It is a top view showing an example of a specified area.

[0075] Figure 48This is a table showing an example of the specifications of a shooting system.

[0076] Figure 49 This is a block diagram showing the structure of a robot system equipped with a measurement system. Detailed Implementation

[0077] Next, the implementation methods of the control device, control system, robot system, control method, and computer program will be described. Hereinafter, the implementation methods of the control device, control system, robot system, control method, and computer program will be described using the robot system SYS.

[0078] (1) Structure of the robot system SYS

[0079] First, the structure of the robot system SYS will be explained.

[0080] (1-1) Overall structure of robot system SYS

[0081] First, refer to Figure 1 The overall structure of the robot system SYS is explained. Figure 1 This is a block diagram showing the overall structure of the robot system SYS.

[0082] like Figure 1 As shown, the robot system SYS includes a robot 1, a camera system 2, a control device 3, and an end effector 4. Furthermore, the camera system 2 can also be referred to as a camera unit.

[0083] Robot 1 is a device capable of performing prescribed processing on object OBJ. Figure 2 An example of robot 1 is shown in the figure. Figure 2 This is a side view showing the appearance of robot 1. (As shown) Figure 2 As shown, robot 1 includes, for example, a base 11, a robotic arm 12, and a robot control device 13.

[0084] The base 11 is a component that serves as the foundation of the robot 1. The base 11 is disposed on a supporting surface S, such as the ground. The base 11 may be fixed to the supporting surface S. Alternatively, the base 11 may be movable relative to the supporting surface S. As an example, the base 11 may also be able to move independently on the supporting surface S. In this case, the base 11 may also be mounted on an automated guided vehicle (AGV). When the base 11 is mounted on an AGV, the AGV can be considered part of the robot 1. Alternatively, the AGV may also be used as the base 11. Furthermore, Figure 2 An example is shown where the base 11 is fixed to the support surface S.

[0085] Robotic arm 12 is mounted on base 11. Robotic arm 12 is a device that connects multiple links 121 via joints 122. Each joint 122 has a built-in actuator. The links 121 can rotate about an axis defined by the joint 122 via the actuators built into the joints 122. Alternatively, at least one link 121 can be extended or retracted along the direction in which it extends. Alternatively, the device comprising multiple links 121 connected via joints 122 and the base 11 can also be referred to as robotic arm 12.

[0086] The robotic arm 12 is equipped with an end effector 4. That is, the robot 1 is equipped with an end effector 4. Figure 2 In the example shown, the end effector 4 is mounted on the front end of the robotic arm 12. The end effector 4 is movable by the movement of the robotic arm 12. That is, the robotic arm 12 moves the end effector 4. In other words, the robot 1 moves the end effector 4.

[0087] The end effector 4 is a device that performs prescribed processing (in other words, prescribed actions) on the object OBJ. The end effector 4 that performs prescribed processing on the object OBJ can also be called a processing device.

[0088] For example, as one example of a specified process, the end effector 4 can perform a holding process for holding the object OBJ. In this case, the end effector 4 can be considered as performing a holding process on the object OBJ that the end effector 4 should hold. The end effector 4 capable of performing the holding process can be referred to as a holding device.

[0089] For example, holding the object OBJ may include clamping the object OBJ. Holding the object OBJ may include using a manual gripper (described later as an example of end effector 4) to clamp the object OBJ. Holding the object OBJ may include adsorbing the object OBJ. Holding the object OBJ may include using a vacuum gripper (described later as an example of end effector 4) to adsorb (vacuum adsorption) the object OBJ. Holding the object OBJ may include using a magnetic gripper (described later as an example of end effector 4) to adsorb the object OBJ.

[0090] For example, as an example of a specified process, the end effector 4 can perform a release process (i.e., separation) of the held object OBJ (in other words, a release action). In this case, the end effector 4 can be regarded as performing a release process on the object OBJ held by the end effector 4. The end effector 4 capable of performing the release process can be referred to as a release device.

[0091] As an example of an end effector 4 capable of performing holding and releasing operations, a manual gripper can be cited. A manual gripper is an end effector 4 capable of physically gripping an object OBJ using multiple (e.g., 2, 3, or 4) finger or claw components to hold the object OBJ. As another example of an end effector 4 capable of performing holding and releasing operations, a vacuum gripper can be cited. A vacuum gripper is an end effector 4 capable of holding an object OBJ by vacuum suction. As another example of an end effector 4 capable of performing holding and releasing operations, a magnetic gripper can be cited. Figure 2 An example of an end effector 4 as a manual gripper is shown. However, the end effector 4, capable of performing hold and release operations, can also be other existing end effectors.

[0092] As an example of the specified processing, robot 1 can use end effector 4, which is capable of holding and releasing, to perform a configuration process (in other words, a configuration action) for positioning object OBJ in a desired position. For example, robot 1 can use end effector 4 to hold a first object OBJ, and then perform a configuration process for positioning the first object OBJ held by end effector 4 in a desired position of a second object OBJ, which is different from the first object OBJ. In this case, end effector 4 can be regarded as performing a release process on the second object OBJ that end effector 4 is to position the first object OBJ as. Similarly, it can be regarded as end effector 4 performing a release process on the first object OBJ that end effector 4 should release.

[0093] As a concrete example of configuration processing (in other words, configuration action), robot 1 can use an end effector 4 capable of holding and releasing to perform an embedding process (in other words, embedding action) for embedding a first object OBJ into a second object OBJ that is different from the first object OBJ. For example, robot 1 can use the end effector 4 to hold the first object OBJ and then perform an embedding process for embedding the first object OBJ held by the end effector 4 into a second object OBJ that is different from the first object OBJ. In this case, the end effector 4 can be regarded as the end effector 4 releasing the second object OBJ into which the first object OBJ is to be embedded. Similarly, it can be regarded as the end effector 4 releasing the first object OBJ that the end effector 4 should release. In addition, the embedding process can also be called a processing process.

[0094] As an example, the embedding process can include a process for embedding (i.e., inserting) a first object OBJ into a hole formed in a second object OBJ. For example, robot 1 can use an end effector 4 to hold the first object OBJ, and then perform an embedding process for embedding (i.e., inserting) the first object OBJ held by the end effector 4 into a hole formed in the second object OBJ. In this case, it can be viewed as the end effector 4 performing a release process on the second object OBJ with the hole in which the end effector 4 should embed (i.e. insert) the first object OBJ. Similarly, it can be viewed as the end effector 4 performing a release process on the first object OBJ that the end effector 4 should release.

[0095] As a specific example of configuration processing (in other words, configuration action), robot 1 can use an end effector 4 capable of holding and releasing to perform a pasting process (in other words, pasting action) for pasting a first object OBJ to a second object OBJ that is different from the first object OBJ. For example, robot 1 can use the end effector 4 to hold the first object OBJ and then perform a pasting process for pasting the first object OBJ held by the end effector 4 to the second object OBJ that is different from the first object OBJ. In this case, it can be regarded as the end effector 4 releasing the second object OBJ to which the end effector 4 should have pasted the first object OBJ. Similarly, it can be regarded as the end effector 4 releasing the first object OBJ that the end effector 4 should have released. In addition, pasting processing can also be called processing.

[0096] As a specific example of configuration processing (in other words, configuration action), robot 1 can use an end effector 4 capable of performing hold and release processing to perform a bonding process (in other words, bonding action) for attaching a first object OBJ to a second object OBJ that is different from the first object OBJ. For example, robot 1 can use the end effector 4 to hold the first object OBJ and then perform a bonding process for attaching the first object OBJ held by the end effector 4 to a second object OBJ that is different from the first object OBJ. In this case, the end effector 4 can be regarded as performing a release process on the second object OBJ to which the end effector 4 should attach the first object OBJ. Similarly, it can be regarded as the end effector 4 performing a release process on the first object OBJ to which the end effector 4 should release the first object OBJ. Furthermore, the bonding process can be referred to as a machining process.

[0097] As a specific example of configuration processing (in other words, configuration action), robot 1 can use an end effector 4 capable of holding and releasing to perform a welding process (in other words, welding action) for welding a first object OBJ to a second object OBJ that is different from the first object OBJ. For example, robot 1 can use the end effector 4 to hold the first object OBJ and then perform a welding process for welding the first object OBJ held by the end effector 4 to the second object OBJ that is different from the first object OBJ. In this case, it can be regarded as the end effector 4 releasing the second object OBJ to which the end effector 4 should weld the first object OBJ. Similarly, it can be regarded as the end effector 4 releasing the first object OBJ that the end effector 4 should release. Furthermore, the welding process can be referred to as a machining process.

[0098] As a specific example of configuration processing (in other words, configuration action), robot 1 can use an end effector 4 capable of holding and releasing to perform a screw-tightening process (in other words, screw-tightening action) for screwing a first object OBJ, which functions as a screw, into a screw hole formed on a second object OBJ that is different from the first object OBJ. For example, robot 1 can use the end effector 4 to hold the first object OBJ and then perform a screw-tightening process for screwing the first object OBJ held by the end effector 4 into the second object OBJ that is different from the first object OBJ. In this case, it can be regarded as the end effector 4 releasing the second object OBJ that the end effector 4 should tighten on the first object OBJ. Similarly, it can be regarded as the end effector 4 releasing the first object OBJ that the end effector 4 should release. In addition, the screw-tightening process can also be referred to as a machining process.

[0099] For example, the end effector 4 can perform specified processing on each of multiple object OBJs. That is, the end effector 4 can perform specified processing on multiple object OBJs sequentially. In this case, the robot 1 can move the end effector 4 to a first position where it can perform specified processing on the first object OBJ. After moving to the first position, the end effector 4 can perform specified processing on the first object OBJ. Then, the robot 1 can move the end effector 4 to a second position where it can perform specified processing on a second object OBJ that is different from the first object OBJ. After moving to the second position, the end effector 4 can perform specified processing on the second object OBJ.

[0100] For example, the end effector 4 can perform predefined processing on each of multiple parts of a single object OBJ. That is, the end effector 4 can sequentially perform predefined processing on multiple parts of a single object OBJ. In this case, the robot 1 can move the end effector 4 to a third position where the end effector 4 can perform predefined processing on the first part of the object OBJ. After moving to the third position, the end effector 4 can perform predefined processing on the first part of the object OBJ. Then, the robot 1 can move the end effector 4 to a fourth position where the end effector 4 can perform predefined processing on the second part of the object OBJ. After moving to the fourth position, the end effector 4 can perform predefined processing on the second part of the object OBJ.

[0101] As an example of an action that performs a prescribed process on each of multiple parts of a single object OBJ, robot 1 can repeatedly perform an embedding process on a second object OBJ having multiple holes, embedding the first object OBJ into each hole. Furthermore, a portion of the object OBJ that is subjected to prescribed processing by the end effector 4 can be referred to as an object part or object component. For example, each part (component) of the second object OBJ having each hole can be referred to as an object part or object component. Furthermore, when the object OBJ is composed of multiple parts, each part can be referred to as an object part or object component. When the object OBJ is composed of multiple components, each component can be considered an object part or object component.

[0102] like Figure 2 As shown, the object OBJ subjected to prescribed processing by the end effector 4 may contain a workpiece W. The workpiece W may include, for example, components or parts used to manufacture the desired product. The workpiece W may include, for example, components or parts processed for manufacturing the desired product. The workpiece W may include, for example, components or parts transported for manufacturing the desired product. The workpiece W may contain, for example, components or parts that move by transport for manufacturing the desired product. The workpiece W may contain, for example, components or parts that move for manufacturing the desired product.

[0103] like Figure 2As shown, the object OBJ subjected to specified processing by the end effector 4 may include a loading device T for placing the workpiece W. As an example of the loading device T, a container (storage box) CB can be cited. The container CB can be the loading device T, which includes a bottom wall BS and a side wall SS projecting upward from the bottom wall BS. The workpiece W is placed on the bottom wall BS within a storage space SP surrounded by the bottom wall BS and the side wall SS. The container CB can be the loading device T, which includes a bottom wall BS and a side wall SS projecting upward from the bottom wall BS, capable of storing the workpiece W within the storage space SP surrounded by the bottom wall BS and the side wall SS. Furthermore, in addition to or as an alternative to at least a portion of the space surrounded by the bottom wall BS and the side wall SS, the storage space SP may also include at least a portion of the space above the opening AP of the container CB, connecting the space surrounded by the bottom wall BS and the side wall SS to the external space of the container CB. However, the container CB may also lack the side wall SS. A container CB without the side wall SS can be referred to as a tray. Furthermore, the loading device T can be a tray. Furthermore, the mounting device T is not limited to containers CB or pallets, but can also be any existing object capable of mounting workpiece W. Additionally, the mounting device T can also be referred to as a mounting component. The mounting device T can also be disposed on a supporting surface S. The mounting device T can also be fixed to the supporting surface S. Alternatively, at least a portion of the mounting device T can be movable relative to the supporting surface S.

[0104] As a first example, where at least a part of the carrying device T can move relative to the support surface S, an example can be given of the carrying device T being supported by a conveyor device that allows the carrying device T to move (in other words, is transferable). As a first example of a conveyor device, a conveyor device capable of moving independently on the support surface S can be cited. In this case, the carrying device T can be placed on or held by the self-moving conveyor device. Furthermore, the self-moving conveyor device can be referred to as an Automatic Guided Vehicle (AGV). As a second example of a conveyor device, a belt conveyor can be cited. In this case, the carrying device T can be placed on a conveyor belt that is part of the belt conveyor and movable relative to the support surface S. As a third example of a conveyor device, a flying device capable of flying on the support surface S can be cited. In this case, the carrying device T can be placed on or held by the flying conveyor device. Furthermore, the flying conveyor device can be referred to as an unmanned aerial vehicle (UAV) or a drone. As a fourth example of a conveying device, another robotic arm, different from robotic arm 12, may be used, and may be equipped with an end effector capable of holding the loading device T. In this case, the loading device T may be held by the end effector mounted on the other robotic arm.

[0105] As a second example of a movable loading device T, at least a portion thereof may be movable relative to the support surface S. An example where the loading device T itself is movable relative to the support surface S may be given. As a first example of a movable loading device T, a loading device T capable of moving autonomously on the support surface S may be given. Furthermore, a autonomously movable loading device T may be referred to as an Automatic Guided Vehicle (AGV). As a second example of a movable loading device T, a belt conveyor may be given. In this case, the conveyor belt, which is part of the belt conveyor and carries the workpiece W, can move relative to the support surface S. As a third example of a movable loading device T, a loading device T capable of flying on the support surface S may be given. In this case, the loading device T can move relative to the support surface S by flying on it. Furthermore, a flying loading device T may be referred to as an unmanned aerial vehicle (UAV) or drone. As a fourth example of a movable loading device T, another robotic arm, different from robotic arm 12, may be given, which is equipped with an end effector capable of holding the workpiece W. In this case, the end effector mounted on other robotic arms can move relative to the support surface S.

[0106] When the conveying device moves the mounting device T, the workpiece W mounted on the mounting device T also moves relative to the supporting surface S. Therefore, the conveying device of the movable mounting device T can be considered equivalent to a conveying device capable of conveying the workpiece W. Similarly, when the mounting device T moves, the workpiece W mounted on the mounting device T also moves relative to the supporting surface S. Therefore, the movable mounting device T can be considered equivalent to a moving device capable of moving the workpiece W (in other words, a conveying device capable of conveying the workpiece W). Furthermore, Figure 2 An example is shown where the mounting device T can move independently on the support surface S.

[0107] Furthermore, if at least a portion of the mounting device T is movable relative to the support surface S, the first mounting device T on which the workpiece W is mounted can be mounted on a second mounting device T that is movable relative to the support surface S. In this case, the device comprising the first mounting device T and the second mounting device T can be referred to as the mounting device T.

[0108] However, the object OBJ may not include the mounting device T. Furthermore, the workpiece W may not be mounted on the mounting device T, or may not have a mounting device T at all. For example, the workpiece W may be mounted on the support surface S.

[0109] When the object OBJ includes a workpiece W and a mounting device T, the above-described holding process may include holding the workpiece W mounted on the mounting device T, whether it is stationary or moving. The above-described holding process may also include holding the workpiece W mounted on the support surface S. The above-described release process may include releasing the workpiece W held by the end effector 4 to position the workpiece W held by the end effector 4 at a desired position on the mounting device T, whether it is stationary or moving. The above-described release process may include releasing the workpiece W held by the end effector 4 to position the workpiece W held by the end effector 4 at a desired position on the support surface S. The above-described release process may include releasing a first workpiece W held by the end effector 4 to embed the first workpiece W held by the end effector 4 into a second workpiece W mounted on the mounting device T, whether it is stationary or moving. The aforementioned release process may include releasing the first workpiece W held by the end effector 4 to embed the first workpiece W held by the end effector 4 into a second workpiece W placed on the support surface S. The aforementioned release process may also include releasing the first workpiece W held by the end effector 4 to embed (i.e., insert) the first workpiece W held by the end effector 4 into a hole formed in the second workpiece W placed on the mounting device T, which is either stationary or moving. The aforementioned release process may also include releasing the first workpiece W held by the end effector 4 to embed (i.e., insert) the first workpiece W held by the end effector 4 into a hole formed in the second workpiece W placed on the support surface S. Furthermore, the object OBJ may include either the workpiece W or the mounting device T.

[0110] The robot control device 13 controls the actions of the robot 1.

[0111] Specifically, the robot control device 13 can control the movements of the robotic arm 12. For example, the robot control device 13 can control the movements of the robotic arm 12 such that the desired link 121 rotates about an axis specified by the desired connector 122. For example, the robot control device 13 can control the movements of the robotic arm 12 such that the end effector 4 mounted on the robotic arm 12 is positioned in a desired location. For example, the robot control device 13 can control the movements of the robotic arm 12 such that the end effector 4 mounted on the robotic arm 12 moves to a desired position.

[0112] In addition to controlling the actions of the robot 1, the robot control device 13 can also control the actions of the end effector 4 installed on the robot 1. For example, the robot control device 13 can control the actions of the end effector 4 so that the end effector 4 holds the object OBJ at a desired timing. That is, the robot control device 13 can control the actions of the end effector 4 so that the end effector 4 performs holding operations at a desired timing. For example, the robot control device 13 can control the actions of the end effector 4 so that the end effector 4 releases the held object OBJ at a desired timing. That is, the robot control device 13 can control the actions of the end effector 4 so that the end effector 4 performs release operations at a desired timing. When the end effector 4 is a manual gripper, the robot control device 13 can also control the timing of opening and closing the manual gripper. When the end effector 4 is a vacuum gripper, the robot control device 13 can also control the timing of turning the vacuum device of the vacuum gripper on / off. When the end effector 4 is a magnetic gripper, the robot control unit 13 can also control the timing of turning the magnetic device of the magnetic gripper on / off.

[0113] in addition, Figure 2 An example of robot 1 being a robotic arm 12 (i.e., a vertical jointed robot) is shown. However, robot 1 can also be a robot different from a vertical jointed robot. For example, robot 1 can also be a SCARA robot (i.e., a horizontal jointed robot). For example, robot 1 can also be a parallel linkage robot. For example, robot 1 can also be a dual-arm robot with two robotic arms 12. For example, robot 1 can also be an orthogonal coordinate robot. For example, robot 1 can also be a cylindrical coordinate robot. Robot 1 can also be referred to as a movable device. In addition to robot 1, the movable device can also include at least one of an automated guided vehicle and a drone. For example, robot 1 can also be configured with at least one of an automated guided vehicle and a drone.

[0114] exist Figure 1 In this process, the imaging system 2 again captures an image of the object OBJ. To capture the object OBJ, the imaging system 2 includes an imaging device 21, an imaging device 22, and a projection device 23. Furthermore, the imaging system 2 can also be referred to as an imaging unit.

[0115] The imaging device 21 is a camera capable of capturing images of the object OBJ. In this embodiment, the imaging device 21 is a monocular camera. The imaging device 21 generates image data IMG_2D by capturing images of the object OBJ. That is, the imaging device 21 generates the image data IMG_2D of the object OBJ. The image data IMG_2D generated by the imaging device 21 is output from the imaging device 21 to the control device 3. As a result, the control device 3 acquires the image data IMG_2D obtained by the imaging device 21 capturing images of the object OBJ. In this embodiment, the imaging device 21 is a monocular camera. Specifically, the imaging device 21 can use a monocular camera (in other words, an imaging element) to capture images of the object OBJ. Specifically, since the imaging device 21 is a monocular camera, the imaging device 21 generates one image data generated by the monocular camera as the image data IMG_2D. In this case, the image data IMG_2D, which is equivalent to one image data (i.e., representing an image), can be called monocular image data. In addition, the imaging device 21 is not limited to a monocular camera. The imaging device 21 can be a stereo camera capable of capturing images of the object OBJ using two monocular cameras, or it can include three or more monocular cameras. Alternatively, the imaging device 21 can be at least one of a light field camera, a pre-positioning camera, and a multispectral camera.

[0116] Similar to the imaging device 21, the imaging device 22 is a camera capable of capturing images of the object OBJ. In this embodiment, the imaging device 22 is a stereo camera. Specifically, the imaging device 22 is a stereo camera capable of capturing images of the object OBJ using two monocular cameras (in other words, two imaging elements). The imaging device 22 generates image data IMG_3D by capturing images of the object OBJ. That is, the imaging device 22 generates the image data IMG_3D of the object OBJ. Specifically, since the imaging device 22 is a stereo camera, the imaging device 22 generates image data IMG_3D that includes two image data generated by the two monocular cameras respectively. In this case, the image data IMG_3D containing two image data (i.e., representing two images) can be referred to as stereo image data. The image data IMG_3D generated by the imaging device 22 is output from the imaging device 22 to the control device 3. As a result, the control device 3 acquires the image data IMG_3D obtained by the imaging device 22 capturing images of the object OBJ. In addition, the imaging device 22 is not limited to a stereo camera. The shooting device 22 can be a monocular camera, or it can include three or more monocular cameras. Alternatively, the shooting device 22 can also be at least one of a light field camera, a pre-positioning camera, and a multispectral camera.

[0117] Projection device 23 is a device capable of irradiating projection light onto object OBJ. Specifically, projection device 23 is a device capable of projecting a desired projection pattern onto object OBJ by irradiating projection light onto object OBJ. Furthermore, the projection pattern can also be referred to as the intensity distribution of the light projected onto object OBJ. If the intensity distribution of the projected light changes, the projection pattern will also change. The desired projection pattern can include, for example, a random pattern. A random pattern can be a projection pattern where each unit illumination area has a different pattern. A random pattern can also include a random dot pattern. The desired projection pattern can include, for example, a one-dimensional or two-dimensional grid pattern. The desired projection pattern can include, for example, a linear pattern. The desired projection pattern can include, for example, a striped pattern. The desired projection pattern can include other projection patterns. Imaging device 22 captures an image of object OBJ onto which the projection pattern from projection device 23 has been projected. In this case, the image showing the image data IMG_3D captures an image of object OBJ onto which the projection pattern has been projected. On the other hand, imaging device 21 may not capture an image of object OBJ onto which the projection pattern has been projected. The imaging device 21 can also capture images of the object OBJ without a projected projection pattern. In this case, the object OBJ with the projected projection pattern may not be captured in the image shown in the image data IMG_2D. Alternatively, the object OBJ without a projected projection pattern may be captured in the image shown in the image data IMG_2D. Furthermore, the projection light used to project the desired projection pattern onto the object OBJ can be called pattern light or structured light. In this case, the projection light may include both pattern light and structured light. Additionally, the projection light may be light with a uniform projection pattern (i.e., a uniform intensity distribution).

[0118] Alternatively, the projection device 23 can also be considered as illuminating the object OBJ by projecting projection light onto it. In this case, the projection device 23 can function as an illumination device for illuminating the object OBJ. When the projection device 23 functions as an illumination device, the projection light can also be called illumination light. The projection light (illumination light) used as an illumination device can also be light with a uniform intensity distribution. When the projection device 23 functions as an illumination device, the projection light may not be light capable of projecting the desired projection pattern onto the object OBJ. When the projection device 23 functions as an illumination device, the projection light (i.e., illumination light) can be any light capable of illuminating the object OBJ.

[0119] The imaging device 21 can photograph the entire object OBJ. Alternatively, the imaging device 21 can photograph a portion of the object OBJ. That is, the imaging device 21 can photograph a portion of the object OBJ without photographing the other portion. Similarly, the imaging device 22 can photograph the entire object OBJ. Alternatively, the imaging device 22 can photograph a portion of the object OBJ. That is, the imaging device 22 can photograph a portion of the object OBJ without photographing the other portion.

[0120] The imaging device 21 can capture images of a single object OBJ. That is, a single object OBJ can be captured in the image data IMG_2D. Alternatively, the imaging device 21 can capture images of multiple object OBJs. That is, multiple object OBJs can be captured in the image data IMG_2D. In this case, as detailed later, the control device 3 can decide (in other words, select) one of the multiple object OBJs captured by the imaging device 21 as the object OBJ that the end effector 4 actually performs the prescribed processing on. Furthermore, the object OBJ that the end effector 4 actually performs the prescribed processing on from the multiple object OBJs captured by the imaging device 21 can be referred to as the processing execution object.

[0121] The imaging device 22 can capture images of a single object OBJ. That is, a single object OBJ can be captured in the image shown in the image data IMG_3D. Alternatively, the imaging device 22 can capture images of multiple object OBJs. That is, multiple object OBJs can be captured in the image shown in the image data IMG_3D. In this case, as detailed later, the control device 3 can decide (in other words, select) one of the multiple object OBJs captured by the imaging device 22 as the object OBJ that the end effector 4 actually performs the prescribed processing on. Furthermore, the object OBJ that the end effector 4 actually performs the prescribed processing on from the multiple object OBJs captured by the imaging device 21 can be referred to as the processing execution object.

[0122] When imaging devices 21 and 22 respectively capture multiple object OBJs, the multiple object OBJs captured by imaging device 21 can be the same as the multiple object OBJs captured by imaging device 22. That is, the multiple object OBJs captured in the image data IMG_2D can be the same as the multiple object OBJs captured in the image data IMG_3D. Alternatively, at least one of the multiple object OBJs captured by imaging device 21 can be different from at least one of the multiple object OBJs captured by imaging device 21. That is, at least one of the multiple object OBJs captured in the image data IMG_2D can be the same as at least one of the multiple object OBJs captured in the image data IMG_3D.

[0123] Multiple object objects OBJ captured by at least one of the imaging devices 21 and 22 can be configured such that at least two of the multiple object objects OBJ at least partially overlap. As an example, in the case where the object objects OBJ are workpieces W equivalent to components used to manufacture a desired product, multiple workpieces W (i.e., multiple components) can be configured such that at least two of the multiple workpieces W at least partially overlap. In this case, robot 1 can perform bulk picking, picking up workpieces W one by one from the randomly configured multiple workpieces W.

[0124] The imaging system 2 is mounted on the robotic arm 12 in the same manner as the end effector 4. That is, the imaging devices 21 and 22 and the projection device 23 are mounted on the robotic arm 12. For example, as... Figure 2 As shown, the imaging devices 21 and 22 and the projection device 23 can also be mounted on the front end of the robotic arm 12 in the same manner as the end effector 4. In this case, the imaging devices 21 and 22 and the projection device 23 can be moved by the movement of the robotic arm 12. That is, the robotic arm 12 moves the imaging devices 21 and 22 and the projection device 23.

[0125] However, the imaging system 2 may not be mounted on the robotic arm 12. The imaging system 2 can be mounted at any location capable of projecting light onto the object OBJ and capturing images of the object OBJ. In this case, for example, the imaging system 2 can be mounted on a structure such as a column, so that it can project light onto the object OBJ and capture images of the object OBJ. Alternatively, at least one of the imaging device 21, imaging device 22, and projection device 23 may be mounted on the robotic arm 12, and at least one other of the imaging device 21, imaging device 22, and projection device 23 may be mounted at a different location from the robotic arm 12 (e.g., a structure such as a column). If at least one of the imaging device 21 and imaging device 22 is located at a different location from the robotic arm 12, at least one of the imaging device 21 and imaging device 22 may also be mounted on a structure such as a column configured to capture images of the object OBJ. Furthermore, if the projection device 23 is mounted at a different location from the robotic arm 12, the projection device 23 may also be mounted on a structure such as a column configured to project light onto the object OBJ.

[0126] The imaging device 21 can capture images of the object OBJ during the period of relative displacement between the imaging device 21 and the object OBJ. Furthermore, the state of relative displacement between the imaging device 21 and the object OBJ can represent the state of change in their relative positional relationship. The state of relative displacement between the imaging device 21 and the object OBJ can also represent the state of relative movement between them. For example, the state of phase displacement between the imaging device 21 and the object OBJ can include the state of movement of the object OBJ relative to the imaging device 21. In this case, the imaging device 21 does not need to remain stationary to capture images of the object OBJ; therefore, the robot system SYS can efficiently perform prescribed processing on the object OBJ using the end effector 4.

[0127] Alternatively, the imaging device 21 can capture images of the object OBJ while there is no relative displacement between the imaging device 21 and the object OBJ. Furthermore, the state where there is no relative displacement between the imaging device 21 and the object OBJ can represent a state where the relative positional relationship between the imaging device 21 and the object OBJ remains unchanged. The state where there is no relative displacement between the imaging device 21 and the object OBJ can also represent a state where there is no relative movement between the imaging device 21 and the object OBJ. The state where there is no relative displacement between the imaging device 21 and the object OBJ can also represent a state where the imaging device 21 and the object OBJ are stationary. The state where there is no relative displacement between the imaging device 21 and the object OBJ can also represent a state where the imaging device 21 and the object OBJ are moving at the same speed in the same direction.

[0128] The imaging device 22 can capture images of the object OBJ during the period of relative displacement between the imaging device 22 and the object OBJ. Furthermore, the state of relative displacement between the imaging device 22 and the object OBJ can represent the change in their relative positional relationship. The state of relative displacement between the imaging device 22 and the object OBJ can also represent the state of relative movement between them. For example, the state of phase displacement between the imaging device 22 and the object OBJ can include the state of movement of the object OBJ relative to the imaging device 22. In this case, the imaging device 22 does not need to remain stationary to capture the object OBJ; therefore, the robot system SYS can efficiently perform prescribed processing on the object OBJ using the end effector 4.

[0129] Alternatively, the imaging device 22 can capture images of the object OBJ while there is no relative displacement between the imaging device 22 and the object OBJ. Furthermore, the state where there is no relative displacement between the imaging device 22 and the object OBJ can represent a state where the relative positional relationship between the imaging device 22 and the object OBJ remains unchanged. The state where there is no relative displacement between the imaging device 22 and the object OBJ can also represent a state where there is no relative movement between the imaging device 22 and the object OBJ. The state where there is no relative displacement between the imaging device 22 and the object OBJ can also represent a state where the imaging device 22 and the object OBJ are stationary. The state where there is no relative displacement between the imaging device 22 and the object OBJ can also represent a state where the imaging device 22 and the object OBJ are moving at the same speed in the same direction.

[0130] The imaging devices 21 and 22 can also simultaneously capture images of the object OBJ. For example, imaging devices 21 and 22 can capture images of the object OBJ at the same time. That is, imaging devices 21 and 22 can capture images of the object OBJ such that the 2D capture time of imaging device 21 capturing the object OBJ and the 3D capture time of imaging device 22 capturing the object OBJ are the same. Imaging devices 21 and 22 can capture images of the object OBJ such that the 2D capture time of imaging device 21 capturing the object OBJ to generate image data IMG_2D and the 3D capture time of imaging device 22 capturing the object OBJ to generate image data IMG_3D are the same.

[0131] The shooting devices 21 and 22 can also capture images of the object OBJ under the control of the control device 3. In this case, the control device 3 can control the timing (in other words, the timing) at which the shooting devices 21 and 22 capture images of the object OBJ. For example, the control device 3 can control the shooting devices 21 and 22 so that they capture images of the object OBJ simultaneously and synchronously. That is, the control device 3 can control the shooting devices 21 and 22 so that the 2D shooting time and the 3D shooting time are the same.

[0132] Here, the state of "the 2D shooting time and the 3D shooting time are the same" can include the state of "the 2D shooting time and the 3D shooting time are literally exactly the same." The state of "the 2D shooting time and the 3D shooting time are the same" can also include the state of "although the 2D shooting time and the 3D shooting time are not exactly the same, because the time deviation between the 2D shooting time and the 3D shooting time is smaller than the first allowable upper limit, they are considered to be substantially the same." Here, the first allowable upper limit can also be a first allowable upper limit based on the control error of the robotic arm 12. For example, there are cases where, due to the time deviation between the 2D shooting time and the 3D shooting time, errors occur in the calculation results of at least one of the position and orientation of the object OBJ (described later) (i.e., the accuracy of at least one of the calculated position and orientation of the object OBJ is reduced). In this case, due to the errors generated in the calculation results of at least one of the position and orientation of the object OBJ, control errors of the robotic arm 12 may sometimes occur. The control error of the robotic arm 12 becomes the movement error of the end effector 4, which sometimes prevents the end effector 4 from properly processing the object OBJ. The first permissible upper limit can be set to an appropriate value that avoids a situation where the end effector 4 cannot properly process the object OBJ due to this control error of the robotic arm 12. Alternatively, the first permissible upper limit can be considered equivalent to the permissible upper limit of the movement error of the end effector 4 caused by the robotic arm 12. Furthermore, for example, even if there is a time deviation between the 2D and 3D shooting times due to the synchronization error of the shooting processes of the shooting devices 21 and 22, the 2D and 3D shooting times can be considered to be substantially the same time. Additionally, the synchronization error of the shooting processes of the shooting devices 21 and 22 can also be the synchronization control error of the shooting processes of the shooting devices 21 and 22 performed by the control device 3.

[0133] However, the imaging devices 21 and 22 may not simultaneously image the object OBJ. That is, the imaging devices 21 and 22 can image the object OBJ such that the 2D imaging time of the imaging device 21 and the 3D imaging time of the imaging device 22 are the same. Furthermore, the state where "the 2D imaging time and the 3D imaging time are different times" can also include the state where "the time difference between the 2D imaging time and the 3D imaging time is greater than the first allowable upper limit, therefore they cannot be considered to be substantially the same time."

[0134] In this embodiment, when the imaging devices 21 and 22 are not relatively displaced from the object OBJ, they can still capture images of the object OBJ, making the 2D and 3D shooting times the same. In other words, when the imaging devices 21 and 22 are capturing images of the object OBJ during periods of relative displacement, the control device 3 can control the imaging devices 21 and 22 to make the 2D and 3D shooting times the same.

[0135] On the other hand, when the shooting devices 21 and 22 are shooting the object OBJ during a period when there is no relative displacement between them and the object OBJ, the shooting devices 21 and 22 may not shoot the object OBJ, so that the 2D shooting time and the 3D shooting time are the same. That is, the control device 3 may not control the shooting devices 21 and 22 to make the 2D shooting time and the 3D shooting time the same. For example, the shooting devices 21 and 22 may shoot the object OBJ, so that the 2D shooting time and the 3D shooting time are different. That is, the control device 3 may control the shooting devices 21 and 22 to make the 2D shooting time and the 3D shooting time different. However, when the shooting devices 21 and 22 are shooting the object OBJ during a period when there is no relative displacement between them and the object OBJ, the shooting devices 21 and 22 may also shoot the object OBJ, so that the 2D shooting time and the 3D shooting time are the same. That is, the control device 3 can control the shooting devices 21 and 22 so that the 2D shooting time and the 3D shooting time are the same.

[0136] The control device 3 performs robot control processing. This robot control processing may include generating robot control signals for controlling the robot 1. Specifically, the control device 3 generates robot control signals based on at least one of the image data IMG_2D and IMG_3D output from the imaging system 2. In this embodiment, the control device 3 calculates at least one of the position and orientation of the object OBJ within the global coordinate system of the robot system SYS based on at least one of the image data IMG_2D and IMG_3D, and generates robot control signals based on at least one of the calculated position and orientation of the object OBJ.

[0137] The global coordinate system is the coordinate system that serves as the reference for the robot system SYS. For example, the global coordinate system can also be the coordinate system that serves as the reference for robot 1. Furthermore, it can be said that the global coordinate system is the coordinate system used for controlling robot 1. As a global coordinate system, for example, a world coordinate system defined with the support surface S on which robot 1 is configured can be used. That is, as a global coordinate system, a world coordinate system that is fixed relative to the support surface S on which robot 1 is configured can be used.

[0138] However, the control device 3 can calculate at least one of the position and orientation of the object in a coordinate system different from the global coordinate system based on at least one of the image data IMG_2D and IMG_3D. As a first example, the control device 3 can calculate at least one of the position and orientation of the object OBJ in the robot coordinate system based on at least one of the image data IMG_2D and IMG_3D. The robot coordinate system can be a coordinate system determined with respect to the robot 1. That is, the robot coordinate system can be a coordinate system that is fixed relative to the robot 1 (e.g., fixed relative to the base 11 of the robot 1). Furthermore, since it is a coordinate system determined with respect to the robot 1, the robot coordinate system can be referred to as the global coordinate system. Moreover, it can be said that the robot coordinate system is a coordinate system used for controlling the robot 1.

[0139] As a second example, the control device 3 can calculate at least one of the position and orientation of the object OBJ in the 2D shooting coordinate system based on at least one of the image data IMG_2D and IMG_3D. The 2D shooting coordinate system can be a coordinate system determined with respect to the shooting device 21. That is, the 2D shooting coordinate system can be a coordinate system fixed relative to the shooting device 21. As an example of a 2D shooting coordinate system, an example can be based on the optical axis AX21 (refer to...) of the optical system (especially terminal optical elements such as objective lenses) provided by the shooting device 21. Figure 2 The coordinate system is determined by this. As an example of a 2D shooting coordinate system, any one of the three coordinate axes constituting the 2D shooting coordinate system can be taken as the optical axis AX21 along the optical system (especially terminal optical elements such as objective lenses) provided by the shooting device 21 (refer to...). Figure 2 The coordinate system of the axes.

[0140] As a third example, the control device 3 can calculate at least one of the position and orientation of the object OBJ in the 3D shooting coordinate system based on at least one of the image data IMG_2D and IMG_3D. The 3D shooting coordinate system can be a coordinate system determined with respect to the shooting device 22. That is, the 3D shooting coordinate system can be a coordinate system fixed relative to the shooting device 22. As an example of a 3D shooting coordinate system, an example can be based on the optical axis AX22 (refer to...) of the optical system (especially terminal optical elements such as objective lenses) provided by the shooting device 22. Figure 2 The coordinate system is determined by this. As an example of a 3D shooting coordinate system, one of the three coordinate axes constituting the 3D shooting coordinate system can be taken as the optical axis AX22 along the optical system (especially terminal optical elements such as objective lenses) provided by the shooting device 22 (refer to...). Figure 2 The coordinate system of the axes.

[0141] In addition to performing robot control processing, the control device 3 can also perform end effector control processing. End effector control processing may include generating end effector control signals for controlling the end effector 4. Specifically, the control device 3 can generate the end effector control signals based on at least one of the calculated position and orientation of the object OBJ.

[0142] Furthermore, the end effector control processing may or may not be included in the robot control processing. That is, the end effector control signal generated by the control device 3 may or may not be included in the robot control signal. In the following description, for ease of explanation, an example in which the end effector control processing is included in the robot control processing (i.e., the end effector control signal is included in the robot control signal) will be described. Therefore, in the following description, robot control processing may also mean the processing of generating at least one of robot control signal and end effector control signal. In addition, in the following description, robot control signal may also mean at least one of signal for controlling robot 1 and signal for controlling end effector 4. In addition, robot control signal may also be simply referred to as control signal.

[0143] Therefore, the control device 3 and the imaging system 2 can be used to control the robot 1. Thus, the system including the control device 3 and the imaging system 2 can be referred to as a robot control system or a control system.

[0144] The robot control signal generated by control device 3 is output to robot control device 13 of robot 1. Robot control device 13 controls the actions of robot 1 based on the robot control signal generated by control device 3. Therefore, the robot control signal may also include signals for controlling the actions of robot 1.

[0145] As described above, when the robot control signal includes a signal for controlling the robotic arm 12, the robot control device 13 can also control the robotic arm 12 based on the robot control signal. For example, the robot control device 13 can control the movement of the robotic arm 12 by controlling the movement of the actuator built into the joint 122 based on the robot control signal.

[0146] For example, as described above, the robotic arm 12 moves the end effector 4. In this case, the robot control signal may include a signal for controlling the robotic arm 12 to position the end effector 4 in a desired location. The robot control signal may include a signal for controlling the robotic arm 12 to move the end effector 4 to the desired location. The robot control signal may include a signal for controlling the robotic arm 12 to achieve the desired positional relationship between the end effector 4 and the object OBJ. In this case, the robot control device 13 may control the robotic arm 12 based on the robot control signal to position the end effector 4 in the desired location. The robot control device 13 may control the robotic arm 12 based on the robot control signal to move the end effector 4 to the desired location. The robot control device 13 may control the robotic arm 12 based on the robot control signal to achieve the desired positional relationship between the end effector 4 and the object OBJ.

[0147] As an example, when the end effector 4 is performing a holding process to hold the object OBJ, the robot control signal may include a signal for controlling the robotic arm 12 to move (i.e., approach) the end effector 4 toward the holding position where the end effector 4 can hold the object OBJ. That is, the robot control signal may include a signal for controlling the robotic arm 12 to position the end effector 4 in the holding position. In this case, the robot control device 13 may control the robotic arm 12 based on the robot control signal to move (i.e., approach) the end effector 4 toward the holding position. In other words, the robot control signal may control the robotic arm 12 to position the end effector 4 in the holding position.

[0148] As another example, in the case of performing a release process for releasing the object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the robotic arm 12 to move (i.e., approach) the end effector 4 toward the release position where the object OBJ held by the end effector 4 should be released. That is, the robot control signal may include a signal for controlling the robotic arm 12 to position the end effector 4 in the release position. In this case, the robot control device 13 may control the robotic arm 12 based on the robot control signal to move (i.e., approach) the end effector 4 toward the release position. In other words, the robot control signal may control the robotic arm 12 to position the end effector 4 in the release position.

[0149] As described above, when the robot control signal includes a signal for controlling the end effector 4, the robot control device 13 can also control the end effector 4 based on the robot control signal. For example, the robot control device 13 can also control the operation of the end effector 4 by controlling the operation of the actuator that moves the manual gripper constituting the end effector 4 based on the robot control signal. For example, the robot control device 13 can also control the operation of the vacuum device of the vacuum gripper constituting the end effector 4 based on the robot control signal, thereby controlling the operation of the end effector 4.

[0150] As an example, when the end effector 4 performs a holding process to hold the object OBJ, the robot control signal may include a signal for controlling the end effector 4 so that the end effector 4, located in the holding position, holds the object OBJ. In this case, the robot control device 13 can control the end effector 4 based on the robot control signal so that the end effector 4, located in the holding position, holds the object OBJ.

[0151] As another example, in the case of performing a release process for releasing the object OBJ held by the end effector 4, the robot control signal may include a signal for controlling the end effector 4 to release the object OBJ held by the end effector 4 from the release position. In this case, the robot control device 13 can control the end effector 4 based on the robot control signal to release the object OBJ held by the end effector 4 from the release position.

[0152] The robot control signal may also include signals that can be directly used to control the movement of the robot 1 by the robot control device 13. The robot control signal may also include signals that can be directly used as robot drive signals for controlling the movement of the robot 1 by the robot control device 13. In this case, the robot control device 13 may also directly use the robot control signal to control the movement of the robot 1. For example, the control device 3 may generate drive signals for actuators built into the joints 122 of the robotic arm 12 as robot control signals, and the robot control device 13 may directly use the robot control signals generated by the control device 3 to control the actuators built into the joints 122 of the robotic arm 12. The robot control signal may also include signals that can be directly used to control the movement of the end effector 4 by the robot control device 13. The robot control signal may also include signals that can be directly used as end effector drive signals for controlling the movement of the end effector 4 by the robot control device 13. In this scenario, for example, control device 3 can generate a drive signal (end-effector drive signal) for the actuator that moves the manual gripper constituting end-effector 4 as a robot control signal, and robot control device 13 can directly use the robot control signal generated by control device 3 to control the actuator of end-effector 4. For example, control device 3 can generate a drive signal (end-effector drive signal) for the vacuum device that drives the vacuum gripper constituting end-effector 4 as a robot control signal, and robot control device 13 can directly use the robot control signal generated by control device 3 to control the vacuum device of end-effector 4.

[0153] Furthermore, as described above, if the robot control signal includes a signal that can be directly used to control the action of at least one of the robot 1 and the end effector 4 by the robot control device 13, the robot 1 may not have the robot control device 13.

[0154] Alternatively, the robot control signal may also include a signal that can be used by the robot control device 13 to generate robot drive signals for controlling the movement of the robot 1. In this case, the robot control device 13 may also generate robot drive signals for controlling the movement of the robot 1 based on the robot control signal, and control the movement of the robot 1 based on the generated robot drive signals. For example, the robot control device 13 may generate robot drive signals for driving actuators built into the joints 122 of the robotic arm 12 based on the robot control signal, and control the actuators built into the joints 122 of the robotic arm 12 based on the generated robot drive signals.

[0155] The robot control signal may also include a signal that can be used by the robot control device 13 to generate an end effector drive signal for controlling the movement of the end effector 13. In this case, the robot control device 13 may also generate an end effector drive signal for controlling the movement of the end effector 4 based on the robot control signal, and control the movement of the end effector 4 based on the generated end effector drive signal. For example, if the end effector 4 is a manual gripper, the robot control device 13 may generate an end effector drive signal for driving the actuator of the manual gripper based on the robot control signal, and control the actuator of the manual gripper based on the generated end effector drive signal. For example, if the end effector 4 is a magnetic gripper, the robot control device 13 may generate an end effector drive signal for driving the magnetic device of the magnetic gripper based on the robot control signal, and control the magnetic device based on the generated end effector drive signal.

[0156] Furthermore, the signals that can be used by the robot control device 13 to generate robot drive signals may include signals representing at least one of the position and orientation of the object OBJ in the global coordinate system. In this case, for example, the robot control device 13 may generate robot drive signals based on the robot control signals to drive the actuators built into the joints 122 of the robotic arm 12, so that the end effector 4 approaches the object OBJ whose position and orientation in the global coordinate system are calculated by the robot control signals (i.e., the positional relationship between the robot 1 (end effector 4) and the object OBJ becomes the desired positional relationship), and control the movement of the robotic arm 12 based on the generated robot drive signals.

[0157] The signals that can be used by the robot control unit 13 to generate robot drive signals may include signals representing the desired positional relationship between the robot 1 and the object OBJ in the global coordinate system. In this case, for example, the robot control unit 13 may generate robot drive signals based on the robot control signals to drive the actuators built into the joints 122 of the robotic arm 12, so that the positional relationship between the robot 1 (end effector 4) and the object OBJ becomes the desired positional relationship represented by the robot control signals, and control the movement of the robotic arm 12 based on the generated robot drive signals.

[0158] The signals that can be used by the robot control unit 13 to generate robot drive signals may include signals representing the desired position of the end effector 4 in the global coordinate system. In this case, for example, the robot control unit 13 may generate robot drive signals based on the robot control signals to drive the actuators built into the joints 122 of the robotic arm 12, such that the end effector 4 is located at the desired position represented by the robot control signals (i.e., the positional relationship between the robot 1 (end effector 4) and the object OBJ becomes the desired positional relationship), and control the movement of the robotic arm 12 based on the generated robot drive signals. As an example of the desired position, a processing position where the end effector 4 should handle the object OBJ can be cited. As a specific example of the desired position, a holding position where the end effector 4 should hold the object OBJ can be cited. In this case, the robot control unit 13 may generate robot drive signals based on the robot control signals to drive the actuators built into the joints 122 of the robotic arm 12, such that the end effector 4 moves to the holding position represented by the robot control signals, and control the movement of the robotic arm 12 based on the generated robot drive signals. As another example of the desired position, the release position where the end effector 4 should release the object OBJ can be cited. In this case, the robot control unit 13 can generate robot drive signals based on robot control signals to drive the actuators built into the joints 122 of the robotic arm 12, so that the end effector 4 is in the release position represented by the robot control signals, and control the movement of the robotic arm 12 based on the generated robot drive signals.

[0159] Furthermore, the signals that can be used by the robot control device 13 to generate robot drive signals may include, for example, signals indicating the desired position of the tip of the robotic arm 12 (e.g., the tool center point) in the global coordinate system, and signals indicating the desired position of the imaging system 2 in the global coordinate system. Additionally, the signals that can be used by the robot control device 13 to generate robot drive signals may be signals indicating the amount and direction of movement from the current position of the end effector 4 to the desired position of the end effector 4.

[0160] In addition, the coordinate system used as a reference in robot control signals can also be a coordinate system other than the global coordinate system (e.g., robot coordinate system, 2D shooting coordinate system or 3D shooting coordinate system).

[0161] (1-2) Structure of control device 3

[0162] Next, refer to Figure 3 Explain the structure of control device 3. Figure 3 This is a block diagram showing the structure of the control device 3.

[0163] like Figure 3As shown, the control device 3 includes a processing unit 31, a storage unit 32, and a communication unit 33. Furthermore, the control device 3 may include an input unit 34 and an output unit 35. Alternatively, the control device 3 may not include at least one of the input unit 34 and the output unit 35. The processing unit 31, storage unit 32, communication unit 33, input unit 34, and output unit 35 can be connected via a data bus 36.

[0164] The computing device 31 includes at least one of, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a FPGA (Field Programmable Gate Array). The computing device 31 reads a computer program. For example, the computing device 31 may also read a computer program stored in the storage device 32. For example, the computing device 31 may also use a recording medium reading device (not shown) provided with the control device 3 to read a computer program stored on a computer-readable and non-temporary recording medium. The computing device 31 can acquire (i.e., download or read) a computer program from a device (not shown) configured outside the control device 3 via a communication device 33 (or other communication device). The computing device 31 executes the read computer program. As a result, a logical function module for performing the processing to be performed by the control device 3 (as an example, the robot control processing described above) is implemented within the computing device 31. That is, the computing device 31 can function as a controller for implementing the logical function module for performing the processing to be performed by the control device 3.

[0165] Within the computing device 31, a computational model capable of being constructed through machine learning can be installed by executing a computer program. As an example of a computational model capable of being constructed through machine learning, a computational model including a neural network (so-called Artificial Intelligence (AI)) can be listed. In this case, learning the computational model may include learning the parameters of the neural network (e.g., at least one of weights and biases). The computing device 31 can use the computational model to perform robot control processing. That is, actions that perform robot control processing may include actions performed using the computational model. Furthermore, the computing device 31 can be equipped with a computational model constructed using offline machine learning with teacher data. Additionally, the computational model installed on the computing device 31 can be updated on the computing device 31 via online machine learning. Alternatively, based on or instead of the computational model installed on the computing device 31, the computing device 31 may also use a computational model installed on a device external to the computing device 31 (i.e., a device located external to the control device 3) to perform robot control processing.

[0166] Furthermore, the recording medium for the computer program executed by the arithmetic device 31 may be at least one of the following: CD-ROM, CD-R, CD-RW or floppy disk, MO, DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW and Blu-ray (registered trademark), magnetic media such as magnetic tape, optical disk, semiconductor memory such as USB memory, and any medium capable of storing other programs. The recording medium may include a device capable of recording a computer program (as an example, a general-purpose or special-purpose device that installs the computer program in a state capable of executing in at least one manner, such as software and firmware). In addition, the various processes or functions included in the computer program may be implemented by a logic processing module implemented within the arithmetic device 31 by executing the computer program, or by hardware such as a specified gate array (FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit)) provided by the arithmetic device 31, or by a combination of a logic processing module and a partial hardware module that implements a portion of the hardware elements.

[0167] Figure 3 An example of a logic function module implemented within the computing device 31 to perform robot control processing is shown. Figure 3As shown, a three-dimensional position data generation unit 311, a position and attitude calculation unit 312, and a signal generation unit 313 are implemented within the computing device 31. Furthermore, the processing performed by each of the three-dimensional position data generation unit 311, the position and attitude calculation unit 312, and the signal generation unit 313 will be described later. Figure 4 Detailed explanations will be provided later, so they are omitted here. Additionally, the arithmetic unit 31 can also be referred to as the arithmetic section.

[0168] Storage device 32 is capable of storing desired data. For example, storage device 32 can also temporarily store computer programs executed by computing device 31. Storage device 32 can also temporarily store data temporarily used by computing device 31 while computing device 31 is executing computer programs. Storage device 32 can also store data that control device 3 will store permanently. In addition, storage device 32 may include at least one of RAM (Random Access Memory), ROM (Read Only Memory), hard disk drive, magneto-optical disk drive, SSD (Solid State Drive), and disk array drive. That is, storage device 32 may include non-temporary recording media.

[0169] The communication device 33 can communicate with both the robot 1 and the imaging system 2 via a communication network (not shown). Alternatively, the communication device 33 can also communicate with other devices different from the robot 1 and the imaging system 2 via a communication network (not shown), in addition to or replacing at least one of the robot 1 and the imaging system 2. In this embodiment, the communication device 33 can also receive (i.e., acquire) at least one of image data IMG_2D and IMG_3D from the imaging system 2. Furthermore, the communication device 33 can also send (i.e., output) robot control signals to the robot 1. Moreover, the communication device 33 that outputs robot control signals to the robot 1 can be referred to as an output unit.

[0170] Input device 34 is a device that accepts input of information for control device 3 from outside the control device 3. For example, input device 34 may include user-operable operating devices of control device 3 (e.g., at least one of keyboard, mouse, and touch panel). For example, input device 34 may include a recording medium reading device capable of reading information recorded as data on a recording medium externally connected to control device 3.

[0171] Furthermore, information can be input as data from an external device to the control device 3 via the communication device 33. In this case, the communication device 33 can also function as an input device that receives information from an external source for the control device 3.

[0172] Output device 35 is a device for externally outputting information to control device 3. For example, output device 35 can output information as an image. That is, output device 35 can include a display device (so-called a monitor) capable of displaying images. For example, output device 35 can output information as sound. That is, output device 35 can include a sound device (so-called a speaker) capable of outputting sound. For example, output device 35 can output information onto paper. That is, output device 35 can include a printing device (so-called a printer) capable of printing desired information onto paper. For example, output device 35 can output information as data to a recording medium that can be externally connected to control device 3.

[0173] Furthermore, information can be output as data from the control device 3 to an external device of the control device 3 via the communication device 33. In this case, the communication device 33 can function as an output device that outputs information to the external device of the control device 3.

[0174] (2) Robot control processing

[0175] Next, the robot control processing performed by control device 3 will be explained.

[0176] (2-1) Robot control and processing flow

[0177] First, refer to Figure 4 This will illustrate the robot control and processing flow. Figure 4 This is a flowchart illustrating the robot control process.

[0178] like Figure 4 As shown, the control device 3 acquires image data IMG_3D from the imaging device 22 using the communication device 33 (step S1). Specifically, the imaging device 22 captures the object OBJ at a predetermined 3D imaging rate. For example, the imaging device 22 can capture the object OBJ at a 3D imaging rate that means capturing the object OBJ tens to hundreds of times per second (for example, 500 times). As a result, the imaging device 22 generates image data IMG_3D at a period corresponding to the predetermined 3D imaging rate. For example, the imaging device 22 can generate tens to hundreds of (for example, 500) image data IMG_3D per second. Whenever the imaging device 22 generates image data IMG_3D, the control device 3 acquires the image data IMG_3D. That is, the control device 3 can acquire tens to hundreds of (for example, 500) image data IMG_3D per second.

[0179] However, the imaging device 22 may also periodically photograph the object OBJ without adhering to the prescribed 3D imaging rate. For example, when the imaging device 22 receives a control signal from the control device 3 to control the imaging device 22 to photograph the object OBJ, the imaging device 22 may photograph the object OBJ.

[0180] In addition to the object OBJ, which is processed by robot 1, the imaging device 22 can also capture images of other objects different from the object OBJ. Furthermore, these other objects are not objects processed by robot 1; therefore, in the following description, these other objects different from the object OBJ will be referred to as non-object objects. Examples of non-object objects include at least a portion of the end effector 4, at least a portion of the robotic arm 12, and at least one object located around robot 1, i.e., a peripheral object. Examples of peripheral objects include at least one of the loading device T and the container CB. For example, when both the object OBJ and the non-object object are included within the imaging range (field of view) of the imaging device 22, the imaging device 22 can capture images of both the object OBJ and the non-object object. As a result, the imaging device 22 can generate image data IMG_3D representing images of both the object OBJ and the non-object object. However, the imaging device 22 can capture images of the object OBJ but not of the non-object object. That is, the imaging device 22 can generate image data IMG_3D, whereby image data IMG_3D represents an image that captures the object OBJ but not non-object objects. In either case, the imaging device 22 generates at least image data IMG_3D representing an image that captures the object OBJ. That is, the imaging device 22 generates image data IMG_3D that at least contains image data of the object OBJ.

[0181] As described above, the imaging device 22 sometimes images multiple object OBJs that are sequentially processed by the end effector 4. In this case, the end effector 4 may perform prescribed processing on the first object OBJ among the multiple object OBJs in a first period, and perform prescribed processing on the second object OBJ among the multiple object OBJs that is different from the first object OBJ in a second period after the first period. In this case, the object OBJs among the multiple object OBJs in the first period other than the first object OBJ processed by the end effector 4 can be regarded as object OBJs (or non-object objects) that are not processed by the robot 1 in the first period. Similarly, the object OBJs among the multiple object OBJs in the second period other than the second object OBJ processed by the end effector 4 can be regarded as object OBJs (or non-object objects) that are not processed by the robot 1 in the second period. Furthermore, the first object OBJ and the second object OBJ can each be different object OBJs among the multiple object OBJs.

[0182] In addition to acquiring image data IMG_3D from the imaging device 22, or as an alternative, the control device 3 can also acquire image data IMG_2D from the imaging device 21 using the communication device 33 (step S1). Specifically, the imaging device 21 captures the object OBJ at a predetermined 2D shooting rate. The 2D shooting rate is the same as the 3D shooting rate. However, the 2D shooting rate can also be different from the 3D shooting rate. For example, the imaging device 21 can capture the object OBJ at a 2D shooting rate, which means capturing the object OBJ dozens to hundreds of times per second (for example, 500 times). As a result, the imaging device 21 generates image data IMG_2D at a period corresponding to the predetermined 2D shooting rate. For example, the imaging device 21 can generate dozens to hundreds (for example, 500) of image data IMG_2D per second. Whenever the imaging device 21 generates image data IMG_2D, the control device 3 acquires the image data IMG_2D. That is, the control device 3 can acquire tens to hundreds (500 as an example) of image data IMG_2D per second.

[0183] However, the imaging device 21 may also periodically photograph the object OBJ without using the prescribed 2D imaging rate. For example, when the imaging device 21 receives a control signal from the control device 3 to control the imaging device 21 to photograph the object OBJ, the imaging device 21 may photograph the object OBJ.

[0184] Additionally, image data IMG_2D can also be referred to as two-dimensional position data representing the two-dimensional position of the object OBJ captured in the image data IMG_2D. Image data IMG_2D (two-dimensional position data) can be data representing the two-dimensional position of at least a portion of the surface of the object OBJ. Specifically, image data IMG_2D (two-dimensional position data) is data representing the individual two-dimensional positions of multiple points on the object OBJ. For example, image data IMG_2D (two-dimensional position data) can be data representing the individual two-dimensional positions of multiple points on the surface of the object OBJ. For example, image data IMG_2D (two-dimensional position data) can be data representing the individual two-dimensional positions of multiple points respectively corresponding to multiple locations on the surface of the object OBJ.

[0185] In addition to the object OBJ, which is processed by robot 1, the imaging device 21 can also capture images of non-object objects different from the object OBJ. For example, if both the object OBJ and the non-object objects are included within the imaging range (field of view) of the imaging device 21, the imaging device 21 can capture images of both the object OBJ and the non-object objects. As a result, the imaging device 21 can generate image data IMG_2D representing images of both the object OBJ and the non-object objects. However, the imaging device 21 can also capture images of the object OBJ without capturing images of non-object objects. That is, the imaging device 21 can generate image data IMG_2D representing images of the object OBJ without capturing images of non-object objects. In either case, the imaging device 21 generates at least image data IMG_2D representing images of the object OBJ. That is, the imaging device 21 generates image data IMG_2D that at least includes image data of the object OBJ.

[0186] As described above, the imaging device 21 sometimes images multiple object OBJs that are sequentially processed by the end effector. In this case, the end effector 4 may perform specified processing on the third object OBJ among the multiple object OBJs in the third period, and perform specified processing on the fourth object OBJ among the multiple object OBJs that is different from the third object OBJ in the fourth period after the third period. In this case, the object OBJs among the multiple object OBJs in the third period other than the third object OBJ processed by the end effector can be regarded as object OBJs (or non-object objects) that are not processed by the robot in the third period. Similarly, the object OBJs among the multiple object OBJs in the fourth period other than the fourth object OBJ processed by the end effector 4 can be regarded as object OBJs (or non-object objects) that are not processed by the robot 1 in the fourth period. Furthermore, the third object OBJ and the fourth object OBJ can each be different object OBJs among the multiple object OBJs.

[0187] When the control device 3 acquires image data IMG_3D, the 3D position data generation unit 311 generates 3D position data WSD based on the acquired image data IMG_3D whenever the control device 3 acquires image data IMG_3D (step S2). Furthermore, the 3D position data generation unit 311 outputs the generated 3D position data WSD to the position and attitude calculation unit 312. However, if the control device 3 does not acquire image data IMG_3D (for example, if the control device 3 acquires image data IMG_2D), the 3D position data generation unit 311 may not generate 3D position data WSD. In this case, the control device 3 may not perform step S2.

[0188] Furthermore, the control device 3 can generate three-dimensional position data WSD based on each of the multiple image data IMG_3D generated by the imaging device 22. Alternatively, the control device 3 can generate three-dimensional position data WSD without using another portion of the multiple image data IMG_3D generated by the imaging device 22.

[0189] The three-dimensional position data (WSD) represents the three-dimensional position of the object OBJ captured in the image data IMG_3D. For example, the three-dimensional position data WSD may represent the three-dimensional position of at least a portion of the surface of the object OBJ. In particular, the three-dimensional position data WSD represents the three-dimensional position of each of multiple points of the object OBJ. That is, the three-dimensional position data WSD represents the three-dimensional position of each of multiple points of the object OBJ captured by the imaging device 22. For example, the three-dimensional position data WSD may represent the three-dimensional position of each of multiple points on the surface of the object OBJ. For example, the three-dimensional position data WSD may also represent the three-dimensional position of multiple points corresponding to multiple parts of the surface of the object OBJ. Furthermore, in the following description, unless otherwise stated, the three-dimensional position of the object OBJ may represent at least one of the following: the three-dimensional position of at least a portion of the surface of the object OBJ, the three-dimensional position of each of multiple points of the object OBJ, the three-dimensional position of each of multiple points on the surface of the object OBJ, and the three-dimensional position of each of multiple points corresponding to multiple parts of the surface of the object OBJ.

[0190] Specifically, as described above, an object OBJ with a projected pattern is captured in the image shown in image data IMG_3D. In this case, the projected pattern in the image shown in image data IMG_3D reflects the three-dimensional shape of at least a portion of the surface of the object OBJ with the projected pattern. The shape of the projected pattern in the image shown in image data IMG_3D reflects the three-dimensional shape of at least a portion of the surface of the object OBJ with the projected pattern. Therefore, the three-dimensional position generation unit 311 can calculate the three-dimensional shape of at least a portion of the surface of the object OBJ based on the projected pattern in the image shown in image data IMG_3D. The three-dimensional shape of at least a portion of the surface of the object OBJ substantially represents the three-dimensional position of each of a plurality of points of the object OBJ. This is because each of the plurality of points of the object OBJ is contained within the surface of the object OBJ. Therefore, the process of calculating the three-dimensional shape of at least a portion of the surface of the object OBJ can be considered substantially equivalent to the process of calculating the three-dimensional position of each of the plurality of points of the object OBJ. Therefore, the three-dimensional position data generation unit 311 can generate three-dimensional position data WSD based on image data IMG_3D.

[0191] Given that each of the multiple points of an object OBJ is contained on the surface of the object OBJ, the three-dimensional position data WSD representing the three-dimensional positions of the multiple points of the object OBJ can be regarded as equivalent to the three-dimensional shape data representing at least a portion of the three-dimensional shape of the object OBJ (in particular, the three-dimensional shape of at least a portion of the surface of the object OBJ).

[0192] To generate three-dimensional position data WSD, the three-dimensional position data generation unit 311 can calculate disparity by associating the portions (e.g., pixels) of the images shown by the two image data contained in the image data IMG_3D with each other. Specifically, in this association, the three-dimensional position data generation unit 311 can calculate disparity by associating the portions of the projection patterns captured by the images shown by the two image data (i.e., the portions of the projection patterns captured in each image with each other). The three-dimensional position data generation unit 311 can calculate the three-dimensional positions of multiple points of the object OBJ using a known method based on the triangulation principle using the calculated disparity. As a result, three-dimensional position data WSD representing the three-dimensional positions of multiple points of the object OBJ is generated. In this case, the accuracy of disparity calculation is higher when the portions of the images with projection patterns captured are correlated (i.e., the portions of the captured projection patterns) compared to correlating the portions of the images without projection patterns captured with each other. Therefore, the accuracy of the generated 3D position data WSD (i.e., the accuracy of the calculation of the 3D position of each point of the object OBJ) becomes higher.

[0193] The 3D position data (WSD) can be any data as long as it can represent the individual 3D positions of multiple points on the object OBJ. That is, the 3D position data (WSD) can be any data that directly or indirectly shows the individual 3D positions of multiple points on the object OBJ. For example, the 3D position data (WSD) can contain coordinate information representing the individual 3D positions of multiple points on the object OBJ. For example, the 3D position data (WSD) can contain information representing the 3D shape of at least a portion of the object OBJ.

[0194] As an example of 3D position data (WSD), depth image data can be used. Depth image data is image data obtained by associating brightness and depth information with each pixel of the depth image represented by the depth image data, or by associating depth information with each pixel of the depth image represented by the depth image data instead of brightness information. Depth information represents the distance (i.e., depth) between each part of the object OBJ captured in each pixel and the imaging device 22. The distance (i.e., depth) between each part of the object OBJ captured in each pixel and the imaging device 22 can be calculated based on the parallax mentioned above. Alternatively, depth image data can also be image data where the brightness information of each pixel represents the depth of each part of the object OBJ (i.e., the distance between each part of the object OBJ and the imaging device 22). The three-dimensional position data generation unit 311 can also calculate the distance between each part of the object OBJ in the image shown in the image data IMG_3D and the shooting device 22 based on the projection pattern in the image shown in the image data IMG_3D, and generate a depth image by associating the calculated distance as depth information with each pixel of the image shown in the image data IMG_3D.

[0195] As another example of 3D position data WSD, point cloud data can be used. Point cloud data is data representing a set of points in three-dimensional space corresponding to each part of an object OBJ captured in the image shown in image data IMG_3D. The 3D position data generation unit 311 can generate point cloud data based on depth image data and camera parameters of the imaging device 22. Furthermore, the following description will illustrate an example of using point cloud data as 3D position data WSD.

[0196] As described above, when the shooting range (field of view) of the shooting device 22 includes not only the object OBJ processed by the robot 1, but also non-object objects different from the object OBJ, the image data IMG_3D represents images captured from both the object OBJ and the non-object objects. In this case, in addition to the three-dimensional position of the object OBJ captured in the image data IMG_3D, the three-dimensional position data WSD can also be data representing the three-dimensional position of the non-object objects captured in the image data IMG_3D. That is, the three-dimensional position data WSD can include data representing the three-dimensional positions of multiple points of the non-object objects captured by the shooting device 22. Even in this case, as long as the three-dimensional position data WSD contains data representing the three-dimensional position of the object OBJ, it can be considered that the three-dimensional position data WSD represents the three-dimensional positions of multiple points of the object OBJ. This is because even if the data representing the three-dimensional position of the non-object objects is included in the three-dimensional position data WSD, the data representing the three-dimensional position of the object OBJ is still included in the three-dimensional position data WSD.

[0197] Then, the position and pose calculation unit 312 calculates at least one of the position and pose of the object OBJ based on at least one of the image data IMG_2D acquired in step S1 and the three-dimensional position data WSD generated in step S2 (step S3). As a result, the position and pose calculation unit 312 generates position and pose data POI representing at least one of the position and pose of the object OBJ (step S3). The position and pose data POI may contain position data representing the position of the object OBJ. In addition to or as a substitute for position data representing the position of the object OBJ, the position and pose data POI may also contain pose data representing the pose of the object OBJ. That is, the position and pose data POI may contain at least one of position data and pose data.

[0198] In step S3, the position and attitude calculation unit 312 calculates at least one of the position and attitude of the object OBJ in the global coordinate system. That is, the position and attitude calculation unit 312 generates position and attitude data POI representing at least one of the position and attitude of the object OBJ in the global coordinate system. For example, the robot control device 13 can control the robotic arm 12 so that the end effector 4 is located at a desired position in the global coordinate system. The global coordinate system is defined by the mutually orthogonal X-axis (GL), Y-axis (GL), and Z-axis (GL). The X-axis (GL) can be an axis along the horizontal plane. The Y-axis (GL) can be an axis along the horizontal plane. The Z-axis (GL) can be an axis orthogonal to the horizontal plane. The Z-axis (GL) can also be an axis extending along the direction of gravity. Furthermore, Figure 2The X-axis, Y-axis, and Z-axis shown can be the X-axis (GL), Y-axis (GL), and Z-axis (GL), respectively. Furthermore, the origin of the global coordinate system does not have to be... Figure 2 The origins of the X-axis (GL), Y-axis (GL), and Z-axis (GL) are shown. For example, the origin of the global coordinate system can be set at... Figure 2 The origin of the global coordinate system can be set at any position on the support surface S. For example, the origin of the global coordinate system can be set at any position of the contact surface of the base 11 relative to the support surface S (e.g., the center or centroid of the contact surface).

[0199] The position and attitude calculation unit 312 can calculate at least one of the following: the position Tx(GL) of the object OBJ in the X-axis direction (GL) parallel to the X-axis (GL); the position Ty(GL) of the object OBJ in the Y-axis direction (GL) parallel to the Y-axis (GL); and the position Tz(GL) of the object OBJ in the Z-axis direction (GL) parallel to the Z-axis (GL), to determine the position of the object OBJ in the global coordinate system. The position and attitude calculation unit 312 can also calculate at least one of the following: the rotation amount Rx(GL) of the object OBJ around the X-axis (GL); the rotation amount Rey(GL) of the object OBJ around the Y-axis (GL); and the rotation amount Rz(GL) of the object OBJ around the Z-axis (GL), to determine the attitude of the object OBJ in the global coordinate system. This is because the rotations of object OBJ around the X-axis (GL) Rx(GL), around the Y-axis (GL) Ry(GL), and around the Z-axis (GL) Rz(GL) are equivalent to the parameters representing the orientation of object OBJ around the X-axis (GL), around the Y-axis (GL), and around the Z-axis (GL), respectively. Therefore, in the following description, the rotations of object OBJ around the X-axis (GL) Rx(GL), around the Y-axis (GL) Ry(GL), and around the Z-axis (GL) Rz(GL) are referred to as the orientations of object OBJ around the X-axis (GL) Rx(GL), around the Y-axis (GL) Ry(GL), and around the Z-axis (GL) Rz(GL), respectively.

[0200] Alternatively, the poses Rx(GL) of object OBJ around the X-axis (GL), Ry(GL) of object OBJ around the Y-axis (GL), and Rz(GL) of object OBJ around the Z-axis (GL) can be viewed as representing the positions of object OBJ in the rotational directions around the X-axis (GL), Y-axis (GL), and Z-axis (GL), respectively. In other words, the poses Rx(GL) of object OBJ around the X-axis (GL), Ry(GL) of object OBJ around the Y-axis (GL), and Rz(GL) of object OBJ around the Z-axis (GL) can all be considered as parameters representing the position of object OBJ.

[0201] Therefore, in Figure 4 In step S3, the position and attitude calculation unit 312 can calculate at least one of position Tx (GL), position Ty (GL), position Tz (GL), attitude Rx (GL), attitude Ry (GL), and attitude Rz (GL) as at least one of the position and attitude of the object OBJ in the global coordinate system.

[0202] Figure 4 In step S3, the position and pose calculation unit 312 can calculate at least one of the position and pose of the object OBJ by performing a matching process using at least one of the image data IMG_2D and the three-dimensional position data WSD. For ease of explanation, the following example will be used primarily: the position and pose calculation unit 312 calculates at least one of the position and pose of the object OBJ by performing a matching process using the three-dimensional position data WSD. Furthermore, in the following explanation, the matching process using the three-dimensional position data WSD will be referred to as 3D matching process.

[0203] Specifically, the position and attitude calculation unit 312 can perform 3D matching processing, which uses 3D position data WSD and 3D model data WMD representing a 3D model that serves as the reference for the object OBJ. In this case, the 3D matching processing can be viewed, for example, as a matching process using a point cloud represented by the 3D position data WSD and a 3D model represented by the 3D model data. Furthermore, the 3D matching processing itself can be the same as existing matching processes. For example, the position and attitude calculation unit 312 can also use a known method including at least one of RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), ICP (Iterative Closest Point), and DSO (Direct Sparse Odometry) to perform 3D matching processing.

[0204] A 3D model based on WSD (Web Surface Deposition Data) can include a point cloud model, which is a 3D model that uses the point cloud shown in the WSD to represent the 3D shape of an object OBJ. A 3D model based on WSD can include a polygonal model or a mesh model generated by combining multiple points constituting the point cloud shown in the WSD on faces (especially faces of polygons with each point as a vertex). A 3D model based on WSD can also include a solid model generated by combining multiple points constituting the point cloud shown in the WSD on a smooth surface.

[0205] The 3D model data WMD is data representing a template model TM3 that serves as a 3D model of the object OBJ. Specifically, the 3D model data WMD is data representing the template model TM3, which is a 3D model with a 3D shape that serves as a reference for the object OBJ. The template model can be a CAD model of the object OBJ. The template model can be a 3D model with the same shape as the 3D shape of the object OBJ obtained by pre-measuring the actual 3D shape of the object OBJ. The template model can be a polygonal model, a mesh model, or a solid model. The 3D model data WMD can be point cloud data representing the template model TM3 of the object OBJ. The 3D model data WMD can be polygonal model data, mesh model data, or solid model data generated from point cloud data. Furthermore, the actual object OBJ measured beforehand to generate the 3D model data WMD can be a reference or a prototype object OBJ.

[0206] In this case, such as Figure 5As shown, the position and attitude calculation unit 312 can perform 3D matching processing (in other words, template matching processing) on ​​the three-dimensional position data WSD. This 3D matching processing uses the template model TM3 shown in the three-dimensional model data WMD as a template. Specifically, the position and attitude calculation unit 312 can move, enlarge, shrink, and / or rotate the template model TM3 in parallel within the 3D shooting coordinate system so that the feature parts (e.g., at least one feature point and edge) of the template model TM3 shown in the three-dimensional model data WMD approach (e.g., match) the feature parts of the object OBJ (e.g., the point cloud corresponding to the object OBJ shown in the three-dimensional position data WSD, or the three-dimensional model (e.g., the point cloud model) reflecting the actual three-dimensional position of the object OBJ) represented by the three-dimensional position data WSD. That is, the position and attitude calculation unit 312 can change the positional relationship between the coordinate system of the 3D model data WMD (e.g., the coordinate system of the CAD model) and the 3D shooting coordinate system, so that the feature parts of the object OBJ represented by the 3D position data WSD (e.g., the point cloud corresponding to the object OBJ shown in the 3D position data WSD, or the 3D model reflecting the actual 3D position of the object OBJ (e.g., the point cloud model)) approach (e.g., match) the feature parts of the template model TM3. As a result, the position and attitude calculation unit 312 can determine the positional relationship between the coordinate system of the 3D model data WMD and the 3D shooting coordinate system. Then, based on the positional relationship between the coordinate system of the 3D model data WMD and the 3D shooting coordinate system, the position and attitude calculation unit 312 can calculate at least one of the position and attitude of the object OBJ in the 3D shooting coordinate system according to at least one of the position and attitude of the template model TM3 in the coordinate system of the 3D model data WMD.

[0207] Then, the position and pose calculation unit 312, based on a transformation matrix used to convert the three-dimensional coordinates in either the robot coordinate system or the 3D imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system and the 3D imaging coordinate system, transforms at least one of the position and pose of the object OBJ in the 3D imaging coordinate system into at least one of the position and pose of the object OBJ in the robot coordinate system. When the robot coordinate system is used as the global coordinate system, at least one of the position and pose of the object OBJ in the robot coordinate system is equivalent to at least one of the position and pose of the object OBJ in the global coordinate system. However, when a coordinate system different from the robot coordinate system is used as the global coordinate system, the position and pose calculation unit 312 can use a transformation matrix used to convert the three-dimensional coordinates in either the robot coordinate system or the global coordinate system into three-dimensional coordinates in the other of the robot coordinate system and the global coordinate system, and calculate at least one of the position and pose of the object OBJ in the global coordinate system based on at least one of the position and pose of the object OBJ in the robot coordinate system.

[0208] When a feature part of the 3D model shown in the 3D model data WMD is brought close to a feature part of the object OBJ whose 3D position is represented by the 3D position data WSD, the position and pose calculation unit 312 can calculate a matching similarity. This matching similarity is the similarity between the 3D model of the object OBJ shown in the 3D model data WMD and the object OBJ whose 3D position is represented by the 3D position data WSD (e.g., a point cloud of the object OBJ). Furthermore, in the following description, the similarity between the 3D model of the object OBJ shown in the 3D model data WMD and the object OBJ whose 3D position is represented by the 3D position data WSD (e.g., a point cloud of the object OBJ), i.e., the matching similarity, can be simply referred to as the matching similarity of the object OBJ. The position and pose calculation unit 312 can move, enlarge, reduce, and / or rotate the 3D model shown in the 3D model data WMD to maximize the matching similarity. Furthermore, the matching similarity can also be considered equivalent to the correlation degree representing the relationship between the 3D model shown in the 3D model data WMD and the object OBJ whose 3D position is represented by the 3D position data WSD. Furthermore, relevance can also be described as an indicator of the correlation between the 3D model represented by the 3D model data WMD and the object OBJ represented by the 3D position data WSD. Additionally, matching similarity can be referred to as the matching score.

[0209] If an object OBJ is detected by 3D matching processing and its calculated matching similarity exceeds a predetermined matching threshold, the position and attitude calculation unit 312 may select this object OBJ as an object OBJ for which the end effector 4 should perform the prescribed processing (i.e., the processing execution object). On the other hand, if an object OBJ is detected by 3D matching processing and its calculated matching similarity is lower than the predetermined matching threshold, the position and attitude calculation unit 312 may not select this object OBJ as an object OBJ for which the end effector 4 should perform the prescribed processing (i.e., the processing execution object).

[0210] Based on the three-dimensional position data WSD, the three-dimensional position data WSD may represent the positions of multiple object objects OBJ. For example, when multiple workpieces W are randomly or neatly placed on the mounting device T, the imaging device 22 may capture images of the multiple workpieces W and generate image data IMG_3D that captures the multiple workpieces W as multiple object objects OBJ. For example, when multiple workpieces W are randomly or neatly stored in a container (storage box) CB, which is an example of the mounting device T, the imaging device 22 may capture images of the multiple workpieces W and generate image data IMG_3D that captures the multiple workpieces W as multiple object objects OBJ. As a result, the position and attitude calculation unit 312 may generate three-dimensional position data WSD representing the positions of the multiple object objects OBJ. In this case, the position and attitude calculation unit 312 can perform the above-mentioned 3D matching processing on each of the multiple object objects OBJ. For example, the position and attitude calculation unit 312 can perform 3D matching processing sequentially on the multiple object objects OBJ. Alternatively, for example, the position and attitude calculation unit 312 can simultaneously perform multiple 3D matching processes, with multiple object objects OBJ as objects. Then, the position and attitude calculation unit 312 can select the object object OBJ among the multiple object objects whose matching similarity exceeds the matching determination threshold and has the largest matching similarity, as the processing execution object for the end effector 4 to perform the prescribed processing. Alternatively, the position and attitude calculation unit 312 can select the object object OBJ among the multiple object objects whose matching similarity exceeds the matching determination threshold and has the largest matching similarity (where N is a constant representing an integer greater than 2) as the processing execution object for the end effector 4 to perform the prescribed processing. Alternatively, the position and attitude calculation unit 312 can select the object object OBJ among the multiple object objects whose matching similarity exceeds the predetermined matching determination threshold as the processing execution object. Alternatively, the position and attitude calculation unit 312 can select the object object OBJ among the multiple object objects whose matching similarity exceeds the matching determination threshold and is closest to the end effector 4 as the processing execution object. Alternatively, for example, when multiple object objects OBJ are stored in the container CB (loading device T) (e.g., when multiple objects are stacked separately), the position and orientation calculation unit 312 can select the object object OBJ among the multiple object objects OBJ that corresponds to a matching similarity exceeding the matching determination threshold and has the largest Z-coordinate along the Z-axis (located at the highest position) as the processing execution object. Furthermore, the Z-coordinates of each of the multiple object objects OBJ can also be the Z-positions along the Z-axis of the robot 1's global coordinate system, calculated through the aforementioned 3D matching process.Alternatively, the position and attitude calculation unit 312 may select one of the multiple object objects OBJ that corresponds to a matching similarity exceeding the matching determination threshold and that the end effector 4 can perform specified processing on, as the processing execution object.

[0211] However, when the end effector 4 can perform specified processing on two or more object objects OBJ simultaneously, the position and attitude calculation unit 312 can select at least two of the multiple object objects OBJ as processing execution objects. That is, the position and attitude calculation unit 312 can select at least two processing execution objects that the end effector 4 should perform specified processing on simultaneously. Furthermore, in the following description, for ease of explanation, an example will be given of the position and attitude calculation unit 312 selecting one object object OBJ from the multiple object objects OBJ as the processing execution object.

[0212] Alternatively, the position and attitude calculation unit 312 can calculate the matching similarity of multiple object OBJs, and then, based on the matching similarity of the multiple object OBJs, select one object OBJ for which the end effector 4 should perform the prescribed processing, and then calculate at least one of the position and attitude of the selected object OBJ. In this case, the position and attitude calculation unit 312 may not calculate the position and attitude of other object OBJs that were not selected as the object OBJ for which the end effector 4 should perform the prescribed processing. Or, as described above, since at least one of the position and attitude of the object OBJs is calculated as the result of the 3D matching processing, it can also be considered that the position and attitude calculation unit 312 calculates the matching similarity together with at least one of the position and attitude of the object OBJs. In this case, the position and attitude calculation unit 312 can calculate the matching similarity of multiple object OBJs together with at least one of the position and attitude of each of the multiple object OBJs, and then select one object OBJ for which the end effector 4 should perform the prescribed processing based on the matching similarity of the multiple object OBJs. Alternatively, the position and attitude calculation unit 312 can calculate at least one of the positions and attitudes of multiple object objects OBJ, then calculate the matching similarity of the multiple object objects OBJ, and then select an object object OBJ that the end effector 4 should perform specified processing based on the calculated matching similarity.

[0213] However, in Figure 4 In step S3, in addition to performing 3D matching processing using three-dimensional position data WSD, or as an alternative, the position and pose calculation unit 312 can also calculate at least one of the position and pose of the object OBJ by performing matching processing using image data IMG_2D. Furthermore, in the following description, the matching processing using image data IMG_2D will be referred to as 2D matching processing.

[0214] Specifically, the position and pose calculation unit 312 can perform 2D matching processing, which uses image data IMG_2D (two-dimensional position data) and two-dimensional model data representing a two-dimensional model that serves as a reference for the object OBJ. Specifically, as 2D matching processing, the position and pose calculation unit 312 can use the two-dimensional model shown in the two-dimensional model data as a template image within the image shown in the image data IMG_2D, thereby performing object detection processing to detect the object OBJ shown in the template image. In other words, as matching processing, the position and pose calculation unit 312 can also perform object detection processing, which detects the object OBJ within the image shown in the image data IMG_2D by detecting similar image portions within the image shown in the image data IMG_2D that are similar to the template image. Furthermore, the 2D matching processing (in this case, object detection processing) itself can be the same as existing matching processing. For example, the position and pose calculation unit 312 can use known methods such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed-Upped Robust Feature) to perform 2D matching processing.

[0215] Two-dimensional model data can be data based on the three-dimensional model data of the object OBJ. Here, the two-dimensional model data can be generated from the CAD (Computer-Aided Design) model data of the object OBJ, or from model data (representing a polygonal model or mesh model) of the object OBJ generated based on prior measurements using a known three-dimensional shape measuring device or imaging device 22. For example, the two-dimensional model data can be two-dimensional image data representing at least a portion of the two-dimensional model of the object OBJ generated by virtually projecting at least a portion of the three-dimensional model shown in the three-dimensional model data of the object OBJ onto a virtual plane. For example, the two-dimensional model data can be two-dimensional image data representing at least a portion of the edge of the two-dimensional model of the object OBJ generated by virtually projecting at least a portion of the three-dimensional model shown in the three-dimensional model data of the object OBJ onto a virtual plane. Alternatively, the two-dimensional model data may not be data based on the three-dimensional model data of the object OBJ. For example, the two-dimensional model data can be image data IMG_2D generated by prior imaging of the object OBJ using imaging device 21. For example, the two-dimensional model data can be feature regions on an image shown in image data IMG_2D generated by pre-capturing an object OBJ using the capturing device 21. For example, the feature regions on the image shown in IMG_2D can be at least one of feature points and edges of the object OBJ captured in the image shown in IMG_2D. Furthermore, if the feature regions on the image shown in IMG_2D include the edges of the object OBJ, the two-dimensional model can also be considered as a model (edge ​​model) of at least a portion of the edges of the object OBJ. The feature regions on the image shown in IMG_2D can be at least one of at least a plurality of feature points of the object OBJ captured in the image shown in IMG_2D, and at least a portion of the edges of the object OBJ. Additionally, the feature regions on the image shown in IMG_2D can also be detected using known image processing techniques.

[0216] The position and pose calculation unit 312 can move, enlarge, reduce, and / or rotate the two-dimensional model of the object OBJ captured in the template image in parallel, so that the feature parts (e.g., at least one feature point and edge) of the two-dimensional model of the object OBJ captured in the template image are close to (e.g., matched) the feature parts of the object OBJ captured in the image data IMG_2D. As a result, the position and pose calculation unit 312 can determine the positional relationship between the coordinate system of the two-dimensional model data and the 2D shooting coordinate system. Then, based on the positional relationship between the coordinate system of the two-dimensional model data and the 2D shooting coordinate system, the position and pose calculation unit 312 can calculate at least one of the position and pose of the object OBJ in the 2D shooting coordinate system according to at least one of the position and pose of the object OBJ in the coordinate system of the two-dimensional model data.

[0217] Subsequently, the position and orientation calculation unit 312, based on a transformation matrix used to convert the three-dimensional coordinates in either the robot coordinate system or the 2D imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system and the 2D imaging coordinate system, transforms at least one of the position and orientation of the object OBJ in the 2D imaging coordinate system into at least one of the position and orientation of the object OBJ in the robot coordinate system. When the robot coordinate system is used as the global coordinate system, at least one of the position and orientation of the object OBJ in the robot coordinate system is equivalent to at least one of the position and orientation of the object OBJ in the global coordinate system. However, when a coordinate system different from the robot coordinate system is used as the global coordinate system, the position and orientation calculation unit 312 can use a transformation matrix used to convert the three-dimensional coordinates in either the robot coordinate system or the global coordinate system into three-dimensional coordinates in the other of the robot coordinate system and the global coordinate system, and calculate at least one of the position and orientation of the object OBJ in the global coordinate system based on at least one of the position and orientation of the object OBJ in the robot coordinate system.

[0218] In the case of 2D matching processing, the position and pose calculation unit 312 can also calculate the matching similarity, similar to the case of 3D matching processing. The matching similarity calculated in 2D matching processing is the similarity between the template image (i.e., the two-dimensional model of the object OBJ) and the image shown in the image data IMG_2D (particularly the image portion of the object OBJ that incorporates the template image). Furthermore, in the following description, the similarity between the template image (i.e., the two-dimensional model of the object OBJ) and the image shown in the image data IMG_2D (particularly the image portion of the object OBJ that incorporates the template image), i.e., the matching similarity, can be simply referred to as the matching similarity of the object OBJ. Moreover, the purpose of the matching similarity calculated in 2D matching processing is the same as that of the matching similarity calculated in the 3D matching processing described above, therefore, its detailed explanation is omitted.

[0219] In the case of 2D matching processing, similarly to the case of 3D matching processing, the position and pose calculation unit 312 can also perform 2D matching processing sequentially or simultaneously in parallel on multiple object objects OBJ captured in the image shown in the image data IMG_2D. Then, similarly to the case of 3D matching processing, the position and pose calculation unit 312 can select at least one object object OBJ from the multiple object objects OBJ as the processing execution object. In this case, the position and pose calculation unit 312 can select at least one object object OBJ from the multiple object objects OBJ as the processing execution object based on matching similarity. Furthermore, Figure 4 In step S3, the position and pose calculation unit 312 performs matching processing using image data IMG_2D instead of 3D matching processing using 3D position data WSD, thereby calculating at least one of the position and pose of the object OBJ. Figure 4 The robot control process shown may not include Figure 4 Step S2.

[0220] Or, in Figure 4 In step S3, in addition to performing at least one of the 3D matching processing using three-dimensional position data WSD and 2D matching processing using image data IMG_2D, or as an alternative, the position and pose calculation unit 312 may perform matching processing using both three-dimensional position data WSD and image data IMG_2D to calculate at least one of the position and pose of the object OBJ.

[0221] For example, the position and pose calculation unit 312 can calculate at least one of the position and pose of the object OBJ in the global coordinate system based on a portion of the position and pose of the object OBJ in the 2D shooting coordinate system calculated based on image data IMG_2D, and a portion of the position and pose of the object OBJ in the 3D shooting coordinate system calculated based on image data IMG_3D. For example, the position and pose calculation unit 312 can set at least one of the initial position and initial pose of the three-dimensional model (template model TM3) used for 3D matching processing based on at least one of the position and pose of the object OBJ in the 2D shooting coordinate system calculated based on image data IMG_2D, and perform 3D matching processing using the three-dimensional model configured at the set initial position and / or with the set initial pose, thereby calculating at least one of the position and pose of the object OBJ in the 3D shooting coordinate system, and calculate at least one of the position and pose of the object OBJ in the global coordinate system based on at least one of the position and pose of the object OBJ in the 3D shooting coordinate system.

[0222] When calculating the position and orientation of an object OBJ using both image data IMG_2D and 3D position data WSD, the position and orientation calculation unit 312 can select an object OBJ that satisfies the condition that the sum or product of the matching similarity calculated through 2D matching processing (hereinafter referred to as 2D matching similarity) and the matching similarity calculated through 3D matching processing (hereinafter referred to as 3D matching similarity) is the maximum, and use this object OBJ as the object object OBJ for which the end effector 4 should perform the prescribed processing. The position and orientation calculation unit 312 can also select an object OBJ that satisfies the condition that the 2D matching similarity exceeds the matching determination threshold and is the maximum, and use this object object OBJ as the object object OBJ for which the end effector 4 should perform the prescribed processing. The position and attitude calculation unit 312 may also select an object OBJ that satisfies both the 2D matching similarity and 3D matching similarity exceeding the matching determination threshold, as an object OBJ for which the end effector 4 should perform the prescribed processing. Alternatively, the position and attitude calculation unit 312 may select an object OBJ that satisfies both the 2D matching similarity and 3D matching similarity exceeding the matching determination threshold and is closest to the end effector 4, as an object OBJ for which the end effector 4 should perform the prescribed processing. Furthermore, the position and attitude calculation unit 312 may select an object OBJ that satisfies both the 2D matching similarity and 3D matching similarity exceeding the matching determination threshold and has the largest Z-coordinate along the Z-axis (located at the highest position), as an object OBJ for which the end effector 4 should perform the prescribed processing.

[0223] Furthermore, if the position and attitude calculation unit 312 performs matching processing using three-dimensional position data WSD but not matching processing using image data IMG_2D, the control device 3 may not acquire image data IMG_2D in step S1. If the control device 3 does not acquire image data IMG_2D, the imaging device 21 may not capture the object OBJ. If the control device 3 does not acquire image data IMG_2D, the imaging system 2 may not include the imaging device 21.

[0224] If the position and attitude calculation unit 312 performs matching processing using image data IMG_2D but not matching processing using three-dimensional position data WSD, the control device 3 may not acquire image data IMG_3D in step S1. If the control device 3 does not acquire image data IMG_3D, the imaging device 22 may not capture the object OBJ. If the control device 3 does not acquire image data IMG_3D, the imaging system 2 may not include the imaging device 22. If the control device 3 does not acquire image data IMG_3D, the control device 3 may not include the three-dimensional position data generation unit 311.

[0225] Furthermore, in the following explanation, for the sake of simplicity, the explanation mainly focuses on... Figure 4 The example of 3D matching processing performed by the position and attitude calculation unit 312 in step S3 will be explained.

[0226] Next, the signal generation unit 313 uses the position and orientation data POI generated in step S3 to generate a robot control signal (step S4). For example, the signal generation unit 313 can generate a robot control signal so that the end effector 4 can perform specified processing on the object OBJ. For example, the signal generation unit 313 can generate a robot control signal so that the end effector 4 approaches the object OBJ so that the end effector 4 can perform specified processing on the object OBJ. For example, the signal generation unit 313 can generate a robot control signal so that the positional relationship between the end effector 4 and the object OBJ becomes the desired positional relationship. For example, the signal generation unit 313 can also generate a robot control signal for controlling the movement of the robotic arm 12 so that the positional relationship between the end effector 4 and the object OBJ becomes the desired positional relationship. For example, the signal generation unit 313 can generate a robot control signal so that the end effector 4 performs specified processing on the object OBJ at the moment when the positional relationship between the end effector 4 and the object OBJ becomes the desired positional relationship. For example, the signal generation unit 313 can generate a robot control signal for controlling the action of the end effector 4 so that the object OBJ is subjected to prescribed processing at the moment when the positional relationship between the end effector 4 and the object OBJ is the desired positional relationship. Furthermore, as described above, the robot control signal for controlling the action of the end effector 4 can be referred to as the end effector control signal.

[0227] Furthermore, the signal generation unit 313 can use the position and orientation data POI generated in step S3 to generate a robot control signal for feedback control of robot 1. For example, the signal generation unit 313 can use the position and orientation data POI to generate a robot control signal for feedback control including P (Proportional) control. For example, the signal generation unit 313 can use the position and orientation data POI to generate a robot control signal for feedback control including PI (Proportional-Integral) control. For example, the signal generation unit 313 can use the position and orientation data POI to generate a robot control signal for feedback control including PID (Proportional-Integral-Differential) control.

[0228] As an example, Figure 6 (a) to Figure 6 (d) are side views showing the positional relationship between robot 1 and workpiece W at a certain moment during a holding process performed by a holding device T#1, such as an AGV (Automatic Guided Vehicle), which moves on support surface S. In this case, control device 3 can generate position and orientation data POI representing at least one of the position and orientation of workpiece W (i.e., an example of object OBJ in the holding process), which is the object of the holding process, and use the generated position and orientation data POI to generate robot control signals. For example, as Figure 6 As shown in (a), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4 moves toward the space directly above the moving workpiece W. As an example, the signal generation unit 313 can generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4 is positioned (in other words, close to) the successively updated target location of the end effector 4 in a desired orientation, based on at least one of the position and orientation of the workpiece W calculated successively by the position and orientation calculation unit 312. Figure 6 As shown in (b), the signal generation unit 313 can also generate robot control signals to control the movement of the robotic arm 12 so that the end effector 4, located directly above the moving workpiece W, continues to approach the workpiece W from directly above it until the workpiece W can be held. Figure 6As shown in (c), the signal generation unit 313 can also generate robot control signals for controlling the movements of the robotic arm 12 and the end effector 4, so that the end effector 4, located at a position capable of maintaining the movement of the workpiece W, follows the moving workpiece W while maintaining the workpiece W. For example... Figure 6 As shown in (d), the signal generation unit 313 can also generate robot control signals for controlling the movements of the robotic arm 12 and the end effector 4 so that the end effector 4, which holds the workpiece W, leaves the moving mounting device T#1 while keeping the workpiece W unchanged.

[0229] When robot 1 uses end effector 4 to hold workpiece W placed on loading device T#1, robot 1 can perform a release process to release the held workpiece W, thereby placing the held workpiece W onto loading device T#2 (i.e., an example of the object OBJ in the release process), which is different from loading device T#1. That is, robot 1 can also perform a configuration process for placing workpiece W on loading device T#2 by continuously performing holding and release processes. In this case, control device 3 can generate position and orientation data POI representing at least one of the position and orientation of loading device T#2, which is the object of the release process, and use the generated position and orientation data POI to generate robot control signals. For example, Figure 7 (a) to Figure 7 (d) are side views showing the positional relationship between the robot 1 and the workpiece W at a certain moment during the release process of placing the workpiece W on the mounting device T#2, which moves on the support surface S. In this case, as... Figure 7 As shown in (a), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4, which holds the workpiece W, moves to the space directly above the moving mounting device T#2 while keeping the workpiece W unchanged. As an example, the signal generation unit 313 can generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4 is located (in other words, close to) the successively updated moving target location based on the moving mounting device T#2, while successively updating the moving target location of the end effector 4 according to the moving mounting device T#2, based on at least one of the position and orientation of the mounting device T#2 calculated successively by the position and orientation calculation unit 312. Figure 7 As shown in (b), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4, which is located directly above the moving mounting device T#2 and keeps the workpiece W directly above the mounting device T#2, approaches the mounting device T#2 while keeping the workpiece W unchanged until the workpiece W can be positioned on the mounting device T#2. Figure 7As shown in (c), the signal generation unit 313 can also generate robot control signals for controlling the actions of the robotic arm 12 and the end effector 4, so that the end effector 4, located at a position capable of placing the workpiece W on the loading device T#2, follows the moving loading device T#2 while loading the workpiece W onto the loading device T#2 (i.e., releasing the held workpiece W). Figure 7 As shown in (d), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4, which places the workpiece W behind the loading device T#2, leaves the loading device T#2.

[0230] In the case of performing a release process (especially a release process performed for the purpose of configuration processing), in addition to position and attitude data POI representing at least one of the position and attitude of the loading device T#2 (i.e., an example of the object OBJ in the release process) which is the first object of the release process, the control device 3 may further generate position and attitude data POI representing at least one of the position and attitude of the workpiece W (i.e., an example of the object OBJ in the release process) which is the second object of the release process. That is, in addition to generating position and attitude data POI representing at least one of the position and attitude of the workpiece W which is not yet held by the end effector 4 before the end effector 4 holds the workpiece W, the control device 3 may also generate position and attitude data POI representing at least one of the position and attitude of the workpiece W held by the end effector 4 after the end effector 4 holds the workpiece W.

[0231] In the case of release processing, before releasing the workpiece W held by the end effector 4 through the release processing, in addition to the position and orientation data POI regarding the loading device T#2, the control device 3 may also use position and orientation data POI representing at least one of the position and orientation of the workpiece W held by the end effector 4 to generate a robot control signal. That is, the control device 3 may use position and orientation data POI representing at least one of the position and orientation of the workpiece W held by the end effector 4 to generate a robot control signal during at least a portion of the period during which the end effector 4 holds the workpiece W.

[0232] For example, the signal generation unit 313 can also generate robot control signals to control the robotic arm 12 to move the workpiece W held by the end effector 4 to a desired position (e.g., the position where the workpiece W should be released). In this case, compared to not using the position and orientation data (POI) of the workpiece W, the robot 1 can properly position the workpiece W held by the end effector 4 to the desired position of the mounting device T (perform configuration processing). This is because the position of the workpiece W held by the end effector 4 is known information to the control device 3. If the position of the workpiece W is not known, there is a technical problem that the workpiece W may collide with the mounting device T. Or, if the position of the workpiece W is not known, the configuration processing described above may result in the workpiece W being positioned in an unexpected position on the mounting device T (or other object). Or, if the position of the workpiece W is not known, the embedding processing described above may result in the workpiece W being embedded in an unexpected position on another object. Or, if the position of the workpiece W is not known, the pasting processing described above may result in the workpiece W being pasted to an unexpected position on another object. Alternatively, if the position of workpiece W is not known, the above-described bonding process may result in the workpiece W being bonded to an unintended position on another object. Alternatively, if the position of workpiece W is not known, the above-described welding process may result in the workpiece W being welded to an unintended position on another object. However, in this embodiment, since the position of workpiece W is known, the possibility of such a technical problem is nonexistent or low.

[0233] Furthermore, for example, the signal generation unit 313 can also generate robot control signals for controlling the end effector 4 to change the posture of the workpiece W held by the end effector 4 to the desired posture. In this case, compared to the case where the position and posture information POI0 related to the workpiece W is not used, the robot 1 can use the end effector 4 to place the workpiece W in the desired posture on the mounting device T (and can perform the configuration process). This is because the posture of the workpiece W held by the end effector 4 is known information to the control device 3. Suppose that if the posture of the workpiece W is not the desired posture, a technical problem may occur where the workpiece W collides with the mounting device T. Or, if the posture of the workpiece W is not the desired posture, the above configuration process may result in a technical problem where the workpiece W is placed on the mounting device T (or other object) in an unexpected posture. Or, if the posture of the workpiece W is not the desired posture, the above embedding process may result in a technical problem where the workpiece W is embedded into other objects in an unexpected posture. Alternatively, if the orientation of workpiece W is not the desired orientation, the above-described adhesive process may result in the workpiece W being adhered to other objects in an unexpected orientation. Similarly, if the orientation of workpiece W is not the desired orientation, the above-described adhesive process may result in the workpiece W being adhered to other objects in an unexpected orientation. Or, if the orientation of workpiece W is not the desired orientation, the above-described welding process may result in the workpiece W being welded to other objects in an unexpected orientation. However, in this embodiment, since the robot 1 can be controlled to make the orientation of workpiece W the desired orientation, the possibility of such technical problems occurring is non-existent or low.

[0234] Furthermore, not limited to the case of release processing, in any scenario where the end effector 4 holds the workpiece W, the control device 3 can use position and orientation data POI, which represents at least one of the position and orientation of the workpiece W held by the end effector 4, to generate robot control signals.

[0235] Figure 6 (a) to Figure 6 (d) shows the mounting device T#1 and Figure 7 (a) to Figure 7 (d) It is not necessary for at least one of the mounting devices T#2 shown to move on the support surface S. For example, Figure 8 (a) to Figure 8 (b) are side views showing the positional relationship between the robot 1 and the workpiece W at a certain moment during the holding process of the workpiece W placed on the mounting device T#1, which is stationary on the support surface S. In this case, as Figure 8As shown in (a), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robot arm 12 to bring the end effector 4 close to the workpiece W until the workpiece W can be kept stationary. As an example, the signal generation unit 313 can generate robot control signals for controlling the movement of the robot arm 12 to bring the end effector 4 to a position (in other words, close to) that can be maintained by the end effector 4 at the workpiece W, whose position and orientation have been calculated, based on at least one of the position and orientation of the workpiece W calculated by the position and orientation calculation unit 312. Figure 8 As shown in (b), the signal generation unit 313 can generate robot control signals for controlling the movements of the robotic arm 12 and the end effector 4 so that the end effector 4, located at a position capable of holding the workpiece W stationary, holds the workpiece W. Furthermore, Figure 8 (c) to Figure 8 (e) are side views showing the positional relationship between the robot 1 and the workpiece W at a certain moment during the release process of placing the workpiece W on the mounting device T#2, which is stationary on the support surface S. In this case, as Figure 8 As shown in (c), the signal generation unit 313 can also generate robot control signals to control the movement of the robotic arm 12 so that the end effector 4 holding the workpiece W approaches the mounting device T#2 while keeping the workpiece W unchanged until the workpiece W can be positioned on the stationary mounting device T#2. Figure 8 As shown in (d), the signal generation unit 313 can also generate robot control signals for controlling the actions of the robotic arm 12 and the end effector 4 so that the end effector 4, located at a position capable of placing the workpiece W in the mounting device T#2, places the workpiece W in the stationary mounting device T#2 (i.e., releases the held workpiece W). Figure 8 As shown in (e), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4, which places the workpiece W behind the loading device T#2, leaves the loading device T#2.

[0236] The loading device T#1 can load multiple workpieces W. For example, multiple workpieces W can be loaded on the loading device T#1 in a manner in which multiple workpieces W are arranged on the loading device T#1 according to a certain reference. For example, multiple workpieces W can be loaded on the loading device T#1 in a manner in which multiple workpieces W are randomly stacked on the loading device T#1. In this case, the robot 1 can perform a holding process for selectively holding one desired workpiece W among the multiple workpieces W loaded on the loading device T#1. In particular, the robot 1 can perform a holding process for sequentially holding the multiple workpieces W loaded on the loading device T#1 one by one.

[0237] Furthermore, robot 1 can perform a release process for configuring multiple workpieces W onto the mounting device T#2. That is, robot 1 can perform a holding process and a release process to sequentially configure multiple workpieces W, which are mounted on mounting device T#1, onto mounting device T#2 (or further onto other mounting devices). In this case, robot 1 can perform a release process where multiple workpieces W are arranged sequentially on mounting device T#2 according to a certain reference. Alternatively, robot 1 can perform a release process where multiple workpieces W are randomly stacked on mounting device T#2.

[0238] As an example, Figure 9 (a) to Figure 9 (e) and Figure 10 (a) to Figure 10 (e) are side views showing the positional relationship between the robot 1 and the workpiece W at a certain moment during the release process of sequentially holding the two workpieces W#1 and W#2 mounted on the mounting device T#1 and sequentially positioning the two workpieces W#1 and W#2 on the mounting device T#2. In this case, as Figure 9 As shown in (a), the signal generation unit 313 can generate actions to control the robotic arm 12 so that the end effector 4 approaches the workpiece W#2 until either workpiece W#1 or W#2 can be held in place. Figure 9 In the example shown in (a), the robot control signal is up to workpiece W#2). As an example, the signal generation unit 313 can generate a robot control signal for controlling the movement of the robotic arm 12 so that the end effector 4 is positioned (in other words, close to) a position that can be maintained by the end effector 4 at least one of the positions and orientations of workpiece W#2 calculated by the position and orientation calculation unit 312. Then, as... Figure 9 As shown in (b), the signal generation unit 313 can generate robot control signals for controlling the movements of the robotic arm 12 and the end effector 4 so that the end effector 4, located in a position capable of holding the workpiece W#2, holds the workpiece W#2. Then, as... Figure 9As shown in (c), the signal generation unit 313 can generate a robot control signal for controlling the movement of the robotic arm 12 so that the end effector 4, which holds the workpiece W#2, approaches the loading device T#2 while keeping the workpiece W#2 unchanged until the workpiece W#2 can be positioned on the loading device T#2. As an example, the signal generation unit 313 can generate a robot control signal for controlling the movement of the robotic arm 12 so that the end effector 4 is positioned (in other words, approaching) in a desired posture at a position on the loading device T#2 where the end effector 4 can position the workpiece W2, based on at least one of the position and posture of the loading device T#2 calculated by the position and posture calculation unit 312. Then, as... Figure 9 As shown in (d), the signal generation unit 313 can also generate robot control signals for controlling the actions of the robotic arm 12 and the end effector 4 so that the end effector 4, located at a position capable of placing the workpiece W#2 on the loading device T#2, loads the workpiece W#2 onto the loading device T#2 (i.e., releases the held workpiece W#2). Then, as... Figure 9 As shown in (e), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robotic arm 12 to cause the end effector 4, which positions the workpiece W#2 behind the mounting device T#2, to leave the mounting device T#2. Then, as... Figure 10 As shown in (a), the signal generation unit 313 can generate actions to control the robotic arm 12 so that the end effector 4 approaches the workpiece W#1 until it can hold one of the remaining workpieces W#1 and W#2. Figure 10 In the example shown in (a), the robot control signal is up to workpiece W#1. As an example, the signal generation unit 313 can generate a robot control signal for controlling the movement of the robotic arm 12 so that the end effector 4 is positioned (in other words, close to) a position that can be maintained by the end effector 4 on workpiece W#1, whose position and orientation have been calculated, based on at least one of the position and orientation of workpiece W#1 calculated by the position and orientation calculation unit 312. Then, as Figure 10 As shown in (b), the signal generation unit 313 can generate robot control signals for controlling the movements of the robotic arm 12 and the end effector 4 so that the end effector 4, located in a position capable of holding the workpiece W#1, holds the workpiece W#1. Then, as... Figure 10As shown in (c), the signal generation unit 313 can generate a robot control signal for controlling the movement of the robotic arm 12 so that the end effector 4, which holds the workpiece W#1, approaches the mounting device T#2 while keeping the workpiece W#1 unchanged until the workpiece W#1 can be positioned on the mounting device T#2. As an example, the signal generation unit 313 can generate a robot control signal for controlling the movement of the robotic arm 12 so that the end effector 4 is positioned (in other words, approaching) in a desired posture at a position on the mounting device T#2 where the end effector 4 can position the workpiece W#2 at the position and posture calculated by the position and posture calculation unit 312. Then, as... Figure 10 As shown in (d), the signal generation unit 313 can also generate robot control signals for controlling the actions of the robotic arm 12 and the end effector 4 so that the end effector 4, located at a position capable of placing the workpiece W#1 on the loading device T#2, loads the workpiece W#1 onto the loading device T#2 (i.e., releases the held workpiece W#1). Then, as... Figure 10 As shown in (e), the signal generation unit 313 can also generate robot control signals for controlling the movement of the robotic arm 12 so that the end effector 4, which places the workpiece W#1 behind the loading device T#2, leaves the loading device T#2.

[0239] Robot 1 can hold multiple workpieces W on a mounting device T#1 that moves sequentially on the support surface S. Alternatively, robot 1 can hold multiple workpieces W on a mounting device T#1 that is stationary on the support surface S. Robot 1 can perform a release process that sequentially positions the multiple workpieces W onto a mounting device T2 that moves on the support surface S. Robot 1 can also perform a release process that sequentially positions the multiple workpieces W onto a mounting device T2 that is stationary on the support surface S.

[0240] Again in Figure 4 In this process, the signal generation unit 313 uses the communication device 33 to output the robot control signal generated in step S4 to the robot 1 (specifically the robot control device 13). As a result, the robot control device 13 controls at least one of the actions of the robot 1 (for example, the actions of the robotic arm 12) and the actions of the end effector 4 based on the robot control signal.

[0241] Subsequently, the control device 3 repeats the series of processes from steps S1 to S4 (step S5) until it is determined that the robot control process is to end. That is, while the control device 3 is controlling the movement of at least one of the robotic arm 12 and the end effector 4 based on the robot control signal, it also continuously acquires at least one of the image data IMG_2D and IMG_3D from the imaging devices 21 and 22. For example, as Figure 6 (a) to Figure 6 As shown in (d), when the end effector 4 is performing a holding process, the control device 3 can repeat a series of processes from steps S1 to S4 until the end effector 4 holds the workpiece W (furthermore, until the end effector 4 holding the workpiece W leaves the mounting device T#1). For example, as Figure 7 (a) to Figure 7 As shown in (d), when the end effector 4 performs the release process, the control device 3 can repeat a series of processes from step S1 to step S4 until the end effector 4 places the workpiece W on the loading device T#2 (further, until the end effector 4 leaves the loading device T#2 after placing the workpiece W on the loading device T#2).

[0242] In this case, as described above, since the movement of at least one of the robotic arm 12 and the end effector 4 is controlled based on robot control signals, the capturing devices 21 and 22 can capture images of the object OBJ during the relative movement of the object OBJ and the capturing devices 21 and 22, respectively. For example, the capturing devices 21 and 22 can capture images of the object OBJ during a period when the object OBJ is stationary and the capturing devices 21 and 22 are moving, respectively. For example, the capturing devices 21 and 22 can capture images of the object OBJ during a period when the object OBJ is moving and the capturing devices 21 and 22 are stationary, respectively. For example, the capturing devices 21 and 22 can capture images of the object OBJ during a period when the object OBJ is moving and the capturing devices 21 and 22 are moving, respectively. That is, the control device 3 can continuously operate during the relative movement of the object OBJ and the capturing devices 21 and 22 (i.e., during the period when the capturing devices 21 and 22 and at least one of the object OBJ are moving). Figure 4 The robot control process shown can be repeated. Figure 4 (The robot control processing shown). As a result, even during the period when the control device 3 controls the actions of the robot 1 based on the robot control signal, it is able to regenerate (i.e., update) the position and orientation data POI representing at least one of the position and orientation of the object OBJ based on the newly acquired image data IMG_2D and IMG_3D.

[0243] The imaging devices 21 and 22 can respectively capture images of the object OBJ while both the object OBJ and the imaging devices 21 and 22 are stationary. The control device 3 can operate during the periods when the imaging devices 21 and 22 and the object OBJ are stationary. Figure 4 The robot control process is shown. Furthermore, control device 3 can repeat the process while the imaging devices 21 and 22 are stationary and while the object OBJ is at rest. Figure 4 The robot control process is shown.

[0244] Furthermore, as described above, the position and attitude calculation unit 312 may not calculate at least one of the position and attitude of the object OBJ in the global coordinate system in step S3. In this case, the position and attitude calculation unit 312 may... Figure 4 In step S3, at least one of the position and orientation of the object OBJ in a coordinate system different from the global coordinate system (e.g., the robot coordinate system, the 2D camera coordinate system, or the 3D camera coordinate system) is calculated. That is, the position and orientation calculation unit 312 can generate position and orientation data POI representing at least one of the position and orientation of the object OBJ in a coordinate system different from the global coordinate system (e.g., the robot coordinate system, the 2D camera coordinate system, or the 3D camera coordinate system). In this case, the signal generation unit 313 can use the position and orientation data POI calculated in step S3, representing at least one of the position and orientation of the object OBJ in a coordinate system different from the global coordinate system, to generate a robot control signal in step S4.

[0245] (4) Variations

[0246] Next, variations of the robot system SYS will be described. Furthermore, the variations described below can be combined with other variations. Alternatively, detailed descriptions of structural elements already described will be omitted in the following description. Furthermore, detailed descriptions of processes already described will be omitted in the following description. That is, the following description will focus on structural elements different from those already described, and processes different from those already described. Therefore, unless otherwise stated, in the variations described below, the robot system SYS may also possess the structural elements already described, and the robot system SYS may also perform the processes already described.

[0247] (4-1) Variation Example 1

[0248] As described above, the position and attitude calculation unit 312 performs 3D matching processing (in other words, template matching processing) on ​​the three-dimensional position data WSD using the three-dimensional model data WMD to generate position and attitude data POI. Specifically, in order to generate position and attitude data POI, the position and attitude calculation unit 312 performs 3D matching processing (in other words, template matching processing) on ​​the three-dimensional position data WSD, which uses the template model TM3 (three-dimensional model) shown in the three-dimensional model data WMD as a template. In Variation 1, in addition to performing 3D matching processing that uses the entire template model TM3 (three-dimensional model) shown in the three-dimensional model data WMD as a template, or as an alternative, the position and attitude calculation unit 312 can also perform 3D matching processing that uses a part of the template model TM3 (three-dimensional model) shown in the three-dimensional model data WMD as a template. That is, the position and attitude calculation unit 312 can perform 3D matching processing that uses three-dimensional model data representing a part of the template model TM3 (three-dimensional model), i.e., local model data. Furthermore, in the following description, the template model that is entirely equivalent to the template model TM3 shown in the 3D model data WDM is referred to as the overall model TM3_W. That is, in the following description, the template model TM3 (3D model) itself is referred to as the overall model TM3_W. On the other hand, in the following description, the template model that is equivalent to a part of the template model TM3 shown in the 3D model data WDM is referred to as the local model TM3_P. In this case, the local model TM3_P can be considered as a part of the overall model TM3_W. The template model that is equivalent to a part of the template model TM3 can be referred to as a 3D local model.

[0249] The control device 3 (particularly the arithmetic device 31) can generate a local model TM3_P. For example, the control device 3 (particularly the arithmetic device 31) can extract at least a portion of the overall model TM3_W from the overall model TM3_W, thereby generating a local model TM3_P that is equivalent to at least a portion of the extracted overall model TM3_W. However, a device different from the control device 3 (e.g., an arithmetic device different from the arithmetic device 31) can also generate the local model TM3_P.

[0250] As an example, control device 3 can generate a local model TM3_P based on instructions from the user. For instance, control device 3 can generate the local model TM3_P based on instructions from the user to extract at least a portion of the overall model TM3_W as the local model TM3_P. In other words, control device 3 can generate the local model TM3_P based on instructions from the user to generate the local model TM3_P.

[0251] The user can use input device 34 to input model generation information related to user instructions for generating the local model TM3_P to control device 3. For example, control device 3 can display a model generation screen 351, operable by the user to generate the local model TM3_P, on output device 35, which functions as a display device. Figure 11 (a) through 11(b) show an example of model generation screen 351. (e.g.) Figure 11 As shown in (a) to 11(b), the overall model TM3_W can be displayed on the model generation screen 351. On the model generation screen 351, the user can input an instruction (model generation information) to the control device 3 for generating a local model TM3_P using the overall model TM3_W displayed on the model generation screen 351.

[0252] like Figure 11 As shown in (b), for example, the user can input the following instruction into the control device 3: that is, select at least a portion of the overall model TM3_W displayed on the model generation screen 351 as the selected model TM3_S. Furthermore, Figure 11 In (b), the user-selected model TM3_S in the overall model TM3_W is shown with a solid line, and the unselected model portion in the overall model TM3_W is shown with a dashed line. Furthermore, the user inputs the following instruction to the control device 3: specifying the user-selected model TM3_S as the model portion to be included in the local model TM3_P. In this case, the control device 3 can extract the user-selected model portion from the overall model TM3_W to generate a local model TM3_P that includes at least a portion of the extracted overall model TM3_W. Alternatively, the user inputs the following instruction to the control device 3: specifying the user-selected model TM3_S as the model portion that should not be included in the local model TM3_P. In this case, the control device 3 can extract the unselected model portion from the overall model TM3_W to generate a local model TM3_P that includes at least a portion of the extracted overall model TM3_W.

[0253] As another example, control device 3 can generate a local model TM3_P without using instructions from the user. For example, control device 3 can generate the local model TM3_P based on simulated image data. The simulated image data can be generated by assuming that the imaging system 2 is photographing the object OBJ. For example, image data expected to be generated by the imaging system 2 in the case of assuming that the imaging system 2 is photographing the object OBJ can be generated as simulated image data. The simulated image data used to generate the local model TM3_P used in the 3D matching process can be, for example, image data in the same form as the image data IMG_3D (e.g., stereo image data). Then, control device 3 can select a local model TM3_P from the overall model TM3_W that satisfies the following condition to generate a model based on the simulated image data (e.g., using simulated point cloud data calculated from the simulated image data): that is, "the object OBJ is detected with high accuracy by using the matching process of the local model TM3_P (as a result, at least one of the position and orientation of the object OBJ is calculated with high accuracy)".

[0254] After generating the local model TM3_P, in addition to performing 3D matching processing using the three-dimensional position data WSD and the overall model TM3_W, or as an alternative, the control device 3 can also perform 3D matching processing using the three-dimensional position data WSD and the local model TM3_P.

[0255] As a first example, the control device 3 (particularly the position and attitude calculation unit 312) can first perform 3D matching processing using the overall model TM3_W. Then, the control device 3 can determine whether the calculation result condition is met, which is related to the result of the position and attitude calculation (position and attitude calculation processing), including the 3D matching processing using the overall model TM3_W. If the calculation result condition is met, the control device 3 can generate robot control signals based on the result of the 3D matching processing using the overall model TM3_W (i.e., position and attitude data POI). On the other hand, if the calculation result condition is not met, the control device 3 (particularly the position and attitude calculation unit 312) can further perform 3D matching processing using the local model TM3_P. Then, the control device 3 can determine whether the calculation result condition is met, which is related to the result of the position and attitude calculation (position and attitude calculation processing), including the 3D matching processing using the local model TM3_P. If the conditions for the calculation result are met, the control device 3 can generate robot control signals based on the results of 3D matching processing using the local model TM3_P (i.e., position and orientation data POI).

[0256] As a second example, the control device 3 (particularly the position and attitude calculation unit 312) can first perform 3D matching processing using the local model TM3_P. Then, the control device 3 can determine whether the calculation result condition is met, which is related to the result of the position and attitude calculation including the 3D matching processing using the local model TM3_P. If the calculation result condition is met, the control device 3 can generate a robot control signal based on the result of the 3D matching processing using the local model TM3_P (i.e., position and attitude data POI). On the other hand, if the calculation result condition is not met, the control device 3 (particularly the position and attitude calculation unit 312) can further perform 3D matching processing using the global model TM3_W. Then, the control device 3 can determine whether the calculation result condition is met, which is related to the result of the position and attitude calculation including the 3D matching processing using the global model TM3_W. If the calculation result condition is met, the control device 3 can generate a robot control signal based on the result of the 3D matching processing using the global model TM3_W (i.e., position and attitude data POI).

[0257] As a third example, the control device 3 (particularly the position and pose calculation unit 312) can perform both 3D matching processing using the local model TM3_P and 3D matching processing using the global model TM3_W. Then, the control device 3 can compare the matching similarity calculated by the 3D matching processing using the local model TM3_P with a matching decision threshold, and also compare the matching similarity calculated by the 3D matching processing using the global model TM3_W with the matching decision threshold, thereby generating position and pose data POI. For example, the control device 3 can select an object OBJ whose matching similarity calculated by the 3D matching processing using the local model TM3_P exceeds the matching decision threshold, and whose matching similarity calculated by the 3D matching processing using the global model TM3_W exceeds the matching decision threshold, as the processing execution object, and generate position and pose data POI for that object OBJ. Then, the control device 3 can generate robot control signals based on the generated position and pose POI.

[0258] For example, in at least one of the first to third examples, the matching threshold used in the position and pose calculation including 3D matching processing using the global model TM3_W can be the same as the matching threshold used in the position and pose calculation including 3D matching processing using the local model TM3_P. Alternatively, the matching threshold used in the position and pose calculation including 3D matching processing using the global model TM3_W can be different from the matching threshold used in the position and pose calculation including 3D matching processing using the local model TM3_P. For example, the matching threshold used in the position and pose calculation including 3D matching processing using the global model TM3_W can be larger than the matching threshold used in the position and pose calculation including 3D matching processing using the local model TM3_P. For example, the matching threshold used in the position and pose calculation including 3D matching processing using the global model TM3_W can be smaller than the matching threshold used in the position and pose calculation including 3D matching processing using the local model TM3_P.

[0259] As a fourth example, the control device 3 (particularly the position and attitude calculation unit 312) can perform 3D matching processing using a first local model TM3_P representing the first model part of the overall model TM3_W (three-dimensional model) before, after, or without performing 3D matching processing using the overall model TM3_W. Then, the control device 3 can determine whether the calculation result condition is met, which is related to the result of the position and attitude calculation including the 3D matching processing using the first local model TM3_P. If the calculation result condition is met, the control device 3 can generate a robot control signal based on the result of the 3D matching processing using the first local model TM3_P (i.e., position and attitude data POI). On the other hand, if the calculation result condition is met, the control device 3 (particularly the position and attitude calculation unit 312) can further perform 3D matching processing using a second local model TM3_P that is different from the first local model TM3_P. The second local model TM3_P differs from the first model portion representing the overall model TM3_W in that it shows a second model portion of the overall model TM3_W that is at least partially different from the first model portion. The control device 3 can then determine whether the calculation result condition is met, which is related to the result of position and pose calculations including 3D matching processing using the second local model TM3_P. If the calculation result condition is met, the control device 3 can generate robot control signals based on the result of 3D matching processing using the second local model TM3_P (i.e., position and pose data POI). Conversely, if the calculation result condition is not met, the control device 3 (particularly the position and pose calculation unit 312) can further perform 3D matching processing using a third local model TM3_P that is different from both the first and second local models TM3_P. The difference lies in that it shows a third model portion of the overall model TM3_W that is at least partially different from both the first and second model portions. Then, repeat the same action until the condition for the calculation result is determined to be true.

[0260] Therefore, in the fourth example, multiple local models TM3_P are used. Thus, the control device 3 can generate multiple local models TM3_P that are different from each other. That is, the control device 3 can generate multiple local models TM3_P, which are at least partially different three-dimensional models. Figure 12 An example of multiple local models TM3_P is shown. Figure 12The example shown illustrates how control device 3 generates three local models TM3_P (specifically, local model TM3_P#1, local model TM3_P#2, and local model TM3_P#3). Furthermore, control device 3 can also generate two or more different local models TM3_P.

[0261] For example, such as Figure 12 As shown in the local model TM3_P#1, the control device 3 can generate a local model TM3_P after removing at least one curved portion (corner portion) from the overall model TM3_W. In this case, as described above, the control device 3 can generate the local model TM3_P based on instructions from the user. As another example, the control device 3 can generate the local model TM3_P by identifying curved portions (corner portions) and other portions (e.g., planar portions) in the overall model TM3_W without using instructions from the user, and by removing the curved portions (corner portions) from the overall model TM3_W. Furthermore, it is not limited to... Figure 12 The control device 3 can also generate a model TM3_P that reduces the thickness of at least a portion of the model in the overall model TM3_W, as the local model TM3_P. For example, the control device 3 can also generate a model that reduces the thickness of at least a portion of the model in the overall model TM3_W, as the local model TM3_P. The model that reduces the thickness of at least a portion of the overall model TM3_W is a model of at least a portion of the overall model TM3_W, and therefore can be called the local model TM3_P. In this case, the control device 3 can generate the local model TM3_P based on instructions from the user, or the control device 3 can generate the local model TM3_P without using instructions from the user.

[0262] Furthermore, the position and pose calculation in this embodiment is a calculation used to generate position and pose data (POI). That is, the position and pose calculation in this embodiment is a calculation used to calculate at least one of the position and pose of the object OBJ. As described above, when generating position and pose data (POI) by performing 3D matching processing, the position and pose calculation includes 3D matching processing. As described above, when generating position and pose data (POI) by performing 2D matching processing, the position and pose calculation includes 2D matching processing.

[0263] In this embodiment, an example is shown where the result of the position and attitude calculation (position and attitude calculation processing) is good (a good calculation result condition). In this case, the control device 3 can determine whether the calculation result condition (a good calculation result condition) is met by determining whether the result of the position and attitude calculation is good. If the result of the position and attitude calculation is good, the signal generation unit 313 can determine that the calculation result condition (a good calculation result condition) is met. If the result of the position and attitude calculation is bad (i.e., the result of the position and attitude calculation is poor), the signal generation unit 313 can determine that the calculation result condition (a good calculation result condition) is not met.

[0264] The control device 3 can use the result of the position and attitude calculation to determine whether there is an object OBJ that the end effector 4 can process according to a specified procedure, thereby determining whether the result of the position and attitude calculation is good. Alternatively, the signal generation unit 313 can use the result of the position and attitude calculation to determine whether there is an object OBJ that the end effector 4 cannot process according to a specified procedure, or as an alternative, to determine whether the result of the position and attitude calculation is good. The reason for this is explained below.

[0265] As described above, since the robot control signal is generated using the result of position and orientation calculations (i.e., information including the result of matching processing, such as information including position and orientation data POI), it is conceivable that the control device 3 cannot generate an appropriate robot control signal if the result of the position and orientation calculations is poor. Consequently, since an appropriate robot control signal cannot be generated, it is conceivable that the end effector 4 may be unable to perform the prescribed processing on the object OBJ. That is, it is conceivable that there may be object OBJs that cannot be processed by the end effector 4 using the result of position and orientation calculations. Therefore, if the end effector 4 cannot perform the prescribed processing on the object OBJ, the result of the position and orientation calculations may be poor. Therefore, the existence of object OBJs that cannot be processed by the end effector 4 using the result of position and orientation calculations can be used as one of the reasons for determining that the result of the position and orientation calculations is poor. On the other hand, if the result of the position and orientation calculations is good, it is conceivable that the control device 3 can generate an appropriate robot control signal. Consequently, since an appropriate robot control signal can be generated, it is conceivable that the end effector 4 is more likely to perform the prescribed processing on the object OBJ. That is, it can be assumed that there is a high probability that there exists an object OBJ whose position and orientation calculation results can be processed by the end effector 4 according to specified procedures. Therefore, if the end effector 4 can process the object OBJ according to specified procedures, the position and orientation calculation results are likely to be good. Therefore, the existence of an object OBJ whose position and orientation calculation results can be processed by the end effector 4 according to specified procedures can be used as one of the reasons for determining that the position and orientation calculation results are good.

[0266] As a first example, if the result of the position and attitude calculation indicates that there is at least one object OBJ that can be processed according to the specified procedure by the end effector 4, the control device 3 can determine that the result of the position and attitude calculation is good. On the other hand, if the result of the position and attitude calculation indicates that there is no object OBJ that can be processed according to the specified procedure by the end effector 4, the control device 3 can determine that the result of the position and attitude calculation is bad.

[0267] As a second example, if the position and attitude calculation results in the presence of at least one object OBJ that cannot be processed by the end effector 4 according to the specified procedure, the control device 3 can determine that the position and attitude calculation result is poor. More specifically, if the position and attitude calculation results in the number of object OBJs that cannot be processed by the end effector 4 according to the specified procedure being above a specified detection threshold, the control device 3 can determine that the position and attitude calculation result is poor. In other words, if the position and attitude calculation results in the number of object OBJs that can be processed by the end effector 4 according to the specified procedure being below a specified detection threshold, the control device 3 can determine that the position and attitude calculation result is poor. On the other hand, if the position and attitude calculation results in the number of object OBJs that cannot be processed by the end effector 4 according to the specified procedure being below a specified detection threshold, the control device 3 can determine that the position and attitude calculation result is good. In other words, if the position and attitude calculation results in the number of object OBJs that can be processed by the end effector 4 according to the specified procedure being above a specified detection threshold, the control device 3 can determine that the position and attitude calculation result is good.

[0268] Furthermore, the specified detection threshold used in the second example can be set by the user. Alternatively, the specified detection threshold used in the second example can be automatically set by the control device 3. Moreover, the first example described above can be considered equivalent to the second example where the specified detection threshold is set to "1".

[0269] As a first example of an object OBJ that the end effector 4 can perform the prescribed processing on, an example of an object OBJ whose position and attitude data POI can be calculated through position and attitude calculation can be given. On the other hand, as a first example of an object OBJ that the end effector 4 cannot perform the prescribed processing on, an example of an object OBJ whose position and attitude data POI cannot be calculated through position and attitude calculation can be given. In this case, the control device 3 can determine whether the position and attitude data POI of the object OBJ can be calculated through position and attitude calculation, and thus determine whether the calculation result condition is met.

[0270] As a second example of an object OBJ that the end effector 4 can process according to regulations, an example is an object OBJ whose matching similarity calculated by position and attitude calculation exceeds the matching determination threshold. On the other hand, as a second example of an object OBJ that the end effector 4 cannot process according to regulations, an example is an object OBJ whose matching similarity calculated by position and attitude calculation is lower than the matching determination threshold. In this case, the control device 3 can determine whether an object OBJ with a matching similarity exceeding the matching determination threshold is detected by position and attitude calculation, thereby determining whether the calculation result condition is met. The control device 3 can determine whether the position and attitude data POI of an object OBJ with a matching similarity exceeding the matching determination threshold is calculated by position and attitude calculation, thereby determining whether the calculation result condition is met.

[0271] Furthermore, it can be assumed that the higher the matching similarity of the object OBJ, the higher the accuracy of the object OBJ's position and pose data POI (i.e., the calculation accuracy of at least one of position and pose). Therefore, the position and pose data POI of the object OBJ with a matching similarity exceeding the matching determination threshold can be considered equivalent to its relatively high accuracy (specifically, exceeding the specified accuracy threshold). On the other hand, the position and pose data POI of the object OBJ with a matching similarity below the matching determination threshold can be considered equivalent to its relatively low accuracy (specifically, below the specified accuracy threshold). In this case, the matching determination threshold can be set as an expected value that can distinguish the following states based on the matching similarity: a state where the accuracy (in other words, reliability) of the object OBJ's position and pose data POI is high enough to enable the end effector 4 to successfully perform the specified processing of the object OBJ; and a state where the accuracy (in other words, reliability) of the object OBJ's position and pose data POI is low enough to cause the end effector 4 to fail in performing the specified processing of the object OBJ.

[0272] Furthermore, if the control device 3 (especially the position and attitude calculation unit 312) is unable to perform position and attitude calculations normally, the probability of the position and attitude calculation result being good is low. Therefore, the control device 3 can determine whether the position and attitude calculation result is good by judging whether the position and attitude calculation unit 312 can perform position and attitude calculations normally. Specifically, if the position and attitude calculation unit 312 can perform position and attitude calculations normally, the control device 3 can determine that the position and attitude calculation result is good. On the other hand, if the position and attitude calculation unit 312 cannot perform position and attitude calculations normally, the control device 3 can determine that the position and attitude calculation result is not good. The state in which the position and attitude calculation unit 312 cannot perform position and attitude calculations normally can include the state in which the position and attitude calculation unit 312 cannot start position and attitude calculations normally. The state in which the position and attitude calculation unit 312 cannot perform position and attitude calculations normally can include the state in which the position and attitude calculation started by the position and attitude calculation unit 312 ends abnormally.

[0273] Furthermore, if the operation result condition is determined to be false, the end effector 4 will not perform the specified processing on the object OBJ. Therefore, determining whether the operation result condition is true or false can include determining whether the end effector 4 performs the specified processing on the object OBJ. Determining that the operation result condition is false can include determining that the end effector 4 does not perform the specified processing on the object OBJ.

[0274] In this embodiment, specifically, when the three-dimensional position data WSD represents the three-dimensional positions of multiple object objects OBJ (especially multiple object objects OBJ that at least partially overlap), the control device 3 can perform 3D matching processing using the local model TM3_P. In this case, compared to the case where only 3D matching processing using the global model TM3_W is performed, the control device 3 is more likely to appropriately detect (identify) an object object OBJ for which the end effector 4 should perform the prescribed processing as the processing execution object. That is, when multiple object objects OBJ (especially multiple object objects OBJ that at least partially overlap) are captured in an image shown by image data IMG_3D, the control device 3 is more likely to appropriately detect (identify) an object object OBJ for which the end effector 4 should perform the prescribed processing as the processing execution object.

[0275] As an example, Figure 13 (a) shows a component with a U-shape as an example of an object OBJ. Figure 13In the example shown in (a), the object OBJ comprises a plate-shaped first portion OB1 and a plate-shaped second portion OB2 that is opposite to and larger than the first portion OB1. In this case, as Figure 13 As shown in (b), when using the overall model TM3_W representing both the three-dimensional shape of part OB1 and the three-dimensional shape of part OB2, the following technical problems may arise. Specifically, as Figure 13 (c) illustrates the following example: a portion of the second part OB2 of the first object OBJ#1 is partially hidden by a portion of the second object OBJ#2 located below the first object OBJ#1. Furthermore, when the object OBJ has a U-shape, by interlocking the first object OBJ#1 and the second object OBJ#2, a portion of the first object OBJ#1 may be hidden by a portion of the second object OBJ#2 located below the first object OBJ#1. In this case, the position and pose calculation unit 312 may not detect the edge of the second part OB2 of the first object OBJ#1 partially hidden by the second object OBJ#2 (especially its first part OB1), but instead detects the edge E_OB1 of the first part OB1 of the first object OBJ#1 located above the second object OBJ#2 (especially its first part OB1). Figure 13 (c) thick line). However, due to the detected edge E_OB1 and the edge E_TM3 of the overall model TM3_W (refer to Figure 13 (b) The similarity between the thick lines is low, and the position and pose calculation unit 312 may not be able to properly detect the first object OBJ#1 (i.e., it may not be able to properly calculate at least one of the position and pose of the first object OBJ#1). In this case, such as Figure 13 As shown in (d), when the model portion corresponding to the first part OB1 in the template model TM3 is used as the local model TM3_P, due to the detected edge E_OB1 and the edge E_TM3' of the local model TM3_P (refer to...), Figure 13 (d) The similarity between the thick lines is high, and the position and pose calculation unit 312 can properly detect the first object OBJ#1 (that is, it can properly calculate at least one of the position and pose of the first object OBJ#1).

[0276] Therefore, in Modified Example 1, the position and attitude calculation unit 312 is more likely to be able to properly detect the object OBJ in 3D matching processing.

[0277] In this scenario, the likelihood of a second object object OBJ#2, which is different from the first object object OBJ#1 that should be selected as the object to be processed, being unintentionally selected as the object to be processed becomes lower. If the second object object OBJ#2, which should not be selected as the object to be processed, is unintentionally selected as the object to be processed, then as the end effector 4 approaches the second object object OBJ#2, which should not be selected as the object to be processed, the end effector 4 may collide with the first object object OBJ#1 that should have been selected as the object to be processed (e.g., the first object object OBJ#1 located on the second object object OBJ#2), causing damage to the first object object OBJ#1. Alternatively, after the end effector 4 holds the second object object OBJ#2, which should not be selected as the object to be processed, the first object object OBJ#1 located on the second object object OBJ#2 may be dragged along by the second object object OBJ#2. As a result, the first object OBJ#1 held by the end effector 4 falls, potentially damaging surrounding object OBJs. However, Modification 1 is advantageous in reducing the likelihood of this technical problem occurring.

[0278] For reference Figure 12 As explained, when generating multiple local models TM3_P, the control device 3 can select from the multiple local models TM3_P that have a higher probability of properly detecting (identifying) an object OBJ that the end effector 4 should process according to the overlap tendency of the multiple object OBJs, and then perform 3D matching processing using the selected local model TM3_P. As a result, compared with the case of using a single local model TM3_P, the control device 3 has a higher probability of properly detecting (identifying) an object OBJ that the end effector 4 should process according to the specified procedure.

[0279] Furthermore, since the local model TM3_P is part of the overall model TM3_W, the data size of the local model TM3_P is smaller than the data size of the overall model TM3_W. Therefore, in Modification 1, the control device 3, by performing 3D matching processing using the local model TM3_P, can shorten the time required for 3D matching processing compared to the case where 3D matching processing using only the overall model TM3_W is not performed.

[0280] Furthermore, in the case of performing 2D matching processing using a template image as a template, similar to the case of performing 3D matching processing, the position and attitude calculation unit 312 may perform 2D matching processing using a local model equivalent to a portion of the template image as a template, or alternatively, in addition to performing 2D matching processing using an overall model equivalent to the entire template image as a template. Furthermore, when the local model used for 2D matching processing is generated based on the aforementioned simulated image data, the simulated image data may be, for example, image data in the same form as the image data IMG_2D (e.g., monocular image data). Furthermore, the method for generating the local model used for 2D matching processing can be the same as the method for generating the local model TM3_P used for the aforementioned 3D matching processing, therefore its detailed description is omitted. Furthermore, 2D matching processing using a local model can be the same as 3D matching processing using the aforementioned local model TM3_P, therefore its detailed description is omitted. Furthermore, the local model equivalent to a portion of the template image can be referred to as a two-dimensional local model.

[0281] Even when using a two-dimensional local model in 2D matching processing, similar to using a local model TM3_P in 3D matching processing, multiple two-dimensional local models can be used. In this case, the control device 3 can generate a two-dimensional local model obtained by removing the image portion corresponding to the curved portion (corner portion) from the object captured in the template image corresponding to the overall model. Alternatively, the control device 3 can identify the curved portion (corner portion) and other portions (e.g., planar portions) in the overall model TM3_W, remove the curved portion (corner portion) from the overall model TM3_W to generate a local model TM3_P, and generate a two-dimensional local model obtained by projecting the local model TM3_P after removing the curved portion onto a two-dimensional plane. In this case, as described above, the control device 3 can generate the two-dimensional local model based on instructions from the user. As another example, the control device 3 can generate a two-dimensional local model by identifying curved portions (corner portions) and other portions (e.g., planar portions) of an object captured in a template image equivalent to the overall model, without using instructions from the user, and by removing the image portions corresponding to the curved portions (corner portions) from the template image equivalent to the overall model. Furthermore, the control device 3 can generate a model with at least a thinned portion of the object captured in the template image equivalent to the overall model as a local model TM3_P, and generate a two-dimensional local model obtained by projecting the local model TM3_P, which is equivalent to the thinned model, onto a two-dimensional plane.

[0282] (4-2) Variation Example 2

[0283] Next, the robot system SYS in Modification 2 will be described. Furthermore, in the following description, the robot system SYS in Modification 2 will be referred to as "robot system SYSb".

[0284] (4-2-1) Structure of the robot system SYSb in Variation Example 2

[0285] The robot system SYSb in Variation Example 2 differs from the robot system SYS described above in that it has a control device 3b instead of the control device 3. Other features of the robot system SYSb may be the same as those of the robot system SYS. Therefore, refer to the following... Figure 14 To illustrate the control device 3b in variant example 2. Figure 14 This is a block diagram showing the structure of the control device 3b in variant example 2.

[0286] like Figure 14 As shown, the control device 3b in Modification 2 differs from the control device 3 described above in that the arithmetic unit 31 includes a movement control unit 314b as a logic processing module. Other features of the control device 3b may be the same as those of the control device 3.

[0287] The motion control unit 314b can determine whether the calculation result condition described in Modification 1 (i.e., the result of the position and attitude calculation including matching processing is good) is met. Position and attitude calculation uses... Figure 4 The process involves using at least one of the image data IMG_2D and IMG_3D obtained in step S1. Therefore, determining whether the condition for the calculation result is true can be considered equivalent to determining whether the condition is based on... Figure 4 Whether the position and pose calculation result of at least one of the image data IMG_2D and IMG_3D obtained in step S1 is good.

[0288] Furthermore, if the calculation result is determined to be invalid, the movement control unit 314b moves the imaging system 2 to change the positional relationship between the imaging system 2 and the object OBJ. In addition, in Modification 2, the positional relationship between the imaging system 2 and the object OBJ can refer to the positional relationship between at least one of the imaging devices 21 and 22 and the object OBJ. In Modification 2, moving the imaging system 2 can mean moving at least one of the imaging devices 21 and 22.

[0289] Specifically, if the positional relationship between the imaging system 2 and the object OBJ is determined to be a first positional relationship, indicating that the end effector 4 cannot perform the prescribed processing on the object OBJ, the movement control unit 314b moves the imaging system 2 to change the positional relationship between the imaging system 2 and the object OBJ from the first positional relationship to a second positional relationship. That is, if at least one of the image data IMG_2D and IMG_3D acquired based on the first positional relationship between the imaging system 2 and the object OBJ is determined to indicate that the end effector 4 cannot perform the prescribed processing on the object OBJ, the movement control unit 314b moves the imaging system 2 to change the positional relationship between the imaging system 2 and the object OBJ from the first positional relationship to the second positional relationship. In other words, if the end effector 4 is determined to be unable to perform the specified processing on the object OBJ based on the result of matching processing of at least one of the image data IMG_2D and IMG_3D acquired when the positional relationship between the shooting system 2 and the object OBJ is the first positional relationship, the movement control unit 314b moves the shooting system 2 so that the positional relationship between the shooting system 2 and the object OBJ changes from the first positional relationship to the second positional relationship.

[0290] The positional relationship between the imaging system 2 and the object OBJ can include the relationship between the position of the imaging system 2 and the position of the object OBJ. Furthermore, in Modification 2, the position of the imaging system 2 can refer to the position of at least one of the imaging devices 21 and 22. Therefore, in Modification 2, the positional relationship between the imaging system 2 and the object OBJ can include the relationship between the position of at least one of the imaging devices 21 and 22 and the position of the object OBJ. In this case, the movement control unit 314b can change the position of the imaging system 2 by moving it, thereby changing the positional relationship between the imaging system 2 and the object OBJ from a first positional relationship to a second positional relationship. Specifically, the movement control unit 314b can move the imaging system 2 so that the position of the imaging system 2 changes from a first position corresponding to the first positional relationship to a second position corresponding to the second positional relationship, thereby changing the positional relationship between the imaging system 2 and the object OBJ from a first positional relationship to a second position.

[0291] The positional relationship between the imaging system 2 and the object OBJ can include the relationship between the posture of the imaging system 2 and the posture of the object OBJ. Furthermore, in Modification 2, the posture of the imaging system 2 can refer to the posture of at least one of the imaging devices 21 and 22. Therefore, in Modification 2, the positional relationship between the imaging system 2 and the object OBJ can include the relationship between the posture of at least one of the imaging devices 21 and 22 and the posture of the object OBJ. In this case, the movement control unit 314b can change the posture of the imaging system 2 by moving the imaging system 2, thereby changing the positional relationship between the imaging system 2 and the object OBJ from a first positional relationship to a second positional relationship. Specifically, the movement control unit 314b can move the imaging system 2 so that the posture of the imaging system 2 changes from a first posture corresponding to the first positional relationship to a second posture corresponding to the second positional relationship, thereby changing the positional relationship between the imaging system 2 and the object OBJ from a first positional relationship to a second positional relationship. Furthermore, when the positional relationship between the imaging system 2 and the object OBJ represents the relationship between the posture of at least one of the imaging devices 21 and 22 and the posture of the object OBJ, the positional relationship between the imaging system 2 and the object OBJ can be referred to as the posture relationship between the imaging system 2 and the object OBJ. In this case, the movement control unit 314b can move the imaging system 2 so that the posture of the imaging system 2 changes from a first posture corresponding to a first posture relationship to a second posture corresponding to a second posture relationship, thereby changing the posture relationship between the imaging system 2 and the object OBJ from the first posture relationship to the second posture relationship.

[0292] Furthermore, the processing performed by the motion control unit 314b can be considered to include the following actions: moving the imaging system 2 with respect to an object OBJ, so as to cause at least one change in the position and orientation of the imaging system 2. Furthermore, the processing performed by the motion control unit 314b can be said to include the following actions: moving the imaging system 2, so as to cause at least one change in the position and orientation of the imaging system 2 around an object OBJ.

[0293] To move the imaging system 2, the motion control unit 314b generates a robot control signal to control the robotic arm 12, thereby changing the positional relationship between the imaging system 2 and the object OBJ by moving the imaging system 2. Specifically, the motion control unit 314b generates a robot control signal to control the robotic arm 12, thereby changing the positional relationship between the imaging system 2 and the object OBJ from a first positional relationship to a second positional relationship by moving the imaging system 2.

[0294] Subsequently, after the robotic arm 12 moves the imaging system 2 based on the robot control signal generated by the motion control unit 314b, the control device 3b performs position and pose calculations again to generate position and pose data POI. That is, after the robotic arm 12 moves the imaging system 2, the imaging system 2 re-images the object OBJ, thereby generating at least one of image data IMG_2D and IMG_3D. The control device 3b uses at least one of the image data IMG_2D and IMG_3D regenerated by the imaging system 2 to generate position and pose data POI. In other words, the control device 3b uses at least one of the image data IMG_2D and IMG_3D regenerated by the imaging system 2 when the positional relationship between the imaging system 2 and the object OBJ becomes a second positional relationship to generate position and pose data POI.

[0295] Here, because the imaging system 2 moves, it captures the object OBJ from a position different from its original position before the movement. Alternatively, it captures the object OBJ from an orientation different from its original orientation before the movement. Therefore, the method of capturing the object OBJ in the images shown by image data IMG_2D and IMG_3D changes. As a result, even if it is determined that at least one of the image data IMG_2D and IMG_3D acquired before the movement of the imaging system 2 cannot be used to perform specified processing on the object OBJ by the end effector 4, it is still possible to perform specified processing on the object OBJ based on at least one of the image data IMG_2D and IMG_3D acquired after the movement of the imaging system 2.

[0296] As an example, as described above, if the position and attitude calculation result is poor (i.e., unsatisfactory), it can be determined that the end effector 4 cannot perform the prescribed processing on the object OBJ. In this case, the method of capturing the object OBJ in the images shown by image data IMG_2D and IMG_3D respectively changes. Therefore, even if the position and attitude calculation performed before the capturing system 2 moves has a poor result, the result of the position and attitude calculation performed after the capturing system 2 moves may no longer be unsatisfactory (i.e., become good). Therefore, when the capturing system 2 moves due to a poor result of the position and attitude calculation, the probability of the position and attitude calculation result becoming good is higher than the probability of the capturing system 2 not moving even if the position and attitude calculation result is poor. As a result, the position and attitude calculation unit 312 can properly calculate at least one of the position and attitude of the object OBJ. As a result, the probability of determining that the calculation result condition is met is higher.

[0297] As described above, as another example, if the position and attitude calculation unit 312 fails to perform position and attitude calculations normally, it can also be determined that the calculation result condition is not met. In this case, since the method of capturing the object OBJ in the images shown by the image data IMG_2D and IMG_3D respectively changes, even if the position and attitude calculation unit 312 failed to perform position and attitude calculations normally before the shooting system 2 moved, it is possible that the position and attitude calculation unit 312 can perform position and attitude calculations normally after the shooting system 2 moves. Therefore, when the shooting system 2 moves because the position and attitude calculation unit 312 fails to perform position and attitude calculations normally, the probability that the position and attitude calculation unit 312 can perform position and attitude calculations normally is higher than the probability that the shooting system 2 does not move even if the position and attitude calculation unit 312 fails to perform position and attitude calculations normally. That is, the probability that the result of the position and attitude calculation is good is higher. As a result, the position and attitude calculation unit 312 can properly calculate at least one of the position and attitude of the object OBJ. As a result, the probability that the calculation result condition is met is higher.

[0298] Then, after properly calculating at least one of the position and orientation of the object OBJ (i.e., properly generating position and orientation data POI), in Figure 4 In step S4, the control device 3b (especially the signal generation unit 313) generates a robot control signal based on the generated position and attitude data POI, so that the robot 1 (end effector 4) can perform specified processing on the object OBJ.

[0299] Furthermore, the robot control signals generated by the motion control unit 314b are primarily signals used to move the shooting system 2, which can be considered as... Figure 4 The robot control signal generated by the signal generation unit 313 in step S4 (i.e., the robot control signal mainly used to cause the robot 1 (end effector 4) to perform specified processing) is different. In this case, the robot control signal generated by the motion control unit 314b can be referred to as the imaging device motion signal.

[0300] However, the robot control signal generated by the motion control unit 314b is the signal that controls robot 1, and in this respect it can be considered as... Figure 4 In step S4, the robot control signal generated by the signal generation unit 313 is the same. That is, the motion control unit 314b and the signal generation unit 313 can be common in that they both generate robot control signals. In this case, the signal generation unit 313 can perform the actions performed by the motion control unit 314b. In other words, the signal generation unit 313 can function as the motion control unit 314b. In this case, the control device 3b may not need to have a motion control unit 314b.

[0301] Furthermore, since the robotic arm 12 is equipped with both shooting devices 21 and 22, it can, for example, move both shooting devices 21 and 22. Therefore, the motion control unit 314b generates, for example, a robot control signal to control the robotic arm 12, which enables the movement of both shooting devices 21 and 22, so as to change the positional relationship between the shooting devices 21 and 22 and the object OBJ by moving them. In the following description, the following example will also be described: that is, the motion control unit 314b generates a robot control signal to control the robotic arm 12, so as to change the positional relationship between the shooting devices 21 and 22 and the object OBJ by moving them. However, depending on the configuration of the shooting devices 21 and 22, the motion control unit 314b can also generate a robot control signal for controlling the robotic arm 12 that can move either of the shooting devices 21 and 22, so that the positional relationship between either of the shooting devices 21 and 22 and the object OBJ can be changed by moving either of the shooting devices 21 and 22.

[0302] (4-2-2) The robot control process in variation example 2

[0303] Next, refer to Figure 15 This will illustrate the robot control processing performed by the control device 3b in Modified Example 2. Figure 15 This is a flowchart illustrating the robot control process performed by the control device 3b in Modified Example 2.

[0304] like Figure 15 As shown, in Modification 2, the control device 3b also uses the communication device 33 to acquire at least one of the image data IMG_2D and IMG_3D from the imaging system 2 (step S1). Furthermore, whenever the control device 3b acquires the image data IMG_3D, the three-dimensional position data generation unit 311 of the control device 3b generates three-dimensional position data WSD based on the acquired image data IMG_3D (step S2). Afterwards, the position and attitude calculation unit 312 of the control device 3b performs a position and attitude calculation (i.e., a calculation including matching processing) based on at least one of the image data IMG_2D acquired in step S1 and the three-dimensional position data WSD generated in step S2 (step S3). In other words, the position and attitude calculation unit 312 performs a position and attitude calculation to generate position and attitude data POI representing at least one of the position and attitude of the object OBJ based on at least one of the image data IMG_2D acquired in step S1 and the three-dimensional position data WSD generated in step S2 (step S3). Furthermore, in Modification 2, compared with the use of... Figure 4 Similarly, for the sake of simplicity, the above statement will be used to... Figure 15An example of the 3D matching processing performed by the position and attitude calculation unit 312 in step S3 (i.e., an example where the control device 3b acquires image data IMG_3D in step S1 and performs 3D matching processing in step S3) will be described. However, in modified example 2, in Figure 15 In step S3, the position and attitude calculation unit 312 can perform 2D matching processing, or both 2D matching processing and 3D matching processing.

[0305] Subsequently, in Modification 2, the movement control unit 314b of the control device 3b determines whether the calculation result condition is met (step S61b). Specifically, the movement control unit 314b determines whether the calculation result condition is met by judging whether the result of the position and pose calculation based on at least one of the image data IMG_2D and IMG_3D obtained in step S1 is good. That is, the movement control unit 314b determines whether the calculation result condition is met by judging whether the result of the position and pose calculation based on at least one of the image data IMG_2D and IMG_3D obtained in step S1 is good. Figure 4 The result of the position and attitude calculation in step S3 is used to determine whether the calculation result condition is met.

[0306] Furthermore, the details of the "conditions for the result of the operation" have already been explained in Variation Example 1, so their detailed explanation is omitted in Variation Example 2.

[0307] If the determination result in step S61b is that the calculation result condition is met (step S61b: Yes), the signal generation unit 313 uses the position and orientation data POI generated in step S3 to generate a robot control signal (step S4). That is, the signal generation unit 313 uses the position and orientation data POI generated in step S3 to generate a robot control signal for controlling the robot 1 so that the end effector 4 approaches the object OBJ (step S4).

[0308] On the other hand, if the determination result in step S61b is that the calculation result condition is not met (step S61b: Yes), the signal generation unit 313 may not use the position and attitude data POI generated in step S3 to generate the robot control signal. Alternatively, if the position and attitude calculation unit 312 is unable to perform position and attitude calculation normally in step S3, it may not generate the position and attitude data POI. Therefore, the signal generation unit 313 cannot generate the robot control signal. In this case, the motion control unit 314b moves the imaging system 2 to change the positional relationship between the imaging system 2 and the object OBJ (steps S62b to S64b).

[0309] Specifically, the motion control unit 314b acquires image data IMG_2D from the imaging system 2 (step S62b). However, as will be explained later, in step S62b, in addition to or as a substitute for image data IMG_2D, the motion control unit 314b may also acquire image data IMG_3D from the imaging system 2.

[0310] For example, after determining in step S61b that the calculation result condition is not met, the imaging system 2 can generate the image data IMG_2D acquired by the motion control unit 314b in step S62b by photographing the object OBJ. Furthermore, the imaging system 2 can re-photograph the object OBJ from the same direction as when it photographed the object OBJ to generate the image data IMG_3D acquired in step S1, thereby generating the image data IMG_2D acquired by the motion control unit 314b in step S62b. In this case, the motion control unit 314b can acquire the image data IMG_2D generated by the imaging system 2 after determining that the calculation result condition is not met.

[0311] For example, before determining in step S61b that the calculation result condition is not met, the imaging system 2 can generate the image data IMG_2D acquired by the motion control unit 314b in step S62b by photographing the object OBJ. Furthermore, the imaging system 2 can re-photograph the object OBJ from the same direction as when it photographed the object OBJ to generate the image data IMG_3D acquired in step S1, thereby generating the image data IMG_2D acquired by the motion control unit 314b in step S62b. In this case, the motion control unit 314b can acquire the image data IMG_2D generated by the imaging system 2 before determining that the calculation result condition is not met. As an example, after the imaging system 2 photographs the object OBJ in step S1 to generate at least one of the image data IMG_2D and IMG_3D for position and attitude calculation, and before determining in step S61b that the calculation result condition is not met, the imaging system 2 can photograph the object OBJ to generate the image data IMG_2D acquired by the motion control unit 314b in step S62b. As another example, the imaging system 2 can generate image data IMG_3D for position and attitude calculation in step S1, and simultaneously generate image data IMG_2D acquired by the motion control unit 314b in step S62b. That is, the imaging system 2 can generate image data IMG_3D for position and attitude calculation and image data IMG_2D acquired by the motion control unit 314b in step S62b at the same timing.

[0312] Furthermore, as described above, in variation example 2, it is illustrated that... Figure 15The example of 3D matching processing performed by the position and attitude calculation unit 312 in step S3, but assuming that in Figure 15 In step S3, when the position and attitude calculation unit 312 performs 2D matching processing, the imaging system 2 generates image data IMG_2D for position and attitude calculation in step S1. In this case, the motion control unit 314b can acquire the image data IMG_2D generated in step S1 in step S62b. That is, the image data IMG_2D used for position and attitude calculation in step S3 can be the same image data IMG_2D acquired by the motion control unit 314b in step S62b. Alternatively, besides using the image data IMG_2D or as a replacement, it can also use the image data obtained through... Figure 15 The position and attitude data POI generated by the position and attitude calculation in step S3 is used to... Figure 15 In step S63b, the movement mode of the shooting system 2 is set by the movement control unit 314b.

[0313] In any case, the image data IMG_2D acquired by the motion control unit 314b in step S62b is the image data IMG_2D generated by the shooting system 2 capturing the object OBJ before the motion control unit 314b changes the positional relationship between the shooting system 2 and the object OBJ. Specifically, as described above, if the calculation result condition is determined to be invalid when the positional relationship between the shooting system 2 and the object OBJ is the first positional relationship, the motion control unit 314b moves the shooting system 2 so that the positional relationship between the shooting system 2 and the object OBJ changes from the first positional relationship to the second positional relationship. In this case, the image data IMG_2D acquired by the motion control unit 314b in step S62b can be the image data IMG_2D generated by the shooting system 2 capturing the object OBJ when the positional relationship between the shooting system 2 and the object OBJ is the first positional relationship. In other words, the image data IMG_2D acquired by the motion control unit 314b in step S62b can be image data IMG_2D generated by the imaging system 2 capturing the object OBJ when the positional relationship between the imaging system 2 and the object OBJ is different from the second positional relationship. However, the image data IMG_2D acquired by the motion control unit 314b in step S62b can also be image data IMG_2D generated by the imaging system 2 capturing the object OBJ when the positional relationship between the imaging system 2 and the object OBJ is different from the first positional relationship.

[0314] Subsequently, the motion control unit 314b sets the movement mode of the imaging system 2 relative to an object (e.g., the observation object OBJ_g, described later) based on the image data IMG_2D obtained in step S62b (step S63b). Furthermore, the movement mode of the imaging system 2 can refer to the movement pattern or method of the imaging system 2 relative to an object (e.g., the observation object OBJ_g, described later). Additionally, the movement mode of the imaging system 2 can refer to a movement pattern or method indicating how the imaging system 2 moves relative to an object (e.g., the observation object OBJ_g, described later). Furthermore, the movement mode can also be considered as the movement pattern or method of the imaging system 2 based on an object (e.g., the observation object OBJ_g, described later). Furthermore, the movement mode can also be said to refer to the movement pattern or method of the imaging system 2 around an object (e.g., the observation object OBJ_g, described later).

[0315] Specifically, the motion control unit 314b can automatically set the movement mode of the shooting system 2 based on the image data IMG_2D acquired in step S62b (step S63b). For example, the motion control unit 314b can perform 2D matching processing using the image data IMG_2D to calculate at least one of the position and orientation of the object OBJ. That is, the motion control unit 314b can perform 2D matching processing using the image data IMG_2D to generate position and orientation data POI representing at least one of the position and orientation of the object OBJ. Specifically, the position and orientation calculation unit 312 can generate data representing the position and orientation of the object OBJ. Figure 15 The position and orientation data POI of at least one of the position and orientation of an object OBJ obtained in the latest image data IMG_3D (or image data IMG_2D) in step S1, but determined by position and orientation calculation to be unsuitable for the end effector 4 to perform the specified processing. Furthermore, as explained below, this includes the following action: moving the imaging system 2 so that at least one of the position and orientation of the object OBJ relative to the position and orientation data POI calculated in step S63b changes. Therefore, in the following description, the object OBJ for which the position and orientation data POI is calculated in step S63b is referred to as "observation object OBJ_g". Furthermore, the 2D matching process in step S63b can be the same as the 2D matching process in step S3 described above, so its detailed explanation is omitted. Afterwards, the motion control unit 314b can set the movement mode of the imaging system 2 based on the generated position and orientation data POI.

[0316] Furthermore, as described above, when image data IMG_3D is acquired in step S62b, the motion control unit 314b can set the movement mode of the shooting system 2 based on the image data IMG_3D in step S63b. For example, the motion control unit 314b can perform 3D matching processing using three-dimensional position data WSD generated from the image data IMG_3D, thereby generating position and pose data POI representing at least one of the position and pose of the object OBJ. The 3D matching processing in step S63b can be the same as the 3D matching processing in step S3 described above, so its detailed description is omitted. Afterwards, the motion control unit 314b can set the movement mode of the shooting system 2 based on the generated position and pose data POI. Alternatively, the motion control unit 314b can set the movement mode of the shooting system 2 based on both image data IMG_2D and IMG_3D in step S63b. For example, the motion control unit 314b can perform both 2D matching processing using image data IMG_2D and 3D matching processing using three-dimensional position data WSD generated from image data IMG_3D, thereby generating position and pose data POI representing at least one of the position and pose of the object OBJ. The processing of both 2D and 3D matching in step S63b can be the same as that in step S3 described above, so a detailed explanation is omitted. Afterwards, the motion control unit 314b can set the movement mode of the shooting system 2 based on the generated position and pose data POI. In the following description, for the sake of simplicity, the following example will be used: that is, in step S63b, the motion control unit 314b sets the movement mode of the shooting system 2 based on image data IMG_2D.

[0317] The motion control unit 314b can set the movement mode of the shooting system 2 to increase the probability that the calculation result condition is met (i.e., the position and attitude calculation result is good) after the shooting system 2 moves relative to the observed object OBJ_g according to the movement mode set in step S63b. Specifically, the motion control unit 314b can set the movement mode of the shooting system 2 based on position and attitude data POI, which represents at least one of the position and attitude of the observed object OBJ_g, so that the positional relationship between the shooting system 2 and the observed object OBJ_g is such that the probability of the calculation result condition related to the result of the re-performed position and attitude calculation after the shooting system 2 moves according to the movement mode set in step S63b is met is increased. The motion control unit 314b can set the movement mode of the shooting system 2 based on position and attitude data POI, which represents at least one of the position and attitude of the observed object OBJ_g, so that the positional relationship between the shooting system 2 and the observed object OBJ_g is such that "the probability that the calculation result condition related to the result of the position and attitude calculation performed again after the shooting system 2 moves according to the movement mode set in step S63b is determined to be true becomes higher than the probability that the calculation result condition related to the result of the position and attitude calculation performed before the shooting system 2 moves according to the movement mode set in step S63b is determined to be true".

[0318] If the calculation result condition is met (i.e., the position and attitude calculation result is good), then it can be expected that the end effector 4 can perform the prescribed processing on the observed object OBJ_g. Therefore, the motion control unit 314b can set the movement mode of the shooting system 2 so that the probability of the end effector 4 being able to perform the prescribed processing on the observed object OBJ_g is increased after the shooting system 2 moves relative to the observed object OBJ_g according to the movement mode set in step S63b. Specifically, the motion control unit 314b can set the movement mode of the shooting system 2 based on the position and attitude data POI representing at least one of the position and attitude of the observed object OBJ_g, so that the positional relationship between the shooting system 2 and the observed object OBJ_g is such that the probability of the end effector 4 being able to perform the prescribed processing on the observed object OBJ_g is increased after the shooting system 2 moves relative to the observed object OBJ_g according to the movement mode set in step S63b is increased. The motion control unit 314b can set the movement mode of the shooting system 2 based on position and attitude data POI, which represents at least one of the position and attitude of the observed object OBJ_g, so that the positional relationship between the shooting system 2 and the observed object OBJ_g is such that "the probability that the end effector 4 can perform the prescribed processing on the observed object OBJ_g becomes higher after the shooting system 2 moves according to the movement mode set in step S63b than the probability that the end effector 4 can perform the prescribed processing on the observed object OBJ_g is higher before the shooting system 2 moves relative to the observed object OBJ_g according to the movement mode set in step S63b".

[0319] As an example, in Figure 4In step S3, when position and pose calculations including 3D matching are performed to generate position and pose data (POI), the more points of the object OBJ contained in the point cloud shown by the 3D position data WSD used for 3D matching, the more information the 3D matching process can utilize. Therefore, the more points of the object OBJ contained in the point cloud shown by the 3D position data WSD, the higher the probability of detecting the object OBJ through 3D matching. Furthermore, the more points of the object OBJ contained in the point cloud shown by the 3D position data WSD, the higher the accuracy (reliability) of the position and pose data (POI) generated by the position and pose calculations including 3D matching. Therefore, the motion control unit 314b can set the movement mode of the shooting system 2 to increase the number of points of the observed object OBJ_g contained in the point cloud shown by the 3D position data WSD. Specifically, the motion control unit 314b can set the movement mode of the imaging system 2 so that the number of points of the observed object OBJ_g contained in the point cloud shown in the regenerated 3D position data WSD in step S2 after the imaging system 2 moves is greater than the number of points of the observed object OBJ_g contained in the point cloud shown in the 3D position data WSD generated in step S2 before the imaging system 2 moves. As a result, the probability of obtaining a good result from the position and pose calculation including 3D matching processing is increased.

[0320] The larger the proportion of the object OBJ in the field of view of the imaging system 2 (especially the field of view of the imaging device 22), the higher the likelihood that the number of points of the object OBJ contained in the point cloud shown by the three-dimensional position data WSD will increase. Therefore, the movement control unit 314b can set the movement mode of the imaging system 2 to increase the proportion of the observed object OBJ_g in the field of view of the imaging system 2 (especially the field of view of the imaging device 22). Specifically, the movement control unit 314b can set the movement mode of the imaging system 2 so that the proportion of the observed object OBJ_g in the field of view of the imaging system 2 (especially the field of view of the imaging device 22) after the imaging system 2 has moved is larger than the proportion of the observed object OBJ_g in the field of view of the imaging system 2 (especially the field of view of the imaging device 22) before the imaging system 2 has moved. In other words, the motion control unit 314b can set the movement mode of the shooting system 2 so that after the shooting system 2 moves, the proportion of the observed object OBJ_g in the shooting field of view (especially the shooting field of view of the shooting device 22) of the shooting system 2 used to generate image data IMG_3D becomes larger than the proportion of the observed object OBJ_g in the shooting field of view (especially the shooting field of view of the shooting device 22) of the shooting system 2 used to generate image data IMG_3D before the shooting system 2 moves. As a result, the probability of obtaining a good result from the position and pose calculation including 3D matching processing increases.

[0321] Furthermore, if the proportion of a portion of the object OBJ within the field of view of the shooting system 2 (especially the field of view of the shooting device 22) increases, the proportion of the object OBJ within the field of view of the shooting system 2 may also increase. Therefore, the movement control unit 314b can set the movement mode of the shooting system 2 to increase the proportion of a portion of the observed object OBJ_g within the field of view of the shooting system 2 (especially the field of view of the shooting device 22). Specifically, the movement control unit 314b can set the movement mode of the shooting system 2 so that the proportion of a portion of the observed object OBJ_g within the field of view of the shooting system 2 (especially the field of view of the shooting device 22) after the shooting system 2 has moved is larger than the proportion of a portion of the observed object OBJ_g within the field of view of the shooting system 2 (especially the field of view of the shooting device 22) before the shooting system 2 has moved.

[0322] Furthermore, if the proportion of the outline of the object OBJ in the field of view of the shooting system 2 (especially the field of view of the shooting device 22) increases, the proportion of the object OBJ in the field of view of the shooting system 2 may also increase. Therefore, the movement control unit 314b can set the movement mode of the shooting system 2 to increase the proportion of the outline of the observed object OBJ_g in the field of view of the shooting system 2 (especially the field of view of the shooting device 22). Specifically, the movement control unit 314b can set the movement mode of the shooting system 2 so that the proportion of the outline of the observed object OBJ_g in the field of view of the shooting system 2 (especially the field of view of the shooting device 22) after the shooting system 2 has moved is larger than the proportion of the outline of the observed object OBJ_g in the field of view of the shooting system 2 (especially the field of view of the shooting device 22) before the shooting system 2 has moved. Here, if the entire outline of the object OBJ enters the field of view of the shooting system 2 (especially the field of view of the shooting device 22), the proportion of the object OBJ's outline within the field of view of the shooting system 2 (especially the field of view of the shooting device 22) is at its maximum. Therefore, the movement control unit 314b can set the movement mode of the shooting system 2 so that the entire outline of the observed object OBJ_g enters the field of view of the shooting system 2 (especially the field of view of the shooting device 22).

[0323] Furthermore, the movement control unit 314b can set the movement mode of the shooting system 2 so that the proportion of a portion of the observed object OBJ_g in the shooting field of view of the shooting system 2 (especially the shooting field of view of the shooting device 22) and the proportion of the outline of the observed object OBJ_g in the shooting field of view of the shooting system 2 (especially the shooting field of view of the shooting device 22) are both maximized (or become a suitable ratio). Alternatively, the movement control unit 314b can set the movement mode of the shooting system 2 so that the ratio between the proportion of a portion of the observed object OBJ_g in the shooting field of view of the shooting system 2 (especially the shooting field of view of the shooting device 22) and the proportion of the outline of the observed object OBJ_g in the shooting field of view of the shooting system 2 (especially the shooting field of view of the shooting device 22) becomes a suitable ratio. In other words, the movement control unit 314b can set the movement mode of the shooting system 2 so that the proportion of a part of the object OBJ_g in the shooting field of the shooting system 2 (especially the shooting field of the shooting device 22) is balanced with the proportion of the outline of the object OBJ_g in the shooting field of the shooting system 2 (especially the shooting field of the shooting device 22).

[0324] For example, Figure 16(a) Shows the imaging device 22 before and after it moves under the control of the motion control unit 314b. Figure 16 In the example shown in (a), the imaging device 22, before moving under the control of the motion control unit 314b, is located at the first position P1. On the other hand, in Figure 16In the example shown in (a), the imaging device 22, after being moved under the control of the motion control unit 314b, is located at a second position P2, which is different from the first position P1. In this case, the motion control unit 314b can set the movement mode of the imaging system 2 relative to the object being observed, OBJ_g, so that the number of points of the object being observed, OBJ_g, contained in the point cloud of the three-dimensional position data WSD generated based on the image data IMG_3D generated by the imaging device 22 located at the second position P2 becomes greater than the number of points of the object being observed, OBJ_g, contained in the point cloud of the three-dimensional position data WSD generated based on the image data IMG_3D generated by the imaging device 22 located at the first position P1. The movement control unit 314b can set the movement mode of the shooting system 2 based on the object being observed, OBJ_g, so that the number of points of the object being observed, OBJ_g contained in the point cloud of the three-dimensional position data WSD generated based on the image data IMG_3D generated by the shooting device 22 located at the second position P2 becomes greater than the number of points of the object being observed, OBJ_g contained in the point cloud of the three-dimensional position data WSD generated based on the image data IMG_3D generated by the shooting device 22 located at the first position P1. The motion control unit 314b can set the movement mode of the imaging system 2 around the object OBJ_g, so that the number of points of the object OBJ_g contained in the point cloud of the three-dimensional position data WSD generated based on the image data IMG_3D generated by the imaging device 22 at the second position P2 becomes greater than the number of points of the object OBJ_g contained in the point cloud of the three-dimensional position data WSD generated based on the image data IMG_3D generated by the imaging device 22 at the first position P1. The motion control unit 314b can set the movement mode of the imaging system 2 relative to the object OBJ_g, so that the proportion of the object OBJ_g shown in the field of view of the imaging device 22 at the second position P2 becomes greater than the proportion of the object OBJ_g shown in the field of view of the imaging device 22 at the first position P1. The movement control unit 314b can set the movement mode of the shooting system 2 based on the object being observed, OBJ_g, so that the scale of the object being observed, OBJ_g shown in the field of view of the shooting device 22 at the second position P2 is larger than the scale of the object being observed, OBJ_g shown in the field of view of the shooting device 22 at the first position P1. The movement control unit 314b can also set the movement mode of the shooting system 2 around the object being observed, OBJ_g, so that the scale of the object being observed, OBJ_g shown in the field of view of the shooting device 22 at the second position P2 is larger than the scale of the object being observed, OBJ_g shown in the field of view of the shooting device 22 at the first position P1.For example, the motion control unit 314b can set the second position P2 where the moving shooting device 22 should be located, so that the number of points of the observed object OBJ_g contained in the point cloud of the three-dimensional position data WSD generated based on the image data IMG_3D generated by the shooting device 22 located at the second position P2 becomes more than the number of points of the observed object OBJ_g contained in the point cloud of the three-dimensional position data WSD generated based on the image data IMG_3D generated by the shooting device 22 located at the first position P1, and set the movement mode of the shooting system 2 relative to the observed object OBJ_g so that the shooting device 22 moves from the first position P1 to the second position P2. For example, the motion control unit 314b can set the second position P2 where the moving shooting device 22 should be located, so that the number of points of the observed object OBJ_g contained in the point cloud of the three-dimensional position data WSD generated based on the image data IMG_3D generated by the shooting device 22 located at the second position P2 becomes more than the number of points of the observed object OBJ_g contained in the point cloud of the three-dimensional position data WSD generated based on the image data IMG_3D generated by the shooting device 22 located at the first position P1, and set the movement mode of the shooting system 2 based on the observed object OBJ_g, so that the shooting device 22 moves from the first position P1 to the second position P2. For example, the motion control unit 314b can set the second position P2 where the moving imaging device 22 should be located, so that the number of points of the observed object OBJ_g contained in the point cloud of the three-dimensional position data WSD generated based on the image data IMG_3D generated by the imaging device 22 located at the second position P2 becomes more than the number of points of the observed object OBJ_g contained in the point cloud of the three-dimensional position data WSD generated based on the image data IMG_3D generated by the imaging device 22 located at the first position P1, and set the movement mode of the imaging system 2 around the observed object OBJ_g so that the imaging device 22 moves from the first position P1 to the second position P2. For example, the motion control unit 314b can set the second position P2 where the moving shooting device 22 should be, so that the proportion of the observed object OBJ_g shown in the shooting field of the shooting device 22 at the second position P2 becomes larger than the proportion of the observed object OBJ_g shown in the shooting field of the shooting device 22 at the first position P1, and set the movement mode of the shooting system 2 relative to the observed object OBJ_g so that the shooting device 22 moves from the first position P1 to the second position P2.For example, the motion control unit 314b can set the second position P2 where the moving shooting device 22 should be, so that the proportion of the observed object OBJ_g shown in the shooting field of view of the shooting device 22 at the second position P2 becomes larger than the proportion of the observed object OBJ_g shown in the shooting field of view of the shooting device 22 at the first position P1, and set the movement mode of the shooting system 2 based on the observed object OBJ_g, so that the shooting device 22 moves from the first position P1 to the second position P2. For example, the motion control unit 314b can set the second position P2 where the moving shooting device 22 should be, so that the proportion of the observed object OBJ_g shown in the shooting field of view of the shooting device 22 at the second position P2 becomes larger than the proportion of the observed object OBJ_g shown in the shooting field of view of the shooting device 22 at the first position P1, and set the movement mode of the shooting system 2 around the observed object OBJ_g, so that the shooting device 22 moves from the first position P1 to the second position P2. Furthermore, setting the movement mode of the shooting system 2 relative to the observed object OBJ_g can be considered equivalent to setting the movement mode of the shooting system 2 based on the observed object OBJ_g, and setting the movement mode of the shooting system 2 around the observed object OBJ_g.

[0325] Figure 16 In the example shown in (a), the imaging device 22 located at the first position P1 faces the first surface S1 of the object OBJ_g, which is narrower than the second surface S2 of the object being observed. On the other hand, the imaging device 22 located at the second position P2 faces the second surface S2 of the object OBJ_g, which is wider than the first surface S1 of the object being observed. Therefore, Figure 16 In the example shown in (a), the number of points containing the observed object OBJ_g in the point cloud generated based on the image data IMG_3D generated by the imaging device 22 at position 2 P2 is greater than the number of points containing the observed object OBJ_g in the point cloud generated based on the image data IMG_3D generated by the imaging device 22 at position 1 P1. Furthermore, the proportion of the observed object OBJ_g shown in the field of view of the imaging device 22 at position 2 P2 is greater than the proportion of the observed object OBJ_g shown in the field of view of the imaging device 22 at position 1 P1.

[0326] Furthermore, as described above, in Variation Example 2, it is illustrated that in Figure 15 The example of the position and attitude calculation unit 312 performing 3D matching processing in step S3, but assuming that in Figure 15In step S3, when position and pose calculations including 2D matching are performed to generate position and pose data POI, the more pixels in the image data IMG_2D used for 2D matching that capture at least a portion of the object OBJ, the more information can be utilized through 2D matching. Therefore, the more pixels in the image data IMG_2D that capture at least a portion of the object OBJ, the higher the probability of detecting the object OBJ through 2D matching. Furthermore, the more pixels in the image data IMG_2D that capture at least a portion of the object OBJ, the higher the accuracy (reliability) of the position and pose data POI generated by position and pose calculations including 2D matching. Therefore, the motion control unit 314b can set the movement mode of the shooting system 2 to increase the number of pixels in the image data IMG_2D that capture at least a portion of the observed object OBJ_g. Specifically, the motion control unit 314b can set the movement mode of the imaging system 2 such that the number of pixels in the image data IMG_2D re-acquired in step S1 after the imaging system 2 moves is greater than the number of pixels in the image data IMG_2D already acquired in step S1 before the imaging system 2 moves. As a result, the probability of obtaining a good result from the position and pose calculation including 2D matching processing increases.

[0327] The larger the proportion of the object OBJ in the field of view of the imaging system 2 (especially the field of view of the imaging device 21), the higher the likelihood that the number of pixels in the image shown by the image data IMG_2D that capture at least a portion of the object OBJ will increase. Therefore, the movement control unit 314b can set the movement mode of the imaging system 2 to increase the proportion of the observed object OBJ_g in the field of view of the imaging system 2 (especially the field of view of the imaging device 21). Specifically, the movement control unit 314b can set the movement mode of the imaging system 2 so that the proportion of the observed object OBJ_g in the field of view of the imaging system 2 (especially the field of view of the imaging device 21) after the imaging system 2 has moved is larger than the proportion of the observed object OBJ_g in the field of view of the imaging system 2 (especially the field of view of the imaging device 21) before the imaging system 2 has moved. In other words, the motion control unit 314b can set the movement mode of the shooting system 2 so that after the shooting system 2 moves, the proportion of the observed object OBJ_g in the shooting field of view (especially the shooting field of view of the shooting device 21) of the shooting system 2 used to generate image data IMG_2D becomes larger than the proportion of the observed object OBJ_g in the shooting field of view (especially the shooting field of view of the shooting device 21) of the shooting system 2 used to generate image data IMG_2D before the shooting system 2 moves. As a result, the probability of obtaining a good result from the position and pose calculation including 2D matching processing increases.

[0328] Furthermore, if the proportion of a portion of the object OBJ within the field of view of the shooting system 2 (especially the field of view of the shooting device 21) increases, the proportion of the object OBJ within the field of view of the shooting system 2 may also increase. Therefore, the movement control unit 314b can set the movement mode of the shooting system 2 to increase the proportion of a portion of the observed object OBJ_g within the field of view of the shooting system 2 (especially the field of view of the shooting device 21). Specifically, the movement control unit 314b can set the movement mode of the shooting system 2 so that the proportion of a portion of the observed object OBJ_g within the field of view of the shooting system 2 (especially the field of view of the shooting device 21) after the shooting system 2 has moved is larger than the proportion of a portion of the observed object OBJ_g within the field of view of the shooting system 2 (especially the field of view of the shooting device 21) before the shooting system 2 has moved.

[0329] Furthermore, if the proportion of the outline of the object OBJ in the field of view of the shooting system 2 (especially the field of view of the shooting device 21) increases, the proportion of the object OBJ in the field of view of the shooting system 2 may also increase. Therefore, the movement control unit 314b can set the movement mode of the shooting system 2 to increase the proportion of the outline of the observed object OBJ_g in the field of view of the shooting system 2 (especially the field of view of the shooting device 21). Specifically, the movement control unit 314b can set the movement mode of the shooting system 2 so that the proportion of the outline of the observed object OBJ_g in the field of view of the shooting system 2 (especially the field of view of the shooting device 21) after the shooting system 2 has moved is larger than the proportion of the outline of the observed object OBJ_g in the field of view of the shooting system 2 (especially the field of view of the shooting device 21) before the shooting system 2 has moved. Here, if the entire outline of the object OBJ enters the field of view of the shooting system 2 (especially the field of view of the shooting device 21), the proportion of the outline of the object OBJ in the field of view of the shooting system 2 (especially the field of view of the shooting device 21) is the largest. Therefore, the movement control unit 314b can set the movement mode of the shooting system 2 so that the entire outline of the observed object OBJ_g enters the field of view of the shooting system 2 (especially the field of view of the shooting device 21).

[0330] Furthermore, the movement control unit 314b can set the movement mode of the shooting system 2 so that the proportion of a portion of the observed object OBJ_g in the shooting field of view of the shooting system 2 (especially the shooting field of view of the shooting device 21) and the proportion of the outline of the observed object OBJ_g in the shooting field of view of the shooting system 2 (especially the shooting field of view of the shooting device 21) are both maximized (or become a suitable ratio). Alternatively, the movement control unit 314b can set the movement mode of the shooting system 2 so that the ratio between the proportion of a portion of the observed object OBJ_g in the shooting field of view of the shooting system 2 (especially the shooting field of view of the shooting device 21) and the proportion of the outline of the observed object OBJ_g in the shooting field of view of the shooting system 2 (especially the shooting field of view of the shooting device 21) becomes a suitable ratio. In other words, the movement control unit 314b can set the movement mode of the shooting system 2 so that the proportion of a part of the object OBJ_g in the shooting field of the shooting system 2 (especially the shooting field of the shooting device 21) is balanced with the proportion of the outline of the object OBJ_g in the shooting field of the shooting system 2 (especially the shooting field of the shooting device 21).

[0331] For example, Figure 16(b) Shows the shooting device 21 before it moves under the control of the motion control unit 314b and the shooting device 21 after it moves under the control of the motion control unit 314b. Figure 16 In the example shown in (b), the imaging device 21, before moving under the control of the motion control unit 314b, is located at the third position P3. On the other hand, in Figure 16 In the example shown in (b), the imaging device 21, after being moved under the control of the motion control unit 314b, is located at a fourth position P4, different from the third position P3. The motion control unit 314b can set the movement mode of the imaging system 2 relative to the object OBJ_g, such that the number of pixels in the image data IMG_2D generated by the imaging device 21 at the fourth position P4 that capture at least a portion of the object OBJ_g is more than the number of pixels in the image data IMG_2D generated by the imaging device 21 at the third position P3 that capture at least a portion of the object OBJ_g is more. The motion control unit 314b can set the movement mode of the imaging system 2 relative to the object OBJ_g, such that the proportion of the object OBJ_g shown in the field of view of the imaging device 21 at the fourth position P2 is larger than the proportion of the object OBJ_g shown in the field of view of the imaging device 21 at the third position P3. For example, the movement control unit 314b can set the fourth position P4 where the moving shooting device 21 should be, so that the number of pixels in the image data IMG_2D generated by the shooting device 21 at the fourth position P4 that captures at least a portion of the observed object OBJ_g is more than the number of pixels in the image data IMG_2D generated by the shooting device 21 at the third position P3 that captures at least a portion of the observed object OBJ_g, and set the movement mode of the shooting system 2 relative to the observed object OBJ_g so that the shooting device 21 moves from the third position P3 to the fourth position P4. For example, the motion control unit 314b can set the fourth position P4 where the moving shooting device 21 should be, so that the proportion of the observed object OBJ_g shown in the shooting field of the shooting device 21 at the fourth position P4 becomes larger than the proportion of the observed object OBJ_g shown in the shooting field of the shooting device 21 at the third position P3, and set the movement mode of the shooting system 2 relative to the observed object OBJ_g so that the shooting device 21 moves from the third position P3 to the fourth position P4.

[0332] Furthermore, as described above, setting the movement mode of the imaging system 2 relative to the observed object OBJ_g can be considered equivalent to setting the movement mode of the imaging system 2 based on the observed object OBJ_g, and setting the movement mode of the imaging system 2 around the observed object OBJ_g. Therefore, the signal generation unit 313 can set the movement mode of the imaging system 2 based on the observed object OBJ_g to satisfy the conditions described in this paragraph. The signal generation unit 313 can set the movement mode of the imaging system 2 around the observed object OBJ_g to satisfy the conditions described in this paragraph.

[0333] Figure 16 In the example shown in (b), the imaging device 21 located at position 3 P3 faces the first surface S1 of the object OBJ_g, which is narrower than the second surface S2 of the object being observed. On the other hand, the imaging device 21 located at position 4 P4 faces the second surface S2 of the object OBJ_g, which is wider than the first surface S1 of the object being observed. Therefore, Figure 16 In the example shown in (b), the number of pixels in the image data IMG_2D generated by the imaging device 21 at position 4 P4 that captures at least a portion of the observed object OBJ_g is greater than the number of pixels in the image data IMG_2D generated by the imaging device 21 at position 3 P3 that captures at least a portion of the observed object OBJ_g. Furthermore, the proportion of the observed object OBJ_g shown in the field of view of the imaging device 22 at position 4 P4 is greater than the proportion of the observed object OBJ_g shown in the field of view of the imaging device 22 at position 3 P3.

[0334] Furthermore, the larger the area of ​​the object OBJ in the image shown in the image data IMG_2D, the more pixels that capture at least a portion of the object OBJ in the image shown in the image data IMG_2D. Therefore, the motion control unit 314b can set the movement mode of the shooting system 2 so that the area of ​​the observed object OBJ_g captured in the image shown in the image data IMG_2D acquired again in step S1 after the shooting system 2 moves becomes larger than the area of ​​the observed object OBJ_g captured in the image shown in the image data IMG_2D acquired in step S1 before the shooting system 2 moves. As a result, the probability of obtaining a good result from the position and pose calculation including 2D matching processing increases.

[0335] Furthermore, the number of pixels in the image data IMG_2D that capture at least a portion of the object OBJ can be considered equivalent to the number of points of the object OBJ captured in the image data IMG_2D (i.e., the points of the object OBJ captured by the imaging device 21). Therefore, the movement control unit 314b can set the movement mode of the imaging system 2 so that the number of points of the observed object OBJ_g captured in the image data IMG_2D that is reacquired in step S1 after the imaging system 2 has moved (i.e., the points of the observed object OBJ_g captured by the imaging device 21) becomes greater than the number of points of the observed object OBJ_g captured in the image data IMG_2D that was acquired in step S1 before the imaging system 2 moved (i.e., the points of the observed object OBJ_g captured by the imaging device 21).

[0336] The motion control unit 314 can set the movement mode of the imaging system 2 so that at least one of the position and orientation of the imaging system 2 relative to at least one object OBJ captured by the imaging system 2 (i.e., at least one object OBJ captured in the image data IMG_2D obtained by the motion control unit 314b in step S62b) changes. Specifically, the motion control unit 314b can set the movement mode of the imaging system 2 so that at least one of the object OBJs captured by the imaging system 2 changes in position and orientation relative to one object OBJ that is determined to be unprocessable by the end effector 4. For example, the object OBJ_g being observed is an object OBJ that, although captured by the imaging system 2, cannot be processed by the end effector 4. In this case, the motion control unit 314b can set the movement mode of the imaging system 2 so that at least one of the position and orientation of the imaging system 2 relative to the object OBJ_g being observed changes.

[0337] When at least one of the positions and orientations of the imaging system 2 relative to the object OBJ (particularly the observed object OBJ_g) changes, the direction in which the imaging system 2 captures the object OBJ (particularly the observed object OBJ_g) can also change. In this case, the movement control unit 314b can set the movement mode of the imaging system 2 so that the direction in which the imaging system 2 captures the object OBJ (particularly the observed object OBJ_g) changes. Specifically, in Figure 15 In step S1, if the imaging system 2 has completed imaging of the object OBJ (especially the observed object OBJ_g) from the first direction, the movement control unit 314b can set the movement mode of the imaging system 2 so that the imaging system 2 moves in accordance with... Figure 15The camera system 2 moves according to the movement mode set in step S63b, thereby enabling it to re-capture the object OBJ (especially the object being observed, OBJ_g) from a second direction different from the first direction. For example, the movement control unit 314b can set the movement mode of the camera system 2 so that, after moving according to the set movement mode, the camera system 2 is positioned such that it can capture the object OBJ (especially the object being observed, OBJ_g) from the second direction. For example, the movement control unit 314b can set the movement mode of the camera system 2 so that, after moving according to the set movement mode, the camera system 2 adopts an attitude in which it can capture the object OBJ (especially the object being observed, OBJ_g) from the second direction.

[0338] like Figure 17 As shown in (a), the motion control unit 314b can set the movement mode of the imaging system 2 so that the imaging system 2 rotates relative to the observed object OBJ_g around a predetermined rotation axis. For example, the motion control unit 314b can set the movement mode of the imaging system 2 so that the imaging system 2 rotates around at least one of the rotation axes along the X-axis (GL) of the global coordinate system (or other coordinate systems such as the robot coordinate system, the same in Modification 5 below), the rotation axis along the Y-axis (GL) of the global coordinate system, and the rotation axis along the Z-axis (GL) of the global coordinate system. As an example, such as Figure 17 As shown in (a), the movement control unit 314b can set the movement mode of the imaging system 2 so that the imaging system 2 rotates around a predetermined rotation axis, thereby causing at least one of the optical axis directions extending from the imaging device 21 along the optical axis AX21 and extending from the imaging device 22 along the optical axis AX22 with respect to the object OBJ (especially the observed object OBJ_g). When the imaging system 2 rotates in this way, the possibility of the direction in which the imaging system 2 captures the object OBJ (especially the observed object OBJ_g) changes is increased.

[0339] However, as Figure 17 As shown in (b), in addition to rotating the imaging system 2 relative to the object OBJ_g about a predetermined rotation axis, or as an alternative, the movement control unit 314b can set the movement mode of the imaging system 2 so that the imaging system 2 translates relative to the object OBJ_g along a predetermined translation axis. For example, the movement control unit 314b can set the movement mode of the imaging system 2 so that the imaging system 2 translates along at least one of the X-axis (GL), Y-axis (GL), and Z-axis (GL) of the global coordinate system.

[0340] Furthermore, when the imaging system 2 rotates via the motion control unit 314b, it can be considered that the direction in which the imaging system 2 observes the object OBJ (especially the object being observed, OBJ_g) changes. In this case, the process of rotating the imaging system 2 via the motion control unit 314b can be referred to as observation processing or rotational movement processing. Furthermore, when the movement mode of the imaging system 2 is set to rotate relative to the object being observed, and to move parallel to the object being observed, the amount of movement of the rotating imaging system 2 (i.e., the amount of rotational movement) can be greater than the amount of movement of the translating imaging system 2 (i.e., the amount of translational movement). In this case, the process of rotating the imaging system 2 via the motion control unit 314b can also be referred to as observation processing or rotational movement processing. However, the amount of movement of the rotating imaging system 2 may not be greater than the amount of movement of the translating imaging system 2.

[0341] The movement of the imaging system 2 can include its direction of movement. Furthermore, in Modification 2, the direction of movement of the imaging system 2 can refer to the direction of movement of at least one of the imaging devices 21 and 22. In this case, the direction of movement of the imaging system 2 relative to the observed object OBJ_g can be set to increase the likelihood of obtaining a good result from the position and orientation calculation. For example... Figure 17 As shown in (a), when the shooting system 2 rotates and moves, the direction of movement can refer to the direction of rotation. For example... Figure 17 As shown in (b), when the shooting system 2 moves in translation, the direction of movement can refer to the translation direction.

[0342] The movement direction of the shooting system 2 can be automatically set by the control device 3. For example, the control device 3 can be set to a random movement direction as the movement direction of the shooting system 2. Alternatively, the movement direction of the shooting system 2 can be set by the user.

[0343] The movement of the imaging system 2 can include the amount of movement of the imaging system 2. Furthermore, in Modification 2, the amount of movement of the imaging system 2 can refer to the movement of at least one of the imaging devices 21 and 22. In this case, the amount of movement of the imaging system 2 relative to the observed object OBJ_g can be set to increase the likelihood of obtaining a good result from the position and orientation calculation. For example... Figure 17 As shown in (a), when the shooting system 2 rotates, the amount of movement can refer to the amount of rotation (e.g., rotation angle). Figure 17 As shown in (b), when the shooting system 2 moves in translation, the amount of movement can refer to the amount of translation.

[0344] The amount of movement of the shooting system 2 can be automatically set by the control device 3. For example, the control device 3 can set a random amount of movement as the amount of movement of the shooting system 2. Alternatively, the amount of movement of the shooting system 2 can be set by the user.

[0345] In step S63b, in addition to automatically setting the movement mode of the shooting system 2 based on image data IMG_2D (specifically, based on position and attitude data POI generated from image data IMG_2D), or as an alternative, the movement control unit 314b may also set the movement mode of the shooting system 2 based on instructions from the user of the robot system SYSb. Furthermore, for example, the user can set the movement mode of the shooting system 2. The user can input instructions for setting the movement mode of the shooting system 2 using the input device 34 via a GUI (Graphical User Interface) displayed on an output device 35 that functions as a display device. The movement control unit 314b can set the movement mode of the shooting system 2 based on the user's instructions input using the input device 34. In this case, the movement control unit 314b can set the movement mode of the shooting system 2 based on the instructions input by the user using the input device 34 and the position and attitude data POI generated from image data IMG_2D (specifically, based on image data IMG_2D). Alternatively, the motion control unit 314b may set the movement mode of the shooting system 2 based on the instructions input by the user using the input device 34, rather than based on the image data IMG_2D (specifically, the position and attitude data POI generated from the image data IMG_2D).

[0346] The user can input at least one of the target values ​​for the position (target position) and the orientation (target orientation) of the shooting system 2 when shooting the object OBJ, as information for setting the movement mode of the shooting system 2. As an example, the user can set at least one of the following: the target position of the shooting system 2 along the X-axis (GL) of the global coordinate system, the target position of the shooting system 2 along the Y-axis (GL) of the global coordinate system, the target position of the shooting system 2 along the Z-axis (GL) of the global coordinate system, the target orientation of the shooting system 2 around the X-axis (GL) of the global coordinate system, the target orientation of the shooting system 2 around the Y-axis (GL) of the global coordinate system, and the target orientation of the shooting system 2 around the Z-axis (GL) of the global coordinate system.

[0347] When the user sets the target position of the shooting system 2, the motion control unit 314b can set the movement mode of the shooting system 2 based on the image data IMG_2D (specifically, the position and attitude data POI generated from the image data IMG_2D) so that the position of the shooting system 2 in the global coordinate system is consistent with the target position in the global coordinate system set by the user. When the user sets the target attitude of the shooting system 2, the motion control unit 314b can set the movement mode of the shooting system 2 based on the image data IMG_2D (specifically, the position and attitude data POI generated from the image data IMG_2D) so that the attitude of the shooting system 2 in the global coordinate system is consistent with the target attitude in the global coordinate system set by the user.

[0348] In addition to setting at least one of the target position and target orientation of the shooting system 2, or as an alternative, the user can also input the positional relationship between the shooting system 2 and the object OBJ when the shooting system 2 is shooting the object OBJ, as information for setting the movement mode of the shooting system 2. For example, the user can input at least one of the relative position and relative orientation of the shooting system 2 relative to the object OBJ when the shooting system 2 is shooting the object OBJ (relative position and orientation information), as information for setting the movement mode of the shooting system 2.

[0349] Users can set the positional relationship between the imaging system 2 and the object OBJ using the input device 34 via a GUI or similar interface displayed on the output device 35, which functions as a display device. For example, users can set the positional relationship between the imaging system 2 and the object OBJ by aligning the 3D model of the imaging system 2 with the 3D model of the object OBJ on the GUI displaying a 3D model (e.g., a CAD model). As another example, users can use the input device 34 to input numerical values ​​to determine the positional relationship between the imaging system 2 and the object OBJ. However, the relative position and orientation information is not limited to user input to the control device 3; it can also be automatically generated by the control device 3. In this case, the control device 3 can generate the relative position and orientation information based on the shape of the object OBJ (e.g., the shape of the CAD model).

[0350] The relative position and attitude information can be set to increase the likelihood that the calculation result condition is met after the imaging system 2 moves relative to the observed object OBJ_g according to the movement method set based on the relative position and attitude information (i.e., the position and attitude data POI of the observed object OBJ_g is appropriately generated). Furthermore, examples of setting the movement method of the imaging system 2 to increase the likelihood that the calculation result condition is met after the imaging system 2 moves have already been described, so detailed explanations are omitted here.

[0351] Specifically, since the position and orientation data (POI) of the observed object OBJ_g is generated as described above to set the movement mode of the shooting system 2, the user or control device 3 can set relative position and orientation information based on the POI of the observed object OBJ_g. This increases the likelihood that the calculation result condition after the shooting system 2 moves relative to the observed object OBJ_g according to the movement mode set based on the relative position and orientation information is determined to be true (i.e., the position and orientation data (POI) of the observed object OBJ_g is properly generated). In this case, it can be expected that the position and orientation data (POI) of the observed object OBJ_g is reliably generated, and as a result, it can be expected that the end effector 4 will reliably perform the prescribed processing on the observed object OBJ_g.

[0352] When the user sets the positional relationship between the shooting system 2 and the object OBJ, the motion control unit 314b can calculate the target value (i.e., the target position) of the shooting system 2 in the global coordinate system after movement, based on the position and attitude data POI of the observed object OBJ_g generated in step S63b (i.e., at least one of the position and attitude of the object OBJ in the global coordinate system) and the relative positional relationship set by the user. Then, the motion control unit 314b can set the movement mode of the shooting system 2 so that the position of the shooting system 2 in the global coordinate system matches the calculated target position. Furthermore, the motion control unit 314b can calculate the target value (i.e., the target attitude) of the shooting system 2 in the global coordinate system after movement, based on the position and attitude data POI of the observed object OBJ_g generated in step S63b (i.e., at least one of the position and attitude of the observed object OBJ_g in the global coordinate system) and the relative positional relationship set by the user. Then, the motion control unit 314b can set the movement mode of the shooting system 2 so that the attitude of the shooting system 2 in the global coordinate system is consistent with the calculated target attitude.

[0353] In step S63b, in addition to setting the movement mode of the shooting system 2 based on the image data IMG_2D, or as an alternative, the movement control unit 314b may also set the movement mode shown in the change information as the movement mode of the shooting system 2. The movement mode shown in the change information can be a fixed movement mode. That is, the change information can represent a fixed movement mode as the movement mode of the shooting system 2 (in other words, it can be specified or defined). For example, the change information can represent a fixed movement direction. For example, the change information can represent a fixed movement amount. In this case, the change information can be regarded as representing at least one of the direction and amount that causes at least one of the position and orientation of the shooting system 2 to change. The change information can be set by the user or automatically set by the movement control unit 314b.

[0354] When the motion control unit 314b sets the motion mode indicated by the change information to the motion mode of the shooting system 2, the motion control unit 314b may not need to... Figure 15 In step S62b, the image data IMG_2D is newly (in other words, re-acquired) or not... Figure 15 In step S62c, at least one of the position and orientation of the object OBJ is calculated based on the image data IMG_2D in order to set the movement mode of the shooting system 2. Therefore, in Figure 15 In steps S62b to S64b, the processing load of the movement control unit 314b for moving the imaging system 2 is reduced. As a result, in Figure 15 In steps S62b to S64b, the time required for the shooting system 2 to move is shortened.

[0355] Furthermore, when the motion control unit 314b sets the motion mode indicated by the change information to the motion mode of the shooting system 2, the shooting system 2 can be without... Figure 15 In step S62b, the image data IMG_2D acquired by the motion control unit 314b is generated. That is, the shooting system 2 can, without... Figure 15 In step S62b, the image data IMG_2D acquired by the motion control unit 314b is generated to photograph the object OBJ. Therefore, the time required for the imaging system 2 to photograph the object OBJ and generate the image data IMG_2D is eliminated, so in Figure 15 In steps S62b to S64b, the time required for the shooting system 2 to move is reduced.

[0356] Back Figure 15Then, the motion control unit 314b generates a robot control signal to control the robotic arm 12 so that the imaging system 2 moves according to the movement pattern set in step S63b (step S64b). As a result, the imaging system 2 moves according to the movement pattern set in step S63b (step S64b). For example, the imaging system 2 may move in the movement direction set in step S63b. For example, the imaging system 2 may move an amount of movement equivalent to that set in step S63b.

[0357] Then, the process after step S1 is repeated. That is, the process of moving the shooting system 2 (steps S62b to S64b) and the position and attitude calculation (steps S1 to S3) are repeated alternately until it is determined in step S61b that the end effector 4 can perform the specified processing on the object OBJ (especially the result of the position and attitude calculation is good).

[0358] Here, when the imaging system 2 moves in step S64b, at least one of its position and orientation changes. For example, the position of the imaging system 2 after the movement changes to a position different from the position of the imaging system 2 at the moment when it captured the object OBJ (especially the observed object OBJ_g) using at least one of the image data IMG_2D and IMG_3D used in the past determination that the condition for generating the computation result is not met. As a result, the position of the imaging system 2 relative to the object OBJ (especially the observed object OBJ_g) changes at the moment when it captured the object OBJ (especially the observed object OBJ_g) using at least one of the image data IMG_2D and IMG_3D used in the past determination that the condition for generating the computation result is not met. For example, the orientation of the imaging system 2 after the movement changes to an orientation different from the orientation of the imaging system 2 at the moment when it captured the object OBJ (especially the observed o...

Claims

1. A control device, The control device generates control signals for controlling a robot equipped with a processing device for handling objects and a camera system, and moves the processing device and the camera system. The control device is characterized by comprising: A computing device for generating the control signal; as well as A communication device that outputs the control signals generated by the computing device. If, based on the first positional relationship between the object and the imaging system, the first image data generated by the imaging system of the object indicates that no object exists that can be processed by the processing device, the computing device generates a first control signal based on the second image data generated by the imaging system of the object to control the robot to change the positional relationship to a second positional relationship different from the first positional relationship. A second control signal is generated based on position data and attitude data representing the position and attitude of the object, generated from the third image data generated by the shooting system when the object is photographed in the second positional relationship.

2. The control device as described in claim 1, characterized in that, The first control signal is a signal used to control the robot so that the position and orientation of the imaging system relative to the object change.

3. The control device as described in claim 1 or 2, characterized in that, The first control signal is a signal used to control the robot so that the shooting direction of the shooting system relative to the object changes from the first direction to the second direction.

4. The control device as described in claim 3, characterized in that, At least one of the first image data and the second image data is generated by photographing the object from the first direction using the shooting system.

5. The control device as described in any one of claims 1 to 4, characterized in that, When the position data and posture data, calculated based on the third image data, representing the position and posture of the object, are set as the second position data and the second posture data, The computing device generates first position data and first pose data representing the position and pose of the object based on the second image data. The first control signal is generated based on the first position data, the first attitude data, and the pre-set position and attitude information representing the relative position and attitude of the object and the shooting system.

6. The control device as described in claim 2, characterized in that, The position and attitude information is set in advance by the user.

7. The control device as described in any one of claims 1 to 6, characterized in that, Under the second positional relationship, the proportion of the object in the field of view of the shooting system is larger than that under the first positional relationship.

8. The control device as described in claim 7, characterized in that, The first control signal is used to control the robot so that the proportion of the object in the field of view under the second positional relationship is larger than the proportion of the object in the field of view under the first positional relationship.

9. The control device as described in any one of claims 1 to 8, characterized in that, Based on first three-dimensional position data generated from the first image data and representing the three-dimensional positions of multiple points of the object, the processing device determines that there is no object that can be processed by the processing device. The position data and pose data of the object are calculated based on the second three-dimensional position data generated from the third image data and representing the three-dimensional positions of multiple points of the object. The number of points of the object whose three-dimensional position is represented by the second three-dimensional position data is greater than the number of points of the object whose three-dimensional position is represented by the first three-dimensional position data.

10. The control device as claimed in claim 9, characterized in that, The first control signal is a signal used to control the robot such that the number of points of the object whose three-dimensional position is represented by the second three-dimensional position data is greater than the number of points of the object whose three-dimensional position is represented by the first three-dimensional position data.

11. The control device as claimed in any one of claims 1 to 10, characterized in that, The first control signal is a signal used to control the robot so that the imaging system rotates and moves around a predetermined rotation axis.

12. The control device as described in any one of claims 1 to 10, characterized in that, The first control signal is a signal used to control the robot so that the direction of the optical axis extending from the shooting system along the optical axis of the shooting system changes relative to the object so as to cause the shooting system to rotate and move.

13. The control device as described in any one of claims 1 to 12, characterized in that, The computing device generates the position data and pose data of the object based on local model data representing a portion of the model of the object and the third image data. The second control signal is generated based on the generated position data and attitude data.

14. The control device as described in any one of claims 1 to 13, characterized in that, When the position data and posture data representing the position and posture of the object generated based on the third image data are set as the second position data and the second posture data, The computing device generates first position data and first pose data representing the position and pose of the object based on model data representing the model of the object and the second image data. The first control signal is generated based on the generated first position data and the first attitude data.

15. The control device as described in any one of claims 1 to 12, characterized in that, The computing device generates first position data and first pose data representing the position and pose of the object based on model data representing the model of the object and the second image data. The first control signal is generated based on the first position data and the first attitude data. Based on local model data representing a portion of the model of the object and the third image data, second position data and second pose data are generated as position data and pose data of the object. The second control signal is generated based on the generated second position data and the second attitude data.

16. The control device as described in any one of claims 13 to 15, characterized in that, The portion of the model shown in the local model data corresponds to the region where the density of points representing the three-dimensional position of the object shown in the three-dimensional position data generated from the third image data is above a threshold.

17. The control device as claimed in any one of claims 13 to 16, characterized in that, The portion of the model shown in the local model data is obtained by removing the portion of the model from which the density of points representing the three-dimensional position of the object shown in the three-dimensional position data generated from the third image data is less than a threshold.

18. The control device as claimed in any one of claims 13 to 17, characterized in that, The local model data is generated based on simulated image data, which is generated by assuming that the object is photographed by the camera system when the object and the camera system are in a specified positional relationship.

19. The control device as claimed in claim 18, characterized in that, The specified positional relationship is based on a pre-defined positional relationship that represents the relative position and posture of the object and the shooting system.

20. The control device as described in claim 18 or 19, characterized in that, The local model data is generated based on the contrast of the image shown in the simulated image data.

21. The control device as described in any one of claims 18 to 20, characterized in that, The computing device generates the local model data based on the simulated image data.

22. The control device as claimed in any one of claims 13 to 17, characterized in that, The computing device generates the local model data based on at least one of information pre-input by the user relating to regions removed from the model and information relating to regions remaining in the model.

23. The control device as described in any one of claims 13 to 22, characterized in that, The computing device calculates the position data and pose data of the object based on the three-dimensional position data generated from the third image data and representing the three-dimensional positions of multiple points of the object, and the local model data.

24. The control device as described in any one of claims 1 to 23, characterized in that, The second control signal is a signal used to control the robot so that the processing device approaches the object in order to perform the processing on the object.

25. The control device as described in any one of claims 1 to 24, characterized in that, The second image data is generated by the shooting system capturing the object in the first positional relationship.

26. The control device as described in any one of claims 1 to 25, characterized in that, The second image data is generated by the shooting system capturing the object in a positional relationship different from the second positional relationship.

27. The control device as described in any one of claims 1 to 26, characterized in that, After determining, based on the first image data, that there is no object that can be processed by the processing device, the shooting system generates the second image data by shooting the object.

28. The control device as claimed in any one of claims 1 to 26, characterized in that, Before determining, based on the first image data, that there is no object that can be processed by the processing device, the shooting system generates the second image data by shooting the object.

29. The control device as claimed in claim 28, characterized in that, The shooting system generates the second image data together with the first image data by shooting the object in the first positional relationship.

30. The control device as described in any one of claims 1 to 29, characterized in that, If the computing device determines that there is no object that can be processed by the processing device, and the computing device determines that the position and pose calculation of the object based on the first image data is poor.

31. The control device as described in any one of claims 1 to 30, characterized in that, The shooting system includes a first shooting device and a second shooting device that is different from the first shooting device. The first imaging device generates the first image data by photographing the object at the first positional relationship. The second imaging device generates the second image data by photographing the object. The first imaging device generates the third image data by photographing the object in the second positional relationship.

32. The control device as described in claim 31, characterized in that, The first shooting device is a stereo camera equipped with two monocular cameras and one of the monocular cameras that is different from the two monocular cameras. The second shooting device is the stereo camera and another of the monocular cameras that are different from the two monocular cameras.

33. The control device as described in claim 32, characterized in that, The first shooting device is the stereo camera. The second shooting device is a monocular camera that is different from the two monocular cameras.

34. The control device as described in any one of claims 1 to 30, characterized in that, The shooting system includes a stereo camera or a monocular camera equipped with two monocular cameras.

35. The control device as described in claim 34, characterized in that, The shooting system is equipped with the stereo camera. The two monocular cameras of the stereo camera capture images of the object in the first positional relationship, thereby generating the first image data, which includes two image data generated by each of the two monocular cameras. The two monocular cameras of the stereo camera generate a second image data by photographing the object, which includes at least one of the two image data generated by each of the two monocular cameras.

36. The control device as described in claim 35, characterized in that, The two monocular cameras of the stereo camera capture images of the object in the first positional relationship, thereby generating the second image data, which includes two image data generated by each of the two monocular cameras. The second image data is the same as the first image data.

37. The control device as described in claim 34, characterized in that, The shooting system is equipped with the monocular camera. The monocular camera generates the first image data by capturing images of the object in the first positional relationship. The monocular camera generates the second image data by photographing the object.

38. The control device as claimed in claim 37, characterized in that, The monocular camera generates the second image data by capturing images of the object in the first positional relationship. The second image data is the same as the first image data.

39. The control device as described in any one of claims 1 to 4, characterized in that, When the position data and posture data, calculated based on the third image data, representing the position and posture of the object, are set as the second position data and the second posture data, The computing device generates first position data and first pose data representing the position and orientation of the object based on the second image data. The first control signal is generated based on the first position data, the first attitude data, and the pre-set position and attitude information representing the relative position and attitude of the object and the shooting system.

40. A control system, characterized in that, include: The control device according to any one of claims 1 to 39; and The shooting system.

41. A robot system, characterized in that, include: The control device according to any one of claims 1 to 39; The shooting system; as well as The robot.

42. A control method, Generating control signals for controlling a robot equipped with a processing device for handling objects and a camera system, and moving the processing device and the camera system, the control method is characterized by comprising: If, based on the first positional relationship between the object and the imaging system, the first image data generated by the imaging system of the object is determined to be unavailable and can be processed by the processing device, a first control signal is generated based on the second image data generated by the imaging system of the object to control the robot to change the positional relationship to a second positional relationship different from the first positional relationship, and this control signal is used as the control signal. as well as A second control signal is generated based on position data and attitude data representing the position and attitude of the object, generated from the third image data generated by the shooting system when the object is photographed in the second positional relationship.

43. A computer program, characterized in that, The computer is made to execute the control method of claim 42.

44. A control device, The control device generates control signals for controlling a robot equipped with a processing device for handling objects and a camera system, and moves the processing device and the camera system. The control device is characterized by comprising: A computing device for generating the control signal; as well as A communication device that outputs the control signals generated by the computing device. If, based on the first image data generated by the imaging system when it is in the first position and the first posture, it is determined that there is no object in the first object group that can be processed by the processing device, the computing device generates a first control signal for controlling the robot to change the imaging system to a second position and a second posture that are different from the first position and the first posture, based on the second image data generated by the imaging system when it is in the first position and the first posture.

45. The control device as claimed in claim 44, characterized in that, The first control signal is a signal used to control the robot so that the position and orientation of the imaging system relative to one of the object objects in the second object group changes.

46. ​​The control device as described in claim 44 or 45, characterized in that, The first control signal is a signal used to control the robot so that the shooting direction of the shooting system relative to one of the object objects in the second object group changes from the first direction to the second direction.

47. The control device as claimed in claim 46, characterized in that, The first object group contains the one object. The first image data is generated by the shooting system capturing the object from the first direction.

48. The control device as described in claim 46 or 47, characterized in that, The second image data is generated by the shooting system capturing the object from the first direction.

49. The control device as claimed in any one of claims 44 to 48, characterized in that, The first control signal is a signal used to control the robot so that the proportion of one of the object objects in the second object group included in the field of view of the shooting system after the shooting system moves under the control of the robot based on the first control signal is greater than the proportion of the object object included in the field of view before the shooting system moves under the control of the robot based on the first control signal.

50. The control device as described in claim 44 or 45, characterized in that, The first control signal is used to control the robot so that the imaging system moves to a position and orientation that allows it to capture images of one of the objects in the second object group from the first direction. After the imaging system moves to a position and orientation that can be photographed from the first direction by the robot controlled by the first control signal, the computing device generates a second control signal for controlling the robot so that the imaging system moves to a position and orientation that can be photographed from the second direction that is different from the first direction, as the control signal.

51. The control device as described in any one of claims 44 to 50, characterized in that, At least one of the object objects in the second object group is the same as at least one of the object objects in the first object group.

52. The control device according to any one of claims 44 to 51, characterized in that, The computing device generates the first control signal based on third position data and third posture data representing the position and posture of one of the object objects in the second object group generated according to the second image data, and pre-set relative position and posture information representing the relative position and posture of the object object and the shooting system.

53. The control device as described in claim 52, characterized in that, The relative position and attitude information is set in advance by the user.

54. The control device as described in any one of claims 44 to 53, characterized in that, If, when the third image data generated by the imaging system from a third object group containing multiple object objects, based on the second position and the second posture, is determined that there is no object object in the third object group that can be processed by the processing device, the computing device generates a second control signal for controlling the robot to change the imaging system to a fourth position and a fourth posture that are different from the second position and the second posture, as the control signal.

55. The control device as described in claim 54, characterized in that, The computing device generates the second control signal without using new image data that is the result of the shooting system.

56. The control device as described in claim 54 or 55, characterized in that, The computing device generates the first control signal based on third position data and third posture data representing the position and posture of one of the object objects in the second object group, generated according to the second image data, and pre-set relative position and posture information representing the relative position and posture of the object object and the shooting system. The second control signal is generated based on pre-defined change information indicating at least one of the direction and amount of change in the relative position and orientation of the object and the shooting system.

57. The control device as claimed in claim 56, characterized in that, The change information is set in advance by the user.

58. The control device as described in any one of claims 54 to 57, characterized in that, The computing device generates a third control signal as the control signal based on fifth position data and fifth posture data, which represent the position and posture of one of the object objects in the fourth object object group, generated by the imaging system from the fourth image data generated according to the fourth position and the fourth posture.

59. The control device as described in claim 58, characterized in that, The third control signal is a signal used to control the robot so that the processing device approaches one of the object objects in the fourth object group in order to perform the processing on the object object.

60. The control device as described in claim 58 or 59, characterized in that, The computing device generates the fifth position data and the fifth pose data of one object in the fourth object group based on local model data representing a portion of the model of the object and the fourth image data. The third control signal is generated based on the generated fifth position data and fifth attitude data.

61. The control device as claimed in claim 60, characterized in that, The portion of the model shown in the local model data corresponds to the region where the density of points representing the three-dimensional position of one of the objects in the fourth object group, as shown in the three-dimensional position data generated from the fourth image data, is above a threshold.

62. The control device as described in claim 60 or 61, characterized in that, The portion of the model shown in the local model data is obtained by removing the part of the model corresponding to the region where the density of points representing the three-dimensional position of one of the object objects in the fourth object group shown in the three-dimensional position data generated from the fourth image data is less than a threshold.

63. The control device as described in any one of claims 58 to 62, characterized in that, The computing device generates the first control signal based on model data representing the model of the object and the second image data.

64. The control device as described in any one of claims 58 to 63, characterized in that, The computing device determines, based on model data representing the model of the object and the first image data, whether at least one object in the first object group can be processed by the processing device, and the result is that there is no object that can be processed by the processing device.

65. The control device as described in any one of claims 58 to 64, characterized in that, The processing device determines, based on first local model data representing a portion of the model of the object and the third image data, whether at least one of the object objects in the third object group can be processed by the processing device, and determines that there is no object object in the third object group that can be processed by the processing device. Based on the second local model data representing a portion of the model and the fourth image data, the fifth position data and the fifth pose data of one object in the fourth object group are generated. The third control signal is generated based on the generated fifth position data and fifth attitude data.

66. The control device as claimed in claim 62, characterized in that, The first local model data and the second local model data are the same data representing the same part of the model.

67. The control device as claimed in claim 65, characterized in that, The second local model data is data representing a portion of the model that is either partially repeated or different from the portion shown in the first local model data.

68. The control device as claimed in any one of claims 65 to 67, characterized in that, The third object group includes the second object as the object. When setting one of the objects in the fourth object group as the fourth object object. The computing device determines, based on the first local model data and the third image data, whether one of the object objects in the third object group, namely the second object object, can be processed by the processing device. The result is that there is no second object object in the third object group that can be processed by the processing device. The third control signal is generated based on the fifth position data and the fifth pose data of the fourth object generated according to the second local model data and the fourth image data.

69. The control device as claimed in claim 68, characterized in that, The fourth object is the second object.

70. The control device as claimed in claim 68, characterized in that, The fourth object is a different object from the second object.

71. The control device as described in any one of claims 68 to 70, characterized in that, The computing device generates the first control signal based on third position data and third attitude data, generated according to the second image data, which represent the position and attitude of one of the object objects in the second object group, namely the first object object. A second control signal is generated to control the robot so that the position and orientation of the imaging system change relative to one of the object objects in the third object group, i.e., the third object object.

72. The control device as claimed in claim 71, characterized in that, The fourth object is the same object as the first, second, and third objects.

73. The control device as claimed in claim 71, characterized in that, The fourth object is an object that is different from at least one of the first, second, and third objects.

74. The control device according to any one of claims 65 to 73, characterized in that, The second local model data is data representing a portion of the model that is either partially repeated or different from the portion shown in the first local model data. The first local model data is generated based on the first simulated image data, which is generated by assuming that the object was photographed by the imaging system when the object and the imaging system are in a first positional relationship. The second local model data is generated based on the second simulated image data, which is generated by assuming that the object is photographed by the shooting system when the object and the shooting system are in a second positional relationship different from the first positional relationship.

75. The control device as claimed in claim 74, characterized in that, The computing device generates the first local model data based on the first simulated image data. The second local model data is generated based on the second simulated image data.

76. The control device as described in claim 74 or 75, characterized in that, The first positional relationship and the second positional relationship are respectively positional relationships based on pre-set relative position and attitude information representing the relative position and attitude of the object and the shooting system.

77. The control device as claimed in claim 76, characterized in that, The first positional relationship is a positional relationship based on the relative position and attitude information. The second positional relationship is a positional relationship based on the relative positional attitude information and a pre-set positional relationship that indicates at least one of the changes in direction and amount of the relative position and attitude of the object and the shooting system.

78. The control device according to any one of claims 65 to 73, characterized in that, The second local model data is data representing a portion of the model that is either partially identical to or different from that portion of the model shown in the first local model data. The computing device generates the first local model data based on at least one of first removal information related to regions removed from the model and first residual information related to regions remaining in the model, which are pre-input by the user. The second local model data is generated based on at least one of the second removal information and the second residual information pre-input by the user.

79. The control device as described in any one of claims 54 to 57, characterized in that, The shooting system includes a first shooting device and a second shooting device that is different from the first shooting device. The first imaging device generates the first image data and the third image data. The second imaging device generates the second image data. The first shooting device is a stereo camera equipped with two monocular cameras and one of the monocular cameras that is different from the two monocular cameras. The second shooting device is the stereo camera and another of the monocular cameras that are different from the two monocular cameras.

80. The control device as described in any one of claims 58 to 78, characterized in that, The shooting system includes a first shooting device and a second shooting device that is different from the first shooting device. The first imaging device generates the first image data, the third image data, and the fourth image data. The second imaging device generates the second image data. The first shooting device is a stereo camera equipped with two monocular cameras and one of the monocular cameras that is different from the two monocular cameras. The second shooting device is the stereo camera and another of the monocular cameras that are different from the two monocular cameras.

81. The control device as described in claim 79 or 80, characterized in that, The first shooting device is the stereo camera. The second shooting device is a monocular camera that is different from the two monocular cameras.

82. The control device as described in any one of claims 54 to 56, characterized in that, If, when the fourth image data generated by the imaging system from a fourth object group containing multiple object objects, based on the fourth position and the fourth posture, is determined that there is no object object in the fourth object group that can be processed by the processing device, the computing device generates a fourth control signal for controlling the robot so that the imaging system changes from the fourth position and the fourth posture to a pre-set fifth position and fifth posture that indicates a re-shooting position and posture, as the control signal.

83. The control device as described in claim 82, characterized in that, The computing device generates the fourth control signal without using new image data that is the result of the shooting system.

84. The control device as claimed in claim 83, characterized in that, The fourth control signal is a control signal used to control the robot such that at least half of the object objects in the fifth object group included in the field of view of the shooting system at the fifth position and the fifth posture are object objects that are different from the object objects in the fourth object group included in the field of view of the shooting system at the fourth position and the fourth posture.

85. The control device as claimed in claim 84, characterized in that, The fourth control signal is a control signal used to control the robot such that all objects in the fifth object group contained in the field of view at the fifth position and the fifth posture are different from objects in the fourth object group contained in the field of view at the fourth position and the fourth posture.

86. The control device as described in any one of claims 82 to 85, characterized in that, The computing device sets the fifth position and the fifth posture based on the reshooting position and posture information pre-input by the user, which represents the position and posture of the reshooting by the shooting system.

87. The control device as described in any one of claims 82 to 85, characterized in that, The computing device sets the fifth position and the fifth posture based on the shooting field of view of the shooting system.

88. The control device as claimed in claim 87, characterized in that, At least the plurality of object objects from the fourth object group and the plurality of object objects from the fifth object group enter the container. The computing device sets the fifth position and the fifth posture based on the shooting field of view and the size of the container.

89. The control device as described in any one of claims 82 to 88, characterized in that, The computing device generates a fifth control signal as the control signal based on a sixth position data and a sixth posture data representing the position and posture of one of the object objects in the fifth object object group, generated by the imaging system from the fifth image data generated according to the fifth position and the fifth posture.

90. The control device as claimed in claim 89, characterized in that, The fifth control signal is a signal used to control the robot so that the processing device approaches one of the object objects in the fifth object group in order to perform the processing on the object object.

91. The control device as described in claim 89 or 90, characterized in that, The shooting system includes a first shooting device and a second shooting device that is different from the first shooting device. The first imaging device generates the first image data, the third image data, the fourth image data, and the fifth image data. The second imaging device generates the second image data. The first shooting device is a stereo camera equipped with two monocular cameras and one of the monocular cameras that is different from the two monocular cameras. The second shooting device is the stereo camera and another of the monocular cameras that are different from the two monocular cameras.

92. The control device as described in any one of claims 44 to 91, characterized in that, The shooting system includes a stereo camera or a monocular camera equipped with two monocular cameras.

93. A control system, characterized in that, include: The control device according to any one of claims 44 to 92; as well as The shooting system.

94. A robot system, characterized in that, include: The control device according to any one of claims 44 to 92; The shooting system; as well as The robot.

95. A control method, Generating control signals for controlling a robot equipped with a processing device for handling objects and a camera system, and moving the processing device and the camera system, the control method is characterized by comprising: If, when the first image data generated by the imaging system from a first group of object objects containing multiple object objects, based on the first position and first posture, is determined that there is no object object in the first group of object objects that can be processed by the processing device, a first control signal is generated based on the second image data generated by the imaging system from a second group of object objects containing multiple object objects, to control the robot to change the imaging system to a second position and a second posture different from the first position and the first posture, and this control signal is used as the control signal.

96. A computer program, characterized in that, The computer is made to execute the control method as described in claim 95.

97. A control device, The control device generates control signals for controlling a robot equipped with a processing device for handling objects and a camera system, and moves the processing device and the camera system. The control device is characterized by comprising: The computing device generates the control signal based on first three-dimensional position data representing the three-dimensional positions of each of the multiple points of the first object group generated from the imaging system of the imaging system, and second three-dimensional position data representing the three-dimensional positions of each of the multiple points of the first object group, which is less than the first three-dimensional position data. as well as A communication device that outputs the control signals generated by the computing device. Based on the second three-dimensional position data, the computing device generates multiple first position data and first pose data representing the position and orientation of each object in the second object group, which is composed of multiple object objects in the first object group. Based on the generated multiple first position data and first pose data, as well as the first three-dimensional position data, second position data and second pose data representing the position and pose of one of the objects in the second object group are generated. The control signal is generated based on the generated second position data and the generated second attitude data.

98. The control device as claimed in claim 97, characterized in that, The computing device generates the first position data and the first pose data of each object in the second object group based on local three-dimensional model data representing a portion of the three-dimensional model of the object and the second three-dimensional position data.

99. The control device as claimed in claim 98, characterized in that, The computing device generates the first position data and the first pose data of each object in the second object group based on multiple local three-dimensional model data that respectively represent different parts of the three-dimensional model and the second three-dimensional position data.

100. The control device as claimed in claim 99, characterized in that, The plurality of local 3D model data are assigned priority for generating the first position data and the first pose data, respectively.

101. The control device as described in claim 99 or 100, characterized in that, The plurality of local 3D model data includes first local 3D model data and second local 3D model data. After performing first position and pose calculations on multiple object objects in the first object group based on the first local three-dimensional model data and the second three-dimensional position data, the computing device performs second position and pose calculations on at least one object object in the first object group based on the second local three-dimensional model data and the second three-dimensional position data.

102. The control device as claimed in claim 100, characterized in that, The plurality of local 3D model data includes first local 3D model data assigned a first priority as the priority, and second local 3D model data assigned a second priority, which is lower than the first priority. If the number of objects whose first position and pose calculation based on the first local 3D model data and the second 3D position data with a first priority is good is less than a threshold, the computing device performs a second position and pose calculation on at least one of the objects in the first object group based on the second local 3D model data and the second 3D position data with a second priority that is lower than the first priority.

103. The control device as described in claim 102, characterized in that, The object whose first position pose calculation is good includes the object whose similarity to a portion of the three-dimensional model shown in the first local three-dimensional model data and the object whose three-dimensional position is represented by multiple points in the second three-dimensional position data representing the three-dimensional position of the object of the first object group is above a threshold.

104. The control device as described in any one of claims 101 to 103, characterized in that, The first position data and the first pose data of each object in the second object group include: the first position data and the first pose data of multiple objects in the second object group generated by the first position and pose calculation; and the first position data and the first pose data of at least one object in the second object group generated by the second position and pose calculation.

105. The control device as described in any one of claims 101 to 104, characterized in that, The computing device generates the second position data and the second posture data based on the first position data and the first posture data of a plurality of objects in the second object group generated by the first position and posture calculation process, the first position data and the first posture data of at least one object in the second object group generated by the second position and posture calculation process, and the first three-dimensional position data.

106. The control device as claimed in any one of claims 101 to 105, characterized in that, The computing device generates the second position data and the second pose data based on the first position data and the first pose data of a plurality of object objects in the second object group generated by the first position and pose calculation processing, the first position data and the first pose data of at least one object object in the second object group generated by the second position and pose calculation processing, the first three-dimensional position data, the first local three-dimensional model data, and the second local three-dimensional model data.

107. The control device as claimed in any one of claims 101 to 106, characterized in that, The computing device selects one object from the second object group based on the first position data and the first posture data of multiple object objects in the second object group generated by the first position and posture calculation process, the first position data and the first posture data of at least one object object in the second object group generated by the second position and posture calculation process, and the first three-dimensional position data.

108. The control device as claimed in any one of claims 101 to 107, characterized in that, The computing device calculates a first similarity between a portion of the 3D model shown in the first local 3D model data and a plurality of points representing the 3D positions of the respective objects in the first object group, based on the first local 3D model data and the second 3D position data. Based on the second local 3D model data and the second 3D position data, a second similarity is calculated between a portion of the 3D model shown in the second local 3D model data and multiple points representing the 3D positions of the respective objects in the first object group shown in the second 3D position data. If the first similarity score corresponding to the object whose first position data and first pose data are calculated by the first position and pose calculation is above a threshold, The second similarity to the object corresponding to the object whose first position data and first pose data are calculated by the second position pose calculation is above a threshold.

109. The control device as described in any one of claims 101 to 108, characterized in that, The computing device selects portions of the first three-dimensional position data based on the plurality of first positions and first poses generated by the first position and pose calculation process, as multiple first interest data including data corresponding to at least a portion of multiple points of each of the plurality of object objects in the second object group. Based on at least one of the first position data and the first pose data generated by the second position and pose calculation, at least a portion of the first three-dimensional position data is selected as at least one second interest data, including data corresponding to at least a portion of multiple points of at least one of the object objects in the second object group. Based on the first local 3D model data, the second local 3D model data, the plurality of first attention data and the at least one second attention data, select the object from the second object group.

110. The control device as claimed in claim 109, characterized in that, The computing device selects the object from the second object group based on the similarity between multiple points representing the three-dimensional positions of multiple object objects in the second object group as shown by the multiple first attention data and a portion of the three-dimensional model shown by the first local three-dimensional model data, and the similarity between multiple points representing the three-dimensional positions of at least one object in the second object group as shown by the at least one second attention data and a portion of the three-dimensional model shown by the second local three-dimensional model data.

111. The control device as described in claim 109 or 110, characterized in that, The computing device generates the second position data and the second pose data based on one of the plurality of first attention data corresponding to the selected object and the first local 3D model data.

112. The control device as described in claim 109 or 110, characterized in that, The computing device generates the second position data and the second pose data based on one of the at least one second concern data corresponding to the selected object and the second local 3D model data.

113. The control device as described in any one of claims 97 to 104, characterized in that, The computing device generates the second position data and the second pose data based on local three-dimensional model data representing a portion of the object's three-dimensional model, the generated plurality of first position data and first pose data, and the first three-dimensional position data.

114. The control device as claimed in claim 113, characterized in that, The computing device generates the second position data and the second pose data based on multiple local three-dimensional model data that respectively represent different parts of the three-dimensional model, multiple generated first position data and first pose data, and the first three-dimensional position data.

115. The control device as described in claim 113 or 114, characterized in that, The computing device selects one of the object objects from the second object group based on multiple local 3D model data representing different parts of the 3D model, multiple generated first position data and first pose data, and the first 3D position data. The second position data and the second pose data are generated based on one of the first position data and the first pose data corresponding to the selected object, and one of the local 3D model data from a plurality of local 3D model data.

116. The control device as claimed in claim 97, characterized in that, Based on the generated plurality of first position data and first pose data, as well as the first three-dimensional position data, the computing device selects one object from the second object group. The second position data and the second pose data are generated based on the first position data and the first pose data of the selected object, as well as the first three-dimensional position data.

117. The control device as claimed in claim 116, characterized in that, The computing device selects portions of each of the first three-dimensional position data based on the generated plurality of first position data and first pose data, as multiple data of interest, including data corresponding to at least a portion of multiple points of the object in each of the second object group captured by the imaging system. Based on model data representing the three-dimensional model of the object and the selected plurality of interest data, the object is selected from the second object group.

118. The control device as claimed in claim 117, characterized in that, The computing device selects one object from the second object group based on the similarity between multiple points representing the three-dimensional position of the object in each of the multiple interest data and the three-dimensional model.

119. The control device as described in claim 117 or 118, characterized in that, The computing device generates the second position data and the second pose data based on the model data and the attention data corresponding to the object.

120. The control device as claimed in any one of claims 97 and 116 to 119, characterized in that, The computing device selects portions of each of the first three-dimensional position data based on the generated plurality of first position data and first pose data, as multiple data of interest, including data corresponding to at least a portion of multiple points of the object in each of the second object group captured by the imaging system. Based on the multiple attention data, second position data and second pose data representing the position and pose of one of the objects in the second object group are generated.

121. The control device as claimed in any one of claims 97 to 120, characterized in that, The shooting system includes a stereo camera. The computing device generates the first three-dimensional position data and the second three-dimensional position data based on the stereo image data generated by the stereo camera as a result of the shooting.

122. The control device as claimed in claim 121, characterized in that, The computing device generates depth image data with distance information for each pixel based on the stereo image data. The first three-dimensional position data and the second three-dimensional position data are generated based on the depth image data.

123. The control device as claimed in claim 121, characterized in that, The computing device generates first depth image data, with distance information corresponding to each pixel, based on the stereo image data. Second depth image data is generated by reducing a portion of the pixel data in the generated first depth image data. The first three-dimensional position data is generated based on the first depth image data. The second three-dimensional position data is generated based on the second depth image data.

124. The control device as claimed in claim 121, characterized in that, The computing device generates the second three-dimensional position data by reducing the data representing a portion of the three-dimensional position of the first three-dimensional position data.

125. The control device as claimed in claim 121, characterized in that, As the result of the stereo camera's capture, a first stereo image data and a second stereo image data with fewer pixels than the first stereo image data are generated. The computing device generates the first three-dimensional position data based on the first stereoscopic image data. The second three-dimensional position data is generated based on the second stereo image data.

126. A control system, characterized in that, include: The control device according to any one of claims 97 to 125; and The shooting system.

127. A robot system, characterized in that, include: The control device according to any one of claims 97 to 125; The shooting system; as well as The robot.

128. A control method, Generating control signals for controlling a robot equipped with a processing device for handling objects and a camera system, and moving the processing device and the camera system, the control method is characterized by comprising: The control signal is generated based on first three-dimensional position data representing the three-dimensional positions of multiple points in the first object group, generated from the imaging system's imaging results of a first object group containing multiple said object objects, and second three-dimensional position data, which is less than the first three-dimensional position data but represents the three-dimensional positions of multiple points in the first object group. Generating the control signal includes: Based on the second three-dimensional position data, multiple first position data and first pose data are generated to represent the position and pose of each object in the second object group, which is composed of multiple object objects in the first object group; Based on the generated multiple first position data and first pose data, and the first three-dimensional position data, second position data and second pose data representing the position and pose of one of the object objects in the second object group are generated; and The control signal is generated based on the generated second position data and the generated second attitude data.

129. A computer program, characterized in that, The computer is made to execute the control method of claim 128.

130. A control device, The control device generates control signals for controlling a robot equipped with a processing device and a camera system for processing objects contained in a container, and moves the processing device and the camera system. The control device is characterized by comprising: A computing device for generating the control signal; as well as A communication device that outputs the control signals generated by the computing device. The computing device generates position and orientation data representing at least one of the container's position and orientation based on first image data generated by the imaging system capturing at least a portion of the container. Based on the second image data generated by the imaging system from a group of objects containing a group of objects, three-dimensional position data representing the three-dimensional positions of multiple points in the group of objects is generated, wherein the group of objects includes at least a portion of a plurality of objects housed in the container. A portion of the three-dimensional position data is selected based on the given position and attitude data to serve as the selection data. Based on the selected data, a control signal is generated to control the robot so that the processing device approaches the object so that the processing device can process one of the object objects in the group of object objects.

131. The control device as claimed in claim 130, characterized in that, The object group includes at least a portion of the object group and the container.

132. The control device as claimed in claim 130 or 131, characterized in that, The selected data does not include data representing the three-dimensional positions of multiple points representing at least a portion of the container.

133. The control device as described in any one of claims 130 to 132, characterized in that, The selected data includes data representing the three-dimensional positions of multiple points representing at least a portion of the object group.

134. The control device according to any one of claims 130 to 133, characterized in that, Based on the position and orientation data, the computing device defines a designated area within the container that includes at least a portion of the space capable of accommodating multiple of the object objects. A portion of the three-dimensional location data is selected as the selection data corresponding to the specified region.

135. The control device as claimed in claim 134, characterized in that, The object group includes at least a portion of the object group and the container. The designated area does not include multiple points representing at least a portion of the container shown in the three-dimensional position data.

136. The control device as described in claim 134 or 135, characterized in that, The designated area includes multiple points representing at least a portion of the object group housed in the container, as shown by the three-dimensional position data.

137. The control device according to any one of claims 134 to 136, characterized in that, When the container is set as the first container and the designated area is set as the first designated area, Based on the position and orientation data, the computing device transforms a second designated region within a second container (different from the first container), which includes at least a portion of a space capable of accommodating multiple objects, into the first designated region. A portion of the three-dimensional location data is selected as the selection data corresponding to the first specified region.

138. The control device as claimed in claim 137, characterized in that, The computing device, based on the position and attitude data, changes the second designated region to the first designated region by changing at least one of the position and attitude of the second designated region.

139. The control device as described in claim 137 or 138, characterized in that, The object group includes at least a portion of the object object group and the first container. The computing device changes the second designated region to the first designated region based on the position and orientation data, so that multiple points of at least a portion of the first container shown by the three-dimensional position data are not included in the first designated region.

140. The control device according to any one of claims 137 to 139, characterized in that, The computing device transforms the second designated region into the first designated region based on the position and orientation data, such that at least a portion of the points of the object group contained in the first container, as shown by the three-dimensional position data, are included in the first designated region.

141. The control device according to any one of claims 137 to 140, characterized in that, The position and orientation of the first container are at least different from the position and orientation of the second container.

142. The control device as described in any one of claims 134 to 135, characterized in that, When the container is set as the first container and the designated area is set as the first designated area, The computing device uses the position and orientation data to set at least one of the position and orientation of a second designated region within a second container (different from the first container), which includes at least a portion of a space capable of accommodating multiple objects, as at least one of the position and orientation of the first designated region. A portion of the three-dimensional location data is selected as the selection data corresponding to the first specified region.

143. The control device as claimed in claim 142, characterized in that, The position and orientation of the first container are approximately the same as those of the second container.

144. The control device according to any one of claims 137 to 143, characterized in that, The second designated area is a pre-defined initial designated area.

145. The control device as claimed in claim 144, characterized in that, The second designated area is pre-specified by the user.

146. The control device as claimed in claim 144 or 145, characterized in that, The second container is used to define the second designated area.

147. The control device according to any one of claims 144 to 146, characterized in that, Based on the position and attitude data, the computing device causes the second specified region to change from an initial position and attitude representing at least one of the initial position and initial attitude of the second specified region to a first position and attitude representing at least one of the first position and first attitude, thereby changing the second specified region to the first specified region.

148. The control device according to any one of claims 137 to 143, characterized in that, Before the object object stored in the first container is processed by the processing device, the processing device processes a plurality of the object objects stored in the second container.

149. The control device as claimed in claim 148, characterized in that, After the processing device processes the plurality of object objects stored in the second container, the first container containing the plurality of object objects is exchanged for the second container.

150. The control device as claimed in claim 148 or 149, characterized in that, When the position and attitude data is set as the first position and attitude data... Before the object housed in the first container is processed by the processing device. The second designated area is set based on second position and attitude data, which represents at least one of the position and attitude of the second container generated from third image data generated by the shooting system based on at least a portion of the second container.

151. The control device as claimed in claim 150, characterized in that, Set the object group as the first object group. Set the object group as the first object group. Set the three-dimensional position data as the first three-dimensional position data. Set the selection as the first selection data. Set the control signal as the first control signal. Before the object housed in the first container is processed by the processing device. The computing device selects a portion of a second three-dimensional position data, generated from the fourth image data produced by the imaging system that captures a second group of objects including the second object group, representing the three-dimensional positions of multiple points in the second object group, as the second selection data corresponding to the second designated region. The second object group comprises at least a portion of the multiple object groups housed in the second container. Based on the second selection data, a second control signal is generated for controlling the robot so that the processing device approaches the object so that the processing device can process one of the object objects in the second group of object objects.

152. The control device as claimed in claim 150 or 151, characterized in that, The computing device, based on the first position and attitude data, causes the second specified region to change from a second position attitude representing at least one of a second position and a second attitude to a third position attitude representing at least one of a third position and a third attitude, thereby changing the second specified region to the first specified region.

153. The control device as claimed in any one of claims 134 to 136, 142, and 143, characterized in that, When the container is set as the first container and the designated area is set as the first designated area, Based on the position and orientation data, the computing device defines a second designated region within a second container, different from the first container, that includes at least a portion of a space capable of accommodating multiple objects, as the first designated region. A portion of the three-dimensional location data is selected as the selection data corresponding to the first specified region.

154. The control device according to any one of claims 137 to 153, characterized in that, The second container is approximately the same size as the first container.

155. The control device as described in any one of claims 130 to 136, characterized in that, The computing device selects a portion of the three-dimensional position data as the selection data based on dimensional information related to the size of the container.

156. The control device as claimed in claim 155, characterized in that, Based on the size information, the computing device defines a designated area within the container that includes at least a portion of the space capable of accommodating multiple of the object objects. A portion of the three-dimensional location data is selected as the selection data corresponding to the specified region.

157. The control device according to any one of claims 137 to 152, characterized in that, The computing device, based on size difference information relating to the difference between the size of the first container and the size of a second container different from the first container, and the position and orientation data, changes a second designated region within the second container, including at least a portion of the space capable of accommodating the plurality of object objects, to the first designated region. A portion of the three-dimensional location data is selected as the selection data corresponding to the first specified region.

158. The control device as claimed in claim 157, characterized in that, The computing device changes at least one of the size of the second specified region and the position and orientation of the second specified region based on the size difference information and the position and orientation data, thereby changing the second specified region into the first specified region.

159. The control device as described in claim 157 or 158, characterized in that, The object group includes at least a portion of the object object group and the first container. The computing device transforms the second designated region into the first designated region based on the size difference information and the position and orientation data, so that multiple points of at least a portion of the first container shown in the three-dimensional position data are not included in the first designated region.

160. The control device as claimed in any one of claims 157 to 159, characterized in that, The computing device transforms the second designated region into the first designated region based on the size difference information and the position and orientation data, so that multiple points of at least a portion of the object group housed in the first container, as shown by the three-dimensional position data, are included in the first designated region.

161. The control device according to any one of claims 157 to 160, characterized in that, The computing device, based on the size difference information, causes the second designated region to change from an initial size to a first size, and, based on the position and attitude data, causes the second designated region to change from an initial position and attitude representing at least one of the initial position and initial attitude of the second designated region to a first position and attitude representing at least one of the first position and first attitude, thereby changing the second designated region to the first designated region.

162. The control device as described in any one of claims 157 to 161, characterized in that, The computing device, based on the size difference information, causes the second designated region to change from a second size to a third size, and, based on the position and attitude data, causes the second designated region to change from a second position attitude representing at least one of a second position and a second attitude to a third position attitude representing at least one of a third position and a third attitude, thereby changing the second designated region into the first designated region.

163. The control device as described in any one of claims 157 to 162, characterized in that, The second container has a different size than the first container.

164. The control device according to any one of claims 137 to 154 and 157 to 163, characterized in that, Set the position and attitude data as the first position and attitude data. After the processing device processes the plurality of object objects housed in the first container The computing device generates third position and orientation data, representing at least one of the position and orientation of the third container, based on fifth image data generated by the imaging system of at least a portion of a third container that is different from the first and second containers and contains multiple of the object objects. Based on the third position and pose data, the second designated area is changed to a third designated area within the third container that includes at least a portion of the space capable of accommodating multiple objects.

165. The control device according to any one of claims 137 to 154 and 157 to 163, characterized in that, Set the position and attitude data as the first position and attitude data. After the processing device processes the plurality of object objects housed in the first container The computing device generates third position and orientation data, representing at least one of the position and orientation of the third container, based on fifth image data generated by the imaging system of at least a portion of a third container that is different from the first and second containers and contains multiple of the object objects. Based on the third positional attitude data, the first designated area is changed to a third designated area within the third container that includes at least a portion of the space capable of accommodating multiple objects.

166. The control device as described in claim 164 or 165, characterized in that, Set the object group as the first object group. Set the object group as the first object group. Set the three-dimensional position data as the first three-dimensional position data. Set the selected data as the first selected data. The computing device selects a portion of third three-dimensional position data, generated from the sixth image data (which is captured by the imaging system and represents the three-dimensional positions of multiple points within the third object group), as third selection data corresponding to the third designated region. This third object group includes at least a portion of the multiple object groups housed in the third container. Based on the third selection data, a third control signal is generated for controlling the robot so that the processing device approaches the object so that the processing device can process one of the object objects in the third group of object objects.

167. The control device as claimed in claim 166, characterized in that, The third object group includes at least a portion of the third object group and the third container. The third designated region does not contain multiple points that constitute at least a portion of the third container as shown by the third three-dimensional location data.

168. The control device as claimed in claim 166 or 167, characterized in that, The third designated region includes multiple points representing at least a portion of the third object group housed in the third container, as shown by the third three-dimensional position data.

169. The control device as claimed in any one of claims 134 to 154 and 157 to 168, characterized in that, The designated area includes multiple local areas. At least a portion of one of the plurality of local regions differs from the other local regions of the plurality of local regions.

170. The control device according to any one of claims 130 to 169, characterized in that, The computing device selects a portion of the three-dimensional position data as the selected data based on robot position and attitude data representing at least one of the position and attitude of the reference point of the processing device and the position and attitude data.

171. The control device as claimed in any one of claims 134 to 154 and 157 to 169, characterized in that, When the specified area is set as the first specified area The computing device, based on robot position and attitude data representing at least one of the position and attitude of the reference point of the processing device, sets a fourth designated region within the space where the robot is configured, which includes at least a portion of the processing device. A portion of the three-dimensional location data is selected as the selected data that does not correspond to the fourth specified region but corresponds to the first specified region.

172. The control device as claimed in claim 170 or 171, characterized in that, The selected data does not include data representing the three-dimensional positions of multiple points representing at least a portion of the processing device and data representing the three-dimensional positions of multiple points representing at least a portion of the container.

173. The control device as described in any one of claims 170 to 174, characterized in that, The selection data includes data representing the three-dimensional positions of multiple points representing at least a portion of the object group.

174. The control device according to any one of claims 170 to 173, characterized in that, The reference point of the processing device is the tool center point of the processing device.

175. The control device as claimed in any one of claims 130 to 174, characterized in that, The processing device is a holding device capable of holding the object.

176. A control system, characterized in that, include: The control device according to any one of claims 130 to 175; and The shooting system.

177. A robot system, characterized in that, include: The control device according to any one of claims 130 to 175; The shooting system; as well as The robot.

178. A control method, Control signals are generated to control the robot, which is equipped with a processing device for handling objects and a camera system, and the processing device and the camera system are moved. Furthermore, it generates control signals for controlling a robot equipped with a processing device and a camera system for processing objects contained in a container, and moves the processing device and the camera system. The control method is characterized by comprising: Based on the first image data generated by the shooting system capturing at least a portion of the container, position and orientation data representing at least one of the position and orientation of the container are generated; Based on the second image data generated by the shooting system that includes a group of object objects, three-dimensional position data representing the three-dimensional positions of multiple points in the group of object objects is generated, wherein the group of object objects includes at least a portion of a plurality of object objects housed in the container. A portion of the three-dimensional position data is selected based on the position and attitude data to serve as the selection data; and Based on the selected data, a control signal is generated to control the robot so that the processing device approaches the object so that the processing device can process one of the object objects in the group of object objects.

179. A computer program, characterized in that, The computer is made to execute the control method of claim 178.

180. A control device, The control device generates control signals for controlling a robot equipped with a processing device for handling objects and a camera system, and moves the processing device and the camera system. The control device is characterized by comprising: A computing device for generating the control signal; as well as A communication device that outputs the control signals generated by the computing device. If, based on the imaging results of the imaging system on a group of object objects containing multiple object objects, the imaging device determines that at least one of the object objects in the group of object objects cannot be processed by the processing device, the computing device generates a control signal for controlling the robot so that the position and orientation of the imaging system relative to one of the object objects in the group of object objects change.

Images