Control device, control system, robot system, control method, and computer program

Abstract

This control device generates a control signal for controlling a robot arm comprising: a processing device that processes an object; a first imaging device that outputs first image data; and a second imaging device that outputs second image data. The control device: generates first information indicating a position and an attitude of the object, by using the first image data generated by the first imaging device capturing an image of the object; generates second information indicating a position and an attitude of the object, by using three-dimensional position data that is generated from the second image data generated by the second imaging device capturing an image of the object, and that indicates the respective three-dimensional positions of a plurality of points of the object, and three-dimensional model data of the object in the position and attitude determined on the basis of the first information; and generates the control signal on the basis of the second information.

Description

Control device, control system, robot system, control method, and computer program

[0001] The present invention relates to the technical fields of a control device, a control system, a robot system, a control method, and a computer program that can generate a control signal for controlling a robot arm, for example.

[0002] An example of a control device that calculates at least one of the position and orientation of an object that is the target of processing by a robot and controls the robot based on the calculated at least one of the position and orientation is described in Patent Document 1. Such a control device is required to accurately calculate at least one of the position and orientation of the object.

[0003] US Patent Application Publication No. 2013 / 0230235

[0004] According to a first aspect, there is provided a control device that generates a control signal for controlling a robot arm provided with a processing device that processes an object, a first imaging device that outputs first image data, and a second imaging device that outputs second image data, wherein the control signal controls the robot arm to move the processing device and the first and second imaging devices, and the control device includes an arithmetic unit that generates the control signal and an output unit that outputs the control signal generated by the arithmetic unit, wherein the arithmetic unit generates first information indicating the position and orientation of the object using the first image data generated by the first imaging device imaging the object, generates second information indicating the position and orientation of the object using three-dimensional position data generated from the second image data generated by the second imaging device imaging the object and indicating the three-dimensional positions of each of multiple points of the object, and three-dimensional model data of the object at a position and orientation determined based on the first information, and generates the control signal based on the second information.

[0005] According to a second aspect, there is provided a control device that generates a control signal for controlling a robot arm provided with a processing device that performs processing on an object, a first imaging device that outputs first image data, and a second imaging device that outputs second image data, wherein the control signal controls the robot arm to move the processing device and the first and second imaging devices, and the control device comprises an arithmetic unit that generates the control signal, and an output unit that outputs the control signal generated by the arithmetic unit, and the arithmetic unit calculates an amount of change in position and orientation of the object between the first and second times based on two sets of first image data that are generated by the first imaging device imaging the object at different first and second times, respectively, and three-dimensional position data that is generated from the second image data that is generated by the second imaging device imaging the object at a third time and indicates the three-dimensional positions of each of multiple points on the object, and generates the control signal based on the amount of change.

[0006] According to a third aspect, there is provided a control device that generates a control signal for controlling a robot arm provided with a processing device that processes at least one of a first object and a second object that differs from the first object in at least one of a position and an attitude, a first imaging device that outputs first image data, and a second imaging device that outputs second image data, wherein the control signal controls the robot arm to move the processing device and the first and second imaging devices, and the control device includes a calculation unit that generates the control signal, and an output unit that outputs the control signal generated by the calculation unit, and the calculation unit controls the first image generated by the first imaging device capturing the first and second objects. A control device is provided that determines, based on data, whether or not to select the first or second object as the object to be processed by the processing device, and if the result of the determination is that the first or second object is selected as the object, generates second information indicating the position and orientation of the object using three-dimensional position data that is generated from the second image data generated by the second imaging device imaging the first and second objects and that indicates the three-dimensional positions of each of multiple points of the object, and three-dimensional model data of the object at a position and orientation determined based on first information that indicates the position and orientation of the object generated using the first image data, and generates the control signal based on the second information.

[0007] According to a fourth aspect, there is provided a control system including the control device provided by any one of the first to third aspects, the first imaging device, and the second imaging device.

[0008] According to a fifth aspect, there is provided a robot system comprising a control device provided by any one of the first to third aspects, the first imaging device, the second imaging device, and the robot arm.

[0009] According to a sixth aspect, there is provided a control method for generating a control signal for controlling a robot arm provided with a processing device for processing an object, a first imaging device for outputting first image data, and a second imaging device for outputting second image data, the control method including: generating first information indicating the position and orientation of the object using the first image data generated by the first imaging device imaging the object; generating second information indicating the position and orientation of the object using three-dimensional position data generated from the second image data generated by the second imaging device imaging the object and indicating the three-dimensional positions of each of a plurality of points on the object, and three-dimensional model data of the object at a position and orientation determined based on the first information; and generating the control signal based on the second information, wherein the robot arm is controlled to move the processing device and the first and second imaging devices based on the generated control signal in order to process the object with the processing device.

[0010] According to a seventh aspect, there is provided a control method for generating a control signal for controlling a robot arm provided with a processing device for processing an object, a first imaging device for outputting first image data, and a second imaging device for outputting second image data, the control method including: calculating an amount of change in position and orientation of the object between the first and second times based on two sets of first image data generated by the first imaging device capturing images of the object at different first and second times, respectively, and three-dimensional position data generated from the second image data generated by the second imaging device capturing images of the object at a third time and indicating the three-dimensional positions of each of a plurality of points on the object; and generating the control signal based on the amount of change; and in order to process the object with the processing device, the control method controls the robot arm to move the processing device and the first and second imaging devices based on the generated control signal.

[0011] According to an eighth aspect, there is provided a control method for generating a control signal for controlling a robot arm provided with a processing device that performs processing on at least one of a first object and a second object that is different from the first object in at least one of a position and an attitude, a first imaging device that outputs first image data, and a second imaging device that outputs second image data, the method including the steps of: determining whether or not to select the first or second object as a target object for the processing by the processing device based on the first image data generated by the first imaging device capturing images of the first and second objects; and, when the first or second object is selected as the target object as a result of the determination, causing the second imaging device to select the first and second objects. A control method is provided which includes generating second information indicating the position and orientation of the object using three-dimensional position data generated from the second image data generated by imaging the second object and indicating the three-dimensional positions of each of multiple points of the object, and three-dimensional model data of the object at a position and orientation determined based on first information indicating the position and orientation of the object generated using the first image data, and generating the control signal based on the second information, wherein the robot arm is controlled to move the processing device and the first and second imaging devices based on the generated control signal in order to process the object with the processing device.

[0012] According to a ninth aspect, there is provided a computer program causing a computer to execute the control method provided by any one of the sixth to eighth aspects.

[0013] The functions and other advantages of the present invention will become apparent from the following detailed description of the preferred embodiments.

[0014] FIG. 1 is a block diagram showing the configuration of a robot system according to this embodiment. FIG. 2 is a side view showing the appearance of a robot according to this embodiment. FIG. 3 is a block diagram showing the configuration of a control device according to this embodiment. FIG. 4 is a flowchart showing the flow of robot control processing. FIGS. 5( a) to 5(d) are side views showing the positional relationship between a robot and a workpiece at a certain point in time during a holding process for holding a workpiece placed on a mounting device moving on a support surface. FIGS. 6( a) to 6(d) are side views showing the positional relationship between a robot and a workpiece at a certain point in time during a release process for placing a workpiece on a mounting device moving on a support surface. FIGS. 7( a) to 7(b) are side views showing the positional relationship between a robot and a workpiece at a certain point in time during a holding process for holding a workpiece placed on a mounting device stationary on a support surface. FIGS. 7( c) to 7(e) are side views showing the positional relationship between a robot and a workpiece at a certain point in time during a release process for placing a workpiece on a mounting device stationary on a support surface. FIGS. 8( a ) to 8 ( e ) are side views showing the positional relationship between the robot and the workpieces at a certain point during a period in which a holding process for holding multiple workpieces placed on the placement device one by one and a release process for placing the multiple workpieces one by one on the placement device are being performed. FIGS. 9( a ) to 9 ( e ) are side views showing the positional relationship between the robot and the workpieces at a certain point during a period in which a holding process for holding multiple workpieces placed on the placement device one by one and a release process for placing the multiple workpieces one by one on the placement device are being performed. FIG. 7 is a block diagram showing logical processing blocks for calculating the position and orientation of an object by performing matching and tracking processes. FIG. 11 is a schematic diagram showing the 2D matching process. FIG. 12 is a timing chart showing the relationship between the timing at which the imaging device generates image data and the timing at which the 2D matching unit performs the 2D matching process. FIG. 13 is a schematic diagram showing the 3D matching process. FIG. 14 shows the initial position and initial orientation of a three-dimensional model.FIG. 15 is a timing chart showing the relationship between the timing at which the imaging device generates image data and the timing at which the 3D matching unit performs 3D matching processing. FIG. 16 schematically shows 3D matching target data, which is part of the three-dimensional position data. FIG. 17 schematically shows tracking processing. FIG. 18 is a timing chart showing the relationship between the timing at which the imaging device generates image data and the timing at which the tracking unit performs tracking processing. FIG. 19 is a timing chart showing the relationship between the timing at which the 2D matching unit performs 2D matching processing, the timing at which the 3D matching unit performs 3D matching processing, and the timing at which the tracking unit performs tracking processing. FIG. 20 is a block diagram showing the configuration of a control device in a first modified example. FIG. 21 is a flowchart showing the flow of robot control processing in the first modified example. FIG. 22 is a plan view showing two objects detected by 2D matching processing. FIG. 23(a) shows an imaging device moving in a rotational manner, and FIG. 23(b) shows an imaging device moving in a translational manner. Fig. 24(a) shows a robot arm that moves an end effector toward a first object, and Fig. 24(b) shows a robot arm that moves an end effector toward a second object. Fig. 25 is a perspective view showing an imaging device that images an object from a first position and an imaging device that images the object from a second position different from the first position. Fig. 26 is a block diagram showing the configuration of a control device in a second modified example. Fig. 27 is a schematic diagram showing plane removal processing. Fig. 28 is a block diagram showing the configuration of an imaging unit. Fig. 29 is a schematic diagram showing high dynamic range synthesis processing. Fig. 30 is a schematic diagram showing noise removal processing. Fig. 31 is a block diagram showing the configuration of a robot system in a third modified example.

[0015] Hereinafter, embodiments of a control device, a control system, a robot system, a control method, and a computer program will be described with reference to the drawings. Hereinafter, embodiments of a control device, a control system, a robot system, a control method, and a computer program will be described using a robot system SYS.

[0016] (1) Configuration of the Robot System SYS First, the configuration of the robot system SYS will be described.

[0017] (1-1) Overall Configuration of Robot System SYS First, the overall configuration of the robot system SYS will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the overall configuration of the robot system SYS.

[0018] As shown in FIG. 1, the robot system SYS includes a robot 1, an imaging unit 2, a control device 3, and an end effector 13.

[0019] The robot 1 is a device capable of performing a predetermined process on an object OBJ. An example of the robot 1 is shown in FIG. 2. FIG. 2 is a side view showing the appearance of the robot 1. As shown in FIG. 2, the robot 1 includes, for example, a base 11, a robot arm 12, and a robot control device 14.

[0020] The base 11 is a component that forms the base of the robot 1. The base 11 is placed on a support surface S such as a floor surface. The base 11 may be fixed to the support surface S. Alternatively, the base 11 may be movable relative to the support surface S. As an example, the base 11 may be self-propelled on the support surface S. In this case, the base 11 may be installed on an automatic guided vehicle (AGV). Note that FIG. 2 shows an example in which the base 11 is fixed to the support surface S.

[0021] The robot arm 12 is attached to the base 11. The robot arm 12 is a device in which a plurality of links 121 are connected via joints 122. An actuator is built into the joint 122. The link 121 may be rotatable around an axis defined by the joint 122 by the actuator built into the joint 122. At least one link 121 may be extendable and contractible along the direction in which the link 121 extends. A device including the device in which a plurality of links 121 are connected via joints 122 and the base 11 may be referred to as the robot arm 12.

[0022] An end effector 13 is attached to the robot arm 12. That is, the end effector 13 is attached to the robot 1. In the example shown in FIG. 2 , the end effector 13 is attached to the tip of the robot arm 12. The end effector 13 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the end effector 13. That is, the robot 1 moves the end effector 13.

[0023] The end effector 13 is a device that performs a predetermined process (in other words, a predetermined operation) on the object OBJ. The end effector 13 may be called a processing device because it performs a predetermined process on the object OBJ.

[0024] For example, the end effector 13 may perform a holding process (in other words, a holding operation) to hold an object OBJ as an example of the predetermined process. In this case, the end effector 13 may be considered to be performing the holding process on the object OBJ that the end effector 13 is to hold. An end effector 13 capable of performing the holding process may be referred to as a holding device.

[0025] For example, as an example of the predetermined process, the end effector 13 may perform a release process (in other words, a release operation) to release (i.e., let go of) the object OBJ that it is holding. In this case, the end effector 13 may be considered to be performing the release process on the object OBJ that it is holding. When the end effector 13 is referred to as a holding device, the holding device may perform the release process on the object OBJ that it is holding. An end effector 13 that is capable of performing the release process may be referred to as a release device.

[0026] The release process may include a process of releasing (i.e., releasing) the first object OBJ held by the end effector 13 so that the first object OBJ is placed on the second object OBJ. In this case, the end effector 13 may be considered to be performing the release process on the first object OBJ held by the end effector 13. In other words, the end effector 13 may be considered to be performing the release process on the first object OBJ to be released by the end effector 13. Furthermore, the end effector 13 may be considered to be performing the release process on the second object OBJ on which the first object OBJ to be released by the end effector 13 is to be placed.

[0027] A hand gripper is an example of an end effector 13 capable of holding and releasing an object OBJ. A hand gripper is an end effector 13 that can hold an object OBJ by physically pinching the object OBJ using multiple (e.g., two, three, or four) finger members or claw members. Another example of an end effector 13 that can hold an object OBJ is at least one vacuum gripper that can hold the object OBJ by vacuum suction. Figure 2 shows an example of the end effector 13 being a hand gripper.

[0028] The robot 1 may perform a placement process (in other words, a placement operation) for placing an object OBJ at a desired position using the end effector 13 that is capable of holding and releasing the object OBJ. For example, the robot 1 may hold a first object OBJ using the end effector 13, and then perform a placement process for placing the first object OBJ held by the end effector 13 at a desired position of a second object OBJ that is different from the first object OBJ.

[0029] The robot 1 may perform a fitting process (in other words, a fitting operation) for fitting a first object OBJ into a second object OBJ different from the first object OBJ, using the end effector 13 capable of performing a holding process and a release process. For example, the robot 1 may hold the first object OBJ using the end effector 13, and then perform a fitting process for fitting the first object OBJ held by the end effector 13 into the second object OBJ different from the first object OBJ.

[0030] The object OBJ on which the end effector 13 performs a predetermined process may include a workpiece W, as shown in FIG. 2 . The workpiece W may include, for example, a part or member used to manufacture a desired product. The workpiece W may include, for example, a part or member to be processed to manufacture the desired product. The workpiece W may include, for example, a part or member to be transported to manufacture the desired product.

[0031] As shown in FIG. 2 , the object OBJ on which the end effector 13 performs a predetermined process may include a placement device T on which a workpiece W is placed. An example of the placement device T is a pallet. The placement device T may be disposed on a support surface S. The placement device T may be fixed to the support surface S. Alternatively, the placement device T may be movable relative to the support surface S. As an example, the placement device T may be self-propelled on the support surface S. Note that FIG. 2 shows an example in which the placement device T is self-propelled on the support surface S. In this case, the placement device T may be referred to as an automatic guided vehicle (AGV). When the placement device T is movable relative to the support surface S, the workpiece W placed on the placement device T also moves relative to the support surface S as the placement device T moves. Therefore, the movable placement device T may function as a moving device that moves the workpiece W. In addition, a belt conveyor may be used as the placement device T.

[0032] However, the object OBJ does not have to include the placement device T. In addition, when the placement device T is capable of self-propelling on the support surface S, the self-propelled device capable of self-propelling on the support surface S and the pallet may be separate entities. In this case, the pallet may be installed on the self-propelled device. A device including both the self-propelled device and the pallet may be called the placement device T. Alternatively, the object OBJ does not have to include at least one of the self-propelled device and the pallet. The self-propelled device may be called an automatic guided vehicle.

[0033] The placement device T may be capable of flying above the support surface S. In this case, the placement device T may be referred to as an unmanned aerial vehicle. The flying device capable of flying above the support surface S and the pallet may be separate entities. The pallet may be set in the flying device. An apparatus including the flying device and the pallet may be referred to as the placement device T. The flying device may be referred to as an unmanned aerial vehicle. The workpiece W does not have to be placed on the placement device T. For example, the workpiece W may be placed on the support surface S.

[0034] When the object OBJ includes a workpiece W and a mounting device T, the holding process described above may include a process of holding a workpiece W placed on a stationary or moving mounting device T. The holding process described above may include a process of holding a workpiece W placed on a support surface S. The release process described above may include a process of releasing a workpiece W held by the end effector 13 in order to place the workpiece W held by the end effector 13 at a desired position on a stationary or moving mounting device T. The release process described above may include a process of releasing a workpiece W held by the end effector 13 in order to place the workpiece W held by the end effector 13 at a desired position on the support surface S. The release process described above may include a process of releasing a first workpiece W held by the end effector 13 in order to fit the first workpiece W held by the end effector 13 into a second workpiece W placed on a stationary or moving mounting device T. The above-mentioned release process may include a process of releasing the first workpiece W held by the end effector 13 in order to fit the first workpiece W held by the end effector 13 into a second workpiece W placed on the support surface S.

[0035] The robot control device 14 controls the operation of the robot 1 .

[0036] Specifically, the robot controller 14 may control the movement of the robot arm 12. For example, the robot controller 14 may control the movement of the robot arm 12 so that a desired link 121 rotates around an axis defined by a desired joint 122. For example, the robot controller 14 may control the movement of the robot arm 12 so that the end effector 13 attached to the robot arm 12 is positioned at a desired position. For example, the robot controller 14 may control the movement of the robot arm 12 so that the end effector 13 attached to the robot arm 12 moves to a desired position.

[0037] In addition to controlling the operation of the robot 1, the robot controller 14 may also control the operation of the end effector 13 attached to the robot 1. For example, the robot controller 14 may control the operation of the end effector 13 so that the end effector 13 holds an object OBJ at a desired timing. That is, the robot controller 14 may control the operation of the end effector 13 so that the end effector 13 performs a holding process at a desired timing. For example, the robot controller 14 may control the operation of the end effector 13 so that the end effector 13 releases the held object OBJ at a desired timing. That is, the robot controller 14 may control the operation of the end effector 13 so that the end effector 13 performs a release process at a desired timing. If the end effector 13 is a hand gripper, the robot controller 14 may control the timing of opening and closing the hand gripper. If the end effector 13 is a vacuum gripper, the robot controller 14 may control the timing of turning on and off a vacuum device of the vacuum gripper.

[0038] Note that FIG. 2 shows an example in which the robot 1 is a robot arm 12 (i.e., a vertical articulated robot). However, the robot 1 may be a robot different from a vertical articulated robot. For example, the robot 1 may be a SCARA robot (i.e., a horizontal articulated robot). For example, the robot 1 may be a parallel link robot. For example, the robot 1 may be a dual-arm robot having two robot arms 12. For example, the robot 1 may be a Cartesian coordinate robot. For example, the robot 1 may be a cylindrical coordinate robot. The robot 1 may be referred to as a movable device. The movable device may include, in addition to the robot 1, at least one of an automatic guided vehicle and an unmanned aerial vehicle. For example, the robot 1 may be installed in at least one of an automatic guided vehicle and an unmanned aerial vehicle.

[0039] 1 again, the imaging unit 2 captures an image of an object OBJ. To capture an image of the object OBJ, the imaging unit 2 includes an imaging device 21, an imaging device 22, and a projection device 23. The imaging unit 2 may also be referred to as an imaging section.

[0040] The imaging device 21 is a camera capable of capturing an image of the object OBJ. The imaging device 21 captures an image of the object OBJ to generate image data IMG_2D. The image data IMG_2D generated by the imaging device 21 is output from the imaging device 21 to the control device 3. In this embodiment, the imaging device 21 is a monocular camera. Specifically, the imaging device 21 is capable of capturing an image of the object OBJ using a monocular camera (in other words, an image sensor). Note that the imaging device 21 is not limited to a monocular camera. The imaging device 21 may be a stereo camera capable of capturing an image of the object OBJ using two monocular cameras, or may include three or more monocular cameras. Note that the imaging device 21 may be at least one of a light field camera, a plenoptic camera, and a multispectral camera.

[0041] Like the imaging device 21, the imaging device 22 is a camera capable of capturing an image of the object OBJ. In this embodiment, the imaging device 22 is a stereo camera. Specifically, the imaging device 22 is a stereo camera capable of capturing an image of the object OBJ using two monocular cameras (in other words, two image sensors). The imaging device 22 captures an image of the object OBJ to generate image data IMG_3D. Specifically, because the imaging device 22 is a stereo camera, the imaging device 22 generates image data IMG_3D including two pieces of image data generated by the two monocular cameras. The image data IMG_3D generated by the imaging device 22 is output from the imaging device 22 to the control device 3. Note that the imaging device 22 is not limited to a stereo camera. The imaging device 22 may be a monocular camera or may include three or more monocular cameras. Note that the imaging device 22 may be at least one of a light field camera, a plenoptic camera, and a multispectral camera.

[0042] The projection device 23 is a device capable of irradiating projection light onto an object OBJ. In particular, the projection device 23 is a device capable of projecting a desired projection pattern onto the object OBJ by irradiating the object OBJ with projection light. The desired projection pattern may include, for example, a random pattern. The random pattern may be a projection pattern having a different pattern for each unit irradiation area. The random pattern may include a random dot pattern. The desired projection pattern may include, for example, a one-dimensional or two-dimensional grid pattern. The desired projection pattern may include other projection patterns. The imaging device 22 captures an image of the object OBJ onto which the projection pattern from the projection device 23 is projected. In this case, the image represented by the image data IMG_3D includes the object OBJ onto which the projection pattern is projected. On the other hand, the imaging device 21 does not need to capture an image of the object OBJ onto which the projection pattern is projected. The imaging device 21 may also capture an object OBJ onto which no projection pattern is projected. In this case, the image represented by the image data IMG_2D does not need to include an object OBJ onto which a projection pattern is projected. The image represented by the image data IMG_2D may include an object OBJ onto which no projection pattern is projected. Note that the projection light for projecting a desired projection pattern onto the object OBJ may be referred to as pattern light or structured light. In this case, the projection light may include pattern light or structured light.

[0043] It should be noted that the projection device 23 may be considered to illuminate the object OBJ with the projection light by irradiating the object OBJ with the projection light. In this case, the projection device 23 may function as an illumination device that illuminates the object OBJ. When the projection device 23 functions as an illumination device, the projection light may be referred to as illumination light. When the projection device 23 functions as an illumination device, the projection light does not need to be light that can project a desired projection pattern onto the object OBJ.

[0044] The imaging unit 2 is provided on the robot arm 12, similar to the end effector 13. That is, the imaging devices 21 and 22 and the projection device 23 are attached to the robot arm 12. For example, as shown in FIG. 2 , the imaging devices 21 and 22 and the projection device 23 may be attached to the tip of the robot arm 12, similar to the end effector 13. In this case, the imaging devices 21 and 22 and the projection device 23 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the imaging devices 21 and 22 and the projection device 23.

[0045] However, the imaging unit 2 does not have to be provided on the robot arm 12. The imaging unit 2 may be provided at any position where it can irradiate the object OBJ with projection light and capture an image of the object OBJ. At least one of the imaging devices 21, 22, and projection device 23 may be provided on the robot arm 12, and at least one of the imaging devices 21, 22, and projection device 23 may be provided at a location different from the robot arm 12. When at least one of the imaging devices 21 and 22 is provided at a location different from the robot arm 12, at least one of the imaging devices 21 and 22 may be installed on a structure such as a pillar arranged so as to be able to capture an image of the object OBJ. When the projection device 23 is provided at a location different from the robot arm 12, the projection device 23 may be installed on a structure such as a pillar arranged so as to be able to irradiate the object OBJ with projection light.

[0046] The imaging devices 21 and 22 may capture images of the object OBJ in synchronization with each other. For example, the imaging devices 21 and 22 may capture images of the object OBJ simultaneously. That is, the imaging devices 21 and 22 may capture images of the object OBJ such that the 2D imaging time at which the imaging device 21 captures the object OBJ and the 3D imaging time at which the imaging device 22 captures the object OBJ are the same. The imaging devices 21 and 22 may capture images of the object OBJ such that the 2D imaging time at which the imaging device 21 captures the object OBJ to generate image data IMG_2D and the 3D imaging time at which the imaging device 22 captures the object OBJ to generate image data IMG_3D are the same.

[0047] The imaging devices 21 and 22 may capture images of the object OBJ under the control of the control device 3. In this case, the time (in other words, the timing) at which each of the imaging devices 21 and 22 captures an image of the object OBJ may be controlled by the control device 3. For example, the control device 3 may control the imaging devices 21 and 22 so that the imaging devices 21 and 22 capture images of the object OBJ in synchronization with each other. For example, the control device 3 may control the imaging devices 21 and 22 so that the imaging devices 21 and 22 capture images of the object OBJ simultaneously. In other words, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same time.

[0048] Here, the state in which "the 2D imaging time and the 3D imaging time are the same" may include a state in which "the 2D imaging time and the 3D imaging time are literally exactly the same." The state in which "the 2D imaging time and the 3D imaging time are the same" may include a state in which "the 2D imaging time and the 3D imaging time are not exactly the same, but the time difference between the 2D imaging time and the 3D imaging time is smaller than the allowable upper limit, so the 2D imaging time and the 3D imaging time can be considered to be substantially the same." Here, the allowable upper limit may be an allowable upper limit based on a control error of the robot arm 12. For example, a time difference between the 2D imaging time and the 3D imaging time may cause an error in the calculation results of at least one of the position and orientation of the object OBJ (described later) (i.e., the accuracy of at least one of the calculated position and orientation of the object OBJ may decrease). In this case, an error in the calculation results of at least one of the position and orientation of the object OBJ may cause a control error of the robot arm 12. A control error of the robot arm 12 results in a movement error of the end effector 13, which may prevent the end effector 13 from properly performing a predetermined process on the object OBJ. The allowable upper limit may be considered equivalent to the allowable upper limit of the movement error of the end effector 13 by the robot arm 12. Furthermore, even if there is a time difference between the 2D imaging time and the 3D imaging time due to a synchronization error in the imaging processes of the imaging devices 21 and 22, for example, the 2D imaging time and the 3D imaging time may be considered to be substantially the same time. The synchronization error in the imaging processes of the imaging devices 21 and 22 may be a synchronization control error of the imaging processes of the imaging devices 21 and 22 by the control device 3.

[0049] However, the imaging devices 21 and 22 do not have to image the object OBJ at the same time. In other words, the imaging devices 21 and 22 may image the object OBJ such that the 2D imaging time at which the imaging device 21 images the object OBJ and the 3D imaging time at which the imaging device 22 images the object OBJ are different times. Note that the state in which "the 2D imaging time and the 3D imaging time are different times" may include a state in which "the 2D imaging time and the 3D imaging time cannot be considered to be substantially the same time because the time difference between the 2D imaging time and the 3D imaging time is larger than the allowable upper limit value."

[0050] In this embodiment, when the imaging devices 21 and 22 image the object OBJ during a period in which the imaging devices 21 and 22 are displaced relative to the object OBJ, the imaging devices 21 and 22 may image the object OBJ so that the 2D imaging time and the 3D imaging time are the same. In other words, when the imaging devices 21 and 22 image the object OBJ during a period in which the imaging devices 21 and 22 are displaced relative to the object OBJ, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same. The reason for this will be explained again when explaining the effects of the robot system SYS. Note that a state in which the imaging devices 21 and 22 are displaced relative to the object OBJ may mean a state in which the relative positional relationship between the imaging devices 21 and 22 and the object OBJ is changing. The state in which each of the imaging devices 21 and 22 and the object OBJ are displaced relative to each other may also mean a state in which each of the imaging devices 21 and 22 and the object OBJ are moving relative to each other.

[0051] On the other hand, when the imaging devices 21 and 22 capture images of the object OBJ during a period in which there is no relative displacement between each of the imaging devices 21 and 22 and the object OBJ, the imaging devices 21 and 22 do not need to capture images of the object OBJ so that the 2D imaging time and the 3D imaging time are the same. That is, the control device 3 does not need to control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same. For example, the imaging devices 21 and 22 may capture images of the object OBJ so that the 2D imaging time and the 3D imaging time are different. That is, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are different. Note that a state in which there is no relative displacement between each of the imaging devices 21 and 22 and the object OBJ may include a state in which the relative positional relationship between each of the imaging devices 21 and 22 and the object OBJ is not changed. The state in which there is no relative displacement between each of the imaging devices 21 and 22 and the object OBJ may include a state in which there is no relative movement between each of the imaging devices 21 and 22 and the object OBJ. The state in which there is no relative displacement between each of the imaging devices 21 and 22 and the object OBJ may include a state in which each of the imaging devices 21 and 22 and the object OBJ are stationary. Note that when the imaging devices 21 and 22 image the object OBJ during a period in which there is no relative displacement between each of the imaging devices 21 and 22 and the object OBJ, the imaging devices 21 and 22 may image the object OBJ so that the 2D imaging time and the 3D imaging time are the same. In other words, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same.

[0052] The control device 3 performs robot control processing. The robot control processing includes processing for generating a robot control signal for controlling the robot 1. Specifically, the control device 3 generates the robot control signal based on the image data IMG_2D and IMG_3D output from the imaging unit 2. In this embodiment, the control device 3 calculates at least one of the position and orientation of the object OBJ in a global coordinate system of the robot system SYS based on the image data IMG_2D and IMG_3D, and generates the robot control signal based on the calculated at least one of the position and orientation of the object OBJ. The global coordinate system is a coordinate system that serves as a reference for the robot system SYS. For example, the global coordinate system may be a coordinate system that serves as a reference for the robot 1. Note that when the global coordinate system is a coordinate system that serves as a reference for the robot 1, the global coordinate system may be referred to as a robot coordinate system.

[0053] The control device 3 may perform end effector processing in addition to performing robot control processing. The end effector control processing may include processing for generating an end effector control signal for controlling the end effector 13. Specifically, the control device 3 may generate the end effector control signal based on at least one of the calculated position and orientation of the object OBJ. The end effector processing may or may not be included in the robot control processing. In other words, the end effector control signal generated by the control device 3 may or may not be included in the robot control signal.

[0054] For convenience of explanation, the following description will be given assuming that the end effector control process is included in the robot control process (i.e., the end effector control signal is included in the robot control signal). Therefore, in the following description, the robot control process may refer to a process for generating at least one of a robot control signal and an end effector control signal. Furthermore, in the following description, the robot control signal may refer to at least one of a signal for controlling the robot 1 and a signal for controlling the end effector 13.

[0055] The robot control signal may be simply referred to as a control signal.

[0056] Furthermore, the control device 3 may calculate at least one of the position and orientation of the object OBJ in a coordinate system different from the global coordinate system of the robot system SYS (for example, in a 2D imaging coordinate system, a 3D imaging coordinate system, or a common coordinate system described below) based on the image data IMG_2D and IMG_3D, and generate a robot control signal based on at least one of the calculated position and orientation of the object OBJ.

[0057] In this way, the control device 3 and the imaging unit 2 are used to control the robot 1. Therefore, a system including the control device 3 and the imaging unit 2 may be referred to as a robot control system or a control system.

[0058] The robot control signal generated by the control device 3 is output to the robot control device 14 of the robot 1. The robot control device 14 controls the operation of the robot 1 based on the robot control signal generated by the control device 3. For this reason, the robot control signal may include a signal for controlling the operation of the robot 1.

[0059] The robot control signal may include a signal for controlling the robot arm 12. That is, the robot arm 12 may be controlled by the robot control signal. In this case, the robot control device 14 may control the operation of the robot arm 12 based on the robot control signal. For example, the robot control device 14 may control the operation of the robot arm 12 by controlling the driving of an actuator built into the joint 122 based on the robot control signal.

[0060] As described above, the robot arm 12 moves the end effector 13. In this case, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 13 is located at a desired position. The robot control signal may include a signal for controlling the robot arm 12 so that the end effector 13 moves to a desired position. The robot control signal may include a signal for controlling the robot arm 12 so that the positional relationship between the end effector 13 and the object OBJ is a desired positional relationship. In this case, the robot control device 14 may control the robot arm 12 based on the robot control signal so that the end effector 13 is located at a desired position. The robot control device 14 may control the robot arm 12 based on the robot control signal so that the end effector 13 moves to a desired position. The robot control device 14 may control the robot arm 12 based on the robot control signal so that the positional relationship between the end effector 13 and the object OBJ is a desired positional relationship.

[0061] As an example, when the end effector 13 performs a holding process to hold the object OBJ, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 13 moves toward (i.e., approaches) a holding position where the end effector 13 can hold the object OBJ. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 13 is located at the holding position. In this case, the robot control device 14 may control the robot arm 12 based on the robot control signal so that the end effector 13 moves toward (i.e., approaches) the holding position. That is, the robot control signal may control the robot arm 12 so that the end effector 13 is located at the holding position.

[0062] As another example, when performing a holding process to release an object OBJ held by the end effector 13, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 13 moves toward (i.e., approaches) a release position where the object OBJ held by the end effector 13 should be released. That is, the robot control signal may include a signal for controlling the robot arm 12 so that the end effector 13 is located at the release position. In this case, the robot control device 14 may control the robot arm 12 based on the robot control signal so that the end effector 13 moves toward (i.e., approaches) the release position. That is, the robot control signal may control the robot arm 12 so that the end effector 13 is located at the release position.

[0063] As described above, the robot control signal may include a signal for controlling the end effector 13 (i.e., an end effector control signal) in addition to or instead of a signal for controlling the robot arm 12. That is, the end effector 13 may be controlled by the robot control signal. In this case, the robot control device 14 may control the operation of the end effector 13 based on the robot control signal. For example, the robot control device 14 may control the operation of the end effector 13 by controlling the drive of an actuator that moves a hand gripper that constitutes the end effector 13 based on the robot control signal. For example, the robot control device 14 may control the operation of the end effector 13 by controlling the drive of a vacuum device of a vacuum gripper that constitutes the end effector 13 based on the robot control signal.

[0064] As an example, when the end effector 13 performs a holding process to hold the object OBJ, the robot control signal may include a signal for controlling the end effector 13 so that the end effector 13 located at the holding position holds the object OBJ. In this case, the robot control device 14 may control the end effector 13 based on the robot control signal so that the end effector 13 located at the holding position holds the object OBJ.

[0065] As another example, when a release process is performed to release the object OBJ held by the end effector 13, the robot control signal may include a signal for controlling the end effector 13 to release the object OBJ held by the end effector 13 located at the release position described above. In this case, the robot control device 14 may control the end effector 13, based on the robot control signal, to release the object OBJ held by the end effector 13 located at the release position.

[0066] The robot control signal may include a signal that can be used directly by the robot control device 14 to control the operation of the robot 1. The robot control signal may include a signal that can be used directly as a robot drive signal used by the robot control device 14 to control the operation of the robot 1. In this case, the robot control device 14 may use the robot control signal as is to control the operation of the robot 1. For example, the control device 3 may generate a drive signal for an actuator built into the joint 122 of the robot arm 12 as the robot control signal, and the robot control device 14 may use the robot control signal generated by the control device 3 as is to control the actuator built into the joint 122 of the robot arm 12. The robot control signal may include a signal that can be used directly by the robot control device 14 to control the operation of the end effector 13. The robot control signal may include a signal that can be used directly as an end effector drive signal used by the robot control device 14 to control the operation of the end effector 13. In this case, for example, the control device 3 may generate, as a robot control signal, a drive signal (end effector drive signal) for an actuator that moves a hand gripper that constitutes the end effector 13, and the robot control device 14 may use the robot control signal generated by the control device 3 as is to control the actuator of the end effector 13. For example, the control device 3 may generate, as a robot control signal, a drive signal (end effector drive signal) for driving a vacuum device of a vacuum gripper that constitutes the end effector 13, and the robot control device 14 may use the robot control signal generated by the control device 3 as is to control the vacuum device of the end effector 13.

[0067] As described above, if the robot control signal includes a signal that can be used directly by the robot control device 14 to control the operation of the robot 1, and a signal that can be used directly by the robot control device 14 to control the operation of the end effector 13, the robot 1 does not need to be equipped with a robot control device 14.

[0068] Alternatively, the robot control signal may include a signal that the robot control device 14 can use to generate a robot drive signal for controlling the operation of the robot 1. In this case, the robot control device 14 may generate a robot drive signal for controlling the operation of the robot 1 based on the robot control signal, and control the operation of the robot 1 based on the generated robot drive signal. For example, the robot control device 14 may generate a drive signal for an actuator built into the joint 122 of the robot arm 12 based on the robot control signal, and control the actuator built into the joint 122 of the robot arm 12 based on the generated signal.

[0069] The robot control signal may include a signal that can be used by the robot control device 14 to generate an end effector drive signal for controlling the operation of the end effector 13. In this case, the robot control device 14 may generate the end effector drive signal for controlling the operation of the end effector 13 based on the end effector control signal, and control the operation of the end effector 13 based on the generated end effector drive signal. For example, the robot control device 14 may generate a drive signal for an actuator or a vacuum device of the end effector 13 based on the robot control signal, and control the actuator or the vacuum device of the end effector 13 based on the generated signal.

[0070] The signal that the robot control device 14 can use to generate the robot drive signal may be a signal representing the position of the object OBJ in the global coordinate system. In this case, for example, the robot control device 14 may generate a drive signal for an actuator built into the joint 122 of the robot arm 12 based on the robot control signal so that the end effector 13 approaches the object OBJ whose position in the global coordinate system is represented by the robot control signal (i.e., so that the positional relationship between the robot 1 (end effector 13) and the object OBJ becomes a desired positional relationship), and control the operation of the robot arm 12 based on the generated drive signal. The signal that the robot control device 14 can use to generate the robot drive signal may be a signal representing a desired positional relationship between the robot 1 and the object OBJ in the global coordinate system. In this case, for example, the robot control device 14 may generate a drive signal for an actuator built into the joint 122 of the robot arm 12 based on the robot control signal so that the positional relationship between the robot 1 (end effector 13) and the object OBJ becomes the desired positional relationship represented by the robot control signal, and control the operation of the robot arm 12 based on the generated drive signal. Furthermore, the signals that the robot control device 14 can use to generate the robot drive signals may include signals representing a desired position of the end effector 13 in a global coordinate system. In this case, for example, the robot control device 14 may generate drive signals for actuators incorporated in the joints 122 of the robot arm 12 based on the robot control signals so that the end effector 13 is positioned at the desired position represented by the robot control signals (i.e., so that the positional relationship between the robot 1 (end effector 13) and the object OBJ is a desired positional relationship), and control the operation of the robot arm 12 based on the generated drive signals. As an example, the robot control device 14 may generate drive signals for actuators incorporated in the joints 122 of the robot arm 12 based on the robot control signals so that the end effector 13 moves to a holding position where the end effector 13 can hold the object OBJ located at the position represented by the robot control signals, and control the operation of the robot arm 12 based on the generated drive signals.As another example, the robot control device 14 may generate a drive signal for an actuator built into the joint 122 of the robot arm 12 based on the robot control signal so that the end effector 13, which holds the object OBJ located at the position indicated by the robot control signal, moves away from the position where the object OBJ was placed, and control the operation of the robot arm 12 based on the generated drive signal. Furthermore, signals that the robot control device 14 can use to generate the robot drive signal may include, for example, a signal representing a desired position of the tip of the robot arm 12 in the global coordinate system, or a signal representing a desired position of the imaging unit 2 in the global coordinate system. The coordinate system that serves as the reference for the robot control signal may be a coordinate system other than the global coordinate system (for example, a 2D imaging coordinate system, a 3D imaging coordinate system, or a common coordinate system, which will be described later).

[0071] (1-2) Configuration of the Control Device 3 Next, the configuration of the control device 3 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the control device 3.

[0072] 3 , the control device 3 includes a calculation device 31, a storage device 32, and a communication device 33. The control device 3 may further include an input device 34 and an output device 35. However, the control device 3 does not necessarily have to include at least one of the input device 34 and the output device 35. The calculation device 31, the storage device 32, the communication device 33, the input device 34, and the output device 35 may be connected via a data bus 36.

[0073] The arithmetic device 31 includes, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a field programmable gate array (FPGA). The arithmetic device 31 reads a computer program. For example, the arithmetic device 31 may read a computer program stored in the storage device 32. For example, the arithmetic device 31 may read a computer program stored in a computer-readable, non-transitory storage medium using a storage medium reading device (not shown) included in the control device 3. The arithmetic device 31 may acquire (i.e., download or read) the computer program from a device (not shown) located outside the control device 3 via the communication device 33 (or another communication device). The arithmetic device 31 executes the read computer program. As a result, a logical function block for executing the processing to be performed by the control device 3 (for example, the robot control processing described above) is realized within the arithmetic device 31. In other words, the arithmetic device 31 can function as a controller for realizing the logical function block for executing the processing to be performed by the control device 3.

[0074] A computational model that can be constructed by machine learning may be implemented in the computational device 31 by the computational device executing a computer program. An example of a computational model that can be constructed by machine learning is a computational model including a neural network (so-called artificial intelligence (AI)). In this case, learning of the computational model may include learning of parameters of the neural network (e.g., at least one of a weight and a bias). The computational device 31 may execute a robot control process using the computational model. In other words, the operation of executing the robot control process may include the operation of executing the robot control process using the computational model. Note that a computational model that has been constructed by offline machine learning using teacher data may be implemented in the computational device 31. Furthermore, the computational model implemented in the computational device 31 may be updated by online machine learning on the computational device 31. Alternatively, the calculation device 31 may perform robot control processing using a calculation model implemented in a device external to the calculation device 31 (i.e., a device provided outside the control device 3) in addition to or instead of the calculation model implemented in the calculation device 31.

[0075] The recording medium for recording the computer program executed by the arithmetic unit 31 may be at least one of the following: a CD-ROM, CD-R, CD-RW, a flexible disk, an MO, a DVD-ROM, a DVD-RAM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, and an optical disk such as Blu-ray (registered trademark), a magnetic medium such as a magnetic tape, a magneto-optical disk, a semiconductor memory such as a USB memory, and any other medium capable of storing a program. The recording medium may also include a device capable of recording a computer program (for example, a general-purpose device or a dedicated device in which a computer program is implemented in an executable state in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program may be realized by a logical processing block realized within the arithmetic device 31 (i.e., a computer) when the arithmetic device 31 executes the computer program, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) included in the arithmetic device 31, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.

[0076] 3 shows an example of logical functional blocks realized in the arithmetic device 31 for executing robot control processing. As shown in Fig. 3, a three-dimensional position data generation unit 311, a position and orientation calculation unit 312, and a signal generation unit 313 are realized in the arithmetic device 31. Note that the processing performed by each of the three-dimensional position data generation unit 311, the position and orientation calculation unit 312, and the signal generation unit 313 will be described in detail later with reference to Fig. 4 etc., and therefore will not be described here.

[0077] The storage device 32 can store desired data. For example, the storage device 32 may temporarily store a computer program executed by the arithmetic device 31. The storage device 32 may temporarily store data that the arithmetic device 31 temporarily uses when the arithmetic device 31 is executing a computer program. The storage device 32 may also store data that the control device 3 stores long-term. The storage device 32 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 32 may include a non-temporary recording medium.

[0078] The communication device 33 is capable of communicating with each of the robot 1 and the imaging unit 2 via a communication network (not shown). Alternatively, the communication device 33 may be capable of communicating with another device different from the robot 1 and the imaging unit 2, in addition to or instead of at least one of the robot 1 and the imaging unit 2, via a communication network (not shown). In this embodiment, the communication device 33 may receive (i.e., acquire) image data IMG_2D and IMG_3D from the imaging unit 2. Furthermore, the communication device 33 may transmit (i.e., output) a robot control signal to the robot 1. The communication device 33 that outputs the robot control signal to the robot 1 may be referred to as an output unit.

[0079] The input device 34 is a device that accepts information input to the control device 3 from outside the control device 3. For example, the input device 34 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by a user of the control device 3. For example, the input device 34 may include a recording medium reading device that can read information recorded as data on a recording medium that can be externally attached to the control device 3.

[0080] It should be noted that information can be input as data to the control device 3 from a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an input device that accepts information input to the control device 3 from outside the control device 3.

[0081] The output device 35 is a device that outputs information to the outside of the control device 3. For example, the output device 35 may output information as an image. That is, the output device 35 may include a display device (a so-called display) that can display an image. For example, the output device 35 may output information as sound. That is, the output device 35 may include an audio device (a so-called speaker) that can output sound. For example, the output device 35 may output information on paper. That is, the output device 35 may include a printing device (a so-called printer) that can print desired information on paper. For example, the output device 35 may output information as data to a recording medium that can be externally attached to the control device 3.

[0082] The control device 3 can output information as data to a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an output device that outputs information to a device external to the control device 3.

[0083] (2) Robot Control Processing Next, the robot control processing performed by the control device 3 will be described.

[0084] (2-1) Flow of Robot Control Processing First, the flow of the robot control processing will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of the robot control processing.

[0085] As shown in FIG. 4 , the control device 3 acquires image data IMG_2D from the imaging device 21 using the communication device 33 (step S1). Specifically, the imaging device 21 captures an image of the object OBJ at a predetermined 2D imaging rate. For example, the imaging device 21 may capture an image of the object OBJ at a 2D imaging rate of tens to hundreds (e.g., 500 times) per second. As a result, the imaging device 21 generates image data IMG_2D at a period corresponding to the predetermined 2D imaging rate. For example, the imaging device 21 may generate tens to hundreds (e.g., 500) pieces of image data IMG_2D per second. The control device 3 acquires the image data IMG_2D each time the imaging device 21 generates image data IMG_2D. In other words, the control device 3 may acquire tens to hundreds (e.g., 500) pieces of image data IMG_2D per second.

[0086] The imaging device 21 may capture an image of an object other than the object OBJ, in addition to the object OBJ on which the robot 1 performs a predetermined process. For example, if both the object OBJ and the other object are included in the imaging range (field of view) of the imaging device 21, the imaging device 21 may capture images of both the object OBJ and the other object. As a result, the imaging device 21 may generate image data IMG_2D that shows an image in which both the object OBJ and the other object are captured.

[0087] The control device 3 further acquires image data IMG_3D from the imaging device 22 using the communication device 33 (step S1). Specifically, the imaging device 22 captures an image of the object OBJ at a predetermined 3D imaging rate. The 3D imaging rate is the same as the 2D imaging rate. However, the 3D imaging rate may be different from the 2D imaging rate. For example, the imaging device 22 may capture an image of the object OBJ at a 3D imaging rate that captures an image of the object OBJ tens to hundreds of times (e.g., 500 times) per second. As a result, the imaging device 22 generates image data IMG_3D at a period corresponding to the predetermined 3D imaging rate. For example, the imaging device 22 may generate tens to hundreds of pieces of image data IMG_3D per second (e.g., 500 pieces). The control device 3 acquires the image data IMG_3D each time the imaging device 22 generates image data IMG_3D. That is, the control device 3 may acquire several tens to several hundreds (for example, 500) of image data IMG_3D per second.

[0088] The imaging device 22 may capture an image of an object other than the object OBJ, in addition to the object OBJ on which the robot 1 performs a predetermined process. For example, if both the object OBJ and the other object are included in the imaging range (field of view) of the imaging device 22, the imaging device 22 may capture images of both the object OBJ and the other object. As a result, the imaging device 22 may generate image data IMG_3D that shows an image in which both the object OBJ and the other object are captured.

[0089] Each time the control device 3 acquires image data IMG_3D, the three-dimensional position data generator 311 generates three-dimensional position data WSD based on the acquired image data IMG_3D (step S2). Furthermore, the three-dimensional position data generator 311 outputs the generated three-dimensional position data WSD to the position and orientation calculator 312.

[0090] The three-dimensional position data WSD is data indicating the three-dimensional position of the object OBJ reflected in the image data IMG_3D. For example, the three-dimensional position data WSD may be data indicating the three-dimensional position of at least a portion of the surface of the object OBJ. In particular, the three-dimensional position data WSD is data indicating the three-dimensional position of each of multiple points on the object OBJ. For example, the three-dimensional position data WSD may be data indicating the three-dimensional position of each of multiple points on the surface of the object OBJ. For example, the three-dimensional position data WSD may be data indicating the three-dimensional position of each of multiple points corresponding to multiple portions on the surface of the object OBJ.

[0091] Specifically, as described above, the image represented by the image data IMG_3D reflects the three-dimensional shape of at least a portion of the surface of the object OBJ onto which the projection pattern is projected. In this case, the projection pattern reflected in the image represented by the image data IMG_3D reflects the three-dimensional shape of at least a portion of the surface of the object OBJ onto which the projection pattern is projected. The shape of the projection pattern reflected in the image represented by the image data IMG_3D reflects the three-dimensional shape of at least a portion of the surface of the object OBJ onto which the projection pattern is projected. Therefore, the three-dimensional position data generation unit 311 can calculate the three-dimensional shape of at least a portion of the surface of the object OBJ based on the projection pattern reflected in the image represented by the image data IMG_3D. The three-dimensional shape of at least a portion of the surface of the object OBJ essentially indicates the three-dimensional position of each of multiple points on the object OBJ. This is because each of the multiple points on the object OBJ is contained within the surface of the object OBJ. Therefore, the process of calculating the three-dimensional shape of at least a part of the surface of the object OBJ may be considered to be substantially equivalent to the process of calculating the three-dimensional position of each of multiple points on the object OBJ. Therefore, the three-dimensional position data generator 311 can generate the three-dimensional position data WSD based on the image data IMG_3D.

[0092] Furthermore, considering that each of the multiple points of object OBJ is contained within the surface of object OBJ, the three-dimensional position data WSD indicating the three-dimensional position of each of the multiple points of object OBJ may be considered to be essentially equivalent to three-dimensional shape data indicating the three-dimensional shape of at least a portion of object OBJ (in particular, the three-dimensional shape of at least a portion of the surface of object OBJ).

[0093] To generate the three-dimensional position data WSD, the three-dimensional position data generation unit 311 may calculate the parallax by associating each portion (e.g., each pixel) of the images represented by each of the two image data sets included in the image data IMG_3D. Specifically, in this association, the three-dimensional position data generation unit 311 may calculate the parallax by associating each portion of the projection pattern reflected in the images represented by each of the two image data sets (i.e., each portion of the projection pattern reflected in each image). The three-dimensional position data generation unit 311 may calculate the three-dimensional position of each of multiple points on the object OBJ using a well-known method based on the principle of triangulation using the calculated parallax. As a result, three-dimensional position data WSD indicating the three-dimensional position of each of the multiple points on the object OBJ is generated. In this case, the accuracy of the parallax calculation is higher when associating each portion of the images in which the projection pattern is reflected (i.e., each portion of the reflected projection pattern) rather than when associating each portion of the images in which the projection pattern is not reflected. Therefore, the accuracy of the generated three-dimensional position data WSD (that is, the accuracy of calculating the three-dimensional positions of each of the multiple points of the object OBJ) is improved.

[0094] The three-dimensional position data WSD may be any data as long as it can indicate the three-dimensional position of each of the multiple points of the object OBJ. In other words, the three-dimensional position data WSD may be any data that directly or indirectly indicates the three-dimensional position of each of the multiple points of the object OBJ. For example, the three-dimensional position data WSD may include coordinate information indicating the three-dimensional position of each of the multiple points of the object OBJ. For example, the three-dimensional position data WSD may include information indicating the three-dimensional shape of at least a portion of the object OBJ.

[0095] Because the control device 3 sequentially acquires multiple pieces of image data IMG_3D as described above, the three-dimensional position data generator 311 also sequentially generates multiple pieces of three-dimensional position data WSD from the multiple pieces of image data IMG_3D. In this case, the three-dimensional position data WSD generated from the image data IMG_3D at a certain time may indicate the three-dimensional positions of each of multiple first points of the object OBJ. The three-dimensional position data WSD generated from the image data IMG_3D at another time different from the one time may indicate the three-dimensional positions of each of multiple second points of the object OBJ. The multiple first points of the object OBJ may be the same as the multiple second points of the object OBJ. Alternatively, at least one of the multiple first points of the object OBJ may be different from at least one of the multiple second points of the object OBJ.

[0096] An example of the three-dimensional position data WSD is depth image data. The depth image data is image data in which depth information is associated with each pixel of a depth image represented by the depth image data in addition to brightness information. The depth information is information indicating the distance (i.e., depth) between each part of the object OBJ reflected in each pixel and the imaging device 22. Note that the depth image data may also be image data in which the brightness information of each pixel indicates the depth of each part of the object OBJ (i.e., the distance between each part of the object OBJ and the imaging device 22). The three-dimensional position data generation unit 311 may calculate the distance between each part of the object OBJ reflected in the image represented by the image data IMG_3D and the imaging device 22 based on the projection pattern reflected in the image represented by the image data IMG_3D, and generate a depth image by associating the calculated distance with each pixel of the image represented by the image data IMG_3D as depth information.

[0097] Another example of the three-dimensional position data WSD is point cloud data. The point cloud data is data that represents a collection of points in three-dimensional space that correspond to each part of the object OBJ that appears in the image represented by the image data IMG_3D. The three-dimensional position data generator 311 may generate the point cloud data based on the depth image data and the camera parameters of the imaging device 22.

[0098] As described above, the three-dimensional position data WSD is generated based on the image data IMG_3D, and therefore may be data indicating the three-dimensional position of the object OBJ in a 3D imaging coordinate system, which will be described later.

[0099] As described above, when the imaging range (field of view) of the imaging device 22 includes an object other than the object OBJ on which the robot 1 performs a predetermined process, the image data IMG_3D represents an image in which both the object OBJ and the other object are captured. In this case, the three-dimensional position data WSD may represent not only the three-dimensional position of the object OBJ captured in the image data IMG_3D, but also the three-dimensional position of the other object captured in the image data IMG_3D. Even in this case, as long as the three-dimensional position data WSD includes data indicating the three-dimensional position of the object OBJ, the three-dimensional position data WSD may be considered to represent the three-dimensional position of each of multiple points on the object OBJ. This is because, even if the three-dimensional position data WSD includes data indicating the three-dimensional position of the other object, the three-dimensional position data WSD still includes data indicating the three-dimensional position of the object OBJ.

[0100] Thereafter, the position and orientation calculation unit 312 calculates at least one of the position and orientation of the object OBJ based on the image data IMG_2D acquired in step S1 and the three-dimensional position data WSD generated in step S2 (step S3). As a result, the position and orientation calculation unit 312 generates position and orientation information POI0 indicating at least one of the position and orientation of the object OBJ (step S3).

[0101] In step S3, the position and orientation calculation unit 312 calculates at least one of the position and orientation of the object OBJ in the global coordinate system. That is, the position and orientation calculation unit 312 generates position and orientation information POI0 indicating at least one of the position and orientation of the object OBJ in the global coordinate system. The global coordinate system is a coordinate system that serves as a reference for the robot system SYS. Specifically, the global coordinate system is a coordinate system used to control the robot 1. For example, the robot control device 14 may control the robot arm 12 so that the end effector 13 is positioned at a desired position in the global coordinate system. The global coordinate system is a coordinate system defined by an X-axis (GL), a Y-axis (GL), and a Z-axis (GL) that are orthogonal to each other. The X-axis (GL) may be an axis along a horizontal plane. The Y-axis (GL) may be an axis along a horizontal plane. The Z-axis (GL) may be an axis perpendicular to the horizontal plane. The Z-axis (GL) may be an axis extending in the direction of gravity. The X-axis, Y-axis, and Z-axis shown in FIG. 2 may be the X-axis (GL), Y-axis (GL), and Z-axis (GL), respectively. The origin of the global coordinate system does not have to be the origin of the X-axis (GL), Y-axis (GL), and Z-axis (GL) shown in FIG. 2. For example, the origin of the global coordinate system may be set at any position on the base 11 in FIG. 2. For example, the origin of the global coordinate system may be set at any position on the support surface S. For example, the origin of the global coordinate system may be set at any position on the contact surface of the base 11 with the support surface S (for example, the center or center of gravity of the contact surface).

[0102] The position and orientation calculation unit 312 may calculate, as the position of the object OBJ in the global coordinate system, at least one of a position Tx(GL) of the object OBJ in an X-axis direction (GL) parallel to the X-axis (GL), a position Ty(GL) of the object OBJ in a Y-axis direction (GL) parallel to the Y-axis (GL), and a position Tz(GL) of the object OBJ in a Z-axis direction (GL) parallel to the Z-axis (GL). The position and orientation calculation unit 312 may calculate, as the orientation of the object OBJ in the global coordinate system, at least one of a rotation amount Rx(GL) of the object OBJ around the X-axis (GL), a rotation amount Ry(GL) of the object OBJ around the Y-axis (GL), and a rotation amount Rz(GL) of the object OBJ around the Z-axis (GL). This is because the rotation amount Rx(GL) of the object OBJ around the X-axis (GL), the rotation amount Ry(GL) of the object OBJ around the Y-axis (GL), and the rotation amount Rz(GL) of the object OBJ around the Z-axis (GL) are equivalent to the parameters representing the orientation of the object OBJ around the X-axis (GL), the parameters representing the orientation of the object OBJ around the Y-axis (GL), and the parameters representing the orientation of the object OBJ around the Z-axis (GL), respectively. For this reason, in the following description, the amount of rotation Rx(GL) of the object OBJ around the X-axis (GL), the amount of rotation Ry(GL) of the object OBJ around the Y-axis (GL), and the amount of rotation Rz(GL) of the object OBJ around the Z-axis (GL) will be referred to as the orientation Rx(GL) of the object OBJ around the X-axis (GL), the orientation Ry(GL) of the object OBJ around the Y-axis (GL), and the orientation Rz(GL) of the object OBJ around the Z-axis (GL), respectively.

[0103] Note that the orientation Rx(GL) of the object OBJ around the X-axis (GL), the orientation Ry(GL) of the object OBJ around the Y-axis (GL), and the orientation Rz(GL) of the object OBJ around the Z-axis (GL) may be considered to indicate the position of the object OBJ in the rotation direction around the X-axis (GL), the position of the object OBJ in the rotation direction around the Y-axis (GL), and the position of the object OBJ in the rotation direction around the Z-axis (GL), respectively. In other words, the orientation Rx(GL) of the object OBJ around the X-axis (GL), the orientation Ry(GL) of the object OBJ around the Y-axis (GL), and the orientation Rz(GL) of the object OBJ around the Z-axis (GL) may all be considered to be parameters representing the position of the object OBJ.

[0104] 4, the position and orientation calculation unit 312 may calculate at least one of the position Tx(GL), position Ty(GL), position Tz(GL), orientation Rx(GL), orientation Ry(GL), and orientation Rz(GL) as the position and orientation of the object OBJ in the global coordinate system. Note that the process of calculating at least one of the position and orientation of the object OBJ in the global coordinate system in step S3 of Fig. 4 (i.e., the process of generating position and orientation information POI0) will be described in detail later with reference to Fig. 10 etc., and therefore will not be described here.

[0105] Thereafter, the signal generator 313 generates a robot control signal using the position and orientation information POI0 generated in step S3 (step S4). For example, the signal generator 313 may generate a robot control signal so that the robot 1 (end effector 13) can perform a predetermined process on the object OBJ. For example, the signal generator 313 may generate a robot control signal so that the positional relationship between the robot 1 (end effector 13) and the object OBJ becomes a desired positional relationship. For example, the signal generator 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the positional relationship between the robot 1 (end effector 13) and the object OBJ becomes a desired positional relationship. For example, the signal generator 313 may generate a robot control signal so that the end effector 13 performs a predetermined process on the object OBJ when the positional relationship between the robot 1 and the object OBJ becomes the desired positional relationship. For example, the signal generating unit 313 may generate a robot control signal for controlling the operation of the end effector 13 so as to perform a predetermined process on the object OBJ when the positional relationship between the robot 1 and the object OBJ reaches a desired positional relationship. As already described above, the robot control signal for controlling the operation of the end effector 13 may also be referred to as an end effector control signal.

[0106] The signal generation unit 313 may generate a robot control signal under feedback control using the position and orientation information POI0 generated in step S3. The signal generation unit 313 may generate a robot control signal under P (Proportional) control using the position and orientation information POI0. The signal generation unit 313 may generate a robot control signal under PI (Proportional-Integral) control using the position and orientation information POI0. The signal generation unit 313 may generate a robot control signal under PID (Proportional-Integral-Differential) control using the position and orientation information POI0.

[0107] As an example, each of FIGS. 5( a) to 5(d) is a side view showing the positional relationship between the robot 1 and the workpiece W at a certain point during a holding process for holding a workpiece W (i.e., another example of an object OBJ) placed on a placement device T#1 (i.e., an example of an object OBJ), such as an AGV (Automatic Guided Vehicle), moving on a support surface S. In this case, the control device 3 may generate position and orientation information POI0 indicating at least one of the position and orientation of the workpiece W to be held, and generate a robot control signal using the generated position and orientation information POI0. For example, as shown in FIG. 5(a), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 13 moves toward the space directly above the moving workpiece W. As shown in FIG. 5( b), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 13 located directly above the moving workpiece W approaches the workpiece W until it can hold the workpiece W while remaining located directly above the workpiece W. As shown in FIG. 5( c), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 13 so that the end effector 13 located in a position where it can hold the moving workpiece W holds the workpiece W while following the moving workpiece W. As shown in FIG. 5( d), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 13 so that the end effector 13 holding the workpiece W moves away from the moving mounting device T#1 while holding the workpiece W.

[0108] When the robot 1 holds a workpiece W placed on the placement device T#1 using the end effector 13, the robot 1 may perform a release process to release the held workpiece W so that the held workpiece W is placed on a placement device T#2 different from the placement device T#1. That is, the robot 1 may perform a placement process (or a fitting process) to place the workpiece W on the placement device T#2 by successively performing a holding process and a release process. In this case, the control device 3 may generate position and orientation information POI0 indicating at least one of the position and orientation of the placement device T#2 that is the target of the release process, and may generate a robot control signal using the generated position and orientation information POI0. For example, each of FIGS. 6( a) to 6(d) is a side view showing the positional relationship between the robot 1 and the workpiece W at a certain point during a release process to place the workpiece W on the placement device T#2 moving on the support surface S. In this case, as shown in FIG. 6( a), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 13 holding the workpiece W moves to the space directly above the moving placement device T#2 while still holding the workpiece W. As shown in FIG. 6( b), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 13 located directly above the moving placement device T#2 and holding the workpiece W approaches the placement device T#2 until it can place the workpiece W on the placement device T#2, while remaining located directly above the placement device T#2 and holding the workpiece W. As shown in FIG. 6( c), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 13 so that the end effector 13 located in a position where it can place the workpiece W on the placement device T#2 places the workpiece W on the placement device T#2 (i.e., releases the workpiece W it was holding) while following the moving placement device T#2. As shown in Figure 6 (d), the signal generating unit 313 may generate a robot control signal to control the operation of the robot arm 12 so that the end effector 13, which has finished placing the workpiece W on the placement device T#2, moves away from the placement device T#2.

[0109] When a release process is performed (particularly when the release process is performed to perform a placement process or a fitting process), the control device 3 may generate position and orientation information POI0 indicating at least one of the position and orientation of the placement device T#2 that is the first target of the release process, as well as position and orientation information POI0 indicating at least one of the position and orientation of the workpiece W that is the second target of the release process. That is, the control device 3 may generate position and orientation information POI0 indicating at least one of the position and orientation of the workpiece W that is not yet held by the end effector 13 before the end effector 13 holds the workpiece W, and may also generate position and orientation information POI0 indicating at least one of the position and orientation of the workpiece W that is held by the end effector 13 after the end effector 13 holds the workpiece W.

[0110] When a release process is performed, before releasing the workpiece W held by the end effector 13 through the release process, the control device 3 may generate a robot control signal using the position and orientation information POI0 indicating at least one of the position and orientation of the workpiece W held by the end effector 13, in addition to the position and orientation information POI0 regarding the mounting device T#2. That is, the control device 3 may generate a robot control signal using the position and orientation information POI0 indicating at least one of the position and orientation of the workpiece W held by the end effector 13 during at least a portion of the period during which the end effector 13 holds the workpiece W. For example, the signal generator 313 may generate a robot control signal for controlling the robot arm 12 to move the workpiece W held by the end effector 13 to a desired position (e.g., a position where the workpiece W is to be released). In this case, the robot 1 can appropriately move the workpiece W held by the end effector 13 to the desired position, compared to when the position and orientation information POI0 regarding the workpiece W is not used. This is because the position of the workpiece W held by the end effector 13 becomes known information to the control device 3. Furthermore, for example, the signal generating unit 313 may generate a robot control signal for controlling the end effector 13 to change the posture of the workpiece W held by the end effector 13 to a desired posture. In this case, compared to when the position and posture information POI0 regarding the workpiece W is not used, the robot 1 can use the end effector 13 to place or fit the workpiece W in a desired posture on the mounting device T. This is because at least one of the position and posture of the workpiece W held by the end effector 13 becomes known information to the control device 3. If the posture of the workpiece W is not the desired posture, there is a possibility that the workpiece W will collide with the mounting device T. However, in this embodiment, the robot 1 can be controlled so that the posture of the workpiece W is the desired posture, and therefore there is no or low possibility that the workpiece W will collide with the mounting device T.In addition, not only when a release process is performed, but also in any situation where the end effector 13 is holding the workpiece W, the control device 3 may generate a robot control signal using position and orientation information POI0 that indicates at least one of the position and orientation of the workpiece W held by the end effector 13.

[0111] At least one of the placement device T#1 shown in FIGS. 5( a) to 5(d) and the placement device T#2 shown in FIGS. 6( a) to 6(d) does not necessarily have to move on the support surface S. For example, FIGS. 7( a) and 7(b) are side views showing the positional relationship between the robot 1 and the workpiece W at a certain point during a holding process for holding the workpiece W placed on the placement device T#1 that is stationary on the support surface S. In this case, as shown in FIG. 7( a), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 13 approaches the stationary workpiece W until it is able to hold the stationary workpiece W. As shown in FIG. 7( b), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 13 so that the end effector 13, which is positioned at a position where it can hold the stationary workpiece W, holds the workpiece W. 7(c) to 7(e) are side views showing the positional relationship between the robot 1 and the workpiece W at a certain point during a period in which a release process is being performed to place the workpiece W on the placement device T#2 that is stationary on the support surface S. In this case, as shown in FIG. 7(c), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 13 holding the workpiece W approaches the placement device T#2 while still holding the workpiece W until it can place the workpiece W on the placement device T#2. As shown in FIG. 7(d), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 13 so that the end effector 13, which is positioned to place the workpiece W on the placement device T#2, places the workpiece W on the placement device T#2 (i.e., releases the workpiece W that it was holding). As shown in Figure 7 (e), the signal generating unit 313 may generate a robot control signal to control the operation of the robot arm 12 so that the end effector 13, which has finished placing the workpiece W on the placement device T#2, moves away from the placement device T#2.

[0112] A plurality of workpieces W may be placed on the placement device T#1. For example, the plurality of workpieces W may be placed on the placement device T#1 so that they are arranged according to a certain standard on the placement device T#1. For example, the plurality of workpieces W may be placed on the placement device T#1 so that they are randomly stacked on the placement device T#1. In this case, the robot 1 may perform a holding process to selectively hold a desired workpiece W from the plurality of workpieces W placed on the placement device T#1. In particular, the robot 1 may perform a holding process to hold the plurality of workpieces W placed on the placement device T#1 one by one in order.

[0113] Furthermore, the robot 1 may perform a release process to place multiple workpieces W on the placement device T#2. That is, the robot 1 may perform a placement process to place multiple workpieces W placed on the placement device T#1 one by one on the placement device T#2 (or another placement device) by performing a holding process and a release process. In this case, the robot 1 may perform a release process to place multiple workpieces W one by one on the placement device T#2 so that the multiple workpieces W are lined up according to a certain standard on the placement device T#2. The robot 1 may perform a release process to place multiple workpieces W one by one on the placement device T#2 so that the multiple workpieces W are randomly stacked on the placement device T#2.

[0114] As an example, each of FIGS. 8( a) to 8(e) and 9(a) to 9(e) is a side view showing the positional relationship between the robot 1 and the workpiece W at a certain point during a period in which a holding process for holding two workpieces W#1 and W#2 placed on the placement device T#1 in order one by one and a release process for placing the two workpieces W#1 and W#2 in order one by one on the placement device T#2 are being performed. In this case, as shown in FIG. 8(a), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 13 approaches the workpiece W#2 until it is able to hold either one of the workpieces W#1 or W#2 (in the example shown in FIG. 8(a)), workpiece W#2. Thereafter, as shown in FIG. 8(b), the signal generation unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 13 so that the end effector 13, positioned in a position where it can hold the workpiece W#2, holds the workpiece W#2. 8(c), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 13 holding the workpiece W#2 approaches the placement device T#2 while still holding the workpiece W#2 until it can place the workpiece W#2 on the placement device T#2. Then, as shown in FIG. 8(d), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 13 so that the end effector 13, which is positioned so that it can place the workpiece W#2 on the placement device T#2, places the workpiece W#2 on the placement device T#2 (i.e., releases the workpiece W#2 it was holding). Then, as shown in FIG. 8(e), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 13, which has placed the workpiece W#2 on the placement device T#2, moves away from the placement device T#2.9( a), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 13 approaches the remaining one of the workpieces W#1 and W#2 (workpiece W#1 in the example shown in FIG. 9( a)) until it is able to hold the remaining one of the workpieces W#1 and W#2. Then, as shown in FIG. 9( b), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 13 so that the end effector 13, which is positioned so that it can hold the workpiece W#1, holds the workpiece W#1. Then, as shown in FIG. 9( c), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 13, which is holding the workpiece W#1, approaches the placement device T#2 until it is able to place the workpiece W#1 on the placement device T#2 while still holding the workpiece W#1. 9(d), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 and the end effector 13 so that the end effector 13, which is positioned at a position where it can place the workpiece W#1 on the placement device T#2, places the workpiece W#1 on the placement device T#2 (i.e., releases the workpiece W#1 that it was holding). Then, as shown in FIG. 9(e), the signal generating unit 313 may generate a robot control signal for controlling the operation of the robot arm 12 so that the end effector 13, which has finished placing the workpiece W#1 on the placement device T#2, moves away from the placement device T#2.

[0115] The robot 1 may perform a holding process to hold, one by one, a plurality of workpieces W placed on a placement device T#1 that moves on the support surface S. Alternatively, the robot 1 may perform a holding process to hold, one by one, a plurality of workpieces W placed on a placement device T#1 that is stationary on the support surface S. The robot 1 may perform a release process to place, one by one, a plurality of workpieces W on a placement device T#2 that moves on the support surface S. The robot 1 may perform a release process to place, one by one, a plurality of workpieces W on a placement device T#2 that is stationary on the support surface S.

[0116] 4, the signal generator 313 outputs the robot control signal generated in step S4 to the robot 1 (particularly, the robot controller 14) using the communication device 33. As a result, the robot controller 14 controls at least one of the operation of the robot 1 (for example, the operation of the robot arm 12) and the operation of the end effector 13 based on the robot control signal.

[0117] Thereafter, the control device 3 repeats the series of processes from step S1 to step S4 until it determines to end the robot control process (step S5). That is, the control device 3 continues to acquire image data IMG_2D and IMG_3D from the imaging devices 21 and 22, respectively, even during the period when the operation of at least one of the robot arm 12 and the end effector 13 is being controlled based on the robot control signal.

[0118] In this case, because the operation of at least one of the robot arm 12 and the end effector 13 is controlled based on the robot control signal as described above, each of the imaging devices 21 and 22 may capture an image of the object OBJ during relative movement between the object OBJ and the imaging devices 21 and 22. For example, each of the imaging devices 21 and 22 may capture an image of the object OBJ while the object OBJ is stationary and the imaging devices 21 and 22 are moving. For example, each of the imaging devices 21 and 22 may capture an image of the object OBJ while the object OBJ is moving and the imaging devices 21 and 22 are stationary. For example, each of the imaging devices 21 and 22 may capture an image of the object OBJ while the object OBJ is moving and the imaging devices 21 and 22 are moving. In other words, the control device 3 may continue to perform the robot control process shown in FIG. 4 during relative movement between the object OBJ and the imaging devices 21 and 22 (i.e., while at least one of the imaging devices 21 and 22 and the object OBJ is moving). As a result, even during the period when the operation of the robot 1 is being controlled based on the robot control signal, the control device 3 can generate (i.e., update) new position and orientation information POI0 indicating at least one of the position and orientation of the object OBJ based on the newly acquired image data IMG_2D and IMG_3D.

[0119] However, each of the imaging devices 21 and 22 may capture an image of the object OBJ while the object OBJ is stationary and while the imaging devices 21 and 22 are stationary. The control device 3 may perform the robot control process shown in FIG. 4 while the imaging devices 21 and 22 and the object OBJ are stationary.

[0120] Note that the position and orientation calculation unit 312 does not have to calculate at least one of the position and orientation of the object OBJ in the global coordinate system in step S3. That is, the position and orientation calculation unit 312 does not have to generate position and orientation information POI0 indicating at least one of the position and orientation of the object OBJ in the global coordinate system. For example, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the object OBJ in a coordinate system different from the global coordinate system (e.g., a 2D imaging coordinate system, a 3D imaging coordinate system, or a common coordinate system, which will be described later) in step S3. That is, the position and orientation calculation unit 312 may generate position and orientation information POI0 indicating at least one of the position and orientation of the object OBJ in a coordinate system different from the global coordinate system (e.g., a 2D imaging coordinate system, a 3D imaging coordinate system, or a common coordinate system, which will be described later). In this case, in step S4, the signal generation unit 313 may generate a robot control signal using position and orientation information POI0 indicating at least one of the position and orientation of the object OBJ in a coordinate system different from the global coordinate system calculated in step S3.

[0121] (2-2) Details of the Process for Calculating At Least One of the Position and Orientation of the Object OBJ (i.e., the Process for Generating Position and Orientation Information POI0) Next, details of the process for calculating at least one of the position and orientation of the object OBJ in step S3 of Fig. 4 (i.e., the process for generating position and orientation information POI0) will be described. Note that in the following description, as an example, details of the process for calculating both the position and orientation of the object OBJ (i.e., the process for generating position and orientation information POI0 indicating both the position and orientation of the object OBJ) will be described. However, the position and orientation calculation unit 312 may calculate at least one of the position and orientation of the object OBJ by performing an operation similar to the process for calculating both the position and orientation of the object OBJ.

[0122] In this embodiment, the position and orientation calculation unit 312 calculates the position and orientation of the object OBJ by performing a matching process using the image data IMG_2D and the three-dimensional position data WSD, and a tracking process using the image data IMG_2D and the three-dimensional position data WSD. Hereinafter, the process of calculating the position and orientation of the object OBJ by performing the matching process and the tracking process will be described with reference to Fig. 10. Fig. 10 is a block diagram showing logical processing blocks for calculating the position and orientation of the object OBJ by performing the matching process and the tracking process.

[0123] 10 , the position and orientation calculation unit 312 includes a 2D matching unit 3121, a 3D matching unit 3122, a tracking unit 3123, and a coordinate conversion unit 3125. The 2D matching unit 3121 performs 2D matching processing, which is an example of matching processing. The 3D matching unit 3122 performs 3D matching processing, which is an example of matching processing. The tracking unit 3123 performs tracking processing. The coordinate conversion unit 3125 performs position and orientation calculation processing, which calculates the position and orientation of the object OBJ in the global coordinate system, based on the results of the matching processing and the tracking processing. Therefore, the 2D matching processing, 3D matching processing, tracking processing, and position calculation processing will be described below in that order.

[0124] (2-2-1) 2D Matching Process Performed by the 2D Matching Unit 3121 The 2D matching unit 3121 performs 2D matching processing to calculate the position and orientation of the object OBJ in the 2D imaging coordinate system. That is, by performing 2D matching processing, the 2D matching unit 3121 generates position and orientation information POI1 that indicates the position and orientation of the object OBJ in the 2D imaging coordinate system. The 2D imaging coordinate system is the coordinate system of the imaging device 21. For example, the 2D imaging coordinate system is a coordinate system based on the imaging device 21. Note that because the 2D imaging coordinate system is the coordinate system of the imaging device 21, the position and orientation information POI1 may be considered to indicate the position and orientation of the object OBJ relative to the imaging device 21. The position and orientation information POI1 may be considered to indicate the position and orientation of the object OBJ as seen from the imaging device 21.

[0125] The 2D imaging coordinate system is a coordinate system defined by an X-axis (2D), a Y-axis (2D), and a Z-axis (2D) that are orthogonal to each other. At least one of the X-axis (2D), the Y-axis (2D), and the Z-axis (2D) may be an axis along the optical axis AX21 (see FIG. 2 ) of the optical system (particularly, the terminal optical element such as an objective lens) included in the imaging device 21. Note that the optical axis AX21 may be considered to be the optical axis of the imaging device 21. In the following description, an example will be described in which the Z-axis (2D) is an axis along the optical axis of the optical system included in the imaging device 21.

[0126] The 2D matching unit 3121 performs a matching process using the image data IMG_2D as the 2D matching process. The matching process using the image data IMG_2D is a matching process using the image data IMG_2D and the two-dimensional model data IMG_2M.

[0127] The two-dimensional model data IMG_2M is data representing a two-dimensional model WM2 of the object OBJ. The two-dimensional model data IMG_2M is data representing a two-dimensional model WM2 having a two-dimensional shape that serves as a reference for the object OBJ. In this embodiment, the two-dimensional model data IMG_2M is image data representing a two-dimensional image that serves as a reference for the object OBJ. More specifically, the two-dimensional model data IMG_2M is image data representing a two-dimensional image that includes the two-dimensional model WM2 of the object OBJ. The two-dimensional model data IMG_2M is image data representing a two-dimensional image that includes the two-dimensional model WM2 having a two-dimensional shape that serves as a reference for the object OBJ.

[0128] The two-dimensional model data IMG_2M may be, for example, two-dimensional image data representing two-dimensional models WM2 of multiple object OBJs, each generated by virtually projecting a three-dimensional model WM3 of the object OBJ from multiple different directions onto virtual planes perpendicular to each of the multiple different directions. The three-dimensional model WM3 of the object OBJ is a three-dimensional model having a reference three-dimensional shape of the object OBJ. One example of the three-dimensional model WM3 of the object OBJ is a CAD (Computer Aided Design) model of the object OBJ. Another example of the three-dimensional model WM3 of the object OBJ is a three-dimensional model having the same shape as the three-dimensional shape of the object OBJ obtained by measuring the actual object OBJ in advance. The three-dimensional model WM3 may be generated in advance by the three-dimensional position data generation unit 311 based on image data IMG_3D generated by using the imaging device 22 to capture an image of the object OBJ onto which a projection pattern from the projection device 23 is projected. Alternatively, the three-dimensional model WM3 may be generated in advance using a well-known three-dimensional shape measurement device different from the robot system SYS. The actual object OBJ whose shape is measured in advance may be a reference or non-defective object OBJ.

[0129] The two-dimensional model data IMG_2M may be image data representing a two-dimensional image generated by capturing an image of an actual object OBJ in advance. In this case, the two-dimensional model data IMG_2M may be image data representing multiple two-dimensional images generated by capturing images of the actual object OBJ from multiple different shooting directions. In this case, image data IMG_2D generated by capturing an image of the object OBJ with the imaging device 21 may be used as the two-dimensional model data IMG_2M. Alternatively, image data generated by capturing an image of the object OBJ with an imaging device other than the robot system SYS may be used as the two-dimensional model data IMG_2M. The object OBJ captured in the two-dimensional image represented by the image data IMG_2D as the two-dimensional model data IMG_2M may be referred to as a two-dimensional model WM2 of the object OBJ. Note that the actual object OBJ captured in advance may be a reference or non-defective object OBJ.

[0130] 11 , the 2D matching unit 3121 may perform a matching process on the image indicated by the image data IMG_2D using the two-dimensional image indicated by the two-dimensional model data IMG_2M as a template image. Specifically, the 2D matching unit 3121 may perform, as the matching process, an object detection process that detects an object OBJ indicated by the template image in the image indicated by the image data IMG_2D. In other words, the 2D matching unit 3121 may perform, as the matching process, an object detection process that detects an object OBJ in the image indicated by the image data IMG_2D by detecting a similar image portion in the image indicated by the image data IMG_2D that is similar to the template image. Note that the 2D matching process (in this case, the object detection process) itself may be the same as an existing matching process. For example, the 2D matching unit 3121 may perform the 2D matching process using a well-known method such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed-Up Robust Feature).

[0131] The 2D matching unit 3121 may change from one template image to another template image having a different projection direction or imaging direction, and may perform at least one of parallel translation, enlargement, reduction, and / or rotation of the two-dimensional model WM2 of the object OBJ reflected in the template image, so that characteristic locations (e.g., at least one of the characteristic points and edges) of the entire two-dimensional model WM2 of the object OBJ reflected in the template image approach (typically, match) the characteristic locations of the entire object OBJ reflected in the image indicated by the image data IMG_2D. In other words, the 2D matching unit 3121 may change the positional relationship between the coordinate system of the 2D model data IMG_2M (e.g., the coordinate system of a CAD model) and the 2D imaging coordinate system of the imaging device 21 that captured the object OBJ (generated the image data IMG_2D) so that characteristic locations in the entire 2D model WM2 of the object OBJ reflected in the template image approach (typically coincide) with characteristic locations in the entire object OBJ reflected in the image represented by the image data IMG_2D. As a result, the 2D matching unit 3121 can identify the positional relationship between the coordinate system of the 2D model data IMG_2M and the 2D imaging coordinate system. Thereafter, the 2D matching unit 3121 may calculate the position and orientation of the object OBJ in the 2D imaging coordinate system from the position and orientation of the object OBJ in the coordinate system of the 2D model data IMG_2M based on the positional relationship between the coordinate system of the 2D model data IMG_2M and the 2D imaging coordinate system.

[0132] The 2D matching unit 3121 may calculate a matching similarity, which is the similarity between the template image (i.e., the two-dimensional model WM2 of the object OBJ) and the image represented by the image data IMG_2D (particularly, the image portion to which the object OBJ of the template image is fitted), when characteristic locations throughout the two-dimensional model WM2 of the object OBJ reflected in the template image are brought closer to characteristic locations throughout the object OBJ reflected in the image represented by the image data IMG_2D. The 2D matching unit 3121 may change from one template image to another template image having a different projection direction or imaging direction, and may perform at least one of translation, enlargement, reduction, and / or rotation of the two-dimensional model WM2 of the object OBJ reflected in the template image, so as to maximize the matching similarity. As a result, under conditions where the matching similarity is maximized, the image portion of the image represented by the image data IMG_2D into which the two-dimensional model WM2 of the object OBJ in the template image is fitted is detected as a similar image portion in the image represented by the image data IMG_2D that is similar to the template image. In other words, the object OBJ represented by the similar image portion is detected. Note that the matching similarity may be considered equivalent to the correlation degree indicating the correlation between the template image and the image represented by the image data IMG_2D. Note that the correlation degree can also be considered an index indicating the correlation between the two-dimensional model WM2 of the object OBJ reflected in the template image and the object OBJ reflected in the image represented by the image data IMG_2D. Note that the matching similarity may also be referred to as a matching score.

[0133] The 2D matching unit 3121 may change the positional relationship between the coordinate system of the two-dimensional model data IMG_2M (e.g., the coordinate system of a CAD model) and the 2D imaging coordinate system of the imaging device 21 that captured the object OBJ, so that a characteristic location in a part of the two-dimensional model WM2 of the object OBJ reflected in the template image approaches (typically matches) a characteristic location in a part of the object OBJ reflected in the image indicated by the image data IMG_2D. Even in this case, the 2D matching unit 3121 may calculate the matching similarity.

[0134] An example of a situation in which 2D matching processing is performed to bring a characteristic location in a portion of the two-dimensional model WM2 closer to a characteristic location in a portion of the object OBJ reflected in the image represented by the image data IMG_2D is a situation in which the object OBJ is so large that as the object OBJ approaches the imaging device 21, a portion of the object OBJ falls outside the imaging range (field of view) of the imaging device 21. Specifically, when the object OBJ is large, the shorter the distance between the object OBJ and the imaging device 21 (i.e., the closer the object OBJ is to the imaging device 21), the smaller the proportion of the entire object OBJ that is occupied by the portion of the object OBJ located within the imaging range (field of view) of the imaging device 21. In other words, the proportion of the entire object OBJ that is actually reflected in the image represented by the image data IMG_2D decreases. If 2D matching processing is performed using the entire two-dimensional model WM2 under such circumstances, the 2D matching processing is likely to fail. Note that the state in which the "2D matching process fails" referred to here may refer to a state in which the object OBJ indicated by the template image cannot be detected in the image represented by the image data IMG_2D, even though the object OBJ is reflected in the image represented by the image data IMG_2D. Specifically, the smaller the proportion of the portion of the object OBJ actually reflected in the image represented by the image data IMG_2D, the higher the likelihood of the 2D matching process failing. This is because the 2D matching process is performed using the entire two-dimensional model WM2 of the object OBJ, even though only a portion of the object OBJ is reflected in the image represented by the image data IMG_2D. Therefore, in this case, the 2D matching unit 3121 may perform the 2D matching process using a portion of the two-dimensional model WM2 that corresponds to a portion of the object OBJ included in the imaging range (field of view) of the imaging device 21. As a result, even if the object is large, the 2D matching unit 3121 can detect the object OBJ indicated by the template image in the image represented by the image data IMG_2D.

[0135] When the 2D matching process is performed using a portion of the two-dimensional model WM2, a portion of the two-dimensional model WM2 of the object OBJ that appears in each template image having a different projection direction or imaging direction may be designated (in other words, selected) in advance as a portion of the two-dimensional model WM2 to be used in the 2D matching process. For example, when the object OBJ is large enough that a portion of the object OBJ falls outside the imaging range (field of view) of the imaging device 21 when the object OBJ approaches the imaging device 21, a portion of the two-dimensional model WM2 that corresponds to a portion of the object OBJ that is expected to be included in the imaging range (field of view) of the imaging device 21 may be designated in advance as a portion of the two-dimensional model WM2 to be used in the 2D matching process.

[0136] Alternatively, a portion of the two-dimensional model WM2 used in the 2D matching process may be specified (in other words, selected) while the 2D matching process is being performed. That is, one range of the two-dimensional model WM2 used in the 2D matching process may be changed while the 2D matching process is being performed. For example, considering that the shorter the distance between the object OBJ and the imaging device 21, the smaller the proportion of the portion of the object OBJ located within the imaging range (field of view) of the imaging device 21 to the entire object OBJ, the range of the portion of the two-dimensional model WM2 used in the 2D matching process may be changed depending on the distance between the object OBJ and the imaging device 21. For example, the portion of the two-dimensional model WM2 used in the 2D matching process may be specified so that the range of the portion of the two-dimensional model WM2 used in the 2D matching process becomes smaller (in other words, narrower) as the distance between the object OBJ and the imaging device 21 becomes shorter.

[0137] A single portion of the two-dimensional model WM2 may be designated as part of the two-dimensional model WM2 used in the 2D matching process. Alternatively, multiple portions of the two-dimensional model WM2 may be designated as part of the two-dimensional model WM2 used in the 2D matching process. For example, multiple portions of the two-dimensional model WM2, each of which can be used as a characteristic part of the two-dimensional model WM2, may be designated as part of the two-dimensional model WM2 used in the 2D matching process. In this case, improved accuracy of the 2D matching process can be expected compared to when a single portion of the two-dimensional model WM2 is designated as part of the two-dimensional model WM2 used in the 2D matching process. In other words, the object OBJ indicated by the template image is more likely to be detected in the image indicated by the image data IMG_2D.

[0138] The portion of the two-dimensional model WM2 used in the 2D matching process may be designated by a user of the robot system SYS. For example, the user may designate a portion of the two-dimensional model WM2 displayed on a display (not shown). For example, the user may estimate a portion of the object OBJ that is assumed to be included in the imaging range (field of view) of the imaging device 21, and designate a portion of the two-dimensional model WM2 that corresponds to the estimated portion of the object OBJ. Alternatively, the designation of a portion of the two-dimensional model WM2 used in the 2D matching process may be automatically performed by the control device 3. For example, the control device 3 may estimate a portion of the object OBJ that is assumed to be included in the imaging range (field of view) of the imaging device 21, and designate a portion of the two-dimensional model WM2 that corresponds to the estimated portion of the object OBJ. In this case, the control device 3 may estimate a portion of the object OBJ that is assumed to be included in the imaging range (field of view) of the imaging device 21, based on information regarding at least one of, for example, the direction of relative movement between the object OBJ and the imaging device 21 (or the direction in which the robot 1 approaches the object OBJ), the distance between the object OBJ and the imaging device 21 (or the distance between the robot 1 and the object OBJ), and the imaging range (field of view) of the imaging device 21.

[0139] Depending on the image represented by the image data IMG_2D, there is a possibility that multiple objects OBJ represented by the template image are captured in the image. In this case, the 2D matching unit 3121 may sequentially perform a process of bringing characteristic features of the two-dimensional model WM2 of the object OBJ captured in the template image closer to characteristic features of the object OBJ captured in the image represented by the image data IMG_2D, targeting the multiple objects OBJ captured in the image represented by the image data IMG_2D. Thereafter, the 2D matching unit 3121 may select, as the target for subsequent processing, one object OBJ among the multiple objects OBJ captured in the image represented by the image data IMG_2D that has the highest matching similarity. Alternatively, the 2D matching unit 3121 may select, as the target for subsequent processing, one object OBJ among the multiple objects OBJ captured in the image represented by the image data IMG_2D that has a matching similarity that exceeds a matching determination threshold.

[0140] As will be described in a first modified example below, the matching determination threshold may be set to an appropriate value that allows appropriate distinction, based on the matching similarity, between a state in which the object OBJ detected by the 2D matching process is identical to the object OBJ on which the end effector 13 is to perform a predetermined process and a state in which the object OBJ detected by the 2D matching process is different from the object OBJ on which the end effector 13 is to perform a predetermined process. In other words, the matching determination threshold may be set to an appropriate value that allows appropriate distinction, based on the matching similarity, between a state in which the object OBJ detected by the 2D matching process is identical to the object OBJ reflected in the template image and a state in which the object OBJ detected by the 2D matching process is different from the object OBJ reflected in the template image.

[0141] Note that the method for calculating the position and orientation of the object OBJ is not limited to the matching process using the image data IMG_2D described above. The 2D matching unit 3121 may calculate the position and orientation of the object OBJ using another well-known method for calculating the position and orientation of the object OBJ using the image data IMG_2D. The method for calculating the position and orientation of the object OBJ may be a well-known method for calculating the position and orientation of the object OBJ based on the image data IMG_2D without using the two-dimensional model data IMG_2M, or a well-known method for calculating the position and orientation of the object OBJ using data other than the two-dimensional model data IMG_2M and the image data IMG_2D. For example, the method for calculating the position and orientation of the object OBJ may be a method for calculating the position and orientation of the object OBJ based on the image data IMG_2D using machine learning or deep learning. In this case, a prediction model for outputting the position and orientation of the object OBJ when image data IMG_2D is input may be constructed in advance by machine learning or deep learning, and the position and orientation of the object OBJ may be calculated by inputting the image data IMG_2D into this prediction model. This prediction model may be stored in the 2D matching unit 3121. Note that the 2D matching unit 3121 may read out this prediction model stored in the storage device 32.

[0142] The 2D matching unit 3121 may calculate, as the position of the object OBJ in the 2D imaging coordinate system, at least one of a position Tx(2D) of the object OBJ in an X-axis direction (2D) parallel to the X-axis (2D), a position Ty(2D) of the object OBJ in a Y-axis direction (2D) parallel to the Y-axis (2D), and a position Tz(2D) of the object OBJ in a Z-axis direction (2D) parallel to the Z-axis (2D). The 2D matching unit 3121 may calculate, as the orientation of the object OBJ in the 2D imaging coordinate system, at least one of a rotation amount Rx(2D) of the object OBJ around the X-axis (2D), a rotation amount Ry(2D) of the object OBJ around the Y-axis (2D), and a rotation amount Rz(2D) of the object OBJ around the Z-axis (2D).

[0143] In the following description, similarly to the rotation amounts Rx, Ry, and Rz described above, in order to unify terminology, the rotation amount Rx(2D) of the object OBJ around the X-axis (2D), the rotation amount Ry(2D) of the object OBJ around the Y-axis (2D), and the rotation amount Rz(2D) of the object OBJ around the Z-axis (2D) will be referred to as the orientation Rx(2D) of the object OBJ around the X-axis (2D), the orientation Ry(2D) of the object OBJ around the Y-axis (2D), and the orientation Rz(2D) of the object OBJ around the Z-axis (2D), respectively.

[0144] Furthermore, the orientation Rx(2D) of the object OBJ around the X-axis (2D), the orientation Ry(2D) of the object OBJ around the Y-axis (2D), and the orientation Rz(2D) of the object OBJ around the Z-axis (2D) may be considered to indicate the position of the object OBJ in the rotation direction around the X-axis (2D), the position of the object OBJ in the rotation direction around the Y-axis (2D), and the position of the object OBJ in the rotation direction around the Z-axis (2D), respectively. In other words, the orientation Rx(2D) of the object OBJ around the X-axis (2D), the orientation Ry(2D) of the object OBJ around the Y-axis (2D), and the orientation Rz(2D) of the object OBJ around the Z-axis (2D) may all be considered to be parameters representing the position of the object OBJ.

[0145] Every time the 2D matching unit 3121 generates position and orientation information POI1, it outputs the generated position and orientation information POI1 to the 3D matching unit 3122. As an example, the 2D matching unit 3121 may calculate all of the positions and orientations of 6 DOF (Degree Of Freedom) (that is, position Tx(2D), position Ty(2D), position Tz(2D), orientation Rx(2D), orientation Ry(2D), and orientation Rz(2D)), and output the position and orientation information POI1 indicating the calculated positions and orientations of the 6 DOF to the 3D matching unit 3122. As another example, the 2D matching unit 3121 may calculate the positions and orientations of the 6 DOF, and output position and orientation information POI1 indicating part of the calculated positions and orientations of the 6 DOF (i.e., part of the position Tx(2D), position Ty(2D), position Tz(2D), orientation Rx(2D), orientation Ry(2D), and orientation Rz(2D)) to the 3D matching unit 3122. As another example, the 2D matching unit 3121 may calculate part of the positions and orientations of the 6 DOF, and output position and orientation information POI1 indicating part of the calculated positions and orientations of the 6 DOF to the 3D matching unit 3122.

[0146] For convenience of explanation, the following description will be given of an example in which the 2D matching unit 3121 calculates the position Tx(2D), position Ty(2D), position Tz(2D), orientation Rx(2D), orientation Ry(2D), and orientation Rx(2D) as the position and orientation of the object OBJ, as shown in Fig. 10. That is, for convenience of explanation, the following description will be given of an example in which the 2D matching unit 3121 generates position and orientation information POI1 indicating the position Tx(2D), position Ty(2D), position Tz(2D), orientation Rx(2D), orientation Ry(2D), and orientation Rx(2D) and outputs the information to the 3D matching unit 3122, as shown in Fig. 10.

[0147] The 2D matching unit 3121 may calculate the position and orientation of the object OBJ at a predetermined 2D matching cycle. That is, the 2D matching unit 3121 may calculate the position and orientation of the object OBJ each time a time equivalent to a 2D matching cycle has elapsed. Conversely, it may be considered that a time equivalent to the 2D matching cycle is required from the start to completion of the 2D matching process by the 2D matching unit 3121. In this case, the 2D matching unit 3121 may output position and orientation information POI1 indicating the position and orientation of the object OBJ to the 3D matching unit 3122 at a predetermined 2D matching cycle.

[0148] As shown in FIG. 12 , the 2D matching period may be longer than the 2D imaging period corresponding to the 2D imaging rate at which the imaging device 21 captures an image of the object OBJ. For example, the 2D matching period may be a period at which the 2D matching unit 3121 can perform the 2D matching process several dozen to several tens of times (e.g., 15 to 30 times) per second. In this case, the imaging device 21 generates new image data IMG_2D between the start and completion of the 2D matching process by the 2D matching unit 3121. In this case, the 2D matching unit 3121 may not start a new 2D matching process using the newly generated image data IMG_2D until the completion of the 2D matching process that has already started. After completing the 2D matching process that has already started, the 2D matching unit 3121 may start a new 2D matching process using the image data IMG_2D that is the most recent at that time. Note that even if the 2D matching unit 3121 does not complete the 2D matching process that has already started, when the imaging device 21 generates new image data IMG_2D, the 2D matching unit 3121 may start a new 2D matching process using the image data IMG_2D newly generated by the imaging device 21. Note that the timing at which the image data IMG_2D shown in Fig. 12 is generated may be considered to be the timing at which the imaging device 21 captures an image of the object OBJ in order to generate the image data IMG_2D.

[0149] However, the 2D matching period does not have to be longer than the 2D imaging period. For example, the 2D matching period may be shorter than the 2D imaging period. For example, the 2D matching period may be the same as the 2D imaging period.

[0150] (2-2-2) 3D Matching Process Performed by the 3D Matching Unit 3122 The 3D matching unit 3122 performs 3D matching processing to calculate the position and orientation of the object OBJ in the 3D imaging coordinate system. That is, by performing 3D matching processing, the 3D matching unit 3122 generates position and orientation information POI2 that indicates the position and orientation of the object OBJ in the 3D imaging coordinate system. The 3D imaging coordinate system is the coordinate system of the image capture device 22. For example, the 3D imaging coordinate system is a coordinate system based on the image capture device 22. Note that because the 3D imaging coordinate system is a coordinate system based on the image capture device 22, the position and orientation information POI2 may be considered to indicate the position and orientation of the object OBJ relative to the image capture device 22. The position and orientation information POI2 may be considered to indicate the position and orientation of the object OBJ as seen from the image capture device 22.

[0151] The 3D imaging coordinate system is a coordinate system defined by an X-axis (3D), a Y-axis (3D), and a Z-axis (3D), which are orthogonal to each other. At least one of the X-axis (3D), the Y-axis (3D), and the Z-axis (3D) may be an axis along the optical axis AX22 (see FIG. 2 ) of the optical system (particularly, the terminal optical element such as an objective lens) included in the imaging device 22. The optical axis AX22 may also be considered to be the optical axis of the imaging device 22. Here, if the imaging device 22 is a stereo camera including two monocular cameras as described above, the optical axis AX22 may be the optical axis of the optical system included in either of the two monocular cameras. In other words, the optical axis AX22 may be the optical axis of either of the two monocular cameras. In the following description, an example will be described in which the Z-axis (3D) is an axis along the optical axis of the optical system included in the imaging device 22.

[0152] The 3D matching unit 3122 performs a matching process using the three-dimensional position data WSD as at least a part of the 3D matching process. That is, the 3D matching unit 3122 performs a 3D matching process that includes a matching process using the three-dimensional position data WSD. The matching process using the three-dimensional position data WSD is a matching process using the three-dimensional position data WSD and the three-dimensional model data WMD.

[0153] The three-dimensional model data WMD is data representing a three-dimensional model WM3 of the object OBJ. In other words, the three-dimensional model data WMD is data representing a three-dimensional model WM3 having a reference three-dimensional shape of the object OBJ. The three-dimensional model WM3 may be a CAD model of the object OBJ, which is an example of the three-dimensional model WM3 of the object OBJ. The three-dimensional model WM3 may be a three-dimensional model having a shape similar to the three-dimensional shape of the object OBJ obtained by measuring the three-dimensional shape of the actual object OBJ in advance. In this case, the three-dimensional model data WMD may be generated in advance by the three-dimensional position data generation unit 311 based on image data IMG_3D generated by using the imaging device 22 to capture an image of the object OBJ onto which the projection pattern from the projection device 23 is projected. Alternatively, the three-dimensional model data WMD may be generated in advance by shape measurement using a well-known three-dimensional shape measurement device different from the robot system SYS. In this case, the three-dimensional model data WMD may be depth image data representing the three-dimensional model WM3 of the object OBJ. The three-dimensional model data WMD may be point cloud data representing a three-dimensional model WM3 of the object OBJ. Note that the actual object OBJ that is imaged or measured in advance to generate the three-dimensional model data WMD may be a reference or non-defective object OBJ.

[0154] As shown in FIG. 13 , the 3D matching unit 3122 may perform a matching process on the three-dimensional position data WSD using the three-dimensional model WM3 indicated by the three-dimensional model data WMD as a template model. The matching process itself using the three-dimensional position data WSD may be the same as an existing matching process. For example, the 3D matching unit 3122 may perform a matching process using the three-dimensional position data WSD using a well-known method including at least one of RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), ICP (Iterative Closest Point), and DSO (Direct Sparse Odometry). Therefore, a detailed description of the matching process using the three-dimensional position data WSD will be omitted, but an overview will be provided below.

[0155] For example, the 3D matching unit 3122 may translate, enlarge, reduce, and / or rotate the three-dimensional model WM3 represented by the three-dimensional model data WMD within the 3D imaging coordinate system so that characteristic locations throughout the three-dimensional model WM3 represented by the three-dimensional model data WMD approach (typically, coincide) with characteristic locations throughout the object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD. In other words, the 3D matching unit 3122 may change the positional relationship between the coordinate system of the three-dimensional model data WMD (e.g., the coordinate system of a CAD model) and the 3D imaging coordinate system so that characteristic locations throughout the object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD approach (typically, coincide) with characteristic locations throughout the three-dimensional model WM3 represented by the three-dimensional model data WMD. As a result, the 3D matching unit 3122 can identify the positional relationship between the coordinate system of the three-dimensional model data WMD and the 3D imaging coordinate system. Thereafter, the 3D matching unit 3122 may calculate the position and orientation of the object OBJ in the 3D imaging coordinate system from the position and orientation of the three-dimensional model WM3 in the coordinate system of the three-dimensional model data WMD, based on the positional relationship between the coordinate system of the three-dimensional model data WMD and the 3D imaging coordinate system. Note that the coordinate system of the three-dimensional model data WMD may also be referred to as the coordinate system of the three-dimensional model WM3 indicated by the three-dimensional model data WMD.

[0156] The 3D matching unit 3122 may change the positional relationship between the coordinate system of the three-dimensional model data WMD (e.g., the coordinate system of a CAD model) and the 3D imaging coordinate system of the imaging device 22 that captured the object OBJ, so that a characteristic location in a part of the three-dimensional model WM3 indicated by the three-dimensional model data WMD approaches (typically coincides with) a characteristic location in a part of the object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD.

[0157] An example of a situation in which 3D matching processing is performed to bring a characteristic location in a portion of the three-dimensional model WM3 closer to a characteristic location in a portion of the object OBJ whose three-dimensional position data WSD indicates is a situation in which the object OBJ is so large that as the object OBJ approaches the imaging device 22, a portion of the object OBJ falls outside the imaging range (field of view) of the imaging device 22. Specifically, when the object OBJ is large, the shorter the distance between the object OBJ and the imaging device 22 (i.e., the closer the object OBJ is to the imaging device 22), the smaller the proportion of the entire object OBJ that is occupied by the portion of the object OBJ located within the imaging range (field of view) of the imaging device 22. In other words, the proportion of the entire object OBJ that is occupied by the portion of the object OBJ that is actually captured in the image represented by the image data IMG_3D decreases. Therefore, the proportion of the entire object OBJ that is occupied by the portion of the object OBJ whose three-dimensional position data WSD generated from the image data IMG_3D indicates is reduced. If the 3D matching process is performed using the entire three-dimensional model WM3 under such circumstances, the 3D matching process is likely to fail. Note that the state in which the "3D matching process fails" here may also refer to a state in which the object OBJ represented by the three-dimensional model WM3 cannot be detected from the three-dimensional position data WSD, even though the three-dimensional position data of the object OBJ is included in the three-dimensional position data WSD. Specifically, the smaller the proportion of the portion of the object OBJ that is actually reflected in the image represented by the image data IMG_3D (i.e., the smaller the proportion of the portion of the object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD generated from the image data IMG_3D), the higher the likelihood of the 3D matching process failing. This is because the 3D matching process is performed using the entire three-dimensional model WM3 of the object OBJ, even though only a portion of the object OBJ is reflected in the image represented by the image data IMG_3D. Therefore, in this case, the 3D matching unit 3122 may perform the 3D matching process using a part of the three-dimensional model WM3 corresponding to a part of the object OBJ included in the imaging range (field of view) of the imaging device 22 (a part of the object OBJ whose three-dimensional position data WSD indicates the three-dimensional position).As a result, even if the object is large, the 3D matching unit 3122 can detect the object OBJ indicated by the three-dimensional model WM3 from the three-dimensional position data WSD.

[0158] When the 3D matching process is performed using a portion of the three-dimensional model WM3, a portion of the three-dimensional model WM3 of the object OBJ may be designated (in other words, selected) in advance as a portion of the three-dimensional model WM3 to be used in the 3D matching process. For example, when the object OBJ is large enough that a portion of the object OBJ falls outside the imaging range (field of view) of the imaging device 22 when the object OBJ approaches the imaging device 22, a portion of the three-dimensional model WM3 corresponding to a portion of the object OBJ that is expected to be included in the imaging range (field of view) of the imaging device 22 may be designated in advance as a portion of the three-dimensional model WM3 to be used in the 3D matching process.

[0159] Alternatively, a portion of the three-dimensional model WM3 used in the 3D matching process may be specified (in other words, selected) while the 3D matching process is being performed. That is, one range of the three-dimensional model WM3 used in the 3D matching process may be changed while the 3D matching process is being performed. For example, considering that the shorter the distance between the object OBJ and the imaging device 22, the smaller the proportion of the portion of the object OBJ located within the imaging range (field of view) of the imaging device 22 to the entire object OBJ, the range of the portion of the three-dimensional model WM3 used in the 3D matching process may be changed depending on the distance between the object OBJ and the imaging device 22. For example, the portion of the three-dimensional model WM3 used in the 3D matching process may be specified so that the range of the portion of the three-dimensional model WM3 used in the 3D matching process becomes smaller (in other words, narrower) as the distance between the object OBJ and the imaging device 22 becomes shorter.

[0160] A single portion of the three-dimensional model WM3 may be designated as part of the three-dimensional model WM3 used in the 3D matching process. Alternatively, multiple portions of the three-dimensional model WM3 may be designated as part of the three-dimensional model WM3 used in the 3D matching process. For example, multiple portions of the three-dimensional model WM3, each of which can be used as a characteristic part of the three-dimensional model WM3, may be designated as part of the three-dimensional model WM3 used in the 3D matching process. In this case, improved accuracy of the 3D matching process can be expected compared to when a single portion of the three-dimensional model WM3 is designated as part of the three-dimensional model WM3 used in the 3D matching process. In other words, the three-dimensional position data WSD generated from the image data IMG_3D is more likely to detect the object OBJ whose three-dimensional position is indicated.

[0161] The portion of the three-dimensional model WM3 used in the 3D matching process may be designated by a user of the robot system SYS. For example, the user may designate a portion of the three-dimensional model WM3 displayed on a display (not shown). For example, the user may estimate a portion of the object OBJ assumed to be included in the imaging range (field of view) of the imaging device 22, and designate a portion of the three-dimensional model WM3 corresponding to the estimated portion of the object OBJ. Alternatively, the designation of a portion of the three-dimensional model WM3 used in the 3D matching process may be automatically performed by the control device 3. For example, the control device 3 may estimate a portion of the object OBJ assumed to be included in the imaging range (field of view) of the imaging device 22 (a portion of the object OBJ whose three-dimensional position data WSD assumed to be generated from the image data IMG_3D indicates a three-dimensional position), and designate a portion of the three-dimensional model WM3 corresponding to the estimated portion of the object OBJ. In this case, the control device 3 may estimate a portion of the object OBJ that is assumed to be included in the imaging range (field of view) of the imaging device 22, based on information regarding at least one of, for example, the direction of relative movement between the object OBJ and the imaging device 22 (or the direction in which the robot 1 approaches the object OBJ), the distance between the object OBJ and the imaging device 22 (or the distance between the robot 1 and the object OBJ), and the imaging range (field of view) of the imaging device 22.

[0162] In particular, in this embodiment, before starting a process of bringing characteristic features of the three-dimensional model WM3 indicated by the three-dimensional model data WMD closer to characteristic features of the object OBJ indicated by the three-dimensional position data WSD (i.e., a matching process using the three-dimensional position data WSD), the 3D matching unit 3122 determines at least one of the initial position and initial orientation of the three-dimensional model WM3. In other words, before starting the matching process using the three-dimensional position data WSD, the 3D matching unit 3122 performs a position and orientation determination process that determines at least one of the initial position and initial orientation of the three-dimensional model WM3 as at least part of the 3D matching process. Therefore, in this embodiment, the 3D matching process performed by the 3D matching unit 3122 may include a position and orientation determination process that determines at least one of the initial position and initial orientation of the three-dimensional model WM3, and a matching process using the three-dimensional position data WSD.

[0163] However, the 3D matching process may include a matching process using the three-dimensional position data WSD but may not include a position and orientation determination process. In this case, the 3D matching unit 3122 may perform the position and orientation determination process as a process separate from the 3D matching process. Alternatively, a processing block or device different from the 3D matching unit 3122 may perform the position and orientation determination process.

[0164] In particular, in this embodiment, as described above, the 3D matching process includes a process of bringing characteristic features of the three-dimensional model WM3 indicated by the three-dimensional model data WMD closer to characteristic features of the object OBJ indicated by the three-dimensional position data WSD within the 3D imaging coordinate system (i.e., a matching process using the three-dimensional position data WSD). Therefore, the 3D matching unit 3122 may determine an initial position and initial orientation of the three-dimensional model WM3 within the 3D imaging coordinate system. The 3D matching unit 3122 then places the three-dimensional model WM3 at the determined initial position and orientation within the 3D imaging coordinate system. The 3D matching unit 3122 then begins a process of bringing the characteristic features of the three-dimensional model WM3 placed at the initial position and orientation closer to characteristic features of the object OBJ indicated by the three-dimensional position data WSD. In other words, the 3D matching unit 3122 begins a matching process using the three-dimensional model WM3 at the initial position and orientation and the three-dimensional position data WSD.

[0165] As described above, the 3D matching process can also be said to include a process of changing the positional relationship between the coordinate system of the three-dimensional model data WMD (e.g., the coordinate system of a CAD model) and the 3D imaging coordinate system so that characteristic locations of the three-dimensional model WM3 indicated by the three-dimensional model data WMD approach characteristic locations of the object OBJ whose three-dimensional position is indicated by the three-dimensional position data WSD. In this case, the operation of determining the initial position and initial orientation of the three-dimensional model WM3 in the 3D imaging coordinate system may be considered substantially equivalent to the operation of determining the initial state of the positional relationship between the coordinate system of the three-dimensional model data WMD and the 3D imaging coordinate system. In this case, the 3D matching unit 3122 may align the coordinate system of the three-dimensional model data WMD and the 3D imaging coordinate system so that the positional relationship between the coordinate system of the three-dimensional model data WMD and the 3D imaging coordinate system is in the determined initial state. Thereafter, the 3D matching unit 3122 may start processing to bring the characteristic parts of the three-dimensional model WM3 closer to the characteristic parts of the object OBJ indicated by the three-dimensional position data WSD.

[0166] Note that, in this embodiment, as described above, an example has been described in which the position and orientation calculation unit 312 calculates both the position and orientation of the object OBJ in step S3 of Fig. 4 , and therefore the 3D matching unit 3122 determines both the initial position and initial orientation of the three-dimensional model WM3 in the 3D imaging coordinate system. However, when calculating either the position or orientation of the object OBJ in step S3 of Fig. 4 , the 3D matching unit 3122 may determine either the initial position or initial orientation of the three-dimensional model WM3 in the 3D imaging coordinate system. Alternatively, even when calculating either the position or orientation of the object OBJ in step S3 of Fig. 4 , the 3D matching unit 3122 may determine both the initial position and initial orientation of the three-dimensional model WM3 in the 3D imaging coordinate system.

[0167] To determine the initial position and initial orientation of the three-dimensional model WM3, the 3D matching unit 3122 may use the position and orientation information POI1 output from the 2D matching unit 3121 to the 3D matching unit 3122. In other words, the 3D matching unit 3122 may determine the initial position and initial orientation of the three-dimensional model WM3 based on the position and orientation information POI1.

[0168] 14 , the 3D matching unit 3122 first converts position and orientation information POI1, which indicates the position and orientation of the object OBJ in the 2D imaging coordinate system, into position and orientation information POI1_conv, which indicates the position and orientation of the object OBJ in the 3D imaging coordinate system. The position and orientation information POI1_conv may indicate the position Tx (2D to 3D) of the object OBJ in the X-axis direction (3D) parallel to the X-axis (3D) of the 3D imaging coordinate system, calculated by the conversion. The position and orientation information POI1_conv may also indicate the position Ty (2D to 3D) of the object OBJ in the Y-axis direction (3D) parallel to the Y-axis (3D) of the 3D imaging coordinate system, calculated by the conversion. The position and orientation information POI1_conv may indicate a position Tz (2D to 3D) of the object OBJ in the Z-axis direction (3D) parallel to the Z-axis (3D) of the 3D imaging coordinate system, calculated by the transformation. The position and orientation information POI1_conv may indicate a rotation amount Rx (2D to 3D) of the object OBJ around the X-axis (3D) of the 3D imaging coordinate system, calculated by the transformation. The position and orientation information POI1_conv may indicate a rotation amount Ry (2D to 3D) of the object OBJ around the Y-axis (3D) of the 3D imaging coordinate system, calculated by the transformation. The position and orientation information POI1_conv may indicate a rotation amount Rz (2D to 3D) of the object OBJ around the Z-axis (3D) of the 3D imaging coordinate system, calculated by the transformation.

[0169] In the following description, to ensure consistency in terminology, the amount of rotation Rx (2D to 3D) of the object OBJ around the X axis (3D), the amount of rotation Ry (2D to 3D) of the object OBJ around the Y axis (3D), and the amount of rotation Rz (2D to 3D) of the object OBJ around the Z axis (3D) will be referred to as the orientation Rx (2D to 3D) of the object OBJ in the direction of rotation around the X axis (3D), the orientation Ry (2D to 3D) of the object OBJ in the direction of rotation around the Y axis (3D), and the orientation Rz (2D to 3D) of the object OBJ in the direction of rotation around the Z axis (3D), respectively.

[0170] Furthermore, the orientation Rx (2D to 3D) of the object OBJ around the X axis (2D to 3D), the orientation Ry (2D to 3D) of the object OBJ around the Y axis (2D to 3D), and the orientation Rz (2D to 3D) of the object OBJ around the Z axis (2D to 3D) may be considered to indicate the position of the object OBJ in the rotation direction around the X axis (2D to 3D), the position of the object OBJ in the rotation direction around the Y axis (2D to 3D), and the position of the object OBJ in the rotation direction around the Z axis (2D to 3D), respectively. In other words, the orientation Rx (2D to 3D) of the object OBJ around the X axis (2D to 3D), the orientation Ry (2D to 3D) of the object OBJ around the Y axis (2D to 3D), and the orientation Rz (2D to 3D) of the object OBJ around the Z axis (2D to 3D) may all be considered to be parameters representing the position of the object OBJ.

[0171] To convert the position and orientation information POI1 into the position and orientation information POI1_conv, the 3D matching unit 3122 may use coordinate system information indicating the positional relationship between the 2D imaging coordinate system and the 3D imaging coordinate system. The coordinate system information may be stored in advance in, for example, the storage device 32. The coordinate system information may include a transformation matrix (typically, a rigid transformation matrix) for transforming a position in either the 2D imaging coordinate system or the 3D imaging coordinate system into a position in the other of the 2D imaging coordinate system and the 3D imaging coordinate system. The transformation matrix can be calculated using a mathematical method from external parameters indicating the positional relationship between the image capture device 21 and the image capture device 22. As an example, the transformation matrix can be calculated using a mathematical method that solves a PnP (Perspective n Point) problem based on external parameters indicating the positional relationship between the image capture device 21 and the image capture device 22. In this case, the control device 3 can convert the position and orientation information POI1 indicating the position and orientation of the object OBJ in the 2D imaging coordinate system into position and orientation information POI1_conv indicating the position and orientation of the object OBJ in the 3D imaging coordinate system based on the coordinate system information (particularly, the transformation matrix).

[0172] 14 , the 3D matching unit 3122 may set the position indicated by the position and orientation information POI1_conv as the initial position of the three-dimensional model WM3. The 3D matching unit 3122 may also set the orientation indicated by the position and orientation information POI1_conv as the initial orientation of the three-dimensional model WM3. For example, the 3D matching unit 3122 may set the position Tx (2D to 3D) indicated by the position and orientation information POI1_conv as the initial position of the three-dimensional model WM3 in the X-axis direction (3D). For example, the 3D matching unit 3122 may set the position Ty (2D to 3D) indicated by the position and orientation information POI1_conv as the initial position of the three-dimensional model WM3 in the Y-axis direction (3D). For example, the 3D matching unit 3122 may set the position Tz (2D to 3D) indicated by the position and orientation information POI1_conv as the initial position of the three-dimensional model WM3 in the Z-axis direction (3D). For example, the 3D matching unit 3122 may set the orientation Rx (2D to 3D) indicated by the position and orientation information POI1_conv as the initial orientation of the three-dimensional model WM3 around the X-axis (3D). For example, the 3D matching unit 3122 may set the orientation Ry (2D to 3D) indicated by the position and orientation information POI1_conv as the initial orientation of the three-dimensional model WM3 around the Y-axis (3D). For example, the 3D matching unit 3122 may set the orientation Rz (2D to 3D) indicated by the position and orientation information POI1_conv as the initial orientation of the three-dimensional model WM3 around the Z-axis (3D).

[0173] 14 , the 3D matching unit 3122 places the three-dimensional model WM3 at the determined initial position and in the determined initial orientation within the 3D imaging coordinate system. For example, the 3D matching unit 3122 may place the three-dimensional model WM3 in the X-axis direction (3D) within the 3D imaging coordinate system so that the position of the three-dimensional model WM3 is position Tx (2D to 3D). For example, the 3D matching unit 3122 may place the three-dimensional model WM3 in the Y-axis direction (3D) within the 3D imaging coordinate system so that the position of the three-dimensional model WM3 is position Ty (2D to 3D). For example, the 3D matching unit 3122 may place the three-dimensional model WM3 in the Z-axis direction (3D) within the 3D imaging coordinate system so that the position of the three-dimensional model WM3 is position Tz (2D to 3D). For example, the 3D matching unit 3122 may position the three-dimensional model WM3 in the 3D imaging coordinate system so that the orientation of the three-dimensional model WM3 around the X-axis (3D) is orientation Rx (2D to 3D). For example, the 3D matching unit 3122 may position the three-dimensional model WM3 in the 3D imaging coordinate system so that the orientation of the three-dimensional model WM3 around the Y-axis (3D) is orientation Ry (2D to 3D). For example, the 3D matching unit 3122 may position the three-dimensional model WM3 in the 3D imaging coordinate system so that the orientation of the three-dimensional model WM3 around the Z-axis (3D) is orientation Rz (2D to 3D).

[0174] The 3D matching unit 3122 may place the three-dimensional model WM3 at the determined initial position and in the determined initial orientation within the 3D imaging coordinate system using the procedure described below. Specifically, the 3D matching unit 3122 may first place (set) the three-dimensional model WM3 within the coordinate system of the three-dimensional model WM3 (within the coordinate system of the three-dimensional model data WSD). In this case, the 3D matching unit 3122 may place the three-dimensional model WM3 within the coordinate system of the three-dimensional model WM3 so that the center of gravity of the three-dimensional model WM3 is located at the origin of the coordinate system of the three-dimensional model WM3. The 3D matching unit 3122 then aligns the coordinate system of the three-dimensional model WM3 with the 3D imaging coordinate system. For example, the 3D matching unit 3122 may align the coordinate system of the three-dimensional model WM3 with the 3D imaging coordinate system so that the origin of the coordinate system of the three-dimensional model WM3 coincides with the origin of the 3D imaging coordinate system. For example, the 3D matching unit 3122 may align the coordinate system of the three-dimensional model WM3 with the 3D imaging coordinate system so that the X-axis of the coordinate system of the three-dimensional model WM3 coincides with the X-axis (3D) of the 3D imaging coordinate system. For example, the 3D matching unit 3122 may align the coordinate system of the three-dimensional model WM3 with the 3D imaging coordinate system so that the Y-axis of the coordinate system of the three-dimensional model WM3 coincides with the Y-axis (3D) of the 3D imaging coordinate system. For example, the 3D matching unit 3122 may align the coordinate system of the three-dimensional model WM3 with the 3D imaging coordinate system so that the Z-axis of the coordinate system of the three-dimensional model WM3 coincides with the Z-axis (3D) of the 3D imaging coordinate system. For example, the 3D matching unit 3122 may align the coordinate system of the three-dimensional model WM3 with the 3D imaging coordinate system so that the scaling of the coordinates in the coordinate system of the three-dimensional model WM3 coincides with the scaling of the coordinates in the 3D imaging coordinate system. The 3D matching unit 3122 may then place the three-dimensional model WM3 at the determined initial position and in the determined initial orientation within the coordinate system of the three-dimensional model WM3 in which the three-dimensional model WM3 has been placed. Here, because the alignment between the coordinate system of the three-dimensional model WM3 and the 3D imaging coordinate system has already been completed, placing the three-dimensional model WM3 within the coordinate system of the three-dimensional model WM3 is equivalent to placing the three-dimensional model WM3 within the 3D imaging coordinate system.As a result, the 3D matching unit 3122 can place the three-dimensional model WM3 at the determined initial position and in the determined initial orientation within the 3D imaging coordinate system.

[0175] In addition to or instead of aligning the coordinate system of the three-dimensional model WM3 with the 3D imaging coordinate system, the 3D matching unit 3122 may place the three-dimensional model WM3 in the 3D imaging coordinate system at the determined initial position and in the determined initial orientation using a transformation matrix (typically, a rigid transformation matrix) for transforming a position in either the 3D imaging coordinate system or the coordinate system of the three-dimensional model WM3 into a position in the other coordinate system of the 3D imaging coordinate system or the coordinate system of the three-dimensional model WM3. Specifically, after placing the three-dimensional model WM3 in the coordinate system of the three-dimensional model WM3, the 3D matching unit 3122 may then use the transformation matrix to transform the position and orientation of the three-dimensional model WM3 placed in the coordinate system of the three-dimensional model WM3 (i.e., the position and orientation in the coordinate system of the three-dimensional model WM3) into the position and orientation of the three-dimensional model WM3 in the 3D imaging coordinate system. Thereafter, the 3D matching unit 3122 may place the three-dimensional model WM3 in the determined initial position and in the determined initial orientation within the 3D imaging coordinate system.

[0176] Alternatively, the 3D matching unit 3122 may determine the initial position and initial orientation of the three-dimensional model WM3 in the 2D imaging coordinate system, in addition to or instead of determining the initial position and initial orientation of the three-dimensional model WM3 in the 3D imaging coordinate system. In this case, the 3D matching unit 3122 may set the position and orientation indicated by the position and orientation information POI1 as the initial position and initial orientation of the three-dimensional model WM3 in the 2D imaging coordinate system, respectively, without converting the position and orientation information POI1 to the position and orientation information POI1_conv. Then, in addition to or instead of aligning the coordinate system of the three-dimensional model WM3 with the 3D imaging coordinate system, the 3D matching unit 3122 may align the coordinate system of the three-dimensional model WM3 with the 2D imaging coordinate system. Specifically, the 3D matching unit 3122 may position the three-dimensional model WM3 in the coordinate system of the three-dimensional model WM3, and then align the coordinate system of the three-dimensional model WM3 with the 2D imaging coordinate system. For example, the 3D matching unit 3122 may align the coordinate system of the three-dimensional model WM3 with the 2D imaging coordinate system so that the origin of the coordinate system of the three-dimensional model WM3 coincides with the origin of the 2D imaging coordinate system. For example, the 3D matching unit 3122 may align the coordinate system of the three-dimensional model WM3 with the 2D imaging coordinate system so that the X axis of the coordinate system of the three-dimensional model WM3 coincides with the X axis (2D) of the 2D imaging coordinate system. For example, the 3D matching unit 3122 may align the coordinate system of the three-dimensional model WM3 with the 2D imaging coordinate system so that the Y axis of the coordinate system of the three-dimensional model WM3 coincides with the Y axis (2D) of the 2D imaging coordinate system. For example, the 3D matching unit 3122 may align the coordinate system of the three-dimensional model WM3 with the 3D imaging coordinate system so that the Z axis of the coordinate system of the three-dimensional model WM3 coincides with the Z axis (2D) of the 2D imaging coordinate system. For example, the 3D matching unit 3122 may align the coordinate system of the three-dimensional model WM3 with the 2D imaging coordinate system so that the scaling of the coordinates in the coordinate system of the three-dimensional model WM3 matches the scaling of the coordinates in the 2D imaging coordinate system. After that, the 3D matching unit 3122 may place the three-dimensional model WM3 at the determined initial position and in the determined initial orientation within the coordinate system of the three-dimensional model WM3 in which the three-dimensional model WM3 is placed.Here, because the alignment between the coordinate system of the three-dimensional model WM3 and the 2D imaging coordinate system has already been completed, placing the three-dimensional model WM3 in the coordinate system of the three-dimensional model WM3 is equivalent to placing the three-dimensional model WM3 in the 2D imaging coordinate system. As a result, the 3D matching unit 3122 can place the three-dimensional model WM3 at the determined initial position and with the determined initial orientation in the 2D imaging coordinate system. The 3D matching unit 3122 may then convert the position and orientation of the three-dimensional model WM3 in the 2D imaging coordinate system to the position and orientation of the three-dimensional model WM3 in the 3D imaging coordinate system, respectively, using a transformation matrix (typically, a rigid transformation matrix) for transforming a position in one of the 2D imaging coordinate system and the 3D imaging coordinate system into a position in the other of the 2D imaging coordinate system and the 3D imaging coordinate system. The position and orientation of the three-dimensional model WM3 in the 3D imaging coordinate system obtained by the transformation are considered to be the initial position and initial orientation of the three-dimensional model WM3 in the 3D imaging coordinate system, respectively. As a result, the 3D matching unit 3122 can place the three-dimensional model WM3 at the determined initial position and in the determined initial orientation in the 3D imaging coordinate system.

[0177] Thereafter, the 3D matching unit 3122 starts a process of bringing the characteristic features of the three-dimensional model WM3 closer to the characteristic features of the object OBJ indicated by the three-dimensional position data WSD. As a result, the 3D matching unit 3122 can calculate the position and orientation of the object OBJ. In other words, the 3D matching unit 3122 can generate position and orientation information POI2.

[0178] The method for calculating the position and orientation of the object OBJ is not limited to the matching process using the three-dimensional position data WSD described above. The 3D matching unit 3122 may calculate the position and orientation of the object OBJ using other well-known methods for calculating the position and orientation of the object OBJ using the three-dimensional position data WSD. The method for calculating the position and orientation of the object OBJ may be a well-known method for calculating the position and orientation of the object OBJ based on the three-dimensional position data WSD without using the three-dimensional model data WMD, or a well-known method for calculating the position and orientation of the object OBJ using the three-dimensional position data WSD and data other than the three-dimensional model data WMD. For example, the method for calculating the position and orientation of the object OBJ may be a method for calculating the position and orientation of the object OBJ based on the three-dimensional position data WSD using machine learning or deep learning. In this case, a prediction model for outputting the position and orientation of the object OBJ when the three-dimensional position data WSD is input may be constructed in advance using machine learning or deep learning, and the position and orientation of the object OBJ may be calculated by inputting the three-dimensional position data WSD into this prediction model. This prediction model may be stored in the 3D matching unit 3122. The 3D matching unit 3122 may read out this prediction model stored in the storage device 32.

[0179] The 3D matching unit 3122 performs 3D matching processing using the above-mentioned point cloud data as the three-dimensional position data WSD. As a result, the accuracy of calculating the position and orientation of the object OBJ is higher than when depth image data is used as the three-dimensional position data WSD. In this case, the 3D matching unit 3122 uses point cloud data (e.g., point cloud data representing a CAD model) representing the three-dimensional shape that serves as the reference for the object OBJ as the three-dimensional model data WMD. However, the 3D matching unit 3122 may also perform 3D matching processing using the above-mentioned depth image data as the three-dimensional position data WSD. In this case, the 3D matching unit 3122 may also use depth image data (e.g., depth image data representing a CAD model) representing the three-dimensional shape that serves as the reference for the object OBJ as the three-dimensional model data WMD.

[0180] The 3D matching unit 3122 may calculate, as the position of the object OBJ in the 3D imaging coordinate system, at least one of a position Tx(3D) of the object OBJ in an X-axis direction (3D) parallel to the X-axis (3D), a position Ty(3D) of the object OBJ in a Y-axis direction (3D) parallel to the Y-axis (3D), and a position Tz(3D) of the object OBJ in a Z-axis direction (3D) parallel to the Z-axis (3D). The 3D matching unit 3122 may calculate, as the orientation of the object OBJ in the 3D imaging coordinate system, at least one of a rotation amount Rx(3D) of the object OBJ around the X-axis (3D), a rotation amount Ry(3D) of the object OBJ around the Y-axis (3D), and a rotation amount Rz(3D) of the object OBJ around the Z-axis (3D).

[0181] In the following description, similarly to the rotation amounts Rx, Ry, and Rz described above, in order to unify terminology, the rotation amount Rx(3D) of the object OBJ around the X-axis (3D), the rotation amount Ry(3D) of the object OBJ around the Y-axis (3D), and the rotation amount Rz(3D) of the object OBJ around the Z-axis (3D) will be referred to as the orientation Rx(3D) of the object OBJ around the X-axis (3D), the orientation Ry(3D) of the object OBJ around the Y-axis (3D), and the orientation Rz(3D) of the object OBJ around the Z-axis (3D), respectively.

[0182] Furthermore, the orientation Rx(3D) of the object OBJ around the X-axis (3D), the orientation Ry(3D) of the object OBJ around the Y-axis (3D), and the orientation Rz(3D) of the object OBJ around the Z-axis (3D) may be considered to indicate the position of the object OBJ in the rotation direction around the X-axis (3D), the position of the object OBJ in the rotation direction around the Y-axis (3D), and the position of the object OBJ in the rotation direction around the Z-axis (3D), respectively. In other words, the orientation Rx(3D) of the object OBJ around the X-axis (3D), the orientation Ry(3D) of the object OBJ around the Y-axis (3D), and the orientation Rz(3D) of the object OBJ around the Z-axis (3D) may all be considered to be parameters representing the position of the object OBJ.

[0183] Each time the 3D matching unit 3122 generates position and orientation information POI2, it outputs the generated position and orientation information POI2 to the coordinate conversion unit 3125. As an example, the 3D matching unit 3122 calculates all of the positions and orientations of the 6 DOF (that is, position Tx(3D), position Ty(3D), position Tz(3D), orientation Rx(3D), orientation Ry(3D), and orientation Rz(3D)), and outputs position and orientation information POI1 indicating the calculated positions and orientations of the 6 DOF to the coordinate conversion unit 3125. As another example, the 3D matching unit 3122 may calculate the positions and orientations of the 6 DOF, and output position and orientation information POI2 indicating a part of the calculated positions and orientations of the 6 DOF to the coordinate conversion unit 3125. As another example, the 3D matching unit 3122 may calculate part of the position and orientation of the 6 DOF, and output position and orientation information POI2 indicating part of the calculated position and orientation of the 6 DOF to the coordinate conversion unit 3125.

[0184] The 3D matching unit 3122 may calculate the position and orientation of the object OBJ at a predetermined 3D matching cycle. In other words, the 3D matching unit 3122 may calculate the position and orientation of the object OBJ each time a time equivalent to a 3D matching cycle has elapsed. Conversely, it may be considered that a time equivalent to the 3D matching cycle is required from the time the 3D matching unit 3122 starts the 3D matching process to the time it completes. In this case, the 3D matching unit 3122 may output position and orientation information POI2 indicating the position and orientation of the object OBJ to the coordinate transformation unit 3125 at a predetermined 3D matching cycle.

[0185] As shown in FIG. 15 , the 3D matching period may be longer than the 3D imaging period corresponding to the 3D imaging rate at which the imaging device 22 captures the object OBJ. For example, the 3D matching period may be a period at which the 3D matching unit 3122 can perform the 3D matching process several times to several tens of times (e.g., 1 to 30 times) per second. In this case, the imaging device 22 generates new image data IMG_3D between the start and completion of the 3D matching process by the 3D matching unit 3122. In this case, the 3D matching unit 3122 may not start a new 3D matching process using the newly generated image data IMG_3D until the completion of the previously started 3D matching process, even if the imaging device 22 generates new image data IMG_3D. The 3D matching unit 3122 may start a new 3D matching process using the new image data IMG_3D after completing the previously started 3D matching process. Note that even if the imaging device 22 generates new image data IMG_3D, the 3D matching unit 3122 may start a new 3D matching process using the newly generated image data IMG_3D by the imaging device 22, even if the started 3D matching process has not been completed. Note that the timing at which the image data IMG_3D shown in FIG. 15 is generated may be considered to be the timing at which the imaging device 22 captures an image of the object OBJ to generate the image data IMG_3D. However, the 3D matching cycle does not have to be longer than the 3D imaging cycle. For example, the 3D matching cycle may be shorter than the 3D imaging cycle. For example, the 3D matching cycle may be the same as the 3D imaging cycle.

[0186] Furthermore, in this embodiment, as described above, the 3D matching unit 3122 performs 3D matching processing using the result of the 2D matching processing performed by the 2D matching unit 3121 (i.e., the position and orientation information POI1). Therefore, when the imaging devices 21 and 22 capture images of an object OBJ at a certain time ta to generate image data IMG_2D and IMG_3D, respectively, the control device 3 may perform, at separate times, the 2D matching processing using the image data IMG_2D generated at the time ta and the 3D matching processing using the three-dimensional position data WSD generated from the image data IMG_3D generated at the time ta. Specifically, the 2D matching unit 3121 starts the 2D matching processing using the image data IMG_2D generated at the time ta. On the other hand, the 3D matching unit 3122 may not start the 3D matching process using the three-dimensional position data WSD generated from the image data IMG_3D generated at time ta until the 2D matching process using the image data IMG_2D generated at time ta is completed. This is because, as described above, the 3D matching unit 3122 determines the initial position and initial orientation of the three-dimensional model WM3 using the result of the 2D matching process. After the 2D matching process using the image data IMG_2D generated at time ta is completed at time tb, the 3D matching unit 3122 may start the 3D matching process using the result of the 2D matching process (i.e., the position and orientation information POI1 indicating the position and orientation of the object OBJ in the 2D imaging coordinate system at time ta) and the three-dimensional position data WSD generated from the image data IMG_3D generated at time ta. As a result, the 3D matching unit 3122 may generate position and orientation information POI2 that indicates the position and orientation of the object OBJ in the 3D image capture coordinate system at time ta.

[0187] Thus, in this embodiment, after the 2D matching unit 3121 generates position and orientation information POI1 indicating the position and orientation of the object OBJ in the 2D imaging coordinate system at a certain time (for example, time ta), the 3D matching unit 3122 generates position and orientation information POI2 indicating the position and orientation of the object OBJ in the 3D imaging coordinate system at the same time (for example, time ta). For this reason, in this embodiment, the 3D matching unit 3122 may generate position and orientation information POI2 indicating the position and orientation of the object OBJ in the 3D imaging coordinate system at time ta when a period equivalent to the sum of the 2D matching cycle and the 3D matching cycle has elapsed since the imaging devices 21 and 22 captured an image of the object OBJ at time ta.

[0188] However, the 2D matching unit 3121 may perform the 2D matching process using image data IMG_2D generated at a certain time, and the 3D matching unit 3122 may perform the 3D matching process using three-dimensional position data WSD generated from image data IMG_3D generated at a time different from the certain time. For example, when the 2D matching unit 3121 performs the 2D matching process using image data IMG_2D generated at time ta, the 3D matching unit 3122 may perform the 3D matching process using three-dimensional position data WSD generated from image data IMG_3D generated during the period in which the 2D matching process is being performed. For example, when the 2D matching unit 3121 performs the 2D matching process using image data IMG_2D generated at time ta, the 3D matching unit 3122 may perform the 3D matching process using three-dimensional position data WSD generated from image data IMG_3D generated at a time when the remaining time until the completion of the 2D matching process falls below a predetermined time.

[0189] The 3D matching cycle may be the same as the 2D matching cycle. Specifically, the length of the 3D matching cycle may be the same as the length of the 2D matching cycle. In other words, the time from when the 3D matching unit 3122 starts the 3D matching process to when it is completed may be the same as the time from when the 2D matching unit 3121 starts the 2D matching process to when it is completed. However, the 3D matching cycle does not have to be the same as the 2D matching cycle. For example, the length of the 3D matching cycle may be longer than the length of the 2D matching cycle. For example, the length of the 3D matching cycle may be shorter than the length of the 2D matching cycle.

[0190] Typically, the data size of the three-dimensional position data WSD and three-dimensional model data WMD used in the 3D matching process is larger than the data size of the image data IMG_2D and two-dimensional model data IMG_2M used in the 2D matching process. Therefore, the 3D matching cycle, which is the time required for the 3D matching unit 3122 to complete the 3D matching process, may be longer than the 2D matching cycle, which is the time required for the 2D matching unit 3121 to complete the 2D matching process. Furthermore, in this embodiment, the 3D matching process includes, in addition to the matching process using the three-dimensional position data WSD, a position and orientation determination process that determines the initial position and initial orientation of the three-dimensional model WM3. Therefore, the 3D matching cycle may be longer than the 2D matching cycle. Therefore, the 3D matching unit 3122 may perform processing to shorten the 3D matching cycle so that the 3D matching cycle is the same as the 2D matching cycle or so that the 3D matching cycle is shorter than the 2D matching cycle.

[0191] As an example, the 3D matching unit 3122 may select a portion of the three-dimensional position data WSD as 3D matching target data for performing the 3D matching process based on the results of the 2D matching process (e.g., the position and orientation information POI1 indicating the above-described position Tx(2D), position Ty(2D), position Tz(2D), orientation Rx(2D), orientation Ry(2D), and orientation Rz(2D)). Specifically, as shown in FIG. 16 , the 3D matching unit 3122 may estimate an area in the 3D imaging coordinate system where the object OBJ is estimated to exist, based on the position and orientation information POI1. As an example, the 3D matching unit 3122 may convert the position and orientation information POI1 indicating the position and orientation of the object OBJ in the 2D imaging coordinate system into position and orientation information POI1_conv indicating the position and orientation of the object OBJ in the 3D imaging coordinate system. The 3D matching unit 3122 may then estimate an area in the 3D imaging coordinate system where the object OBJ is estimated to exist, based on the position and orientation information POI1_conv. For example, the 3D matching unit 3122 may estimate an area of ​​a predetermined size that includes the position indicated by the position and orientation information POI1_conv as an area in the 3D imaging coordinate system where the object OBJ is estimated to exist. For example, the 3D matching unit 3122 may estimate an area of ​​a predetermined shape that includes the position indicated by the position and orientation information POI1_conv as an area in the 3D imaging coordinate system where the object OBJ is estimated to exist. The 3D matching unit 3122 may then select, as 3D matching target data, a portion of the three-dimensional position data WSD that corresponds to the area where the object OBJ is estimated to exist. The 3D matching unit 3122 may then perform 3D matching processing on the 3D matching target data that is part of the three-dimensional position data WSD.

[0192] Here, the three-dimensional position data WSD may include not only a data portion related to the object OBJ but also a data portion related to other objects different from the object OBJ. This is because the imaging range (field of view) of the imaging device 22 that generates the image data IMG_3D used to generate the three-dimensional position data WSD may include not only the object OBJ but also other objects different from the object OBJ. For example, when the three-dimensional position data WSD is generated from image data IMG_3D generated by the imaging device 22 capturing an image of a workpiece W placed on a support surface S, the three-dimensional position data WSD may include not only a data portion related to the workpiece W but also a data portion related to the support surface S. In this case, the process of selecting the 3D matching target data described above is equivalent to the process of selecting the data portion related to the workpiece W.

[0193] In this case, the 3D matching target data may be considered to be equivalent to data obtained by removing data portions relating to objects other than the object OBJ from the three-dimensional position data WSD. In other words, the 3D matching target data may be considered to be equivalent to data obtained by selectively extracting data portions relating to the object OBJ from the three-dimensional position data WSD. As a result, the time required to complete the 3D matching process performed on the 3D matching target data (i.e., a portion of the three-dimensional position data WSD) is shorter than the time required to complete the 3D matching process performed on the entire three-dimensional position data WSD. This is because the time required to complete the 3D matching process depends on the size of the data on which the 3D matching process is performed. As a result, the 3D matching cycle is shorter than when the 3D matching process is performed on the entire three-dimensional position data WSD.

[0194] Note that the operation of selecting 3D matching target data that is part of the three-dimensional position data WSD may be considered equivalent to the operation of not selecting (or deleting) data portions of the three-dimensional position data WSD other than the 3D matching target data. Therefore, the 3D matching unit 3122 may generate 3D matching target data that is part of the three-dimensional position data WSD based on the result of the 2D matching process by deleting some data portions of the three-dimensional position data WSD that correspond to areas where it is estimated that no object OBJ exists (for example, data portions related to objects other than the object OBJ, as described above).

[0195] As another example, the 3D matching unit 3122 may select 3D matching target data based on a user instruction specifying a portion of the three-dimensional position data WSD. Specifically, the control device 3 may use an output device 35 including a display device to display a GUI (Graphical User Interface) that the user can operate to specify a portion of the three-dimensional position data WSD as 3D matching target data. For example, the control device 3 may use the output device 35 including a display device to display multiple points indicated by point cloud data, which is an example of the three-dimensional position data WSD, and display a GUI that the user can operate to specify a point among the multiple points that corresponds to an object OBJ. For example, the control device 3 may use the output device 35 including a display device to display a depth image indicated by depth image data, which is an example of the three-dimensional position data WSD, and display a GUI that the user can operate to specify a pixel in the depth image that corresponds to the object OBJ. Even in this case, the 3D matching unit 3122 may perform the 3D matching process on the 3D matching target data, which is a portion of the three-dimensional position data WSD designated by the user. As a result, the 3D matching cycle is shorter than when the 3D matching process is performed on the entire three-dimensional position data WSD.

[0196] (2-2-3) Tracking Process Performed by Tracking Unit 3123 The tracking unit 3123 performs tracking processing using two pieces of image data IMG_2D#t1 and IMG_2D#t2, which are generated by the imaging device 21 capturing images of the object OBJ at a first time t1 and a second time t2 that is later than the first time t1. The tracking processing using the two pieces of image data IMG_2D#t1 and IMG_2D#t2 is a tracking processing that tracks, in the image data IMG_2D#t2, at least one feature location that is the same as at least one feature location (e.g., at least one of a feature point and an edge) of the object OBJ captured in the image data IMG_2D#t1, as shown in FIG. 17 . Note that the tracking processing itself may be the same as existing tracking processing. Therefore, a detailed description of the tracking processing will be omitted, but an overview will be provided below.

[0197] The tracking unit 3123 performs tracking processing to calculate the amount of change in the position of at least one characteristic location between the first time t1 and the second time t2. Specifically, the tracking unit 3123 generates position information PI4 indicating the position of at least one characteristic location of the object OBJ at the first time t1 based on the image data IMG_2D#t1 for the first time t1. For example, the tracking unit 3123 may generate position information PI4 indicating the position of the characteristic location in at least one of the X-axis direction (2D) and the Y-axis direction (2D) in a 2D imaging coordinate system. Furthermore, the tracking unit 3123 may generate position information PI5 indicating the position of at least one characteristic location of the object OBJ at the second time t2 based on the image data IMG_2D#t2 for the second time t2. For example, the tracking unit 3123 may generate position information PI5 indicating the position of the characteristic location in at least one of the X-axis direction (2D) and the Y-axis direction (2D) in the 2D imaging coordinate system. For convenience of explanation, the following description will be given assuming that the position information PI4 and PI5 indicate the positions of characteristic locations in the X-axis direction (2D) and the Y-axis direction (2D), respectively. The tracking unit 3123 then calculates the amount of change in at least one of the position and orientation of the object OBJ between the first time t1 and the second time t2 based on the position information PI4 and PI5. As a result, the tracking unit 3123 generates change amount information VI indicating the amount of change in at least one of the position and orientation of the object OBJ between the first time t1 and the second time t2.

[0198] As described above, in this embodiment, an example has been described in which the position and orientation calculation unit 312 calculates both the position and orientation of the object OBJ. In this case, the tracking unit 3123 may generate change amount information VI that indicates the amount of change in both the position and orientation of the object OBJ. For convenience of explanation, the following description will be given of an example in which the tracking unit 3123 generates change amount information VI that indicates the amount of change in both the position and orientation of the object OBJ.

[0199] The tracking unit 3123 may calculate the amount of change in the position of the object OBJ in the 2D imaging coordinate system. The tracking unit 3123 may calculate, as the amount of change in the position of the object OBJ in the 2D imaging coordinate system, at least one of the amount of change ΔTx(2D) in the position Tx(2D) of the object OBJ in the X-axis direction (2D), the amount of change ΔTy(2D) in the position Ty(2D) of the object OBJ in the Y-axis direction (2D), and the amount of change ΔTz(2D) in the position Tz(2D) of the object OBJ in the Z-axis direction (2D). The tracking unit 3123 may calculate at least one of the change in orientation of the object OBJ around the X axis (2D), the change in orientation Rx(2D) of the object OBJ around the X axis (2D), the change in orientation Ry(2D) of the object OBJ around the Y axis (2D), and the change in orientation Rz(2D) of the object OBJ around the Z axis (2D), ΔRz(2D), as the change in orientation of the object OBJ in the 2D imaging coordinate system.

[0200] Alternatively, the tracking unit 3123 may calculate the amount of change in the position of the object OBJ in the 3D imaging coordinate system. The tracking unit 3123 may calculate, as the amount of change in the position of the object OBJ in the 3D imaging coordinate system, at least one of the amount of change ΔTx(3D) in the position Tx(3D) of the object OBJ in the X-axis direction (3D), the amount of change ΔTy(3D) in the position Ty(3D) of the object OBJ in the Y-axis direction (3D), and the amount of change ΔTz(3D) in the position Tz(3D) of the object OBJ in the Z-axis direction (3D). The tracking unit 3123 may calculate at least one of the change in orientation of the object OBJ around the X axis (3D), the change in orientation Rx(3D) of the object OBJ around the X axis (3D), the change in orientation Ry(3D) of the object OBJ around the Y axis (3D), and the change in orientation Rz(3D) of the object OBJ around the Z axis (3D), ΔRz(3D), as the change in orientation of the object OBJ in the 3D imaging coordinate system.

[0201] The tracking unit 3123 may calculate the amount of change in the position of the object OBJ in the global coordinate system. The tracking unit 3123 may calculate, as the amount of change in the position of the object OBJ in the global imaging coordinate system, at least one of the amount of change ΔTx(GL) in the position Tx(GL) of the object OBJ in the X-axis direction (GL), the amount of change ΔTy(GL) in the position Ty(GL) of the object OBJ in the Y-axis direction (GL), and the amount of change ΔTz(GL) in the position Tz(GL) of the object OBJ in the Z-axis direction (GL). The tracking unit 3123 may calculate at least one of the change in orientation of the object OBJ around the X axis (GL), the change in orientation Rx(GL), the change in orientation Ry(GL), the change in orientation Rz(GL), and the change in orientation Rz(GL), the change in orientation Rz(GL), and the change in orientation Rz(GL), respectively, around the Z axis (GL) in the global imaging coordinate system.

[0202] Note that if the object OBJ moves during the period from the first time t1 to the second time t2, at least one of the position and orientation of the object OBJ in the 2D imaging coordinate system may change. Furthermore, if the imaging device 21 that captures the object OBJ moves during the period from the first time t1 to the second time t2, at least one of the position and orientation of the object OBJ in the 2D imaging coordinate system may also change. Therefore, the amount of change in the position of the object OBJ calculated by the tracking unit 3123 may be considered equivalent to the amount of change in the relative position between the object OBJ and the imaging device 21. Similarly, the amount of change in the orientation of the object OBJ calculated by the tracking unit 3123 may be considered equivalent to the amount of change in the relative orientation between the object OBJ and the imaging device 21.

[0203] Particularly in this embodiment, the tracking unit 3123 generates the change amount information VI based on the three-dimensional position data WSD in addition to the two image data IMG_2D#t1 and IMG_2D#t2. Specifically, the tracking unit 3123 generates the change amount information VI based on the three-dimensional position data WSD generated from image data IMG_3D generated by the imaging device 22 capturing an image of the object OBJ at a third time t3 in addition to the two image data IMG_2D#t1 and IMG_2D#t2 generated by the imaging device 22 capturing an image of the object OBJ at a first time t1 and a second time t2, respectively.

[0204] In particular, in this embodiment, the third time t3 at which the imaging device 22 captures an image of the object OBJ for the tracking process is the same time as the second time t2 at which the imaging device 21 captures an image of the object OBJ for the tracking process. That is, the tracking unit 3123 generates the change amount information VI based on not only the two image data IMG_2D#t1 and IMG_2D#t2, but also on the three-dimensional position data WSD generated from image data IMG_3D generated by the imaging device 22 capturing an image of the object OBJ at the third time t3, which is the same as the second time t2.

[0205] Note that the state in which "the second time t2 and the third time t3 are the same" may include a state in which "the second time t2 and the third time t3 are literally exactly the same." The state in which "the second time t2 and the third time t3 are the same" may include a state in which "although the second time t2 and the third time t3 are not exactly the same, the second time t2 and the third time t3 can be considered to be substantially the same because the time difference between the second time t2 and the third time t3 is smaller than the allowable upper limit." Here, the allowable upper limit may be an allowable upper limit based on a control error of the robot arm 12. For example, a time difference between the second time t2 and the third time t3 may cause an error in the calculation results of at least one of the position and orientation of the object OBJ (described later) (i.e., the accuracy of at least one of the calculated position and orientation of the object OBJ may be reduced). In this case, an error in the calculation results of at least one of the position and orientation of the object OBJ may cause a control error of the robot arm 12. A control error of the robot arm 12 results in a movement error of the end effector 13, which may prevent the end effector 13 from properly performing a predetermined process on the object OBJ. The allowable upper limit can also be considered as an allowable upper limit for a movement error of the end effector 13 by the robot arm 12. Even if there is a time difference between the second time t2 and the third time t3 due to a synchronization error in the imaging processes of the imaging devices 21 and 22, for example, the second time t2 and the third time t3 may be considered to be substantially the same time. The synchronization error in the imaging processes of the imaging devices 21 and 22 may be a synchronization control error of the imaging processes of the imaging devices 21 and 22 by the control device 3.

[0206] However, the second time t2 and the third time t3 may be different times. The state in which "the second time t2 and the third time t3 are different times" may include a state in which "the second time t2 and the third time t3 cannot be regarded as substantially the same time because the time difference between the second time t2 and the third time t3 is greater than the allowable upper limit value."

[0207] In the present embodiment, when the imaging devices 21 and 22 capture images of the object OBJ during a period in which each of the imaging devices 21 and 22 and the object OBJ are displaced relative to each other, the imaging devices 21 and 22 may capture images of the object OBJ so that the second time t2 and the third time t3 are the same. In other words, when the imaging devices 21 and 22 capture images of the object OBJ during a period in which each of the imaging devices 21 and 22 and the object OBJ are displaced relative to each other, the control device 3 may control the imaging devices 21 and 22 so that the second time t2 and the third time t3 are the same. The reason for this will be explained again when explaining the effects of the robot system SYS.

[0208] On the other hand, when the imaging devices 21 and 22 capture images of the object OBJ during a period in which there is no relative displacement between each of the imaging devices 21 and 22 and the object OBJ, the imaging devices 21 and 22 do not need to capture images of the object OBJ so that the second time t2 and the third time t3 are the same. That is, the control device 3 does not need to control the imaging devices 21 and 22 so that the second time t2 and the third time t3 are the same. For example, the imaging devices 21 and 22 may capture images of the object OBJ so that the second time t2 and the third time t3 are different times. That is, the control device 3 may control the imaging devices 21 and 22 so that the second time t2 and the third time t3 are different times. When the imaging devices 21 and 22 capture images of the object OBJ during a period when there is no relative displacement between the imaging devices 21 and 22 and the object OBJ, the imaging devices 21 and 22 may capture images of the object OBJ so that the second time t2 and the third time t3 are the same. In other words, the control device 3 may control the imaging devices 21 and 22 so that the second time t2 and the third time t3 are the same.

[0209] To generate the change amount information VI based on the three-dimensional position data WSD, the tracking unit 3123 first generates position information PI3 indicating the position of at least one characteristic location of the object OBJ based on the three-dimensional position data WSD. Because the three-dimensional position data WSD is generated from image data IMG_3D generated by the imaging device 22 capturing an image of the object OBJ at the third time t3, the tracking unit 3123 generates position information PI3 indicating the position of at least one characteristic location of the object OBJ at the third time t3 based on the three-dimensional position data WSD.

[0210] In this embodiment, the tracking unit 3123 may calculate the position of at least one characteristic location of the object OBJ in the Z-axis direction (3D) of the 3D imaging coordinate system. In this case, the tracking unit 3123 may calculate the position of at least one characteristic location of the object OBJ in the Z-axis direction (3D) of the 3D imaging coordinate system based on the three-dimensional positions of each of multiple points of the object OBJ in the 3D imaging coordinate system indicated by the three-dimensional position data WSD. As a result, the tracking unit 3123 may generate position information PI3 that indicates the position of the at least one characteristic location of the object OBJ in the Z-axis direction (3D).

[0211] Alternatively, the tracking unit 3123 may calculate the position of at least one characteristic location of the object OBJ in the Z-axis direction (2D) of the 2D imaging coordinate system. In this case, the tracking unit 3123 may use the coordinate system information described above (e.g., a transformation matrix for converting a position in either the 2D imaging coordinate system or the 3D imaging coordinate system into a position in the other of the 2D imaging coordinate system or the 3D imaging coordinate system) to convert the three-dimensional positions of each of the multiple points of the object OBJ in the 3D imaging coordinate system indicated by the three-dimensional position data WSD into the three-dimensional positions of each of the multiple points of the object OBJ in the 2D imaging coordinate system. The tracking unit 3123 may then calculate the position of at least one characteristic location of the object OBJ in the Z-axis direction (2D) based on the three-dimensional positions of each of the multiple points of the object OBJ in the 2D imaging coordinate system. As a result, the tracking unit 3123 may generate position information PI3 indicating the position of at least one characteristic location of the object OBJ in the Z-axis direction (2D).

[0212] Thereafter, the tracking unit 3123 generates change amount information VI based on position information PI4 and PI5 generated from the two image data IMG_2D#t1 and IMG_2D#t2, respectively, and position information PI3 generated from the three-dimensional position data WSD. Specifically, the tracking unit 3123 reflects position information PI3, which indicates the position of at least one characteristic location of the object OBJ at a third time t3 that is the same as (but may differ in some cases from) the second time t2, in position information PI5, which indicates the position of at least one characteristic location of the object OBJ at a second time t2. Specifically, the position information PI5 indicates the position of the characteristic location in each of the X-axis direction (2D) and the Y-axis direction (2D), as described above, and the position information PI3 indicates the position of the characteristic location in the Z-axis direction (2D) or the Z-axis direction (3D), as described above. In this case, the tracking unit 3123 may add the position of the characteristic location in the Z-axis direction (2D) or the Z-axis direction (3D) indicated by the position information PI3 to the position information PI5 that does not indicate the position of the characteristic location in the Z-axis direction (2D). In other words, the tracking unit 3123 reflects the position information PI3 in the position information PI5 so that the position information PI5 becomes information that indicates the position of the characteristic location in the Z-axis direction (2D) or the Z-axis direction (3D) calculated from the three-dimensional position data WSD, in addition to the positions of the characteristic location in each of the X-axis direction (2D) and Y-axis direction (2D) calculated from the image data IMG_2D.

[0213] When the position information PI3 indicates the position of the characteristic location in the Z-axis direction (2D), both the position information PI5 and the position information PI3 indicate positions in the 2D imaging coordinate system. That is, the coordinate system of the position indicated by the position information PI5 is the same as the coordinate system of the position indicated by the position information PI3. In this case, the tracking unit 3123 may simply add the position information PI3 indicating the position of the characteristic location in the 2D imaging coordinate system to the position information PI5 indicating the position of the characteristic location in the 2D imaging coordinate system.

[0214] On the other hand, when the position information PI3 indicates the position of the characteristic location in the Z-axis direction (3D), the position information PI3 indicates a position in the 3D imaging coordinate system, while the position information PI5 indicates a position in the 2D imaging coordinate system. That is, the coordinate system of the position indicated by the position information PI5 is different from the coordinate system of the position indicated by the position information PI3. In this case, the tracking unit 3123 may convert one of the position information PI3 and PI5 so that the coordinate system of the position indicated by the position information PI5 is the same as the coordinate system of the position indicated by the position information PI3, and then add the position information PI3 to the position information PI5. For example, the tracking unit 3123 may convert the position of the characteristic location in the 2D imaging coordinate system indicated by the position information PI5 into the position of the characteristic location in the 3D imaging coordinate system using the above-described coordinate system information (e.g., a transformation matrix for converting a position in one of the 2D imaging coordinate system and the 3D imaging coordinate system into a position in the other of the 2D imaging coordinate system and the 3D imaging coordinate system). The tracking unit 3123 may add position information PI3 converted to indicate the position of the characteristic location in the 3D imaging coordinate system to position information PI5 indicating the position of the characteristic location in the 3D imaging coordinate system.

[0215] Thereafter, the tracking unit 3123 generates change amount information VI based on the position information PI4 and the position information PI5 reflecting the position information PI3. That is, the tracking unit 3123 calculates the amount of change in at least one of the position and orientation of the object OBJ between the first time t1 and the second time t2 based on the position information PI4 and the position information PI5 reflecting the position information PI3.

[0216] Specifically, as described above, the position information PI4 indicates the positions of the characteristic locations in each of the X-axis direction and the Y-axis direction. Meanwhile, the position information PI5 indicates the positions of the characteristic locations in each of the X-axis direction, the Y-axis direction, and the Z-axis direction. In this case, the process of calculating the change amount information VI indicating the change amount in the position and orientation of the object OBJ can be considered to be substantially equivalent to the process of solving a PnP (Perspective n Point) problem. Specifically, a typical PnP problem is a problem of estimating the position and orientation (specifically, the translation matrix and the rotation matrix) of the image capture device 21 or 22 that captures the object OBJ from the relationship between the positions (three-dimensional positions) of n characteristic locations of the object OBJ existing in three-dimensional space and the positions (two-dimensional positions) of n characteristic locations of the object OBJ on a two-dimensional image. The estimated translation matrix and rotation matrix can be used to convert the position and orientation of the object OBJ existing in three-dimensional space into the position and orientation of the object OBJ on a two-dimensional image, or conversely, to convert the position and orientation of the object OBJ on a two-dimensional image into the position and orientation of the object OBJ existing in three-dimensional space. Therefore, in this embodiment, the tracking unit 3123 may set, as the PnP problem, a problem of estimating the translation matrix and rotation matrix necessary to convert the position and orientation of the object OBJ at the first time t1 into the position and orientation of the object OBJ at the second time t2, based on the relationship between position information PI4 indicating the positions (two-dimensional positions) of n characteristic locations of the object OBJ at the first time t1 and position information PI5 indicating the positions (three-dimensional positions) of n characteristic locations of the object OBJ at the second time t2. In this case, the tracking unit 3123 may estimate the translation matrix and rotation matrix by solving the PnP problem. Thereafter, the tracking unit 3123 may calculate the amount of change in the position of the object OBJ between the first time t1 and the second time t2 based on the estimated translation matrix. The tracking unit 3123 may calculate the amount of change in the orientation of the object OBJ between the first time t1 and the second time t2 based on the estimated rotation matrix.

[0217] The method for calculating the amount of change in the position and orientation of the object OBJ is not limited to tracking processing using the two pieces of image data IMG_2D#t1 and IMG_2D#t2 (and, if necessary, three-dimensional position data WSD). The tracking unit 3123 may calculate the amount of change in the position and orientation of the object OBJ using other well-known methods for calculating the amount of change in the position and orientation of the object OBJ using the two pieces of image data IMG_2D#t1 and IMG_2D#t2 (and, if necessary, three-dimensional position data WSD). For example, the method for calculating the amount of change in the position and orientation of the object OBJ may be a method for calculating the amount of change in the position and orientation of the object OBJ based on the two pieces of image data IMG_2D#t1 and IMG_2D#t2 (and, if necessary, three-dimensional position data WSD) using machine learning or deep learning. In this case, a prediction model for outputting the amount of change in the position and orientation of the object OBJ when two pieces of image data IMG_2D#t1 and IMG_2D#t2 (and, if necessary, three-dimensional position data WSD) are input may be constructed in advance using machine learning or deep learning, and the amount of change in the position and orientation of the object OBJ may be calculated by inputting the two pieces of image data IMG_2D#t1 and IMG_2D#t2 (and, if necessary, three-dimensional position data WSD) into this prediction model. The tracking unit 3123 may include this prediction model.

[0218] Each time the tracking unit 3123 calculates the change amount information VI, it outputs the generated change amount information VI to the coordinate conversion unit 3125. As an example, the tracking unit 3123 may calculate all of the changes in position and attitude of the 6 DOF (that is, the change amount ΔTx(2D), the change amount ΔTy(2D), the change amount ΔTz(2D), the change amount ΔRx(2D), the change amount ΔRy(2D), and the change amount ΔRz(2D), or the change amount ΔTx(3D), the change amount ΔTy(3D), the change amount ΔTz(3D), the change amount ΔRx(3D), the change amount ΔRy(3D), and the change amount ΔRz(3D)), and output the change amount information VI indicating the calculated changes in position and attitude of the 6 DOF to the coordinate conversion unit 3125. As another example, the tracking unit 3123 may calculate a portion of the change in position and posture of the 6 DOF, and output change amount information VI indicating a portion of the calculated change in position and posture of the 6 DOF to the coordinate conversion unit 3125.

[0219] The tracking unit 3123 may calculate the amount of change in the position and orientation of the object OBJ at a predetermined tracking period. In other words, the tracking unit 3123 may calculate the amount of change in the position and orientation of the object OBJ each time a time period equivalent to a tracking period has elapsed. Conversely, it may be considered that a time period equivalent to the tracking period is required from when the tracking unit 3123 starts the tracking process until it is completed. In this case, the tracking unit 3123 may output change amount information VI indicating the amount of change in the position and orientation of the object OBJ to the coordinate conversion unit 3125 at a predetermined tracking period.

[0220] As shown in FIG. 18 , the tracking period may be the same as the 2D imaging period corresponding to the 2D imaging rate at which the imaging device 21 captures images of the object OBJ. For example, the tracking period may be a period at which the tracking unit 3123 can perform tracking processing tens to hundreds of times (for example, 500 times) per second. In this case, for example, as shown in FIG. 18 , simultaneously with or after image data IMG_2D#ta is generated at time ta, the tracking unit 3123 may perform tracking processing using image data IMG_2D#ta and IMG_2D#ta-1 generated at time ta and time ta-1 prior to time ta, and three-dimensional position data WSD generated using image data IMG_3D#ta generated at time ta. As a result, the amount of change in the position and orientation of the object OBJ between times ta-1 and ta is calculated. Similarly, for example, as shown in FIG. 18 , the tracking unit 3123 may perform tracking processing using three-dimensional position data WSD generated using image data IMG_2D#ta+1 and IMG_2D#ta generated at time ta+1 and a time ta before time ta+1, and image data IMG_3D#ta+1 generated at time ta+1, simultaneously with or after image data IMG_2D#ta+1 is generated at time ta+1. As a result, the amount of change in the position and orientation of the object OBJ between times ta and ta+1 is calculated. Note that the timing at which the image data IMG_2D shown in FIG. 18 is generated may be considered to be the timing at which the imaging device 21 captures an image of the object OBJ to generate the image data IMG_2D.

[0221] Furthermore, as shown in Fig. 18, the tracking period may be the same as the 3D imaging period corresponding to the 3D imaging rate at which the imaging device 22 captures images of the object OBJ. In this case, for example, as shown in Fig. 18, the tracking unit 3123 may perform tracking processing using the three-dimensional position data WSD generated using the image data IMG_3D#ta generated at time ta, and the image data IMG_2D#ta and IMG_2D#ta-1 generated at time ta and time ta-1, which is before time ta, simultaneously with or after the image data IMG_3D#ta is generated at time ta. As a result, the amount of change in the position and orientation of the object OBJ between time ta-1 and ta is calculated. Similarly, for example, as shown in FIG. 18 , the tracking unit 3123 may perform tracking processing using the three-dimensional position data WSD generated using the image data IMG_3D#ta+1 generated at time ta+1, and the image data IMG_2D#ta+1 and IMG_2D#ta generated at time ta+1 and a time ta before time ta+1, simultaneously with or after the generation of image data IMG_3D#ta+1 at time ta+1. As a result, the amount of change in the position and orientation of the object OBJ between times ta and ta+1 is calculated. Note that the timing at which the image data IMG_3D shown in FIG. 18 is generated may be considered to be the timing at which the imaging device 22 captures the object OBJ to generate the image data IMG_3D.

[0222] However, the tracking period does not have to be the same as the 2D imaging period. For example, the tracking period may be shorter than the 2D imaging period. For example, the tracking period may be longer than the 2D imaging period. Similarly, the tracking period does not have to be the same as the 3D imaging period. For example, the tracking period may be shorter than the 3D imaging period. For example, the tracking period may be longer than the 3D imaging period.

[0223] Furthermore, as shown in FIG. 18 , the tracking period is shorter than the 2D matching period in which the 2D matching unit 3121 performs the 2D matching process. In other words, the 2D matching period is longer than the tracking period. Specifically, in the tracking process, the calculation device 31 only needs to perform calculations focusing on data portions of the image data IMG_2D and the three-dimensional position data WSD related to characteristic locations of the object OBJ, whereas in the matching process, the calculation device 31 needs to perform calculations to search for the object OBJ from the entire image data IMG_2D and the three-dimensional position data WSD. As a result, the amount of calculation required for the tracking process is usually less than the amount of calculation required for the matching process. For this reason, the tracking period is usually shorter than the 2D matching period. For this reason, the tracking unit 3123 may perform the tracking process multiple times between the start and end of one 2D matching period. In other words, the tracking unit 3123 may perform the tracking process multiple times between the start and end of the 2D matching process by the 2D matching unit 3121.

[0224] 18 , the 2D matching period may be considered to be equivalent to the time from when the object OBJ is imaged by the imaging device 21 until the position and orientation information POI1 is generated based on the image data IMG_2D. Similarly, the tracking period may be considered to be equivalent to the time from when the object OBJ is imaged by the imaging device 21 at the second time t2 until the change amount information VI is generated based on the two image data IMG_2D#t1 and IMG_2D#t2 generated at the first time t1 and the second time t2, and the three-dimensional position data WSD generated using the image data IMG_3D generated at the second time t2. For example, in the example shown in Figure 18, the tracking period may be considered to be equivalent to the time from when the object OBJ is imaged by the imaging device 21 at time ta, which is an example of the second time t2, to when the change amount information VI is generated based on the two image data IMG_2D generated at time ta-1, which is an example of the first time t1 and the second time t2, and the three-dimensional position data WSD generated using the image data IMG_3D#ta generated at time ta.

[0225] The tracking period may be considered to be equivalent to the time from when the object OBJ is imaged by the imaging device 21 at a first time t1 to when the change amount information VI is generated based on the two image data IMG_2D#t1 and IMG_2D#t2 generated at the first time t1 and the second time t2, and the three-dimensional position data WSD generated using the image data IMG_3D generated at the second time t2. For example, in the example shown in FIG. 18 , the tracking period may be considered to be equivalent to the time from when the object OBJ is imaged by the imaging device 21 at time ta-1, which is an example of the first time t1, to when the change amount information VI is generated based on the two image data IMG_2D generated at times ta-1 and ta, which are examples of the first time t1 and the second time t2, and the three-dimensional position data WSD generated using the image data IMG_3D#ta generated at time ta. Even in this case, the tracking period may be shorter than the 2D matching period.

[0226] Furthermore, when the tracking period is shorter than the 2D matching period, the tracking unit 3123 may perform the tracking process using image data IMG_2D that is different from the image data IMG_2D used for the 2D matching process, as shown in Fig. 18. For example, as shown in Fig. 18, the 2D matching unit 3121 performs the 2D matching process using image data IMG_2D generated at time ta, and after the 2D matching process using the image data IMG_2D generated at time ta is completed, performs the 2D matching process using image data IMG_2D generated at time tb. In this case, the tracking unit 3123 may perform the tracking process using two pieces of image data IMG_2D generated at two times different from times ta and tb, and three-dimensional position data WSD generated using image data IMG_3D generated at a time different from times ta and tb. For example, in the example shown in FIG. 18, the tracking unit 3123 may calculate the amount of change in the position and attitude of the object OBJ between times ta+1 and ta+2 by performing a tracking process using two pieces of image data IMG_2D generated at two times ta+1 and ta+2 (or any two times in the period between time ta and time tb) and three-dimensional position data WSD generated using image data IMG_3D generated at time ta+2 (or any time in the period between time ta and time tb).

[0227] However, the tracking unit 3123 may perform the tracking process using image data IMG_2D used for the 2D matching process. For example, in the example shown in Fig. 18 , the tracking unit 3123 may calculate the amount of change in the position and orientation of the object OBJ between times ta and ta+1 by performing the tracking process using two pieces of image data IMG_2D generated at times ta and ta+1, respectively, and three-dimensional position data WSD generated using image data IMG_3D generated at time ta+1.

[0228] 18 , the tracking period is shorter than the 3D matching period in which the above-described 3D matching unit 3122 performs the 3D matching process. In other words, the 3D matching period is longer than the tracking period. Therefore, the tracking unit 3123 may perform the tracking process multiple times between the start and end of one 3D matching period. In other words, the tracking unit 3123 may perform the tracking process multiple times between the start and end of the 3D matching process by the 3D matching unit 3122.

[0229] As shown in FIG. 18, the 3D matching period may be considered to be equivalent to the time from when the object OBJ is captured by the imaging device 22 until the position and orientation information POI2 is generated based on the three-dimensional position data WSD generated using the image data IMG_3D.

[0230] If the tracking period is shorter than the 3D matching period, as shown in FIG. 18 , the tracking unit 3123 may perform the tracking process using three-dimensional position data WSD generated using image data IMG_3D that is different from the image data IMG_3D used for the 3D matching process. For example, as shown in FIG. 18 , the 3D matching unit 3122 performs the 3D matching process using the three-dimensional position data WSD generated from the image data IMG_3D generated at time ta, and after the 3D matching process using the three-dimensional position data WSD generated from the image data IMG_3D generated at time ta is completed, performs the 3D matching process using the three-dimensional position data WSD generated from the image data IMG_3D generated at time tb. In this case, the tracking unit 3123 may perform the tracking process using the three-dimensional position data WSD generated from the image data IMG_3D generated at a time different from times ta and tb, and two pieces of image data IMG_2D generated at two times different from times ta and tb. For example, in the example shown in Figure 18, the tracking unit 3123 may calculate the amount of change in the position and orientation of the object OBJ between times ta+1 and ta+2 by performing tracking processing using three-dimensional position data WSD generated from image data IMG_3D generated at time ta+2 (or any time in the period between time ta and time tb) and two image data IMG_2D#ta+2 and IMG_2D#ta+1 generated at time ta+2 and time ta+1, which is before time ta+2, respectively.

[0231] However, the tracking unit 3123 may perform the tracking process using three-dimensional position data WSD generated using image data IMG_3D used for the 3D matching process. For example, in the example shown in Fig. 18, the tracking unit 3123 may calculate the amount of change in the position and orientation of the object OBJ between times ta-1 and ta by performing the tracking process using three-dimensional position data WSD generated from image data IMG_3D generated at time ta and image data IMG_2D#ta and IMG_2D#ta-1 generated at time ta and time ta-1 before time ta.

[0232] However, the tracking period does not have to be shorter than the 2D matching period. For example, the tracking period may be the same as the 2D matching period. For example, the tracking period may be longer than the 2D matching period. Similarly, the tracking period does not have to be shorter than the 3D matching period. For example, the tracking period may be the same as the 3D matching period. For example, the tracking period may be longer than the 3D matching period.

[0233] The tracking unit 3123 does not have to calculate the amount of change in the position and orientation of the object OBJ in the 2D imaging coordinate system between the first time t1 and the second time t2. For example, the tracking unit 3123 may calculate the position and orientation of the object OBJ in the 2D imaging coordinate system at the first time t1 and the second time t2 by performing the above-described matching process on each of the two image data IMG_2D#t1 and IMG_2D#t2. Note that, because the position and orientation of the object OBJ are calculated for different times (e.g., times t1 and t2), the process performed by the tracking unit 3123 here can also be considered a tracking process. Here, the tracking unit 3123 may perform the matching process on each of the two image data IMG_2D#t1 and IMG_2D#t2 using a smaller number of feature locations than the feature locations of the object OBJ used in the matching process by the above-described 2D matching unit 3121. In this case, the tracking unit 3123 can reduce the calculation time required for the matching process, and therefore can perform the tracking process in a shorter cycle than the matching process performed by the 2D matching unit 3121.

[0234] (2-2-4) Position Calculation Process Performed by Coordinate Transformation Unit 3125 The coordinate transformation unit 3125 calculates the position of the object OBJ in the global coordinate system based on the results of the matching process and the tracking process. That is, the coordinate transformation unit 3125 calculates the position Tx(GL), position Ty(GL), position Tz(GL), orientation Rx(GL), orientation Ry(GL), and orientation Rz(GL) as the position of the object OBJ in the global coordinate system. The coordinate transformation unit 3125 generates position and orientation information POI0 indicating the position Tx(GL), position Ty(GL), position Tz(GL), orientation Rx(GL), orientation Ry(GL), and orientation Rz(GL).

[0235] In order to calculate the position of the object OBJ in the global coordinate system, the coordinate transformation unit 3125 corrects the result of the matching process based on the result of the tracking process. Specifically, as shown in Fig. 10 , the coordinate transformation unit 3125 acquires position and orientation information POI2 indicating position Tx(3D), position Ty(3D), position Tz(3D), orientation Rx(3D), orientation Ry(3D), and orientation Rz(3D) from the 3D matching unit 3122 as a result of the matching process. Furthermore, as a result of the tracking process, the coordinate conversion unit 3125 acquires, from the tracking unit 3123, change amount information VI indicating the change amount ΔTx(2D), the change amount ΔTy(2D), the change amount ΔTz(2D), the change amount ΔRx(2D), the change amount ΔRy(2D), and the change amount ΔRz(2D); change amount information VI indicating the change amount ΔTx(3D), the change amount ΔTy(3D), the change amount ΔTz(3D), the change amount ΔRx(3D), the change amount ΔRy(3D), and the change amount ΔRz(3D); or change amount information VI indicating the change amount ΔTx(GL), the change amount ΔTy(GL), the change amount ΔTz(GL), the change amount ΔRx(GL), the change amount ΔRy(GL), and the change amount ΔRz(GL).

[0236] In this case, the coordinate conversion unit 3125 first matches the coordinate system of the position and orientation information POI2 with the coordinate system of the change amount information VI.

[0237] For example, when the coordinate conversion unit 3125 acquires change amount information VI indicating the change amounts ΔTx(2D), ΔTy(2D), ΔTz(2D), ΔRx(2D), ΔRy(2D), and ΔRz(2D), the coordinate system of the position and orientation information POI2 is a 3D imaging coordinate system, while the coordinate system of the change amount information VI is a 2D imaging coordinate system. In this case, the coordinate conversion unit 3125 may use the above-described coordinate system information (e.g., a transformation matrix M(3D to 2D) for transforming a position in the 3D imaging coordinate system into a position in the 2D imaging coordinate system) to convert the position and orientation information POI2 indicating the position and orientation of the object OBJ in the 3D imaging coordinate system into position and orientation information POI2 indicating the position and orientation of the object OBJ in the 2D imaging coordinate system, thereby matching the coordinate system of the position and orientation information POI2 with the coordinate system of the change amount information VI. Alternatively, the coordinate conversion unit 3125 may use the above-mentioned coordinate system information (for example, a transformation matrix M (2D to 3D) for converting a position in the 2D imaging coordinate system into a position in the 3D imaging coordinate system) to convert the change amount information VI indicating the change amount of the position and orientation of the object OBJ in the 2D imaging coordinate system into change amount information VI indicating the change amount of the position and orientation of the object OBJ in the 3D imaging coordinate system, thereby aligning the coordinate system of the position and orientation information POI2 with the coordinate system of the change amount information VI.

[0238] When the above-described coordinate system information (for example, the transformation matrix M (3D to 2D) or the transformation matrix M (2D to 3D)) is used, the imaging devices 21 and 22 may be arranged so that the origin of the 2D imaging coordinate system is as close as possible to the origin of the 3D imaging coordinate system. The imaging devices 21 and 22 may be arranged so that the X-axis (2D) of the 2D imaging coordinate system is as close as possible to the X-axis (3D) of the 3D imaging coordinate system. The imaging devices 21 and 22 may be arranged so that the Y-axis (2D) of the 2D imaging coordinate system is as close as possible to the Y-axis (3D) of the 3D imaging coordinate system. The imaging devices 21 and 22 may be arranged so that the Z-axis (2D) of the 2D imaging coordinate system is as close as possible to the Z-axis (3D) of the 3D imaging coordinate system.

[0239] Alternatively, the coordinate conversion unit 3125 may align both the coordinate system of the position and orientation information POI2 and the coordinate system of the change amount information VI with the global coordinate system. In this case, the coordinate conversion unit 3125 may convert the position and orientation information POI2 indicating the position and orientation of the object OBJ in the 3D imaging coordinate system into position and orientation information POI2 indicating the position and orientation of the object OBJ in the global coordinate system using a transformation matrix M(3DtoGL) for converting a position in the 3D imaging coordinate system into a position in the global coordinate system. Furthermore, the coordinate conversion unit 3125 may convert the change amount information VI indicating the change amount of the position and orientation of the object OBJ in the 2D imaging coordinate system into change amount information VI indicating the change amount of the position and orientation of the object OBJ in the global coordinate system using a transformation matrix M(2DtoGL) for converting a position in the 2D imaging coordinate system into a position in the global coordinate system.

[0240] The transformation matrix M(2D to GL) may include, for example, a product of transformation matrices that reflect changes in the position coordinates of the image capture device 21 due to rotation of the link 121 around an axis defined by each joint 122 of the robot arm 12. Similarly, the transformation matrix M(3D to GL) may include, for example, a product of transformation matrices that reflect changes in the position coordinates of the image capture device 22 due to rotation of the link 121 around an axis defined by each joint 122 of the robot arm 12. This transformation matrix may be a so-called rotation matrix, a matrix that includes a translational component in a rotation matrix, or a matrix based on Euler angles. Since an existing transformation method may be used for the coordinate transformation of the robot arm using the transformation matrix, a detailed description thereof will be omitted.

[0241] On the other hand, for example, when the coordinate conversion unit 3125 acquires change amount information VI indicating the change amounts ΔTx(3D), ΔTy(3D), ΔTz(3D), ΔRx(3D), ΔRy(3D), and ΔRz(3D), the coordinate system of the position and orientation information POI2 and the coordinate system of the change amount information VI are both 3D imaging coordinate systems. In other words, the coordinate system of the position and orientation information POI2 and the coordinate system of the change amount information VI already match. In this case, the coordinate conversion unit 3125 does not need to perform additional processing to match the coordinate system of the position and orientation information POI2 and the coordinate system of the change amount information VI.

[0242] Alternatively, even when the coordinate conversion unit 3125 acquires change amount information VI indicating the change amounts ΔTx(3D), ΔTy(3D), ΔTz(3D), ΔRx(3D), ΔRy(3D), and ΔRz(3D), the coordinate conversion unit 3125 may align both the coordinate system of the position and orientation information POI2 and the coordinate system of the change amount information VI with the global coordinate system. In this case, the coordinate conversion unit 3125 may use a transformation matrix M(3DtoGL) to convert the position and orientation information POI2 indicating the position and orientation of the object OBJ in the 3D imaging coordinate system into position and orientation information POI2 indicating the position and orientation of the object OBJ in the global coordinate system. Furthermore, the coordinate conversion unit 3125 may use a conversion matrix M(3DtoGL) to convert change amount information VI indicating the change amount of the position and orientation of the object OBJ in the 3D imaging coordinate system into change amount information VI indicating the change amount of the position and orientation of the object OBJ in the global coordinate system.

[0243] On the other hand, for example, when the coordinate conversion unit 3125 acquires change amount information VI indicating the change amounts ΔTx(GL), ΔTy(GL), ΔTz(GL), ΔRx(GL), ΔRy(GL), and ΔRz(GL), the coordinate system of the position and orientation information POI2 is the 3D imaging coordinate system, while the coordinate system of the change amount information VI is the global coordinate system. In this case, the coordinate conversion unit 3125 may use a transformation matrix M(3DtoGL) to convert the position and orientation information POI2 indicating the position and orientation of the object OBJ in the 3D imaging coordinate system into position and orientation information POI2 indicating the position and orientation of the object OBJ in the global coordinate system, thereby matching the coordinate system of the position and orientation information POI2 with the coordinate system of the change amount information VI. However, the coordinate conversion unit 3125 may use a transformation matrix M (GLto3D) for converting a position in the global coordinate system into a position in the 3D imaging coordinate system to convert the change amount information VI indicating the change amount of the position and orientation of the object OBJ in the global coordinate system into change amount information VI indicating the change amount of the position and orientation of the object OBJ in the 3D imaging coordinate system, thereby aligning the coordinate system of the position and orientation information POI2 with the coordinate system of the change amount information VI.

[0244] After matching the coordinate system of the position and orientation information POI2 with the coordinate system of the change amount information VI, the coordinate conversion unit 3125 corrects the position and orientation information POI2, which is the result of the matching process, based on the change amount information VI, which is the result of the tracking process.

[0245] For example, if the coordinate system of the position and orientation information POI2 and the coordinate system of the change amount information VI are both 3D imaging coordinate systems, the coordinate conversion unit 3125 may calculate the position Tx'(3D) of the object OBJ in the X-axis direction (3D) of the 3D imaging coordinate system by correcting the position Tx(3D) based on the change amount ΔTx(3D). The coordinate conversion unit 3125 may calculate the position Ty'(3D) of the object OBJ in the Y-axis direction (3D) of the 3D imaging coordinate system by correcting the position Ty(3D) based on the change amount ΔTy(3D). The coordinate conversion unit 3125 may calculate the position Tz'(3D) of the object OBJ in the Z-axis direction (3D) of the 3D imaging coordinate system by correcting the position Tz(3D) based on the change amount ΔTz(3D). The coordinate conversion unit 3125 may calculate the orientation Rx'(3D) of the object OBJ about the X-axis (3D) of the 3D imaging coordinate system by correcting the orientation Rx(3D) based on the amount of change ΔRx(3D). The coordinate conversion unit 3125 may calculate the orientation Ry'(3D) of the object OBJ about the Y-axis (3D) of the 3D imaging coordinate system by correcting the orientation Ry(3D) based on the amount of change ΔRy(3D). The coordinate conversion unit 3125 may calculate the orientation Rz'(3D) of the object OBJ about the Z-axis (3D) of the 3D imaging coordinate system by correcting the orientation Rz(3D) based on the amount of change ΔRz(3D). In this case, the corrected position and orientation information POI2 becomes position and orientation information POI2' indicating position Tx' (3D), position Ty' (3D), position Tz' (3D), orientation Rx' (3D), orientation Ry' (3D), and orientation Rz' (3D).

[0246] For example, if the coordinate system of the position and orientation information POI2 and the coordinate system of the change amount information VI are both 2D imaging coordinate systems, the coordinate conversion unit 3125 may calculate the position Tx'(2D) of the object OBJ in the X-axis direction (2D) of the 2D imaging coordinate system by correcting the position Tx(2D) based on the change amount ΔTx(2D). The coordinate conversion unit 3125 may calculate the position Ty'(2D) of the object OBJ in the Y-axis direction (2D) of the 2D imaging coordinate system by correcting the position Ty(2D) based on the change amount ΔTy(2D). The coordinate conversion unit 3125 may calculate the position Tz'(2D) of the object OBJ in the Z-axis direction (2D) of the 2D imaging coordinate system by correcting the position Tz(2D) based on the change amount ΔTz(2D). The coordinate conversion unit 3125 may calculate the orientation Rx'(2D) of the object OBJ about the X-axis (2D) of the 2D imaging coordinate system by correcting the orientation Rx(2D) based on the amount of change ΔRx(2D). The coordinate conversion unit 3125 may calculate the orientation Ry'(2D) of the object OBJ about the Y-axis (2D) of the 2D imaging coordinate system by correcting the orientation Ry(2D) based on the amount of change ΔRy(2D). The coordinate conversion unit 3125 may calculate the orientation Rz'(2D) of the object OBJ about the Z-axis (2D) of the 2D imaging coordinate system by correcting the orientation Rz(2D) based on the amount of change ΔRz(2D). In this case, the corrected position and orientation information POI2 becomes position and orientation information POI2' indicating position Tx' (2D), position Ty' (2D), position Tz' (2D), orientation Rx' (2D), orientation Ry' (2D), and orientation Rz' (2D).

[0247] For example, if the coordinate system of the position and orientation information POI2 and the coordinate system of the change amount information VI are both global coordinate systems, the coordinate conversion unit 3125 may calculate the position Tx'(GL) of the object OBJ in the X-axis direction (GL) of the global coordinate system by correcting the position Tx(GL) based on the change amount ΔTx(GL). The coordinate conversion unit 3125 may calculate the position Ty'(GL) of the object OBJ in the Y-axis direction (GL) of the global coordinate system by correcting the position Ty(GL) based on the change amount ΔTy(GL). The coordinate conversion unit 3125 may calculate the position Tz'(GL) of the object OBJ in the Z-axis direction (GL) of the global coordinate system by correcting the position Tz(GL) based on the change amount ΔTz(GL). The coordinate conversion unit 3125 may calculate the orientation Rx'(GL) of the object OBJ around the X-axis (GL) of the global coordinate system by correcting the orientation Rx(GL) based on the amount of change ΔRx(GL). The coordinate conversion unit 3125 may calculate the orientation Ry'(GL) of the object OBJ around the Y-axis (GL) of the global coordinate system by correcting the orientation Ry(GL) based on the amount of change ΔRy(GL). The coordinate conversion unit 3125 may calculate the orientation Rz'(GL) of the object OBJ around the Z-axis (GL) of the global coordinate system by correcting the orientation Rz(GL) based on the amount of change ΔRz(GL). In this case, the corrected position and orientation information POI2 becomes position and orientation information POI2' indicating position Tx'(GL), position Ty'(GL), position Tz'(GL), orientation Rx'(GL), orientation Ry'(GL), and orientation Rz'(GL).

[0248] In the following description, in order to avoid redundant explanation, the processing performed by the coordinate conversion unit 3125 will be described when the coordinate system of the position and orientation information POI2 and the coordinate system of the change amount information VI are both 3D imaging coordinate systems. However, even when the coordinate system of the position and orientation information POI2 and the coordinate system of the change amount information VI are both 2D imaging coordinate systems, the coordinate conversion unit 3125 can perform operations similar to those described below. Specifically, by replacing the word "3D" with the word "2D," the description of the operation of the coordinate conversion unit 3125 described below can be used as a description of the operation of the coordinate conversion unit 3125 when the coordinate system of the position and orientation information POI2 and the coordinate system of the change amount information VI are both 2D imaging coordinate systems. Alternatively, even when the coordinate system of the position and orientation information POI2 and the coordinate system of the change amount information VI are both global coordinate systems, the coordinate conversion unit 3125 can perform operations similar to those described below. Specifically, the explanation of the operation of the coordinate conversion unit 3125 described below can be used as an explanation of the operation of the coordinate conversion unit 3125 when the coordinate system of the position and orientation information POI2 and the coordinate system of the change amount information VI are both global coordinate systems by replacing the terms "3D" and "3D imaging coordinate system" with the terms "GL" and "global coordinate system."

[0249] The process of correcting the position and orientation information POI2 based on the change amount information VI may include a process of adding the change amount information VI to the position and orientation information POI2. For example, the coordinate conversion unit 3125 may calculate a position Tx'(3D) by adding a change amount ΔTx(3D) to a position Tx(3D). The coordinate conversion unit 3125 may calculate a position Ty'(3D) by adding a change amount ΔTy(3D) to a position Ty(3D). The coordinate conversion unit 3125 may calculate a position Tz'(3D) by adding a change amount ΔTz(3D) to a position Tz(3D). The coordinate conversion unit 3125 may calculate a orientation Rx'(3D) by adding a change amount ΔRx(3D) to an orientation Rx(3D). The coordinate conversion unit 3125 may calculate the orientation Ry'(3D) by adding the change amount ΔRy(3D) to the orientation Ry(3D). The coordinate conversion unit 3125 may calculate the orientation Rz'(3D) by adding the change amount ΔRz(3D) to the orientation Rz(3D).

[0250] As described above, the tracking period, which is the period during which the change amount ΔTx(3D) is calculated, is shorter than the 3D matching period, which is the period during which the position Tx(3D) is calculated. In this case, the tracking unit 3123 calculates the change amount ΔTx(3D) multiple times between when the position Tx(3D) is newly calculated and when the position Tx(3D) is next calculated. In this case, the coordinate conversion unit 3125 may calculate the position Tx′(3D) based on the newly calculated position Tx(3D) and the newly calculated change amount ΔTx(3D). Specifically, each time the change amount ΔTx(3D) is newly calculated, the coordinate conversion unit 3125 may calculate the position Tx′(3D) by adding the sum of the change amounts ΔTx(3D) calculated from the start of the matching process for calculating the position Tx(3D) to the current time to the position Tx(3D). Alternatively, the coordinate conversion unit 3125 may calculate a new position Tx'(3D) by adding the newly calculated amount of change ΔTx(3D) to the position Tx'(3D) at that time (i.e., the position Tx(3D) to which the calculated amount of change ΔTx(3D) has already been added) each time a new amount of change ΔTx(3D) is calculated. Therefore, each time a change amount ΔTx(3D) is calculated, the total sum of the amounts of change ΔTx(3D) added to the position Tx(3D) increases.

[0251] In this case, the update frequency of the position Tx'(3D) is higher than in a comparative example in which the position Tx(3D) is used as the position Tx'(3D) without modification. Specifically, when the position Tx(3D) is used as the position Tx'(3D) without modification, the update frequency of the position Tx'(3D) depends on the 3D matching cycle, which is the cycle for calculating the position Tx(3D). On the other hand, when the position Tx'(3D) is calculated by correcting the position Tx(3D) based on the change amount ΔTx(3D), the update frequency of the position Tx'(3D) depends on the tracking cycle, which is the cycle for calculating the change amount ΔTx(3D). As a result, because the tracking cycle is shorter than the 3D matching cycle, the update frequency of the position Tx'(3D) is higher. Because the position of the object OBJ in the global coordinate system is calculated based on the position Tx'(3D), as will be described later, the coordinate conversion unit 3125 can calculate the position of the object OBJ in the global coordinate system with the same cycle as the tracking cycle. Therefore, the position and orientation calculation unit 312 can calculate the position of the object OBJ in the global coordinate system at shorter intervals. In other words, the position and orientation calculation unit 312 can calculate the position of the object OBJ in the global coordinate system more frequently (in other words, at higher speed).

[0252] However, the change amount ΔTx(3D) calculated by the tracking process may contain an error corresponding to the deviation from the actual position of the object OBJ. As a result, the error contained in the sum of the change amounts ΔTx(3D), which continues to increase each time the change amount ΔTx(3D) is calculated, may also increase. As a result, each time the change amount ΔTx(3D) is added, the deviation between the position Tx'(3D) of the object OBJ calculated by the control device 3 and the actual position of the object OBJ may increase. On the other hand, the accuracy of the position calculated by the matching process is higher than the accuracy of the position calculated by the tracking process. Therefore, when a new position Tx(3D) is calculated, the coordinate conversion unit 3125 may reset the range of the change amount ΔTx(3D) to be added to the position Tx(3D) to calculate the position Tx'(3D). Specifically, when a position Tx(3D) at a certain time is newly calculated, the coordinate conversion unit 3125 may calculate a position Tx'(3D) by adding to the position Tx(3D) the sum of the amount of change ΔTx(3D) calculated from the certain time to the current time. When a position Tx(3D) at a certain time is newly calculated, the coordinate conversion unit 3125 may calculate a position Tx'(3D) by adding to the position Tx(3D) the sum of the amount of change ΔTx(3D) calculated from the time when the matching process for calculating the position Tx(3D) at the certain time was started to the current time. As a result, the position and orientation calculation unit 312 can calculate the position Tx'(3D) with high accuracy.

[0253] An example of the flow for calculating the position Tx' (3D) is shown in Fig. 19. Fig. 19 shows an example in which the imaging devices 21 and 22 capture images of the object OBJ at time t#0_0, thereby generating image data IMG_2D#0_0 and IMG_3D#0_0, and the 2D matching unit 3121 starts the 2D matching process at time t#0_0. In this case, at time t#1_0, when a 2D matching cycle has elapsed since time t#0_0, the 2D matching unit 3121 completes generation of position and orientation information POI1#0_0, which indicates the position and orientation of the object OBJ at time t#0_0.

[0254] Furthermore, while the 2D matching unit 3121 is performing 2D matching processing using the image data IMG_2D#0_0, the three-dimensional position data generation unit 311 is performing processing to generate three-dimensional position data WSD#0_0 from the image data IMG_3D#0_0. For example, the three-dimensional position data generation unit 311 may start processing to generate three-dimensional position data WSD#0_0 from the image data IMG_3D#0_0 at the same time that the 2D matching unit 3121 starts 2D matching processing using the image data IMG_2D#0_0. For example, after the 2D matching unit 3121 starts 2D matching processing using the image data IMG_2D#0_0, the three-dimensional position data generation unit 311 may start processing to generate three-dimensional position data WSD#0_0 from the image data IMG_3D#0_0. As a result, three-dimensional position data WSD#0_0 indicating the three-dimensional position of the object OBJ at time t#0_0 is generated. Furthermore, the three-dimensional position data generation unit 311 may perform a process of generating three-dimensional position data WSD#0_0 from image data IMG_3D#0_0 so that the generation of three-dimensional position data WSD#0_0 is completed at the time the 2D matching unit 3121 completes the 2D matching process using image data IMG_2D#0_0.

[0255] Thereafter, at time t#1_0, the 3D matching unit 3122 starts 3D matching processing based on the three-dimensional position data WSD#0_0 and the position and orientation information POI1#0_0. That is, the 3D matching unit 3122 performs position and orientation determination processing to determine the initial position and initial orientation of the three-dimensional model WM3 based on the position and orientation information POI1#0_0 generated at time t#1_0, and then performs matching processing using the three-dimensional position data WSD#0_0 and the three-dimensional model WM3 placed at the determined initial position and in the determined initial orientation.

[0256] As described above, if the 3D matching process does not include a position and orientation determination process, the 3D matching unit 3122 may perform the position and orientation determination process at time t#1_0 based on the position and orientation information POI1#0_0. After that, after the initial position and initial orientation of the three-dimensional model WM3 are determined by the position and orientation determination process, the 3D matching unit 3122 may start the 3D matching process using the three-dimensional position data WSD#0_0 and the three-dimensional model WM3 placed at the initial position and in the initial orientation. Therefore, if the 3D matching process does not include a position and orientation determination process, the 3D matching unit 3122 may start the 3D matching process after taking into account the time required to complete the position and orientation determination process.

[0257] As a result, at time t#2_0, when the 3D matching cycle has elapsed since time t#1_0, the 3D matching unit 3122 completes the generation of position and orientation information POI2#0_0 indicating the position and orientation of the object OBJ at time t#0_0 (in the example shown in FIG. 19 , calculation of position Tx(3D)#0_0 in the X-axis direction (3D)). Here, the position and orientation of the object OBJ at time t#2_0, when the 3D matching unit 3122 completes the generation of the position and orientation information POI2#0_0, may differ from the position and orientation of the object OBJ at time t#0_0 indicated by the position and orientation information POI2#0_0 (i.e., the time when the matching process for generating the position and orientation information POI2#0_0 was started). However, the amount of change in the position and orientation of the object between time t#0_0 and time t#2_0 has been calculated by the tracking process. For this reason, the coordinate conversion unit 3125 adds the sum of the amounts of change calculated from time t#0_0 to time t#2_0, which corresponds to the current time, to position and orientation information POI2#0_0, which indicates the position and orientation of the object at time t#0_0. For example, the coordinate conversion unit 3125 calculates the position Tx'(3D) of the object OBJ at time t#2_0 by adding the sum of the amounts of change calculated from time t#0_0 to time t#2_0, which corresponds to the current time, to position Tx(3D)#0_0. In the example shown in Figure 19, between time t#0_0 and time t#2_0, the tracking process calculates the change ΔTx(3D)#0_0 between time t#0_0 and time t#0_1, the change ΔTx(3D)#0_1 between time t#0_1 and time t#0_2, ..., the change ΔTx(3D)#0_N between time t#0_N and time t#1_0, the change ΔTx(3D)#1_0 between time t#1_0 and time t#1_1, the change ΔTx(3D)#1_1 between time t#1_1 and time t#1_2, ..., the change ΔTx(3D)#1_N between time t#1_N and time t#2_0.Therefore, from time t#2_0 onwards, position Tx(3D)#0_0 + change amount ΔTx(3D)#0_0 + change amount ΔTx(3D)#0_1 + ... + change amount ΔTx(3D)#0_N + change amount ΔTx(3D)#1_0 + change amount ΔTx(3D)#1_1 + ... + change amount ΔTx(3D)#1_N is used as position Tx'(3D).

[0258] Thereafter, at time t#2_1, a change amount ΔTx(3D)#2_0 between time t#2_0 and time t#2_1 is newly calculated by tracking processing. In this case, the coordinate conversion unit 3125 further adds the newly calculated change amount ΔTx(3D)#2_0 to the current position Tx'(3D). In other words, from time t#2_1 onwards, the position Tx(3D)#0_0 + change amount ΔTx(3D)#0_0 + change amount ΔTx(3D)#0_1 + ... + change amount ΔTx(3D)#0_N + change amount ΔTx(3D)#1_0 + change amount ΔTx(3D)#1_1 + ... + change amount ΔTx(3D)#1_N + change amount ΔTx(3D)#2_0 is used as the position Tx'(3D).

[0259] Thereafter, at time t#2_2, a change amount ΔTx(3D)#2_1 between time t#2_1 and time t#2_2 is newly calculated by tracking processing. In this case, the coordinate conversion unit 3125 further adds the newly calculated change amount ΔTx(3D)#2_1 to the current position Tx'(3D). In other words, from time t#2_1 onwards, the position Tx(3D)#0_0 + change amount ΔTx(3D)#0_0 + change amount ΔTx(3D)#0_1 + ... + change amount ΔTx(3D)#0_N + change amount ΔTx(3D)#1_0 + change amount ΔTx(3D)#1_1 + ... + change amount ΔTx(3D)#1_N + change amount ΔTx(3D)#2_0 + change amount ΔTx(3D)#2_1 is used as the position Tx'(3D).

[0260] On the other hand, at time t#1_0, the 2D matching unit 3121 completes generation of position and orientation information POI1#0_0 indicating the position and orientation of object OBJ at time t#0_0, and then starts a new 2D matching process using image data IMG_2D#1_0 generated by capturing an image of object OBJ by the imaging device 21 at time t#1_0. In this case, at time t#2_0, when a 2D matching cycle has elapsed since time t#1_0, the 2D matching unit 3121 completes generation of position and orientation information POI1#1_0 indicating the position and orientation of object OBJ at time t#1_0.

[0261] Furthermore, while the 2D matching unit 3121 is performing 2D matching processing using the image data IMG_2D#1_0, the three-dimensional position data generation unit 311 is performing processing to generate three-dimensional position data WSD#1_0 from the image data IMG_3D#1_0. For example, the three-dimensional position data generation unit 311 may start processing to generate three-dimensional position data WSD#1_0 from the image data IMG_3D#1_0 at the same time that the 2D matching unit 3121 starts 2D matching processing using the image data IMG_2D#1_0. For example, after the 2D matching unit 3121 starts 2D matching processing using the image data IMG_2D#1_0, the three-dimensional position data generation unit 311 may start processing to generate three-dimensional position data WSD#1_0 from the image data IMG_3D#1_0. As a result, three-dimensional position data WSD#1_0 indicating the three-dimensional position of the object OBJ at time t#1_0 is generated. Furthermore, the three-dimensional position data generation unit 311 may perform a process of generating three-dimensional position data WSD#1_0 from image data IMG_3D#1_0 so that the generation of three-dimensional position data WSD#1_0 is completed at the time the 2D matching unit 3121 completes the 2D matching process using image data IMG_2D#1_0.

[0262] Thereafter, at time t#2_0, the 3D matching unit 3122 starts 3D matching processing based on the three-dimensional position data WSD#1_0 and the position and orientation information POI1#1_0. As a result, at time t#3_0, when the 3D matching cycle has elapsed since time t#2_0, the 3D matching unit 3122 completes the generation of position and orientation information POI2#1_0 indicating the position and orientation of the object OBJ at time t#1_0 (in the example shown in FIG. 19, the calculation of position Tx(3D)#1_0 in the X-axis direction (3D)). Because the position and orientation information POI2 has been newly generated in this way (in the example shown in FIG. 19, the position Tx(3D) has been newly calculated), the coordinate conversion unit 3125 resets the range of the change amount ΔTx(3D) to be added to the position Tx(3D) to calculate the position Tx'(3D). Specifically, because the position Tx(3D)#1_0 at time t#1_0 is newly calculated, the coordinate conversion unit 3125 sets the range of the change amount ΔTx(3D) calculated from time t#1_0 to the current time as a new range of the change amount ΔTx(3D) to be added to the position Tx(3D)#1_0. Therefore, from time t#3_0 onwards, the position Tx(3D)#1_0+change amount ΔTx(3D)#1_0+change amount ΔTx(3D)#1_1+...+change amount ΔTx(3D)#1_N+change amount ΔTx(3D)#2_0+change amount ΔTx(3D)#2_1+...+change amount ΔTx(3D)#2_N is used as the position Tx'(3D).

[0263] Thereafter, the same operation is repeated.

[0264] Although not shown in the figure to avoid redundant description, the coordinate conversion unit 3125 calculates the position Ty'(3D), the position Tz'(3D), the orientation Rx'(3D), the orientation Ry'(3D), and the orientation Rz'(3D) in the same way as when calculating the position Tx'(3D). That is, the coordinate conversion unit 3125 may calculate the position Ty'(3D) by adding, to the position Ty(3D) at a certain time, the sum of the amount of change ΔTy(3D) calculated from the time when the matching process for calculating the position Ty(3D) at a certain time was started to the current time. The coordinate conversion unit 3125 may calculate the position Ty'(3D) by adding, to the position Ty(3D) at a certain time, the sum of the amount of change ΔTy(3D) calculated from the time when the matching process for calculating the position Ty(3D) at a certain time was started to the current time. Furthermore, the coordinate conversion unit 3125 may calculate the position Tz'(3D) by adding, to the position Tz(3D) at a certain time, the sum of the change amounts ΔTz(3D) calculated from the time the matching process for calculating the position Tz(3D) at a certain time was started to the current time. The coordinate conversion unit 3125 may calculate the position Tz'(3D) by adding, to the position Tz(3D) at a certain time, the sum of the change amounts ΔTz(3D) calculated from the time the matching process for calculating the position Rx(3D) at a certain time was started to the current time. Furthermore, the coordinate conversion unit 3125 may calculate the position Rx'(3D) by adding, to the position Rx(3D) at a certain time, the sum of the change amounts ΔRx(3D) calculated from the time the matching process for calculating the position Rx(3D) at a certain time was started to the current time. The coordinate conversion unit 3125 may calculate the position Rx'(3D) by adding the sum of the change amounts ΔRx(3D) calculated from one time to the current time to the position Rx(3D) at one time. The coordinate conversion unit 3125 may also calculate the position Ry'(3D) by adding the sum of the change amounts ΔRy(3D) calculated from the time the matching process for calculating the position Ry(3D) at one time was started to the current time to the position Ry(3D) at one time. The coordinate conversion unit 3125 may also calculate the position Ry'(3D) by adding the sum of the change amounts ΔRy(3D) calculated from one time to the current time to the position Ry(3D) at one time.Furthermore, the coordinate conversion unit 3125 may calculate the position Rz'(3D) by adding, to the position Rz(3D) at a certain time, the sum of the amount of change ΔRz(3D) calculated from the time when the matching process for calculating the position Rz(3D) at a certain time was started to the current time. The coordinate conversion unit 3125 may calculate the position Rz'(3D) by adding, to the position Rz(3D) at a certain time, the sum of the amount of change ΔRz(3D) calculated from the time when the matching process for calculating the position Rz(3D) at a certain time was started to the current time.

[0265] Thereafter, the coordinate conversion unit 3125 calculates the position Tx(GL), position Ty(GL), position Tz(GL), orientation Rx(GL), orientation Ry(GL), and orientation Rz(GL) of the object OBJ in the global coordinate system based on the position Tx'(3D), position Ty'(3D), position Tz'(3D), orientation Rx'(3D), orientation Ry'(3D), and orientation Rz'(3D). In other words, the coordinate conversion unit 3125 calculates the position of the 6 DOF of the object OBJ in the global coordinate system based on the position of the 6 DOF of the object OBJ in the 3D imaging coordinate system (or, in some cases, the position of the 6 DOF of the object OBJ in the 2D imaging coordinate system). The coordinate conversion unit 3125 generates position and orientation information POI0 indicating the position Tx(GL), the position Ty(GL), the position Tz(GL), the orientation Rx(GL), the orientation Ry(GL), and the orientation Rz(GL).

[0266] For example, the coordinate conversion unit 3125 may convert the 6 DOF position of the object OBJ in the 3D imaging coordinate system (or the 6 DOF position of the object OBJ in the 2D imaging coordinate system) into the 6 DOF position of the object OBJ in the global coordinate system using a conversion matrix M(3DtoGL) for converting a position in the 3D imaging coordinate system (or the 3D imaging coordinate system) into a position in the global coordinate system.

[0267] However, if position and orientation information POI2' indicating position Tx'(GL), position Ty'(GL), position Tz'(GL), orientation Rx'(GL), orientation Ry'(GL), and orientation Rz'(GL) is generated as a result of correcting position and orientation information POI2 based on change amount information VI, position and orientation information POI2' may be used as position and orientation information POI0. In this case, the coordinate conversion unit 3125 does not necessarily have to perform coordinate conversion based on the above-mentioned conversion matrix M(3DtoGL).

[0268] (3) Technical Effects of the Robot System SYS As described above, the control device 3 calculates at least one of the position and orientation of the object OBJ in the global coordinate system by correcting the result of the matching process based on the result of the tracking process. Therefore, as described in detail below, the control device 3 can calculate at least one of the position and orientation of the object OBJ in the global coordinate system in a shorter cycle (i.e., more frequently) than when at least one of the position and orientation of the object OBJ in the global coordinate system is calculated based on the result of the matching process without using the result of the tracking process. In other words, the position and orientation calculation unit 312 can calculate at least one of the position and orientation of the object OBJ in the global coordinate system more quickly. As a result, the control device 3 can generate (i.e., update) robot control signals more frequently and output them to the robot 1. It can also be said that the control device 3 can generate (update) robot control signals more quickly and output them to the robot 1.

[0269] For example, the control device 3 may generate a robot control signal each time at least one of the position and orientation of the object OBJ is calculated. In this case, the control device 3 can generate a robot control signal at the same period as the period (e.g., the tracking period described above) in which at least one of the position and orientation of the object OBJ is calculated. As a result, the control device 3 can output a robot control signal to the robot 1 at the same period as the period (e.g., the tracking period described above) in which at least one of the position and orientation of the object OBJ is calculated. If at least one of the position and orientation of the object OBJ in the global coordinate system is calculated based on the result of the matching process without using the result of the tracking process, the control device 3 would generate and output a robot control signal at the same period (typically a period shorter than the tracking period) as the 2D matching period or 3D matching period in which at least one of the position and orientation of the object OBJ is calculated. Therefore, in this embodiment, compared to this case, the control device 3 can generate (i.e., update) and output a robot control signal to the robot 1 more frequently (at a higher speed). The control device 3 may generate and output the robot control signal at a cycle different from the cycle at which the position of the object OBJ is calculated (for example, a cycle different from the tracking cycle described above).

[0270] This effect is particularly beneficial when at least one of the robot 1 and the object OBJ is moving (particularly when each of the imaging devices 21 and 22 and the object OBJ are displaced relative to each other). This is because the less frequently the position of the object OBJ is calculated when at least one of the robot 1 and the object OBJ is moving, the longer the period during which the deviation between the position of the object OBJ calculated by the control device 3 and the actual position of the object OBJ is large. Similarly, the less frequently the orientation of the object OBJ is calculated when at least one of the robot 1 and the object OBJ is moving, the longer the period during which the deviation between the orientation of the object OBJ calculated by the control device 3 and the actual orientation of the object OBJ is large. In this embodiment, the control device 3 calculates the position of the object OBJ in the global coordinate system more frequently (at a higher speed), so the deviation between the position of the object OBJ calculated by the control device 3 and the actual position of the object OBJ is relatively small. Similarly, because the control device 3 calculates the orientation of the object OBJ in the global coordinate system more frequently (at a higher speed), the degree of discrepancy between the orientation of the object OBJ calculated by the control device 3 and the actual orientation of the object OBJ becomes relatively small. Therefore, by calculating at least one of the position and orientation of the object OBJ at a high frequency, the control device 3 can essentially calculate at least one of the position and orientation of the object OBJ with high accuracy. As a result, the control device 3 can appropriately generate (i.e., update) and output to the robot 1 a robot control signal for controlling the robot 1 so that the desired processing is performed on the object OBJ whose position and / or orientation is calculated with high accuracy and frequency. As a result, the possibility that the robot 1 will move differently from the intended movement is reduced.

[0271] For example, when the robot 1 holds a moving workpiece W as shown in FIGS. 5( a) to 5(d), the control device 3 can control the movement of the robot 1 following the moving workpiece W (i.e., generate and output a robot control signal) in accordance with the calculation cycle of at least one of the position and posture of the object OBJ. Therefore, even if the movement of the robot 1 deviates from the desired movement for following the moving workpiece W, the movement of the robot 1 is corrected frequently (high speed) in accordance with the calculation cycle of at least one of the position and posture of the workpiece W. As an example, even if the movement of the moving workpiece W is irregular, the movement of the robot 1 is corrected frequently (high speed) to match the irregular movement of the workpiece W. As another example, even if the workpiece W moves at high speed, the movement of the robot 1 is corrected frequently (high speed) to match the high-speed movement of the workpiece W. Therefore, the robot 1 can follow the moving workpiece W while reducing unnecessary movement. As a result, the robot 1 can hold the moving workpiece W while reducing unnecessary movement. 5( a) to 5(d), the robot 1 can move relative to the object OBJ while reducing unnecessary movements. Furthermore, because at least one of the position and orientation of the object OBJ is calculated with high frequency, the possibility that the object OBJ will move out of the imaging range (field of view) of the imaging unit 2 is reduced. Therefore, the robot 1 can move accurately and quickly relative to the object OBJ (for example, track the object OBJ accurately and quickly) without losing sight of the object OBJ, and perform desired processing on the object OBJ with the end effector 13.

[0272] Considering that the effects that can be obtained by the robot system SYS are particularly beneficial when at least one of the robot 1 and the object OBJ is moving, the imaging devices 21 and 22 may capture images of the object OBJ while at least one of the robot 1 and the object OBJ is moving. That is, the imaging devices 21 and 22 may capture images of the object OBJ at a timing (in other words, a time) when the imaging devices 21 and 22 and the object OBJ are moving relative to each other. For example, the imaging device 21 may capture images of the object OBJ at a timing when the imaging device 21 and the object OBJ are moving relative to each other, thereby generating image data IMG_2D used in the above-described 2D matching process. For example, the imaging device 22 may capture images of the object OBJ at a timing when the imaging device 22 and the object OBJ are moving relative to each other, thereby generating image data IMG_3D used in the above-described 3D matching process. For example, the imaging device 21 may generate image data IMG_2D used in the tracking process described above by capturing images of the object OBJ at a first time t1 and a second time t2, when the imaging device 21 and the object OBJ are moving relative to each other. For example, the imaging device 21 may generate image data IMG_3D used in the tracking process described above by capturing images of the object OBJ at a third time t3, when the imaging device 21 and the object OBJ are moving relative to each other.

[0273] Furthermore, in this embodiment, the control device 3 can determine at least one of the initial position and initial orientation of the three-dimensional model WM3 used for the 3D matching process based on the results of the 2D matching process (i.e., the position and orientation information POI1). As a result, the control device 3 can determine the initial position and initial orientation of the three-dimensional model WM3 such that the degree of deviation from the actual position and orientation of the object OBJ is within a tolerable amount. The reasons for this are as follows: First, the position and orientation information POI1 obtained by the 2D matching process is less likely to lack information regarding at least one of the position and orientation of the object OBJ compared to the position and orientation information POI2 obtained by the 3D matching process. This is because the 2D matching process detects the object OBJ using the image represented by the image data IMG_2D itself, and therefore there is a low possibility that information regarding at least one of the position and orientation of the object OBJ is lacking in the first place. Therefore, the 2D matching process is more likely to successfully detect the object OBJ. In other words, the 2D matching process is more likely to detect an object OBJ that matches the two-dimensional model WM2. In other words, the 2D matching process is less likely to detect an object different from the object OBJ corresponding to the two-dimensional model WM2. On the other hand, the 3D matching process detects the object OBJ using three-dimensional position data WSD that may be missing some information about the object OBJ (e.g., point cloud data that may be missing a point cloud corresponding to a portion of the object OBJ). Therefore, a lack of data in the three-dimensional position data WSD (e.g., missing points) significantly affects the success of the detection of the object OBJ by the 3D matching process. Therefore, the likelihood of successfully detecting the object OBJ by the 3D matching process is lower than the likelihood of successfully detecting the object OBJ by the 2D matching process. In other words, the likelihood of detecting an object OBJ that matches the three-dimensional model WM3 by the 3D matching process is lower than the likelihood of detecting an object OBJ that matches the two-dimensional model WM2 by the 2D matching process.In other words, the possibility that an object different from the object OBJ corresponding to the three-dimensional model WM3 will be detected by the 3D matching process is higher than the possibility that an object different from the object OBJ corresponding to the two-dimensional model WM2 will be detected by the 2D matching process.

[0274] In this embodiment, taking this into consideration, at least one of the initial position and initial orientation of the three-dimensional model WM3 can be determined based on the results of the 2D matching process, which reduces the likelihood of detecting an object different from the object OBJ corresponding to the two-dimensional model WM2. As a result, the control device 3 can reliably determine the initial position and initial orientation of the three-dimensional model WM3 such that the degree of deviation from the actual position and orientation of the object OBJ that should be detected by the 2D matching process (and, naturally, should also be detected by the 3D matching process) is within an allowable amount. As a result, compared to a case in which at least one of the initial position and initial orientation of the three-dimensional model WM3 is not determined, the likelihood that an object different from the object OBJ corresponding to the three-dimensional model WM3 will be detected by the 3D matching process or that the object OBJ corresponding to the three-dimensional model WM3 will not be detected (a detection error will occur) is reduced. This is because the greater the deviation between the initial position and initial orientation of the three-dimensional model WM3 and the actual position and orientation of the object OBJ, the higher the possibility that an object different from the object OBJ corresponding to the three-dimensional model WM3 will be detected by the 3D matching process, or that the object OBJ corresponding to the three-dimensional model WM3 cannot be detected. Therefore, the control device 3 can generate position and orientation information POI2 that indicates at least one of the position and orientation of the object OBJ corresponding to the three-dimensional model WM3 with higher accuracy than when at least one of the initial position and initial orientation of the three-dimensional model WM3 is not determined.

[0275] Furthermore, in this embodiment, the control device 3 can determine at least one of the initial position and initial orientation of the three-dimensional model WM3 used for the 3D matching process based on the result of the 2D matching process (i.e., the position and orientation information POI1). Here, the position and orientation information POI1 obtained by the 2D matching process indicates the position of the object OBJ in the X-axis direction (2D) and the Y-axis direction (2D) with high accuracy due to the characteristics of the 2D matching process using two-dimensional images. Furthermore, the position and orientation information POI1 indicates the orientation of the object OBJ around the Z-axis (2D) with high accuracy due to the characteristics of the 2D matching process using two-dimensional images. Meanwhile, the position and orientation information POI1 also indicates the position of the object OBJ in the Y-axis direction (2D) and the orientation of the object OBJ around the X-axis (2D) and the Y-axis (2D) with a certain degree of accuracy, although not as accurately as the position of the object OBJ in the X-axis direction (2D) and the Y-axis direction (2D) and the orientation of the object OBJ around the Z-axis (2D). Therefore, in this embodiment, the control device 3 combines 2D matching processing using the image data IMG_2D and 3D matching processing using the three-dimensional position data WSD to calculate at least one of the position and orientation of the object OBJ with higher accuracy. Specifically, the control device 3 determines at least one of the initial position and initial orientation of the three-dimensional model WM3 based on the position and orientation information POI1. Therefore, in the 3D imaging coordinate system in which the 3D matching processing is performed, the initial position of the three-dimensional model WM3 is unlikely to differ significantly from the actual position of the object OBJ. In particular, the initial position of the three-dimensional model WM3 is likely to be close to the actual position of the object OBJ in both the X-axis direction (3D) and the Y-axis direction (3D). Furthermore, in the Z-axis direction (3D), the initial position of the three-dimensional model WM3 is unlikely to differ significantly from the actual position of the object OBJ. Similarly, in the 3D imaging coordinate system, the initial orientation of the three-dimensional model WM3 is unlikely to differ significantly from the actual orientation of the object OBJ. In particular, around the Z axis (3D), the initial orientation of the three-dimensional model WM3 is likely to be close to the actual orientation of the object OBJ.Furthermore, the initial orientation of the three-dimensional model WM3 around the X-axis (3D) and the Y-axis (3D) is unlikely to differ significantly from the actual orientation of the object OBJ. In this state, the 3D matching unit 3122 performs a matching process. As a result, the 3D matching unit 3122 can accurately detect the object OBJ through the 3D matching process compared to when the initial position and orientation of the three-dimensional model WM3 are not determined based on the position and orientation information POI1. This is because the three-dimensional model WM3 is already positioned near the object OBJ indicated by the three-dimensional position data WSD. Furthermore, the 3D matching unit 3122 can accurately approximate, in the Z-axis direction (3D), the characteristic features of the three-dimensional model WM3 to the characteristic features of the object OBJ indicated by the three-dimensional position data WSD, using the three-dimensional position data WSD that indicates the position of the object OBJ in the Z-axis direction (3D) with high accuracy. Therefore, compared to when the three-dimensional position data WSD is not used, the 3D matching unit 3122 can generate position and orientation information POI2 that indicates at least one of the position and orientation of the object OBJ with higher accuracy. In particular, the 3D matching unit 3122 can generate position and orientation information POI2 that indicates the position Tz(3D), orientation Rx(3D), and orientation Ry(3D) with higher accuracy.

[0276] Here, as described above, if at least one of the robot 1 and the object OBJ is moving (i.e., the imaging devices 21 and 22 are displaced relative to the object OBJ), the 2D imaging time at which the imaging device 21 captures the object OBJ for the matching process and the 3D imaging time at which the imaging device 22 captures the object OBJ for the matching process may be set to the same time in order to improve the accuracy of the position and orientation information POI2. Specifically, if the 2D imaging time and the 3D imaging time are different times, the 3D matching unit 3122 determines at least one of the initial position and initial orientation of the three-dimensional model WM3 for calculating the position of the object OBJ at a 3D imaging time different from the 2D imaging time, based on the position and orientation information POI1 indicating at least one of the position and orientation of the object OBJ at the 2D imaging time. However, if at least one of the robot 1 and the object OBJ is moving, the position of the object OBJ at the 2D imaging time is likely to differ from the position of the object OBJ at the 3D imaging time. For this reason, when the 2D imaging time and the 3D imaging time are different times, the initial position of the three-dimensional model WM3 determined based on the position of the object OBJ at the 2D imaging time is likely to be significantly different from the actual position of the object OBJ at the 3D imaging time. Similarly, the initial orientation of the three-dimensional model WM3 determined based on the orientation of the object OBJ at the 2D imaging time is likely to be significantly different from the actual orientation of the object OBJ at the 3D imaging time. As a result, the accuracy of the position and orientation information POI2 generated using the three-dimensional model WM3 may be reduced compared to when the initial position and initial orientation of the three-dimensional model WM3 are the same as or close to the actual position and orientation of the object OBJ. However, when the 2D imaging time and the 3D imaging time are the same time, the position and orientation of the object OBJ at the 2D imaging time will be the same as the position and orientation of the object OBJ at the 3D imaging time. For this reason, it is unlikely that the initial position and initial orientation of the three-dimensional model WM3 will be significantly different from the actual position and orientation of the object OBJ. Therefore, the control device 3 can prevent a decrease in the accuracy of the position and orientation information POI2 by controlling the image capturing devices 21 and 22 so that the 2D image capturing time and the 3D image capturing time are the same time.

[0277] As already described above, the state in which "the 2D imaging time and the 3D imaging time are the same" may include a state in which "although the 2D imaging time and the 3D imaging time are not exactly the same, the 2D imaging time and the 3D imaging time can be considered to be substantially the same because the time difference between the 2D imaging time and the 3D imaging time is smaller than the allowable upper limit." In this case, the allowable upper limit of the time difference between the 2D imaging time and the 3D imaging time may be set to an appropriate value that can achieve a state in which the accuracy of the position and orientation information POI2 is desired. In other words, the allowable upper limit of the time difference between the 2D imaging time and the 3D imaging time may be set to an appropriate value that can achieve a state in which the amount of reduction in accuracy of the position and orientation information POI2 due to the time difference between the 2D imaging time and the 3D imaging time falls within an allowable range. For example, a time difference between the 2D imaging time and the 3D imaging time may cause an error in the calculation results of at least one of the position and orientation of the object OBJ (described later) (i.e., a case in which the accuracy of the position and orientation information POI2 is reduced). In this case, if the amount of decrease in accuracy of the position and orientation information POI2 exceeds the allowable range, a control error may occur in the robot arm 12. The control error of the robot arm 12 may result in a movement error of the end effector 13, which may prevent the end effector 13 from properly performing a predetermined process on the object OBJ. For this reason, the allowable upper limit of the time lag between the 2D image capture time and the 3D image capture time may be set to an appropriate value based on the accuracy of the position and orientation information POI2.

[0278] However, if there is no relative displacement between the imaging devices 21 and 22 and the object OBJ (e.g., the robot 1 and the object OBJ are not moving), the position of the object OBJ at the 2D imaging time will be the same as the position of the object OBJ at the 3D imaging time, even if the 2D imaging time and the 3D imaging time are different times. Therefore, the initial position and initial orientation of the three-dimensional model WM3 are unlikely to differ significantly from the actual position and orientation of the object OBJ. Therefore, the control device 3 does not need to control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are the same. For example, the control device 3 may control the imaging devices 21 and 22 so that the 2D imaging time and the 3D imaging time are different times. Even in this case, the accuracy of the position and orientation information POI2 is hardly reduced due to a time difference between the 2D imaging time and the 3D imaging time.

[0279] Furthermore, in this embodiment, the control device 3 can perform tracking processing using three-dimensional position data WSD in addition to the two image data IMG_2D#t1 and IMG_2D#t2. In particular, the control device 3 can perform tracking processing based on position information PI3, which indicates the position of a characteristic location of the object OBJ in the Z-axis direction (2D) or Z-axis direction (3D), calculated from the three-dimensional position data WSD. Here, the accuracy of the position of the object OBJ in the Z-axis direction (2D), calculated from the image data IMG_2D representing a two-dimensional image, is not necessarily high. On the other hand, the three-dimensional position data WSD generated from the image data IMG_3D indicates the position of the object OBJ in the Z-axis direction (3D) with high accuracy. Therefore, by performing tracking processing using the position information PI3 as described in this embodiment, the control device 3 can calculate the amount of change in at least one of the position and orientation of the object OBJ with higher accuracy. In particular, the tracking unit 3123 ...

Claims

1. A control device that generates a control signal for controlling a robot arm provided with a processing device that performs processing on an object, a first imaging device that outputs first image data, and a second imaging device that outputs second image data, the control signal controls the robot arm to move the processing device and the first and second imaging devices; The control device includes: A calculation unit that generates the control signal; an output unit that outputs the control signal generated by the calculation unit; Equipped with The calculation unit is generating first information indicating a position and an orientation of the object using the first image data generated by the first imaging device capturing an image of the object; generating second information indicating a position and orientation of the object using three-dimensional position data, which is generated from the second image data generated by the second imaging device capturing an image of the object and indicates a three-dimensional position of each of a plurality of points of the object, and three-dimensional model data of the object at a position and orientation determined based on the first information; generating the control signal based on the second information Control device.

2. the first information indicates a position and orientation of the object in a first coordinate system; the determined position and orientation of the three-dimensional model indicated by the three-dimensional model data is a position and orientation in a second coordinate system different from the first coordinate system, The second information indicates a position and an orientation of the object in the second coordinate system. The control device according to claim 1 .

3. The calculation unit is Transforming the first information in the first coordinate system into a position and orientation in the second coordinate system; determining the position and orientation of the three-dimensional model based on the transformed position and orientation in the second coordinate system; The control device according to claim 2.

4. The calculation unit sets the position and orientation in the transformed second coordinate system as the position and orientation of the three-dimensional model. The control device according to claim 3.

5. The calculation unit is setting a position and an orientation of the three-dimensional model indicated by the three-dimensional model data in the first coordinate system based on the first information; The position and orientation of the three-dimensional model in the set first coordinate system is transformed into a position and orientation in the second coordinate system, thereby determining the position and orientation of the three-dimensional model in the second coordinate system. The control device according to claim 2.

6. The calculation unit sets the position and orientation of the object in a first coordinate system indicated by the first information as the position and orientation of the three-dimensional model in the first coordinate system. The control device according to claim 5.

7. A time when the first imaging device captures an image of the object to generate the first image data is the same as a time when the second imaging device captures an image of the object to generate the second image data. A control device according to any one of claims 1 to 6.

8. The output unit outputs the control signal generated by the calculation unit based on the second information in a first period. A control device according to any one of claims 1 to 6.

9. the calculation unit calculates an amount of change in position and orientation of the object between the first and second times based on two sets of first image data that are respectively generated by the first imaging device capturing images of the object at first and second times that are different from each other, and three-dimensional position data that is generated from the second image data that is generated by the second imaging device capturing images of the object at a third time and indicates three-dimensional positions of each of a plurality of points of the object; The control signal is generated based on the second information and the amount of change. A control device according to any one of claims 1 to 6.

10. the first time and the second time are different from times at which the first imaging device captures an image of the object for generating the first image data used in generating the first information, The third time is different from the time when the second imaging device captures an image of the object for generating the second image data used for generating the three-dimensional position data used for generating the second information. The control device according to claim 9.

11. the second time is a time later than the first time, The second time and the third time are the same. The control device according to claim 9.

12. The calculation unit is using the three-dimensional position data generated from the second image data generated by imaging at the third time by the second imaging device, generate third information indicating a position of a characteristic location of the object in a direction parallel to the first axis in a first coordinate system defined by a first axis along an optical axis of the optical system of the first imaging device, a second axis orthogonal to the first axis, and a third axis orthogonal to the first and second axes, or a position of a characteristic location of the object in a direction parallel to the fourth axis in a second coordinate system defined by a fourth axis along the optical axis of the optical system of the second imaging device, a fifth axis orthogonal to the fourth axis, and a sixth axis orthogonal to the fourth and fifth axes; The amount of change is calculated using the two pieces of first image data and the third information. The control device according to claim 9.

13. The calculation unit is generating fourth information indicating a position of the characteristic location at the first time using the first image data generated by the first imaging device capturing an image of the object at the first time; generating fifth information indicating a position of the characteristic location at the second time using the first image data generated by the first imaging device capturing an image of the object at the second time and the third information; The amount of change is calculated using the fourth information and the fifth information. The control device according to claim 12.

14. The first time, the second time, and the third time are times when the first and second imaging devices and the object are moving relatively to each other. The control device according to claim 9.

15. The output unit outputs the control signal generated by the calculation unit based on the second information and the amount of change in a second period. The control device according to claim 9.

16. A time period from when the object is imaged by the first and second imaging devices to when the second information is generated based on the first and second image data is longer than a time period from when the object is imaged by the first imaging device at an earlier time of the first and second times that are different from each other to when the amount of change is calculated based on the two first image data. The control device according to claim 9.

17. The calculation unit is generating the second information in a third period; The amount of change is calculated in a fourth period that is shorter than the third period. The control device according to claim 9.

18. The calculation unit generates the control signal based on the newly calculated amount of change and the newly generated second information every time the amount of change is newly calculated in the fourth period during the period from when the second information is newly generated as one period of the third period to when the second information is next generated.

20. The control device of claim 17.

19. the output unit outputs the control signal generated by the calculation unit based on the second information and the amount of change in a second period; The second period is the same as the fourth period.

20. The control device of claim 17.

20. The calculation unit determines, based on the first image data, the object as a target object to be processed by the processing device. A control device according to any one of claims 1 to 6.

21. The calculation unit determines whether or not to determine the object as a target object to be processed by the processing device based on the first image data. A control device according to any one of claims 1 to 6.

22. The calculation unit is generating the first information by an object detection process that detects the object in a first image represented by the first image data; Based on the result of the object detection process, it is determined whether or not to determine the object as the target object.

22. The control device of claim 21.

23. The calculation unit is As the object detection process, a matching process is performed using two-dimensional model data indicating a two-dimensional model of the object and the first image data; Detecting an edge of the object from the object detected by the matching process; Detecting edges in the first image using the first image data; determining whether or not to determine the object as the target object based on a first similarity between the edge of the object and the edge in the first image; 23. The control device of claim 22.

24. The calculation unit is As the object detection process, a matching process is performed using two-dimensional model data indicating a two-dimensional model of the object and the first image data; A determination is made as to whether or not the object is to be determined as the target object based on a second similarity between the two-dimensional model of the object and the first image represented by the first image data, the second similarity being calculated in the matching process.

23. The control device of claim 22.

25. The calculation unit is As the object detection process, a matching process is performed using two-dimensional model data indicating a two-dimensional model of the object and the first image data; determining whether or not to determine the object as the target object based on a second similarity between the two-dimensional model of the object and the first image represented by the first image data, the second similarity being calculated in the matching process; determining that the object is the target object when the first and second similarities exceed a predetermined first threshold; When at least one of the first and second similarities is below a predetermined first threshold, it is determined that the object is not to be determined as the target object.

24. The control device of claim 23.

26. The calculation unit generates the first information using the first image data and a calculation model generated by machine learning or deep learning. A control device according to any one of claims 1 to 6.

27. ​​The computational model is generated by machine learning or deep learning so as to output information regarding the position and orientation of the object when the first image data is input.

27. The control device of claim 26.

28. A control device that generates a control signal for controlling a robot arm provided with a processing device that performs processing on an object, a first imaging device that outputs first image data, and a second imaging device that outputs second image data, the control signal controls the robot arm to move the processing device and the first and second imaging devices; The control device includes: A calculation unit that generates the control signal; an output unit that outputs the control signal generated by the calculation unit; Equipped with The calculation unit is calculating an amount of change in position and orientation of the object between the first and second times based on two sets of first image data that are respectively generated by the first imaging device capturing images of the object at first and second times that are different from each other, and three-dimensional position data that is generated from the second image data that is generated by the second imaging device capturing images of the object at a third time and indicates three-dimensional positions of each of a plurality of points of the object; The control signal is generated based on the amount of change. Control device.

29. the second time is a time later than the first time, The second time and the third time are the same.

29. The control device of claim 28.

30. The calculation unit is using the three-dimensional position data, generate information indicating a position of a characteristic location of the object in a direction parallel to the first axis in a first coordinate system defined by a first axis along an optical axis of the optical system of the first image capture device, a second axis orthogonal to the first axis, and a third axis orthogonal to the first and second axes, or generate information indicating a position of a characteristic location of the object in a direction parallel to the fourth axis in a second coordinate system defined by a fourth axis along the optical axis of the optical system of the second image capture device, a fifth axis orthogonal to the fourth axis, and a sixth axis orthogonal to the fourth and fifth axes; The amount of change is calculated using the two pieces of first image data and the information.

30. A control device according to claim 28 or 29.

31. When the information is defined as first information, The calculation unit is generating second information indicating a position of the characteristic location at the first time using the first image data generated by the first imaging device capturing an image of the object at the first time; generating third information indicating a position of the characteristic location at the second time using the first image data generated by the first imaging device capturing an image of the object at the second time and the first information; The amount of change is calculated using the second information and the third information. The control device of claim 30.

32. The first time, the second time, and the third time are times when the first and second imaging devices and the object are moving relatively to each other.

30. A control device according to claim 28 or 29.

33. The output unit outputs the control signal generated by the calculation unit based on the amount of change at a predetermined period.

30. A control device according to claim 28 or 29.

34. The calculation unit generates the first information by a matching process using the first image data and two-dimensional model data indicating a two-dimensional model of the object. A control device according to any one of claims 1 to 6.

35. The calculation unit determines the position and orientation of the three-dimensional model of the object represented by the three-dimensional model data based on the first information. A control device according to any one of claims 1 to 6.

36. The calculation unit generates the second information by a matching process using the three-dimensional position data and the three-dimensional model data. A control device according to any one of claims 1 to 6.

37. The control device controls timing of imaging by the first and second imaging devices so that a time when the first imaging device images the object and a time when the second imaging device images the object are the same. A control device according to any one of claims 1 to 6.

38. A timing when the first imaging device captures an image of the object for generating the first image data used for generating the first information and a timing when the second imaging device captures an image of the object for generating the second image data used for generating the three-dimensional position data used for generating the second information are timings during which the first and second imaging devices and the object are moving relative to each other. A control device according to any one of claims 1 to 6.

39. the second time is a time later than the first time, The control device controls timing of imaging by the first and second imaging devices so that the second time and the third time are the same. The control device according to claim 9.

40. the three-dimensional position data generated from the second image data generated by capturing the image at the third time indicates the three-dimensional positions of each of the multiple points of the object in the second coordinate system; The calculation unit is Transforming the three-dimensional position in the second coordinate system to a three-dimensional position in the first coordinate system; generating third information indicating a position of the characteristic location in a direction parallel to the first axis from the three-dimensional position in the transformed first coordinate system; The control device according to claim 12.

41. the three-dimensional position data generated from the second image data generated by capturing the image at the third time indicates the three-dimensional positions of each of the multiple points of the object in the second coordinate system; The calculation unit generates the third information indicating a position of the characteristic location in a direction parallel to the fourth axis from the three-dimensional position in the second coordinate system. The control device according to claim 12.

42. the three-dimensional position data indicates the three-dimensional position of each of a plurality of points of the object in the second coordinate system; The calculation unit is Transforming the three-dimensional position in the second coordinate system to a three-dimensional position in the first coordinate system; generating information indicating a position of the characteristic location in a direction parallel to the first axis from the three-dimensional position in the transformed first coordinate system; The control device of claim 30.

43. the three-dimensional position data indicates the three-dimensional position of each of a plurality of points of the object in the second coordinate system; The calculation unit generates the information indicating a position of the characteristic location in a direction parallel to the fourth axis from the three-dimensional position in the second coordinate system. The control device of claim 30.

44. The calculation unit is determining that the object is the target object when the first similarity exceeds a predetermined first threshold; If the first similarity is below a predetermined first threshold, it is determined that the object is not to be determined as the target object.

24. The control device of claim 23.

45. The calculation unit is determining that the object is the target object when the second similarity exceeds a predetermined second threshold; If the second similarity is below a predetermined second threshold, it is determined that the object is not to be determined as the target object.

25. The control device of claim 24.

46. The calculation unit is When it is determined that the object is not to be determined as the target object, a signal for controlling the robot arm to move the first imaging device relative to the object is generated as the control signal. The control device according to claim 20.

47. The calculation unit is generating, as the control signal, a signal for controlling the robot arm to rotate and move the first imaging device around a desired rotation axis; After the robot arm is controlled based on the signal, the determination is made based on the first image data generated by the first imaging device capturing an image of the object.

47. The control device of claim 46.

48. The calculation unit is When it is determined that the object is not to be selected as the target object as a result of the determination made after the robot arm has been controlled to rotate and move the first imaging device based on the signal, a signal for controlling the robot arm to translate the first imaging device along a desired translation axis is generated as the control signal.

48. The control device of claim 47.

49. The processing device includes a holding device capable of holding the object, The control signal is a signal for controlling a holding operation of the holding device, based on the control signal generated by the calculation unit, the robot arm is controlled so that the holding device approaches the object, and the holding device is controlled so that the object is held by the holding device; The calculation unit is generating sixth information indicating a position and an orientation of the object held by the holding device using the first image data generated by the first imaging device capturing an image of the object after the object is held by the holding device; Based on the sixth information, the control signal is generated for controlling the robot arm to move the object held by the holding device to a desired position and / or change the posture of the object held by the holding device to a desired posture. A control device according to any one of claims 1 to 6.

50. a processing device that performs processing on at least one of a first object and a second object that is different from the first object in at least one of a position and an attitude; a control device that generates a control signal for controlling a robot arm provided with a first imaging device that outputs first image data and a second imaging device that outputs second image data, the control signal controls the robot arm to move the processing device and the first and second imaging devices; The control device includes: A calculation unit that generates the control signal; an output unit that outputs the control signal generated by the calculation unit; Equipped with The calculation unit is determining whether or not to select the first or second object as a target object for performing the processing by the processing device, based on the first image data generated by the first imaging device capturing images of the first and second objects; When the first or second object is selected as the target object as a result of the determination, second information indicating a position and orientation of the target object is generated using three-dimensional position data that is generated from the second image data generated by the second imaging device capturing images of the first and second objects and that indicates three-dimensional positions of each of a plurality of points of the target object, and three-dimensional model data of the target object at a position and orientation that is determined based on first information indicating the position and orientation of the target object generated using the first image data; generating the control signal based on the second information Control device.

51. The calculation unit is generating the first information by an object detection process that detects the first and second objects in a first image represented by the first image data; Based on a result of the object detection process, it is determined whether or not to select the first or second object as the target object.

51. The control device of claim 50.

52. The calculation unit is performing a matching process as the object detection process, using first two-dimensional model data indicating a two-dimensional model of the first object and second two-dimensional model data indicating a two-dimensional model of the second object, and the first image data; Detecting an edge of the first object and an edge of the second object from the first and second objects detected in the matching process; Detecting edges in the first image using the first image data; determining whether to select the first or second object as the target object based on a first similarity between the edge of the first object and the edge in the first image and a second similarity between the edge of the second object and the edge in the first image; 52. The control device of claim 51.

53. The calculation unit is performing a matching process as the object detection process, using first two-dimensional model data indicating a two-dimensional model of the first object and second two-dimensional model data indicating a two-dimensional model of the second object, and the first image data; Whether or not to select the first or second object as the target object is determined based on a third similarity between the two-dimensional model of the first object and the first image represented by the first image data and a fourth similarity between the two-dimensional model of the second object and the first image represented by the first image data, both calculated in the matching process.

53. A control device as claimed in claim 51 or 52.

54. The calculation unit is performing a matching process as the object detection process, using first two-dimensional model data indicating a two-dimensional model of the first object and second two-dimensional model data indicating a two-dimensional model of the second object, and the first image data; determining whether to select the first or second object as the target object based on a third similarity between the two-dimensional model of the first object and the first image represented by the first image data, a fourth similarity between the two-dimensional model of the second object and the first image represented by the first image data, the first similarity, and the second similarity, which are calculated in the matching process; determining that the first object is to be selected as the target object when the first similarity exceeds a predetermined first threshold, the first similarity is higher than the second similarity, the third similarity exceeds a predetermined second threshold, and the third similarity is higher than the fourth similarity; determining that the second object is to be selected as the target object when the second similarity exceeds a predetermined first threshold, the second similarity is higher than the first similarity, the fourth similarity exceeds a predetermined second threshold, and the fourth similarity is higher than the third similarity; determining that the second object is to be selected as the target object when the first similarity exceeds a predetermined first threshold, the first similarity is higher than the second similarity, the third similarity exceeds a predetermined second threshold, the third similarity is lower than the fourth similarity, and the third similarity / the fourth similarity is smaller than the second similarity / the first similarity; determining that the first object is to be selected as the target object when the first similarity exceeds a predetermined first threshold, the first similarity is higher than the second similarity, the third similarity exceeds a predetermined second threshold, the third similarity is lower than the fourth similarity, and the second similarity / the first similarity is smaller than the third similarity / the fourth similarity; determining that the first object is to be selected as the target object when the second similarity exceeds a predetermined first threshold and is higher than the first similarity, the fourth similarity exceeds a predetermined second threshold and is lower than the third similarity, and the fourth similarity / the third similarity is smaller than the first similarity / the second similarity; determining that the second object is to be selected as the target object when the second similarity exceeds a predetermined first threshold and is higher than the first similarity, the fourth similarity exceeds a predetermined second threshold and is lower than the third similarity, and the first similarity / the second similarity is smaller than the fourth similarity / the third similarity; determining that the first object is selected as the target object when the first similarity exceeds a second threshold and the third similarity exceeds a first threshold, and the second similarity falls below the second threshold and / or the fourth similarity falls below the first threshold; determining that the second object is to be selected as the target object when the second similarity exceeds a predetermined second threshold and the fourth similarity exceeds a predetermined first threshold and the first similarity falls below a predetermined second threshold and / or the third similarity falls below a predetermined first threshold; When the first similarity is below a predetermined second threshold and / or the third similarity is below a predetermined first threshold, and when the second similarity is below a predetermined second threshold and / or the fourth similarity is below a predetermined first threshold, it is determined that the first and second objects are not to be selected as the target objects.

53. The control device of claim 52.

55. the second object has the same three-dimensional shape as the first object; The first two-dimensional model data and the second two-dimensional model data are two-dimensional model data that represent two-dimensional models of the three-dimensional shape.

54. The control device of claim 53.

56. A time when the first imaging device captures images of the first and second objects to generate the first image data is the same as a time when the second imaging device captures images of the first and second objects to generate the second image data.

53. A control device according to any one of claims 50 to 52.

57. the first imaging device is a monocular camera, The second imaging device is a stereo camera having two monocular cameras different from the monocular camera.

53. A control device according to any one of claims 1 to 6, 28 to 29 and 50 to 52.

58. A control device according to any one of claims 1 to 6, 28 to 29 and 50 to 52; The first imaging device; The second imaging device; A control system comprising:

59. a first filter that attenuates a light component of a first wavelength band from the light from the object; a second filter that attenuates a light component of a second wavelength band different from the first wavelength band in the light from the object; Further comprising: the first imaging device captures an image of the object by receiving light from the object via the first filter; The second imaging device receives light from the object via the second filter to capture an image of the object.

59. The control system of claim 58.

60. an illumination device that illuminates the object with illumination light including a light component in the second wavelength band; a light projection device that projects a pattern light including a light component of the first wavelength band onto the object; Further comprising: The second image capturing device captures an image of the object onto which the pattern light is projected from the light projection device, thereby generating the second image data.

60. The control system of claim 59.

61. A control device according to any one of claims 1 to 6, 28 to 29 and 50 to 52; The first imaging device; The second imaging device; The robot arm. A robot system comprising:

62. The processing device further comprises:

62. The robotic system of claim 61.

63. A control method for generating a control signal for controlling a robot arm provided with a processing device that performs processing on an object, a first imaging device that outputs first image data, and a second imaging device that outputs second image data, the method comprising: generating first information indicating a position and an orientation of the object using the first image data generated by the first imaging device capturing an image of the object; generating second information indicating a position and orientation of the object using three-dimensional position data, which is generated from the second image data generated by the second imaging device capturing an image of the object and indicates a three-dimensional position of each of a plurality of points of the object, and three-dimensional model data of the object at a position and orientation determined based on the first information; generating the control signal based on the second information; Including, The robot arm is controlled to move the processing device and the first and second imaging devices based on the generated control signal to process the object with the processing device. Control methods.

64. A control method for generating a control signal for controlling a robot arm provided with a processing device that performs processing on an object, a first imaging device that outputs first image data, and a second imaging device that outputs second image data, the method comprising: calculating an amount of change in position and orientation of the object between the first and second times based on two sets of first image data that are respectively generated by the first imaging device capturing images of the object at first and second times that are different from each other, and three-dimensional position data that is generated from the second image data that is generated by the second imaging device capturing images of the object at a third time and indicates three-dimensional positions of each of a plurality of points of the object; generating the control signal based on the amount of change; Including, The robot arm is controlled to move the processing device and the first and second imaging devices based on the generated control signal to process the object with the processing device. Control methods.

65. A control method for generating a control signal for controlling a robot arm provided with a processing device that processes at least one of a first object and a second object that is different from the first object in at least one of a position and an attitude, a first imaging device that outputs first image data, and a second imaging device that outputs second image data, the method comprising: determining whether or not to select the first or second object as a target object to be processed by the processing device, based on the first image data generated by the first imaging device capturing images of the first and second objects; When the first or second object is selected as the target object as a result of the determination, generating second information indicating a position and orientation of the target object using three-dimensional position data generated from the second image data generated by the second imaging device capturing images of the first and second objects and indicating three-dimensional positions of each of a plurality of points of the target object, and three-dimensional model data of the target object at a position and orientation determined based on first information indicating the position and orientation of the target object generated using the first image data; generating the control signal based on the second information; Including, The robot arm is controlled to move the processing device and the first and second imaging devices based on the generated control signal to process the object with the processing device. Control methods.

66. the first imaging device is a monocular camera, The second imaging device is a stereo camera having two monocular cameras different from the monocular camera.

66. A control method according to any one of claims 63 to 65.

67. A computer program product for causing a control method according to any one of claims 63 to 65 to be executed.