Control system, robot system, control method, and computer program
The control system for robots separates rough and fine movement processes to enhance the accuracy and efficiency of processing tasks by using image data for fine movements and independent rough movements, addressing the inefficiencies in existing systems.
Patent Information
- Application Number
- PCT/JP2023/017026
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-01
- Publication Date
- 2024-11-07
AI Technical Summary
Current control systems for robots that perform processing tasks on target objects lack efficient methods to accurately position and orient processing devices without relying on image data, leading to potential inaccuracies and inefficiencies in movement processes.
A control system that includes a combination of a control device and a robot control device, where the control device generates control signals based on image data from an imaging device to perform fine movement processes, while the robot control device performs rough movement processes independently of image data, allowing for precise positioning and orientation of processing devices.
This approach enables efficient and accurate movement of processing devices by separating rough and fine movement processes, improving the precision and efficiency of robot operations in processing tasks without relying solely on image data.
Smart Images

Figure JP2023017026_07112024_PF_FP_ABST
Abstract
Description
Control system, robot system, control method, and computer program
[0001] The present invention relates to the technical fields of a control system, a robot system, a control method, and a computer program that can generate control signals for controlling a robot, for example.
[0002] An example of a control device for controlling a robot equipped with a processing device such as an end effector is described in Patent Document 1. Such a control device is required to control the robot so as to efficiently move the processing device.
[0003] US Patent Application Publication No. 2013 / 0230235
[0004] According to a first aspect, there is provided a control system for generating a control signal for controlling a robot, the robot being equipped with a processing device that performs processing on a target object and an imaging device that generates at least image data of the target object by capturing an image of the target object, the robot moving the processing device and the imaging device, the control system including a control device that generates and outputs the control signal, the control device performing a second movement process after a first movement process, the first movement process including a process of controlling the robot to move the processing device without using the image data, and the second movement process including a process of generating and outputting the control signal for controlling the robot to move the processing device based on a calculation result of the position and orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object after the first movement process.
[0005] According to a second aspect, there is provided a control system for generating control signals for controlling a robot, the robot being equipped with a processing device that performs processing on a target object and an imaging device that generates at least image data of the target object by imaging the target object, the robot moving the processing device and the imaging device, the control system comprising a first control device that performs a first movement process to control the robot to move the processing device in accordance with movement instruction information, and a second control device that performs a second movement process to generate and output the control signal for controlling the robot to move the processing device based on the calculation results of the position and posture of the target object calculated based on the image data generated by the imaging device by imaging the target object, the second control device being a control device different from the first control device.
[0006] According to a third aspect, there is provided a robot system including the control system provided by the first or second aspect and the robot.
[0007] According to a fourth aspect, there is provided a control method for generating a control signal for controlling a robot, the robot being equipped with a processing device that performs processing on a target object and an imaging device that captures an image of the target object and outputs at least image data of the target object, the robot moving the processing device and the imaging device, the control method including performing a second movement process after a first movement process, the first movement process including controlling the robot to move the processing device without using the image data, and performing the second movement process including generating and outputting the control signal for controlling the robot to move the processing device based on a calculation result of the position and orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object after the first movement process.
[0008] According to a fifth aspect, there is provided a control method for generating a control signal for controlling a robot, the robot being equipped with a processing device that performs processing on a target object and an imaging device that captures an image of the target object and outputs at least image data of the target object, the robot moving the processing device and the imaging device, the control method including: performing a first movement process by a first control device to control the robot to move the processing device in accordance with movement instruction information; and then performing a second movement process by a second control device different from the first control device to generate and output the control signal for controlling the robot to move the processing device based on calculation results of the position and orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object.
[0009] According to a sixth aspect, there is provided a computer program that causes a computer to execute the control method provided by the fourth or fifth aspect.
[0010] The functions and other advantages of the present invention will become apparent from the following detailed description of the preferred embodiments.
[0011] FIG. 1 is a block diagram showing the configuration of a robot system according to this embodiment. FIG. 2 is a side view showing the appearance of a robot according to this embodiment. FIGS. 3( a) to 3(d) are cross-sectional views showing a first example of robot movement controlled by a robot control process including a rough movement process and a fine movement process. FIGS. 4( a) to 4(d) are cross-sectional views showing a second example of robot movement controlled by a robot control process including a rough movement process and a fine movement process. FIGS. 5( a) to 5(d) are cross-sectional views showing a third example of robot movement controlled by a robot control process including a rough movement process and a fine movement process. FIGS. 6( a) to 6(c) are cross-sectional views showing a third example of robot movement controlled by a robot control process including a rough movement process and a fine movement process. FIG. 7 is a block diagram showing the configuration of a control device according to this embodiment. FIG. 8 is a block diagram showing the configuration of a robot control device according to this embodiment. FIG. 9 is a flowchart showing the flow of robot control processing. FIG. 10 is a flowchart showing the flow of fine movement processing. FIG. 11 is a schematic diagram showing matching processing. FIG. 12 is a flowchart showing the flow of robot control processing in a first modified example. FIG. 13 shows a first example of robot control processing in which rough movement processing and fine movement processing are alternately repeated. FIG. 14 shows a second example of robot control processing in which rough movement processing and fine movement processing are alternately repeated. FIG. 15 shows edge model data. FIG. 16 shows model data indicating an object model and an end effector model. FIG. 17 shows an image in which multiple target objects detected by the matching processing appear, together with model data indicating the object models and end effector models that are aligned with each other. FIG. 18 is a flowchart showing the flow of robot control processing in a fifth modified example. FIG. 19 is a flowchart showing the flow of robot control processing in a sixth modified example. FIG. 20 is a flowchart showing the flow of robot control processing in a seventh modified example. FIG. 21 is a side view showing the appearance of a robot in the seventh modified example. FIG. 22 is a flowchart showing the flow of fine movement processing in an eighth modified example.
[0012] Hereinafter, embodiments of a control system, a robot system, a control method, and a computer program will be described with reference to the drawings. Hereinafter, the embodiments of the control system, the robot system, the control method, and the computer program will be described using a robot system SYS to which the embodiments of the control system, the robot system, the control method, and the computer program are applied.
[0013] (1) Configuration of the Robot System SYS First, the configuration of the robot system SYS will be described.
[0014] (1-1) Overall Configuration of Robot System SYS First, the overall configuration of the robot system SYS will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the overall configuration of the robot system SYS.
[0015] As shown in FIG. 1, the robot system SYS includes a robot 1, an imaging unit 2, a control device 3, a robot control device 4, and an end effector 5.
[0016] The robot 1 is a device capable of performing a predetermined process on a target object OBJ. An example of the robot 1 is shown in FIG. 2. FIG. 2 is a side view showing the appearance of the robot 1. As shown in FIG. 2, the robot 1 includes, for example, a base 11 and a robot arm 12. The robot 1 may also include the robot control device 4 described above. In other words, a device including the base 11, the robot arm 12, and the robot control device 4 may be referred to as the robot 1.
[0017] The base 11 is a component that forms the base of the robot 1. The base 11 is placed on a support surface S, such as a floor surface. The base 11 may be fixed to the support surface S. Alternatively, the base 11 may be movable relative to the support surface S. As an example, the base 11 may be self-propelled on the support surface S. In this case, the base 11 may be installed on an automatic guided vehicle (AGV). When the base 11 is installed on an automatic guided vehicle, the automatic guided vehicle may be considered to be part of the robot 1. Alternatively, the automatic guided vehicle (AGV) may be used as the base 11. Note that FIG. 2 shows an example in which the base 11 is fixed to the support surface S.
[0018] The robot arm 12 is attached to the base 11. The robot arm 12 is a device in which a plurality of links 121 are connected via joints 122. An actuator is built into the joint 122. The link 121 may be rotatable around an axis defined by the joint 122 by the actuator built into the joint 122. At least one link 121 may be extendable and contractible along the direction in which the link 121 extends. A device including the device in which a plurality of links 121 are connected via joints 122 and the base 11 may be referred to as the robot arm 12.
[0019] An end effector 5 is attached to the robot arm 12. That is, the end effector 5 is attached to the robot 1. In the example shown in FIG. 2 , the end effector 5 is attached to the tip of the robot arm 12. The end effector 5 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the end effector 5. That is, the robot 1 moves the end effector 5.
[0020] The end effector 5 is a device that performs a predetermined process (in other words, a predetermined operation) on the target object OBJ. The end effector 5 that performs a predetermined process on the target object OBJ may also be called a processing device.
[0021] For example, the end effector 5 may perform a holding process for holding a target object OBJ as an example of the predetermined process. In this case, the end effector 5 may be considered to be performing the holding process on the target object OBJ that is to be held by the end effector 5. An end effector 5 capable of performing the holding process may be referred to as a holding device.
[0022] Note that holding the target object OBJ may include, for example, gripping the target object OBJ. For example, holding the target object OBJ may include gripping the target object OBJ using a hand gripper (described later), which is an example of the end effector 5. Holding the target object OBJ may include attracting the target object OBJ. For example, holding the target object OBJ may include attracting (vacuum suction) the target object OBJ using a vacuum gripper (described later), which is an example of the end effector 5. For example, holding the target object OBJ may include attracting the target object OBJ using a magnetic attraction gripper (described later), which is an example of the end effector 5.
[0023] For example, as an example of the predetermined process, the end effector 5 may perform a release process (in other words, a release operation) to release (i.e., let go of) the target object OBJ that it is holding. In this case, the end effector 5 may be considered to be performing the release process on the target object OBJ that it is holding. An end effector 5 that is capable of performing the release process may be called a release device.
[0024] A hand clipper is an example of an end effector 5 capable of holding and releasing an object OBJ. A hand clipper is an end effector 5 that can hold a target object OBJ by physically pinching the target object OBJ using multiple (e.g., two, three, or four) finger members or claw members. Another example of an end effector 5 that can hold and release an object OBJ is a vacuum clipper. A vacuum clipper is an end effector 5 that can hold a target object OBJ by vacuum-suctioning the target object OBJ. FIG. 2 shows an example of the end effector 5 being a hand clipper.
[0025] The robot 1 may perform, as an example of a predetermined process, a placement process (in other words, a placement operation) for placing a target object OBJ at a desired position using the end effector 5 capable of holding and releasing processes. For example, the robot 1 may hold a first target object OBJ using the end effector 5, and then perform a placement process for placing the first target object OBJ held by the end effector 5 at a desired position of a second target object OBJ different from the first target object OBJ. In this case, the end effector 5 may be considered to be performing a release process for the second target object OBJ on which the end effector 5 is to place the first target object OBJ. Similarly, the end effector 5 may be considered to be performing a release process for the first target object OBJ to be released by the end effector 5.
[0026] The robot 1 may use the end effector 5 capable of performing a holding process and a release process to perform a fitting process (in other words, a fitting operation) for fitting a first target object OBJ into a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (in other words, a placement operation). For example, the robot 1 may hold the first target object OBJ using the end effector 5, and then perform a fitting process for fitting the first target object OBJ held by the end effector 5 into a second target object OBJ different from the first target object OBJ. In this case, the end effector 5 may be considered to be performing a release process for the second target object OBJ into which the end effector 5 should fit the first target object OBJ. Similarly, the end effector 5 may be considered to be performing a release process for the first target object OBJ to be released by the end effector 5.
[0027] The fitting process may include a process for fitting (i.e., inserting) the first target object OBJ into a hole formed in the second target object OBJ. For example, the robot 1 may hold the first target object OBJ using the end effector 5, and then perform a fitting process for fitting (i.e., inserting) the first target object OBJ held by the end effector 5 into a hole formed in the second target object OBJ. In this case, the end effector 5 may be considered to be performing a release process for the second target object OBJ, which has a hole formed therein into which the first target object OBJ should be fitted (i.e., inserted) by the end effector 5. Similarly, the end effector 5 may be considered to be performing a release process for the first target object OBJ that the end effector 5 should release.
[0028] The robot 1 may use the end effector 5 capable of holding and releasing operations to perform a pasting process (i.e., a pasting operation) for pasting a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (i.e., a placement operation). For example, the robot 1 may hold the first target object OBJ using the end effector 5, and then perform a pasting process for pasting the first target object OBJ held by the end effector 5 to a second target object OBJ different from the first target object OBJ. In this case, the end effector 5 may be considered to be performing a release process on the second target object OBJ to which the end effector 5 is to paste the first target object OBJ. Similarly, the end effector 5 may be considered to be performing a release process on the first target object OBJ to be released by the end effector 5.
[0029] The robot 1 may use the end effector 5 capable of holding and releasing operations to perform a bonding process (i.e., a bonding operation) for bonding a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (i.e., a placement operation). For example, the robot 1 may hold the first target object OBJ using the end effector 5, and then perform a bonding process for bonding the first target object OBJ held by the end effector 5 to a second target object OBJ different from the first target object OBJ. In this case, the end effector 5 may be considered to be performing a release process for the second target object OBJ to which the end effector 5 is to bond the first target object OBJ. Similarly, the end effector 5 may be considered to be performing a release process for the first target object OBJ to which the end effector 5 is to release.
[0030] The robot 1 may use the end effector 5 capable of holding and releasing operations to perform a welding process (i.e., a welding operation) for welding a first target object OBJ to a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (i.e., a placement operation). For example, the robot 1 may hold the first target object OBJ using the end effector 5, and then perform a welding process for welding the first target object OBJ held by the end effector 5 to a second target object OBJ different from the first target object OBJ. In this case, the end effector 5 may be considered to be performing a release process for the second target object OBJ to which the end effector 5 is to weld the first target object OBJ. Similarly, the end effector 5 may be considered to be performing a release process for the first target object OBJ to be released by the end effector 5.
[0031] The robot 1 may use the end effector 5 capable of holding and releasing operations to perform a screw tightening process (i.e., a screw tightening operation) for tightening a first target object OBJ capable of functioning as a screw into a screw hole formed in a second target object OBJ different from the first target object OBJ, as a specific example of a placement process (i.e., a placement operation). For example, the robot 1 may hold the first target object OBJ using the end effector 5, and then perform a screw tightening process for tightening the first target object OBJ held by the end effector 5 into a second target object OBJ different from the first target object OBJ. In this case, the end effector 5 may be considered to be performing a release process for the second target object OBJ into which the end effector 5 is to tighten the first target object OBJ. Similarly, the end effector 5 may be considered to be performing a release process for the first target object OBJ to be released by the end effector 5.
[0032] For example, the end effector 5 may perform a predetermined process on each of multiple target objects OBJ. That is, the end effector 5 may perform the predetermined process on the multiple target objects OBJ in sequence. In this case, the robot 1 may move the end effector 5 to a first position where the end effector 5 can perform the predetermined process on a first target object OBJ, and after the end effector 5 has moved to the first position, the end effector 5 may perform the predetermined process on the first target object OBJ. Thereafter, the robot 1 may move the end effector 5 to a second position where the end effector 5 can perform the predetermined process on a second target object OBJ different from the first target object OBJ, and after the end effector 5 has moved to the second position, the end effector 5 may perform the predetermined process on the second target object OBJ.
[0033] For example, the end effector 5 may perform a predetermined process on each of multiple portions of a single target object OBJ. That is, the end effector 5 may sequentially perform a predetermined process on multiple portions of a single target object OBJ. In this case, the robot 1 may move the end effector 5 to a third position where the end effector 5 can perform a predetermined process on a first portion of the target object OBJ, and after the end effector 5 has moved to the third position, the end effector 5 may perform the predetermined process on the first portion of the target object OBJ. Thereafter, the robot 1 may move the end effector 5 to a fourth position where the end effector 5 can perform a predetermined process on a second portion of the target object OBJ, and after the end effector 5 has moved to the fourth position, the end effector 5 may perform the predetermined process on the second portion of the target object OBJ.
[0034] As an example of an operation in which the robot 1 performs a predetermined process on each of multiple portions of a single target object OBJ, the robot 1 may repeatedly perform a fitting process on a second target object OBJ having multiple holes formed therein, fitting the first target object OBJ into each hole. Note that a portion of the target object OBJ on which the end effector 5 performs a predetermined process may be referred to as a target portion or a target member. For example, each portion (each member) of the second target object OBJ in which each hole is formed may be referred to as a target portion or a target member. Note that when the target object OBJ is composed of multiple parts, each part may be referred to as a target portion or a target member. When the target object OBJ is composed of multiple members, each member may be considered as a target portion or a target member.
[0035] As shown in FIG. 2 , the target object OBJ on which the end effector 5 performs a predetermined process may include a workpiece W. The workpiece W may include, for example, a part or member used to manufacture a desired product. The workpiece W may include, for example, a part or member to be processed to manufacture the desired product. The workpiece W may include, for example, a part or member to be transported to manufacture the desired product. The workpiece W may include, for example, a part or member that is moving due to transportation to manufacture the desired product. The workpiece W may include, for example, a part or member that is moving to manufacture the desired product.
[0036] As shown in FIG. 2 , the target object OBJ on which the end effector 5 performs a predetermined process may include a placement device T on which a workpiece W is placed. An example of the placement device T is a pallet. The placement device T may be disposed on a support surface S. The placement device T may be fixed to the support surface S. Alternatively, at least a portion of the placement device T may be movable relative to the support surface S. As a first example, the placement device T may be self-propelled on the support surface S. A self-propelled placement device T may be referred to as an automatic guided vehicle (AGV). As a second example, the placement device T may be a belt conveyor. In this case, a belt that is part of the belt conveyor and on which the workpiece W is placed may move relative to the support surface S. As a third example, the placement device T may be capable of flying above the support surface S. In other words, the placement device T may be capable of moving relative to the support surface S by flying over the support surface S. In this case, the placement device T may be referred to as an unmanned aerial vehicle. As a fourth example, the placement device T may be another robot arm that is different from the robot arm 12 and is equipped with an end effector capable of holding the workpiece W. In this case, the end effector attached to the other robot arm may move relative to the support surface S. Note that FIG. 2 shows an example in which the placement device T is self-propelled on the support surface S. When at least the placement device T is movable relative to the support surface S, the workpiece W placed on the placement device T also moves relative to the support surface S as the placement device T moves. Therefore, the placement device T may be capable of functioning as a moving device that moves the workpiece W. The placement device T may be capable of functioning as a transport device that transports the workpiece W. The transport device may be a device that moves the workpiece W. Note that the transport device may be referred to as a moving device.
[0037] In addition, when at least a part of the placement device T is movable relative to the support surface S, a first placement device T corresponding to a pallet on which the workpiece W is placed may be placed on a second placement device T movable relative to the support surface S. In this case, an apparatus including the first placement device T and the second placement device T may be referred to as the placement device T.
[0038] However, the target object OBJ does not have to include the mounting device T. For example, the workpiece W may be placed on a support surface S.
[0039] When the target object OBJ includes a workpiece W and a mounting device T, the holding process described above may include a process of holding a workpiece W placed on a stationary or moving mounting device T. The holding process described above may include a process of holding a workpiece W placed on a support surface S. The release process described above may include a process of releasing a workpiece W held by the end effector 5 in order to place the workpiece W held by the end effector 5 at a desired position on a stationary or moving mounting device T. The release process described above may include a process of releasing a workpiece W held by the end effector 5 in order to place the workpiece W held by the end effector 5 at a desired position on the support surface S. The release process described above may include a process of releasing a first workpiece W held by the end effector 5 in order to fit the first workpiece W held by the end effector 5 into a second workpiece W placed on a stationary or moving mounting device T. The above-described release process may include a process of releasing a first workpiece W held by the end effector 5 in order to fit the first workpiece W held by the end effector 5 into a second workpiece W placed on the support surface S. The above-described release process may include a process of releasing the first workpiece W held by the end effector 5 in order to fit (i.e., insert) the first workpiece W held by the end effector 5 into a hole formed in a second workpiece W placed on a stationary or moving mounting device T. The above-described release process may include a process of releasing the first workpiece W held by the end effector 5 in order to fit (i.e., insert) the first workpiece W held by the end effector 5 into a hole formed in the second workpiece W placed on the support surface S.
[0040] Note that FIG. 2 shows an example in which the robot 1 is a robot arm 12 (i.e., a vertical articulated robot). However, the robot 1 may be a robot different from a vertical articulated robot. For example, the robot 1 may be a SCARA robot (i.e., a horizontal articulated robot). For example, the robot 1 may be a parallel link robot. For example, the robot 1 may be a dual-arm robot having two robot arms 12. For example, the robot 1 may be a Cartesian coordinate robot. For example, the robot 1 may be a cylindrical coordinate robot. The robot 1 may be referred to as a movable device. The movable device may include, in addition to the robot 1, at least one of an automatic guided vehicle and an unmanned aerial vehicle. For example, the robot 1 may be installed in at least one of an automatic guided vehicle and an unmanned aerial vehicle.
[0041] 1, the imaging unit 2 captures an image of the target object OBJ. To capture the image of the target object OBJ, the imaging unit 2 includes an imaging device 21 and an illumination device 23.
[0042] The imaging device 21 is a camera capable of capturing an image of the target object OBJ. For example, the imaging device 21 may capture an image of the target object OBJ under the control of the control device 3. The imaging device 21 generates image data IMG by capturing an image of the target object OBJ. That is, the imaging device 21 generates image data IMG that is the result of capturing the image of the target object OBJ. The image data IMG generated by the imaging device 21 is output from the imaging device 21 to the control device 3. As a result, the control device 3 acquires the image data IMG acquired by capturing the image of the target object OBJ by the imaging device 21. In this embodiment, the imaging device 21 is a monocular camera. Specifically, the imaging device 21 is capable of capturing an image of the target object OBJ using a monocular camera (in other words, an image sensor). However, as will be described in detail later in a modified example, the imaging device 21 is not limited to being a monocular camera.
[0043] The imaging device 21 may capture an image of the entire target object OBJ. Alternatively, the imaging device 21 may capture an image of a portion of the target object OBJ. In other words, the imaging device 21 may capture an image of a portion of the target object OBJ while not capturing an image of another portion of the target object OBJ.
[0044] The imaging device 21 may capture an image of a single target object OBJ. In other words, a single target object OBJ may appear in the image represented by the image data IMG. Alternatively, the imaging device 21 may capture an image of multiple target objects OBJ. In other words, a plurality of target objects OBJ may appear in the image represented by the image data IMG. In this case, as will be described in detail later, the control device 3 may determine (in other words, select) one of the multiple target objects OBJ captured by the imaging device 21 as the target object OBJ on which the end effector 5 will actually perform a predetermined process. Note that one of the multiple target objects OBJ captured by the imaging device 21 on which the end effector 5 will actually perform a predetermined process may be referred to as a process execution object.
[0045] The multiple target objects OBJ captured by the imaging device 21 may be arranged such that at least two of the multiple target objects OBJ at least partially overlap. As an example, if the target objects OBJ are workpieces W corresponding to parts used to manufacture a desired product, the multiple workpieces W (i.e., multiple parts) may be arranged such that at least two of the multiple workpieces W at least partially overlap. In this case, the robot 1 may perform bulk picking, in which the robot 1 picks workpieces W one by one from the multiple workpieces W that are arranged in a disorderly manner.
[0046] The imaging unit 2 is attached to the robot arm 12, similar to the end effector 5. That is, the imaging device 21 and the lighting device 23 are attached to the robot arm 12. For example, as shown in FIG. 2 , the imaging device 21 and the lighting device 23 may be attached to the tip of the robot arm 12, similar to the end effector 5. In this case, the imaging device 21 and the lighting device 23 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the imaging device 21 and the lighting device 23.
[0047] The imaging device 21 may capture an image of the target object OBJ while the imaging device 21 and the target object OBJ are displaced relative to each other. Note that the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may refer to a state in which the relative positional relationship between the imaging device 21 and the target object OBJ is changing. The state in which the imaging device 21 and the target object OBJ are displaced relative to each other may refer to a state in which the imaging device 21 and the target object OBJ are moving relative to each other. For example, the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the target object OBJ is moving relative to the imaging device 21. For example, the state in which the imaging device 21 and the target object OBJ are displaced relative to each other may include a state in which the imaging device 21 is moving relative to the target object OBJ. In this case, the imaging device 21 does not need to be stationary to capture an image of the target object OBJ, and the robot system SYS can efficiently perform a predetermined process on the target object OBJ using the end effector 5.
[0048] Alternatively, the imaging device 21 may capture an image of the target object OBJ during a period in which there is no relative displacement between the imaging device 21 and the target object OBJ. Note that the state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the relative positional relationship between the imaging device 21 and the target object OBJ is not changing. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which there is no relative movement between the imaging device 21 and the target object OBJ. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the imaging device 21 and the target object OBJ are stationary. The state in which there is no relative displacement between the imaging device 21 and the target object OBJ may refer to a state in which the imaging device 21 and the target object OBJ are moving at the same moving speed in the same moving direction.
[0049] The illumination device 23 is a device capable of irradiating illumination light onto the target object OBJ. For example, the illumination device 23 may irradiate the target object OBJ with illumination light under the control of the control device 3. In particular, the illumination device 23 is a device capable of irradiating the target object OBJ with illumination light, thereby illuminating the target object OBJ with the illumination light. The illumination light may be light having a uniform intensity distribution. Note that the illumination light may be light having a non-uniform intensity distribution. In this case, the imaging device 21 may image the target object OBJ illuminated with the illumination light. However, the illumination device 23 does not need to irradiate the target object OBJ with illumination light. In this case, the imaging unit 2 (robot system SYS) does not need to be equipped with the illumination device 23.
[0050] The control device 3 performs robot control processing. The robot control device 4 also performs robot control processing. In other words, the control device 3 performs robot control processing in cooperation with the robot control device 4. The robot control device 4 performs robot control processing in cooperation with the control device 3. In this way, in this embodiment, the control device 3 and the robot control device 4, which are different control devices, perform robot control processing while cooperating with each other.
[0051] However, the control device 3 and the robot control device 4 may be integrated. In other words, a control device that can function as both the control device 3 and the robot control device 4 may perform the robot control process.
[0052] The robot control process may include a process of generating a control signal for controlling the robot 1. In other words, the robot control process may include a process of generating a control signal for controlling the operation of the robot 1. The control signal for controlling the robot 1 may be referred to as a robot control signal. The robot control process may include a process of generating a control signal for controlling the end effector 5 attached to the robot 1. In other words, the robot control process may include a process of generating a control signal for controlling the operation of the end effector 5 attached to the robot 1. The control signal for controlling the end effector 5 may be referred to as an end effector control signal.
[0053] The control signal for controlling the robot 1 may include a signal for controlling the robot arm 12. For example, as described above, the robot arm 12 moves the end effector 5. In this case, the control signal may include a signal for controlling the robot arm 12 so that the end effector 5 is located at a desired position. The control signal may include a signal for controlling the robot arm 12 so that the end effector 5 moves to a desired position. The control signal may include a signal for controlling the robot arm 12 so that the positional relationship between the end effector 5 and the target object OBJ is a desired positional relationship.
[0054] When the end effector 5 performs a holding process to hold the target object OBJ, the control signal may include a signal for controlling the robot arm 12 so that the end effector 5 moves toward (i.e., approaches) a holding position where the end effector 5 can hold the target object OBJ. In other words, the control signal may include a signal for controlling the robot arm 12 so that the end effector 5 is positioned at the holding position. Furthermore, the control signal may include a signal for controlling the end effector 5 so that the end effector 5 positioned at the above-mentioned holding position holds the target object OBJ.
[0055] When performing a release process to release the target object OBJ held by the end effector 5, the control signal may include a signal for controlling the robot arm 12 so that the end effector 5 moves toward (i.e., approaches) a release position where the end effector 5 should release the target object OBJ held by the end effector 5. In other words, the control signal may include a signal for controlling the robot arm 12 so that the end effector 5 is located at the release position. Furthermore, the control signal may include a signal for controlling the end effector 5 so that the end effector 5 located at the release position releases the target object OBJ held by the end effector 5.
[0056] The control signal may include a signal that can be used as is to control the operation of the robot 1. Specifically, the control signal may include a drive signal (robot drive signal) that can be input to the robot 1 to control the operation of the robot 1. The control signal may include a drive signal (robot drive signal) that can be input to the robot 1 to control the drive amount of the robot 1. For example, the control signal may include a drive signal for driving an actuator built into the joint 122 of the robot arm 12. For example, the control signal may include a drive signal for controlling the drive amount of an actuator built into the joint 122 of the robot arm 12.
[0057] The control signal may include a signal that can be used directly to control the operation of the end effector 5. Specifically, the control signal may include a drive signal (end effector drive signal) that can be input to the end effector 5 to control the operation of the end effector 5. The control signal may include a drive signal (end effector drive signal) that can be input to the end effector 5 to control the drive amount of the end effector 5. For example, the control signal may include a drive signal for driving an actuator that operates a hand clipper that constitutes the end effector 5. For example, the control signal may include a drive signal for controlling the drive amount of the actuator that operates the hand clipper that constitutes the end effector 5. For example, the control signal may include a drive signal for driving a vacuum device of a vacuum clipper that constitutes the end effector 5. For example, the control signal may include a drive signal for controlling the drive amount of the vacuum device of the vacuum clipper that constitutes the end effector 5.
[0058] The control signal may include a signal that can be used to generate the above-mentioned drive signal that can be input to the robot 1 to control the movement of the robot 1. For example, the control signal may include a signal that can be used to generate a drive signal for an actuator built into the joint 122 of the robot arm 12.
[0059] The control signal may include a signal usable to generate the above-mentioned drive signal that can be input to the end effector 5 to control the operation of the end effector 5. For example, the control signal may include a signal usable to generate a drive signal for an actuator that moves a hand clipper that constitutes the end effector 5. For example, the control signal may include a signal usable to generate a drive signal for a vacuum device of a vacuum clipper that constitutes the end effector 5.
[0060] The control device 3 may generate a control signal by performing robot control processing. When the control device 3 generates a control signal, the control signal generated by the control device 3 may be input from the control device 3 to the robot control device 4. In other words, the control device 3 may output the control signal generated by the control device 3 to the robot control device 4.
[0061] When the control device 3 generates a signal that can be used as is to control the operation of the robot 1 (i.e., a drive signal that can be input to the robot 1 to control the operation of the robot 1) as a control signal, the robot control device 4 may use the control signal generated by the control device 3 as is to control the operation of the robot 1. Specifically, the robot control device 4 may control the operation of the robot 1 by outputting the control signal generated by the control device 3 to the robot 1 as a drive signal that can be input to the robot 1 to control the operation of the robot 1.
[0062] When the control device 3 generates a signal that can be used as is to control the operation of the end effector 5 (i.e., a drive signal that can be input to the end effector 5 to control the operation of the end effector 5) as a control signal, the robot control device 4 may use the control signal generated by the control device 3 as is to control the operation of the end effector 5. Specifically, the robot control device 4 may control the operation of the end effector 5 by outputting the control signal generated by the control device 3 to the end effector 5 as a drive signal that can be input to the end effector 5 to control the operation of the end effector 5. Note that outputting the control signal to the end effector 5 may include at least one of directly outputting the control signal to the end effector 5 and indirectly outputting the control signal to the end effector 5 via the robot 1.
[0063] When the control device 3 generates, as a control signal, a signal that can be used to generate a drive signal that can be input to the robot 1 to control the operation of the robot 1, the robot control device 4 may generate, as a control signal, a drive signal that can be input to the robot 1 to control the operation of the robot 1, based on the control signal generated by the control device 3. Thereafter, the robot control device 4 may control the operation of the robot 1 by outputting the control signal generated by the robot control device 4 to the robot 1.
[0064] When the control device 3 generates, as a control signal, a signal that can be used to generate a drive signal that can be input to the end effector 5 to control the operation of the end effector 5, the robot control device 4 may generate, as a control signal, a drive signal that can be input to the end effector 5 to control the operation of the end effector 5, based on the control signal generated by the control device 3. Thereafter, the robot control device 4 may control the operation of the end effector 5 by outputting the control signal generated by the robot control device 4 to the end effector 5.
[0065] Alternatively, when the control device 3 generates a control signal, the control signal generated by the control device 3 may be input from the control device 3 to the robot 1 or the end effector 5 without going through the robot control device 4. For example, when the control device 3 generates a signal that can be used as is to control the operation of the robot 1 (i.e., a drive signal that can be input to the robot 1 to control the operation of the robot 1) as a control signal, the control device 3 may input the control signal generated by the control device 3 to the robot 1 without going through the robot control device 4. For example, when the control device 3 generates a signal that can be used as is to control the operation of the end effector 5 (i.e., a drive signal that can be input to the end effector 5 to control the operation of the end effector 5) as a control signal, the control device 3 may input the control signal generated by the control device 3 to the end effector 5 without going through the robot control device 4.
[0066] In addition to or instead of the control device 3, the robot control device 4 may generate a control signal by performing robot control processing. For example, the robot control device 4 may generate, as a control signal, a drive signal that can be input to the robot 1 to control the operation of the robot 1. The robot control device 4 may then control the operation of the robot 1 by outputting the control signal generated by the robot control device 4 to the robot 1. For example, the robot control device 4 may generate, as a control signal, a drive signal that can be input to the end effector 5 to control the operation of the end effector 5. The robot control device 4 may then control the operation of the end effector 5 by outputting the control signal generated by the robot control device 4 to the end effector 5.
[0067] In particular, in this embodiment, the control device 3 performs fine movement processing as at least a part of the robot control processing. Furthermore, the robot control device 4 performs rough movement processing as at least a part of the robot control processing. However, in addition to or instead of the control device 3, the robot control device 4 may perform fine movement processing as at least a part of the robot control processing. In this case, the robot control device 4 may perform both the rough movement processing and the fine movement processing, or the robot system SYS may not include the control device 3. Similarly, in addition to or instead of the robot control device 4, the control device 3 may perform rough movement processing as at least a part of the robot control processing. In this case, the control device 3 may perform both the rough movement processing and the fine movement processing. Note that if the control device 3 that performs both the rough movement processing and the fine movement processing generates a control signal that can be used directly to control the operation of the robot 1, the robot system SYS may not include the robot control device 4. In the following explanation, an example will be described in which the control device 3 performs the fine movement processing and the robot control device 4 performs the rough movement processing.
[0068] Both the fine movement processing and the rough movement processing are processing for moving the end effector 5. Specifically, both the fine movement processing and the rough movement processing are processing for controlling the robot 1 to move the end effector 5. In other words, both the fine movement processing and the rough movement processing are processing for generating control signals for controlling the robot 1 to move the end effector 5, and outputting the generated signals.
[0069] As described above, the moved end effector 5 performs a predetermined process on the target object OBJ. In this case, both the fine movement process and the rough movement process may be considered to be processes (movement processes) performed by the end effector 5 on the target object OBJ. In other words, both the fine movement process and the rough movement process may be considered to be processes performed by the end effector 5 on the target object OBJ. Specifically, both the fine movement process and the rough movement process may be considered to be processes for moving the end effector 5 with respect to the target object OBJ. For example, both the fine movement process and the rough movement process may be considered to be processes for moving the end effector 5 so that the end effector 5 approaches the target object OBJ. In other words, both the fine movement process and the rough movement process may be considered to be processes for controlling the robot 1 so that the end effector 5 approaches the target object OBJ by moving the end effector 5.
[0070] As a first example, as described above, the end effector 5 may perform a holding process, which is an example of a predetermined process, on the target object OBJ in order to hold the target object OBJ. In this case, both the fine movement process and the rough movement process may be considered to be processes on the target object OBJ that the end effector 5 is to hold. Specifically, both the fine movement process and the rough movement process may be considered to be processes for moving the end effector 5 toward the target object OBJ that the end effector 5 is to hold. For example, both the fine movement process and the rough movement process may be considered to be processes for controlling the robot 1 so that the end effector 5 approaches the target object OBJ that the end effector 5 is to hold by moving the end effector 5.
[0071] As a second example, as described above, the end effector 5 may perform a release process, which is an example of a predetermined process, to release the held first target object OBJ. For example, as described above, the end effector 5 may perform a release process, which is an example of a predetermined process, on a second target object OBJ from which the held first target object OBJ is to be released, in order to place the held first target object OBJ at a desired position of the second target object OBJ. For example, as described above, the end effector 5 may perform a release process, which is an example of a predetermined process, on a second target object OBJ from which the held first target object OBJ is to be released, in order to fit the held first target object OBJ into the second target object OBJ. For example, as described above, the end effector 5 may perform a release process, which is an example of a predetermined process, on a second object OBJ from which the held first object OBJ is to be released in order to fit (i.e., insert) the held first object OBJ into a hole formed in the second object OBJ. In this case, both the fine movement process and the rough movement process may be considered to be processes performed by the end effector 5 on the second object OBJ from which the first object OBJ is to be released. Specifically, both the fine movement process and the rough movement process may be considered to be processes of moving the end effector 5 holding the first object OBJ toward the second object OBJ from which the end effector 5 is to release the first object OBJ. For example, both the fine movement process and the rough movement process may be considered to be processes that control the robot 1 by moving the end effector 5 so that the end effector 5 approaches the second target object OBJ from which the end effector 5 should release the first target object OBJ.
[0072] At least one of the fine movement processing and the rough movement processing may be performed on a moving target object OBJ. For example, as described above, if a workpiece W, which is an example of the target object OBJ, is placed on a placement device T that can function as a transport device for transporting the workpiece W, at least one of the fine movement processing and the rough movement processing may be performed on the workpiece W that is being moved by the transport device. For example, as described above, if the placement device T, which is an example of the target object OBJ, is movable, at least one of the fine movement processing and the rough movement processing may be performed on the moving placement device T. However, at least one of the fine movement processing and the rough movement processing may also be performed on a stationary target object OBJ.
[0073] The fine movement processing and the rough movement processing differ from each other in that the method of controlling the robot 1 to move the end effector 5 by the fine movement processing is different from the method of controlling the robot 1 to move the end effector 5 by the rough movement processing. The difference between the fine movement processing and the rough movement processing will be described in detail later when explaining the flow of the robot control processing, but the difference will be briefly explained below.
[0074] The fine movement process includes a process of controlling the robot 1 to move the end effector 5 based on at least one of the calculation results of the position and the orientation of the target object OBJ, which are based on the image data IMG generated by the imaging device 21. In other words, the fine movement process includes a process of generating a control signal for controlling the robot 1 to move the end effector 5, based on at least one of the calculation results of the position and the orientation of the target object OBJ, which are based on the image data IMG generated by the imaging device 21, and outputting the generated control signal (for example, outputting it to the robot 1 or the robot control device 4).
[0075] As an example, the fine movement processing may include a process of acquiring image data IMG generated by the imaging device 21 from the imaging device 21. Because the imaging device 21 generates the image data IMG to generate a control signal, the fine movement processing may include a process of the imaging device 21 generating image data by capturing an image of the target object OBJ. Because the imaging device 21 generates the image data IMG to generate a control signal, the fine movement processing may be performed in parallel with a process of the imaging device 21 generating image data by capturing an image of the target object OBJ. In this case, the fine movement processing may be considered to include a process of the imaging device 21 generating image data by capturing an image of the target object OBJ. The fine movement processing may include a process of calculating at least one of the position and orientation of the target object OBJ based on the image data IMG generated by the imaging device 21. The fine movement processing may include a process of generating a control signal for controlling the robot 1 to move the end effector 5 based on the calculation result of at least one of the position and orientation of the target object OBJ. The fine movement process may include a process of outputting the generated control signal (for example, outputting it to the robot 1 or the robot control device 4).
[0076] On the other hand, the rough movement processing includes processing for controlling the robot 1 to move the end effector 5 without using image data IMG generated by the imaging device 21. The rough movement processing includes processing for controlling the robot 1 to move the end effector 5 without using calculation results of at least one of the position and orientation of the target object OBJ based on the image data IMG. In other words, the rough movement processing includes processing for generating a control signal for controlling the robot 1 to move the end effector 5 without using image data IMG generated by the imaging device 21, and outputting the generated control signal (e.g., outputting to the robot 1). The rough movement processing includes processing for generating a control signal for controlling the robot 1 to move the end effector 5 without using calculation results of at least one of the position and orientation of the target object OBJ based on the image data IMG, and outputting the generated control signal (e.g., outputting to the robot 1).
[0077] However, the control signals used by the rough movement processing may be generated in advance before the rough movement processing is performed. In this case, the rough movement processing may not include processing for generating control signals that control the robot 1 to move the end effector 5. The rough movement processing may include processing for outputting (e.g., generating) the control signals that were generated in advance before the rough movement processing is started. Furthermore, when the rough movement processing is performed multiple times, the control signals used by at least one of the multiple rough movement processing operations may be generated in advance.
[0078] In this embodiment, the control device 3 starts the fine movement processing after the rough movement processing. That is, the control device 3 performs the fine movement processing after the rough movement processing. In this case, the control device 3 may generate a control signal for controlling the robot 1 to move the end effector 5 based on a calculation result of at least one of the position and orientation of the target object OBJ based on image data IMG generated by the imaging device 21 capturing an image of the target object OBJ after the rough movement processing.
[0079] Specifically, the control device 3 may start the fine movement processing after the robot control device 4 has started the rough movement processing. In other words, the control device 3 may perform the fine movement processing after the robot control device 4 has started the rough movement processing. Conversely, the robot control device 4 may start the rough movement processing before the control device 3 has started the fine movement processing. In other words, the robot control device 4 may perform the rough movement processing before the control device 3 has started the fine movement processing. In this case, the control device 3 may generate a control signal for controlling the robot 1 to move the end effector 5 based on a calculation result of at least one of the position and orientation of the target object OBJ based on image data IMG generated by the imaging device 21 capturing an image of the target object OBJ after the robot control device 4 has started the rough movement processing.
[0080] As an example, the control device 3 starts the fine movement process after the robot control device 4 has performed the rough movement process. In other words, the control device 3 starts the fine movement process after the robot control device 4 has performed the rough movement process. In other words, the control device 3 starts the fine movement process after the robot control device 4 has completed (i.e., ended) the rough movement process. In other words, the control device 3 performs the fine movement process after the robot control device 4 has completed (i.e., ended) the rough movement process. In this case, the control device 3 generates a control signal for controlling the robot 1 to move the end effector 5 based on a calculation result of at least one of the position and orientation of the target object OBJ, which is based on image data IMG generated by the imaging device 21 capturing an image of the target object OBJ after the robot control device 4 has performed the rough movement process. In other words, the control device 3 generates a control signal for controlling the robot 1 to move the end effector 5 based on a calculation result of at least one of the position and orientation of the target object OBJ, which is based on image data IMG generated by the imaging device 21 capturing an image of the target object OBJ after the robot control device 4 has completed (i.e., ended) the rough movement process.
[0081] However, as will be described later in a first modified example, the control device 3 may start the fine movement process before the robot control device 4 completes (i.e., ends) the rough movement process. In other words, the control device 3 may perform at least a part of the fine movement process before the robot control device 4 completes (i.e., ends) the rough movement process.
[0082] The rough movement processing performed before the fine movement processing may include processing for moving the end effector 5 so that the end effector 5 approaches the target object OBJ until the distance between the end effector 5 and the target object OBJ becomes a first distance (i.e., processing for controlling the robot 1 to move the end effector 5; the same applies hereinafter). On the other hand, the fine movement processing performed after the rough movement processing may include processing for moving the end effector 5 so that the end effector 5 approaches the target object OBJ until the distance between the end effector 5 and the target object OBJ becomes a second distance that is shorter than the first distance. In other words, the fine movement processing may include processing for moving the end effector 5, which has approached the target object OBJ by the rough movement processing, so that it approaches even closer to the target object OBJ.
[0083] For example, the fine movement process may include a process of moving the end effector 5 so that it is located at a target processing position TPP where the end effector 5 should be located when performing a predetermined process on the target object OBJ. On the other hand, the rough movement process may include a process of moving the end effector 5 so that it is located at a target movement position TMP that is different from the target processing position TPP. The rough movement process may include a process of moving the end effector 5 so that it is located at a target movement position TMP that is away from the target processing position TPP. Note that the distance from the target object OBJ to the target processing position TPP is shorter than the distance from the target object OBJ to the target movement position TMP.
[0084] As a first example, as described above, the end effector 5 may perform a holding process, which is an example of a predetermined process, on the target object OBJ in order to hold the target object OBJ. In this case, the fine movement process may include a process of moving the end effector 5 so that the end effector 5 is located at a target processing position TPP where the end effector 5 should be located when the end effector 5 holds the target object OBJ. Note that when the holding process is performed, the target processing position TPP may also be referred to as a target holding position. On the other hand, the rough movement process may include a process of moving the end effector 5 so that the end effector 5 is located at a target movement position TMP that is different from the target processing position TPP.
[0085] As a second example, as described above, the end effector 5 may perform a release process, which is an example of a predetermined process, to release the held target object OBJ. In this case, the fine movement process may include a process of moving the end effector 5 so that it is positioned at a target processing position TPP where the end effector 5 should be positioned when the end effector 5 releases the target object OBJ. Note that when the release process is performed, the target processing position TPP may also be referred to as a target release position. On the other hand, the rough movement process may include a process of moving the end effector 5 so that it is positioned at a target movement position TMP that is different from the target processing position TPP.
[0086] If the rough movement processing is started before the fine movement processing is started, the robot control device 4 may perform the rough movement processing to roughly adjust the position of the end effector 5 relative to the target object OBJ. On the other hand, if the fine movement processing is started after the rough movement processing is started, the control device 3 may perform the fine movement processing to finely adjust the position of the end effector 5 relative to the target object OBJ. In this case, the movement distance of the end effector 5 by the rough movement processing may be longer than the movement distance of the end effector 5 by the fine movement processing. The movement distance of the end effector 5 by the fine movement processing may be shorter than the movement distance of the end effector 5 by the rough movement processing. The movement speed of the end effector 5 by the rough movement processing may be faster than the movement speed of the end effector 5 by the fine movement processing. The movement speed of the end effector 5 by the fine movement processing may be slower than the movement speed of the end effector 5 by the rough movement processing. The movement speed of the end effector 5 referred to here may mean the maximum movement speed, the average movement speed, or any other movement speed.
[0087] As described above, in this embodiment, the robot system SYS performs robot control processing using the control device 3 and the robot control device 4. Therefore, an apparatus including the control device 3 and the robot control device 4 may be referred to as a control system performing robot control processing. However, an apparatus including the control device 3 but not the robot control device 4 may also be referred to as a control system performing robot control processing. An apparatus including the robot control device 4 but not the control device 3 may also be referred to as a control system performing robot control processing. Furthermore, since the imaging unit 2 (particularly the imaging device 21) is used for the fine movement processing as described above, an apparatus including the imaging unit 2 and at least one of the control device 3 and the robot control device 4 may also be referred to as a control system performing robot control processing. An apparatus including the imaging unit 2 and at least one of the control device 3 and the robot control device 4 may also be referred to as an imaging system.
[0088] 3 to 6, an example of the operation of the robot 1 controlled by the robot control process including the rough movement process and the fine movement process will be described. Specifically, an example of the movement of the end effector 5 by the robot control process including the rough movement process and the fine movement process will be described.
[0089] Each of Figures 3(a) to 3(d) shows a first example of the operation of the robot 1 controlled by a robot control process including a rough movement process and a fine movement process. Specifically, each of Figures 3(a) to 3(d) shows an example of the operation of the robot 1 for holding (i.e., picking up) a workpiece W stored in a storage box CB, which is an example of a mounting device T, using the end effector 5. That is, each of Figures 3(a) to 3(d) shows an example of the operation of the robot 1 when the end effector 5 performs a holding process.
[0090] 3(a) to 3(d) show an example in which a single workpiece W is accommodated in the storage box CB. However, multiple workpieces W may be accommodated in the storage box CB. In this case, the multiple workpieces W may be accommodated regularly in the storage box CB. For example, the multiple workpieces W may be located at regular positions in the storage box CB. For example, the multiple workpieces W may be located at predetermined regular positions in the storage box CB. The multiple workpieces W having regular orientations may be located in the storage box CB. Alternatively, the multiple workpieces W may be accommodated irregularly in the storage box CB. For example, the multiple workpieces W may be located at irregular positions in the storage box CB. The multiple workpieces W having irregular orientations may be located in the storage box CB. The storage box CB may also be referred to as a container.
[0091] Note that a state in which multiple workpieces W are irregularly stored in the storage box CB may be considered equivalent to a state in which multiple workpieces W are randomly or carelessly stored in the storage box CB. When multiple workpieces W are irregularly stored in the storage box CB, the orientation of a first workpiece W among the multiple workpieces W may be different from the orientation of a second workpiece W among the multiple workpieces W that is different from the first workpiece W. When multiple workpieces W are irregularly stored in the storage box CB, the posture of the first workpiece W may be different from the posture of the second workpiece W. When multiple workpieces W are irregularly stored in the storage box CB, a third workpiece W different from the first workpiece W may overlap the first workpiece W, while a fourth workpiece W different from the second workpiece W may not overlap the second workpiece W. When multiple workpieces W are stored irregularly in the storage box CB, a third workpiece W may overlap a first workpiece W, a fourth workpiece W may overlap a second workpiece W, and the size of the area where the first workpiece W and the third workpiece W overlap may be different from the size of the area where the second workpiece W and the fourth workpiece W overlap.
[0092] In this case, as shown in Fig. 3(a), the robot control device 4 may first perform a rough movement process. Specifically, the robot control device 4 may perform a rough movement process to move the end effector 5 so that the end effector 5 is positioned at a target movement position TMP#1. In the example shown in Fig. 3(a), a position directly above the workpiece W is used as the target movement position TMP#1.
[0093] In this embodiment, the position of the end effector 5 may refer to any position determined based on the end effector 5. An example of the position of the end effector 5 is the tool center point of the end effector 5. The tool center point of the end effector 5 may refer to the point at which the end effector 5 contacts the target object OBJ (e.g., the workpiece W). As described above, if the end effector 5 is a hand clipper, the tool center point may refer to a point in the space between the tips of multiple jaw members of the hand clipper. As described above, if the end effector 5 is a vacuum clipper, the tool center point may refer to a point in the space between multiple suction ports formed on the vacuum clipper, each capable of sucking gas. Such a tool center point may be set by the user of the robot system SYS. Alternatively, the tool center point may be set in advance for each end effector 5. The tool center point is not limited to the point at which the end effector 5 contacts the target object OBJ, but may be any point on the end effector 5. Alternatively, the position of the end effector 5 may refer to a position determined based on a global coordinate system (to be described later) or a robot coordinate system (to be described later).
[0094] After the end effector 5 is positioned at the target movement position TMP#1, the control device 3 may perform a fine movement process as shown in Fig. 3(b). Specifically, the control device 3 may perform a fine movement process to move the end effector 5 so that the end effector 5 is positioned at the target processing position TPP#1. The target processing position TPP#1 is the target processing position TPP where the end effector 5 should be positioned when the end effector 5 holds the workpiece W.
[0095] After the end effector 5 is positioned at the target processing position TPP#1, the control device 3 (or the robot control device 4) may generate a control signal for controlling the end effector 5 so that the end effector 5 holds the workpiece W, and output the generated control signal to the end effector 5. As a result, as shown in FIG. 3( c), the end effector 5 holds the workpiece W under the control of the control device 3 (or under the control of the robot control device 4).
[0096] Thereafter, the robot control device 4 may perform a rough movement process so that the end effector 5 holding the workpiece W is retracted. In other words, the robot control device 4 may perform the rough movement process by retracting the end effector 5 holding the workpiece W. As an example, the robot control device 4 may perform the rough movement process so that the end effector 5 moves from the target processing position TPP#1 to the target movement position TMP#1 along the same movement path as the movement path along which the end effector 5 moved from the target movement position TMP#1 to the target processing position TPP#1. Thereafter, as shown in FIG. 3( d ), if necessary, the robot control device 4 may perform a rough movement process so that the end effector 5 holding the workpiece W is retracted.
[0097] Next, each of Figures 4(a) to 4(d) shows a second example of the operation of the robot 1 controlled by the robot control process including the rough movement process and the fine movement process. Specifically, each of Figures 4(a) to 4(d) shows an example of the operation of the robot 1 that releases the workpiece W held by the end effector 5 so as to place the workpiece W held by the end effector 5 on a pallet PLT, which is an example of a placement device T. In other words, each of Figures 4(a) to 4(d) shows an example of the operation of the robot 1 when the end effector 5 performs the release process.
[0098] In this case, as shown in Fig. 4(a), the robot control device 4 may first perform a rough movement process. Specifically, the robot control device 4 may perform a rough movement process to move the end effector 5 so that the end effector 5 is positioned at a target movement position TMP#2. In the example shown in Fig. 4(a), a position directly above the pallet PLT is used as the target movement position TMP#2.
[0099] After the end effector 5 is positioned at the target movement position TMP#2, the control device 3 may perform a fine movement process as shown in FIG. 4( b). Specifically, the control device 3 may perform a fine movement process to move the end effector 5 so that the end effector 5 is positioned at the target processing position TPP#2. The target processing position TPP#2 is the target processing position TPP where the end effector 5 should be positioned when the end effector 5 releases the workpiece W. Note that, when the pallet PLT moves as described above, the target processing position TPP#2 may also move in accordance with the movement of the pallet PLT#2.
[0100] After the end effector 5 is positioned at the target processing position TPP#2, the control device 3 (or the robot control device 4) may generate a control signal for controlling the end effector 5 so that the end effector 5 releases the workpiece W, and output the generated control signal to the end effector 5. As a result, as shown in FIG. 4( c), the end effector 5 releases the workpiece W under the control of the control device 3 (or under the control of the robot control device 4).
[0101] Thereafter, as shown in FIG. 4( c), if necessary, the control device 3 may perform a fine movement process to retract the end effector 5 that has released the workpiece W. Alternatively, the robot control device 4 may perform a rough movement process to retract the end effector 5 that has released the workpiece W. In other words, the robot control device 4 may perform the rough movement process to retract the end effector 5 that has released the workpiece W. As an example, the robot control device 4 may perform a rough movement process to move the end effector 5 from the target processing position TPP#2 to the target processing position TMP#2 along the same movement path along which the end effector 5 moved from the target movement position TMP#2 to the target processing position TPP#2. Thereafter, as shown in FIG. 4( d), if necessary, the robot control device 4 may perform a rough movement process to retract the end effector 5 that has released the workpiece W.
[0102] Next, Figures 5(a) to 5(d) and Figures 6(a) to 6(b) each show a third example of the operation of the robot 1 controlled by a robot control process including a rough movement process and a fine movement process. Specifically, Figures 5(a) to 5(d) and Figures 6(a) to 6(b) each show an example of the operation of the robot 1 in which the end effector 5, which houses multiple rod-shaped first workpieces W1, sequentially releases the multiple first workpieces W1 so that the multiple first workpieces W1 are fitted (inserted) into multiple holes HL formed in the second workpiece W2. In other words, Figures 5(a) to 5(d) and Figures 6(a) to 6(b) each show an example of the operation of the robot 1 when the end effector 5 performs a release process.
[0103] In the examples shown in FIGS. 5( a) to 5(d) and 6(a) and 6(b), the end effector 5 performs a release process on each of the multiple holes HL formed in the second workpiece W2. Specifically, the second workpiece W2 has multiple target portions (processing target portions) each having a hole HL formed therein, and the end effector 5 performs a release process on each of the multiple target portions of the second workpiece W2 each having a hole HL formed therein. Note that this is not limited to the case where the first workpiece W1 is fitted into the holes HL formed in the second workpiece W2. The target object OBJ may have multiple target portions on which the end effector 5 performs a predetermined process, and the end effector 5 may perform the predetermined process on the multiple target portions in sequence. Note that the target portions (processing target portions) may also be referred to as target members (processing target members).
[0104] In the following explanation, as shown in Figures 5(a) to 5(d) and Figures 6(a) to 6(b), a plurality of holes HL including holes HL#1 and HL#2 are formed in the second workpiece W2, and the operation of the robot 1 will be described, in which the end effector 5 sequentially releases a plurality of first workpieces W1, including first workpieces W1#1 and W1#2, so that the plurality of first workpieces W1 are fitted (inserted) into the plurality of holes HL including holes HL#1 and HL#2.
[0105] In this case, as shown in FIG. 5A, the robot control device 4 may first perform a rough movement process. Specifically, the robot control device 4 may perform a rough movement process to move the end effector 5 so that the end effector 5 is positioned at a target movement position TMP#31. In the example shown in FIG. 5A, a position near a position directly above the hole HL#1 is used as the target movement position TMP#31. However, the target movement position TMP#31 may be a position directly above the hole HL#1 or another position.
[0106] After the end effector 5 is positioned at the target movement position TMP#31, the control device 3 may perform a fine movement process as shown in Fig. 5(b). Specifically, the control device 3 may perform a fine movement process to move the end effector 5 so that the end effector 5 is positioned at the target processing position TPP#31. The target processing position TPP#31 is the target processing position TPP where the end effector 5 should be positioned when the end effector 5 releases the first workpiece W1#1 in order to fit the first workpiece W1#1 into the hole HL#1.
[0107] After the end effector 5 is positioned at the target processing position TPP#31, the control device 3 (or the robot control device 4) may generate a control signal for controlling the end effector 5 so that the end effector 5 releases the first workpiece W1#1, and output the generated control signal to the end effector 5. As a result, as shown in FIG. 5(c), the end effector 5 releases the first workpiece W1#1 under the control of the control device 3 (or under the control of the robot control device 4). As a result, as shown in FIG. 5(c), the first workpiece W1#1 released by the end effector 5 is fitted (inserted) into the hole HL#1.
[0108] Thereafter, as shown in FIG. 5( d ), the robot control device 4 may perform the rough movement process again. Specifically, the robot control device 4 may perform the rough movement process to move the end effector 5 so that the end effector 5 is positioned at the target movement position TMP#32. In the example shown in FIG. 5( d ), the target movement position TMP#32 is a position near the position directly above the hole HL#2. However, the target movement position TMP#31 may be a position directly above the hole HL#1 or another position.
[0109] 6A, after the end effector 5 is positioned at the target movement position TMP#32, the control device 3 may perform the fine movement process again. Specifically, the control device 3 may perform the fine movement process to move the end effector 5 so that the end effector 5 is positioned at the target processing position TPP#32. The target processing position TPP#32 is the target processing position TPP where the end effector 5 should be positioned when the end effector 5 releases the first workpiece W1#2 to fit the first workpiece W1#2 into the hole HL#2.
[0110] After the end effector 5 is positioned at the target processing position TPP#32, the control device 3 (or the robot control device 4) may generate a control signal for controlling the end effector 5 so that the end effector 5 releases the first workpiece W1#2, and output the generated control signal to the end effector 5. As a result, as shown in FIG. 6(b), the end effector 5 releases the first workpiece W1#2 under the control of the control device 3 (or under the control of the robot control device 4). As a result, as shown in FIG. 6(b), the first workpiece W1#2 released by the end effector 5 is fitted (inserted) into the hole HL#2.
[0111] Thereafter, the control device 3 and the robot control device 4 repeat the same operations. As a result, the plurality of first workpieces W1 are fitted (inserted) into the plurality of holes HL formed in the second workpiece W2.
[0112] (1-2) Configuration of the Control Device 3 Next, the configuration of the control device 3 will be described with reference to Fig. 7. Fig. 7 is a block diagram showing the configuration of the control device 3.
[0113] 7 , the control device 3 includes a calculation device 31, a storage device 32, and a communication device 33. The control device 3 may further include an input device 34 and an output device 35. However, the control device 3 does not necessarily have to include at least one of the input device 34 and the output device 35. The calculation device 31, the storage device 32, the communication device 33, the input device 34, and the output device 35 may be connected via a data bus 36.
[0114] The arithmetic device 31 includes, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a field programmable gate array (FPGA). The arithmetic device 31 reads a computer program. For example, the arithmetic device 31 may read a computer program stored in the storage device 32. For example, the arithmetic device 31 may read a computer program stored in a computer-readable, non-transitory storage medium using a storage medium reading device (not shown) included in the control device 3. The arithmetic device 31 may acquire (i.e., download or read) the computer program from a device (not shown) located outside the control device 3 via the communication device 33 (or another communication device). The arithmetic device 31 executes the read computer program. As a result, logical function blocks for executing the processes to be performed by the control device 3 (for example, the robot control process including the fine movement process described above) are realized within the arithmetic device 31. In other words, the arithmetic device 31 can function as a controller for realizing logical function blocks for executing the processes to be performed by the control device 3.
[0115] A computational model that can be constructed by machine learning may be implemented in the computational device 31 by the computational device executing a computer program. An example of a computational model that can be constructed by machine learning is a computational model including a neural network (so-called artificial intelligence (AI)). In this case, learning of the computational model may include learning of parameters of the neural network (e.g., at least one of a weight and a bias). The computational device 31 may execute a robot control process using the computational model. In other words, the operation of executing the robot control process may include the operation of executing the robot control process using the computational model. Note that a computational model that has been constructed by offline machine learning using teacher data may be implemented in the computational device 31. Furthermore, the computational model implemented in the computational device 31 may be updated by online machine learning on the computational device 31. Alternatively, the calculation device 31 may perform robot control processing using a calculation model implemented in a device external to the calculation device 31 (i.e., a device provided outside the control device 3) in addition to or instead of the calculation model implemented in the calculation device 31.
[0116] The recording medium for recording the computer program executed by the arithmetic unit 31 may be at least one of the following: a CD-ROM, CD-R, CD-RW, a flexible disk, an MO, a DVD-ROM, a DVD-RAM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, and an optical disk such as Blu-ray (registered trademark), a magnetic medium such as a magnetic tape, a magneto-optical disk, a semiconductor memory such as a USB memory, and any other medium capable of storing a program. The recording medium may also include a device capable of recording a computer program (for example, a general-purpose device or a dedicated device in which a computer program is implemented in an executable state in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program may be realized by a logical processing block realized within the arithmetic device 31 (i.e., a computer) when the arithmetic device 31 executes the computer program, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) included in the arithmetic device 31, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.
[0117] 7 shows an example of logical functional blocks implemented in the arithmetic device 31 for executing robot control processing. As shown in FIG. 7, a position and orientation calculation unit 311 and a signal generation unit 312 are implemented in the arithmetic device 31. The position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ based on the image data IMG. The signal generation unit 312 may generate a control signal for controlling the robot 1 based on at least one of the position and orientation of the target object OBJ calculated by the position and orientation calculation unit 311.
[0118] The storage device 32 can store desired data. For example, the storage device 32 may temporarily store a computer program executed by the arithmetic device 31. The storage device 32 may temporarily store data that the arithmetic device 31 temporarily uses when the arithmetic device 31 is executing a computer program. The storage device 32 may also store data that the control device 3 stores long-term. The storage device 32 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 32 may include a non-temporary recording medium.
[0119] The communication device 33 can communicate with each of the imaging unit 2 and the robot control device 4 via a communication network (not shown). Alternatively, the communication device 33 may be able to communicate with devices other than the imaging unit 2 and the robot control device 4, in addition to or instead of at least one of the imaging unit 2 and the robot control device 4, via a communication network (not shown). In this embodiment, the communication device 33 may receive (i.e., acquire) the image data IMG and IMG from the imaging unit 2. Furthermore, the communication device 33 may transmit (i.e., output) a control signal generated by the signal generation unit 312 to the robot control device 4. Furthermore, if the control device 3 generates a signal that can be used as is to control the operation of the robot 1 as a control signal, the communication device 33 may transmit (i.e., output) the control signal generated by the signal generation unit 312 to the robot 1. Note that the communication device 33 that outputs the control signal to the robot 1 or the robot control device 4 may be referred to as an output unit.
[0120] The input device 34 is a device that accepts information input to the control device 3 from outside the control device 3. For example, the input device 34 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by a user of the control device 3. For example, the input device 34 may include a recording medium reading device that can read information recorded as data on a recording medium that can be externally attached to the control device 3.
[0121] It should be noted that information can be input as data to the control device 3 from a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an input device that accepts information input to the control device 3 from outside the control device 3.
[0122] The output device 35 is a device that outputs information to the outside of the control device 3. For example, the output device 35 may output information as an image. That is, the output device 35 may include a display device (a so-called display) that can display an image. For example, the output device 35 may output information as sound. That is, the output device 35 may include an audio device (a so-called speaker) that can output sound. For example, the output device 35 may output information on paper. That is, the output device 35 may include a printing device (a so-called printer) that can print desired information on paper. For example, the output device 35 may output information as data to a recording medium that can be externally attached to the control device 3.
[0123] The control device 3 can output information as data to a device external to the control device 3 via the communication device 33. In this case, the communication device 33 may function as an output device that outputs information to a device external to the control device 3.
[0124] (1-3) Configuration of the Robot Control Device 4 Next, the configuration of the robot control device 4 will be described with reference to Fig. 8. Fig. 8 is a block diagram showing the configuration of the robot control device 4.
[0125] 8 , the robot control device 4 includes a calculation device 41, a storage device 42, and a communication device 43. The robot control device 4 may further include an input device 44 and an output device 45. However, the robot control device 4 does not necessarily have to include at least one of the input device 44 and the output device 45. The calculation device 41, the storage device 42, the communication device 43, the input device 44, and the output device 45 may be connected via a data bus 46.
[0126] The arithmetic device 41 includes, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a field programmable gate array (FPGA). The arithmetic device 41 reads a computer program. For example, the arithmetic device 41 may read a computer program stored in the storage device 42. For example, the arithmetic device 41 may read a computer program stored in a computer-readable, non-transitory storage medium using a storage medium reading device (not shown) included in the robot control device 4. The arithmetic device 41 may acquire (i.e., download or read) the computer program from a device (not shown) located outside the robot control device 4 via the communication device 43 (or another communication device). The arithmetic device 41 executes the read computer program. As a result, logical function blocks for executing the processes to be performed by the robot control device 4 (for example, robot control processes including the rough movement process described above) are realized within the arithmetic device 41. In other words, the arithmetic device 41 can function as a controller for realizing logical function blocks for executing the processes to be performed by the robot control device 4.
[0127] A computational model that can be constructed by machine learning may be implemented in the computational device 41 by the computational device executing a computer program. An example of a computational model that can be constructed by machine learning is a computational model including a neural network (so-called artificial intelligence (AI)). In this case, learning of the computational model may include learning of parameters of the neural network (e.g., at least one of a weight and a bias). The computational device 41 may execute a robot control process using the computational model. In other words, the operation of executing the robot control process may include the operation of executing the robot control process using the computational model. Note that a computational model that has been constructed by offline machine learning using teacher data may be implemented in the computational device 41. Furthermore, the computational model implemented in the computational device 41 may be updated by online machine learning on the computational device 41. Alternatively, the computing device 41 may execute the robot control process using a computational model implemented in a device external to the computing device 41 (i.e., a device provided outside the robot control device 4) in addition to or instead of the computational model implemented in the computing device 41.
[0128] The recording medium for recording the computer program executed by the arithmetic unit 41 may be at least one of the following: a CD-ROM, CD-R, CD-RW, a flexible disk, an MO, a DVD-ROM, a DVD-RAM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, and an optical disk such as Blu-ray (registered trademark), a magnetic medium such as a magnetic tape, a magneto-optical disk, a semiconductor memory such as a USB memory, and any other medium capable of storing a program. The recording medium may also include a device capable of recording a computer program (for example, a general-purpose device or a dedicated device in which a computer program is implemented in an executable state in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program may be realized by a logical processing block realized within the arithmetic device 41 (i.e., a computer) when the arithmetic device 41 executes the computer program, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) included in the arithmetic device 41, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.
[0129] 8 shows an example of logical functional blocks implemented in the arithmetic device 41 for executing robot control processing. As shown in Fig. 8, a signal generation unit 411 is implemented in the arithmetic device 41. The signal generation unit 411 may generate a control signal for controlling the robot 1 without using the image data IMG generated by the imaging device 21.
[0130] The storage device 42 can store desired data. For example, the storage device 42 may temporarily store a computer program executed by the arithmetic device 41. The storage device 42 may temporarily store data that the arithmetic device 41 temporarily uses when the arithmetic device 41 is executing a computer program. The storage device 42 may also store data that the robot control device 4 stores long-term. The storage device 42 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 42 may include a non-temporary recording medium.
[0131] The communication device 43 is capable of communicating with both the robot 1 and the control device 3 via a communication network (not shown). Alternatively, the communication device 43 may be capable of communicating with another device different from the control device 3 in addition to or instead of at least one of the robot 1 and the control device 3 via a communication network (not shown). In this embodiment, the communication device 43 may receive (i.e., acquire) a control signal generated by the control device 3 from the control device 3. Furthermore, the communication device 43 may transmit (i.e., output) a control signal generated by the signal generation unit 411 to the robot 1 (particularly, at least one of the robot drive signal and the end effector drive signal described above). The communication device 43 that outputs the control signal to the robot 1 may be referred to as an output unit.
[0132] The input device 44 is a device that accepts information input to the robot control device 4 from outside the robot control device 4. For example, the input device 44 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by a user of the robot control device 4. For example, the input device 44 may include a recording medium reader that can read information recorded as data on a recording medium that can be externally attached to the robot control device 4.
[0133] It should be noted that information can be input as data to the robot control device 4 from a device external to the robot control device 4 via the communication device 43. In this case, the communication device 43 may function as an input device that accepts information input to the robot control device 4 from outside the robot control device 4.
[0134] The output device 45 is a device that outputs information to the outside of the robot control device 4. For example, the output device 45 may output information as an image. That is, the output device 45 may include a display device (a so-called display) that can display an image. For example, the output device 45 may output information as sound. That is, the output device 45 may include an audio device (a so-called speaker) that can output sound. For example, the output device 45 may output information on paper. That is, the output device 45 may include a printing device (a so-called printer) that can print desired information on paper. For example, the output device 45 may output information as data to a recording medium that can be externally attached to the robot control device 4.
[0135] The robot control device 4 can output information as data to a device external to the robot control device 4 via the communication device 43. In this case, the communication device 43 may function as an output device that outputs information to a device external to the robot control device 4.
[0136] (2) Robot Control Processing Next, the flow of the robot control processing will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the flow of the robot control processing.
[0137] As described above, the rough movement process is performed first, and then the fine movement process is performed to control the robot 1. Therefore, in the following explanation, the rough movement process will be explained first, and then the fine movement process will be explained.
[0138] (2-1) Rough Movement Processing As shown in FIG. 9 , first, the robot control device 4 performs the rough movement processing (step S41). Specifically, as described above, the rough movement processing includes a process of moving the end effector 5 without using image data IMG. For this reason, the signal generation unit 411 included in the robot control device 4 generates control signals (particularly, robot drive signals that can be input to the robot 1 to control the operation of the robot 1) without using image data IMG and outputs the generated control signals to the robot 1. However, as described above, if the control signals used in the rough movement processing are generated in advance before the rough movement processing is performed, the signal generation unit 411 does not need to generate control signals. The signal generation unit 411 may also output control signals generated in advance to the robot 1 before the rough movement processing is started. As a result, the robot 1 moves the end effector 5 in a movement pattern based on the control signals. Note that, when the rough movement processing is performed multiple times, the control signals used in at least one of the multiple rough movement processing operations may be generated in advance.
[0139] In this embodiment, the rough movement processing may include processing for controlling the robot 1 to move the end effector 5 in accordance with movement instruction information without using image data IMG. The movement instruction information includes information instructing the movement mode of the end effector 5. In this case, the signal generation unit 411 generates a control signal (particularly, a robot drive signal that can be input to the robot 1 to control the operation of the robot 1) for controlling the robot 1 to move the end effector 5 in accordance with the movement instruction information, and outputs the generated control signal to the robot 1. As a result, the end effector 5 moves in the movement mode instructed by the movement instruction information.
[0140] The movement mode of the end effector 5 in the rough movement processing may be defined by a computer program executed by the robot control device 4 to perform the rough movement processing. In this case, the movement instruction information may be considered to be included in at least a part of the computer program executed by the robot control device 4 to perform the rough movement processing. For example, the movement instruction information may be considered to be included in a part of the program portion (program information) that describes the movement mode of the end effector 5, of the computer program executed by the robot control device 4 to perform the rough movement processing.
[0141] The movement mode of the end effector 5 by the rough movement processing may include a position to which the end effector 5 should be moved by the rough movement processing (a target movement position TMP shown in FIG. 3 etc.). In this case, the movement instruction information may include the target movement position TMP (see FIG. 3 etc.) to which the end effector 5 should be moved by the rough movement processing. In other words, the movement instruction information may include information indicating the target movement position TMP. As described above, when the movement instruction information is included in a computer program, the information indicating the target movement position TMP may be considered to be included in a part of the program portion (program information) of the computer program in which the target movement position TMP is described.
[0142] The movement mode of the end effector 5 by the rough movement processing may include a target movement distance, which is the distance the end effector 5 should move by the rough movement processing. In this case, the movement instruction information may include the target movement distance that the end effector 5 should move by the rough movement processing. In other words, the movement instruction information may include information that indicates the target movement distance. As described above, when the movement instruction information is included in a computer program, the information that indicates the target movement distance may be considered to be included in a part of the program portion (program information) of the computer program in which the target movement distance is described.
[0143] The movement mode of the end effector 5 by the rough movement processing may include a target movement direction, which is the direction in which the end effector 5 should move by the rough movement processing. In this case, the movement instruction information may include the target movement direction in which the end effector 5 should move by the rough movement processing. In other words, the movement instruction information may include information that indicates the target movement direction. As described above, when the movement instruction information is included in a computer program, the information that indicates the target movement direction may be considered to be included in a part of the program portion (program information) of the computer program in which the target movement direction is described.
[0144] The movement mode of the end effector 5 by the rough movement processing may include a target movement posture, which is the posture of the end effector 5 after movement by the rough movement processing. In this case, the movement instruction information may include the target movement posture. In other words, the movement instruction information may include information instructing the target movement posture. As described above, when the movement instruction information is included in a computer program, the information instructing the target movement posture may be considered to be included in a part of the program portion (program information) of the computer program in which the target movement posture is described.
[0145] The movement instruction information may be set by the user of the robot system SYS. That is, the movement instruction information may be input by the user of the robot system SYS to the robot control device 4. In other words, the movement instruction information may be set by the user of the robot system SYS inputting the movement instruction information.
[0146] However, an information setting device capable of automatically setting movement instruction information may automatically set the movement instruction information. That is, in addition to or instead of a user setting the movement instruction information, the information setting device may automatically set the movement instruction information. For example, a robot control device 4 (particularly, the calculation device 41) capable of functioning as an information setting device may automatically set the movement instruction information. For example, an information setting device different from the robot control device 4 may automatically set the movement instruction information. Note that the information setting device may set the movement instruction information using a calculation model capable of outputting the movement instruction information. The calculation model capable of outputting the movement instruction information may include a rule-based calculation model. The calculation model capable of outputting the movement instruction information may include a calculation model capable of learning by machine learning (so-called artificial intelligence).
[0147] The user may set the movement instruction information by directly setting the movement instruction information. For example, the robot control device 4 may use an output device 35 capable of functioning as a display device to display a GUI (Graphical User Interface) that the user can operate to set the movement instruction information, and the user may set the movement instruction information on the GUI displayed by the output device 35. The GUI may include at least one of a GUI that the user can operate to set the target movement position TMP, a GUI that the user can operate to set the target movement distance, a GUI that the user can operate to set the target movement direction, and a GUI that the user can operate to set the target movement posture. In this case, the movement instruction information directly set by the user may be reflected in a computer program executed by the robot control device 4 to perform the rough movement process. For example, the movement instruction information directly set by the user may be reflected in a registry that stores information regarding the settings of a computer program executed by the robot control device 4 to perform the rough movement process.
[0148] Alternatively, in addition to directly setting the movement instruction information, the user may indirectly set the movement instruction information by creating at least a part of a computer program executed by the robot control device 4 to perform the rough movement processing. For example, the user may indirectly set the movement instruction information by creating a computer program that is executed by the robot control device 4 to perform the rough movement processing and that reflects the movement instruction information that the user wants to set.
[0149] Alternatively, the user may set the movement instruction information by robot teaching. For example, the user may set the movement instruction information by using a teaching pendant to perform robot teaching to actually move the robot 1 in a movement pattern that should be achieved by rough movement processing. For example, the user may set the movement instruction information by performing offline teaching without using the robot 1.
[0150] As described above, since the fine movement process using the image data IMG is performed after the rough movement process, it is desirable that the target object OBJ, on which the end effector 5 is to perform a predetermined process, be included within the imaging field of view of the imaging device 21 when the rough movement process is completed. Therefore, the movement instruction information may be set to satisfy the condition that "the target object OBJ, on which the end effector 5 is to perform a predetermined process, is included within the imaging field of view of the imaging device 21 when the rough movement process is completed." For example, the target movement position TMP indicated by the movement instruction information may be set to satisfy the condition that "when the end effector 5 moves to the target movement position TMP, the target object OBJ is included within the imaging field of view of the imaging device 21, which moves together with the end effector 5." For example, the target movement distance indicated by the movement instruction information may be set to satisfy the condition that "when the end effector 5 moves by the target movement distance, the target object OBJ is included within the imaging field of view of the imaging device 21, which moves together with the end effector 5." For example, the target movement direction indicated by the movement instruction information may be set so as to satisfy the condition that "when the end effector 5 moves toward the target movement direction, the target object OBJ is included in the imaging field of view of the imaging device 21 moving together with the end effector 5." For example, the target movement posture indicated by the movement instruction information may be set so as to satisfy the condition that "when the posture of the end effector 5 becomes the target movement posture, the target object OBJ is included in the imaging field of view of the imaging device 21 moving together with the end effector 5."
[0151] As described above, when a target object OBJ, on which the end effector 5 performs a predetermined process, is moved by a transport device capable of transporting the target object OBJ, the movement instruction information may be set based on the transport mode of the target object OBJ. For example, as described above, the rough movement process may include a process of moving the end effector 5 so that the end effector 5 approaches the target object OBJ. In this case, the movement mode of moving the end effector 5 so that the end effector 5 approaches the target object OBJ when the target object OBJ is not moving (not being transported) is likely to be different from the movement mode of moving the end effector 5 so that the end effector 5 approaches the target object OBJ when the target object OBJ is moving (being transported). Similarly, the movement mode of moving the end effector 5 so that the end effector 5 approaches the target object OBJ when the target object OBJ is moving in a first transport mode is likely to be different from the movement mode of moving the end effector 5 so that the end effector 5 approaches the target object OBJ when the target object OBJ is moving (being transported) in a second transport mode different from the first transport mode. For this reason, the movement instruction information may be set based on the transport mode of the target object OBJ so that the end effector 5 appropriately approaches the target object OBJ being transported in any transport mode (for example, as described above, the end effector 5 is positioned at the target movement position TMP determined based on the target object OBJ being transported in any transport mode). As a result, regardless of differences in the transport mode of the target object OBJ, the robot control device 4 can move the end effector 5 by performing rough movement processing so that the end effector 5 appropriately approaches the target object OBJ. The transport mode of the target object OBJ by the transport device may also be referred to as the movement mode of the target object OBJ.
[0152] The transport mode may include a transport speed. In this case, the movement instruction information may be set based on the transport speed of the target object OBJ so that the end effector 5 appropriately approaches the target object OBJ being transported at a given transport speed (for example, as described above, the end effector 5 is positioned at a target movement position TMP determined based on the target object OBJ being transported at a given transport speed). As an example, when the transport speed of the target object OBJ is a first transport speed, the movement instruction information may be set so that the end effector 5 moves at the first movement speed. As another example, when the transport speed of the target object OBJ is a second transport speed different from the first transport speed, the movement instruction information may be set so that the end effector 5 moves at a second movement speed different from the first movement speed. For example, when the transport speed of the target object OBJ is a second transport speed faster than the first transport speed, the movement instruction information may be set so that the end effector 5 moves at a second movement speed faster than the first movement speed. For example, when the transport speed of the target object OBJ is a second transport speed that is slower than the first transport speed, the movement instruction information may be set so that the end effector 5 moves at the second transport speed that is slower than the first transport speed. Note that the transport speed may also be referred to as the movement speed of the target object OBJ. In other words, the transport mode may include the movement speed.
[0153] The transport mode may include the transport direction. In this case, the movement instruction information may be set based on the transport direction of the target object OBJ so that the end effector 5 appropriately approaches the target object OBJ being transported in an arbitrary transport direction (for example, as described above, the end effector 5 is positioned at the target movement position TMP determined based on the target object OBJ being transported in an arbitrary transport direction). As an example, if the transport direction of the target object OBJ is a first transport direction, the movement instruction information may be set so that the end effector 5 moves in the first movement direction. As another example, if the transport speed of the target object OBJ is a second transport direction different from the first transport direction, the movement instruction information may be set so that the end effector 5 moves in a second movement direction different from the first movement direction. For example, the movement instruction information may be set so that the end effector 5 moves in a movement direction that includes a directional component along the transport direction of the target object OBJ. For example, the movement instruction information may be set so that the end effector 5 moves in the same movement direction as the transport direction of the target object OBJ. For example, the movement instruction information may be set so that the end effector 5 moves in a movement direction that diagonally intersects with the transport direction of the target object OBJ. Note that the transport direction may also be referred to as the movement direction of the target object OBJ. In other words, the transport mode may include the movement direction.
[0154] When the end effector 5 sequentially performs predetermined processing on a plurality of different target objects OBJ, the movement instruction information may be set based on the positional relationship (e.g., relative positional relationship) between at least two of the plurality of different target objects OBJ. As an example, when the end effector 5 sequentially performs predetermined processing on a second target object OBJ different from the first target object OBJ and the second target object OBJ, the movement instruction information may be set based on the positional relationship between the first and second target objects OBJ. For example, the movement instruction information may be set so that the end effector 5 moves from the first target object OBJ toward the second target object OBJ by a rough movement process performed after the end effector 5 performs predetermined processing on the first target object OBJ. For example, the movement instruction information may be set so that the end effector 5 moves in a movement direction including a directional component connecting the first target object OBJ and the second target object OBJ by a rough movement process performed after the end effector 5 performs predetermined processing on the first target object OBJ. For example, the movement instruction information may be set so that the end effector 5 moves at a movement speed corresponding to the relative speed of the second target object OBJ with respect to the first target object OBJ through a rough movement process performed after the end effector 5 performs a predetermined process on the first target object OBJ. For example, the movement instruction information may be set so that the end effector 5 moves from the first target object OBJ to the second target object OBJ at a movement speed that allows movement within a desired time through a rough movement process performed after the end effector 5 performs a predetermined process on the first target object OBJ. For example, the movement instruction information may be set so that the end effector 5 moves a movement distance corresponding to the distance between the first target object OBJ and the second target object OBJ through a rough movement process performed after the end effector 5 performs a predetermined process on the first target object OBJ. As a result, the robot control device 4 can move the end effector 5 so that the end effector 5 approaches multiple target objects OBJ in sequence by performing the rough movement process.
[0155] As described with reference to FIGS. 5 and 6 , when the end effector 5 sequentially performs predetermined processes on multiple different target portions (multiple different target components) of the target object OBJ, the movement instruction information may be set based on the positional relationship (e.g., relative positional relationship) between at least two of the multiple different target portions. As an example, when the end effector 5 sequentially performs predetermined processes on a first target portion (e.g., the target portion in which the hole HL#1 shown in FIG. 5A is formed) and a second target portion different from the first target portion (e.g., the target portion in which the hole HL#2 shown in FIG. 5A is formed), the movement instruction information may be set based on the positional relationship between the first and second target portions. For example, the movement instruction information may be set so that the end effector 5 moves from the first target portion to the second target portion by a rough movement process performed after the end effector 5 performs the predetermined process on the first target portion. For example, the movement instruction information may be set so that the rough movement process performed after the end effector 5 has performed a predetermined process on the first target portion causes the end effector 5 to move in a movement direction that includes a directional component connecting the first target portion and the second target portion. For example, the movement instruction information may be set so that the rough movement process performed after the end effector 5 has performed a predetermined process on the first target portion causes the end effector 5 to move from the first target portion to the second target portion at a movement speed that allows movement within a desired time. For example, the movement instruction information may be set so that the rough movement process performed after the end effector 5 has performed a predetermined process on the first target portion causes the end effector 5 to move a movement distance that corresponds to the distance between the first target portion and the second target portion. As a result, by performing the rough movement process, the robot control device 4 can move the end effector 5 so that the end effector 5 approaches multiple target portions in sequence.
[0156] The user (or the information setting device described above, the same applies hereinafter in this paragraph) may set the movement instruction information in advance. Specifically, the user may set the movement instruction information in advance before the robot control device 4 starts the rough movement process. In this case, the rough movement process may be considered to be a process of controlling the robot 1 based on the movement instruction information set in advance. For example, the rough movement process may be considered to be a process of controlling the robot 1 based on a computer program set (generated) in advance.
[0157] However, the user (or the information setting device described above; the same applies hereinafter in this paragraph) may set the movement instruction information after the robot control device 4 starts the rough movement process. That is, the user may set (in other words, update or change) the movement instruction information instructing the movement mode of the end effector 5 in the rough movement process in real time after the robot control device 4 starts the rough movement process. For example, the user may acquire information about the position of the target object OBJ during the period when the rough movement process is being performed, and set the movement instruction information based on the acquired information. As an example, in a case where a workpiece W, which is an example of the target object OBJ, is placed on a placement device T capable of functioning as a transport device for transporting (moving) the workpiece W, the user may acquire information about the position of the workpiece W moving (being transported) on the placement device T from a position detection device (e.g., an encoder) provided on the placement device T, and set the movement instruction information based on the acquired information.
[0158] If the movement instruction information is set in advance, the movement instruction information may be fixed during the rough movement process. In other words, the movement instruction information may not be changed during the rough movement process. For example, if the target movement position TMP indicated by the movement instruction information is set in advance, the target movement position TMP may be fixed at the preset position during the rough movement process. For example, if the target movement distance indicated by the movement instruction information is set in advance, the target movement distance may be fixed at the preset distance during the rough movement process. For example, if the target movement direction indicated by the movement instruction information is set in advance, the target movement direction may be fixed at the preset direction during the rough movement process. For example, if the target movement posture indicated by the movement instruction information is set in advance, the target movement posture may be fixed at the preset posture during the rough movement process. However, even if the movement instruction information is set in advance, the movement instruction information may be changed during the rough movement process.
[0159] (2-2) Transmission of Control Permission Signal The robot control device 4 transmits a control permission signal to the control device 3 at a desired timing after starting the rough movement process (step S42). For example, the robot control device 4 transmits a control permission signal to the control device 3 after completing the rough movement process (step S42). In other words, the robot control device 4 transmits a control permission signal to the control device 3 after the end effector 5 has finished moving through the rough movement process. However, the robot control device 4 may also transmit a control permission signal to the control device 3 before completing the rough movement process. The robot control device 4 transmits a control permission signal to the control device 3 before the end effector 5 has finished moving through the rough movement process (step S42).
[0160] After the robot control device 4 transmits a control permission signal to the control device 3, the control device 3 receives the control permission signal transmitted from the robot control device 4. The control device 3 that receives the control permission signal starts the fine movement process (step S32). That is, after the control device 3 receives the control permission signal, the control device 3 starts the fine movement process (step S32). In other words, after the control device 3 receives the control permission signal, the control device 3 moves the end effector 5 by performing the fine movement process (step S32). On the other hand, a control device 3 that does not receive a control permission signal does not start the fine movement process. That is, before the control device 3 receives the control permission signal, the control device 3 does not start the fine movement process. In other words, before the control device 3 receives the control permission signal, the control device 3 does not move the end effector 5 by performing the fine movement process.
[0161] Therefore, the control permission signal may be considered to be a signal by which the robot control device 4 permits the control device 3 to control the robot 1 so as to move the end effector 5 by performing fine movement processing. In this case, the robot control device 4 may transmit the control permission signal to the control device 3 at a desired timing when the robot control device 4 may permit the control device 3 to control the robot 1 so as to move the end effector 5 by performing fine movement processing.
[0162] After the control device 3 receives the control permission signal, the control device 3 transmits the control signal generated by the control device 3 to the robot control device 4 in order to control the robot 1 to move the end effector 5 by the fine movement processing. Thereafter, the robot control device 4 controls the robot 1 to move the end effector 5 based on the control signal generated by the control device 3. As a result, after the control device 3 receives the control permission signal, the control signal generated by the control device 3 is actually reflected in the control of the robot 1. In this case, the control device 3 may be considered to be controlling the robot 1 via the robot control device 4. Alternatively, the control device 3 may transmit the control signal generated by the control device 3 to the robot 1 without going through the robot control device 4 in order to control the robot 1 to move the end effector 5 by the fine movement processing. As a result, even when the control device 3 transmits a control signal to the robot 1, after the control device 3 receives the control permission signal, the control signal generated by the control device 3 is actually reflected in the control of the robot 1. In this case, the control device 3 may be considered to be controlling the robot 1 without going through the robot control device 4. On the other hand, before the control device 3 receives the control permission signal, the control device 3 does not need to generate a control signal. Alternatively, before the control device 3 receives the control permission signal, the control device 3 may generate a control signal but may not transmit the control signal generated by the control device 3 to the robot control device 4. Alternatively, before the control device 3 receives the control permission signal (particularly, before the robot control device 4 transmits the control permission signal), the control device 3 may transmit the control signal generated by the control device 3 to the robot control device 4, but the robot control device 4 may control the robot 1 using the control signal generated by the robot control device 4 without using the control signal generated by the control device 3. Alternatively, before the control device 3 receives the control permission signal (particularly, before the robot control device 4 transmits the control permission signal), the control device 3 may transmit the control signal generated by the control device 3 to the robot 1, but the robot 1 may not use the control signal generated by the control device 3. In other words, before the control device 3 receives the control permission signal, the control signal generated by the control device 3 may not actually be reflected in the control of the robot 1.In this case, the control permission signal may be considered as a signal that the robot control device 4 gives permission to the control device 3 to actually reflect the control signal generated by the control device 3 in the control of the robot 1 .
[0163] After the control device 3 receives the control permission signal, the control signal generated by the control device 3 is actually reflected in the control of the robot 1, and therefore the computer program executed by the robot control device 4 to perform the rough movement processing may be at least partially stopped. In this case, the robot 1 operates to move the end effector 5 under the control of the control device 3. In other words, after the control device 3 receives the control permission signal, it may be considered that the control right of the robot 1 is held by the control device 3. On the other hand, before the control device 3 receives the control permission signal, the robot control device 4 controls the robot 1 by executing the computer program executed to perform the rough movement processing. In this case, the robot 1 operates to move the end effector 5 under the control of the robot control device 4. In other words, it may be considered that the control right of the robot 1 is held by the robot control device 4 before the control device 3 receives the control permission signal. In this case, the control permission signal may be considered as a signal that transfers the control right of the robot 1 from the robot control device 4 to the control device 3.
[0164] After the control device 3 receives the control permission signal, the control device 3 may perform the fine movement process without receiving a signal transmitted from the robot control device 4. In other words, after the control device 3 receives the control permission signal, the control device 3 may perform the fine movement process even if the control device 3 does not receive a signal transmitted from the robot control device 4. After the control device 3 receives the control permission signal, the control device 3 may perform the fine movement process even if no signal is transmitted from the robot control device 4 to the control device 3. This is because, after the control device 3 receives the control permission signal, it can be considered that the control right of the robot 1 is held by the control device 3.
[0165] Note that while the control device 3 has control of the robot 1 (i.e., while the fine movement processing is being performed), the robot control device 4 does not perform the rough movement processing. In other words, while the control device 3 is performing the fine movement processing, the robot control device 4 does not perform the rough movement processing. However, even while the control device 3 has control of the robot 1 (i.e., while the fine movement processing is being performed), the robot control device 4 may perform at least a part of the rough movement processing. In other words, even while the control device 3 is performing the fine movement processing, the robot control device 4 may perform at least a part of the rough movement processing. Similarly, while the robot control device 4 has control of the robot 1 (i.e., while the rough movement processing is being performed), the control device 3 does not perform the fine movement processing. In other words, while the robot control device 4 is performing the rough movement processing, the control device 3 does not perform the fine movement processing. However, even while the robot control device 4 has control of the robot 1 (i.e., while the rough movement processing is being performed), the control device 3 may perform at least a part of the fine movement processing. In other words, even while the robot control device 4 is performing the rough movement processing, the control device 3 may perform at least a part of the fine movement processing. That is, at least a part of the rough movement processing and at least a part of the fine movement processing may be performed in parallel. Note that an example in which at least a part of the rough movement processing and at least a part of the fine movement processing are performed in parallel will be described in the first modified example below.
[0166] As described above, when a control signal generated by the control device 3 performing the fine movement processing is input to the robot control device 4, the robot control device 4 may generate a control signal, based on the control signal generated by the control device 3, as a drive signal that can be input to the robot 1 or the end effector 5 to control the operation of the robot 1 or the end effector 5, and continue the process of outputting the control signal generated by the robot control device 4 to the robot 1 or the end effector 5. In other words, even when the computer program executed by the robot control device 4 to perform the rough movement processing is stopped, the robot control device 4 may continue the process of converting the control signal generated by the control device 3 into a control signal (drive signal) that can be input to the robot 1 or the end effector 5, and outputting the converted control signal to the robot 1 or the end effector 5. However, when the control signal generated by the control device 3 is input to the robot 1 or the end effector 5 without going through the robot control device 4, the robot control device 4 may stop the process of converting the control signal generated by the control device 3 into a control signal (drive signal) that can be input to the robot 1 or the end effector 5, and outputting the converted control signal to the robot 1 or the end effector 5.
[0167] (2-3) Fine Movement Processing Next, the fine movement processing that is performed after the control device 3 receives the control permission signal in step S32 of Fig. 9 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the flow of the fine movement processing.
[0168] In the following description, an example will be described in which the fine movement process is started after the rough movement process is completed. However, as will be described later in the first modified example, the fine movement process may be started after the rough movement process has been started but before the rough movement process is completed. In other words, the control device 3 may perform at least a part of the fine movement process shown in FIG. 10 during at least a part of the period during which the rough movement process is being performed.
[0169] As shown in FIG. 10 , the position and orientation calculation unit 311 included in the control device 3 acquires image data IMG from the imaging device 21 using the communication device 33 (step S321). Specifically, the imaging device 21 captures an image of the target object OBJ at a predetermined imaging rate. For example, the imaging device 21 may capture an image of the target object OBJ at an imaging rate of tens to hundreds (e.g., 30 to 500) times per second. As a result, the imaging device 21 generates image data IMG at a period corresponding to the predetermined imaging rate. For example, the imaging device 21 may generate tens to hundreds (e.g., 30 to 500) pieces of image data IMG per second. The control device 3 acquires the image data IMG each time the imaging device 21 generates image data IMG. In other words, the control device 3 may acquire tens to hundreds (e.g., 30 to 500) pieces of image data IMG per second.
[0170] As described above, when the fine movement process is started after the rough movement process is completed, the image capture device 21 may capture an image of the target object OBJ after the rough movement process is completed. As a result, the position and orientation calculation unit 311 acquires image data IMG generated by the image capture device 21 capturing an image of the target object OBJ after the rough movement process is completed.
[0171] The imaging device 21 may capture an image of an object other than the target object OBJ, in addition to the target object OBJ on which the end effector 5 performs a predetermined process. For example, if both the target object OBJ and the other object are included in the imaging range (field of view) of the imaging device 21, the imaging device 21 may capture images of both the target object OBJ and the other object. As a result, the imaging device 21 may generate image data IMG representing an image in which both the target object OBJ and the other object are captured. Note that, because the other object is not a target on which the robot 1 performs a predetermined process, the other object other than the target object OBJ will be referred to as a non-target object in the following description. Examples of the non-target object include at least one of at least a portion of the end effector 5, at least a portion of the robot arm 12, and a peripheral object located in the vicinity of the robot 1. However, the imaging device 21 may capture an image of the target object OBJ but not the non-target object. That is, the imaging device 21 may generate image data IMG showing an image in which the target object OBJ is captured but no non-target objects are captured. In either case, the imaging device 21 generates image data IMG showing an image in which at least the target object OBJ is captured. That is, the imaging device 21 generates image data IMG including image data of at least the target object OBJ.
[0172] Every time the position and orientation calculation unit 311 acquires image data IMG in step S321, the position and orientation calculation unit 311 calculates at least one of the position and orientation of the target object OBJ based on the image data IMG acquired in step S321 (step S322). As a result, the position and orientation calculation unit 311 generates position and orientation information POI that indicates at least one of the position and orientation of the target object OBJ (step S322).
[0173] In step S322, the position and orientation calculation unit 311 calculates at least one of the position and orientation of the target object OBJ in the global coordinate system. That is, the position and orientation calculation unit 311 generates position and orientation information POI that indicates at least one of the position and orientation of the target object OBJ in the global coordinate system.
[0174] The global coordinate system is a coordinate system that serves as the reference for the robot system SYS. Specifically, the global coordinate system is a coordinate system used to control the robot 1. Therefore, a robot coordinate system may be used as the global coordinate system. The robot coordinate system may be a three-dimensional coordinate system defined based on the robot 1. The robot coordinate system may be, for example, a three-dimensional coordinate system set on the base 11 provided on the robot 1. That is, the robot coordinate system is, for example, a three-dimensional coordinate system set on the base 11 that does not move even when the robot 1 moves (i.e., the robot arm 12 moves). However, the robot coordinate system may be, for example, a three-dimensional coordinate system different from the three-dimensional coordinate system set on the base 11 provided on the robot 1. For example, the robot coordinate system may be a three-dimensional coordinate system set on the tip (i.e., the hand) of the robot arm 12. For example, the robot coordinate system may be a three-dimensional coordinate system set on the end effector 5 attached to the robot arm 12. For example, the robot coordinate system may be a three-dimensional coordinate system set at the tool center point of the end effector 5 attached to the robot arm 12. Alternatively, a coordinate system different from the robot coordinate system may be used as the global coordinate system. Note that the global coordinate system is not limited to the robot coordinate system, and may be a three-dimensional coordinate system with a fixed reference point as its origin. The fixed reference point may be set in the space (e.g., a factory) where the robot 1 is installed.
[0175] The control device 3 and the robot control device 4 may each control the robot 1 so that the end effector 5 is positioned at a desired position in a global coordinate system. The global coordinate system is a coordinate system defined by an X-axis, a Y-axis, and a Z-axis that are perpendicular to each other. The X-axis may be an axis along a horizontal plane. The Y-axis may be an axis along a horizontal plane. The Z-axis may be an axis perpendicular to the horizontal plane. The Z-axis may be an axis extending in the direction of gravity. The X-axis, Y-axis, and Z-axis shown in FIG. 2 may be the X-axis, Y-axis, and Z-axis, respectively. The origin of the global coordinate system does not have to be the origin of the X-axis, Y-axis, and Z-axis shown in FIG. 2.
[0176] The position and orientation calculation unit 311 may calculate, as the position of the target object OBJ in the global coordinate system, at least one of a position Tx of the target object OBJ in an X-axis direction parallel to the X-axis, a position Ty of the target object OBJ in a Y-axis direction parallel to the Y-axis, and a position Tz of the target object OBJ in a Z-axis direction parallel to the Z-axis. The position and orientation calculation unit 311 may calculate, as the orientation of the target object OBJ in the global coordinate system, at least one of a rotation amount Rx of the target object OBJ about the X-axis, a rotation amount Ry of the target object OBJ about the Y-axis, and a rotation amount Rz of the target object OBJ about the Z-axis. This is because the rotation amount Rx of the target object OBJ about the X-axis, the rotation amount Ry of the target object OBJ about the Y-axis, and the rotation amount Rz of the target object OBJ about the Z-axis are equivalent to a parameter representing the orientation of the target object OBJ about the X-axis, a parameter representing the orientation of the target object OBJ about the Y-axis, and a parameter representing the orientation of the target object OBJ about the Z-axis, respectively. For this reason, in the following description, the rotation amount Rx of the target object OBJ around the X axis, the rotation amount Ry of the target object OBJ around the Y axis, and the rotation amount Rz of the target object OBJ around the Z axis will be referred to as the orientation Rx of the target object OBJ around the X axis, the orientation Ry of the target object OBJ around the Y axis, and the orientation Rz of the target object OBJ around the Z axis, respectively.
[0177] The orientation Rx of the target object OBJ around the X-axis, the orientation Ry of the target object OBJ around the Y-axis, and the orientation Rz of the target object OBJ around the Z-axis may be considered to indicate the position of the target object OBJ in the rotation direction around the X-axis, the position of the target object OBJ in the rotation direction around the Y-axis, and the position of the target object OBJ in the rotation direction around the Z-axis, respectively. In other words, the orientation Rx of the target object OBJ around the X-axis, the orientation Ry of the target object OBJ around the Y-axis, and the orientation Rz of the target object OBJ around the Z-axis may all be considered to be parameters representing the position of the target object OBJ.
[0178] In this way, in step S322 of Figure 10, the position and orientation calculation unit 311 may calculate at least one of the position Tx, position Ty, position Tz, orientation Rx, orientation Ry, and orientation Rz as at least one of the position and orientation of the target object OBJ in the global coordinate system.
[0179] In this embodiment, the position and orientation calculation unit 311 may calculate the position and orientation of the target object OBJ by performing a matching process using the image data IMG acquired in step S321. Specifically, the position and orientation calculation unit 311 may calculate the position and orientation of the target object OBJ by performing a matching process using the image data IMG and the model data MDL. The model data MDL is data representing an object model OBM that is a model of the target object OBJ. The object model OBM is, for example, a two-dimensional model and includes a two-dimensional image. In this case, the model data MDL is data representing the object model OBM that is a two-dimensional model having a two-dimensional shape that serves as a reference for the target object OBJ. The model data MDL may be, for example, two-dimensional image data representing a two-dimensional model of the target object OBJ that is generated by virtually projecting a CAD model (or any three-dimensional model) of the target object OBJ from a plurality of different directions onto virtual planes that are orthogonal to each of the plurality of different directions. Alternatively, the model data MDL may be image data representing a two-dimensional image generated by capturing an image of an actual target object OBJ in advance. Note that the actual target object OBJ measured in advance for generating the model data MDL may be a reference or non-defective target object OBJ.
[0180] However, as will be explained in a third modified example below, model data indicating a model of at least a part of the edge of the target object OBJ may be used as the model data MDL.
[0181] As shown in FIG. 11 , the position and orientation calculation unit 311 may perform a matching process (in other words, a 2D matching process, or a template matching process) on the image indicated by the image data IMG, using the object model OBM indicated by the model data MDL as a template. Specifically, the position and orientation calculation unit 311 may perform, as the matching process, an object detection process that detects the target object OBJ indicated by the object model OBM in the image indicated by the image data IMG. In other words, the position and orientation calculation unit 311 may perform, as the matching process, an object detection process that detects the target object OBJ in the image indicated by the image data IMG by detecting a similar image portion that is similar to the object model OBM in the image indicated by the image data IMG. Note that the matching process (in this case, the object detection process) itself may be the same as an existing matching process. For example, the position and orientation calculation unit 311 may perform matching processing using a well-known method such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed-Up Robust Feature).
[0182] To perform the matching process, the position and orientation calculation unit 311 may translate, enlarge, reduce, and / or rotate the object model OBM in the image capture coordinate system, which is the coordinate system of the imaging device 21 that generated the image data IMG, so that characteristic locations throughout the entire object model OBM indicated by the model data MDL approach (e.g., match) characteristic locations throughout the entire target object OBJ reflected in the image indicated by the image data IMG. In other words, the position and orientation calculation unit 311 may change the positional relationship between the coordinate system of the model data MDL and the image capture coordinate system so that characteristic locations throughout the entire target object OBJ reflected in the image indicated by the image data IMG approach (e.g., match) characteristic locations throughout the entire object model OBM. As a result, the position and orientation calculation unit 311 can identify the positional relationship between the coordinate system of the model data MDL and the image capture coordinate system. Thereafter, the position and orientation calculation unit 311 may calculate the position and orientation of the target object OBJ in the imaging coordinate system from the position and orientation of the object model OBM in the coordinate system of the model data MDL based on the positional relationship between the coordinate system of the model data MDL and the imaging coordinate system. Thereafter, the position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ in the global coordinate system from at least one of the position and orientation of the target object OBJ in the imaging coordinate system using a transformation matrix for transforming three-dimensional coordinates in either the global coordinate system or the imaging coordinate system into three-dimensional coordinates in the other of the global coordinate system or the imaging coordinate system.
[0183] When approximating characteristic locations of the entire object model OBM to characteristic locations of the entire target object OBJ captured in the image represented by the image data IMG, the position and orientation calculation unit 311 may calculate a matching similarity, which is the similarity between the object model OBM and the image represented by the image data IMG (particularly, the image portion to which the object model OBM is fitted). The position and orientation calculation unit 311 may translate, enlarge, reduce, and / or rotate the object model OBM to maximize the matching similarity. As a result, under conditions where the matching similarity is maximized, the image portion of the image represented by the image data IMG to which the object model OBM is fitted is detected as a similar image portion in the image represented by the image data IMG that is similar to the object model OBM. In other words, the target object OBJ represented by the similar image portion is detected. Note that the matching similarity may be considered equivalent to the correlation indicating the correlation between the object model OBM and the image represented by the image data IMG. The matching similarity may be considered a parameter calculated as a detection result of the target object OBJ. In other words, the matching similarity may be considered to be information related to the detection result of the target object OBJ. The correlation degree may also be considered to be an index showing the correlation between the object model OBM and the target object OBJ appearing in the image represented by the image data IMG. The matching similarity may also be called a matching score.
[0184] When a target object OBJ whose calculated matching similarity exceeds a predetermined matching determination threshold used in the matching process is detected by the matching process, the position and orientation calculation unit 311 may select the target object OBJ as a processing target object on which the end effector 5 should actually perform predetermined processing. On the other hand, when a target object OBJ whose calculated matching similarity is below the matching determination threshold is detected by the matching process, the position and orientation calculation unit 311 does not have to select the target object OBJ as a processing target object on which the end effector 5 should actually perform predetermined processing. The matching determination threshold is a threshold used to detect the target object OBJ from the image indicated by the image data IMG by the matching process.
[0185] As described above, the imaging device 21 may capture images of multiple target objects OBJ. In this case, multiple target objects OBJ may appear in the image represented by the image data IMG generated by the imaging device 21. In this case, the position and orientation calculation unit 311 may select one of the multiple target objects OBJ reflected in the image represented by the image data IMG as the processing target object on which the end effector 5 should actually perform a predetermined process. In other words, the position and orientation calculation unit 311 may select one of the multiple target objects OBJ reflected in the image represented by the image data IMG as the processing target object. For example, the position and orientation calculation unit 311 may select one of the multiple target objects OBJ whose matching similarity is greater than a matching threshold and is the highest among the multiple target objects OBJ as the processing target object.
[0186] Note that if the matching process does not detect a target object OBJ whose matching similarity is below the matching judgment threshold, the control device 3 may move the imaging device 21. The state in which "the matching process does not detect a target object OBJ whose matching similarity is below the matching judgment threshold" may include a state in which the matching similarity of a single target object OBJ detected by the matching process is below the matching judgment threshold. The state in which "the matching process does not detect a target object OBJ whose matching similarity is below the matching judgment threshold" may include a state in which the matching similarities of all of the multiple target objects O detected by the matching process are below the matching judgment threshold. Thereafter, the imaging device 21 may image the target object OBJ again. In this case, the imaging device 21 may image the target object OBJ from a position different from the position of the imaging device 21 before the imaging device 21 moved. The imaging device 21 may image the target object OBJ in a posture different from the posture of the imaging device 21 before the imaging device 21 moved. Thereafter, the control device 3 may select the processing target object by performing the processes of steps S321 and S322 again. In this case, the appearance of the target object OBJ in the image represented by the image data IMG changes as the image capture device 21 moves. Therefore, even if a target object OBJ whose matching similarity is below the matching determination threshold was not detected by the matching process before the image capture device 21 moved (i.e., the processing target object could not be selected), after the image capture device 21 moves, there is a possibility that a target object OBJ whose matching similarity is below the matching determination threshold will be detected by the matching process (i.e., the processing target object can be selected).
[0187] Note that the method for calculating at least one of the position and orientation of the target object OBJ is not limited to the above-described matching process using the image data IMG. The position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ using other well-known methods for calculating at least one of the position and orientation of the target object OBJ using the image data IMG (i.e., a portion of the image data IMG). The method for calculating at least one of the position and orientation of the target object OBJ may be a well-known method for calculating at least one of the position and orientation of the target object OBJ based on the image data IMG (i.e., a portion of the image data IMG) without using the model data MDL, or a well-known method for calculating at least one of the position and orientation of the target object OBJ using data other than the model data MDL and the image data IMG (i.e., a portion of the image data IMG). For example, the method for calculating at least one of the position and orientation of the target object OBJ may be a method for calculating at least one of the position and orientation of the target object OBJ based on the image data IMG (i.e., a portion of the image data IMG) using machine learning or deep learning. In this case, a prediction model for outputting at least one of the position and orientation of the target object OBJ when image data IMG (i.e., a portion of the image data IMG) is input may be constructed in advance by machine learning or deep learning. The position and orientation calculation unit 311 may calculate at least one of the position and orientation of the target object OBJ by inputting the image data IMG (i.e., a portion of the image data IMG) into this prediction model. This prediction model may be stored in the position and orientation calculation unit 311. Note that the position and orientation calculation unit 311 may read out this prediction model stored in the storage device 32.
[0188] Referring back to FIG. 10, the signal generating unit 312 generates a control signal using the position and orientation information POI generated in step S322 (step S323).
[0189] In step S323, the signal generating unit 312 may generate a control signal so that the positional relationship between the end effector 5 and the target object OBJ becomes a predetermined target positional relationship. In other words, the signal generating unit 312 may generate a control signal for controlling the robot 1 so that the positional relationship between the end effector 5 and the target object OBJ becomes a predetermined target positional relationship.
[0190] The predetermined target positional relationship may include a positional relationship in which the end effector 5 is located at a target processing position TPP where the end effector 5 should be located when performing a predetermined process on the target object OBJ. In this case, the signal generating unit 312 may generate a control signal so that the end effector 5 is located at the target processing position TPP determined based on the target object OBJ. For example, the signal generating unit 312 may generate a control signal for controlling the robot 1 so that the end effector 5 is located at the target processing position TPP determined based on the target object OBJ.
[0191] Specifically, the signal generation unit 312 recognizes at least one of the position and orientation of the target object OBJ in the global coordinate system based on the position and orientation information POI generated in step S322. Furthermore, the signal generation unit 312 recognizes the target processing position TPP determined based on the target object OBJ, based on the work information (or user input specifying the target processing position TPP) described below. That is, the signal generation unit 312 recognizes the target processing position TPP in the coordinate system based on the target object OBJ. In this case, the signal generation unit 312 may calculate the target processing position TPP_G in the global coordinate system based on at least one of the position and orientation of the target object OBJ in the global coordinate system and the target processing position TPP determined based on the target object OBJ. That is, the signal generation unit 312 may convert the target processing position TPP in the coordinate system based on the target object OBJ to the target processing position TPP_G in the global coordinate system. Note that both the target processing position TPP in the coordinate system based on the target object OBJ and the target processing position TPP_G in the global coordinate system indicate the position where the end effector 5 should be located when performing a predetermined process on the target object OBJ (i.e., the target processing position in a broad sense). Thereafter, the signal generating unit 312 may generate a control signal for controlling the robot 1 so that the end effector 5 is located at the target processing position TPP_G in the global coordinate system.
[0192] The state in which "the end effector 5 is located at the target processing position TPP" may include a state in which "the position of the end effector 5 (e.g., the position of the tool center point) completely coincides with the target processing position TPP." The state in which "the end effector 5 is located at the target processing position TPP" may also include a state in which "the position of the end effector 5 does not completely coincide with the target processing position TPP, but the amount of deviation between the position of the end effector 5 and the target processing position TPP is small enough to be equal to or less than the upper tolerance threshold for the position of the end effector 5."
[0193] The predetermined target positional relationship may be set by the user. The target processing position TPP that defines the predetermined target positional relationship may be set by the user. The upper tolerance limit threshold (i.e., the upper tolerance limit value for the position of the end effector 5) that defines the predetermined target positional relationship may be set by the user. Alternatively, the predetermined target positional relationship may be automatically set by the control device 3. The target processing position TPP that defines the predetermined target positional relationship may be automatically set by the control device 3. The upper tolerance limit threshold (i.e., the upper tolerance limit value for the position of the end effector 5) that defines the predetermined target positional relationship may be automatically set by the control device 3.
[0194] In step S323, the signal generator 312 may generate a control signal so that the posture relationship between the end effector 5 and the target object OBJ becomes a predetermined target posture relationship. The signal generator 312 may generate a control signal for controlling the robot 1 so that the posture relationship between the end effector 5 and the target object OBJ becomes the predetermined target posture relationship.
[0195] The predetermined desired posture relationship may include a positional relationship in which the end effector 5 assumes a target processing posture that the end effector 5 should assume when performing a predetermined process on the target object OBJ. In this case, the signal generating unit 312 may generate a control signal so that the end effector 5 assumes the target processing posture determined based on the target object OBJ. For example, the signal generating unit 312 may generate a control signal for controlling the robot 1 so that the end effector 5 assumes the target processing posture determined based on the target object OBJ.
[0196] Specifically, the signal generation unit 312 recognizes at least one of the position and orientation of the target object OBJ in the global coordinate system based on the position and orientation information POI generated in step S322. Furthermore, the signal generation unit 312 recognizes a target processing orientation determined based on the target object OBJ as a reference, based on work information (or user input specifying the target processing orientation), which will be described later. That is, the signal generation unit 312 recognizes a target processing orientation in a coordinate system based on the target object OBJ as a reference. In this case, the signal generation unit 312 may calculate a target processing orientation in the global coordinate system based on at least one of the position and orientation of the target object OBJ in the global coordinate system and the target processing orientation determined based on the target object OBJ as a reference. That is, the signal generation unit 312 may convert the target processing orientation in the coordinate system based on the target object OBJ as a reference into a target processing orientation in the global coordinate system. Note that both the target processing posture in the coordinate system based on the target object OBJ and the target processing posture in the global coordinate system indicate the posture that the end effector 5 should take when performing a predetermined process on the target object OBJ (i.e., the target processing posture in a broad sense). Thereafter, the signal generating unit 312 may generate a control signal for controlling the robot 1 so that the end effector 5 takes the target processing posture in the global coordinate system.
[0197] The state in which "the end effector 5 assumes a target processing posture" may mean a state in which "the posture of the end effector 5 matches the target processing posture." The state in which "the posture of the end effector 5 matches the target processing posture" may include a state in which "the posture of the end effector 5 completely matches the target processing posture." The state in which "the posture of the end effector 5 matches the target processing posture" may include a state in which "the posture of the end effector 5 does not completely match the target processing posture, but the deviation between the posture of the end effector 5 and the target processing posture is small enough to be equal to or less than the allowable upper limit value for the posture of the end effector 5."
[0198] The predetermined target posture relationship may be set by the user. The target processing posture that defines the predetermined target posture relationship may be set by the user. The upper tolerance limit threshold that defines the predetermined target posture relationship (i.e., the upper tolerance limit value for the posture of the end effector 5) may be set by the user. Alternatively, the predetermined target posture relationship may be automatically set by the control device 3. The target processing posture that defines the predetermined target posture relationship may be automatically set by the control device 3. The upper tolerance limit threshold that defines the predetermined target posture relationship (i.e., the upper tolerance limit value for the posture of the end effector 5) may be automatically set by the control device 3.
[0199] In step S323, the signal generation unit 312 may use the position and orientation information POI generated in step S322 to generate a control signal to perform feedback control of the robot 1. For example, the signal generation unit 312 may use the position and orientation information POI to generate a control signal to perform feedback control including P (Proportional) control. For example, the signal generation unit 312 may use the position and orientation information POI to generate a control signal to perform feedback control including PI (Proportional-Integral) control. For example, the signal generation unit 312 may use the position and orientation information POI to generate a control signal to perform feedback control including PID (Proportional-Integral-Derivative) control.
[0200] When the signal generator 312 performs feedback control using the position and orientation information POI in this way, the fine movement process may be considered to be a process based on closed-loop control. The closed-loop control may be at least one of the above-mentioned P control, PI control, and PID control. On the other hand, the rough movement process, which moves the end effector 5 in accordance with movement instruction information included in a computer program, may be considered to be a process based on open-loop control. In other words, the fine movement process and the rough movement process may differ in that they use different loop control methods.
[0201] Furthermore, as described above, in order to perform the fine movement processing, the control device 3 needs to calculate the position and orientation information POI. On the other hand, in order to perform the rough movement processing, the robot control device 4 does not need to calculate the position and orientation information POI. Therefore, for example, the computation cost of the fine movement processing that includes the process of calculating the position and orientation information POI is higher than the computation cost of the rough movement processing that does not need to include the process of calculating the position and orientation information POI. In other words, the computation cost of the rough movement processing that does not include the process of calculating the position and orientation information POI is lower than the computation cost of the fine movement processing that includes the process of calculating the position and orientation information POI. Therefore, the fine movement processing and the rough movement processing may differ in that the required computation costs are different from each other.
[0202] After generating the control signal, the signal generating unit 312 outputs the control signal generated in step S323 (step S324). For example, the signal generating unit 312 may output the control signal generated in step S323 to the robot control device 4. For example, the signal generating unit 312 may output the control signal generated in step S323 to the robot 1. As a result, the robot 1 moves the end effector 5 in accordance with the control signal generated by the signal generating unit 312.
[0203] 9 again, the signal generating unit 312 then determines whether a predetermined convergence condition is met (step S33). The predetermined convergence condition may include a position convergence condition that the positional relationship between the end effector 5 and the target object OBJ becomes a predetermined target positional relationship. An example of the position convergence condition is a condition that the end effector 5 is located at a target processing position TPP determined based on the target object OBJ. The predetermined convergence condition may include, in addition to or instead of the position convergence condition, a posture convergence condition that the posture relationship between the end effector 5 and the target object OBJ becomes a predetermined target posture relationship. An example of the posture convergence condition is a condition that the end effector 5 takes a target processing posture determined based on the target object OBJ.
[0204] The signal generating unit 312 may determine that the predetermined convergence condition is met when both the position convergence condition and the attitude convergence condition are met. The signal generating unit 312 may determine that the predetermined convergence condition is not met when at least one of the position convergence condition and the attitude convergence condition is not met. However, the signal generating unit 312 may also determine that the predetermined convergence condition is met when at least one of the position convergence condition and the attitude convergence condition is met.
[0205] If the result of the determination in step S33 is that the predetermined convergence condition is not met (step S33: No), the control device 3 performs the fine movement process again (step S32). Specifically, the control device 3 acquires the image data IMG again (step S321 in FIG. 10 ), recalculates at least one of the position and orientation of the target object OBJ (step S322 in FIG. 10 ), generates the control signal again (step S323 in FIG. 10 ), and outputs the control signal again (step S324 in FIG. 10 ). In other words, the control device 3 may repeat the fine movement process multiple times until the predetermined convergence condition is met. However, if the predetermined convergence condition is met after one fine movement process has been performed, the control device 3 does not need to repeat the fine movement process multiple times.
[0206] Because the control device 3 outputs a control signal each time the control device 3 performs the fine movement process, the robot 1 moves the end effector 5 each time the control device 3 performs the fine movement process. Therefore, the robot 1 essentially gradually moves the end effector 5 in accordance with the control signal that is appropriately generated (updated) by performing the fine movement process multiple times. In other words, each time the control device 3 performs the fine movement process, the positional relationship between the end effector 5 and the target object OBJ gradually changes. In this case, the signal generation unit 312 may generate (in this case, update) the control signal so that the positional relationship between the end effector 5 and the target object OBJ gradually approaches a predetermined target positional relationship. The signal generation unit 312 may also generate (in this case, update) the control signal so that the posture relationship between the end effector 5 and the target object OBJ gradually approaches a predetermined target posture relationship. As a result, by performing the fine movement process multiple times, the possibility that a predetermined convergence condition will be met gradually increases.
[0207] More specifically, because the proportion of the target object OBJ occupies the imaging field of view of the imaging device 21 (the proportion of the target object OBJ captured in the image) increases, the shorter the distance between the imaging device 21 and the target object OBJ, the higher the accuracy of calculating at least one of the position and orientation of the target object OBJ. In other words, the closer the imaging device 21 is to the target object OBJ, the higher the accuracy of calculating at least one of the position and orientation of the target object OBJ. In this case, if the robot 1 moves in accordance with a control signal generated based on at least one of the position and orientation of the target object OBJ calculated at the first timing, the imaging device 21 approaches the target object OBJ. Therefore, at a second timing later than the first timing, the control device 3 can calculate at least one of the position and orientation of the target object OBJ with higher accuracy. As a result, if the robot 1 moves in accordance with a control signal generated based on at least one of the position and orientation of the target object OBJ calculated at the second timing, the end effector 5 approaches the target object OBJ more accurately. That is, the end effector 5 approaches the target processing position TPP more accurately. Conversely, even if the robot 1 continues to move in accordance with a control signal generated based on at least one of the position and orientation of the target object OBJ calculated when the distance between the imaging device 21 and the target object OBJ is not very short, the end effector 5 may not be able to approach the target object OBJ sufficiently. That is, the end effector 5 may not be able to approach the target processing position TPP sufficiently (in other words, may not be able to reach it). For this reason, in this embodiment, by repeating the fine movement process by the control device 3, the accuracy of calculating at least one of the position and orientation of the target object OBJ increases, and as a result, the end effector 5 can reach the target processing position TPP accurately.
[0208] Furthermore, when the target object OBJ is being transported by a transport device, the target object OBJ moves while the robot 1 is moving. Therefore, if the fine movement process is not repeated, the robot 1 can move the end effector 5 to approach the target object OBJ before movement, but cannot move the end effector 5 to approach the target object OBJ after movement. However, in this embodiment, because the control device 3 repeats the fine movement process, the calculation results of at least one of the position and orientation of the moving target object OBJ are successively updated. In other words, the control device 3 can accurately calculate at least one of the current position and current orientation of the moving target object OBJ. Therefore, the robot 1 can move the end effector 5 to approach the moved target object OBJ in accordance with a control signal generated based on at least one of the current position and current orientation of the moving target object OBJ.
[0209] The control device 3 may repeat the fine movement process at a period corresponding to the image capture rate of the imaging device 21. For example, if the imaging device 21 captures an image of the target object OBJ N times (where N is a variable indicating an integer greater than or equal to 1) (e.g., 200 times) per second, the control device 3 may repeat the fine movement process N times per second. In other words, the period in which the imaging device 21 captures an image of the target object OBJ and the period in which the control device 3 repeats the fine movement process may be the same. However, the period in which the imaging device 21 captures an image of the target object OBJ and the period in which the control device 3 repeats the fine movement process may be different. For example, if the imaging device 21 captures an image of the target object OBJ N times (e.g., 200 times) per second, the control device 3 may repeat the fine movement process M times (where M is a variable indicating an integer greater than or equal to 1 and different from N) (e.g., 100 times) per second.
[0210] Note that because the robot 1 moves the end effector 5 every time the control device 3 performs the fine movement processing, the imaging device 21, which moves together with the end effector 5, also moves every time the control device 3 performs the fine movement processing. In this case, the control device 3 may change the imaging conditions of the imaging device 21 every time the control device 3 performs the fine movement processing. For example, the control device 3 may change the imaging conditions so that the imaging device 21, which has moved due to the fine movement processing, can appropriately image the target object OBJ. An example of the imaging conditions is at least one of the exposure time and the intensity of the illumination light.
[0211] The control device 3 may perform predetermined image processing on an image represented by image data IMG acquired from the imaging device 21. In this case, the control device 3 may change the image processing conditions (hereinafter referred to as "image processing conditions") each time the control device 3 performs fine movement processing. The image processing may include gamma correction processing. In this case, the image processing conditions may include gamma correction processing conditions related to the gamma correction processing. The gamma correction processing conditions may include at least one of a condition regarding whether or not to perform gamma correction processing and a condition regarding a gamma value used in the gamma correction processing. The image processing may include HDR (High Dynamic Range) processing. In this case, the image processing conditions may include HDR processing conditions related to HDR processing. The HDR processing conditions may include at least one of a condition regarding whether or not to perform HDR processing and a condition regarding the number of images to be combined in the HDR processing. The image processing may include noise reduction processing. In this case, the image processing conditions may include noise reduction processing conditions related to the noise reduction processing. The noise removal processing conditions may include at least one of a condition regarding whether or not to perform noise removal processing, a condition regarding the type of filter used in the noise removal processing, a condition regarding the combination of at least two filters used in the noise removal processing, and a condition regarding the parameters of the filters used in the noise removal processing.
[0212] The control device 3 may perform fine movement processing by moving the imaging device 21 (i.e., moving the end effector 5) while maintaining the distance between the end effector 5 and the target object OBJ, so that the imaging device 21 can capture images of the target object OBJ from different imaging positions and / or different imaging directions. As an example, if an obstacle other than the target object OBJ exists between the imaging device 21 and the target object OBJ, part of the target object OBJ may be located in a blind spot of the imaging device 21, and therefore the imaging device 21 may not be able to capture an appropriate image of the target object OBJ. Note that an example of an obstacle is the above-mentioned peripheral object (i.e., a peripheral object located around the robot 1). In this case, if the presence of an obstacle is identified based on the results of the previous fine movement process (particularly the matching process), the control device 3 may generate a control signal for controlling the robot 1 to move the imaging device 21 to avoid the obstacle (e.g., by changing the orientation of the imaging device 21) while maintaining the distance between the end effector 5 and the target object OBJ, thereby reducing the impact of the obstacle on the image captured by the imaging device 21. Alternatively, if an obstacle exists between the imaging device 21 and the target object OBJ, the imaging device 21 will capture an image of the obstacle in addition to the target object OBJ. In this case, the control device 3 may calculate at least one of the position and orientation of the obstacle in addition to at least one of the position and orientation of the target object OBJ. Thereafter, based on the calculation result of at least one of the position and orientation of the obstacle, the control device 3 may generate a control signal for controlling the robot 1 to move the imaging device 21 to avoid the obstacle (e.g., by changing the orientation of the imaging device 21) while maintaining the distance between the end effector 5 and the target object OBJ, thereby reducing the impact of the obstacle on the image captured by the imaging device 21. Alternatively, in some cases, the control device 3 may generate a control signal to control the robot 1 so as to move the imaging device 21 to avoid the obstacle while allowing the distance between the end effector 5 and the target object OBJ to change, thereby reducing the impact of the obstacle on the imaging of the imaging device 21.For example, the control device 3 may generate a control signal for controlling the robot 1 so as to move the imaging device 21 to avoid the obstacle while allowing the distance between the end effector 5 and the target object OBJ to increase, thereby reducing the effect of the obstacle on the image captured by the imaging device 21. For example, the control device 3 may generate a control signal for controlling the robot 1 so as to move the imaging device 21 to avoid the obstacle while allowing the distance between the end effector 5 and the target object OBJ to decrease, thereby reducing the effect of the obstacle on the image captured by the imaging device 21. As a result, the accuracy of calculating the position and orientation of the target object OBJ is improved, and the control device 3 can accurately move the end effector 5 so that the end effector 5 approaches the target object OBJ.
[0213] If it is determined that the predetermined convergence condition is not satisfied, the control device 3 does not need to receive a signal (e.g., a control permission signal) from the robot control device 4 in order to perform the fine movement processing again. In other words, even if the control device 3 does not receive a signal from the robot control device 4, the control device 3 may perform the fine movement processing again. Even if the control device 3 does not receive a signal from the robot control device 4, the control device 3 may repeat the fine movement processing multiple times. In other words, even if the robot control device 4 does not send a signal to the control device 3, the control device 3 may perform the fine movement processing again. Even if the robot control device 4 does not send a signal to the control device 3, the control device 3 may repeat the fine movement processing multiple times.
[0214] On the other hand, if it is determined in step S33 that the predetermined convergence condition is met (step S33: Yes), the control device 3 does not need to perform the fine movement process.
[0215] Here, because the fine movement processing is performed after the rough movement processing, the end effector 5, which has approached the target object OBJ through the rough movement processing, will approach the target object OBJ even closer through the fine movement processing. Therefore, the difference between the position of the end effector 5 at the time the fine movement processing is completed and the target processing position TPP where the end effector 5 should be when performing a predetermined process on the target object OBJ will be smaller than, for example, the difference between the position of the end effector 5 at the time the rough movement processing is completed and the target processing position TPP. In other words, the distance between the end effector 5 and the target processing position TPP at the time the fine movement processing is completed will be shorter than, for example, the distance between the end effector 5 and the target processing position TPP at the time the rough movement processing is completed. For this reason, the fine movement processing may be considered to include processing for generating a control signal for controlling the robot 1 so that the difference between the position of the end effector 5 and the target processing position TPP at the time when the fine movement processing is completed is smaller than the difference between the position of the end effector 5 and the target processing position TPP at the time when the rough movement processing is completed. The fine movement processing may be considered to include processing for generating a control signal for controlling the robot 1 so that the distance between the end effector 5 and the target processing position TPP at the time when the fine movement processing is completed is shorter than the distance between the end effector 5 and the target processing position TPP at the time when the rough movement processing is completed.
[0216] Furthermore, because the fine movement processing is performed after the rough movement processing, the difference between the posture of the end effector 5 at the time when the fine movement processing is completed and the target processing posture that the end effector 5 should take when performing a predetermined process on the target object OBJ is smaller than, for example, the difference between the posture of the end effector 5 at the time when the rough movement processing is completed and the target processing posture. For this reason, the fine movement processing may be considered to include processing for generating a control signal for controlling the robot 1 so that the difference between the posture of the end effector 5 at the time when the fine movement processing is completed and the target processing posture is smaller than the difference between the posture of the end effector 5 at the time when the rough movement processing is completed and the target processing posture.
[0217] When the convergence condition includes the above-described position convergence condition, the difference between the positional relationship between the end effector 5 and the target object OBJ and a predetermined target positional relationship at the time when the fine movement process is completed is smaller than, for example, the difference between the positional relationship between the end effector 5 and the target object OBJ and a predetermined target positional relationship at the time when the rough movement process is completed. Therefore, the fine movement process may be considered to include processing for generating a control signal for controlling the robot 1 so that the difference between the positional relationship between the end effector 5 and the target object OBJ and a predetermined target positional relationship at the time when the fine movement process is completed is smaller than the difference between the positional relationship between the end effector 5 and the target object OBJ and a predetermined target positional relationship at the time when the rough movement process is completed.
[0218] When the convergence condition includes the above-described posture convergence condition, the difference between the posture relationship between the end effector 5 and the target object OBJ and the predetermined target posture relationship at the time when the fine movement processing is completed is smaller than, for example, the difference between the posture relationship between the end effector 5 and the target object OBJ and the predetermined target posture relationship at the time when the rough movement processing is completed. Therefore, the fine movement processing may be considered to include processing for generating a control signal for controlling the robot 1 so that the difference between the posture relationship between the end effector 5 and the target object OBJ and the predetermined target posture relationship at the time when the fine movement processing is completed is smaller than the difference between the posture relationship between the end effector 5 and the target object OBJ and the predetermined target posture relationship at the time when the rough movement processing is completed.
[0219] After it is determined that the convergence condition is met (step S33: Yes), the signal generator 312 of the control device 3 controls the end effector 5 to perform a predetermined process on the target object OBJ (step S34). Specifically, the signal generator 312 generates a control signal for controlling the end effector 5 to perform a predetermined process on the target object OBJ, and outputs the generated control signal to the robot control device 4 or the end effector 5. As a result, the end effector 5 performs the predetermined process on the target object OBJ.
[0220] Alternatively, in addition to or instead of the signal generating unit 312 of the control device 3, the signal generating unit 411 of the robot control device 4 may control the end effector 5 to perform a predetermined process on the target object OBJ (step S34). Specifically, the signal generating unit 411 may generate a control signal for controlling the end effector 5 to perform a predetermined process on the target object OBJ, and output the generated control signal to the end effector 5. As a result, the end effector 5 performs the predetermined process on the target object OBJ.
[0221] Even during a processing period in which the end effector 5 performs a predetermined process on the target object OBJ, the control device 3 may perform fine movement processing as necessary. Specifically, the control device 3 may calculate at least one of the position and orientation of the target object OBJ based on image data IMG generated by the imaging device 21 during the processing period. The control device 3 may then generate a control signal based on at least one of the position and orientation of the target object OBJ. For example, the control device 3 may generate a control signal so that the positional relationship between the end effector 5 and the target object OBJ is maintained at a predetermined target positional relationship. For example, the control device 3 may generate a control signal so that the orientation relationship between the end effector 5 and the target object OBJ is maintained at a predetermined target orientation relationship.
[0222] In particular, when the target object OBJ moves during the processing period, the control device 3 may perform the fine movement processing even during the processing period. In this case, the end effector 5 can perform a predetermined processing on the target object OBJ while following the moving target object OBJ. In other words, even when the target object OBJ moves during the processing period, the end effector 5 can perform a predetermined processing on the target object OBJ in the same way as when the target object OBJ is stationary during the processing period.
[0223] Note that, since the end effector 5 is controlled to perform a predetermined process on the target object OBJ after the fine movement process is performed, the state of the end effector 5 may be set to an ON state in which a control signal (particularly, an end effector drive signal) can be input during the period in which the fine movement process is being performed. The ON state may mean a state in which, when a control signal (particularly, an end effector drive signal) is input to the end effector 5, the end effector 5 can perform a predetermined process without requiring any processing other than the processing for inputting the control signal. On the other hand, since the fine movement process is performed without controlling the end effector 5 after the rough movement process is performed, the state of the end effector 5 does not need to be set to an ON state during the period in which the rough movement process is being performed. For example, the state of the end effector 5 may be set to an OFF state in which a control signal (particularly, an end effector drive signal) cannot be input. The off state may refer to a state in which, when a control signal (particularly an end effector drive signal) is input to the end effector 5, the end effector 5 cannot perform a predetermined process unless a process other than the process of inputting the control signal (for example, a startup process) is performed. As a result, the standby power consumption of the end effector 5 can be reduced.
[0224] After the end effector 5 has performed a predetermined process on the target object OBJ, the control device 3 may further perform fine movement processing as necessary. For example, in the case where fine movement processing is performed to retract the end effector 5 after the end effector 5 has performed a predetermined process on the target object OBJ as described using FIG. 3C and the like, the control device 3 may perform fine movement processing to retract the end effector 5 after the end effector 5 has performed the predetermined process on the target object OBJ. However, the control device 3 does not have to perform fine movement processing after the end effector 5 has performed the predetermined process on the target object OBJ.
[0225] If the control device 3 performs the fine movement process again after the end effector 5 has performed a predetermined process on the target object OBJ, the control device 3 may again control the end effector 5 so that the end effector 5 performs the predetermined process on the target object OBJ. That is, the control device 3 may alternately repeat an operation of performing the fine movement process until a convergence condition is met and an operation of controlling the end effector 5 after the convergence condition is met. As an example, an example will be described in which the robot 1 uses the end effector 5 to hold a workpiece W and then performs a series of processes to release the held workpiece W onto the placement device T. In this case, the control device 3 may perform the fine movement process on the workpiece W to be held by the end effector 5 until the convergence condition for the workpiece W is met. Thereafter, the control device 3 may control the end effector 5 to hold the workpiece W. Thereafter, the control device 3 may perform the fine movement process on the placement device T from which the end effector 5 is to release the workpiece W until the convergence condition for the placement device T is met. Thereafter, the control device 3 may control the end effector 5 to release the held workpiece W onto the placement device T.
[0226] (2-4) Transmission of Completion Notification Signal The control device 3 transmits a completion notification signal to the robot control device 4 at a desired timing after controlling the end effector 5 to perform a predetermined process on the target object OBJ (step S35). For example, the control device 3 transmits a completion notification signal to the robot control device 4 after controlling the end effector 5 to perform a predetermined process on the target object OBJ and after determining that no further fine movement processing is required (step S35). That is, the control device 3 transmits a completion notification signal to the robot control device 4 after controlling the end effector 5 to perform a predetermined process on the target object OBJ and after completing the fine movement processing (step S35). In this case, the completion notification signal may be considered to be a signal notifying the robot control device 4 that the control device 3 has completed the fine movement processing. Alternatively, the completion notification signal may be considered to be a signal notifying the robot control device 4 that the control device 3 has completed control of the end effector 5 to perform a predetermined process on the target object OBJ.
[0227] As described above, when the signal generating unit 411 of the robot control device 4 controls the end effector 5 in step S34, the control device 3 may transmit a completion notification signal to the robot control device 4 at a desired timing before the robot control device 4 controls the end effector 5 to perform a predetermined process on the target object OBJ. The control device 3 may transmit a completion notification signal to the robot control device 4 after completing the fine movement process (for example, after it is determined in step S33 that the convergence condition is met).
[0228] After the control device 3 transmits the completion notification signal to the robot control device 4, the robot control device 4 receives the completion notification signal transmitted from the control device 3 (step S43). The robot control device 4 that receives the completion notification signal may start the rough movement process. That is, the robot control device 4 may start the rough movement process after the robot control device 4 receives the completion notification signal. In other words, after the robot control device 4 receives the completion notification signal, the robot control device 4 may move the end effector 5 by performing the rough movement process. On the other hand, a robot control device 4 that has not received the completion notification signal does not start the rough movement process. That is, the robot control device 4 does not start the rough movement process before the robot control device 4 receives the completion notification signal. In other words, before the robot control device 4 receives the completion notification signal, the robot control device 4 does not move the end effector 5 by performing the rough movement process.
[0229] On the other hand, after the control device 3 transmits the completion notification signal to the robot control device 4, the control device 3 does not perform the fine movement process. In particular, the control device 3 may be restricted (e.g., prohibited, the same applies below) from moving the end effector 5 (imaging device 21) by performing the fine movement process. For example, the control device 3 may be restricted from acquiring image data IMG from the imaging device 21. Note that if the control device 3 is restricted from acquiring image data IMG from the imaging device 21, the imaging device 21 may be restricted from capturing an image of the target object OBJ. For example, the control device 3 may be restricted from calculating the position and orientation of the target object OBJ based on the image data IMG. For example, the control device 3 may be restricted from generating a control signal. For example, even if the control device 3 is permitted to generate a control signal, the control device 3 may be restricted from outputting the control signal to the robot control device 4 and the robot 1. For example, even if the control device 3 is permitted to generate and output a control signal, the robot control device 4 and the robot 1 may be restricted from accepting the control signal output by the control device 3.
[0230] Therefore, after the control device 3 transmits the completion notification signal to the robot control device 4, it may be considered that the robot control device 4 holds the control right of the robot 1. In this case, the completion notification signal may be considered as a signal that transfers (in this case, returns) the control right of the robot 1 from the control device 3 to the robot control device 4.
[0231] However, even when the robot control device 4 receives the completion notification signal, there is a possibility that the robot control device 4 does not need to perform further rough movement processing. For example, if the robot system SYS has performed a predetermined process on all of the multiple target objects OBJ using the end effector 5, there is a possibility that the robot control device 4 does not need to perform further rough movement processing. For example, if the robot system SYS has performed a predetermined process on all of the multiple target portions of the target object OBJ using the end effector 5, there is a possibility that the robot control device 4 does not need to perform further rough movement processing. Therefore, after the robot control device 4 receives the completion notification signal, the robot control device 4 may determine whether or not to end the robot control processing (step S44).
[0232] If it is determined in step S44 that the robot control process is to be ended (step S44: Yes), the robot control device 4 does not need to perform the rough movement process again. In this case, the control device 3 and the robot control device 4 may end the robot control process.
[0233] On the other hand, if it is determined in step S44 that the robot control process should not be terminated (step S44: No), the robot control device 4 may perform the rough movement process again (step S41). Thereafter, the control device 3 may perform the fine movement process (step S32). In other words, the robot system SYS may alternately repeat the rough movement process and the fine movement process.
[0234] (2-5) Robot Control Process for Alternately Repeating Rough Movement Processing and Fine Movement Processing (2-5-1) First Example of Robot Control Process for Alternately Repeating Rough Movement Processing and Fine Movement Processing A first example of a robot control process for alternately repeating rough movement processing and fine movement processing is a robot control process performed when the end effector 5 sequentially performs predetermined processing on multiple target portions of the target object OBJ (e.g., the portion where the hole HL shown in FIGS. 5 and 6 is formed). As an example, a robot control process performed when the end effector 5 sequentially performs predetermined processing on a first target portion of the target object OBJ (e.g., the portion where the hole HL#1 shown in FIGS. 5 and 6 is formed) and a second target portion of the target object OBJ different from the first target portion (e.g., the portion where the hole HL#2 shown in FIGS. 5 and 6 is formed) will be described. However, even when the end effector 5 sequentially performs predetermined processing on three or more target portions of the target object OBJ, the following robot control process may be performed in which the rough movement processing and the fine movement processing are alternately repeated as many times as the number of target portions.
[0235] In this case, the robot control device 4 may first perform a rough movement process on the first target portion (step S41 in FIG. 9 ). For example, the robot control device 4 may generate and output a control signal for controlling the robot 1 so that the end effector 5 approaches the first target portion. As a result, the end effector 5 moves under the control of the robot control device 4.
[0236] After the rough movement process for the first target portion, the robot control device 4 transmits a control permission signal to the control device 3 (step S42 in FIG. 9 ), and the control device 3 receives the control permission signal from the robot control device 4 (step S31 in FIG. 9 ). The control device 3 may then perform fine movement process for the first target portion (step S32 in FIG. 9 ). For example, the control device 3 may acquire image data IMG generated by the imaging device 21 capturing an image of the target object OBJ (particularly, the first target portion) after the rough movement process for the first target portion. The control device 3 may then calculate at least one of the position and orientation of the target object OBJ (particularly, at least one of the position and orientation of the first target portion of the target object OBJ) based on the acquired image data IMG. The control device 3 may then generate and output a control signal for controlling the robot 1 to move the end effector 5 based on the calculation result of the position and orientation of the first target portion. As a result, the end effector 5 moves under the control of the control device 3.
[0237] The control device 3 may perform the fine movement process on the first target portion until the above-mentioned predetermined convergence condition is met (step S33 in FIG. 9 ). That is, the control device 3 may repeat the fine movement process on the first target portion until the above-mentioned predetermined convergence condition is met. In particular, the control device 3 may repeat the fine movement process on the first target portion until the above-mentioned predetermined convergence condition is met during the period from the completion of the rough movement process on the first target portion to the start of the rough movement process on the second target portion.
[0238] As one example, the control device 3 may perform the fine movement process on the first target portion until a position convergence condition is met, in which the positional relationship between the end effector 5 and the first target portion becomes a first target positional relationship. An example of the first target positional relationship is a positional relationship in which the end effector 5 is located at a first target processing position TPP, where the end effector 5 should be located when performing a predetermined process on the first target portion. As another example, the control device 3 may perform the fine movement process on the first target portion until a posture convergence condition is met, in which the posture relationship between the end effector 5 and the first target portion becomes a first target posture relationship. An example of the first target posture relationship is a positional relationship in which the end effector 5 assumes a first target processing posture, where the end effector 5 should be located when performing a predetermined process on the first target portion.
[0239] If the convergence condition is met (step S33 in FIG. 9 : Yes), the control device 3 ends the fine movement process for the first target portion and controls the end effector 5 to perform a predetermined process for the first target portion (step S34 in FIG. 9 ).
[0240] Thereafter, the control device 3 transmits a completion notification signal to the robot control device 4 (step S35 in FIG. 9 ), and the robot control device 4 receives the completion notification signal transmitted from the control device 3 (step S43 in FIG. 9 ). Thereafter, the robot control device 4 may perform rough movement processing on the second target portion (step S41 in FIG. 9 ). For example, the robot control device 4 may generate and output a control signal for controlling the robot 1 so that the end effector 5 approaches the second target portion from the first target portion. As a result, the end effector 5 moves under the control of the robot control device 4.
[0241] As described above, the control device 3 may perform the rough movement process in addition to or instead of the robot control device 4. Therefore, after the fine movement process for the first target portion, the control device 3 may perform the rough movement process for the second target portion. In this case, the control device 3 does not need to receive a signal from the robot control device 4 to perform the rough movement process for the second target portion. In other words, the control device 3 may perform the rough movement process for the second target portion even if the control device 3 does not receive a signal from the robot control device 4. In other words, the control device 3 may perform the rough movement process for the second target portion even if the robot control device 4 does not send a signal to the control device 3. The control device 3 may start the rough movement process for the second target portion without sending a completion notification signal to the robot control device 4 in step S35 of FIG. 9 . In this case, the time required to perform the rough movement process for the second target portion is shortened compared to when the control device 3 needs to receive a signal from the robot control device 4 to perform the rough movement process for the second target portion. This is because the control device 3 does not need to wait to start the rough movement process for the second target portion until it receives a signal from the robot control device 4.
[0242] However, when the control device 3 performs the rough movement process on the second target portion, before the control device 3 performs the rough movement process on the second target portion, the robot control device 4 transmits a control permission signal to the control device 3 (step S42 in FIG. 9 ), and the control device 3 receives a control permission signal from the robot control device 4 (step S31 in FIG. 9 ). Thereafter, the control device 3 may start the rough movement process on the second target portion.
[0243] The control device 3 may perform rough movement processing on the first target portion in addition to or instead of the rough movement processing on the second target portion. Since the operation of the control device 3 performing the rough movement processing on the first target portion may be the same as the operation of the control device 3 performing the rough movement processing on the second target portion, detailed description thereof will be omitted to avoid duplication.
[0244] After the rough movement process for the second target portion, the robot control device 4 transmits a control permission signal to the control device 3 (step S42 in FIG. 9 ), and the control device 3 receives the control permission signal from the robot control device 4 (step S31 in FIG. 9 ). The control device 3 may then perform fine movement process for the second target portion (step S32 in FIG. 9 ). For example, the control device 3 may acquire image data IMG generated by the imaging device 21 capturing an image of the target object OBJ (particularly, the second target portion) after the rough movement process for the second target portion. The control device 3 may then calculate at least one of the position and orientation of the target object OBJ (particularly, at least one of the position and orientation of the second target portion of the target object OBJ) based on the acquired image data IMG. The control device 3 may then generate and output a control signal for controlling the robot 1 to move the end effector 5 based on the calculation result of the position and orientation of the second target portion. As a result, the end effector 5 moves under the control of the control device 3.
[0245] The control device 3 may perform the fine movement process on the second target portion until the predetermined convergence condition described above is met (step S33 in FIG. 9 ). In other words, the control device 3 may repeat the fine movement process on the second target portion until the predetermined convergence condition described above is met.
[0246] As one example, the control device 3 may perform the fine movement process on the second target portion until a position convergence condition is met, in which the positional relationship between the end effector 5 and the second target portion becomes a second target positional relationship. An example of the second target positional relationship is a positional relationship in which the end effector 5 is located at a second target processing position TPP, where the end effector 5 should be located when performing a predetermined process on the second target portion. As another example, the control device 3 may perform the fine movement process on the second target portion until a posture convergence condition is met, in which the posture relationship between the end effector 5 and the second target portion becomes a second target posture relationship. An example of the second target posture relationship is a positional relationship in which the end effector 5 assumes a second target processing posture, where the end effector 5 should be located when performing a predetermined process on the second target portion.
[0247] The second target position relationship may be the same as or different from the first target position relationship described above. The second target processing position TPP may be the same as or different from the first target processing position TPP described above. The second target attitude relationship may be the same as or different from the first target attitude relationship described above. The second target attitude may be the same as or different from the first target processing attitude described above.
[0248] If the convergence condition is met (step S33 in FIG. 9 : Yes), the control device 3 ends the fine movement process for the second target portion. In this case, the control device 3 controls the end effector 5 to perform a predetermined process for the second target portion (step S34 in FIG. 9 ). Thereafter, the control device 3 transmits a completion notification signal to the robot control device 4 (step S35 in FIG. 9 ), and the robot control device 4 receives the completion notification signal transmitted from the control device 3 (step S43 in FIG. 9 ).
[0249] (2-5-2) Second Example of Robot Control Processing in which Rough Movement Processing and Fine Movement Processing are Alternately Repeated A second example of robot control processing in which rough movement processing and fine movement processing are alternately repeated is robot control processing performed when the end effector 5 sequentially performs predetermined processing on a plurality of target objects OBJ. As an example, a robot control processing performed when the end effector 5 sequentially performs predetermined processing on a first target object OBJ and a second target object OBJ different from the first target object OBJ will be described. However, even when the end effector 5 sequentially performs predetermined processing on three or more target objects OBJ, the following robot control processing may be performed in which rough movement processing and fine movement processing are alternately repeated the number of times equal to the number of target objects OBJ.
[0250] In this case, the robot control device 4 may first perform a rough movement process on the first target object OBJ (step S41 in FIG. 9 ). For example, the robot control device 4 may generate and output a control signal for controlling the robot 1 so that the end effector 5 approaches the first target object OBJ. As a result, the end effector 5 moves under the control of the robot control device 4.
[0251] After the rough movement process for the first target object OBJ, the robot control device 4 transmits a control permission signal to the control device 3 (step S42 in FIG. 9 ), and the control device 3 receives the control permission signal from the robot control device 4 (step S31 in FIG. 9 ). The control device 3 may then perform a fine movement process for the first target object OBJ (step S32 in FIG. 9 ). For example, the control device 3 may acquire image data IMG generated by the imaging device 21 capturing an image of the first target object OBJ after the rough movement process for the first target object OBJ. The control device 3 may then calculate at least one of the position and orientation of the first target object OBJ based on the acquired image data IMG. The control device 3 may then generate and output a control signal for controlling the robot 1 to move the end effector 5 based on the calculation result of at least one of the position and orientation of the first target object OBJ. As a result, the end effector 5 moves under the control of the control device 3.
[0252] The control device 3 may perform the fine movement processing on the first target object OBJ until the above-mentioned predetermined convergence condition is met (step S33 in FIG. 9 ). That is, the control device 3 may repeat the fine movement processing on the first target object OBJ until the above-mentioned predetermined convergence condition is met. In particular, the control device 3 may repeat the fine movement processing on the first target object OBJ until the above-mentioned predetermined convergence condition is met during the period from the completion of the rough movement processing on the first target object OBJ to the start of the rough movement processing on the second target object OBJ.
[0253] As one example, the control device 3 may perform the fine movement processing on the first target object OBJ until a position convergence condition is met in which the positional relationship between the end effector 5 and the first target object OBJ becomes a third target positional relationship. An example of the third target positional relationship is a positional relationship in which the end effector 5 is located at a third target processing position TPP where the end effector 5 should be located when performing a predetermined process on the first target object OBJ. As another example, the control device 3 may perform the fine movement processing on the first target object OBJ until a posture convergence condition is met in which the posture relationship between the end effector 5 and the first target object OBJ becomes a third target posture relationship. An example of the third target posture relationship is a positional relationship in which the end effector 5 assumes a third target processing posture where the end effector 5 should be located when performing a predetermined process on the first target object OBJ.
[0254] The third target position relationship may be the same as or different from at least one of the first and second target position relationships described above. The third target processing position TPP may be the same as or different from at least one of the first and second target processing positions TPP described above. The third target attitude relationship may be the same as or different from at least one of the first and second target attitude relationships described above. The third target processing attitude may be the same as or different from at least one of the first and second target processing attitudes described above.
[0255] If the convergence condition is met (step S33 in FIG. 9 : Yes), the control device 3 ends the fine movement process for the first target object OBJ. In this case, the control device 3 controls the end effector 5 to perform a predetermined process on the first target object OBJ (step S34 in FIG. 9 ).
[0256] Thereafter, the control device 3 transmits a completion notification signal to the robot control device 4 (step S35 in FIG. 9 ), and the robot control device 4 receives the completion notification signal transmitted from the control device 3 (step S43 in FIG. 9 ). Thereafter, the robot control device 4 may perform a rough movement process on the second target object OBJ (step S41 in FIG. 9 ). For example, the robot control device 4 may generate and output a control signal for controlling the robot 1 so that the end effector 5 moves from the first target object OBJ toward the second target object OBJ. As a result, the end effector 5 moves under the control of the robot control device 4.
[0257] As described above, the control device 3 may perform the rough movement processing in addition to or instead of the robot control device 4. Therefore, after the fine movement processing on the first target object OBJ, the control device 3 may perform the rough movement processing on the second target object OBJ. In this case, the control device 3 does not need to receive a signal from the robot control device 4 to perform the rough movement processing on the second target object OBJ. In other words, the control device 3 may perform the rough movement processing on the second target object OBJ even if the control device 3 does not receive a signal from the robot control device 4. In other words, the control device 3 may perform the rough movement processing on the second target object OBJ even if the robot control device 4 does not send a signal to the control device 3. The control device 3 may start the rough movement processing on the second target object OBJ without sending a completion notification signal to the robot control device 4 in step S35 of FIG. 9 . In this case, the time required to perform the rough movement processing on the second target object OBJ is shortened compared to when the control device 3 needs to receive a signal from the robot control device 4 to perform the rough movement processing on the second target object OBJ. This is because the control device 3 does not need to wait until it receives a signal from the robot control device 4 before starting the rough movement process for the second target object OBJ.
[0258] However, when the control device 3 performs the rough movement processing on the second target object OBJ, before the control device 3 performs the rough movement processing on the second target object OBJ, the robot control device 4 transmits a control permission signal to the control device 3 (step S42 in FIG. 9 ), and the control device 3 receives a control permission signal from the robot control device 4 (step S31 in FIG. 9 ). Thereafter, the control device 3 may start the rough movement processing on the second target object OBJ.
[0259] The control device 3 may perform rough movement processing on the first target object OBJ in addition to or instead of the rough movement processing on the second target object OBJ. The operation of the control device 3 performing the rough movement processing on the first target object OBJ may be the same as the operation of the control device 3 performing the rough movement processing on the second target object OBJ, and therefore detailed description thereof will be omitted to avoid redundant description.
[0260] After the rough movement process for the second target object OBJ, the robot control device 4 transmits a control permission signal to the control device 3 (step S42 in FIG. 9 ), and the control device 3 receives the control permission signal from the robot control device 4 (step S31 in FIG. 9 ). The control device 3 may then perform a fine movement process for the second target object OBJ (step S32 in FIG. 9 ). For example, the control device 3 may acquire image data IMG generated by the imaging device 21 capturing an image of the second target object OBJ after the rough movement process for the second target object OBJ. The control device 3 may then calculate at least one of the position and orientation of the second target object OBJ based on the acquired image data IMG. The control device 3 may then generate and output a control signal for controlling the robot 1 to move the end effector 5 based on the calculation result of at least one of the position and orientation of the second target object OBJ. As a result, the end effector 5 moves under the control of the control device 3.
[0261] The control device 3 may perform the fine movement process on the second target object OBJ until the predetermined convergence condition described above is met (step S33 in FIG. 9 ). In other words, the control device 3 may repeat the fine movement process on the second target object OBJ until the predetermined convergence condition described above is met.
[0262] As one example, the control device 3 may perform the fine movement processing on the second target object OBJ until a position convergence condition is satisfied, in which the positional relationship between the end effector 5 and the second target object OBJ becomes a fourth target positional relationship. An example of the fourth target positional relationship is a positional relationship in which the end effector 5 is located at a fourth target processing position TPP, where the end effector 5 should be located when performing a predetermined process on the second target object OBJ. As another example, the control device 3 may perform the fine movement processing on the second target object OBJ until a posture convergence condition is satisfied, in which the posture relationship between the end effector 5 and the second target object OBJ becomes a fourth target posture relationship. An example of the fourth target posture relationship is a positional relationship in which the end effector 5 assumes a fourth target processing posture, where the end effector 5 should be located when performing a predetermined process on the second target object OBJ.
[0263] The fourth target position relationship may be the same as or different from at least one of the first to third target position relationships described above. The fourth target processing position TPP may be the same as or different from at least one of the first to third target processing positions TPP described above. The fourth target attitude relationship may be the same as or different from at least one of the first to third target attitude relationships described above. The fourth target attitude may be the same as or different from at least one of the first to third target processing attitudes described above.
[0264] If the convergence condition is met (step S33 in FIG. 9 : Yes), the control device 3 ends the fine movement process for the second target object OBJ. In this case, the control device 3 controls the end effector 5 to perform a predetermined process on the second target object OBJ (step S34 in FIG. 9 ). Thereafter, the control device 3 transmits a completion notification signal to the robot control device 4 (step S35 in FIG. 9 ), and the robot control device 4 receives the completion notification signal transmitted from the control device 3 (step S43 in FIG. 9 ).
[0265] (3) Technical Effects As described above, in this embodiment, the robot system SYS can perform robot control processing including rough movement processing and fine movement processing. As a result, the robot system SYS can move the end effector 5 efficiently and with high accuracy, compared to when robot control processing including only rough movement processing or only fine movement processing is performed.
[0266] Specifically, as described above, the rough movement process is a process for roughly moving the end effector 5 without using image data IMG. Therefore, a robot control process that includes only the rough movement process does not necessarily include a process for calculating at least one of the position and orientation of the target object OBJ based on image data IMG. Therefore, while a robot control process that includes only the rough movement process reduces the time required to move the end effector 5, the difference between the position of the end effector 5 and the target processing position TPP described above may become so large as to exceed the tolerance. Similarly, the difference between the orientation of the end effector 5 and the target processing orientation described above may become so large as to exceed the tolerance. On the other hand, the fine movement process is a process for moving the end effector 5 with high precision using the results of calculating at least one of the position and orientation of the target object OBJ based on image data IMG, as described above. However, because it is necessary to calculate at least one of the position and orientation of the target object OBJ based on image data IMG, the time required to move the end effector 5 using only the fine movement process is longer than the time required to move the end effector 5 using only the rough movement process. Therefore, with a robot control process including only the fine movement process, the difference between the position of the end effector 5 and the target processing position TPP described above is likely to become so small as to fall below the tolerance, and the difference between the posture of the end effector 5 and the target processing posture described above is likely to become so small as to fall below the tolerance. However, the time required to move the end effector 5 is increased. However, in this embodiment, the robot system SYS performs the rough movement process to roughly move the end effector 5, and then performs the fine movement process to move the end effector 5 with high precision. Therefore, compared to when a robot control process including only the fine movement process is performed, the time required to move the end effector 5 is reduced. In other words, the robot system SYS can move the end effector 5 efficiently. Furthermore, compared to when a robot control process including only the rough movement process is performed, the robot system SYS can move the end effector 5 with high precision.
[0267] Furthermore, in this embodiment, as described above, the control device 3 may repeat the fine movement process multiple times. In this case, as the end effector 5 (image capture device 21) approaches the target object OBJ each time the fine movement process is repeated, the accuracy of calculating at least one of the position and orientation of the target object OBJ gradually increases, as described above. As a result, the movement accuracy of the end effector 5, which moves in accordance with a control signal generated based on the calculation result of at least one of the position and orientation of the target object OBJ, also increases. Compared to a case in which the fine movement process is performed only once between receiving a control permission signal and transmitting a completion notification signal, the robot system SYS can move the end effector 5 with high accuracy. In other words, the robot system SYS can move the end effector 5 with high accuracy so that it can accurately reach the target processing position TPP.
[0268] In particular, in this embodiment, after the rough movement process, a control permission signal is sent from the robot control device 4 to the control device 3. Thereafter, the control device 3 starts the fine movement process. In other words, when the control permission signal is sent from the robot control device 4 to the control device 3, control of the robot 1 is passed from the robot control device 4 to the control device 3. In this case, the control device 3 can perform the fine movement process without receiving any signal from the robot control device. In particular, the control device 3 can perform the fine movement process multiple times without receiving any signal from the robot control device 4. In other words, the control device 3 can perform the fine movement process even if it does not receive any signal from the robot control device 4. In particular, the control device 3 can perform the fine movement process multiple times even if it does not receive any signal from the robot control device 4. For example, the control device 3 can repeat the fine movement process even if it does not receive any signal (e.g., a control permission signal) for each fine movement process. For this reason, the time required to perform the fine movement processing is reduced compared to a comparative example in which the control device 3 does not generate a control signal based on the calculation results of at least one of the position and orientation of the target object OBJ by the control device 3, but the robot control device 4 generates a control signal based on the calculation results of at least one of the position and orientation of the target object OBJ by the control device 3, and moves the end effector 5 based on the generated control signal. In other words, the time required to perform the fine movement processing is reduced compared to a comparative example in which part of the fine movement processing is performed by the control device 3 and another part of the fine movement processing is performed by the robot control device 4. In particular, the time required to perform the fine movement processing multiple times is reduced. The reasons for this are as follows.
[0269] In a comparative example in which the robot control device 4 generates a control signal based on the calculation result of at least one of the position and orientation of the target object OBJ by the control device 3 without generating a control signal based on the calculation result of at least one of the position and orientation of the target object OBJ by the control device 3, and moves the end effector 5 based on the generated control signal, it is preferable that the process of moving the end effector 5 by the robot control device 4 and the process of calculating at least one of the position and orientation of the target object OBJ by the control device 3 are performed synchronously. For this reason, it is generally desirable that the control device 3, which has calculated at least one of the position and orientation of the target object OBJ, is not permitted to recalculate at least one of the position and orientation of the target object OBJ until after receiving a signal transmitted from the robot control device 4 to the control device 3 as a substantial synchronization signal. An example of a signal transmitted from the robot control device 4 to the control device 3 is a signal notifying the control device 3 that the robot control device 4 has moved the end effector 5 based on the calculation result of at least one of the position and orientation of the target object OBJ by the control device 3. For this reason, after calculating at least one of the position and orientation of the target object OBJ once, the control device 3 must wait without performing the process of calculating at least one of the position and orientation of the target object OBJ again until it receives a signal transmitted from the robot control device 4 to the control device 3. However, in this embodiment, both the process of calculating at least one of the position and orientation of the target object OBJ and the process of moving the end effector 5 based on the calculation result of at least one of the position and orientation of the target object OBJ (specifically, the process of generating a control signal based on the calculation result of at least one of the position and orientation of the target object OBJ) are performed by the same control device 3.For this reason, in this embodiment, the control device 3 can synchronously perform the process of calculating at least one of the position and orientation of the target object OBJ and the process of moving the end effector 5 based on the calculation result of at least one of the position and orientation of the target object OBJ (specifically, the process of generating a control signal based on the calculation result of at least one of the position and orientation of the target object OBJ) even when the control device 3 does not receive a signal transmitted from the robot control device 4 to the control device 3. For this reason, in this embodiment, the control device 3 does not need to wait until it receives a signal transmitted from the robot control device 4 to the control device 3, thereby reducing the time required to perform the fine movement process (particularly the time required to perform the fine movement process multiple times).
[0270] The effect of reducing the time required to perform the fine movement process multiple times becomes greater the more frequently the fine movement process is performed. For example, if the number of times the fine movement process is repeated is N (where N is an integer greater than or equal to 2), and the time required in the comparative example from when the control device 3 transmits the calculation results of at least one of the position and orientation of the target object OBJ to the robot control device 4 until the control device 3 receives the signal transmitted from the robot control device 4 to the control device 3 (i.e., the standby time of the control device 3) is T, the time required to perform the fine movement process N times in this embodiment is shorter by T x N compared to the time required to perform the fine movement process N times in the comparative example. Therefore, the effect of reducing the time required to perform the fine movement process multiple times in this embodiment becomes greater the more frequently the fine movement process is performed.
[0271] Additionally, in this embodiment, when the rough movement process and the fine movement process are alternately performed as described above, the control device 3 may also perform the rough movement process after performing the fine movement process. For example, as described in the first example of the robot control process in which the rough movement process and the fine movement process are alternately repeated, the control device 3 may perform the fine movement process on a first portion of the target object OBJ and then perform the rough movement process on a second portion of the target object OBJ. For example, as described in the second example of the robot control process in which the rough movement process and the fine movement process are alternately repeated, the control device 3 may perform the fine movement process on a first target object OBJ and then perform the rough movement process on a second target object OBJ. In this case, the time required to perform the rough movement process on the second target portion or the second target object is particularly shortened. This is because the control device 3 can start the rough movement process on the second target portion or the second target object OBJ even if the control device 3 does not receive a signal sent to the control device 3 from the robot control device 4 after performing the fine movement process on the first target portion or the first target object OBJ.
[0272] In this way, the robot system SYS of this embodiment can move the end effector 5 efficiently, also in terms of being able to reduce the time required to perform at least one of the rough movement processing and the fine movement processing.
[0273] (4) Modifications Next, modifications of the robot system SYS will be described.
[0274] (4-1) First Modification (4-1-1) Preparatory Processing As shown in FIG. 12 , which shows the flow of robot control processing in the first modification, in the first modification, the control device 3 may perform preparatory processing for performing fine movement processing before the robot control device 4 completes (i.e., ends) the rough movement processing (step S30a). Specifically, the control device 3 may perform preparatory processing while the robot control device 4 is performing the rough movement processing (step S30a). The control device 3 may perform preparatory processing before the control device 3 receives a control permission signal from the robot control device 4 (step S30a). The control device 3 may perform preparatory processing before the control device 3 starts the fine movement processing (step S30a).
[0275] The preparation process may include a pre-processing that is performed to start the fine movement process. In particular, the preparation process may include a pre-processing that the control device 3 can perform in advance even if the control device 3 has not received a control permission signal from the robot control device 4, in order to start the fine movement process after the control device 3 receives a control permission signal.
[0276] When the control device 3 performs preparatory processing before the robot control device 4 completes the rough movement processing, the time required from when the robot control device 4 completes the rough movement processing until the control device 3 starts the fine movement processing (in other words, the time required for the control device 3 to complete the fine movement processing) is shortened compared to when the control device 3 does not perform preparatory processing before the robot control device 4 completes the rough movement processing.
[0277] (4-1-1-1) First Example of Preparatory Processing The preparatory processing may include processing to read work information, which is information used to perform the fine movement processing. In other words, the preparatory processing may include processing to read work information, which is information that can be used to perform the fine movement processing.
[0278] The work information may include target object information, which is information about the target object OBJ on which a predetermined process is to be performed by the end effector 5. The target object information may include information about the type of the target object OBJ.
[0279] The work information may include peripheral object information, which is information about peripheral objects located around a target object OBJ on which the end effector 5 performs a predetermined process. The peripheral object information may include information about at least one of the position, orientation, shape, and size of the peripheral object. Such peripheral object information may be used to generate a control signal for controlling the robot 1 to move the end effector 5 while avoiding interference of the end effector 5 with the peripheral objects indicated by the peripheral object information.
[0280] The work information may include matching information used in a matching process for detecting the target object OBJ. The matching information may include at least one of information regarding the model data MDL, information regarding a matching judgment threshold, and information regarding a convergence condition. That is, the matching information may include at least one of information specifying the model data MDL, information specifying a matching judgment threshold, and information specifying a convergence condition. The information regarding the convergence condition may include at least one of information specifying a target positional relationship, information specifying a target posture relationship, information specifying a target processing position TPP, information specifying a target processing posture, an upper tolerance threshold that determines the target positional relationship (i.e., an upper tolerance value for the position of the end effector 5), and an upper tolerance threshold that determines the target posture relationship (i.e., an upper tolerance value for the posture of the end effector 5).
[0281] The work information may include imaging condition information, which is information related to the imaging conditions of the imaging device 21. That is, the work information may include imaging condition information, which is information specifying the imaging conditions of the imaging device 21. The imaging condition information may include at least one of information related to the exposure time and information related to the intensity of the illumination light. That is, the imaging condition information may include at least one of information specifying the exposure time and information specifying the intensity of the illumination light.
[0282] As described above, when the control device 3 performs predetermined image processing on an image represented by image data IMG acquired from the imaging device 21, the work information may include image processing condition information, which is information related to image processing conditions. That is, the work information may include image processing condition information, which is information specifying image processing conditions. The image processing condition information may include at least one of information related to gamma correction processing conditions, information related to HDR processing conditions, and information related to noise reduction processing conditions. That is, the image processing condition information may include at least one of information specifying gamma correction processing conditions, information specifying HDR processing conditions, and information specifying noise reduction processing conditions.
[0283] The control device 3 may read the work information from a storage device 32 provided in the control device 3. The control device 3 may read the work information from a recording medium that is external to or built into the control device 3. The control device 3 may read the work information from a device (not shown) that is located outside the control device 3 (in this case, the work information may be downloaded).
[0284] In this way, when the preparation process includes a process for reading work information, the control device 3 reads the work information before the robot control device 4 completes the rough movement process. Here, a certain amount of time is required to read the work information. Therefore, if the control device 3 completes reading the work information before the robot control device 4 completes the rough movement process, the control device 3 does not need to read the work information after the robot control device 4 completes the rough movement process. Alternatively, even if the control device 3 has not completed reading the work information before the robot control device 4 completes the rough movement process, the amount of work information that the control device 3 needs to read after the robot control device 4 completes the rough movement process is reduced compared to when the work information is not read before the robot control device 4 completes the rough movement process. This reduces the time required from when the robot control device 4 completes the rough movement process to when the fine movement process actually begins (in other words, the time required for the control device 3 to complete the fine movement process).
[0285] (4-1-1-2) Second Example of Preparatory Processing If it is possible to perform auto exposure (AE) processing for adjusting the exposure conditions of the imaging device 21, the preparatory processing may include processing for starting auto exposure processing related to the auto exposure function. The preparatory processing may include processing for adjusting the exposure conditions in advance by starting the auto exposure processing. An example of the exposure conditions is at least one of an aperture value and a shutter speed.
[0286] Thus, when the preparation process includes processing for starting the automatic exposure process, the control device 3 starts the automatic exposure process before the robot control device 4 completes the rough movement process. Here, a certain amount of time is required from the start of the automatic exposure process until the adjustment of the exposure conditions is actually completed. Therefore, if the control device 3 starts the automatic exposure process and completes the adjustment of the exposure conditions before the robot control device 4 completes the rough movement process, the control device 3 does not need to start the automatic exposure process and adjust the exposure conditions after the robot control device 4 completes the rough movement process. Alternatively, even if the control device 3 does not start the automatic exposure process and complete the adjustment of the exposure conditions before the robot control device 4 completes the rough movement process, the time required from the completion of the rough movement process by the robot control device 4 to the completion of the adjustment of the exposure conditions is shortened compared to when the automatic exposure process is not started before the robot control device 4 completes the rough movement process. As a result, the time the control device 3 waits until the adjustment of the exposure conditions is completed after the robot control device 4 completes the rough movement process is shortened. As a result, the time required from when the robot control device 4 completes the rough movement processing to when it actually starts the fine movement processing using the imaging device 21 whose exposure conditions have been adjusted (in other words, the time required for the control device 3 to complete the fine movement processing) is shortened.
[0287] (4-1-1-3) Third Example of Preparatory Processing The preparatory processing may include a process of controlling the illumination device 23 to start irradiating the target object OBJ with illumination light. In other words, the preparatory processing may include a process of controlling the illumination device 23 to start emitting illumination light.
[0288] Thus, when the preparation process includes a process for controlling the illumination device 23, the illumination device 23 starts emitting illumination light before the robot control device 4 completes the rough movement process. Here, a certain amount of time is required for the illumination light intensity to stabilize (specifically, for the illumination light intensity to reach a certain intensity) after the illumination device 23 starts emitting illumination light. Therefore, if the illumination light intensity stabilizes before the robot control device 4 completes the rough movement process, the control device 3 does not need to wait until the illumination light intensity stabilizes. Alternatively, even if the illumination light intensity is not stable before the robot control device 4 completes the rough movement process, the time required for the illumination light intensity to stabilize after the robot control device 4 completes the rough movement process is shortened compared to when the illumination device 23 is not controlled before the robot control device 4 completes the rough movement process. As a result, the time the control device 3 waits until the illumination light intensity stabilizes after the robot control device 4 completes the rough movement process is shortened. Therefore, the time required for the robot control device 4 to actually start the fine movement process using the illumination device 23 capable of emitting illumination light at a stable intensity after the robot control device 4 completes the rough movement process is shortened.
[0289] (4-1-1-4) Other Examples of Timing for Performing Preparatory Processing The control device 3 may perform preparatory processing after the robot control device 4 has completed the rough movement processing and before the control device 3 receives a control permission signal transmitted by the robot control device 4. Even in this case, the time required from receiving the control permission signal to starting the fine movement processing is reduced compared to when the control device 3 does not perform preparatory processing before receiving the control permission signal.
[0290] Furthermore, even when the control device 3 performs both the rough movement process and the fine movement process, the control device 3 may perform the preparatory process before completing the rough movement process. Even in this case, the time required from when the control device 3 completes the rough movement process until when the control device 3 starts the fine movement process is reduced compared to when the control device 3 does not perform the preparatory process before completing the rough movement process.
[0291] Furthermore, even if the robot control device 4 does not perform the rough movement processing, the control device 3 may perform the preparatory processing before starting the fine movement processing. In other words, if the control device 3 starts the fine movement processing without the robot control device 4 performing the rough movement processing, the control device 3 may perform the preparatory processing before starting the fine movement processing. Even in this case, the time required from when the control device 3 completes the fine movement processing to when the control device 3 starts the fine movement processing is reduced compared to when the control device 3 does not perform the preparatory processing before starting the fine movement processing.
[0292] (4-1-2) Advance Execution of At Least Part of the Fine Movement Process The control device 3 may perform at least part of the fine movement process in advance, in addition to or instead of performing preparatory processing before the robot control device 4 completes (i.e., terminates) the rough movement process. The control device 3 may perform at least part of the fine movement process in advance, in addition to or instead of performing preparatory processing before the control device 3 receives a control permission signal from the robot control device 4. The control device 3 may perform at least part of the fine movement process in advance, in addition to or instead of performing preparatory processing while the robot control device 4 is performing the rough movement process. In other words, at least part of the rough movement process and at least part of the fine movement process may be performed in parallel. As a result, the time required to complete a series of robot control processes including the rough movement process and the fine movement process is reduced compared to when the control device 3 does not perform the fine movement process before the robot control device 4 completes the rough movement process.
[0293] As a first example, the control device 3 may perform a process of acquiring image data IMG generated by the imaging device 21 from the imaging device 21 as at least part of the fine movement process before the robot control device 4 completes the rough movement process. Note that the image data IMG may be considered to be information used to move the end effector 5 through the fine movement process. In this case, the control device 3 may consider the process of acquiring image data IMG generated by the imaging device 21 from the imaging device 21 before the robot control device 4 completes the rough movement process to be performed as at least part of the above-mentioned pre-processing.
[0294] As described above, when the fine movement processing includes processing in which the imaging device 21 generates image data by capturing an image of the target object OBJ, the control device 3 may perform processing in which the imaging device 21 generates image data by capturing an image of the target object OBJ as at least a part of the fine movement processing before the robot control device 4 completes the rough movement processing. In other words, the imaging device 21 may perform processing in which the imaging device 21 generates image data by capturing an image of the target object OBJ as at least a part of the fine movement processing before the robot control device 4 completes the rough movement processing.
[0295] As a second example, the control device 3 may perform a process of calculating at least one of the position and orientation of the target object OBJ based on the image data IMG as at least part of the fine movement process before the robot control device 4 completes the rough movement process. In particular, the control device 3 may perform a process of calculating at least one of the position and orientation of the target object OBJ based on the image data IMG generated by the imaging device 21 capturing an image of the target object OBJ before the robot control device 4 completes the rough movement process. Note that the calculation result of at least one of the position and orientation of the target object OBJ may be considered to be information used to move the end effector 5 through the fine movement process. In this case, the control device 3 may consider the process of calculating at least one of the position and orientation of the target object OBJ before the robot control device 4 completes the rough movement process to be performed as at least part of the above-described pre-processing.
[0296] Here, even before the robot control device 4 completes the rough movement processing, the shorter the time difference between the imaging time when the imaging device 21 images the target object OBJ and the rough movement completion time when the robot control device 4 completes the rough movement processing, the smaller the difference between the position of the target object OBJ at the imaging time and the position of the target object OBJ at the rough movement completion time. In other words, the closer the imaging time is to the rough movement processing, the smaller the difference between the position of the target object OBJ at the imaging time and the position of the target object OBJ at the rough movement completion time. Similarly, the difference between the orientation of the target object OBJ at the imaging time and the orientation of the target object OBJ at the rough movement completion time becomes smaller. Therefore, even if at least one of the position and orientation of the target object OBJ is calculated before the robot control device 4 completes the rough movement process, if the time difference between the image capture time and the rough movement completion time is shorter than a predetermined time threshold, the calculation result of at least one of the position and orientation of the target object OBJ calculated before the robot control device 4 completes the rough movement process may be considered to indicate at least one of the position and orientation of the target object OBJ at the rough movement completion time with a reasonable degree of accuracy. Therefore, if the time difference between the image capture time and the rough movement completion time is shorter than a predetermined time threshold, the control device 3 may calculate at least one of the position and orientation of the target object OBJ based on image data IMG generated by the imaging device 21 capturing an image of the target object OBJ before the robot control device 4 completes the rough movement process. For the same reason, if the time difference between the image capture time and the rough movement completion time is shorter than a predetermined time threshold, the control device 3 may acquire image data IMG from the imaging device 21 before the robot control device 4 completes the rough movement process. Furthermore, before the robot control device 4 completes the rough movement process, the imaging device 21 may capture an image of the target object OBJ and generate image data IMG at an imaging time that is shorter than a predetermined time threshold from the rough movement completion time.
[0297] Furthermore, if the target object OBJ is stationary, the position and orientation of the target object OBJ at the image capture time should match the position and orientation of the target object OBJ at the time the rough movement is completed. Therefore, if the target object OBJ is stationary, the control device 3 may calculate at least one of the position and orientation of the target object OBJ based on the image data IMG generated by the imaging device 21 capturing an image of the target object OBJ before the robot control device 4 completes the rough movement process.
[0298] Alternatively, if the distance traveled by the target object OBJ from the time the imaging device 21 captures an image of the target object OBJ until the rough movement process is completed is smaller than a predetermined distance threshold, the calculation results of at least one of the position and orientation of the target object OBJ calculated before the robot control device 4 completes the rough movement process may be considered to indicate, with a reasonable degree of accuracy, at least one of the position and orientation of the target object OBJ at the time the rough movement process is completed. Therefore, if the distance traveled by the target object OBJ from the time the imaging device 21 captures an image of the target object OBJ until the rough movement process is completed is a predetermined distance threshold, the control device 3 may calculate at least one of the position and orientation of the target object OBJ based on image data IMG generated by the imaging device 21 capturing an image of the target object OBJ before the robot control device 4 completes the rough movement process.
[0299] As a third example, the control device 3 may perform a process to generate a control signal for controlling the robot 1 to move the end effector 5 after the rough movement process is completed, based on at least one of the calculation results of the position and posture of the target object OBJ, before the robot control device 4 completes the rough movement process.
[0300] As described above, if the time difference between the image capture time and the rough movement completion time is shorter than a predetermined time threshold, the calculation results of at least one of the position and orientation of the target object OBJ calculated before the robot control device 4 completes the rough movement process may be considered to indicate, with a reasonable degree of accuracy, at least one of the position and orientation of the target object OBJ at the rough movement completion time. In this case, if the time difference between the image capture time and the rough movement completion time is shorter than a predetermined time threshold before the robot control device 4 completes the rough movement process, the control device 3 may perform processing to generate a control signal for controlling the robot 1 to move the end effector 5 after the rough movement process is completed. The control device 3 may output the control signal generated by the control device 3 before the robot control device 4 completes the rough movement process to the robot 1 or the robot control device 4 after the robot control device 4 completes the rough movement process. In this case, after the robot control device 4 completes the rough movement process, the robot 1 may move the end effector 5 based on the control signal generated by the control device 3 before the robot control device 4 completes the rough movement process.
[0301] Alternatively, the control device 3 may output a control signal generated by the control device 3 before the robot control device 4 completes the rough movement processing to the robot 1 or the robot control device 4 before the robot control device 4 completes the rough movement processing. In this case, before the robot control device 4 completes the rough movement processing, the robot 1 may move the end effector 5 based on the control signal generated by the control device 3 before the robot control device 4 completes the rough movement processing. In other words, the control device 3 may move the end effector 5 by performing the fine movement processing during the period when the robot control device 4 is moving the end effector 5 by performing the rough movement processing.
[0302] However, there is a possibility that a conflict may occur between the process of actually moving the end effector 5 by the robot control device 4 performing the rough movement process and the process of actually moving the end effector 5 by the control device 3 performing the fine movement process. In this case, the control device 3 may output a control signal generated by the control device 3 before the robot control device 4 completes the rough movement process to the robot 1 or the robot control device 4 after the robot control device 4 completes the rough movement process and before the control device 3 receives the control permission signal transmitted by the robot control device 4. In this case, after the robot control device 4 completes the rough movement process and before the control device 3 receives the control permission signal transmitted by the robot control device 4, the robot 1 may move the end effector 5 based on the control signal generated by the control device 3 before the robot control device 4 completes the rough movement process. As a result, there is no possibility of a conflict between the process of actually moving the end effector 5 by the robot control device 4 performing the rough movement process and the process of actually moving the end effector 5 by the control device 3 performing the fine movement process.
[0303] The control device 3 may perform at least a part of the fine movement processing after the robot control device 4 has completed the rough movement processing and before receiving a control permission signal transmitted by the robot control device 4. Even in this case, the time required to complete a series of robot control processing including the rough movement processing and the fine movement processing is reduced compared to when the control device 3 does not perform the fine movement processing before receiving the control permission signal.
[0304] Furthermore, even before the robot control device 4 completes the rough movement process, if the rough movement process will be completed within a certain time, the robot control device 4 may send a control permission signal to the control device 3. In this case, the control device 3 may receive the control permission signal even before the robot control device 4 completes the rough movement process. As a result, the control device 3 may perform at least a part of the fine movement process in response to the reception of the control permission signal. Even in this case, the control device 3 can start the fine movement process at an earlier timing than when the robot control device 4 does not send a control permission signal to the control device 3 before the robot control device 4 completes the rough movement process (i.e., the control device 3 does not receive the control permission signal before the robot control device 4 completes the rough movement process). As a result, the time required to complete a series of robot control processes including the rough movement process and the fine movement process is shortened.
[0305] Furthermore, even when the control device 3 performs both the rough movement processing and the fine movement processing, the control device 3 may perform at least a part of the fine movement processing before completing the rough movement processing. Even in this case, the time required to complete a series of robot control processing including the rough movement processing and the fine movement processing is reduced compared to when the control device 3 does not perform the fine movement processing before completing the rough movement processing.
[0306] (4-2) Second Modification When a robot control process that alternately repeats rough movement processing and fine movement processing is performed as described above, in a second modification, the control device 3 may generate a control signal used in the fine movement processing that is performed after the rough movement processing, based on a control signal generated by the fine movement processing that is performed before the rough movement processing. Below, an example of a method for generating a control signal in the second modification will be described with reference to an example of a robot control process that alternately repeats rough movement processing and fine movement processing.
[0307] (4-2-1) First Example of Method for Generating Control Signals in Second Modification A first example of a robot control process that alternately repeats rough movement processing and fine movement processing is, as described above, a robot control process that is performed when the end effector 5 sequentially performs predetermined processing on a plurality of target portions of the target object OBJ. For example, a first example of a robot control process that alternately repeats rough movement processing and fine movement processing is, as described above, a robot control process that is performed when the end effector 5 sequentially performs predetermined processing on a first target portion of the target object OBJ (for example, the portion where hole HL#1 shown in FIGS. 5 and 6 is formed) and a second target portion of the target object OBJ that is different from the first target portion (for example, the portion where hole HL#2 shown in FIGS. 5 and 6 is formed).
[0308] In the following explanation, a first example of a method for generating a control signal in the second variant will be described with reference to a first example of robot control processing performed when the end effector 5 sequentially performs predetermined processing on a first target portion TP#1 (specifically, the portion where hole HL#1 is formed) of the target object OBJ (workpiece W2) and a second target portion TP#2 (specifically, the portion where hole HL#2 is formed) of the target object OBJ that is different from the first target portion TP#1, as shown in Figure 13.
[0309] In this case, as already described above, the robot control device 4 performs a rough movement process on the first target portion TP#1, and then, as shown in the first row of FIG. 13 , the control device 3 performs a fine movement process on the first target portion TP#1, and then, as shown in the second row of FIG. 13 , the robot control device 4 performs a rough movement process on the second target portion TP#2, and then, as shown in the second row of FIG. 13 , the control device 3 performs a fine movement process on the second target portion TP#2. In this case, in the second modified example, the control device 3 may generate a control signal used in the fine movement process on the second target portion TP#2 based on the control signal generated by the fine movement process on the first target portion TP#1. In other words, the control device 3 may perform a process of generating a control signal based on the control signal generated by the fine movement process on the first target portion TP#1 as at least a part of the fine movement process on the second target portion TP#2. The fine movement process on the second target portion TP#2 may include a process of generating a control signal based on the control signal generated by the fine movement process on the first target portion TP#1.
[0310] Specifically, the control device 3 performs fine movement processing on the first target portion TP#1 during a first period from the completion of the rough movement processing on the first target portion TP#1 to the start of the rough movement processing on the second target portion TP#2. Furthermore, the control device 3 performs fine movement processing on the second target portion TP#2 during a second period after the completion of the rough movement processing on the second target portion TP#2. In this case, the control device 3 may generate control signals during the second period based on the control signals generated during the first period. The control device 3 may generate control signals used to control the robot 1 during the second period based on the control signals generated during the first period.
[0311] In particular, the control device 3 may generate the first control signal in the second period based on the control signal generated last in the first period. For example, an example will be described in which the control device 3 generates N1 control signals in the first period by repeating the fine movement process for the first target portion TP#1 N1 times (where N1 is a variable indicating an integer greater than or equal to 2) and generates N2 control signals in the second period by repeating the fine movement process for the second target portion TP#2 N2 times (where N2 is a variable indicating an integer greater than or equal to 2). In the following description, the N1 control signals generated in the first period will be referred to as control signal #1[1] to control signal #1[N1], respectively. Control signal #1[i] (where i is a variable indicating an integer greater than or equal to 1 and less than or equal to N1) refers to the control signal generated in the i-th fine movement process performed on the first target portion TP#1. Similarly, in the following description, the N2 control signals generated in the second period will be referred to as control signal #2[1] to control signal #2[N2], respectively. Control signal #2[j] (where j is a variable indicating an integer greater than or equal to 1 and less than or equal to N2) indicates a control signal generated in the jth fine movement process performed on the second target portion TP#2. In this case, the control device 3 may generate control signal #2[1] based on control signal #1[N1].
[0312] In this way, when the control signal #2[1] is generated based on the control signal #1[N1], the time required from the start of the fine movement process for the second target portion TP#2 until the convergence condition is satisfied is shorter than when the control signal #2[1] is generated without using the control signal #1[N1]. The reason for this is explained below. First, the control signal #1[N1] is the most recent control signal generated at the time it was determined that the convergence condition was satisfied during the first period in which the fine movement process for the first target portion TP#1 was performed. Therefore, generating the control signal #2[1] based on the control signal #1[N1] is essentially equivalent to reflecting in the control signal #2[1] the control state (e.g., movement) of the robot 1 at the time it was determined that the convergence condition was satisfied. As a result, when the robot 1 is controlled using the control signal #2[1] generated based on the control signal #1[N1], the movement required to satisfy the convergence condition is reflected in the robot 1. As a result, when robot 1 is controlled using control signal #2[1] generated based on control signal #1[N1], the time required for the convergence condition to be satisfied is reduced compared to when robot 1 is controlled using control signal #2[1] generated without being based on control signal #1[N1].
[0313] This effect of reducing the time required for the convergence condition to be satisfied is particularly pronounced when the target object OBJ is moving. The reason for this is explained below. Control signal #1[N1] is the most recent control signal generated at the time it was determined that the convergence condition was satisfied during the first period in which fine movement processing is performed on the first target portion TP#1, which moves in conjunction with the movement of the target object OBJ. Therefore, generating control signal #2[1] based on control signal #1[N1] is essentially equivalent to reflecting in control signal #2[1] the control state (e.g., movement) of the robot 1 for causing the end effector 5 to follow the moving target object OBJ. In other words, generating control signal #2[1] based on control signal #1[N1] is essentially equivalent to reflecting in control signal #2[1] the movement mode of the target object OBJ and the control state (e.g., movement) of the robot 1 at the time it was determined that the convergence condition was satisfied. As a result, when the robot 1 is controlled using the control signal #2[1] generated based on the control signal #1[N1], the movement required to make the end effector 5 follow the moving target object OBJ is reflected in the robot 1. As a result, when the robot 1 is controlled using the control signal #2[1] generated based on the control signal #1[N1], the time required for the end effector 5 to follow the target object OBJ is reduced compared to when the robot 1 is controlled using the control signal #2[1] generated without being based on the control signal #1[N1], and as a result, the time required for the convergence condition to be satisfied is reduced.
[0314] The control device 3 may generate a control signal #2[j] used to control the robot 1 by the j-th fine movement process by adding a correction term based on a difference component U indicating the difference between the actual position of the end effector 5 and the target processing position and the difference between the actual posture of the end effector 5 and the target processing posture to the control signal #2[j-1] generated in the j-1th fine movement process. Note that, as described above, if the fine movement process is a process including feedback control such as P control, PI control, or PID control, the correction term may include a term calculated based on the difference component U and the proportional gain Kp. As described above, if the fine movement process is a process including feedback control such as PI control or PID control, the correction term may include a term calculated based on the difference component U and the integral gain Ki. As described above, if the fine movement process is a process including feedback control such as PID control, the correction term may include a term calculated based on the difference component U and the derivative gain Kd. Here, when j = 1, the control signal #2[0] used to generate the control signal #2[1] is not actually generated by the fine movement process. In this case, the control device 3 may use a default initial signal, which is preset as the initial value of the control signal #2[j], as the control signal #2[0]. In this case, the control device 3 may generate the control signal #2[1] by using the control signal #1[N1] as the control signal #2[0] (i.e., as the initial signal). In other words, generating the control signal #2[1] based on the control signal #1[N1] may include generating the control signal #2[1] by using the control signal #1[N1] as the initial signal, the control signal #2[0].
[0315] In the above description, the control device 3 generates the first control signal #2[1] in the second period based on the last control signal #1[N1] generated in the first period. However, the control device 3 may also generate the control signal #2[j] at any second time in the second period based on the control signal #1[i] generated at any first time in the first period. The control device 3 may also generate the control signal #2[j] used to control the robot 1 at any second time in the second period based on the control signal #1[i] generated at any first time in the first period. For example, the control device 3 may generate the control signal #2[j] by using the control signal #1[i] as the control signal #2[j-1] to which a correction term is added to generate the control signal #2[j]. Even in this case, the movement of the robot 1 in the fine movement process for the first target portion TP#1 is reflected in the control signal #2[j] used to control the robot 1 in the fine movement process for the second target portion TP#2. When the target object OBJ is moving, the control signal #2[j] used to control the robot 1 by the fine movement process for the second target portion TP#2 reflects the movement mode of the first target portion TP#1 (i.e., the movement mode of the target object OBJ) and the movement of the robot 1. Therefore, the time required for the convergence condition to be satisfied is reduced compared to when the robot 1 is controlled using the control signal #2[j] generated without being based on the control signal #1[i].
[0316] However, the closer the time when control signal #1[i] is generated to the end of the first period (i.e., the larger the variable i), the more likely it is that the movement of the robot 1 required to satisfy the convergence condition by the fine movement processing for the first target portion TP#1 will be reflected in the control signal #2[j]. Therefore, the control device 3 may generate control signal #2[j] based on control signal #1[i] that satisfies the condition that the movement of the robot 1 required to satisfy the convergence condition by the fine movement processing for the first target portion TP#1 is appropriately reflected in the control signal #2[j]. As an example, the control device 3 may generate control signal #2[j] based on control signal #1[N1-1]. As another example, the control device 3 may generate control signal #2[j] based on control signal #1[N1-2].
[0317] Furthermore, the closer the time when the control signal #2[j] is generated to the start of the second period (i.e., the smaller the variable j), the shorter the time required for the convergence condition to be satisfied after the fine movement process for the second target portion TP#2 is initiated. This is because the closer the time when the control signal #2[j] is generated to the start of the second period, the earlier the timing at which the movement of the robot 1 required to satisfy the convergence condition by the fine movement process for the first target portion TP#1 is reflected in the control signal #2[j]. For this reason, the control device 3 may generate a control signal #2[j] based on the control signal #1[i] that satisfies the condition that the time required for the convergence condition to be satisfied after the fine movement process for the second target portion TP#2 is initiated is appropriately short. As an example, the control device 3 may generate the control signal #2[2] based on the control signal #1[i]. As another example, the control device 3 may generate the control signal #2[3] based on the control signal #1[i].
[0318] In the above description, the control device 3 generates the control signal #2[j] in the second period based on a single control signal #1[i] generated in the first period. However, the control device 3 may generate the control signal #2[j] in the second period based on multiple control signals #1[i] generated in the first period. For example, the control device 3 may generate the control signal #2[j] in the second period based on the average value of the multiple control signals #1[i] generated in the first period.
[0319] Note that even if the robot control device 4 does not perform the rough movement processing, the control device 3 may generate a control signal used in the fine movement processing for the second target portion TP#2 based on a control signal generated by the fine movement processing for the first target portion TP#1. For example, even if the control device 3 performs the fine movement processing for the first target portion TP#1 and then performs the fine movement processing for the second target portion TP#2 without the robot control device 4 performing the rough movement processing for the second target portion TP#2, the control device 3 may generate a control signal used in the fine movement processing for the second target portion TP#2 based on the control signal generated by the fine movement processing for the first target portion TP#1.
[0320] (4-2-2) Second Example of Method for Generating Control Signals in Second Modification A second example of robot control processing that alternately repeats rough movement processing and fine movement processing is, as described above, robot control processing that is performed when the end effector 5 sequentially performs predetermined processing on a plurality of target objects OBJ. For example, as a second example of robot control processing that alternately repeats rough movement processing and fine movement processing, a robot control processing that is performed when the end effector 5 sequentially performs predetermined processing on a first target object OBJ and a second target object OBJ that is different from the first target object OBJ will be described.
[0321] In the following explanation, a second example of a method for generating a control signal in the second variant will be explained with reference to a second example of a robot control process performed when the end effector 5 sequentially performs predetermined processes on a first target object OBJ#1 and a second target object OBJ#2 different from the first target object OBJ#1, as shown in Figure 14.
[0322] In this case, as already described above, the robot control device 4 performs a rough movement process on the first target object OBJ#1, and then, as shown in the first row of FIG. 14 , the control device 3 performs a fine movement process on the first target object OBJ#1, and then, as shown in the second row of FIG. 14 , the robot control device 4 performs a rough movement process on the second target object OBJ#2, and then, as shown in the second row of FIG. 14 , the control device 3 performs a fine movement process on the second target object OBJ#2. In this case, in the second modified example, the control device 3 may generate a control signal used in the fine movement process for the second target object OBJ#2 based on the control signal generated by the fine movement process for the first target object OBJ#1. In other words, the control device 3 may perform a process of generating a control signal based on the control signal generated by the fine movement process for the first target object OBJ#1 as at least a part of the fine movement process for the second target object OBJ#2. The fine movement process for the second target object OBJ#2 may include a process of generating a control signal based on the control signal generated by the fine movement process for the first target object OBJ#1.
[0323] Specifically, the control device 3 performs fine movement processing on the first target object OBJ#1 during a third period from the completion of the rough movement processing on the first target object OBJ#1 to the start of the rough movement processing on the second target object OBJ#2. Furthermore, the control device 3 performs fine movement processing on the second target object OBJ#2 during a fourth period after the completion of the rough movement processing on the second target object OBJ#2. In this case, the control device 3 may generate control signals during the fourth period based on the control signals generated during the third period. The control device 3 may generate control signals used to control the robot 1 during the fourth period based on the control signals generated during the third period.
[0324] In particular, the control device 3 may generate the first control signal in the fourth period based on the control signal generated last in the third period. For example, an example will be described in which the control device 3 generates N3 control signals in the third period by repeating the fine movement process on the first target object OBJ#1 N3 times (where N3 is a variable indicating an integer greater than or equal to 2) and generates N4 control signals in the fourth period by repeating the fine movement process on the second target object OBJ#2 N4 times (where N4 is a variable indicating an integer greater than or equal to 2). In the following description, the N3 control signals generated in the third period are referred to as control signal #3[1] to control signal #3[N3], respectively. Control signal #3[m] (where m is a variable indicating an integer greater than or equal to 1 and less than or equal to N3) refers to the control signal generated in the mth fine movement process performed on the first target object OBJ#1. Similarly, in the following description, the N4 control signals generated in the fourth period will be referred to as control signal #4[1] to control signal #4[N4], respectively. Control signal #4[n] (where n is a variable indicating an integer greater than or equal to 1 and less than or equal to N4) indicates the control signal generated in the nth fine movement process performed on the second target object OBJ#2. In this case, the control device 3 may generate control signal #4[1] based on control signal #3[N3].
[0325] In this way, when the control signal #4[1] is generated based on the control signal #3[N3], the time required from the start of the fine movement process for the second target object OBJ#2 until the convergence condition is satisfied is shorter than when the control signal #4[1] is generated without using the control signal #3[N3]. The reason for this is explained below. First, the control signal #3[N3] is the most recent control signal generated at the time when it was determined that the convergence condition was satisfied during the third period in which the fine movement process for the first target object OBJ#1 was performed. Therefore, generating the control signal #4[1] based on the control signal #3[N3] is essentially equivalent to reflecting the control state (e.g., movement) of the robot 1 at the time when it was determined that the convergence condition was satisfied in the control signal #2[1]. As a result, when the robot 1 is controlled using the control signal #4[1] generated based on the control signal #3[N3], the movement required to satisfy the convergence condition is reflected in the robot 1. As a result, when robot 1 is controlled using control signal #4[1] generated based on control signal #3[N3], the time required for the convergence condition to be satisfied is reduced compared to when robot 1 is controlled using control signal #4[1] generated without being based on control signal #3[N3].
[0326] This effect of reducing the time required for the convergence condition to be satisfied is particularly noticeable when the first target object OBJ#1 and the second target object OBJ#2 are moving. In particular, this effect of reducing the time required for the convergence condition to be satisfied is particularly noticeable when the first target object OBJ#1 and the second target object OBJ#2 are moving in the same movement pattern. The reason for this is explained below. Control signal #3[N3] is the most recent control signal generated at the time it was determined that the convergence condition was satisfied during the third period in which the fine movement process was performed on the moving first target object OBJ#1. Therefore, generating control signal #4[1] based on control signal #3[N3] is essentially equivalent to reflecting, in control signal #4[1], the control state (e.g., movement) of the robot 1 for causing the end effector 5 to follow the moving first target object OBJ#1. In other words, generating the control signal #4[1] based on the control signal #3[N3] is essentially equivalent to reflecting in the control signal #4[1] the movement mode of the first target object OBJ#1 and the control state (e.g., movement) of the robot 1 at the time when it is determined that the convergence condition is satisfied. Particularly when the first target object OBJ#1 and the second target object OBJ#2 are moving in the same movement mode, generating the control signal #4[1] based on the control signal #3[N3] is essentially equivalent to reflecting in the control signal #4[1] the movement mode of the second target object OBJ#2 and the control state (e.g., movement) of the robot 1 at the time when it is determined that the convergence condition is satisfied. As a result, when the robot 1 is controlled using the control signal #4[1] generated based on the control signal #3[N3], the movement required to cause the end effector 5 to follow the moving second target object OBJ#2 is reflected in the robot 1. As a result, when the robot 1 is controlled using the control signal #4[1] generated based on the control signal #3[N3], the time required for the end effector 5 to follow the second target object OBJ#2 is reduced compared to when the robot 1 is controlled using the control signal #4[1] generated without being based on the control signal #3[N3], and as a result, the time required for the convergence condition to be satisfied is reduced.
[0327] The control device 3 may generate a control signal #4[n] used to control the robot 1 by the n-th fine movement process by adding a correction term based on a difference component U indicating the difference between the actual position of the end effector 5 and the target processing position and the difference between the actual posture of the end effector 5 and the target processing posture to the control signal #4[n-1] generated in the (n-1)th fine movement process. Note that, as described above, if the fine movement process is a process including feedback control such as P control, PI control, or PID control, the correction term may include a term calculated based on the difference component U and the proportional gain Kp. As described above, if the fine movement process is a process including feedback control such as PI control or PID control, the correction term may include a term calculated based on the difference component U and the integral gain Ki. As described above, if the fine movement process is a process including feedback control such as PID control, the correction term may include a term calculated based on the difference component U and the derivative gain Kd. Here, when n = 1, the control signal #4[0] to be used to generate the control signal #4[1] is not actually generated by the fine movement process. In this case, the control device 3 may use a default initial signal, which is preset as the initial value of the control signal #4[n], as the control signal #4[0]. In this case, the control device 3 may generate the control signal #4[1] by using the control signal #3[N3] as the control signal #4[0] (i.e., as the initial signal). In other words, generating the control signal #4[1] based on the control signal #3[N3] may include generating the control signal #4[1] by using...
Claims
1. A control system that generates a control signal for controlling a robot, the robot being equipped with a processing device that performs processing on a target object and an imaging device that generates at least image data of the target object by capturing an image of the target object, the robot moves the processing device and the imaging device, the control system having a control device that generates and outputs the control signal, the control device performing a second movement process after a first movement process, the first movement process including a process of controlling the robot to move the processing device without using the image data, and the second movement process including a process of generating and outputting the control signal for controlling the robot to move the processing device based on a calculation result of the position and orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object after the first movement process.
2. The control system according to claim 1, wherein when the control device is a second control device, the first movement process is performed by a first control device different from the second control device.
3. The control system according to claim 1, wherein the control device performs the first movement process, and performs the second movement process after performing the first movement process.
4. A control system according to any one of claims 1 to 3, wherein the first movement process includes a process of controlling the robot to move the processing device in accordance with movement instruction information without using the image data.
5. A control system that generates control signals for controlling a robot, wherein the robot is equipped with a processing device that performs processing on a target object and an imaging device that generates at least image data of the target object by imaging the target object, and the robot moves the processing device and the imaging device, and the control system includes a second control device that performs a first movement process to control the robot to move the processing device in accordance with movement instruction information using a first control device, and then generates and outputs the control signal for controlling the robot to move the processing device based on the calculation results of the position and orientation of the target object calculated based on the image data generated by the imaging device by imaging the target object, and the second control device is a control device different from the first control device.
6. The control system described in claim 2 or 5, wherein the second control device receives an authorization signal from the first control device that authorizes the second control device to control the robot to move the processing device by performing the second movement processing after the first movement processing is completed by the first control device, and the second control device starts the second movement processing after receiving the authorization signal from the first control device.
7. The control system according to claim 6, wherein the second control device does not start the second movement process when the second control device does not receive the permission signal transmitted by the first control device.
8. The control system according to claim 6 or 7, wherein the second control device repeats the second movement process a plurality of times after the second control device receives the permission signal.
9. The control system according to claim 8, wherein the second control device controls the robot to move the processing device each time the second control device performs the second movement process after receiving the permission signal.
10. A control system as claimed in any one of claims 6 to 9, wherein the second control device repeats the second movement process a plurality of times between the time when the first control device completes the first movement process and the time when the first control device newly performs the first movement process.
11. A control system as described in any one of claims 6 to 10, wherein the second control device repeats the second movement process multiple times even if the second control device does not receive a signal transmitted from the first control device after the second control device receives the permission signal.
12. A control system as claimed in any one of claims 6 to 11, wherein the second control device repeats the second movement process until a predetermined positional relationship between the processing device and the target object and / or a predetermined attitude relationship between the processing device and the target object is achieved during the period from when the second control device receives the permission signal to when the first control device newly performs the first movement process.
13. The control system described in claim 12, wherein the predetermined positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the target object, and the predetermined attitude relationship includes an attitude relationship in which the processing device takes a target processing attitude where the processing device should take when performing processing on the target object.
14. A control system as claimed in any one of claims 6 to 13, wherein the processing device is capable of holding an object, and the second control device repeats the second movement process until a positional relationship between the object held by the processing device and the target object becomes a predetermined positional relationship and / or a posture relationship between the object held by the processing device and the target object becomes a predetermined posture relationship, during the period from when the second control device receives the permission signal to when the first control device newly performs the first movement process.
15. A control system as described in any one of claims 2 and 5 to 14, wherein, after completing the second movement process, the second control device transmits a notification signal to the first control device to notify the first control device that the second control device has completed the second movement process, and the first control device starts the first movement process after receiving the notification signal transmitted from the second control device.
16. The control system described in claim 15, wherein the second control device repeats the second movement process until a predetermined positional relationship between the processing device and the target object and / or a predetermined attitude relationship between the processing device and the target object, and then transmits the notification signal to the first control device, and the first control device starts the first movement process after receiving the notification signal transmitted by the second control device.
17. The control system described in claim 16, wherein the predetermined positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the target object, and the predetermined attitude relationship includes a positional relationship in which the processing device takes a target processing attitude where the processing device should take when performing processing on the target object.
18. A control system according to any one of claims 15 to 17, wherein the first control device does not start the first movement process if the first control device does not receive the notification signal transmitted by the second control device.
19. A control system as claimed in any one of claims 2 and 5 to 18, wherein the second control device generates and outputs a signal for controlling the processing device to perform processing on the target object after performing the second movement processing.
20. A control system as claimed in any one of claims 1 to 19, wherein the second movement process includes a process of generating the control signal for controlling the robot so that the positional relationship between the processing device and the target object becomes a predetermined positional relationship and / or the attitude relationship between the processing device and the target object becomes a predetermined attitude relationship by moving the processing device by the second movement process.
21. The control system of claim 20, wherein the predetermined positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the target object, and the predetermined attitude relationship includes an attitude relationship in which the processing device takes a target processing attitude where the processing device should take when performing processing on the target object.
22. A control system as claimed in any one of claims 1 to 21, wherein the processing device is capable of holding an object, and the second movement process includes a process of generating the control signal for controlling the robot so that, by moving the processing device by the second movement process, a positional relationship between the object held by the processing device and the target object becomes a predetermined positional relationship and / or a posture relationship between the object held by the processing device and the target object becomes a predetermined posture relationship.
23. A control system according to any one of claims 1 to 22, wherein the movement distance of the processing device by the second movement process is shorter than the movement distance of the processing device by the first movement process.
24. A control system according to any one of claims 1 to 23, wherein the maximum movement speed of the processing device by the second movement process is slower than the maximum movement speed of the processing device by the first movement process, or the average movement speed of the processing device by the second movement process is slower than the average movement speed of the processing device by the first movement process.
25. A control system as claimed in any one of claims 1 to 24, wherein the second movement process includes a process of generating the control signal for controlling the robot so that, by moving the processing device by the second movement process, the difference between the position of the processing device at the time when the second movement process is completed and the target processing position is smaller than the difference between the position of the processing device at the time when the first movement process is completed and the target processing position where the processing device should be when the processing device performs processing on the target member.
26. A control system as claimed in any one of claims 1 to 25, wherein the second movement process includes a process of generating the control signal for controlling the robot so that, by moving the processing device by the second movement process, the difference between the attitude of the processing device at the time when the second movement process is completed and the target processing attitude that the processing device should take when processing the target member is smaller than the difference between the attitude of the processing device at the time when the first movement process is completed and the target processing attitude that the processing device should take when processing the target member.
27. A control system according to any one of claims 1 to 26, wherein at least a part of the first movement process is not performed during a period in which the second movement process is being performed.
28. The target object has a first portion that is processed by the processing device, and a second portion that is different from the first portion and is processed by the processing device, the first movement process includes the first movement process for the first portion and the first movement process for the second portion, the second movement process includes the second movement process for the first portion and the second movement process for the second portion, the first movement process for the first portion includes a process of controlling the robot so that the processing device approaches the first portion, the second movement process for the first portion includes a process of generating and outputting a first control signal as the control signal for controlling the robot to move the processing device based on a calculation result of the position and orientation of the first portion calculated based on the image data generated by the imaging device capturing the first portion after the first movement process for the first portion, and the first movement process for the second portion includes a process of controlling the robot so that the processing device approaches the second portion after the second movement process for the first portion, The control system of any one of claims 1 to 27, wherein the second movement process for the second part includes, after the first movement process for the second part, a process of generating and outputting a second control signal as the control signal for controlling the robot to move the processing device based on a calculation result of the position and orientation of the second part calculated based on the image data generated by the imaging device imaging the second part.
29. The control system described in claim 28, wherein the second movement process for the first part is repeated until the positional relationship between the processing device and the first part becomes a first positional relationship and / or the attitude relationship between the processing device and the first part becomes a first attitude relationship between the completion of the first movement process for the first part and the start of the first movement process for the second part.
30. The control system of claim 29, wherein the first positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the first portion, and the first attitude relationship includes an attitude relationship in which the processing device takes a target processing attitude where the processing device should be located when performing processing on the first portion.
31. A control system according to any one of claims 28 to 30, wherein the second movement process for the second part is repeated after the first movement process for the second part until a positional relationship between the processing device and the second part becomes a second positional relationship and / or a posture relationship between the processing device and the second part becomes a second posture relationship.
32. The control system of claim 31, wherein the second positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the second portion, and the second attitude relationship includes an attitude relationship in which the processing device takes a target processing attitude where the processing device should be located when performing processing on the second portion.
33. A control system as claimed in any one of claims 28 to 32, wherein the second control signal is generated based on the first control signal.
34. A control system as described in any one of claims 28 to 33, wherein the second movement process for the first part is repeated during a first period from the completion of the first movement process for the first part to the start of the first movement process for the second part until a positional relationship between the processing device and the first part becomes a first positional relationship and / or an attitude relationship between the processing device and the first part becomes a first attitude relationship; the second movement process for the second part is repeated during a second period after the first movement process for the second part until a positional relationship between the processing device and the second part becomes a second positional relationship and / or an attitude relationship between the processing device and the second part becomes a second attitude relationship; and the second control signal generated during the second period is generated based on the first control signal generated during the first period.
35. The control system according to claim 34, wherein the second control signal that is generated first in the second period is generated based on the first control signal that is generated last in the first period.
36. The target objects include a first target object to be processed by the processing device, and a second target object different from the first target object to be processed by the processing device, the first movement process includes the first movement process for the first target object and the first movement process for the second target object, the second movement process includes the second movement process for the first target object and the second movement process for the second target object, the first movement process for the first target object includes a process in which the processing device controls the robot to approach the first target object, the second movement process for the first target object includes a process, after the first movement process for the first target object, of generating and outputting a third control signal as the control signal for controlling the robot to move the processing device based on a calculation result of the position and orientation of the first target object calculated based on the image data generated by the imaging device capturing an image of the first target object, and the first movement process for the second target object includes a process, after the second movement process for the first target object, of controlling the robot to approach the second target object from the first target object, The control system of any one of claims 1 to 35, wherein the second movement process for the second target object includes, after the first movement process for the second target object, a process of generating and outputting a fourth control signal as the control signal for controlling the robot to move the processing device based on a calculation result of the position and orientation of the second target object calculated based on the image data generated by the imaging device capturing an image of the second target object.
37. The control system described in claim 36, wherein the second movement process for the first target object is repeated until the positional relationship between the processing device and the first target object becomes a third positional relationship and / or the attitude relationship between the processing device and the first target object becomes a third attitude relationship, between the completion of the first movement process for the first target object and the start of the first movement process for the second target object.
38. The control system described in claim 37, wherein the third positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the first target object, and the third attitude relationship includes an attitude relationship in which the processing device takes a target processing attitude where the processing device should take when performing processing on the first target object.
39. A control system as described in claim 37 or 38, wherein the second movement process for the second target object is repeated after the first movement process for the second target object until the positional relationship between the processing device and the second target object becomes a fourth positional relationship and / or the attitude relationship between the processing device and the second target object becomes a fourth attitude relationship.
40. The control system of claim 39, wherein the fourth positional relationship includes a positional relationship in which the processing device is located at a target processing position where the processing device should be located when performing processing on the second target object, and the fourth attitude relationship includes an attitude relationship in which the processing device takes a target processing attitude where the processing device should take when performing processing on the second target object.
41. A control system as claimed in any one of claims 36 to 40, wherein the fourth control signal is generated based on the third control signal.
42. A control system as described in any one of claims 36 to 41, wherein the second movement process for the first target object is repeated during a third period from the completion of the first movement process for the first target object to the start of the first movement process for the second target object until the positional relationship between the processing device and the first target object becomes a third positional relationship and / or the attitude relationship between the processing device and the first target object becomes a third attitude relationship; the second movement process for the second target object is repeated during a fourth period after the first movement process for the second target object until the positional relationship between the processing device and the second target object becomes a fourth positional relationship and / or the attitude relationship between the processing device and the second target object becomes a fourth attitude relationship; and the fourth control signal generated during the fourth period is generated based on the third control signal generated during the third period.
43. The control system according to claim 42, wherein the fourth control signal that is generated first in the fourth period is generated based on the third control signal that is generated last in the third period.
44. A control system according to claim 4 or 5, wherein the movement instruction information is set in advance by a user inputting the movement instruction information.
45. A control system as described in claim 4, 5, or 44, wherein the movement instruction information includes at least one of a target movement position indicating a position to which the processing device should move, a target movement distance indicating a distance to which the processing device should move, and a target movement direction indicating a direction in which the processing device should move.
46. A control system as described in claim 45, wherein the target movement position is fixed to a predetermined position during the period in which the first movement process is being performed, the target movement distance is fixed to a predetermined distance during the period in which the first movement process is being performed, and the target movement direction is fixed to a predetermined direction during the period in which the first movement process is being performed.
47. A control system as described in claim 45 or 46, wherein the target object is moved by a conveying device capable of conveying the target object, and at least one of the target movement position, the target movement distance and the target movement direction is preset based on at least one of the conveying speed of the target member by the conveying device and the conveying direction of the target member by the conveying device.
48. A control system as described in any one of claims 45 to 47, wherein the target object has a first portion that is processed by the processing device and a second portion that is different from the first portion and is processed by the processing device, the processing device processes the first portion and then processes the second portion, and at least one of the target movement position, the target movement distance, and the target movement direction is preset based on a positional relationship between the first portion and the second portion.
49. A control system as described in any one of claims 45 to 48, wherein the target objects include a first target object to be processed by the processing device, and a second target object that is different from the first target object and is processed by the processing device, the processing device processes the second target object after processing the first target object, and at least one of the target movement position, the target movement distance, and the target movement direction is preset based on the positional relationship between the first target object and the second target object.
50. A control system according to any one of claims 1 to 49, wherein the target object is moved by a transport device capable of transporting the target object, and the first movement process and the second movement process are performed on the target object being moved by the transport device.
51. A control system according to any one of claims 1 to 50, wherein the second movement process includes a process of calculating the position and orientation of the target object by performing a matching process using the image data generated by the imaging device capturing an image of the target object after the first movement process and model data indicating an object model that is a model of the target object.
52. The control system of claim 51, wherein the object model is a model of an edge of at least a portion of the target object.
53. The control system according to claim 52, wherein the model data is data indicating an edge of at least a portion of the target object or the sample in an image obtained by imaging the target object or a sample object having the same shape as the target object in advance.
54. A control system according to any one of claims 1 to 53, wherein the second movement process includes a process of generating and outputting the control signal for controlling the robot to move the processing device based on the calculation results of the position and orientation of the target object and the calculation results of the position and orientation of the processing device, calculated based on the image data generated by the imaging device capturing an image of the target object and the processing device after the first movement process.
55. The control system described in claim 54, wherein the second movement process includes a process of calculating the position and orientation of the target object and the position and orientation of the processing device by performing a matching process using the image data generated by the imaging device capturing an image of the target object and the processing device after the first movement process, and model data indicating an object model that is a model of the target object and an device model that is a model of the processing device.
56. The control system of claim 55, wherein the object model is a model of an edge of at least a portion of the target object, and the device model is a model of an edge of at least a portion of the processing device.
57. A control system as described in claim 55 or 56, wherein the model data indicates the object model and the equipment model which are aligned with each other, and the second movement process includes a process of generating the control signal for controlling the robot by moving the processing device so that a difference between a positional relationship between the target object and the processing device and a positional relationship between the object model and the equipment model which are aligned with each other is within a threshold value, based on the calculation results of the position and orientation of the target object and the calculation results of the position and orientation of the processing device.
58. A control system as claimed in any one of claims 55 to 57, wherein the model data indicates the object model and the equipment model which are aligned with each other, and the second movement process includes a process of generating the control signal for controlling the robot by moving the processing device so that a difference between an attitude relationship between the target object and the processing device and an attitude relationship between the object model and the equipment model which are aligned with each other is within a threshold value, based on the calculation results of the position and attitude of the target object and the calculation results of the position and attitude of the processing device.
59. A control system as described in any one of claims 55 to 58, wherein the model data indicates the object model and the device model which are aligned with each other, and the second movement process includes a process of selecting, when a plurality of target objects are detected by the matching process, one target object which satisfies the condition that the positional relationship between the target object and the processing device is closest to the positional relationship between the object model and the device model which are aligned with each other, as the target object to be processed by the processing device.
60. A control system as described in any one of claims 1 to 58, wherein the control system performs a preparatory process for performing the second movement process during a period in which the first movement process is being performed and before the second movement process is started.
61. A control system according to any one of claims 2 and 5 to 19, wherein the second control device performs a preparatory process for performing the second movement process before starting the second movement process.
62. A control system as described in claim 61, wherein said second control device performs a preparatory process for performing said second movement process during a period in which said first movement process is being performed and before starting said second movement process.
63. A control system according to any one of claims 60 to 62, wherein the preparation process includes a process of acquiring information used to perform the second movement process.
64. A control system according to any one of claims 60 to 63, wherein the preparation process includes a process for adjusting an exposure condition of the imaging device.
65. A control system as described in any one of claims 60 to 64, wherein the imaging device is equipped with an illumination device capable of illuminating the target object with illumination light, and the preparation process includes a process of causing the illumination device to start illuminating the target object with the illumination light.
66. A control system as claimed in any one of claims 1 to 65, wherein the second movement process includes, when it becomes impossible to calculate the position and orientation of the target object based on image data generated by imaging by the imaging device at a first time during the period in which the second movement process is being performed, a process of calculating the position and orientation of the target object at the first time based on the position and orientation of the target object calculated based on the image data generated by imaging the target object by the imaging device at a second time prior to the first time.
67. A control system as claimed in any one of claims 1 to 66, wherein the second movement process includes a process of generating and outputting a control signal for controlling the robot to move the processing device based on information regarding contact between the processing device and the target object when it becomes impossible to calculate the position and orientation of the target object based on the image data generated by imaging by the imaging device during the period when the second movement process is being performed.
68. The control system of any one of claims 1 to 67, wherein the control system inspects the target object based on the image data.
69. The control system according to any one of claims 1 to 68, wherein the control system inspects the target object based on a measurement result of the target object by a measurement device capable of measuring the target object.
70. A control system as claimed in claim 68 or 69, wherein the control system inspects the object before the processing device performs processing on the object.
71. A control system as claimed in any one of claims 68 to 70, wherein the control system inspects the object after the processing device has performed processing on the object.
72. A control system according to any one of claims 1 to 71, wherein the control system inspects the results of processing performed by the processing device on the target object based on the image data or the measurement results of a measurement device.
73. A control system as claimed in any one of claims 1 to 72, wherein the target object is moved by a transport device capable of transporting the target object, and the control system performs the second movement process during a period in which the transport device is moving the target object, and the second movement process includes a process of generating a control signal for controlling the robot to move the processing device based on a transport state of the target object when it becomes impossible to calculate the position and orientation of the target object based on image data generated by imaging by the imaging device at a first time during the period in which the second movement process is being performed.
74. A control system according to any one of claims 1 to 73, further comprising the imaging device.
75. The control system of any one of claims 2, 5 to 19, 61 and 62, further comprising the first control device.
76. A robot system comprising: a control system according to any one of claims 1 to 75; and the robot.
77. The robot system of claim 76, further comprising a processing unit for performing the processing on the target object.
78. A control method for generating a control signal for controlling a robot, the robot being equipped with a processing device for processing a target object and an imaging device for capturing an image of the target object and outputting at least image data of the target object, the robot moving the processing device and the imaging device, the control method including performing a second movement process after a first movement process, the first movement process including a process of controlling the robot to move the processing device without using the image data, and performing the second movement process including generating and outputting the control signal for controlling the robot to move the processing device based on a calculation result of the position and orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object after the first movement process.
79. A control method for generating a control signal for controlling a robot, the robot being equipped with a processing device for processing a target object and an imaging device for capturing an image of the target object and outputting at least image data of the target object, the robot moving the processing device and the imaging device, the control method including: performing a first movement process by a first control device to control the robot to move the processing device in accordance with movement instruction information; and performing a second movement process by a second control device different from the first control device to generate and output the control signal for controlling the robot to move the processing device based on the calculation results of the position and orientation of the target object calculated based on the image data generated by the imaging device capturing an image of the target object.
80. A computer program for causing a computer to execute the control method according to claim 78 or 79.
Citation Information
Patent Citations
Information processing apparatus and information processing method
US20130230235A1
Movement control method of robot
JP1997076185A
Robot device and robot control method
JP2015000454A
Robot control system, control device, robot, control method for robot control system and robot control method
JP2015003348A
Robot control device, robot system, robot, robot control method and robot control program
JP2015174206A
