Control device
The control device uses a fixed three-dimensional camera on the robot hand to perform multi-step calibration, maintaining the workpiece within the camera's view and ensuring accurate coordinate conversion, addressing the challenges of conventional systems with separate calibration marks.
Patent Information
- Application Number
- JP2021020016
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-02-10
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-02-10
AI Technical Summary
Conventional hand-eye robot systems require separate calibration marks that are different from the workpieces, necessitating adjustments in imaging parameters, leading to suboptimal calibration and potential loss of workpiece visibility during rotation, which complicates accurate coordinate conversion.
A control device that uses a three-dimensional camera fixed to the robot hand to measure the workpiece's shape, performing multi-step calibration by moving and rotating the robot hand to maintain the workpiece within the camera's field of view, allowing for accurate conversion of camera coordinates to robot coordinates.
Enables precise calibration data acquisition without significant robot hand retraction, improving measurement accuracy by keeping the workpiece within the camera's field of view and reducing errors in coordinate conversion.
Smart Images

Figure 0007591938000001 
Figure 0007591938000002 
Figure 0007591938000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a control device for controlling a robot. [Background technology]
[0002] Conventionally, a robot system known as a hand-eye system is known, which combines a robot with a three-dimensional camera fixed to a robot hand (see, for example, Patent Document 1). This type of robot system requires calibration before operation to convert the camera coordinates into the robot coordinates of the robot hand. In Patent Document 1, one calibration mark is placed in the robot coordinate system, and the calibration mark is observed at a plurality of observation points where a three-dimensional camera is translated while facing the calibration mark, and at a plurality of observation points where the calibration mark is viewed from a different direction from the observation points, and calibration data is acquired by associating mark position recognition data obtained at each observation point with posture data of the robot hand. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2012-91280 A Summary of the Invention [Problem to be solved by the invention]
[0004] Incidentally, in Patent Document 1, it is necessary to prepare a calibration mark when performing calibration. The calibration mark is different in material, shape, size, etc. from the workpiece to be picked up when the robot system is in operation, so it is necessary to image the calibration mark by changing the imaging parameters (exposure time, gain, etc.) of the camera and the focus position, etc., from those used when imaging the workpiece. In other words, since calibration is performed with parameters different from those used during operation, it is difficult to say that the calibration is optimal for operation.
[0005] Therefore, it is conceivable to use the workpiece as the calibration mark instead of the calibration mark.However, when obtaining the parameters for converting the camera coordinates into the robot coordinates, the robot hand needs to be rotated, and in principle, the accuracy increases by rotating the robot hand more widely, so there is a demand for rotating the robot hand more widely during calibration.
[0006] However, because the camera's field of view and the workpiece are small compared to the movable range of the robot hand, if the robot hand is rotated significantly, the workpiece will go outside the camera's field of view, making it impossible to obtain calibration data.
[0007] Furthermore, even if the workpiece does not leave the camera's field of view when the robot hand is rotated, if only a portion of the workpiece is imaged, sufficient data for accurate calibration cannot be obtained.
[0008] Furthermore, while the measurement accuracy of the peripheral areas of a 3D camera's field of view is generally lower than that near the center, if the robot hand is rotated significantly, calibration will be performed using data from the peripheral areas of the 3D camera's field of view, which also raises the risk that accurate calibration will not be possible.
[0009] The present disclosure has been made in consideration of these points, and an object of the present disclosure is to make it possible to obtain accurate calibration data when performing calibration using a workpiece. [Means for solving the problem]
[0010] In order to achieve the above object, a first aspect of the present disclosure can be based on a control device that measures a shape of a workpiece using a three-dimensional camera fixed to a robot hand, and performs calibration to convert the camera coordinates of the three-dimensional camera into robot coordinates of the robot hand based on the measured shape data of the workpiece. The control device includes a calibration execution unit that can execute normal calibration processing and high-precision calibration processing.
[0011] In the normal calibration process, the robot hand is repeatedly moved in the X, Y and Z directions and rotated multiple times around a predetermined axis. After each operation, work coordinates are detected based on shape data obtained by measuring the workpiece with the three-dimensional camera at the stopping position of the robot hand, and the robot coordinates at the time when the workpiece coordinates are obtained are obtained. Based on the multiple pairs of workpiece coordinates and robot coordinates obtained, parameters for converting the position in the camera coordinates to a position on the robot coordinates can be calculated.
[0012] On the other hand, in the high-precision calibration process, a provisional rotation center is determined based on the relative position of the tool center of the robot hand and the three-dimensional camera obtained by the normal calibration process, and the robot hand is rotated multiple times around the provisional rotation center, and at each position, work coordinates are detected based on shape data obtained by measuring the work with the three-dimensional camera, and the robot coordinates at the time the work coordinates are obtained are obtained, and the final rotation center of the robot hand can be determined based on the multiple pairs of work coordinates and robot coordinates obtained.
[0013] In this configuration, since a robot system in which a three-dimensional camera for measuring the three-dimensional shape of a workpiece is fixed to a robot hand can be assumed, during operation of the robot system, the position of the workpiece can be measured by the three-dimensional camera without significantly retracting the robot hand, and the three-dimensional shape of the workpiece can be measured from a location close to the workpiece to be measured, improving measurement accuracy. Note that the subject of the present invention may be a robot system or a robot control device (robot controller).
[0014] When a three-dimensional camera is fixed to a robot hand, it is necessary to perform calibration to convert the camera coordinates of the three-dimensional camera into the robot coordinates of the robot hand. In this configuration, a multi-stage calibration process is performed using the work to be picked up during operation.
[0015] That is, first, as a normal calibration process, the robot hand is repeatedly moved in the XYZ directions and rotated, and the workpiece after each operation is measured by a 3D camera to detect the workpiece coordinates, and the robot coordinates at the time the workpiece coordinates are obtained are also obtained. Based on the multiple pairs of workpiece coordinates and robot coordinates obtained in this way, parameters for converting the position in the camera coordinates to a position on the robot coordinates are calculated, and the relative positional relationship between the tool center of the robot hand and the 3D camera is found.
[0016] After determining the relative positional relationship between the tool center of the robot hand and the three-dimensional camera, a high-precision calibration process is performed. First, a provisional rotation center of the robot hand is determined based on the relative position between the tool center of the robot hand and the three-dimensional camera determined as described above. This provisional rotation center is a rotation center that is temporarily set. By rotating the robot hand multiple times around the provisional rotation center, it is possible to prevent the workpiece from going outside the field of view of the three-dimensional camera, to make the entire workpiece enter the field of view of the three-dimensional camera, and to place the workpiece near the center of the field of view of the three-dimensional camera. Then, at each position, the workpiece is measured by the three-dimensional camera to detect the workpiece coordinates, and the robot coordinates when the workpiece coordinates are obtained are obtained, and the final rotation center of the robot hand can be determined with high precision based on the multiple pairs of workpiece coordinates and robot coordinates obtained.
[0017] The robot coordinates when the workpiece coordinates are obtained can be obtained, for example, based on the operating state of the robot hand, or from a robot controller that controls the robot.
[0018] In a second aspect of the present disclosure, the calibration execution unit rotates the robot hand around the temporary rotation center in the high-precision calibration process to a rotation angle that is greater than the rotation angle of the robot hand rotated around a predetermined axis in the normal calibration process.
[0019] According to this configuration, the accuracy of the calibration data obtained is increased by increasing the rotation angle of the robot hand in the high-precision calibration process. In the high-precision calibration process, the robot hand is rotated around the temporary rotation center, so even if the rotation angle is increased as described above, the possibility that all or part of the workpiece will go outside the field of view of the 3D camera is reduced.
[0020] In a third aspect of the present disclosure, the calibration execution unit can execute a first process to determine a rotation center of the robot hand based on a pair of the multiple work coordinates and the robot coordinates in the high-precision calibration process, and a second process to detect work coordinates based on shape data obtained by measuring the work with the three-dimensional camera at each position where the robot hand is rotated multiple times, using the rotation center determined in the first process as the next rotation center, and obtain the robot coordinates at the time the work coordinates were obtained, and determine a rotation center of the robot hand based on the obtained pairs of the multiple work coordinates and robot coordinates, before determining a final rotation center of the robot hand.
[0021] In other words, since the provisional rotation center may be shifted from the center that should ultimately be the rotation center, instead of directly determining the final rotation center based on this provisional rotation center, a second process is executed to determine a new rotation center of the robot hand using the rotation center determined in the first process as the next rotation center, and the robot hand is rotated using the newly determined rotation center in the second process as the next rotation center to determine the final rotation center of the robot hand, thereby reducing the error in the final rotation center.
[0022] In a fourth aspect of the present disclosure, a process of determining the rotation center of the robot hand using the rotation center determined in the second process as the next rotation center, and a process of determining the rotation center of the robot hand using the rotation center determined in the first process as the next rotation center are repeatedly executed, and then the final rotation center of the robot hand can be determined.
[0023] According to this configuration, the process of finding the rotation center can be repeated multiple times before the final rotation center is determined, so that it is possible to converge to a rotation center with less error.
[0024] In a fifth aspect of the present disclosure, the calibration execution unit can repeatedly execute the high-precision calibration process to obtain multiple rotation centers of the robot hand, and determine a final rotation center based on the multiple rotation centers obtained.
[0025] That is, if the high-precision calibration process is completed only once, the determined final rotation center may be inaccurate due to some factor, but by determining the final rotation center based on multiple rotation centers obtained by executing the high-precision calibration process multiple times as in this configuration, an accurate rotation center can be obtained. When determining the final rotation center, for example, an averaging process of the multiple rotation centers may be performed.
[0026] In a sixth aspect of the present disclosure, the calibration execution unit determines whether or not a plurality of rotation centers obtained by repeatedly executing the high-precision calibration process satisfy a predetermined convergence condition, and if it determines that the predetermined convergence condition is satisfied, can terminate the high-precision calibration process.
[0027] In other words, if the multiple rotation centers acquired by repeatedly executing the high-precision calibration process do not converge, the acquired rotation center may not be accurate. In this configuration, the high-precision calibration process is executed until a predetermined convergence condition is satisfied, so that an accurate rotation center can be acquired. As the predetermined convergence condition, for example, a change in the error of the rotation center acquired by the high-precision calibration process is equal to or less than a certain value.
[0028] In a seventh aspect of the present disclosure, the calibration execution unit can rotate the robot hand by a predetermined angle around a temporary rotation center during the high-precision calibration process, and move the robot hand so that the workpiece after rotating by the predetermined angle is within the field of view of the three-dimensional camera.
[0029] In other words, when the robot hand is rotated a predetermined angle around the temporary rotation center, depending on the position and size of the workpiece, there is a risk that the workpiece will go out of the field of view of the three-dimensional camera after rotating by the predetermined angle. However, with this configuration, the workpiece remains within the field of view of the three-dimensional camera even after rotating by the predetermined angle, so accurate calibration data can be obtained.
[0030] In an eighth aspect of the present disclosure, the calibration execution unit can determine the XYZ coordinates of a representative point of the work based on three-dimensional shape data obtained by measuring the work using the three-dimensional camera, and the determined XYZ coordinates of the representative point can be used as work coordinates.
[0031] In a ninth aspect of the present disclosure, the calibration execution unit can calculate the XY coordinates of a representative point of the work based on two-dimensional shape data obtained by measuring the work using the three-dimensional camera, and can calculate the Z coordinate of the representative point from a plane equation, and can set the calculated XYZ coordinates of the representative point as work coordinates.
[0032] In a tenth aspect of the present disclosure, the calibration execution unit detects work coordinates based on shape data obtained by moving the robot hand to a first position and measuring the work with the three-dimensional camera, and detects work coordinates based on shape data obtained by moving the robot hand to a second position having a different Z coordinate and measuring the work with the three-dimensional camera, and can convert the work coordinates detected at the first position and the work coordinates detected at the second position into pixel positions independent of the angle of view of the three-dimensional camera.
[0033] That is, when the robot hand is in the first position and when the robot hand is in the second position, the position of the 3D camera in the Z direction is different, so the X coordinate of the work coordinates varies depending on the angle of view of the camera, and accurate calibration data may not be obtained. With this configuration, even if the position of the 3D camera in the Z direction is different, the work coordinates can be converted into pixel positions that are independent of the angle of view of the 3D camera, so accurate calibration data can be obtained.
[0034] In an eleventh aspect of the present disclosure, a calibration mode for performing calibration is configured to be selectable, and when the calibration execution unit detects that the calibration mode has been selected, the calibration execution unit sequentially executes the normal calibration process and the high-precision calibration process in sequence.
[0035] According to this configuration, for example, in a case where an operational mode and a calibration mode are provided, when the calibration mode is selected before the operational mode, the normal calibration process is executed first, and then the high-precision calibration process is executed, eliminating the need for the user to switch from the normal calibration process to the high-precision calibration process.
[0036] In a twelfth aspect of the present disclosure, the calibration execution unit generates a user interface having a display area capable of updating and displaying the detection result of the work coordinates detected in the high-precision calibration process and the convergence value of the rotation center. The control device includes a display unit that displays the user interface generated by the calibration execution unit, and thus can present the change in the convergence value to a user. Effect of the Invention
[0037] As described above, the provisional rotation center is determined based on the relative position between the tool center of the robot hand and the three-dimensional camera obtained by normal calibration processing, and the final rotation center of the robot hand can be determined with high accuracy based on multiple pairs of workpiece coordinates and robot coordinates obtained by rotating the robot hand multiple times around the provisional rotation center. This makes it possible to obtain accurate calibration data when performing calibration using a workpiece. [Brief description of the drawings]
[0038] [Figure 1] FIG. 1 is a schematic diagram showing a state in which a workpiece is transported using a robot system. [Diagram 2] FIG. 1 is a block diagram showing a configuration of a robot system. [Diagram 3] FIG. 1 is a perspective view showing an example of a three-dimensional camera. [Figure 4A] 1 is a schematic diagram showing an example of gripping a workpiece with a robot hand; [Figure 4B] 1A and 1B are schematic diagrams showing an example of gripping a workpiece having a cavity from the inside with a robot hand. [Figure 4C] 1 is a schematic diagram showing an example of a robot hand suctioning and gripping a plate-shaped workpiece; FIG. [Diagram 5] 13 is a flowchart showing a procedure for setting a search model. [Figure 6] 11 is a flowchart showing a procedure for setting a hand model and a gripping position. [Figure 7] FIG. 13 is a diagram showing an example of a flowchart of basic control of a robot. [Figure 8] 11 is a flowchart showing an example of a calibration procedure. [Figure 9] 10 is a flowchart showing a detailed procedure of a first step of calibration. [Figure 10] FIG. 13 is a diagram illustrating an example of a user interface for calibration. [Figure 11] FIG. 2 is a schematic diagram showing how the robot hand is translated in a first step. [Figure 12] FIG. 13 is a diagram showing an example of a calibration user interface that is displayed when the robot hand is moved to a first position in the first step. [Figure 13] FIG. 13 is a diagram showing an example of a calibration user interface that is displayed when the robot hand is moved to the 27th position in the first step. [Figure 14A] FIG. 4 is a side view showing the positional relationship between the workpiece and the three-dimensional camera during measurement in the first step. [Figure 14B] FIG. 11 is a diagram showing the relationship between the workpiece and the field of view of the three-dimensional camera during measurement in the first step. [Figure 15] 11 is a table showing distances from the center of the camera to representative points used when correcting the angle of view. [Figure 16] FIG. 13 is a diagram showing an example of a calibration user interface that is displayed upon completion of the first step. [Figure 17] 13 is a flowchart showing a detailed procedure of a second step of the calibration. [Figure 18] FIG. 13 is a diagram showing an example of a user interface for calibration displayed in a second step. [Figure 19] 13 is a diagram illustrating the positional relationship between the field of view and the workpiece when the movement of the robot hand is enlarged in the second step. FIG. [Figure 20] FIG. 13 is a diagram showing an example of a calibration user interface that is displayed upon completion of the second step. [Figure 21] 13 is a flowchart showing a detailed procedure of a third step of calibration. [Figure 22] FIG. 13 is a diagram showing an example of a calibration user interface displayed when high-precision calibration is set. [Figure 23A] 13A to 13C are diagrams illustrating the case where the robot hand is rotated around the provisional center of rotation in the third step. [Figure 23B] 13A to 13C are diagrams illustrating a case where the robot hand is rotated while keeping the workpiece in view in the third step. [Figure 24] 11A and 11B are diagrams for explaining a case where a deviation between an original rotation center and a true rotation center position is obtained. [Diagram 25] 11A and 11B are diagrams illustrating a case where the original rotation center is updated. [Figure 26] FIG. 13 is a diagram showing an example of a user interface for calibration that is displayed when high-precision calibration processing is completed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0039] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. Note that the following description of the preferred embodiments is merely illustrative in nature and is not intended to limit the present invention, its applications, or its uses.
[0040] Fig. 1 is a schematic diagram showing a configuration example of a robot system 1000 equipped with a control device 100 according to an embodiment of the present invention. Fig. 1 shows an example of bulk picking in which a robot RBT grasps multiple workpieces WK arranged in a work space in a manufacturing plant for various products one by one, transports them to a stage STG installed in a predetermined location, and places them on the stage STG.
[0041] The robot system 1000 includes a robot RBT and a robot controller 6 that controls the robot RBT. The robot RBT is a general-purpose industrial robot, and its base is fixed to the floor of a factory or the like. The robot RBT is also called, for example, a manipulator, and is configured to be capable of six-axis control. The robot RBT includes an arm ARM extending from the base, and a robot hand HND provided at the tip of the arm ARM. The arm ARM can be configured as a multi-joint type having multiple joints as movable parts. The robot hand HND can be moved to a desired position within a movable range by the operation of each joint of the arm ARM and the rotational operation of the arm ARM itself.
[0042] The robot hand HND is configured to be able to grip the workpiece WK, and its structure is not particularly limited. For example, the robot hand HND may be a robot hand HND1 having a structure for pinching and gripping the outside of the workpiece WK as shown in FIG. 4A, a robot hand HND2 having a structure for holding the workpiece WK2 having a hollow portion by inserting a claw portion into the inside of the workpiece WK2 and expanding it as shown in FIG. 4B, or a robot hand HND3 for sucking and holding a plate-shaped workpiece WK3 as shown in FIG. 4C, and any of these robot hands may be used. In this specification, the term "hold" is used to include all examples, such as a method of pinching the outside of the workpiece WK shown in FIG. 4A, a method of inserting a claw portion into a hollow portion and expanding it, and a method of sucking.
[0043] The robot controller 6 controls the operation of the arm section ARM and the opening and closing operation (grasping operation) of the robot hand HND. The robot controller 6 also acquires information required for controlling the robot RBT from the control device 100 shown in Fig. 2. For example, the position and orientation of the workpieces WK, which are a number of parts randomly placed in the container BX shown in Fig. 1, are acquired using the three-dimensional camera 2, and the position and orientation of the workpieces WK are detected by the control device 100, and the robot controller 6 acquires this information.
[0044] A functional block diagram of the robot system 1000 is shown in Fig. 2. The robot system 1000 shown in this figure includes a control device 100, an operation unit 4, and a robot operation tool 7 in addition to the robot RBT and the robot controller 6. The robot system 1000 also includes a display unit 3. The robot controller 6 may include the control device 100, or the control device 100 may be configured independently and connected to the robot controller 6. The control device 100 may also include the operation unit 4 and the robot operation tool 7.
[0045] The operation unit 4 is a section for performing various settings. The control device 100 also includes a measurement unit 20, which captures an image of the workpiece WK to measure the position, orientation, etc. of the workpiece WK. Furthermore, the user can use the display unit 3 to perform various settings, check the operating status of the robot RBT, check the operating status of the control device 100, check the calibration described below, and the like. The display unit 3 can also be a part of the control device 100.
[0046] On the other hand, the robot controller 6 is a well-known member configured to control the robot RBT according to a signal output from the control device 100. Also, the robot operation tool 7 sets the operation of the robot RBT. Note that, although the operation unit 4 and the robot operation tool 7 are separate members in the example of Fig. 2, they may be a common member.
[0047] The three-dimensional camera 2 is a component called a robot vision or the like that captures images of the working space and the workpiece WK. The three-dimensional camera 2 is fixed to the robot hand HND so that the relative positional relationship between the three-dimensional camera 2 and the robot hand HND does not change. Therefore, the robot system 1000 of this embodiment is a robot system of a type called a hand-eye.
[0048] The object to be imaged by the three-dimensional camera 2 includes at least the workpiece WK, and may include the container BX. Although it depends on the workpiece WK, the direction of the optical axis of the three-dimensional camera 2 can be set so that the three-dimensional camera 2 can image the downward direction when the robot hand HND is in a position and posture for picking up the workpiece WK below.
[0049] It becomes possible to obtain three-dimensional shape data (image data) showing the three-dimensional shape of the workpiece WK from the image captured by the three-dimensional camera 2. Three-dimensional shape data showing the three-dimensional shape of the container BX can also be obtained from the image captured by the three-dimensional camera 2. It is also possible to obtain two-dimensional shape data showing the two-dimensional shape of the workpiece WK from the image captured by the three-dimensional camera 2. The three-dimensional camera 2 can also output color images or black and white images of the workpiece WK.
[0050] In addition, methods for acquiring the three-dimensional shape of the workpiece WK or the container BX include, for example, a pattern projection method, a stereo method, a lens focusing method, a light section method, an optical radar method, an interference method (white light interference method), a TOF method, and the like, and any of these methods may be used. Each method is conventionally well known, so a detailed description is omitted. In this embodiment, of the pattern projection methods, a phase shift method is used in which a pattern light having a periodic illuminance distribution is irradiated onto the object to be imaged and the light reflected from the surface of the object to be imaged is received.
[0051] The measurement unit 20 is made up of the three-dimensional camera 2 and a shape measurement processing unit 200. The shape measurement processing unit 200 can also be included as part of the three-dimensional camera 2. The shape measurement processing unit 200 can measure not only the position, posture, shape and dimensions of each part of the workpiece WK, but also the position, shape and dimensions of the container BX based on image data acquired by the three-dimensional camera 2. The shape measurement processing unit 200 can be part of the control device 100, but can also be configured as a separate member.
[0052] The measuring unit 20 is a three-dimensional measuring unit capable of performing three-dimensional measurements including the Z direction (height direction) in addition to the X direction and Y direction. The specific configuration of the three-dimensional camera 2 included in the measuring unit 20 is determined according to the measurement technology of the three-dimensional shape. In this example, the three-dimensional camera 2 includes a camera, lighting, a projector, or the like. For example, when measuring the three-dimensional shape of the workpiece WK using the phase shift method, the three-dimensional camera 2 includes a projector (light-projecting unit) PRJ and multiple cameras (light-receiving units) CME1, CME2, CME3, and CME4 as shown in FIG. 3. The projector PRJ is a member that irradiates light onto the workpiece WK. The cameras CME1, CME2, CME3, and CME4 are members having imaging elements that receive the light projected by the projector PRJ and reflected by the surface of the workpiece WK. The light source of the light-projecting unit can be configured, for example, with multiple light-emitting diodes, a liquid crystal panel, an organic EL panel, a digital micromirror device (DMD), or the like. There is no particular limitation on the number of cameras and lights included in the three-dimensional camera 2. Furthermore, the connections between the three-dimensional camera 2, the robot RBT, and the robot controller 6 are not limited to wired connections, and may be well-known wireless connections.
[0053] The three-dimensional camera 2 may be configured with a plurality of members such as the cameras CME1, CME2, CME3, and CME4 and the projector PRJ, or may be configured as an integrated unit. For example, the three-dimensional camera 2 may be a 3D imaging head formed by integrating the cameras CME1, CME2, CME3, and CME4 and the projector PRJ into a head shape.
[0054] Furthermore, the generation of the three-dimensional shape data itself can be performed by the three-dimensional camera 2. In this case, an image processing IC or the like that realizes a function of generating three-dimensional shape data is provided in the three-dimensional camera 2. Alternatively, the generation of the three-dimensional shape data is not performed on the shape measurement processing unit 200 side, and the raw image captured by the three-dimensional camera 2 may be image-processed in another part of the control device 100 to generate three-dimensional shape data such as a three-dimensional image.
[0055] The control device 100 performs three-dimensional search, interference determination, gripping solution calculation, etc., based on three-dimensional shape data such as the position and posture of the workpiece WK obtained by the measurement unit 20. This control device 100 can be configured with a general-purpose computer with a dedicated image processing program installed, a dedicated image processing controller, and dedicated hardware. It can also be configured with an image processing program installed on a dedicated computer with hardware such as a graphics board specialized for image inspection processing.
[0056] 2 shows an example in which the three-dimensional camera 2 and the shape measurement processing unit 200 of the measurement unit 20 are configured as separate members, but the present invention is not limited to this configuration, and for example, the three-dimensional camera 2 and the shape measurement processing unit 200 can be integrated, or a part of the three-dimensional camera 2 can be incorporated into the shape measurement processing unit 200, or a part of the shape measurement processing unit 200 can be incorporated into the three-dimensional camera 2. In this way, the division of the members shown in FIG. 2 is one example, and multiple members can be integrated, or one member can be separated. For example, the operation unit 4 and the robot operation tool 7 can be a common member.
[0057] The display unit 3 can be, for example, a liquid crystal monitor, an organic EL display, or a CRT. The operation unit 4 is a member for performing various settings for various simulations, image processing, and the like, and can use input devices such as a keyboard and a mouse. In addition, the operation unit 4 and the display unit 3 can be integrated by making the display unit 3 a touch panel.
[0058] For example, when the control device 100 is configured with a computer on which an image processing program is installed, as described below, a graphical user interface (GUI) screen generated by executing the image processing program is displayed on the display unit 3. Various settings can be made from the GUI displayed on the display unit 3, and processing results and the like can be displayed. In this case, the display unit 3 can also be used as a setting unit or input unit for making various settings, and the setting unit and input unit may include the operation unit 4.
[0059] The robot controller 6 controls the operation of the robot based on, for example, information captured by the three-dimensional camera 2 and information output from the control device 100. The robot operation tool 7 is a member for setting the operation of the robot RBT, and a so-called pendant or the like can be used.
[0060] The robot controller 6 controls the robot RBT to specify the workpiece WK to be grasped from among the multiple workpieces WK based on the three-dimensional shape of the workpiece WK obtained by the three-dimensional camera 2, and to grasp the workpiece WK. Then, while grasping the workpiece WK at the grasping position, the arm unit ARM is operated to move the workpiece WK to a predetermined placement position, for example, on the stage STG, and place the workpiece WK in a predetermined posture. In other words, the robot controller 6 controls the operation of the robot RBT to grasp the workpiece WK to be picked (grasped) specified by the three-dimensional camera 2 and the control device 100 with the robot hand HND, place the grasped workpiece WK in a predetermined reference posture on a placement location (stage STG), and release the robot hand HND. The stage STG can be, for example, a conveyor belt or a pallet. The grasping operation can also be called picking, and the grasping position can also be called the picking position. The placing operation can also be called placing, and the placing position can also be called the placing position.
[0061] Here, the picking targeted by this embodiment is not limited to bulk picking, and the robot RBT can transport the workpieces WK regardless of how they are placed. In this specification, bulk picking is used to mean an example in which the workpieces WK that are randomly stacked in a container BX are grasped by the robot RBT and placed in a predetermined position, as well as an example in which the workpieces WK stacked in a predetermined area are grasped and placed without using a container, or an example in which the workpieces WK that are arranged and stacked in a predetermined posture are grasped and placed one by one. In addition, the workpieces WK do not necessarily have to be stacked on top of each other, and the workpieces WK that are randomly placed on a plane without overlapping each other are also called bulk picking in this specification (this is the same reason why the workpieces WK are still called bulk picking even when they are picked one by one and there is no overlapping between them at the end of the picking process). Note that the present invention is not necessarily limited to bulk picking, and can also be applied to the application of picking up workpieces WK that are not stacked.
[0062] When performing a bulk picking operation with the robot system 1000, teaching can be performed including settings for performing the bulk picking operation in advance. Specifically, it is possible to register which part of the workpiece WK the robot hand HND will grasp in what posture, the grasping position and the posture when grasping, and the placement position and the posture when placing can be registered. Such settings can be performed with a robot operating tool 7 such as a pendant. Also, settings can be performed in the vision space without operating the actual robot RBT.
[0063] The display unit 3 displays the work model, which virtually represents the three-dimensional shape of the workpiece WK, and the robot hand model, which is configured with three-dimensional CAD data and virtually represents the three-dimensional shape of the robot hand HND, in three-dimensional form in a virtual three-dimensional space. Furthermore, the display unit 3 can also display the basic direction image of the work model as a six-sided view. This makes it possible to display each posture of the work model in a six-sided view and set the gripping position.
[0064] (Settings Procedure) FIG. 5 is a flowchart showing the procedure for setting a search model. In step SB1, a search model of the workpiece WK is registered. The search model of the workpiece WK is a model that represents the shape of the workpiece WK and is used when executing the search process described below. When registering the search model of the workpiece WK in step SB1, three-dimensional CAD data (CAD model) of the workpiece WK is read and temporarily stored in the storage device 42 of the control device 100. The CAD data is data of the search model of the workpiece WK, and data in a format that has been commonly used in the past can be used.
[0065] The search model of the workpiece WK may be, for example, an interference determination model (used in an interference determination unit 25 described later) generated based on a surface shape seen from a certain side. The search model of the workpiece WK may be in the STL format, which is the simplest format for CAD data.
[0066] After reading the CAD data of the search model of the workpiece WK in step SB1, the process proceeds to step SB2, where the search model of the workpiece WK is confirmed and registered.
[0067] In addition, in step SB2, height image data viewed from each direction of the six faces of the CAD model read in step SB1 is generated. First, six height image data are generated so as to be a plan view, a bottom view, a left side view, a right side view, a front view, and a back view of the CAD model. A height image is obtained from this height image data. If there is no CAD data of the workpiece WK, a plurality of height image data viewed from different directions of the search model of the workpiece WK may be registered as a search model. That is, the workpiece WK is placed on a plane with the face to be registered facing upward, and three-dimensional measurement is performed. This three-dimensional measurement can be performed using the three-dimensional camera 2 and the shape measurement processing unit 200 of the robot system 1000. The measurement data obtained by the three-dimensional measurement by the shape measurement processing unit 200 is output from the shape measurement processing unit 200, and height image data of the face to be registered of the workpiece WK can be obtained based on the measurement data. After the height image is obtained, the height image is registered as a search model.
[0068] In step SB3 of the flowchart shown in Fig. 5, search conditions are set. The search conditions include, for example, the number of workpieces WK that can be detected during a search, a lower limit of the correlation value between the search model and the workpieces WK, feature extraction conditions (edge extraction threshold value, etc.), etc. The search conditions are set by the user and are accepted by the setting unit 21.
[0069] Next, a description will be given of the flowchart showing the procedure for setting the hand model and the gripping position shown in Fig. 6. In step SC1 after the start, a robot model that specifies various information related to the robot RBT used at the site is registered.
[0070] After the robot model is registered in step SC1, the process proceeds to step SC2 where a robot hand model is registered. The robot hand model is a model that represents the shape of a robot hand, and for example, polygon data (CAD data) of the robot hand is read in.
[0071] Then, proceed to step SC3, where the relative position and orientation of the robot hand model registered in step SC2 and the flange surface of the robot model (registered in step SC1) to which the robot hand model is to be attached is defined. Specifically, the positional relationship between the mounting surface of the robot hand model and the flange surface of the robot model is defined so that they coincide with each other.
[0072] In step SC4 following step SC3, the tip coordinates of the robot hand model attached to the flange surface of the robot model are specified. These tip coordinates can be obtained from the robot controller 6, but can also be set by the user. After that, in step SC5, it is confirmed that the robot hand model has been registered.
[0073] Then, in step SC6, a surface of the search model to be grasped by the hand model is selected. The surface of the search model can be represented by a height image registered in the flowchart shown in Fig. 5, and the user selects a height image corresponding to a surface that can be grasped by the robot hand model from among multiple registered height images.
[0074] After selecting the height image in step SC6, the process proceeds to step SC7, where the position and posture of the robot hand HND when gripping the surface selected in step SC6 is registered. The gripping information indicating the gripping candidate positions of the workpiece WK and the posture of the robot hand HND can be stored in the gripping information storage unit 42a (shown in FIG. 2) of the storage device 42 of the control device 100 in a state associated with the search model.
[0075] When picking up the workpiece WK with the actual robot hand HND, it is necessary to convert the vision coordinates, which are the coordinates of the three-dimensional space (vision space) in which the workpiece WK is imaged by the three-dimensional camera 2, into the robot coordinates that the robot controller 6 uses when actually operating the robot RBT. The coordinates can be given in robot coordinates (XYZ, RxRyRz), but they can also be given in the axis angle (joint angle) of the arm section ARM. In addition, the control device 100 can obtain the axis angle from the robot coordinates such as the gripping position, and transmit the axis angle to the robot controller 6.
[0076] Specifically, the position and orientation of the search model of the workpiece WK are obtained as the position and orientation in the vision space. Similarly, the orientation of the robot hand model that grasps the workpiece WK is obtained as the position and orientation in the virtual three-dimensional space of the control device 100. In order for the robot controller 6 to operate the robot RBT based on the position and orientation in the vision space, these must be converted into the position and orientation in the robot space. The process of calculating a conversion formula for converting the position and orientation calculated in this displayed coordinate system into the position and orientation in the coordinate system in which the robot controller 6 operates the robot hand HND can be performed by a conventionally known method.
[0077] In step SC8 of the flowchart shown in Fig. 6, it is determined whether or not the required number of gripping candidate positions have been registered. If the determination in step SC8 is NO and the required number of gripping candidate positions have not been registered, the process proceeds to step SC7 via step SC6, where other gripping candidate positions are set and registered. On the other hand, if the determination in step SC8 is YES, the process proceeds to step SC9.
[0078] In step SC9, the start position and the placement position where the grasped workpiece WK is placed are registered. The start position here is the position where the robot hand HND grasps the workpiece WK, and corresponds to the transfer start position. The start position can be a candidate gripping position for the workpiece WK bulk loaded in the container. The start position differs depending on where the workpiece WK is located in the container, but this can be obtained by a three-dimensional search. On the other hand, the placement position is the position where the workpiece WK is placed by the robot hand HND, and can be set, for example, on the stage STG shown in FIG. 1.
[0079] In step SC10, the via points that the workpiece WK will pass through while moving from the gripping position to the placement position are registered. When the via points are registered, the operations of the arm unit ARM and the robot hand HND are set so that a part of the robot hand HND or a part of the workpiece WK passes through the via points while the robot hand HND grips the workpiece WK and transports it from the start position to the placement position.
[0080] (Basic robot control) 7 is a diagram showing an example of a flowchart of the basic control of the robot RBT. In step SD1 after the start, the measurement unit 20 captures an image of the workpiece WK to measure the position and orientation of the workpiece WK, and generates data for search processing (height image data) executed in step SD2. Since the measurement unit 20 is capable of three-dimensional measurement, the data for search processing can include height information.
[0081] Then, proceed to step SD2, and execute a search process to detect the position and orientation of the workpiece WK from the search process data generated in step SD1. Specifically, the orientation and position of the workpiece WK are identified using the registered search model of the workpiece WK, and first, the position and orientation that are most closely matched by each feature point of the search model are searched for in the search process data.
[0082] If the result of searching for the presence or absence of a registered search model is that the search model does not exist in the search processing data, that is, if the work WK cannot be detected, this flowchart ends.
[0083] 2 determines a gripping position of the workpiece WK arranged in the workspace by the robot hand HND based on the measurement result by the measurement unit 20 and the gripping information stored in the gripping information storage unit 42a. After determining the gripping position, the positioning processing unit 22 is configured to calculate a movement target coordinate of the robot hand HND for gripping the gripping position.
[0084] At this time, the judgment result of the interference judgment unit 25 shown in Fig. 2 can be used. The interference judgment unit 25 is a part that judges the interference between the robot hand HND and an obstacle. For example, it can judge whether or not the model of the robot hand HND registered by the process shown in the flowchart in Fig. 6 will come into contact with an obstacle such as a wall of the container BX.
[0085] In step SD3, the path generating unit 23 shown in Fig. 2 generates a transport path for transporting the workpiece WK grasped by the robot hand HND to the place position. In step SD4, the output unit 24 outputs the coordinates of the gripping position, placement position, via point, etc. and the movement target coordinates calculated in step SD2 to the robot controller 6. The robot controller 6 receives the coordinates of the gripping position, placement position, via point, etc. output from the output unit 24, generates a control signal, and outputs the generated control signal to the robot RBT. Thereafter, in step SD5, the robot RBT operates according to the control signal to transport the workpiece WK along a predetermined path.
[0086] (calibration) The robot system 1000 can be operated according to the flowchart shown in FIG. 7. However, since the robot system 1000 is a hand-eye type robot system, before operation, it is necessary to perform a calibration for converting the camera coordinates of the three-dimensional camera 2 into the robot coordinates of the robot hand HND, that is, a so-called hand-eye calibration.
[0087] When performing calibration, calibration marks, a calibration board, and the like may be used as disclosed in the above Patent Document 1, but since the calibration marks and the calibration board are different from the workpiece WK in terms of material, shape, size, and the like, it is necessary to image the calibration marks and the calibration board by changing the imaging parameters and the focal position of the three-dimensional camera 2 from those used when imaging the actual workpiece WK. In other words, since calibration is performed with imaging parameters that are different from those used when the robot system 1000 is in operation, it is difficult to say that a calibration result that is optimal for operation can be obtained.
[0088] Therefore, in this embodiment, the workpiece WK that is actually picked during operation is used for calibration. The control device 100 is configured to measure the shape of the workpiece WK using a three-dimensional camera 2 fixed to the robot hand HND, and to perform multiple stages of calibration based on the shape data of the measured workpiece WK.
[0089] The calibration procedure is roughly divided into first to third steps SE1 to SE3 as shown in FIG. 8. In the first step SE1 after the start, the robot hand HND and the three-dimensional camera 2 are translated in the XYZ directions, the camera coordinates and the robot coordinates are obtained when the movement of the robot hand HND and the three-dimensional camera 2 is stopped, and the correspondence between the camera coordinates and the robot coordinates is obtained. In the second step SE2, the robot hand HND and the three-dimensional camera 2 are rotated several times around a predetermined axis, the camera coordinates and the robot coordinates are obtained when the movement of the robot hand HND and the three-dimensional camera 2 is stopped, and a provisional correspondence between the rotation center and the position of the three-dimensional camera 2 is obtained. In the third step SE3, the robot hand HND and the three-dimensional camera 2 are rotated based on the provisional correspondence obtained in the second step SE2, and a correction value of the rotation center is obtained. Based on the obtained correction value, the provisional rotation center obtained in the second step SE2 is corrected.
[0090] 8, the first step SE1 and the second step SE2 are shown as separate steps, but this is not limiting, and the processing content of the first step SE1 and the processing content of the second step SE2 may be executed in one step. Also, in the second step SE2, a provisional correspondence relationship between the rotation center and the position of the three-dimensional camera 2 is obtained, but if this relationship is determined in advance by a design value, the second step SE2 can be omitted and substituted by reading in the design value.
[0091] Moreover, the first step SE1 and the second step SE2 are defined as normal calibration processing because they are steps for calculating parameters for converting a position in the camera coordinates to a position in the robot coordinates and for determining a provisional correspondence relationship between the center of rotation and the position of the three-dimensional camera 2. The third step SE3 is defined as high-precision calibration processing because it is a step for correcting the provisional center of rotation determined in the second step SE2 to increase accuracy. The normal calibration processing and high-precision calibration processing are performed by the calibration execution unit 26 shown in FIG. 2.
[0092] The control device 100 is configured to be able to execute an operation mode selected when operating the robot system 1000 described above and a calibration mode selected when performing calibration. When a user selects a calibration mode for performing calibration before operating the robot system 1000, the calibration execution unit 26 detects that the calibration mode has been selected by the control device 100. When the calibration execution unit 26 detects that the calibration mode has been selected, the normal calibration process and the high-precision calibration process are executed in sequence and continuously. "Executing continuously" means that after the normal calibration process is completed, the high-precision calibration process is started using the result of the normal calibration process without the user having to perform any special operation, thereby reducing the burden on the user. Note that the high-precision calibration process may be started after the normal calibration process is completed upon receiving an instruction from the user, without the normal calibration process and the high-precision calibration process being executed continuously. Details of each of the first to third steps SE1 to SE3 will be described below.
[0093] (Details of the first step) 9 is a flowchart showing a detailed procedure of the first step SE1 shown in FIG. 8. When the first step SE1 is started, the calibration execution unit 26 generates a calibration user interface 50 consisting of an image as shown in FIG. 10 and displays it on the display unit 3. The calibration user interface 50 is provided with a two-dimensional image display area 51 that displays a color image of the workpiece WK captured by the three-dimensional camera 2, and a three-dimensional image display area 52 that displays a height image based on three-dimensional shape data acquired from the image captured by the three-dimensional camera 2. In reality, the three-dimensional image display area 52 displays a workpiece image colored in different colors according to the height, but coloring is omitted in the figure of this example.
[0094] Furthermore, the calibration user interface 50 is provided with a mode display area 53 for displaying whether or not it is in a calibration mode, a movement amount display area 54 for displaying the movement amount of the robot hand HND, an error display area 55 for displaying the rotation center error, and a movement amount setting area 56 for setting the movement amount of the robot hand HND during normal calibration processing. The movement amount display area 54 displays the movement amount (X) in the X direction, the movement amount (Y) in the Y direction, the movement amount (Z) in the Z direction, the rotation amount (Rx) around the X axis, the rotation amount (Ry) around the Y axis, and the rotation amount (Rz) around the Z axis of the robot hand HND. The unit of the movement amount (X) in the X direction, the movement amount (Y) in the Y direction, and the movement amount (Z) in the Z direction is mm, and the unit of the rotation amount (Rx) around the X axis, the rotation amount (Ry) around the Y axis, and the rotation amount (Rz) around the Z axis is deg.
[0095] The movement amount setting area 56 also displays the amount of movement in the X direction (X), the amount of movement in the Y direction (Y), the amount of movement in the Z direction (Z), the amount of rotation around the X axis (Rx), the amount of rotation around the Y axis (Ry), and the amount of rotation around the Z axis (Rz) similar to those in the movement amount display area 54, and the user can input each value individually. Each value in the movement amount setting area 56 may be a predetermined value input, or the input value may be changeable by the user. The user can input or change each value by operating the operation unit 4.
[0096] In step SF1 of the flowchart shown in Fig. 9, the robot hand HND is moved to the start position of the calibration. After being moved to the start position, the process proceeds to step SF2, where the robot hand HND is translated in a lattice pattern of 3 points vertically × 3 points horizontally × 3 points high (total of 27 points) with the start position as the center, as shown in Fig. 11. The amount of movement at this time is the amount of movement in the X direction (X), the amount of movement in the Y direction (Y), and the amount of movement in the Z direction (Z) inputted in the movement amount setting area 56 of the calibration user interface 50 shown in Fig. 10.
[0097] In this step SF1, the robot hand HND is not rotated, but is moved only in the directions along the X, Y, and Z directions. In other words, the posture (θx, θy, θz) of the robot hand HND is not changed, and θx, θy, and θz are kept constant. This is to prevent the relationship between the tool center and the position of the three-dimensional camera 2 from being lost. Note that the vertical direction is the Y direction, the horizontal direction is the X direction, and the height is the Z direction. The intervals between multiple points in the X direction, the intervals between multiple points in the Y direction, and the intervals between multiple points in the Z direction can all be the same. In addition, in this embodiment, a total of 27 points are moved, but this is just one example, and the number of moving points may be less than or greater than 27. In addition, the robot hand HND does not necessarily have to be moved in the directions along the X, Y, and Z directions, and may be moved linearly with a predetermined inclination angle with respect to the X direction, may be moved linearly with a predetermined inclination angle with respect to the XY direction, or may be moved linearly with a predetermined inclination angle with respect to the Z direction.
[0098] After each operation, the workpiece WK is imaged by the three-dimensional camera 2 at the stop position of the robot hand HND. FIG. 12 is a diagram showing an example of a calibration user interface 50 displayed when the robot hand HND is moved to the position of the first point. FIG. 13 is a diagram showing an example of a calibration user interface 50 displayed when the robot hand HND is moved to the position of the 27th point. Since the position of the three-dimensional camera 2 with respect to the workpiece WK is different between the first point and the 27th point, the three-dimensional image display area 52 in FIG. 12 and the three-dimensional image display area 52 shown in FIG. 13 display three-dimensional images of the workpiece WK viewed from different angles and positions. The three-dimensional images of the workpiece WK imaged at each of the first point to the 27th point may be displayed sequentially in the three-dimensional image display area 52, or the three-dimensional image of the workpiece WK may not be displayed. As shown in FIG. 12 and FIG. 13, when imaging is started at each of the first point to the 27th point, an option display area 57 is displayed in the calibration user interface 50. The option display area 57 displays the position tolerance, normal tolerance, angle tolerance, etc. Note that the tolerances are not relevant during calibration but are applied during operation.
[0099] The calibration execution unit 26 detects the workpiece coordinates based on the three-dimensional shape data obtained by measurement using the three-dimensional camera 2, and acquires the robot coordinates corresponding to the workpiece coordinates. The robot coordinates corresponding to the workpiece coordinates are the robot coordinates when the workpiece coordinates are obtained. This is performed for all 27 points, and the workpiece coordinates and robot coordinates for each point are stored as pairs (27 pairs) in the storage device 42, etc.
[0100] As a method for identifying the position of the workpiece WK at each point, for example, there is a method for searching the three-dimensional shape acquired by the three-dimensional camera 2 and using the XYZ of the representative point (such as the center position) as the detection coordinates, and a method for searching the two-dimensional shape and using the XY coordinates of the representative point as the representative point and calculating the Z of the representative point from a separately obtained plane equation. In this embodiment, the latter identification method is adopted, and after detecting the edges of multiple points, the center coordinates of the circle are calculated by the least squares method so as to follow the change in the appearance of the workpiece WK depending on the angle of view. The coordinates of the representative point may be obtained as the coordinates of a characteristic part of the workpiece WK. For example, if there is a hole in the center of the workpiece WK, the center of the hole may be used as the representative point.
[0101] Then, the process proceeds to step SF3 in the flowchart shown in Fig. 9. In step SF3, a correction process is performed to correct the angle of view of the three-dimensional camera 2. If the angle of view of the three-dimensional camera 2 has been corrected, this step SF3 can be omitted. Furthermore, the three-dimensional camera 2 used in this embodiment is of an interchangeable lens type, and is configured so that the angle of view correction is not performed on the camera side, but it may be configured so that the angle of view correction is performed on the camera side or within the control device 100 when the lens is replaced. A non-interchangeable lens type camera may also be used in this embodiment.
[0102] Here, we will explain how the workpiece WK appears in the field of view when the workpiece WK is imaged by the three-dimensional camera 2. In step SF1 above, the robot hand HND to which the three-dimensional camera 2 is fixed is translated in the vertical, horizontal, and height directions, so the position of the workpiece WK shifts relative to the field of view of the three-dimensional camera 2. In the case of a three-dimensional camera 2 that does not correct the angle of view, if the distance from the three-dimensional camera 2 changes, the area of the workpiece WK that appears in the field of view also changes. In a three-dimensional camera 2 that does not correct the angle of view, the Z direction can be said to be in mm units, and the X and Y directions can be said to be in pixel units, while in a three-dimensional camera 2 that does correct the angle of view, the X and Y directions can also be said to be in mm units.
[0103] This will be explained based on Fig. 14A and Fig. 14B. Fig. 14A is a side view showing the possible positional relationship between the workpiece WK and the three-dimensional camera 2 when measuring the first to 27th points in the first step. The surface S1 is a surface obtained by extending the top surface of the workpiece WK in the horizontal direction. The surface S2 is a horizontal plane passing through the light receiving element (or the light receiving surface of the lens) when the three-dimensional camera 2 is in the reference position, that is, the middle position (first position). The surface S3 is a horizontal plane passing through the light receiving element (or the light receiving surface of the lens) when the three-dimensional camera 2 is in the upper position (second position). If the representative point of the workpiece WK is point P1, the physical coordinates of point P1 can be expressed as (X, Z) at the reference position and as (X, Z') at the upper position.
[0104] A straight line L1 indicates the optical axis of the three-dimensional camera 2 and is aligned along the Z direction. Two straight lines L2, L2 indicate the field of view of the three-dimensional camera 2 when the three-dimensional camera 2 is in the reference position. Furthermore, two straight lines L3, L3 indicate the field of view of the three-dimensional camera 2 when the three-dimensional camera 2 is in the upper position.
[0105] The rectangle indicated by the symbol 2A in Fig. 14B indicates the field of view range when imaged by the three-dimensional camera 2. The workpiece WK imaged when the three-dimensional camera 2 is in the reference position is indicated by a solid line, and the workpiece WK imaged when the three-dimensional camera 2 is in the upper position is indicated by a dashed line. The coordinates of the representative point P1 of the workpiece WK imaged when the three-dimensional camera 2 is in the reference position are (xp, Z), and the coordinates of the representative point P1 of the workpiece WK imaged when the three-dimensional camera 2 is in the upper position are (xp', Z').
[0106] The relationship shown in FIG. 14A and FIG. 14B is expressed in a table as shown in FIG. 15. That is, the horizontal distance from the center (optical axis L1) of the three-dimensional camera 2 to the representative point P1 of the workpiece WK is xp at the middle position and xp' at the upper position when expressed in pixel distance, and these xp and xp' are known because they are detected representative points. Also, the vertical distance (distance in the Z direction) from the three-dimensional camera 2 to the representative point P1 of the workpiece WK is Z at the middle position and Z' at the upper position when expressed in physical distance, and these Z and Z' are known because they are detected representative points. Of the horizontal distances from the center (optical axis L1) of the three-dimensional camera 2 to the representative point P1 of the workpiece WK, the physical distance X is unknown, but the geometric relationship Z':Z=xp:xp' holds.
[0107] The pixel position xp' when the three-dimensional camera 2 is in the upper position is determined by the relationship between the physical size (position) X and the distance Z' from the three-dimensional camera 2 depending on the angle of view. However, if this is left as it is, the camera coordinates themselves depend on the height, making it difficult to use for calibration. Therefore, the calibration execution unit 26 of this embodiment converts the above relationship into a pixel position that is independent of the camera's angle of view (i.e., when a telecentric lens is used). There is a proportional relationship between xcp and X, that is, the corrected pixel position xcp depends on X. From this relationship, xcp can be obtained using the following formula 1 (formula for obtaining the equivalent pixel position on the reference surface). Note that, when considering lens distortion, other calculation methods can be adopted, and the calculation method is not limited to the calculation method using formula 1.
[0108] xcp=xp'*(Z' / Z) … Equation 1
[0109] The above is the correction process executed in step SF3 of the flowchart shown in FIG. 9. Then, proceed to step SF4, where the correspondence from the camera coordinates to the robot coordinates is obtained as an Affine transformation matrix by the least squares method. Specifically, in this example, the workpiece WK is measured at 27 points, so the Affine transformation matrix can be obtained by the least squares method from the relationship between the 27 robot positions Pr and the camera coordinate position Pc. Here, Pc is the corrected camera position. The corrected camera position Pc does not need to be in mm units for X, Y, and Z. In this example, X and Y are pixel unit coordinates, and Z is mm coordinates, and as long as the X, Y, and Z coordinates are in uniform space, they can be used for calculation.
[0110] By going through step SF4, the first step SE1 in the flowchart shown in Fig. 8 is completed. Fig. 16 shows a calibration user interface 50 that is displayed on the display unit 3 when the first step SE1 is completed. This calibration user interface 50 is provided with a calibration result display area 58. The field of view, pixel resolution, coordinate axis angles, coordinate axis rotation angles, rotation center position, etc. of the 3D camera 2 determined in the first step SE1 are displayed in the calibration result display area 58.
[0111] (Details of the second step) Fig. 17 is a flowchart showing the detailed procedure of the second step SE2 shown in Fig. 8. When the second step SE2 is started, the calibration execution unit 26 generates a calibration user interface 50 shown in Fig. 18 and displays it on the display unit 3. The calibration user interface 50 displayed in the second step SE2 displays in a mode display area 53 that it is in the rotation center calculation mode.
[0112] In step SG1 of the flowchart shown in FIG. 17, the robot hand HND is moved to the start position of the calibration. After being moved to the start position, the process proceeds to step SG2, where the robot hand HND is stopped at each point (total of seven points) shifted by the start position, ±θx, ±θy, ±θz with respect to the tool center coordinates, and the workpiece WK is imaged by the three-dimensional camera 2 at the stop position of the robot hand HND. The workpiece coordinates are detected based on the three-dimensional shape data obtained by measurement with the three-dimensional camera 2, and the robot coordinates corresponding to the workpiece coordinates are obtained. The workpiece coordinates and robot coordinates of each point are paired and stored in the storage device 42 or the like. Thereafter, the process proceeds to step SG3, where a provisional rotation center is obtained based on the robot coordinates.
[0113] Specifically, when determining the center of rotation from the detected coordinates of seven points, the least squares method using the equation of a sphere can be used. In this example, the number of points is seven, but it is possible to determine the center of rotation in three dimensions if there are four or more points. The more points there are, the better the results will be.
[0114] It is also possible to find the center of rotation geometrically without using the equation of a sphere. For example, by finding multiple bisectors and planes of the displacement vector, the center of rotation can be estimated from their intersection point. Also, the distance to the center position can be found using arctan from the relationship between the displacement distance and the angle of the displacement, and the central coordinates can be found from these multiple relationships.
[0115] The above is the second step SE2 of the calibration, but in principle, it is difficult to accurately determine the center of rotation unless the rotational movement of the robot hand HND is large. However, as shown in Fig. 19, if the rotational movement of the robot hand HND is large, the workpiece WK does not fit into the field of view, and detection of the representative point P1 becomes unstable. The three symbols 2A in Fig. 19 indicate the field of view of the three-dimensional camera 2, with the right field of view 2A indicating the field of view before rotation, and the center field of view 2A and the left field of view 2A indicating the field of view after rotation.
[0116] When the second step is completed, a calibration user interface 50 as shown in FIG. 20 is displayed, and the third step SE3 shown in FIG. 8 above is started.
[0117] (Details of the 3rd step) Fig. 21 is a flow chart showing a detailed procedure of the third step SE3 shown in Fig. 8. In the third step SE3, the robot hand HND is moved by a movement amount set in a high-precision calibration setting area 60 provided in the calibration user interface 50 shown in Fig. 22. The calibration user interface 50 shown in Fig. 22 may be displayed on the display unit 3 before the start of the first step SE1 so that each value can be input, or may be displayed on the display unit 3 after the second step SE2 is completed so that each value can be input. The movement amount set in the high-precision calibration setting area 60 may be a predetermined value.
[0118] 21, a provisional rotation center (provisional rotation center) is determined based on the relative position between the tool center of the robot hand HND and the three-dimensional camera 2, which is determined by the normal calibration process. The determined provisional rotation center is stored in the storage device 42 or the like as the original rotation center. Then, the process proceeds to step SH2, where the robot hand HND is moved to a position (start position) where the representative point P1 of the workpiece WK can be stably determined.
[0119] Next, proceed to step SH3, and using the relationship between the camera position and the original rotation center position, rotate the robot hand HND (three-dimensional camera 3) at the starting position, stop the robot hand HND at each point (seven points in total) shifted from the starting position by ±θx, ±θy, ±θz, and capture an image of the workpiece WK with the three-dimensional camera 2 at the stopping position of the robot hand HND. Based on the three-dimensional shape data obtained by measurement with the three-dimensional camera 2, the workpiece coordinates (coordinates of the representative point P1) are detected, and the robot coordinates corresponding to the workpiece coordinates are obtained. The workpiece coordinates and robot coordinates of each point are paired and stored in a storage device 42 or the like.
[0120] When setting the rotation angle, the calibration execution unit 26 sets the rotation angle (θx, θy, θz) of the robot hand HND rotated around the temporary rotation center in the high-precision calibration process to a rotation angle larger than the rotation angle of the robot hand HND rotated around the XYZ axes in the normal calibration process. Note that the rotation angle of the robot hand HND may be the same in the high-precision calibration process and the normal calibration process.
[0121] In step SH3, the provisional positional relationship between the tool center and the camera position is obtained by the normal calibration process, and the robot hand HND can be rotated while the workpiece WK is in the field of view so that the representative point P1 of the workpiece WK can be stably obtained by using the provisional positional relationship. That is, as shown in FIG. 23A, if the robot hand HND is simply rotated around the provisional rotation center, a part of the workpiece WK will be out of the field of view 2A after the rotation, and as a result, the representative point P1 will be located outside the field of view 2A and will not be measurable. On the other hand, as shown in FIG. 23B, the calibration execution unit 26 rotates the robot hand HND around the provisional rotation center by a predetermined angle in this high-precision calibration process, and moves the robot hand HND so that the workpiece WK after the predetermined angle rotation is in the field of view 2A of the three-dimensional camera 2. Therefore, the entire workpiece WK is located within the field of view 2A, and as a result, the representative point P1 can be positioned within the field of view 2A.
[0122] In other words, the expected amount of movement when rotated a specified angle is calculated based on the original center of rotation, and the robot hand HND is moved or rotated using that amount of movement and imaged by the three-dimensional camera 2. From the representative point obtained from this, the position of the representative point that was originally desired can be obtained. Specifically, since the provisional center of rotation is obtained in the normal calibration process and the position after movement due to rotation can be estimated, the movement of the camera position can be offset by controlling the tool center to move in the opposite direction of that movement vector. This makes it possible to rotate the robot hand HND significantly while stably detecting the representative point of the workpiece WK.
[0123] For example, if there is a deviation between the original rotation center and the true rotation center position as shown in Figure 24, the deviation can be found as follows. Note that p, c, and d are vectors, and Q is a rotation matrix. If the origin rotation center is c and the detected deviation of the representative point p is d, the following relationship is established.
[0124] Q(pc)+d+c … Formula 2 Q(p-(c+x))+(c+x) … Equation 3
[0125] Since Equation 2 and Equation 3 are equivalent, the center deviation x and the true center of rotation c+x can be calculated. Note that I is a unit vector.
[0126] d=-Qx+x d=(IQ)x x=(IQ)^-1d
[0127] In the case of two dimensions, the inverse matrix of (IQ) can be found stably. In the case of three dimensions, the above relationship alone will result in a rank loss, so we increase the number of relationships and use the least squares method to find the solution. To do this, θx, θy, and θz are each moved in the ± direction.
[0128] Then, the process proceeds to steps SH4 to SH6 in the flow chart shown in FIG. 21, where the final rotation center of the robot hand HND is determined based on the pair of workpiece coordinates and robot coordinates of each point held in step SH3. To be more specific, first, in step SH4, the rotation center is recalculated (corrected) based on the pair of workpiece coordinates and robot coordinates of each point held in step SH3. The recalculated value is saved, and this value is overwritten as the next original rotation center. Steps SH2 to SH4 are repeated N times, and the value of the rotation center for each time is saved. For example, the first loop can be called the first process, and the second loop can be called the second process, and the calibration execution unit 26 can determine the final rotation center of the robot hand HND after at least the first process and the second process.
[0129] After steps SH2 to SH4 are repeated N times, the process proceeds to step SH5. In step SH5, the average of the N rotation center values is calculated. Then, rotation centers that are farther away from the average than a predetermined threshold value are marked in order to remove abnormal values.
[0130] Then, proceed to step SH6, and calculate an average using only the rotation centers that were not marked in step SH5, and the obtained average is used as the final rotation center.
[0131] In this flow chart, the original rotation center is updated for each loop (a loop consisting of steps SH2 to SH4). A conceptual diagram of updating the original rotation center for each loop is shown in FIG. 25. In FIG. 25, the initial original rotation center is the provisional rotation center obtained by the normal calibration process, and as shown by the arrow, the rotation center is updated for each loop. As shown in this example, the rotation center may deviate significantly from the average. If the rotation center deviates significantly, it can be estimated that the rotation center is an abnormal value, so that a process of removing the abnormal value may be executed when updating the original rotation center. Specifically, in this flow, after calculating N rotation centers by going through the above loop N times, those calculated rotation centers that deviate from the average by a predetermined threshold value or more are eliminated. Alternatively, a threshold value can be subtracted from the correction amount for each loop to eliminate rotation centers with a large error. This process can also be performed by the calibration execution unit 26.
[0132] When the original rotation center is updated for each loop, the obtained rotation center gradually approaches the true rotation center, and the obtained rotation center converges. In this embodiment, for example, a convergence value display area 62 is provided in the calibration user interface 50 shown in FIG. 20. The convergence value display area 62 can display the state in which the rotation center converges. As an example of the display, the convergence value (unit: mm) of the rotation center is displayed. For example, the convergence value of the rotation center obtained in the third loop is displayed as a smaller value than the convergence value of the rotation center obtained in the second loop, and the convergence value displayed in the convergence value display area 62 becomes smaller as the number of loops increases. Therefore, by looking at the convergence value displayed in the convergence value display area 62, the user can understand that the obtained rotation center gradually approaches the true rotation center. The number of loops may be two.
[0133] If the convergence value does not change by a predetermined amount or more even if the number of loops is increased, the high-precision calibration process can be terminated at that point. Specifically, the calibration execution unit 26 judges whether or not the multiple rotation centers acquired by repeatedly executing the high-precision calibration process satisfy a predetermined convergence condition. The calibration execution unit 26 can be configured to terminate the high-precision calibration process when it is determined that the predetermined convergence condition is satisfied, and to continue the high-precision calibration process when it is determined that the predetermined convergence condition is not satisfied. An example of the predetermined convergence condition is that the change in the error of the rotation center acquired by the high-precision calibration process is equal to or less than a certain amount. If the change in the error of the rotation center is equal to or less than a certain amount for multiple consecutive times (for example, three times), it may be determined that the predetermined convergence condition is satisfied. The predetermined convergence condition may be determined in advance, or may be set arbitrarily by the user.
[0134] When the high-precision calibration process is completed and the final center of rotation is determined, the calibration execution unit 26 generates a calibration user interface 50 shown in Fig. 26 and displays it on the display unit 3, while notifying the user that the high-precision calibration process is completed. The calibration user interface 50 displayed after the high-precision calibration process is completed has a calibration result display area 61. The calibration result display area 61 displays the field of view, pixel resolution, coordinate axis angles, coordinate axis rotation angles, final rotation center position, etc. of the three-dimensional camera 2 acquired through both the normal calibration process and the high-precision calibration process.
[0135] (Effects of the embodiment) As described above, according to this embodiment, the three-dimensional camera 2 is fixed to the robot hand HND. Therefore, when the robot system 1000 is in operation, the three-dimensional camera 2 can measure the position of the workpiece WK without significantly retracting the robot hand HND, and the three-dimensional shape of the workpiece WK can be measured from close to the workpiece WK to be measured, improving measurement accuracy.
[0136] In addition, during calibration, the workpiece WK can be used without using a calibration mark or a calibration board. At this time, since parameters for converting the position in the camera coordinates to the position on the robot coordinates can be calculated first by the normal calibration process, the relative positional relationship between the tool center of the robot hand HND and the three-dimensional camera 2 can be obtained. After that, a provisional rotation center of the robot hand HND is determined, and the robot hand HND is rotated multiple times around the provisional rotation center, so that the workpiece WK does not go out of the field of view of the three-dimensional camera 2, the entire workpiece WK is placed in the field of view of the three-dimensional camera 2, and the workpiece WK is placed near the center of the field of view of the three-dimensional camera 2. Then, at each position, the workpiece WK is measured by the three-dimensional camera 2 to detect the workpiece coordinates, and the robot coordinates at the time when the workpiece coordinates are obtained are obtained, and the final rotation center of the robot hand HND can be determined with high accuracy based on the obtained pairs of multiple workpiece coordinates and robot coordinates.
[0137] The above-described embodiment is merely illustrative in all respects and should not be construed as limiting. Furthermore, all modifications and variations within the scope of the claims are within the scope of the present invention. [Industrial Applicability]
[0138] As described above, the control device according to the present invention can be applied to a robot system. [Explanation of symbols]
[0139] 2. 3D Camera 3 Display section 26 Calibration execution unit 50 Calibration User Interface 100 Control device HND Robot Hand RBT Robot WK Work
Claims
1. A control device that measures a shape of a workpiece using a three-dimensional camera fixed to a robot hand, and performs calibration to convert camera coordinates of the three-dimensional camera into robot coordinates of the robot hand based on measured shape data of the workpiece, a first normal calibration process for executing an operation of moving the robot hand in X, Y and Z directions, detecting workpiece coordinates based on shape data obtained by measuring the workpiece with the three-dimensional camera at a stop position of the robot hand, acquiring robot coordinates at the time when the workpiece coordinates are obtained, and calculating parameters for converting a position in the camera coordinates to a position on the robot coordinates based on a plurality of pairs of the acquired workpiece coordinates and robot coordinates; a second normal calibration process for executing an operation of rotating the robot hand around a predetermined axis, detecting workpiece coordinates based on shape data obtained by measuring the workpiece with the three-dimensional camera rotated in conjunction with the rotation of the robot hand at a stop position of the robot hand, acquiring robot coordinates at the time when the workpiece coordinates were obtained, and determining a provisional rotation center of the robot hand based on the robot coordinates based on the acquired pairs of multiple workpiece coordinates and robot coordinates; a high-precision calibration process in which the provisional rotation center is set as the original rotation center of the robot hand, an operation is performed to rotate the robot hand around the original rotation center, work coordinates are detected based on shape data obtained by measuring the work at the stop position of the robot hand using the three-dimensional camera that rotates in conjunction with the rotation of the robot hand, the robot coordinates at the time the work coordinates are obtained are obtained, a next rotation center is determined based on the obtained pairs of multiple work coordinates and robot coordinates, the original rotation center of the robot hand is updated to the next rotation center, and the operation of rotating the robot hand around each updated original rotation center is repeated to determine a final rotation center of the robot hand; A calibration execution unit that executes the following: The calibration execution unit is a control device that, during the high-precision calibration process, determines the next rotation center corresponding to the original rotation center based on the deviation between the work coordinates after the robot hand has rotated, which are estimated based on the original rotation center, and the work coordinates after the robot hand has rotated, which are detected based on measurements by the three-dimensional camera.
2. 2. The control device according to claim 1, The calibration execution unit is a control device that sets the rotation angle of the robot hand rotated around the original rotation center in the high-precision calibration process to a rotation angle larger than the rotation angle of the robot hand rotated around the specified axis in the second normal calibration process.
3. 3. The control device according to claim 1, The calibration execution unit repeatedly executes a process of determining the next rotation center corresponding to each of the updated original rotation centers to obtain multiple rotation centers of the robot hand, and determines the final rotation center based on the multiple rotation centers obtained.
4. 4. The control device according to claim 3, The calibration execution unit determines whether the multiple rotation centers obtained by executing the repetition satisfy a predetermined convergence condition, and terminates the repetition when it is determined that the predetermined convergence condition is satisfied.
5. 5. The control device according to claim 1, The calibration execution unit is a control device that rotates the robot hand a predetermined angle around the original rotation center during the high-precision calibration process and moves the robot hand so that the workpiece after rotation by the predetermined angle is within the field of view of the three-dimensional camera.
6. 6. The control device according to claim 1, The calibration execution unit is a control device that calculates the XYZ coordinates of a representative point of the work based on three-dimensional shape data obtained by measuring the work using the three-dimensional camera, and defines the calculated XYZ coordinates of the representative point as work coordinates.
7. 7. The control device according to claim 1, The calibration execution unit is a control device that calculates the XY coordinates of a representative point of the work based on two-dimensional shape data obtained by measuring the work using the three-dimensional camera, and calculates the Z coordinate of the representative point from a planar equation, and sets the calculated XYZ coordinates of the representative point as work coordinates.
8. The control device according to any one of claims 1 to 7, The calibration execution unit detects work coordinates based on shape data obtained by moving the robot hand to a first position and measuring the work with the three-dimensional camera, and detects work coordinates based on shape data obtained by moving the robot hand to a second position having a Z coordinate different from the first position and measuring the work with the three-dimensional camera, and converts the work coordinates detected at the first position and the work coordinates detected at the second position into pixel positions independent of the angle of view of the three-dimensional camera.
9. 9. The control device according to claim 1, It is configured to allow selection of the calibration mode in which calibration is performed, The calibration execution unit is a control device that, when detecting that the calibration mode has been selected, successively executes the first normal calibration process, the second normal calibration process, and the high-precision calibration process in that order.
10. 10. The control device according to claim 1, The calibration execution unit generates a user interface having a display area capable of updating and displaying the detection results of the work coordinates detected in the high-precision calibration process and the convergence value of the rotation center, and is equipped with a display unit that displays the user interface generated by the calibration execution unit.
Citation Information
Patent Citations
Method and system for recognizing three-dimensional position and attitude on the basis of sense of sight
JP1995098208A
Coordinate system calibration method and robot system
JP2012091280A
Data creation system of visual sensor, and detection simulation system
JP2015136770A
Robot system, calibration method of robot system and position correction method of robot system
JP2015174191A
Robot simulation device and robot simulation method
JP2019028773A
