Robot device equipped with a 3D sensor and control method for the robot device

Abstract

This robot device comprises: a positional information generation unit that generates three-dimensional positional information of a surface of a workpiece on the basis of an output from a visual sensor; and a face inference unit that infers face information related to a face including the surface of the workpiece on the basis of the three-dimensional positional information. The robot moves the visual sensor from a first position to a second position. A correction amount setting unit sets a correction amount for driving the robot in the second position so that a first face including the surface of the workpiece detected in the first position matches a second face including the surface of the workpiece detected in the second position.

Description

Technical Field

[0001] The present invention relates to a robot apparatus provided with a three-dimensional sensor and a method for controlling the robot apparatus.

Background Art

[0002] A robot apparatus including a robot and a working tool can perform various operations by changing the position and orientation of the robot. In order for the robot to perform an operation at a position and orientation corresponding to the position and orientation of a workpiece, it is known to detect the position of the workpiece with a three-dimensional sensor (for example, Japanese Patent Laid-Open No. 2004-144557). By driving the robot based on the position and orientation of the workpiece detected by the three-dimensional sensor, the robot apparatus can perform operations with high accuracy.

[0003] By using a three-dimensional sensor, a plurality of three-dimensional points can be set on the surface of a workpiece contained inside a measurement area, and the positions of each of the three-dimensional points can be detected. Further, based on the positions of the plurality of three-dimensional points, a distance image or the like having different densities according to the distance can be generated.

[0004] When the workpiece is large with respect to the measurement area of the three-dimensional sensor, the robot apparatus can perform measurements at a plurality of positions while moving the three-dimensional sensor. The three-dimensional point clouds acquired by arranging the three-dimensional sensor at the plurality of positions can be combined. For example, a three-dimensional camera is fixed to the hand of the robot apparatus. The position and orientation of the robot can be changed to perform imaging at a plurality of positions. Then, the three-dimensional point clouds measured at each position can be combined to generate one large three-dimensional point cloud.

[0005] Alternatively, if the surface of the workpiece is glossy, the position of a part of the workpiece may not be measurable due to light halation (for example, Japanese Patent Publication No. 2019-113895). When such halation occurs, the 3D points of the part whose position could not be measured can be compensated for by taking images at multiple positions while changing the position of the 3D sensor. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2004-144557 [Patent Document 2] Japanese Patent Publication No. 2019-113895 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] When a robot's control system calculates the position of a 3D point set on the surface of a workpiece, it converts the position in the sensor coordinate system set on the 3D sensor to the position in the robot coordinate system. At this time, the position of the 3D point is converted based on the robot's position and orientation. However, if there are errors in the robot's position and orientation, these errors may affect the accuracy of the 3D point position. For example, errors in the robot's position and orientation due to backlash in the reduction gear can cause errors in the position of the 3D point in the robot coordinate system. In particular, when 3D points are measured from multiple locations and a 3D point cloud is synthesized, controlling the robot based on the synthesized 3D point cloud can lead to inaccurate control. [Means for solving the problem]

[0008] A robotic device according to an aspect of the present disclosure includes a three-dimensional sensor for detecting the position of the surface of a workpiece, and a robot for changing the relative position between the workpiece and the three-dimensional sensor. The robotic device includes a position information generation unit for generating three-dimensional position information of the surface of the workpiece based on the output of the three-dimensional sensor, and a surface estimation unit for estimating surface information relating to a surface including the surface of the workpiece based on the three-dimensional position information. The robotic device includes a determination unit that compares the length and direction of normal vectors based on surface information for multiple surfaces. The robot device includes a correction amount setting unit for setting a correction amount for driving the robot. The robot is configured to change the relative position between the workpiece and the 3D sensor from a first relative position to a second relative position different from the first relative position. The surface estimation unit estimates surface information of a first surface including the surface of the workpiece detected at a first relative position, and surface information of a second surface including the surface of the workpiece detected at a second relative position. The determination unit determines whether the difference in length and direction between the normal vector from a predetermined point to the first surface and the normal vector from a predetermined point to the second surface is within a predetermined determination range. The correction amount setting unit is, If the difference in length and the difference in direction are within a predetermined range, Set the correction amount for driving the robot at the second relative position.

[0009] A control method for a robot apparatus according to an aspect of the present disclosure includes the steps of: the robot positioning the relative positions of the workpiece and the 3D sensor to a first relative position; and the position information generation unit generating 3D position information of the workpiece surface at the first relative position based on the output of the 3D sensor. The control method includes the steps of: the robot positioning the relative positions of the workpiece and the 3D sensor to a second relative position different from the first relative position; and the position information generation unit generating 3D position information of the workpiece surface at the second relative position based on the output of the 3D sensor. The control method includes the surface estimation unit determining the 3D position information at each relative position, The control method includes a step of estimating surface information of a first surface including the surface of the workpiece detected at a first relative position, and surface information of a second surface including the surface of the workpiece detected at a second relative position. The control method includes a step in which a determination unit determines, based on the surface information of the first surface and the surface information of the second surface, whether the difference in length and direction between the normal vector from a predetermined point to the first surface and the normal vector from a predetermined point to the second surface is within a predetermined determination range. The control method is The correction amount setting unit, If the difference in length and the difference in direction are within a predetermined range, The process includes setting a correction amount for driving the robot at a second relative position. A control device for controlling a robot according to an embodiment of the present disclosure includes a processing unit that estimates surface information of a surface including the surface of a workpiece based on the output of a 3D sensor. The processing unit estimates surface information of a first surface including the surface of the workpiece detected at a first relative position representing the relative position between the workpiece and the 3D sensor, and surface information of a second surface including the surface of the workpiece detected at a second relative position different from the first relative position. The processing unit determines whether the difference in length and direction between the normal vector from a predetermined point to the first surface and the normal vector from a predetermined point to the second surface is within a determination range. If the difference in length and direction is within the determination range, it sets a correction amount for driving the robot at the second relative position. A control method for controlling a robot according to an embodiment of the present disclosure comprises the steps of: a processing unit of a robot control device estimating, based on the output of a 3D sensor, surface information of a first surface including the surface of a workpiece detected at a first relative position representing the relative position between the workpiece and the 3D sensor, and surface information of a second surface including the surface of a workpiece detected at a second relative position different from the first relative position; a processing unit determining whether the difference in length and direction between a normal vector from a predetermined point to the first surface and a normal vector from a predetermined point to the second surface is within a determination range; and a processing unit setting a correction amount for driving the robot at the second relative position if the difference in length and direction is within a determination range. [Effects of the Invention]

[0010] The robot apparatus and control method for the robot apparatus according to the embodiments of this disclosure can set a robot correction amount that reduces the error of the three-dimensional position information obtained from the output of the three-dimensional sensor. [Brief explanation of the drawing]

[0011] [Figure 1] It is a perspective view of the workpiece and the first robot device in the embodiment. [Figure 2] It is a block diagram of the first robot device in the embodiment. [Figure 3] It is a schematic diagram of the vision sensor in the embodiment. [Figure 4] It is a perspective view of the vision sensor and the workpiece for explaining the three-dimensional point cloud and the distance image. [Figure 5] It is a perspective view for explaining the three-dimensional point cloud set on the surface of the workpiece. [Figure 6] It is an example of a distance image generated based on the output of the vision sensor. [Figure 7] It is a perspective view of the workpiece and the first robot device when the vision sensor is moved to the second position. [Figure 8] It is a schematic cross-sectional view when there is no error in the second position when the vision sensor is moved to the second position. [Figure 9] It is a schematic cross-sectional view when an error occurs in the second position when the vision sensor is moved to the second position. [Figure 10] It is a schematic cross-sectional view for explaining the position of the three-dimensional point cloud in the robot coordinate system when an error occurs in the second position of the vision sensor. [Figure 11] It is a schematic cross-sectional view of the vision sensor and the workpiece for explaining the correction amount of the position of the vision sensor. [Figure 12] It is a flowchart of the control performed during the teaching operation of the robot device in the embodiment. [Figure 13] It is a flowchart of the control of the operation for transporting the workpiece in the embodiment. [Figure 14] It is a perspective view of the second workpiece and the vision sensor in the embodiment. [Figure 15] It is a block diagram of the surface estimation unit in a modified example of the first robot device. [Figure 16] It is a perspective view of the third workpiece and the vision sensor in the embodiment. [Figure 17] It is a schematic cross-sectional view of the fourth workpiece and the vision sensor in the embodiment. [Figure 18] It is a schematic view of the second robot device in the embodiment.

Embodiments for Carrying Out the Invention

[0012] Referring to FIGS. 1 to 18, a robot device and a control method for the robot device in the embodiment will be described. The robot device of this embodiment includes a three-dimensional sensor for detecting the position of the surface of a workpiece as an object to be worked on. By processing the output of the three-dimensional sensor, three-dimensional position information such as the position of three-dimensional points is acquired. First, a first robot device including a robot that changes the position and orientation of the three-dimensional sensor will be described.

[0013] FIG. 1 is a perspective view of the first robot device in this embodiment. FIG. 2 is a block diagram of the first robot device in this embodiment. Referring to FIGS. 1 and 2, the first robot device 3 conveys the workpiece 65. The first robot device 3 includes a hand 5 as a work tool for gripping the first workpiece 65 and a robot 1 as a moving mechanism for moving the hand 5. The robot device 3 includes a control device 2 for controlling the robot 1 and the hand 5. The robot device 3 includes a vision sensor 30 as a three-dimensional sensor that outputs a signal for detecting the position of the surface of the workpiece 65.

[0014] The first workpiece 65 is a plate-shaped member having a planar surface 65a. The workpiece 65 is disposed on the surface 69a of a gantry 69 as a mounting member. In the first robot device 3, the position and orientation of the workpiece 65 are fixed. The hand 5 of this embodiment grips the workpiece 65 by suction. The work tool is not limited to this form, and any work tool corresponding to the work performed by the robot device 3 can be adopted. For example, a work tool for welding or a work tool for applying a sealing material can be adopted.

[0015] Robot 1 is a vertical articulated robot including multiple joints 18. Robot 1 includes an upper arm 11 and a lower arm 12. The lower arm 12 is supported by a swivel base 13. The swivel base 13 is supported by a base 14. The base 14 is fixed to a mounting surface. Robot 1 includes a wrist 15 connected to the end of the upper arm 11. The wrist 15 includes a flange 16 for securing a hand 5. Robot 1 in this embodiment has six drive axes, but is not limited to this configuration. The robot can employ any robot capable of moving a work tool.

[0016] The vision sensor 30 is attached to the flange 16 via a support member 36. In the first robot device 3, the vision sensor 30 is supported by the robot 1 so that its position and orientation change together with the hand 5.

[0017] The robot 1 of this embodiment includes a robot drive unit 21 that drives components of the robot 1, such as the upper arm 11. The robot drive unit 21 includes a plurality of drive motors for driving the upper arm 11, the lower arm 12, the slewing base 13, and the wrist 15. The hand 5 includes a hand drive unit 22 that drives the hand 5. The hand drive unit 22 of this embodiment drives the hand 5 by air pressure. The hand drive unit 22 includes a vacuum pump and a solenoid valve, etc., that supply reduced-pressure air to the hand 5.

[0018] The control device 2 includes an arithmetic processing unit 24 (computer) which includes a CPU (Central Processing Unit) as a processor. The arithmetic processing unit 24 has RAM (Random Access Memory) and ROM (Read Only Memory), etc., which are connected to each other via a bus to the CPU. The robot device 3 is driven by the robot 1 and hand 5 based on the operation program 41. The robot device 3 has the function of automatically transporting the workpiece 65.

[0019] The arithmetic processing unit 24 of the control device 2 includes a storage unit 42 that stores information related to the control of the robot device 3. The storage unit 42 can be configured with a non-temporary storage medium capable of storing information. For example, the storage unit 42 can be configured with a storage medium such as volatile memory, non-volatile memory, magnetic storage medium, or optical storage medium. An operation program 41, which has been created in advance to perform the operation of the robot 1, is stored in the storage unit 42.

[0020] The arithmetic processing unit 24 includes an operation control unit 43 that sends operation commands. The operation control unit 43 sends operation commands to the robot drive unit 44 to drive the robot 1 based on the operation program 41. The robot drive unit 44 includes an electrical circuit that drives the drive motor. The robot drive unit 44 supplies electricity to the robot drive unit 21 based on the operation commands. The operation control unit 43 also sends operation commands to the hand drive unit 45 to drive the hand drive unit 22. The hand drive unit 45 includes an electrical circuit that drives a pump or the like. The hand drive unit 45 supplies electricity to the hand drive unit 22 based on the operation commands.

[0021] The operation control unit 43 corresponds to a processor that operates according to the operation program 41. The processor reads the operation program 41 and functions as the operation control unit 43 by executing the controls defined in the operation program 41.

[0022] Robot 1 includes a state detector for detecting the position and orientation of robot 1. In this embodiment, the state detector includes a position detector 23 attached to the drive motor of each drive axis of the robot drive unit 21. The position detector 23 is configured, for example, by an encoder. The output of the position detector 23 allows for the detection of the position and orientation of robot 1.

[0023] The control device 2 includes a teaching control panel 49, which serves as an operating panel for an operator to manually operate the robot device 3. The teaching control panel 49 includes an input unit 49a for inputting information about the robot device 3. The input unit 49a is composed of operating members such as a keyboard and a dial. The teaching control panel 49 includes a display unit 49b for displaying information related to the control of the robot device 3. The display unit 49b is composed of a display panel such as a liquid crystal display panel.

[0024] In this embodiment, the robot device 3 is configured with a robot coordinate system 71 that remains stationary even when the position and orientation of the robot 1 change. In the example shown in Figure 1, the origin of the robot coordinate system 71 is located on the base 14 of the robot 1. The robot coordinate system 71 is also called the world coordinate system. In the robot coordinate system 71, the position of the origin is fixed, and the orientation of the coordinate axes is fixed. In this embodiment, the robot coordinate system 71 is configured so that the Z axis is parallel to the vertical direction.

[0025] The robot device 3 has a tool coordinate system 73 set up with an origin set at an arbitrary position on the work tool. The position and orientation of the tool coordinate system 73 change together with the hand 5. In this embodiment, the origin of the tool coordinate system 73 is set at the tip of the tool. The position of the robot 1 corresponds to the position of the tip of the tool (the position of the origin of the tool coordinate system 73). Also, the orientation of the robot 1 corresponds to the orientation of the tool coordinate system 73 with respect to the robot coordinate system 71.

[0026] Furthermore, the robot device 3 has a sensor coordinate system 72 set for the visual sensor 30. The sensor coordinate system 72 is a coordinate system whose origin is fixed at an arbitrary position, such as the lens center point of the visual sensor 30. The position and orientation of the sensor coordinate system 72 change together with the visual sensor 30. In this embodiment, the sensor coordinate system 72 is set so that the Z axis is parallel to the optical axis of the camera included in the visual sensor 30.

[0027] The sensor coordinate system 72 has a predetermined relative position and orientation with respect to the flange coordinate system or tool coordinate system 73 set on the surface of the flange 16. The sensor coordinate system 72 is calibrated so that the coordinate values ​​of the robot coordinate system 71 can be calculated from the coordinate values ​​of the sensor coordinate system 72 based on the position and orientation of the robot 1.

[0028] Each coordinate system has defined X, Y, and Z axes. Additionally, it has defined a W axis around the X axis, a P axis around the Y axis, and an R axis around the Z axis.

[0029] Figure 3 shows a schematic diagram of the visual sensor in this embodiment. The visual sensor in this embodiment is a three-dimensional camera capable of acquiring three-dimensional positional information of the surface of an object. Referring to Figures 2 and 3, the visual sensor 30 in this embodiment is a stereo camera including a first camera 31 and a second camera 32. Each of the cameras 31 and 32 is a two-dimensional camera capable of capturing two-dimensional images. The visual sensor 30 in this embodiment includes a projector 33 that projects patterned light, such as stripes, onto the workpiece 65. The cameras 31, 32 and the projector 33 are arranged inside the housing 34.

[0030] Referring to Figure 2, the control device 2 of the robot apparatus 3 includes a vision sensor 30. The robot 1 changes the relative position between the workpiece 65 and the vision sensor 30. The control device 2 includes a processing unit 51 that processes the output of the vision sensor 30. The processing unit 51 includes a position information generation unit 52 that generates three-dimensional position information of the surface of the workpiece 65 based on the output of the vision sensor 30. The processing unit 51 includes a surface estimation unit 53 that estimates surface information relating to the surface including the surface of the workpiece based on the three-dimensional position information. Surface information is information that identifies the surface of the workpiece. For example, if the surface of the workpiece is planar, the surface information includes the equation of the surface in the robot coordinate system.

[0031] The processing unit 51 includes a correction amount setting unit 55 that sets a correction amount for driving the robot 1. The robot 1 changes the relative position between the workpiece 65 and the vision sensor 30 from a first relative position to a second relative position different from the first relative position. The processing unit 51 includes a determination unit 54 that determines whether the first surface, which includes the surface of the workpiece 65 detected at the first relative position, and the second surface, which includes the surface of the workpiece 65 detected at the second relative position, coincide within a predetermined determination range. The correction amount setting unit 55 sets a correction amount for driving the robot at the second relative position so that the first surface and the second surface coincide based on the surface information. For example, the correction amount setting unit 55 sets a correction amount for driving the robot at the second relative position so that the first surface and the second surface coincide within a predetermined range based on the surface information.

[0032] The processing unit 51 includes a synthesis unit 56 that synthesizes three-dimensional position information of multiple workpiece surfaces acquired at multiple relative positions. In this example, the synthesis unit 56 synthesizes three-dimensional position information detected at a first relative position with three-dimensional position information detected at a second relative position. In particular, the synthesis unit 56 uses three-dimensional position information generated at the second relative position that has been corrected based on a correction amount set by the correction amount setting unit 55.

[0033] The processing unit 51 includes an imaging control unit 57 that performs control related to imaging by the vision sensor 30. The processing unit 51 also includes a command unit 58 that sends commands for the movement of the robot 1. In this embodiment, the command unit 58 sends commands to the movement control unit 43 to correct the position and posture of the robot 1 based on the correction amount for the movement of the robot 1 set by the correction amount setting unit 55.

[0034] The processing unit 51 described above corresponds to a processor driven according to the operation program 41. The processor functions as the processing unit 51 by executing the controls defined in the operation program 41. In addition, the position information generation unit 52, surface estimation unit 53, determination unit 54, correction amount setting unit 55, and synthesis unit 56 included in the processing unit 51 correspond to processors driven according to the operation program 41. Furthermore, the imaging control unit 57 and command unit 58 correspond to processors driven according to the operation program 41. The processor functions as each unit by executing the controls defined in the operation program 41.

[0035] In this embodiment, the position information generation unit 52 calculates the distance from the visual sensor 30 to a 3D point set on the surface of the object based on the parallax between the image captured by the first camera 31 and the image captured by the second camera 32. The 3D point can be set, for example, for each pixel of the image sensor. The position information generation unit 52 calculates the distance from the visual sensor 30 for each 3D point. Furthermore, the position information generation unit 52 calculates the coordinate value of the position of the 3D point in the sensor coordinate system 72 based on the distance from the visual sensor 30.

[0036] Figure 4 shows a perspective view of the vision sensor and workpiece to illustrate an example of a 3D point cloud and depth image. In this example, the workpiece 65 is positioned at an angle on the surface 69a of the frame 69. The surface 69a of the frame 69 extends perpendicular to the optical axes of the cameras 31 and 32 of the vision sensor 30. By processing the images captured by the cameras 31 and 32 of the vision sensor 30, the distance from the vision sensor 30 to a 3D point set on the surface of the workpiece 63 can be detected, as shown by arrows 102 and 103.

[0037] Figure 5 shows a perspective view of the point cloud of 3D points generated by the position information generation unit. In Figure 5, the contours of the workpiece 65 and the measurement area 91 are shown by dashed lines. The 3D points 85 are placed on the surface of the object facing the visual sensor 30. The position information generation unit 52 sets 3D points 85 on the surfaces of objects contained within the measurement area 91. Here, a large number of 3D points 85 are placed on the surface 65a of the workpiece 65. In addition, a large number of 3D points 85 are placed on the surface 69a of the frame 69 inside the measurement area 91.

[0038] In this way, the position information generation unit 52 can represent the surface of the workpiece 65 as a three-dimensional point cloud. The position information generation unit 52 can generate three-dimensional position information of the surface of the object in the form of a distance image or three-dimensional point position information (three-dimensional map). A distance image represents the position information of the surface of the object using a two-dimensional image. In a distance image, the intensity or color of each pixel represents the distance from the visual sensor 30 to the three-dimensional point. On the other hand, a three-dimensional map represents the position information of the surface of the object using a set of coordinate values ​​(x, y, z) of three-dimensional points on the surface of the object. Such coordinate values ​​can be represented in the robot coordinate system 71 or the sensor coordinate system 72.

[0039] Figure 6 shows an example of a distance image obtained by the output of the visual sensor. The position information generation unit 52 can generate a distance image 81 in which the intensity of the color changes according to the distance from the visual sensor 30 to the 3D point 85. In this example, the distance image 81 is generated such that the color becomes darker as the distance from the visual sensor 30 increases. On the surface 65a of the workpiece 65, the color becomes lighter as it gets closer to the visual sensor 30. In this embodiment, the position of a 3D point is used as the 3D position information of the surface of the object to be explained, but similar control can be performed using a distance image.

[0040] In this embodiment, the position information generation unit 52 is located in the processing unit 51 of the arithmetic processing unit 24, but the embodiment is not limited to this configuration. The position information generation unit may be located inside the 3D sensor. That is, the 3D sensor may be equipped with an arithmetic processing unit including a processor such as a CPU, and the processor of the 3D sensor's arithmetic processing unit may function as the position information generation unit. In this case, the vision sensor outputs 3D position information such as a 3D map or a distance image.

[0041] Figure 7 shows a perspective view of the robot device and workpiece when the vision sensor is moved to the second position by the first robot device. Referring to Figures 1 and 7, the robot device 3 grasps the surface 65a of the workpiece 65 with the hand 5. The robot device 3 performs control to transport the workpiece 65 from the surface 69a of the stand 69 to a predetermined position. For example, the robot device 3 performs control to transport the workpiece 65 to a nearby conveyor or shelf.

[0042] In this embodiment, when the robot 1 is in a predetermined position and orientation, the workpiece 65 having a surface 65a with an area larger than the measurement area 91 of the vision sensor 30 is measured. That is, the workpiece 65 is too large to capture the entire surface 65a in a single image. The surface 65a is larger than the measurement area 91 and has a portion that extends beyond the measurement area 91. Alternatively, the length of one predetermined direction of the surface 65a is greater than the length of one predetermined direction of the measurement area 91. For this reason, in this embodiment, multiple images are taken by changing the position (viewpoint) of the vision sensor 30. The robot device 3 changes the relative position between the workpiece 65 and the vision sensor from a first relative position to a second relative position different from the first relative position. By taking images at each position, three-dimensional position information is generated for the entire surface 65a of the workpiece 65. Here, three-dimensional points are set for the entire surface 65a of the workpiece 65. Then, based on the three-dimensional position information, the position and orientation of the robot 1 when gripping the workpiece 65 with the hand 5 are calculated.

[0043] In Figure 1, the vision sensor 30 is positioned in a first position and orientation (first viewpoint). The position information generation unit 52 sets a 3D point on the surface 65a located inside the measurement area 91. The position information generation unit 52 sets a 3D point on one end of the surface 65a. Next, the robot 1 changes its position and orientation so that the vision sensor 30 moves as shown by arrow 101. Here, the vision sensor 30 is translated horizontally. In Figure 7, the vision sensor 30 is positioned in a second position and orientation (second viewpoint). The position information generation unit 52 sets a 3D point on the other end of the surface 65a.

[0044] The measurement area 91 of the visual sensor 30 at the first position shown in Figure 1 and the measurement area 91 of the visual sensor 30 at the second position shown in Figure 7 partially overlap. The synthesis unit 56 sets 3D points across the entire surface 65a by combining the 3D point cloud acquired at the first position and the 3D point cloud acquired at the second position. In this example, 3D points can be set across the entire surface 65a by imaging with the visual sensor 30 twice.

[0045] The command unit 58 can then calculate the position and orientation of the surface 65a of the workpiece 65 based on the three-dimensional point cloud set on the surface 65a. Based on the position and orientation of the workpiece 65, the command unit 58 can then calculate the position and orientation of the robot 1 for gripping the workpiece 65.

[0046] The first position and orientation of the vision sensor 30 for measuring the surface of the workpiece 65, and the second position and orientation of the vision sensor 30, can be set by arbitrary control. For example, the operator can display an image captured by one of the two-dimensional cameras of the vision sensor 30 on the display unit 49b of the teaching control panel 49. Then, by operating the input unit 49a while viewing the image displayed on the display unit 49b, the operator can adjust the position and orientation of the robot 1.

[0047] As shown in Figure 1, the operator can adjust the robot's position and orientation so that one side of the workpiece 65 is positioned inside the measurement area 91. Alternatively, as shown in Figure 7, the operator can manually adjust the robot's position and orientation so that the other side of the workpiece 65 is positioned inside the measurement area 91. The operator can store the robot's position and orientation in the memory unit 42 when the vision sensor 30 is positioned in the desired location and orientation. Alternatively, the position and orientation of the vision sensor may be pre-set using a simulation device or the like.

[0048] Figure 8 shows a schematic cross-sectional view of the vision sensor and workpiece when the robot is ideally driven. In this embodiment, the surface 69a of the frame 69 is planar and extends horizontally. Alternatively, the surface 65a of the workpiece 65 is planar and extends horizontally. The vision sensor 30 is moving from a first position P30a to a second position P30b, as shown by arrow 105. In this example, the vision sensor 30 is changing position without changing its orientation. The vision sensor 30 is moving horizontally parallel to the Y-axis of the robot coordinate system 71.

[0049] Figure 8 shows the case where there is no error in the actual position and orientation of robot 1 compared to the command values ​​of robot 1. The 3D points 85a and 85b indicate the positions of the coordinate values ​​in the sensor coordinate system 72. Furthermore, if there is no error in the actual position and orientation of the robot, the 3D points 85a and 85b will be located in the same positions even when the coordinate values ​​in the sensor coordinate system 72 are converted to the coordinate values ​​in the robot coordinate system 71.

[0050] Based on the output of the vision sensor 30 located at the first position P30a, a 3D point 85a is set on the surface 65a of the workpiece 65 and the surface 69a of the stand 69. Similarly, based on the output of the vision sensor 30 located at the second position P30b, a 3D point 85b is set on surfaces 65a and 69a. A portion of the measurement area 91a at the first position P30a and a portion of the measurement area 91b at the second position P30b overlap. The 3D points 85a and 85b are located in the overlapping area. However, because there are no errors in the robot's position and orientation, the 3D points 85a and 85b set on surface 65a are on the same plane. For this reason, the processing unit 51 can accurately estimate the position and orientation of the workpiece 65 by combining the point cloud of 3D point 85a and the point cloud of 3D point 85b.

[0051] Figure 9 shows a schematic cross-sectional view of the vision sensor and workpiece when errors occur in the robot's position and orientation when the vision sensor is moved to a second position. When a robot is driven, errors may occur in the actual position and orientation compared to the command values ​​defined in the motion program. For example, movement errors caused by the drive mechanism, such as backlash in the transmission, may cause the actual robot's position and orientation to deviate from the command values. In this case, the robot's movement error corresponds to an error in the position of a three-dimensional point.

[0052] In the example shown in Figure 9, similar to Figure 8, command values ​​are generated to move the visual sensor 30 horizontally. However, as shown by arrow 106, the visual sensor 30 moves from the first position P30a to the second position P30c, which is shifted upwards. The position information generation unit 52 detects 3D points 85a and 85c in the sensor coordinate system 72. The Z-axis coordinate value of 3D point 85a in the sensor coordinate system 72 is different from the Z-axis coordinate value of 3D point 85c in the sensor coordinate system 72.

[0053] Figure 10 shows the position of a 3D point, expressed in terms of coordinate values ​​in the robot coordinate system, when a 3D point is detected at a second position containing errors. The processing unit 51 converts the coordinate values ​​of the sensor coordinate system 72 to the coordinate values ​​of the robot coordinate system 71, assuming that the vision sensor 30 is located at the second position P30b. Using the coordinate values ​​of the robot coordinate system at the second position P30b, the coordinate values ​​of the sensor coordinate system 72 are converted to the coordinate values ​​of the robot coordinate system 71. For this purpose, the position of the 3D point 85c in the robot coordinate system 71 is calculated under the condition that the vision sensor 30 is located at the second position P30b.

[0054] The Z-axis coordinate value of the 3D point 85c in the sensor coordinate system 72 becomes larger, and the 3D point 85c is positioned offset from the surface 65a of the workpiece 65. In this example, the position of the 3D point 85c is calculated to be below the surface 65a.

[0055] For the 3D points 85a and 85c in the overlapping region of measurement area 91a and measurement area 91b, for example, 3D point 85a, which is close to the vision sensor 30, can be used. In this case, as shown in surface 99, it is determined that there is a step on the surface of the workpiece 65. Thus, there is a problem that if there is an error in the robot's drive, it is not possible to accurately detect the position of the 3D points on the entire surface 65a of the workpiece 65.

[0056] Therefore, in this embodiment, the processing unit 51 sets a correction amount for driving the robot 1 so that when the vision sensor 30 is placed at the second position, the vision sensor 30 is placed at the second position P30b corresponding to the command values ​​for the position and orientation of the robot 1.

[0057] Figure 11 shows a schematic diagram of the vision sensor and workpiece when the robot is driven after calculating the correction value. The processing unit 51 in this embodiment sets the correction amount shown by arrow 107 so that the vision sensor 30, which is positioned at the second position P30c, is positioned at the second position P30b. The correction amount can be a correction amount for the command values ​​of the robot's position and orientation. In particular, the correction amount setting unit 55 searches for a second position such that when the vision sensor 30 is positioned at the second position, the plane determined by the first 3D point 85a acquired at the first position and the plane determined by the second 3D point 85c acquired at the second position are on the same plane. That is, it performs plane alignment control to make the two planes coincide. Based on the corrected second position of the vision sensor 30, the correction amount for driving the robot is set.

[0058] Figure 12 shows a flowchart of the control of the first robot device in this embodiment. The control shown in Figure 12 includes surface alignment control to align a first surface and a second surface. The first surface is the surface that includes the surface of the workpiece 65 detected at a first relative position, and serves as the reference surface for surface alignment control. The second surface is the surface that includes the surface of the workpiece 65 detected at a second relative position. The control shown in Figure 12 can be performed during the teaching process before performing the actual work.

[0059] Referring to Figures 9 and 12, in step 111, the first position P30a and the second position P30c of the visual sensor 30 for imaging the workpiece are set. In this embodiment, the operator sets the first position P30a and the second position P30c by operating the teaching control panel 49. The storage unit 42 stores the command values ​​for the robot 1 at each position.

[0060] Here, the position of the vision sensor 30 is translated so that its orientation at the first position P30a is the same as its orientation at the second position P30b. For example, the vision sensor is moved in the negative direction of the Y-axis in the robot coordinate system 71. However, due to errors in the robot 1's drive mechanism, the vision sensor 30 is also moved in the Z-axis direction.

[0061] Next, in step 112, the command unit 58 drives the robot 1 to move the vision sensor 30 to the first position P30a. In this example, when the vision sensor 30 is positioned at the first position P30a, the robot 1 is driven without any error in the actual position and orientation of the robot 1 relative to the command value of the robot 1.

[0062] In step 113, the imaging control unit 57 sends a command to the vision sensor 30 to capture an image. The vision sensor 30 captures an image. The position information generation unit 52 generates first three-dimensional position information in the measurement area 91a based on the image from the first camera 31 and the image from the second camera 32. Here, first three-dimensional points 85a are set on the surface 65a of the workpiece 65 and the surface 69a of the stand 69. The position information generation unit 52 is calibrated to convert coordinate values ​​in the sensor coordinate system 72 to coordinate values ​​in the robot coordinate system 71. The position information generation unit 52 calculates the position of the three-dimensional point 85a in the sensor coordinate system 72. The position information generation unit 52 converts the coordinate values ​​in the sensor coordinate system 72 to coordinate values ​​in the robot coordinate system 71. The position of the first three-dimensional point 85a as first three-dimensional position information is calculated using the coordinate values ​​in the robot coordinate system 71.

[0063] In step 114, the surface estimation unit 53 calculates surface information for the first surface including the surface 65a of the workpiece 65. The surface estimation unit 53 calculates the equation of a plane including the 3D points 85a in the robot coordinate system 71 as surface information for the first surface. The surface estimation unit 53 excludes 3D points from the acquired 3D points 85a whose coordinate values ​​differ significantly from a predetermined judgment value. Here, 3D points 85a located on the surface 69a of the frame 69 are excluded. Alternatively, the range for estimating the plane in the image may be predetermined. For example, when an operator manually sets the first position and orientation of the vision sensor 30, they may specify the range for estimating the plane on the image while looking at the image captured by the 2D camera. The surface estimation unit 53 extracts the 3D points 85a within the range for plane estimation. Next, the surface estimation unit 53 calculates the equation of a plane in the robot coordinate system 71 along the point cloud of 3D points 85a. For example, the equation of the first plane in the robot coordinate system 71 is calculated using the least squares method to minimize the error in the coordinate values ​​of the three-dimensional points.

[0064] Next, in step 115, the command unit 58 moves the vision sensor 30 from the first position P30a to the second position P30c, as shown by arrow 106. The vision sensor 30 moves by driving the robot 1.

[0065] In step 116, the imaging control unit 57 sends a command to the vision sensor 30 to capture an image. The vision sensor 30 captures an image. The position information generation unit 52 sets a second three-dimensional point 85c corresponding to the surface 65a of the workpiece 65. The position information generation unit 52 calculates the position of the three-dimensional point 85c as the second three-dimensional position information. The position information generation unit 52 calculates the position of the second three-dimensional point 85c using the coordinate values ​​of the robot coordinate system 71.

[0066] Next, in step 117, the surface estimation unit 53 calculates surface information for a second surface including the surface 65a of the workpiece 65. The surface estimation unit 53 can exclude second three-dimensional points 85c located on the surface 69a of the frame 69. Alternatively, the range for estimating a plane in the image may be predetermined. For example, when an operator manually sets the second position and orientation of the vision sensor 30, they may specify the range for estimating a plane on the screen while viewing the image captured by the two-dimensional camera. The surface estimation unit 53 extracts three-dimensional points 85c within the range for estimating a plane. Next, the surface estimation unit 53 calculates surface information for the second surface based on the positions of the multiple second three-dimensional points 85c. The surface estimation unit 53 calculates the equation of the plane of the second surface containing the three-dimensional points 85c in the robot coordinate system 71 using the least squares method.

[0067] Next, in step 118, the determination unit 54 determines whether the first face and the second face coincide within a predetermined determination range. Specifically, the determination unit 54 calculates whether the difference between the position and orientation of the first face and the position and orientation of the second face is within the determination range. In this example, the determination unit 54 calculates a normal vector from the origin of the robot coordinate system 71 toward the first face based on the equation of the first face with respect to the first three-dimensional point 85a. Similarly, the determination unit 54 calculates a normal vector from the origin of the robot coordinate system 71 toward the second face based on the equation of the second face with respect to the second three-dimensional point 85c.

[0068] The determination unit 54 compares the length and direction of the normal vectors for the first and second faces. If the difference in the length of the normal vectors is within a predetermined determination range, and the difference in the direction of the normal vectors is also within a predetermined determination range, then the determination unit 54 can determine that the difference in position and orientation between the first and second faces is within the determination range. The determination unit 54 determines that the degree of agreement between the first and second faces is high. In step 118, if the difference between the position and orientation of the first face and the position and orientation of the second face deviates from the determination range, the control proceeds to step 119. Note that when the relative position between the vision sensor and the workpiece is changed, the relative orientation between the vision sensor and the workpiece may not change. For example, as shown in Figure 9, when the vision sensor is moved relative to the workpiece, it may be known in advance that the orientation of the vision sensor hardly changes. If there is no error in the relative orientation between the workpiece and the vision sensor, the evaluation of the relative orientation based on the first and second faces of the workpiece does not need to be performed in step 118. For example, it is not necessary to evaluate the direction of the normal vector.

[0069] In step 119, the command unit 58 sends a command to change the position and orientation of the robot 1. In this example, the command unit 58 changes the position and orientation of the robot 1 by a small amount. The command unit 58 can perform control to move the position and orientation of the robot 1 slightly in a predetermined direction. For example, it can perform control to move the vision sensor 30 slightly upward or downward in the vertical direction. Alternatively, the command unit 58 may perform control to drive the drive motors for each drive axis so that the component rotates in a predetermined direction at a predetermined angle. Note that if the relative orientation between the vision sensor and the workpiece does not change when the relative position between the vision sensor and the workpiece is changed, the robot's orientation does not need to be changed in step 119.

[0070] After changing the position and orientation of robot 1, the control returns to step 116. The processing unit 51 repeats the control from step 116 to step 118. In this way, the control in Figure 12 performs control to search for the position of the vision sensor 30 where the first and second faces coincide, while changing the position of the vision sensor 30. In step 118, if the difference in position and orientation between the first and second faces is within the determination range, the control proceeds to step 120. In this case, referring to Figure 11, it can be determined that the vision sensor 30 has moved from the second position P30c to the second position P30b.

[0071] Referring to Figure 12, in step 120, the correction amount setting unit 55 sets a correction amount for moving the vision sensor 30 from the second position P30c to the second position P30b. The arrow 107 shown in Figure 11 corresponds to the correction amount. The storage unit 42 stores the correction amount for driving the robot to position the vision sensor 30 at the second position. In this embodiment, the correction amount setting unit 55 sets the correction amount for the command values ​​of the position and posture of the robot 1. The correction amount is not limited to this form and may be determined by the rotation angle of the drive motor on each drive axis. Also, if the relative posture evaluation based on the first and second planes is not performed in step 118, the correction amount for the command value of the robot's posture does not need to be calculated.

[0072] In the above embodiment, a high degree of agreement between the first and second surfaces is determined when the difference between the position and orientation of the first surface and the position and orientation of the second surface is within the determination range, but the embodiment is not limited to this. The robot's position may be changed a predetermined number of times, and the robot's position and orientation with the highest degree of surface agreement may be adopted. The correction amount at the second position may be set based on the position and orientation of robot 1 at this time.

[0073] Alternatively, referring to Figure 9, the correction amount setting unit 55 may set the correction amount based on the coordinate value of the 3D point 85a in the sensor coordinate system 72 at the first position P30a and the coordinate value of the 3D point 85c in the sensor coordinate system 72 at the second position P30c. In this example, the equation of the first plane is calculated in the sensor coordinate system 72 based on the first 3D point 85a, and the equation of the second plane is calculated in the sensor coordinate system 72 based on the second 3D point 85c. Then, the correction amount may be calculated based on the difference in position and orientation between the first plane and the second plane. Here, the correction amount in the Z-axis direction of the sensor coordinate system 72 is calculated. Then, the correction amount in the sensor coordinate system 72 can be converted to the correction amount in the robot coordinate system 71.

[0074] In the examples shown in Figures 8 to 11, the second position is set to maintain a constant posture of the vision sensor 30. That is, the second position is set to move the vision sensor 30 in parallel without changing its posture, but the embodiment is not limited to this. The robot 1 in this embodiment is an articulated robot. The robot 1 can change the relative posture between the workpiece 65 and the vision sensor 30 from a first relative posture to a second relative posture. For this purpose, the position and posture of the vision sensor may be changed from a first position and posture to a second position and posture. The processing unit can control the posture of the vision sensor in the same way as the position control of the vision sensor. The correction amount setting unit can set the correction amount for the second position and the correction amount for the second posture of the vision sensor. That is, the correction amount setting unit may set the posture correction amount in addition to the position correction amount.

[0075] Figure 13 shows a control flowchart for when performing the actual work of transporting a workpiece. In the actual work, the visual sensor is moved to the second position using the correction amount set in the correction amount setting unit 55. In step 131, the operator or other device places the workpiece 65 at a predetermined position on the surface 69a of the stand 69. The workpiece is placed inside the measurement area obtained by adding the measurement area at the first position and the measurement area at the second position of the visual sensor 30.

[0076] In step 132, the motion control unit 43 drives the robot 1 to move the vision sensor 30 to a first position. In step 133, the imaging control unit 57 captures an image with the vision sensor 30. The position information generation unit 52 generates first three-dimensional position information.

[0077] Next, in step 134, the motion control unit 43 drives the robot 1 to move the vision sensor 30 to the corrected second position using the correction amounts for the position and orientation of the robot 1 set by the correction amount setting unit 55 during the teaching operation. The motion control unit 43 positions the vision sensor at a position that reflects the correction amount to the command value. That is, it drives the robot with command values ​​that have been corrected by the correction amount to the command values ​​(coordinate values) of position and orientation. Referring to Figure 11, as shown by arrow 107, the correction amount is applied and the vision sensor 30 is positioned at the second position P30b. Note that it may be known in advance that there is no error in the relative orientation between the workpiece and the vision sensor, and the correction amount setting unit 55 calculates the correction amount for the robot's position, but does not calculate the correction amount for the robot's orientation. In this case, the vision sensor 30 may be moved using only the correction amount for the robot's position.

[0078] Next, in step 135, the imaging control unit 57 captures an image with the vision sensor 30. The position information generation unit 52 acquires the image from the vision sensor 30 and generates second three-dimensional position information. Because the position of the robot 1 at the second position has been corrected, the three-dimensional points placed on the surface 65a of the workpiece 65 can be accurately calculated in the robot coordinate system 71. Here, when the robot command value for the second position is corrected, the position information generation unit 52 uses the robot command value before correction to convert the position (coordinate value) of the three-dimensional point expressed in the sensor coordinate system 72 to the position (coordinate value) of the three-dimensional point expressed in the robot coordinate system 71.

[0079] Next, in step 136, the synthesis unit 56 synthesizes the first 3D position information acquired at the first position and the second 3D position information acquired at the second position. The 3D position information uses the position of a 3D point. In this embodiment, for areas where the measurement area of ​​the visual sensor at the first position and the measurement area of ​​the visual sensor at the second position overlap, the position of the 3D point with the shorter distance from the visual sensor 30 is used. Alternatively, in the overlapping area, the average position of the 3D point acquired at the first position and the 3D point acquired at the second position may be calculated. Alternatively, both 3D points may be used.

[0080] Next, in step 137, the command unit 58 calculates the position and orientation of the workpiece 65. The command unit 58 eliminates 3D points from the acquired 3D points whose coordinate values ​​deviate from a predetermined range. That is, the command unit 58 eliminates 3D points 85a located on the surface 69a of the frame 69. The command unit 58 estimates the contour of the surface 65a of the workpiece 65 using the multiple 3D points. The command unit 58 calculates the gripping position on the surface 65a where the hand 5 is positioned approximately in the center of the surface 65a of the workpiece 65. Furthermore, the command unit 58 calculates the orientation of the workpiece at the gripping position.

[0081] In step 138, the command unit 58 calculates the position and orientation of the robot 1 so that the hand 5 is positioned to grasp the workpiece 65. In step 139, the command unit 58 sends the position and orientation of the robot 1 to the motion control unit 43. The motion control unit 43 drives the robot 1 to grasp the workpiece 65 with the hand 5. After this, the motion control unit 43 drives the robot 1 to transport the workpiece 65 to a predetermined position based on the motion program 41.

[0082] As described above, the control method for the robot device of this embodiment includes the steps of: positioning the robot at a first relative position relative to the workpiece and the vision sensor; and generating three-dimensional position information of the workpiece surface at the first relative position based on the output of the vision sensor. The control method also includes the steps of: positioning the robot at a second relative position different from the first relative position relative to the workpiece and the vision sensor; and generating three-dimensional position information of the workpiece at the second relative position based on the output of the vision sensor. Furthermore, the control method includes the step of a surface estimation unit estimating surface information relating to a surface including the workpiece surface based on the three-dimensional position information. The control method also includes the step of a correction amount setting unit setting a correction amount for driving the robot at the second relative position based on the surface information. The correction amount setting unit sets the correction amount so that the first surface including the workpiece surface detected at the first relative position and the second surface including the workpiece surface detected at the second relative position coincide.

[0083] In this embodiment, when multiple measurements are taken on a single workpiece using a visual sensor, the robot's drive correction amount is set to align the surfaces generated from the 3D position information of each measurement. To achieve this, the robot's correction amount can be set to minimize the error in the 3D position information obtained from the output of the 3D sensor. In actual operation, by correcting the robot's position and orientation with the set correction amount, 3D points can be accurately set on the workpiece surface even when multiple measurements are taken. The workpiece surface can be accurately detected, enabling the robot device to perform its work with high precision. For example, in this embodiment, the position and orientation of the workpiece can be accurately detected, preventing the robot device from failing to grip the workpiece or experiencing unstable gripping. Alternatively, even when halation occurs and 3D points need to be interpolated, 3D points can be accurately set.

[0084] Furthermore, the position and orientation of the robot when the vision sensor is placed at the first position may be pre-adjusted to precisely match the command values ​​for position and orientation in the robot coordinate system. In addition, in the above embodiment, the robot 1 moves the vision sensor 30 to two positions, thereby placing the workpiece 65 and the vision sensor 30 at two relative positions, but the embodiment is not limited to this. The robot may change the relative positions of the workpiece and the vision sensor to three or more different relative positions. For example, the vision sensor can be moved to three or more positions, and measurements can be performed using the vision sensor.

[0085] At this time, the position information generation unit can generate three-dimensional position information of the workpiece surface at each relative position. The surface estimation unit can estimate surface information at each relative position. Furthermore, the correction amount setting unit can set a correction amount for driving the robot at at least one relative position so that the surfaces including the workpiece surface detected at multiple relative positions coincide within a predetermined determination range.

[0086] For example, the correction amount setting unit may create a reference plane based on 3D position information acquired at one relative position, and then correct other relative positions so that the plane generated from 3D position information acquired at other relative positions coincides with the reference plane.

[0087] When using the flat first workpiece 65 described above, the first and second planes are calculated from the three-dimensional points set on the surface 65a. The position of the vision sensor is then corrected so that the first and second planes coincide. Alternatively, the orientation of the vision sensor may be corrected so that the first and second planes coincide. However, the relative position of the second plane to the first plane in the direction in which the first and second planes extend is not specified. Furthermore, the rotation angle around the normal direction of the first and second planes is not specified.

[0088] For example, referring to Figure 11, the position of the workpiece 65 in the Z-axis direction of the robot coordinate system 71 and the orientation of the workpiece 65 around the W-axis and P-axis can be corrected. However, errors in the position in the X-axis and Y-axis directions and the orientation around the R-axis of the robot coordinate system 71 remain. Therefore, a workpiece with a distinctive shape and characteristic parts formed on its surface may be used to correct the position and orientation of the vision sensor and the robot.

[0089] Figure 14 shows a perspective view of the second workpiece and the visual sensor of this embodiment. The second workpiece 66 is formed in a plate shape. The workpiece 66 has a circular hole 66b in its planar shape. The position and orientation of the surface 66a of the workpiece 66 are detected by placing the visual sensor 30 at the first position P30a and the second position P30c and taking measurements.

[0090] Figure 15 shows a block diagram of a modified surface estimation unit of the first robot device according to this embodiment. Referring to Figures 14 and 15, in the modified surface estimation unit 53, the surface estimation unit 53 has a feature detection unit 59. The feature detection unit 59 is configured to detect the position of a feature portion of a workpiece. For example, the feature detection unit 59 is configured to perform pattern matching using three-dimensional position information. Alternatively, the feature detection unit 59 is configured to perform pattern matching using a two-dimensional image.

[0091] When setting the correction amount for the robot 1's drive during the teaching process, the feature detection unit 59 detects the position of the hole 66b of the workpiece 66 in the measurement area 91a based on the first three-dimensional position information acquired at the first position P30a. The feature detection unit 59 also detects the position of the hole 66b of the workpiece 66 in the measurement area 91c based on the second three-dimensional position information acquired at the second position P30c. The surface estimation unit 53 estimates the surface information of the first surface and the surface information of the second surface.

[0092] The determination unit 54 compares the lengths and directions of the normal vectors of the first and second faces, as well as the position of the hole 66b. The robot's position and orientation at the second position can be changed until the difference between the position of the hole in the first 3D position information and the position of the hole in the second 3D position information falls within the determination range.

[0093] The correction amount setting unit 55 sets the correction amount so that the first surface and the second surface coincide within the judgment range. Furthermore, the correction amount setting unit 55 can set the correction amount so that the position of the hole 66b in the first 3D position information acquired at the first position and the position of the hole 66b in the second 3D position information acquired at the second position coincide within the judgment range. By driving the robot with this correction amount, it is possible to align 3D points in directions parallel to the direction in which the first and second surfaces extend. In addition to the W-axis, P-axis, and Z-axis directions of the robot coordinate system 71, the positions of 3D points in the X-axis and Y-axis directions can also be aligned. The correction amount of the robot's position and orientation can be set so that the position of the hole 66b of the second workpiece 66 matches.

[0094] Figure 16 shows a perspective view of the third workpiece and the vision sensor of this embodiment. The hole 66b, which is a feature portion of the second workpiece 66, has a circular planar shape. The hole 66b has a point-symmetric planar shape. In contrast, the third workpiece 67 has a feature portion with an asymmetric planar shape. The third workpiece 67 is formed in a flat plate shape. The third workpiece 67 has a hole 67b with a triangular planar shape. The feature detection unit 59 can detect the position of the hole 67b in the measurement area.

[0095] The determination unit 54 compares the position of the hole 67b in the first 3D position information with the position of the hole 67b in the second 3D position information. It can then change the position and orientation of the robot at the second position until the difference in the position of the hole 67b falls within the determination range. The correction amount setting unit 55 can set a correction amount so that the position of the hole 67b in the 3D position information acquired at the first position matches the position of the hole 67b in the 3D position information acquired at the second position.

[0096] The third workpiece 67 has a characteristic portion with an asymmetrical planar shape. For this reason, alignment can be performed around the normal directions of the first and second faces. Referring to Figure 16, the positions of three-dimensional points in the directions of the X, Y, and R axes, in addition to the W, P, and Z axes of the robot coordinate system 71, can be aligned. The position and orientation correction amounts of the robot 1 can be set so that the position and orientation of the hole 67b of the third workpiece 67 match.

[0097] The third workpiece illustrates an example where the planar shape of the feature portion is neither point-symmetric nor line-symmetric. Asymmetric feature portions may be formed at multiple asymmetrical locations on the workpiece. For example, feature portions such as protrusions may be formed at the points corresponding to the vertices of the triangles in the holes of the third workpiece.

[0098] The above-described embodiment explains the case where the workpiece surface is flat, but it is not limited to this form. The control described in this embodiment can also be applied when the workpiece surface is curved.

[0099] Figure 17 shows a schematic diagram of the fourth workpiece and the vision sensor in this embodiment. The surface 68a of the fourth workpiece 68 is formed in a curved shape. A first three-dimensional point 85a is set by the output of the vision sensor 30 located at the first position P30a, and a second three-dimensional point 85c is set by the output of the vision sensor 30 located at the second position P30c.

[0100] Even in the case of such curved surfaces, the correction amount setting unit 55 can set a correction amount for driving the robot 1 at the second position P30c so that the first surface including the surface of the workpiece 68 detected at the first position P30a and the second surface including the surface of the workpiece 68 detected at the second position P30c coincide within a predetermined determination range, by controlling surface alignment in the teaching process as described above. In the actual operation of the robot device, the position and orientation of the robot can be corrected based on the correction amount to detect three-dimensional position information.

[0101] Alternatively, if the surface of the workpiece is curved, a reference plane can be pre-set in three-dimensional space to serve as the reference for the surface 68a of the workpiece 68. The shape of the surface 68a can be generated, for example, based on three-dimensional shape data output from a CAD (Computer Aided Design) device. To determine the position of the surface 68a of the workpiece 68 in the robot coordinate system 71, first, the workpiece 68 is placed on the stand. Next, a touch-up pen is attached to the robot 1, and the touch-up pen is brought into contact with multiple contact points set at various positions on the surface 68a of the workpiece 68. The positions of the multiple contact points are detected in the robot coordinate system 71. Based on the positions of the multiple contact points, the position of the workpiece 68 in the robot coordinate system 71 can be determined, and a reference plane in the robot coordinate system 71 can be generated. The storage unit stores the generated reference plane of the workpiece 68.

[0102] The processing unit can adjust the first and second positions of the vision sensor 30 to match the shape and position of the reference surface of the workpiece 68. The correction amount setting unit 55 can calculate the position and orientation of the robot so that the first surface matches the reference surface. The correction amount setting unit 55 can also calculate the position and orientation of the robot 1 so that the second surface matches the reference surface. The correction amount setting unit 55 can then calculate the correction amount for driving the robot 1 at each position.

[0103] When a reference surface corresponding to the workpiece surface is generated in advance using the output of a CAD device, it is preferable that the measurement area at the first position and the measurement area at the second position generally overlap. Therefore, this is suitable for control that interpolates the loss of 3D points due to halation.

[0104] In the above embodiment, the position and orientation of the vision sensor are changed by the robot, while the position and orientation of the workpiece remain unchanged. However, the embodiment is not limited to this configuration. The robot device can employ any configuration that changes the relative position between the workpiece and the vision sensor.

[0105] Figure 18 shows a side view of the second robot device in this embodiment. In the second robot device 7, the position and orientation of the vision sensor 30 are fixed, and the robot 4 changes the position and orientation of the workpiece 64. The second robot device 7 comprises the robot 4 and a hand 6 attached to the robot 4 as a working tool. The robot 4 is a 6-axis vertical articulated robot, similar to the robot 1 of the first robot device 3. The hand 6 has two fingers that face each other. The hand 6 is formed to grip the workpiece 64 by pinching it with its fingers.

[0106] The second robot device 7, like the first robot device 3, includes a control device 2 for controlling the robot 4 and the hand 6. The second robot device 7 also includes a vision sensor 30 as a three-dimensional sensor. The position and orientation of the vision sensor 30 are fixed by a frame 35 which serves as a fixing member.

[0107] In the second robot device 7 of this embodiment, the surface of the workpiece 64a is inspected. For example, the processing unit can perform inspections of the shape of the contour of the surface of the workpiece 64 and the shape of characteristic parts formed on the surface of the workpiece 64, based on the synthesized three-dimensional position information of the workpiece 64. The processing unit can determine whether each variable is within a predetermined judgment range.

[0108] In the second robot device 7, three-dimensional position information of the surface 64a of the workpiece 64 is generated based on the output of the vision sensor 30. The area of ​​the surface 64a of the workpiece 64 is larger than the measurement area 91 of the vision sensor 30. For this reason, the robot device 7 places the workpiece 64 at the first position P70a and generates the first three-dimensional position information. The robot device 7 also places the workpiece 64 at the second position P70c and generates the second three-dimensional position information. In this way, the robot 4 changes the relative position between the workpiece 64 and the vision sensor 30 from the first relative position to the second relative position by moving the workpiece 64 from the first position P70a to the second position P70c. In this example, the robot 4 moves the workpiece 64 horizontally as shown by arrow 108. At the second position P70c, the second position P70c may deviate from the desired position due to errors in the robot's drive mechanism.

[0109] The position information generation unit 52 of the second robot device generates first three-dimensional position information based on the output of the vision sensor 30 that has captured an image of the surface 64a of the workpiece 64 located at the first position P70a. The position information generation unit 52 also generates second three-dimensional position information based on the output of the vision sensor 30 that has captured an image of the surface 64a of the workpiece 64 located at the second position P70c.

[0110] The surface estimation unit 53 generates surface information for the first and second surfaces, including surface 64a, based on the respective three-dimensional position information. The correction amount setting unit 55 can set a correction amount for driving the robot 4 at the second position P70c so that the first surface estimated from the first three-dimensional position information and the second surface estimated from the second three-dimensional position information coincide within a predetermined determination range. In actual inspection work, the position and orientation of the robot at the second position can be corrected based on the correction amount set by the correction amount setting unit 55.

[0111] In the second robot device 7, errors in three-dimensional position information caused by driving errors of the robot 4 can also be suppressed. By driving the robot 4 based on a correction amount for driving the robot 4 at the second position, the robot device 7 can perform accurate inspections.

[0112] The other configurations, operations, and effects of the second robotic device are the same as those of the first robotic device, so they will not be explained again here.

[0113] The 3D sensor in this embodiment is a visual sensor including two 2D cameras, but is not limited to this configuration. The 3D sensor can be any sensor capable of generating 3D position information of the workpiece surface. For example, a Time of Flight (TOF) camera that acquires 3D position information based on the time of flight of light can be used as the 3D sensor. Also, the stereo camera as a visual sensor in this embodiment is equipped with a projector, but is not limited to this configuration. The stereo camera does not need to be equipped with a projector.

[0114] In this embodiment, the control device for controlling the robot functions as a processing unit for processing the output of the 3D sensor, but the system is not limited to this configuration. The processing unit may be composed of a different processing unit (computer) than the control device for controlling the robot. For example, a tablet terminal that functions as a processing unit may be connected to the control device for controlling the robot.

[0115] The robot device in this embodiment performs tasks such as transporting or inspecting workpieces, but is not limited to this configuration. The robot device can perform any task. Furthermore, the robot in this embodiment is a vertical articulated robot, but is not limited to this configuration. Any robot capable of moving workpieces can be used. For example, a horizontal articulated robot can be used.

[0116] The above embodiments can be combined as appropriate. In each of the above controls, the order of the steps can be changed as appropriate, as long as the function and operation are not altered.

[0117] In each of the figures described above, identical or equivalent parts are denoted by the same reference numerals. The embodiments described above are illustrative and do not limit the invention. Furthermore, the embodiments include modifications of the embodiments shown in the claims. [Explanation of symbols]

[0118] 1,4 Robots 2 Control device 3.7 Robot equipment 5, 6 hands 24 Arithmetic Processing Unit 30 Vision Sensors P30a,P30b,P30c position 51 Processing Unit 52 Location information generation section 53 plane estimation part 55 Correction amount setting section 56 Synthesis part 64, 65, 66, 67, 68 Work 64a,65a,66a,67a,68a surface 66b,67b Hole P70a,P70c position 81 Distance image 85,85a,85b,85c 3D point

Claims

1. A three-dimensional sensor for detecting the position on the surface of a workpiece, A robot that changes the relative position between the workpiece and the three-dimensional sensor, A position information generation unit generates three-dimensional position information of the surface of the workpiece based on the output of the three-dimensional sensor, A surface estimation unit that estimates surface information relating to a surface including the surface of the workpiece based on three-dimensional position information, A determination unit that compares the length and direction of normal vectors based on surface information for multiple surfaces, The system includes a correction amount setting unit for setting a correction amount for driving the robot, The robot is configured to change the relative position between the workpiece and the three-dimensional sensor from a first relative position to a second relative position different from the first relative position. The surface estimation unit estimates surface information of a first surface including the surface of the workpiece detected at a first relative position, and surface information of a second surface including the surface of the workpiece detected at a second relative position. The determination unit determines whether the difference in length and direction between the normal vector from a predetermined point to the first surface and the normal vector from the predetermined point to the second surface falls within a predetermined determination range. The correction amount setting unit sets a correction amount for driving the robot at a second relative position when the difference in length and the difference in orientation are within a predetermined determination range.

2. The robot apparatus according to claim 1, wherein the robot is configured to change the relative orientation between the workpiece and the three-dimensional sensor from a first relative orientation to a second relative orientation.

3. The aforementioned three-dimensional sensor is attached to the robot. The workpiece is positioned so that its position and orientation are immovable. The robot apparatus according to claim 1 or 2, wherein the robot changes the relative position between the workpiece and the three-dimensional sensor from a first relative position to a second relative position by moving the three-dimensional sensor from a first position to a second position.

4. The robot is equipped with a work tool for gripping the workpiece, The position and orientation of the three-dimensional sensor are fixed by a fixing member. The robot apparatus according to claim 1 or 2, wherein the robot changes the relative position between the workpiece and the three-dimensional sensor from a first relative position to a second relative position by moving the workpiece from a first position to a second position.

5. The robot changes the relative position between the workpiece and the three-dimensional sensor to three or more mutually different relative positions. The position information generation unit generates three-dimensional position information of the surface of the workpiece at each relative position, The surface estimation unit estimates the surface information at each relative position, The robot apparatus according to any one of claims 1 to 4, wherein the correction amount setting unit sets a correction amount for driving the robot at at least one relative position such that the surfaces including the surface of the workpiece detected at a plurality of relative positions coincide within a predetermined determination range.

6. The system includes a synthesis unit that synthesizes multiple three-dimensional positional information of the workpiece surface acquired at multiple relative positions, The robot apparatus according to claim 1, wherein the synthesis unit synthesizes three-dimensional position information generated at a first relative position with three-dimensional position information generated at a second relative position corrected based on a correction amount set by the correction amount setting unit.

7. The process involves the robot positioning the workpiece and the 3D sensor relative to a first relative position, The position information generation unit generates three-dimensional position information of the surface of the workpiece at a first relative position based on the output of the three-dimensional sensor, The robot performs the step of positioning the workpiece and the three-dimensional sensor at a second relative position different from the first relative position, The position information generation unit performs the step of generating three-dimensional position information of the surface of the workpiece at a second relative position based on the output of the three-dimensional sensor, The surface estimation unit estimates, based on three-dimensional positional information at each relative position, surface information of a first surface including the surface of the workpiece detected at a first relative position, and surface information of a second surface including the surface of the workpiece detected at a second relative position. The determination unit determines, based on the surface information of the first surface and the surface information of the second surface, whether the difference in length and direction between the normal vector from a predetermined point to the first surface and the normal vector from the predetermined point to the second surface is within a predetermined determination range. A method for controlling a robot device, comprising the step of setting a correction amount for driving the robot at a second relative position when the correction amount setting unit is within a predetermined determination range of the difference in length and the difference in orientation.

8. A control device for controlling a robot, It includes a processing unit that estimates surface information of a surface including the surface of a workpiece based on the output of a 3D sensor, The processing unit estimates surface information of a first surface including the surface of the workpiece detected at a first relative position representing the relative position between the workpiece and the three-dimensional sensor, and surface information of a second surface including the surface of the workpiece detected at a second relative position different from the first relative position. Determine whether the difference in length and direction between the normal vector from a predetermined point to the first surface and the normal vector from the same predetermined point to the second surface falls within the specified range. A control device that sets a correction amount for driving the robot at a second relative position when the difference in length and the difference in orientation are within a determination range.

9. A robot control method, The processing unit of the robot control device estimates, based on the output of the 3D sensor, surface information of a first surface including the surface of the workpiece detected at a first relative position representing the relative position between the workpiece and the 3D sensor, and surface information of a second surface including the surface of the workpiece detected at a second relative position different from the first relative position. The processing unit performs the step of determining whether the difference in length and direction between the normal vector from a predetermined point to the first surface and the normal vector from the predetermined point to the second surface is within a determination range. A control method comprising the step of setting a correction amount for driving the robot at a second relative position when the processing unit is within a determination range for the difference in length and the difference in orientation.

Images