Parameter correction device

Abstract

The present invention provides a parameter correction device that calculates a three-dimensional estimated position of a workpiece on the basis of a sensor parameter and the position of a characteristic portion in an image of the workpiece. The parameter correction device calculates a measured position of one member out of the sensor and the workpiece on the basis of the drive state of a movement device. The parameter correction device uses the movement device to move one member to first and second positions to capture images of the workpiece. The parameter correction device corrects the sensor parameter on the basis of the amount of movement, which is based on first and second measured positions, and the distance from a first estimated position to a second estimated position.

Description

Parameter Correction Device

[0001] The present disclosure relates to a parameter modification device.

[0002] Conventionally, robotic devices have been known that measure the position of a workpiece using an image captured by a camera and perform tasks based on the measured position of the workpiece. A control device of the robotic device can acquire an image of a characteristic part of the workpiece captured by the camera. The control device calculates the three-dimensional position of the workpiece from the image of the characteristic part of the workpiece based on a reference image of the characteristic part of the workpiece created in advance and camera parameters. The control device then corrects the operating position described in the robot's operating program based on the three-dimensional position of the workpiece and drives the robot. The robotic device can perform accurate tasks corresponding to the actual position of the workpiece.

[0003] International Publication WO2018 / 163450A1 JP2023-69274A International Publication WO2021 / 145280A1

[0004] In a robotic device that corrects the robot's operating position using an image captured by a camera, it is necessary to determine camera parameters in advance. High-precision camera parameters also lead to accurate calculations of the three-dimensional position of the workpiece by the control device. The multiple parameters included in the camera parameters can be determined through a process known as calibration. In calibration, for example, the control device uses a camera to capture an image of a calibration plate on which dot positions are predetermined. The control device can then generate camera parameters based on the positions of the dots in the image and the actual three-dimensional positions of the dots.

[0005] However, after using a robotic device, camera parameters may need to be corrected to correctly correct the robot's operation. In this case, a calibration plate must be prepared again, which poses a problem of time-consuming correction of errors in the camera parameters. Furthermore, the height of the calibration plate may differ from the height of the characteristic part of the workpiece to be actually detected. In the case of a camera capturing two-dimensional images, this may result in an error in the three-dimensional position of the workpiece estimated from the camera image. Alternatively, if the outline or pattern of the workpiece is incorrectly set in the reference image, an error in the three-dimensional position of the workpiece may occur. Such errors are difficult to detect, and furthermore, parameter correction is difficult.

[0006] One aspect of the present disclosure is a parameter modification device that modifies sensor parameters for calculating a three-dimensional position corresponding to a specific position in an image of a workpiece captured by a sensor. The parameter modification device includes a feature detection unit that detects a feature in the image of the workpiece. The parameter modification device includes an estimated position calculation unit that calculates an estimated position, which is the three-dimensional position of the workpiece, based on the position of the feature in the image of the workpiece and the sensor parameters. The parameter modification device includes an actual measured position calculation unit that calculates an actual measured position, which is the three-dimensional position of one of the members, based on a drive state of a movement device that moves at least one of the sensor and the workpiece. The parameter modification device includes a modification unit that modifies the sensor parameters. The feature detection unit detects a feature in an image captured when the one member is positioned at a first position and a feature in an image captured when the one member is positioned at a second position. The estimated position calculation unit calculates a first estimated position when the one member is positioned at the first position and a second estimated position when the one member is positioned at the second position. The actual position calculation unit calculates a first actual position when one of the members is placed at a first position and a second actual position when one of the members is placed at a second position. The correction unit corrects the sensor parameters based on the amount of movement based on the first actual position and the second actual position and the distance from the first estimated position to the second estimated position.

[0007] 1 is a schematic diagram of a first robot device according to an embodiment. FIG. 2 is a block diagram of a robot device according to an embodiment. FIG. 3 is a perspective view of a workpiece according to an embodiment. FIG. 4 is a schematic diagram illustrating the distance from a camera to a characteristic portion of a workpiece. FIG. 5 is a control flowchart for evaluating and correcting camera parameters. FIG. 6 is an image captured by placing a camera at a first position. FIG. 7 is a schematic diagram of the first robot device when the camera is placed at a second position. FIG. 8 is an image captured by placing a camera at a second position. FIG. 9 is a graph showing the relationship between the movement amount of an actual measured position in the X direction and an error. FIG. 10 is a graph showing the relationship between the movement amount of an actual measured position in the Y direction and an error. FIG. 11 is a graph showing the error in the X direction and the error in the Y direction with respect to the movement amount of an actual measured position in the X direction when there is an error in the distance from the camera to the workpiece. FIG. 12 is a graph showing the error in the Y direction and the error in the X direction with respect to the movement amount of an actual measured position in the Y direction when there is an error in the distance from the camera to the workpiece. FIG. 13 is an error in the X direction and the error in the Y direction with respect to the movement amount of an actual measured position in the X direction when there is an error in the distance from the camera to the workpiece. 10 shows the Y-direction error and the X-direction error with respect to the amount of movement of the actual measured position in the Y direction when the setting of the camera coordinate system is inaccurate. FIG. 10 shows an image in which an estimated image estimated without calculating image distortion is superimposed on an actual workpiece image. FIG. 10 shows an image in which an estimated image estimated by calculating image distortion is superimposed on an actual workpiece image. FIG. 10 shows an image displaying an overlapping portion between an actually captured image and an estimated image estimated by an image estimation unit. FIG. 10 is a schematic diagram of a second robot device in an embodiment.

[0008] A parameter modification device according to an embodiment will be described with reference to FIGS. 1 to 18. The parameter modification device according to the present embodiment modifies sensor parameters for calculating a three-dimensional position corresponding to a specific position in an image captured by a sensor. In this embodiment, a control device of a robot device functions as the parameter modification device. In this embodiment, a camera that captures two-dimensional images is exemplified as the sensor. As the sensor parameters, camera parameters including internal parameters and external parameters are exemplified.

[0009] (Configuration of First Robot Device) Fig. 1 is a schematic diagram of a first robot device equipped with a parameter correction device according to this embodiment. Fig. 2 is a block diagram of the robot device according to this embodiment. The robot device 3 according to this embodiment detects the position of a workpiece 31 as an object and transports the workpiece.

[0010] 1 and 2, the first robot device 3 includes a hand 5 as a work tool for gripping a workpiece 31, and a robot 1 for moving the hand 5. The robot device 3 includes a control device 2 as a robot control device for controlling the robot device 3. The robot device 3 also includes a stand 35 on which the workpiece 31 is placed.

[0011] The hand 5 in this embodiment is a work tool that grips and releases the workpiece 31. The work tool attached to the robot 1 is not limited to this form, and any work tool can be used depending on the work to be performed by the robot device 3. For example, a welding gun for welding can be used as the work tool.

[0012] The robot 1 of this embodiment functions as a moving device that moves one of the camera 6 as a sensor and the workpiece 31. In the first robot device 3, the camera 6 corresponds to one of the members that is moved by the moving device, and the workpiece 31 corresponds to the other member whose position is fixed.

[0013] The robot 1 of this embodiment is an articulated robot including multiple joints 18. The robot 1 includes an upper arm 11 and a lower arm 12. The lower arm 12 is supported by a swivel base 13. The swivel base 13 is supported by a base 14. The robot 1 includes a wrist 15 connected to the end of the upper arm 11. The wrist 15 includes a flange 16 to which the hand 5 is fixed. The components of the robot 1 of this embodiment are configured to rotate around six predetermined drive axes. The robot 1 is not limited to this configuration, and any robot that can move a work tool can be used.

[0014] The robot 1 of this embodiment includes a robot driving device 21 having a driving motor that drives components such as the upper arm 11. The hand 5 includes a hand driving device 22 that drives the hand 5. The hand driving device 22 of this embodiment drives the hand 5 by air pressure. The hand driving device 22 includes an air pump or the like for supplying compressed air to a cylinder.

[0015] The control device 2 includes a control device main body 40 and a teaching pendant 26 that allows an operator to operate the control device main body 40. The control device main body 40 includes an arithmetic processing device (computer) having a CPU (Central Processing Unit) as a processor. The arithmetic processing device has a RAM (Random Access Memory) and a ROM (Read Only Memory), etc., connected to the CPU via a bus. The robot device 3 of this embodiment automatically transports a workpiece 31 based on an operation program 67. The robot driving device 21 and the hand driving device 22 are controlled by the control device 2.

[0016] The control device main body 40 includes a storage unit 42 that stores any information related to the robot device 3. The storage unit 42 can be configured with a non-transitory storage medium capable of storing information. For example, the storage unit 42 can be configured with a storage medium such as a volatile memory, a non-volatile memory, a magnetic storage medium, or an optical storage medium.

[0017] An operation program 67 created in advance for operating the robot 1 is input to the control device 2. Alternatively, an operator can set teaching points for the robot 1 by operating the teaching pendant 26 to drive the robot 1. The control device 2 can generate the operation program 67 based on the teaching points.

[0018] The operation program 67 is stored in the memory unit 42. The operation control unit 43 sends operation commands to the robot driving unit 44 for driving the robot 1 based on the operation program 67. The robot driving unit 44 includes an electrical circuit that drives the drive motor, and supplies electricity to the robot driving device 21 based on the operation commands. The operation control unit 43 also sends operation commands to the hand driving unit 45 for driving the hand driving device 22. The hand driving unit 45 includes an electrical circuit that drives an air pump or the like, and supplies electricity to the air pump or the like based on the operation commands.

[0019] The operation control unit 43 corresponds to a processor that operates in accordance with the operation program 67. The processor is configured to be able to read information stored in the storage unit 42. The processor reads the operation program 67 and performs the control defined in the operation program 67, thereby functioning as the operation control unit 43.

[0020] The robot 1 includes a state detector that detects the driving state of the robot 1. The state detector of this embodiment detects the position and posture of the robot as the driving state of the robot 1. The state detector of this embodiment includes a position detector 23 attached to the drive motor of each drive shaft of the robot drive device 21. The position detector 23 can be configured, for example, by an encoder that detects the rotational position of the output shaft of the drive motor. The position and posture of the robot 1 are detected based on the output of the position detector 23.

[0021] The teaching pendant 26 is connected to the control device main body 40 via a communication device. The teaching pendant 26 includes an input unit 27 for inputting information about the robot 1 and the hand 5. The input unit 27 is composed of input members such as a keyboard and a dial. The teaching pendant 26 includes a display unit 28 for displaying information about the robot 1 and the hand 5. The display unit 28 can be composed of any display panel such as a liquid crystal display panel or an organic EL (Electro Luminescence) display panel. When the teaching pendant is equipped with a touch panel type display panel, the display panel functions as both the input unit and the display unit.

[0022] A reference coordinate system 71 that remains stationary even when the position and posture of the robot 1 change is set in the robot device 3. In the example shown in FIG. 1 , the origin of the reference coordinate system 71 is located on the swivel base 13 of the robot 1. The reference coordinate system 71 is also referred to as a world coordinate system. In the reference coordinate system 71, the position of the origin is fixed, and the orientations of the coordinate axes are also fixed. The reference coordinate system 71 has X-axis, Y-axis, and Z-axis as its coordinate axes that are orthogonal to each other.

[0023] A tool coordinate system 74 is set in the robot device 3, with its origin set at an arbitrary position on the work tool. The position and orientation of the tool coordinate system 74 change along with the work tool. In this embodiment, the origin of the tool coordinate system 74 is set at the tool tip point of the hand 5. The position of the robot 1 corresponds to the position of the tool tip point in the reference coordinate system 71 (the position of the origin of the tool coordinate system). The orientation of the robot 1 corresponds to the orientation of the tool coordinate system 74 with respect to the reference coordinate system 71.

[0024] The robot device 3 in this embodiment includes a camera 6 as a sensor for capturing an image of the workpiece 31. The camera 6 in this embodiment is a two-dimensional camera that captures a two-dimensional image. The camera 6 can capture an image in a field of view 6 a.

[0025] In the first robot device 3, one of the camera 6 and the workpiece 31 that is moved by the robot 1 is the camera 6. The other of the camera 6 and the workpiece 31 is the workpiece 31. In this manner, the position and orientation of the camera 6 are changed by changing the position and orientation of the robot 1. The three-dimensional position of the camera 6 is changed by the robot 1. In this embodiment, the camera 6 is fixed to the hand 5 attached to the robot 1, but this is not limited to this. The camera 6 can be fixed to any member whose position and orientation change as the robot 1 operates. Alternatively, the camera 6 may be attached to the robot 1. For example, the camera 6 may be fixed to a flange 16 of the robot 1.

[0026] In the robot device 3, a camera coordinate system 72 is set as a sensor coordinate system relative to the camera 6. The position and orientation of the camera coordinate system 72 change together with the camera 6. The origin of the camera coordinate system 72 is set at a predetermined position on the camera 6, such as the lens center or optical center of the camera 6. The camera coordinate system 72 has an X-axis, a Y-axis, and a Z-axis that are orthogonal to one another. In the present embodiment, the camera coordinate system 72 is set so that the Z-axis is parallel to the optical axis of the lens of the camera 6.

[0027] When the position and orientation of the robot 1 change, the position and orientation of the camera coordinate system 72 also change. The position of the camera 6 corresponds to the position of the origin of the camera coordinate system 72 in the reference coordinate system 71. The orientation of the camera 6 corresponds to the orientation of the camera coordinate system 72 relative to the reference coordinate system 71. In this embodiment, the position and orientation of the camera coordinate system 72 relative to the tool coordinate system 74 are determined in advance.

[0028] The control device 2 detects the three-dimensional position of the workpiece 31 in the work space based on an image of the workpiece 31 captured during work. The control device 2 corrects the position and posture of the robot 1 based on the position of the workpiece 31.

[0029] The control device 2 functions as a parameter correction device that corrects sensor parameters. The control device 2 includes an image processing unit 51 that processes images captured by the camera 6. The image processing unit 51 includes an imaging control unit 53 that sends a command to the camera 6 to capture an image.

[0030] The image processing unit 51 includes a feature detection unit 56 that detects predetermined feature portions in an image of the workpiece 31. The image processing unit 51 includes an estimated position calculation unit 57 that calculates an estimated position, which is the three-dimensional position of the workpiece, in a camera coordinate system 72 based on the positions of the feature portions in the image and camera parameters as sensor parameters. The image processing unit 51 includes an operation command generation unit 64 that generates operation commands for the robot 1 and the hand 5 based on the results of image processing.

[0031] The image processing unit 51 includes an actual position calculation unit 52 that calculates the actual position, which is the three-dimensional position of one of the camera 6 and the workpiece 31, in a reference coordinate system 71 based on the position and posture of the robot 1 as the driving state of the robot. The image processing unit 51 includes an evaluation unit 58 that evaluates camera parameters. The image processing unit 51 is equipped with a selection unit 59 that selects parameters to be corrected from multiple parameters included in the camera parameters. The image processing unit 51 includes a correction unit 60 that corrects the camera parameters.

[0032] The image processing unit 51 includes an image estimation unit 61 that estimates an image of the characteristic part when the camera 6 is disposed at a predetermined position. The image estimation unit 61 of this embodiment estimates an image of the characteristic part when one of the members is disposed at a second position based on an image of the characteristic part captured when the one of the members is disposed at a first position, camera parameters 68, and the amount of movement of the actual position of the one of the members. The image processing unit 51 includes a score calculation unit 62 that calculates a score indicating the degree to which the image of the characteristic part estimated by the image estimation unit 61 overlaps with the image of the characteristic part captured when the one of the members is disposed at the second position. The image processing unit 51 includes a display control unit 63 that controls the image to be displayed on the display unit 28.

[0033] The image processing unit 51 and the units included in the image processing unit 51 correspond to processors that operate in accordance with the operation program 67. The actual position calculation unit 52, the imaging control unit 53, the feature detection unit 56, the estimated position calculation unit 57, and the operation command generation unit 64 correspond to processors that operate in accordance with the operation program 67. The evaluation unit 58, the selection unit 59, the correction unit 60, the image estimation unit 61, the score calculation unit 62, and the display control unit 63 correspond to processors that operate in accordance with the operation program 67. The processors read the operation program 67 and perform the control defined in the operation program 67, thereby functioning as the respective units.

[0034] FIG. 3 shows a perspective view of a workpiece according to this embodiment. Referring to FIGS. 1 and 3, the workpiece 31 according to this embodiment has a plate-shaped portion 31a and a plate-shaped portion 31b formed above the plate-shaped portion 31a. Each of the plate-shaped portions 31a and 31b has a circular disk shape. The workpiece 31 is placed on the surface of the stand 35 so that the upper surface of the upper plate-shaped portion 31b is parallel to the horizontal direction. The plate-shaped portion 31b has an edge portion 31c on its upper surface. In this embodiment, the upper surface of the plate-shaped portion 31b is the characteristic portion of the workpiece 31. In the image of the workpiece 31 according to this embodiment, the characteristic portion is detected based on the image of the edge portion 31c, which has a circular planar shape. The shape of the workpiece may be any shape. Furthermore, the characteristic portion may be any portion that can be detected by image processing.

[0035] 1 to 3, the first robot device 3 places the workpiece 31 on the surface of the stand 35 in a predetermined manner. The position and posture of the robot 1 change, and the hand 5 grasps the workpiece 31 placed on the top surface of the stand 35. The robot device 3 changes the position and posture of the robot 1 to transport the workpiece 31 to a predetermined position.

[0036] When the workpiece 31 is placed on the surface of the stand 35, the position of the workpiece 31 on the stand 35 may be shifted. Before gripping the workpiece 31, the control device 2 places the camera 6 vertically above the workpiece 31. At this time, the distance hz from the surface of the characteristic part of the workpiece 31 to the camera 6 is determined in advance.

[0037] In this example, the attitude of the camera 6 is adjusted so that the optical axis of the camera 6 (the Z axis of the camera coordinate system 72) is approximately perpendicular to the upper surface of the plate-like portion 31b, which is a characteristic portion of the workpiece 31. Furthermore, the position and attitude of the robot 1 are controlled so that the position of the upper surface of the plate-like portion 31b is at a predetermined value on the Z axis of the camera coordinate system 72. Such a position and attitude of the robot are predetermined in the operation program 67.

[0038] The imaging control unit 53 images the workpiece 31 with the camera 6. The feature detection unit 56 detects the feature portions of the workpiece 31 based on a reference image of the feature portions of the workpiece 31 and the image of the workpiece 31 captured by the camera 6. The reference image of the workpiece 31 used to detect the feature portions is created in advance and stored in the storage unit 42. The feature detection unit 56 detects the edge portion 31c of the upper surface of the plate-shaped portion 31b of the workpiece 31 in the image captured by the camera 6 by performing pattern matching.

[0039] The estimated position calculation unit 57 calculates the three-dimensional position of the workpiece 31 based on the position of the characteristic part of the workpiece 31 in the image, the camera parameters 68 stored in the memory unit 42, and the distance hz from the characteristic part of the workpiece 31 to the camera 6. In this embodiment, the three-dimensional position of the workpiece calculated from the image of the workpiece 31 is referred to as the estimated position. As the estimated position, the position of a specific point 31p set on the workpiece 31 can be calculated. In this embodiment, the center of the circle shaped on the upper surface of the plate-shaped portion 31b is set as the specific point 31p. The estimated position calculation unit 57 can calculate the estimated position of the workpiece 31 in the camera coordinate system 72. Furthermore, the estimated position calculation unit 57 can convert coordinate values ​​in the camera coordinate system 72 into coordinate values ​​in the reference coordinate system 71 based on the position and posture of the robot 1.

[0040] The motion command generation unit 64 generates a motion command that corrects the position and posture of the robot 1 included in the motion program 67 so that it corresponds to the estimated position of the workpiece 31. The motion command generation unit 64 sends the corrected motion command to the motion control unit 43. The motion control unit 43 drives the robot 1 based on the corrected motion command. Alternatively, the motion command generation unit 64 may send the position and posture of the robot 1 that correspond to the estimated position of the workpiece 31 to the motion control unit 43. This control causes the position and posture of the robot 1 to match the estimated position of the workpiece 31. The robot device 3 can grasp the workpiece 31 with high precision.

[0041] In this way, the robot device 3 of this embodiment can correct the position and posture of the robot based on the image captured by the camera 6. Since the robot 1 is driven in accordance with the estimated position of the workpiece 31 estimated from the image of the workpiece 31, the hand 5 can be positioned accurately. The robot device 3 of this embodiment can perform predetermined tasks with high precision.

[0042] (Calculation Model) Here, we will explain a calculation model for calculating a three-dimensional position in space corresponding to a specific position in an image captured by a camera. In the calculation model, calculations are performed using camera parameters as sensor parameters that include multiple parameters. The camera parameters include internal parameters and external parameters. The camera parameters also include the distance hz from the camera to a characteristic part of the workpiece. The position in the camera image that corresponds to an arbitrary position in space is generally expressed by the following equation (1) using a pinhole camera model.

[0043]

[0044] The coordinate values ​​(X, Y, Z) of a three-dimensional position are expressed, for example, in a reference coordinate system 71. The coordinate values ​​(u, v) of a position on an image are expressed, for example, in an image coordinate system 73 set on the image. The external parameter matrix is ​​a transformation matrix for converting the coordinate values ​​(X, Y, Z) of the reference coordinate system 71 in space into coordinate values ​​(x, y, z) of a camera coordinate system 72. The internal parameter matrix is ​​a matrix for converting the coordinate values ​​(x, y, z) of the camera coordinate system 72 into coordinate values ​​(u, v) of an image coordinate system 73 in the image. Here, the coordinate value of the Z axis in the camera coordinate system 72 is predetermined corresponding to the distance hz from the camera 6 to the characteristic portion of the workpiece 31.

[0045] The above formula (1) is an ideal example in which there is no image distortion due to lens distortion, etc. In contrast, parameter changes due to lens distortion, etc. can be taken into consideration. First, the calculation of the three-dimensional position and external parameter matrix in formula (1) can be expressed by the following formula (2).

[0046]

[0047] Using equation (2), coordinate values ​​(X, Y, Z) expressed in the reference coordinate system 71 can be converted into coordinate values ​​(x, y, z) expressed in the camera coordinate system 72. Next, in order to take image distortion into consideration, variables x' and y' are defined as shown in the following equations (3) and (4). Furthermore, variables x'' and y'' that take distortion into consideration are calculated as shown in equations (5) and (6). Here, the relationship between variables x', y', and r is as shown in equation (7).

[0048]

[0049] In equations (5) and (6), coefficients k1 to k6 are coefficients relating to lens distortion in the radial direction, and coefficients p1 and p2 are coefficients relating to lens distortion in the circumferential direction. Using variables x'' and y'' that take lens distortion into account, coordinate values ​​(u, v) on the image in image coordinate system 73 can be calculated as shown in the following equations (8) and (9). Equations (8) and (9) correspond to the parts of equation (1) above that perform calculations using the matrix of internal parameters.

[0050]

[0051] The above explanation describes a method for calculating a position in an image from a three-dimensional position in space, but based on the above relational expressions, a three-dimensional position in space (X, Y, Z) can be calculated based on the coordinate values ​​(u, v) of the position on the image and the distance hz from the camera 6 to the characteristic portion of the workpiece 31 in the camera coordinate system 72. The Z-axis coordinate value of the camera coordinate value corresponding to the distance hz from the camera 6 to the characteristic portion of the workpiece 31 can be determined in advance and stored in the storage unit 42. The estimated position calculation unit 57 can calculate a three-dimensional position in space (X, Y, Z) from the coordinate values ​​(u, v) of a specific position on the image based on a calculation model.

[0052] Here, referring to equations (1) to (9), the parameters for converting the coordinate values ​​in the camera coordinate system 72 into the coordinate values ​​in the image coordinate system 73 correspond to internal parameters. In a calculation model for calculating a three-dimensional position from a specific position in an image, for example, the product f of the focal length and the effective size of the image is used. x , f y , image center c x , c y , and distortion-related coefficients k1 to k6, p1, and p2 correspond to internal parameters. Parameters for converting coordinate values ​​in the reference coordinate system 71 into coordinate values ​​in the camera coordinate system 72 correspond to external parameters. For example, the parameters included in the external parameter matrix of equation (1) correspond to external parameters.

[0053] (Parameter Evaluation and Modification Control) Evaluation of camera parameters and modification of camera parameters in this embodiment will be described. When setting new camera parameters 68, any method can be used. For example, camera parameters can be created using a flat calibration plate. For example, a grid pattern of dots of different sizes is drawn on the surface of the calibration plate. The calibration plate is placed at a predetermined position on the mount 35.

[0054] The worker measures in advance the position of the calibration plate in the reference coordinate system 71. The distance between the dots drawn on the calibration plate is measured in advance. The robot 1 is driven so that the camera 6 is positioned at a predetermined distance in the height direction perpendicular to the calibration plate. The camera 6 captures an image of the dot pattern on the calibration plate.

[0055] Next, the camera 6 is moved to an arbitrary position relative to the calibration plate. For example, the camera 6 is moved in the direction of the optical axis of the camera 6. The positions of the dots on the calibration plate in the image coordinate system 73 in images captured at multiple positions by the camera 6, the position and orientation of the robot, and the positions of the dots on the calibration plate measured in advance in the reference coordinate system 71 are acquired. Then, the camera parameters can be generated based on Equation (1) or Equations (8) and (9).

[0056] Fig. 4 shows a schematic side view of the camera and workpiece in this embodiment. In Fig. 4, the calibration plate 32 and the workpiece 31 are shown overlapping each other. A dot pattern is drawn on the upper surface of the calibration plate 32. The upper surface of the calibration plate 32 and the camera 6 are disposed with a distance hc between them.

[0057] In this embodiment, the workpiece 31 is thicker than the calibration plate 32. The distance hz between the upper surface of the plate-like portion 31b, which is a characteristic portion of the workpiece 31, and the camera 6 is shorter than the distance hc by the distance difference Δh. As such, the distance hz used to calculate the three-dimensional position may differ from the preset distance hc. For this reason, it may be difficult to accurately obtain the distance hz that reflects the distance difference Δh.

[0058] Alternatively, the operator can capture an image of the workpiece 31 including its characteristic portions to generate a reference image for pattern matching. This reference image may contain image distortion or errors in the shape of the characteristic portions. Furthermore, as the robot device continues to operate, the values ​​of the camera parameters for detecting the accurate three-dimensional position of the workpiece may deviate over time, making it impossible to accurately calculate the estimated position of the workpiece.

[0059] When new camera parameters are generated, and when the robot device continues to perform work, each parameter included in the camera parameters may deviate from its optimal value. Therefore, the control device 2 of this embodiment evaluates the camera parameters. That is, it determines whether or not the three-dimensional position of the workpiece can be detected from the image with an error within the allowable range when the current camera parameters are used. If the camera parameters are evaluated poorly, the camera parameters are corrected.

[0060] FIG. 5 shows a flowchart of control for evaluating and correcting camera parameters in this embodiment. Here, a reference image for pattern matching is generated in advance. Furthermore, in order to evaluate and correct the camera parameters, the distance hz from the camera 6 to the top surface, which is a characteristic portion of the workpiece 31, when capturing the image is determined in advance. The orientation of the camera 6 is also determined in advance. In this embodiment, the orientation of the robot 1 is determined in advance so that the Z direction of the camera coordinate system 72 is perpendicular to the top surface of the workpiece 31. Furthermore, the orientation of the robot 1 is determined in advance so that the X axis of the camera coordinate system 72 is parallel to the X axis of the reference coordinate system 71. As mentioned above, the distance hz may be corrected as one of the camera parameters.

[0061] 1, 2, and 5, in step 101, the camera 6 is moved to a first position P6a. Any position at which the camera 6 can capture an image of a characteristic portion of the workpiece 31 can be selected as the first position P6a. In this embodiment, as shown in FIG. 1, the position and attitude of the robot 1 are adjusted so that the camera 6 is positioned directly above the workpiece 31. The operator corrects the position and attitude of the robot 1 using the teaching console 26. Alternatively, the operation command generator 64 may control the position and attitude of the robot 1 so that the camera 6 moves to a predetermined first position P6a.

[0062] The actual position calculation unit 52 calculates a first actual position when the camera 6 is placed at the first position P6a, based on the position and orientation of the robot 1. In this embodiment, the position of one of the members calculated based on the driving state of the moving device is referred to as the actual position. The relative position and orientation of the camera coordinate system 72 with respect to the tool coordinate system 74 are determined in advance. The actual position calculation unit 52 can calculate the first actual position of the camera 6 using coordinate values ​​in the reference coordinate system 71, for example.

[0063] In step 102, the imaging control unit 53 causes the camera 6 to capture an image of the workpiece 31. In particular, the camera 6 captures an image of the upper surface of the plate-like portion 31b, which is a characteristic portion of the workpiece 31.

[0064] In step 103, the feature detection unit 56 detects feature portions in the image captured when the camera 6 is positioned at the first position P6a, based on the reference image. The feature detection unit 56 detects feature portions by pattern matching. In this embodiment, the feature detection unit 56 detects the edge portion 31c on the upper surface of the plate-shaped portion 31b of the workpiece 31. The feature detection unit 56 calculates the position of a specific point 31p on the upper surface of the plate-shaped portion 31b in the image coordinate system 73 as the position of the workpiece 31 in the image.

[0065] 6 shows an image captured by the camera when the camera is placed at the first position. An image coordinate system 73 is set in the image 81 captured by the camera 6. The image coordinate system 73 is composed of a U axis and a V axis. The image 81 shows an image 31g of a workpiece. The image 81 also displays a mark 31m indicating the position of a characteristic part detected by the characteristic detection unit 56 through pattern matching. In this example, the mark 31m is composed of two intersecting lines. The intersection of the two lines is a specific point 31p indicating the position of the characteristic part (the position of the workpiece 31).

[0066] 5 , in step 103, the estimated position calculation unit 57 detects a first estimated position of the workpiece 31 when the camera 6 is placed at a first position. In this embodiment, the position of the workpiece calculated based on an image captured by the camera is referred to as the estimated position. As the estimated position, the position of a specific point 31p of the characteristic portion can be calculated using coordinate values ​​in the camera coordinate system 72. The estimated position calculation unit 57 calculates the first estimated position of the workpiece 31 based on the position of the characteristic portion of the workpiece 31 in the image, the position and posture of the robot 1, the camera parameters, and the distance hz, using the above-mentioned formula (1) or formulas (2) to (9).

[0067] Next, in step 104, the position and attitude of the robot 1 are controlled so that the camera 6 is located at a second position P6b. The second position P6b is different from the first position P6a. Any position that allows the camera 6 to capture an image of a characteristic portion of the workpiece 31 can be selected as the second position P6b. In this embodiment, the attitude of the camera 6 is approximately constant at the first position and the second position P6b of each camera. For example, the direction of the Z axis of the camera coordinate system 72 is maintained parallel to the vertical direction. Furthermore, the attitude of the camera 6 is maintained so that the directions of the X axis and the Y axis of the camera coordinate system 72 are constant.

[0068] 7 is a schematic diagram of the first robot device when the camera is positioned at a second position. The operator operates the teaching console 26 to change the position and orientation of the robot, thereby moving the camera 6 in a direction parallel to the surface of the characteristic portion of the workpiece 31, as shown by arrow 98, to position it at a second position P6b. The distance hz from the characteristic portion of the workpiece 31 to the camera 6 is maintained constant. The camera 6 is also moved so that the characteristic portion of the workpiece 31 is included within the field of view 6a of the camera 6. Alternatively, the motion command generator 64 may control the position and orientation of the robot 1 so that the camera 6 moves to a predetermined second position P6b.

[0069] 5 , in step 104, the measured position calculation unit 52 calculates a second measured position of the camera 6 when the camera 6 is placed at the second position P6b, based on the position and posture of the robot 1. The measured position calculation unit 52 can calculate the second measured position of the camera 6 using coordinate values ​​in the reference coordinate system 71, for example.

[0070] In step 105, the camera 6 captures an image of the workpiece 31. In step 106, the same control as in step 103 is performed. The feature detection unit 56 detects a feature portion in the image captured when the camera 6 is placed at the second position P6b. Furthermore, the feature detection unit 56 detects the position of the feature portion in the image.

[0071] 8 shows an image captured by the camera when it is positioned at the second position P6b. In image 82, the position of workpiece image 31g has moved. Also, because image 82 was captured from an oblique direction of workpiece 31, the side surfaces of plate-like portions 31a and 31b of workpiece 31 are slightly captured. Image 82 also shows marks 31m and specific points 31p that indicate the positions of the characteristic portions detected by feature detection unit 56.

[0072] 5, in step 106, estimated position calculation unit 57 calculates a second estimated position in camera coordinate system 72 when camera 6 is placed at the second position. Estimated position calculation unit 57 calculates the second estimated position of workpiece 31 using the position of the characteristic part in the image, the position and orientation of robot 1, camera parameters, and distance hz.

[0073] Next, in step 107, the evaluation unit 58 calculates the amount of movement between the first measured position and the second measured position. The evaluation unit 58 calculates the distance from the first estimated position to the second estimated position. Here, if the camera parameters are accurate, the amount of movement between the first measured position and the second measured position and the distance from the first estimated position to the second estimated position will be approximately the same. In this embodiment, the amount of movement between the first measured position and the second measured position is referred to as the amount of movement of the measured positions. Furthermore, the distance from the first estimated position to the second estimated position is referred to as the distance between the estimated positions.

[0074] In step 108, the evaluation unit 58 evaluates the camera parameters. In this embodiment, the evaluation unit 58 evaluates the camera parameters based on the movement amount of the actual measured position and the distance between the estimated positions. In this embodiment, the evaluation unit 58 calculates an error by subtracting the movement amount of the actual measured position from the distance between the estimated positions. The evaluation unit determines whether the magnitude of the error (absolute value of the error) is smaller than a predetermined judgment value. The judgment value can be set, for example, depending on the magnitude of the movement amount of the actual measured position. Alternatively, a fixed judgment value can be set for all movement amounts of the actual measured position.

[0075] In step 108, if the magnitude of the error is smaller than a predetermined judgment value, it can be determined that the estimated position of the workpiece can be calculated with high accuracy using the currently set camera parameters. It can also be determined that there is no need to modify the current camera parameters. In this case, control ends.

[0076] On the other hand, if the error in the distance between the estimated positions relative to the movement amount of the actual measured position is equal to or greater than a predetermined judgment value, it can be determined that the estimated position of the workpiece calculated using the current camera parameters contains an error equal to or greater than the allowable value. In this case, control proceeds to step 109 to correct the camera parameters.

[0077] The evaluation unit 58 can evaluate the camera parameters using any method. For example, the evaluation unit 58 may evaluate the camera parameters based on the ratio of the distance between the estimated positions to the amount of movement of the actual measured positions. The evaluation unit 58 can determine that the closer this ratio is to 1, the higher the accuracy of the camera parameters. The evaluation unit may determine a judgment value for the ratio to evaluate the camera parameters.

[0078] In step 109, the selection unit 59 selects parameters to be modified from among the multiple parameters included in the camera parameters. The operator can select the parameters to be modified and input them via the input unit 27 of the teaching pendant 26. Alternatively, the operator may select all of the parameters included in the camera parameters. The selection unit 59 can set the parameters to be modified in response to the operator's operation. Alternatively, the selection unit 59 can automatically select the parameters to be modified through predetermined control. Alternatively, the parameters to be modified from among the multiple parameters included in the camera parameters may be predetermined, without the selection unit 59 selecting the parameters.

[0079] Next, in step 110, the modifying unit 60 modifies the parameters selected by the selecting unit 59. The modifying unit 60 modifies the camera parameters based on the amount of movement between the first measured position and the second measured position and the first estimated position and the second estimated position. If the accuracy of the camera parameters is high, the amount of movement of the measured position and the distance between the estimated positions will be approximately the same. In this embodiment, the modifying unit 60 calculates the error between the amount of movement of the measured position and the distance between the estimated positions. For example, the modifying unit 60 calculates the error by subtracting the amount of movement of the measured position from the distance between the estimated positions.

[0080] (Parameter Correction Method) Here, a method for correcting the camera parameters by the correction unit 60 of this embodiment will be described. The control device 2 changes at least one of the first position and the second position of the robot 1 and repeats image capture. Image capture is repeated a number of times corresponding to the number of parameters selected by the selection unit 59. The correction unit 60 can correct the camera parameters based on the images obtained by repeated image capture so as to minimize errors.

[0081] In the first robot device 3 of this embodiment, the transformation from the coordinate values ​​in the camera coordinate system 72 to the coordinate values ​​in the reference coordinate system 71 can be expressed by the following equation (2) including the matrix R and the variable t. Next, the movement amount of the measured position in the camera coordinate system 72 is calculated as (Dx c , Dy c , Dz c), the movement amount of the measured position in the camera coordinate system 72 is expressed as the movement amount (Dx w , Dy w , Dz w ) can be calculated from

[0082]

[0083] Next, in the image coordinate system 73, the position of the workpiece moves from the coordinate value (u1, v1) corresponding to the first position to the coordinate value (u2, v2) corresponding to the second position. The coordinate values ​​in each image coordinate system 73 can be converted into coordinate values ​​(x1, y1, hz) and coordinate values ​​(x2, y2, hz) in the camera coordinate system 72 using equations (1) or (2) to (9) based on the distance hz from the camera to the characteristic portion and the parameters c, f, k, and p. Equation (11) can be derived by modifying equation (10).

[0084]

[0085] Here, [(x1, y1, hz) - (x2, y2, hz)] on the left side of equation (11) corresponds to the distance between the estimated positions in the camera coordinate system 72. w , Dy w , Dz w ) is the amount of movement of the measured position in the camera coordinate system. When the distance between the estimated positions calculated from the images is equal to the amount of movement of the measured position, it can be determined that the camera parameters are correct. In other words, when the accuracy of the camera parameters is high, Equation (11) holds. Alternatively, Equation (12), which is a modification of Equation (11), holds.

[0086] The left side of equation (12) is a measurement value corresponding to the movement amount of the actual measured position. In other words, it corresponds to the movement amount of the actual measured position of the camera calculated from the position and posture of the robot. The right side of equation (12) corresponds to the distance between the estimated positions of the workpiece calculated from the position of the feature part of the image, parameters c, f, k, p, and matrix R. Equation (12) is expressed in the reference coordinate system 71. In reality, equation (12) does not hold due to errors in the camera parameters. By introducing the error ΔD, equation (13) can be derived based on equation (12). (ΔDx, ΔDy, ΔDz) on the left side of equation (13) corresponds to the error ΔD obtained by subtracting the movement amount of the actual measured position from the distance between the estimated positions.

[0087] The correction unit 60 calculates the selected parameters by Newton's method so that (ΔDx, ΔDy, ΔDz) on the left side of equation (13) approaches (0, 0, 0). The correction unit 60 can calculate the selected parameters by calculating a function of Newton's method and repeating the calculation by an iterative method. This method using the error between two points allows the parameters to be calculated one by one.

[0088] Next, another example of calculating the camera parameters will be described. The right side of equation (12) is a function of the parameters c, f, k, p, the matrix R, and the distance hz. Therefore, the movement amount (Dx w , Dy w , Dz w ) can be expressed by the following equations (14) to (16).

[0089]

[0090] For example, Dx on the right side of equation (14) w (c, f, k, p, R, hz, u, v) indicates that it is a function of parameters c, f, k, p, matrix R, distance hz, and coordinate values ​​(u, v) of the image. Next, the coordinate values ​​(u, v) and the amount of movement (Dx w , Dy w , Dz w ) can be calculated using the least squares method based on the measured values ​​of the parameter f. For example, the parameter f can be calculated using the Gauss-Newton method. x, f y When calculating the distance hz, Δf is calculated using the following equation (17): x , Δf y , and Δhz, the parameter f x , f y , and the distance hz can be corrected. By calculating the inverse matrix of the matrix on the left side using the singular value decomposition method, Δf x , Δf y , and Δhz can be calculated. By repeating this operation, the parameter f x , f y and the distance hz can be made closer to the true value.

[0091]

[0092] When data is acquired at a plurality of moving points, the points at each position are numbered and written with subscripts in the formula, resulting in the following formula (18).

[0093]

[0094] When calculating more variables simultaneously, the calculation can be performed by expanding the formula (17). For example, the parameter c x , c y When this is included, the following equation (19) is obtained.

[0095]

[0096] (Display of Error Graph) In the control device 2 of this embodiment, the display control unit 63 can display the relationship between the movement amount of the actual measured position and the error in the distance between the estimated positions relative to the movement amount of the actual measured position on the display unit 28. In this embodiment, the graphs shown in Figs. 9 to 14 can be displayed on the display unit 28.

[0097] 9 is a graph showing the relationship between the amount of movement of the measured position in the X direction in the reference coordinate system and the error in the distance between the estimated positions relative to the amount of movement of the measured position. Fig. 10 is a graph showing the relationship between the amount of movement of the measured position in the Y direction in the reference coordinate system and the error in the distance between the estimated positions relative to the amount of movement of the measured position.

[0098] For example, as shown by arrow 98 in FIG. 7 , the camera 6 is moved in the direction of the X-axis of the reference coordinate system 71. Images of the workpiece 31 can be captured at one first position of the camera 6 and multiple second positions of the camera 6. The correction unit 60 calculates the error in the distance between the estimated positions relative to the amount of movement of the actual measured position at each actual measured position. Note that the display control unit 63 or the evaluation unit 58 may calculate the error instead of the correction unit 60. The display control unit 63 displays the relationship between the amount of movement of the actual measured position and the error on the display unit 28. The error here is an error in any direction, regardless of the X direction or Y direction.

[0099] The graphs in Figures 9 and 10 show the trend of errors relative to the measured position. For example, parameters containing errors can be estimated from the trend of errors relative to the measured position, and the parameters to be corrected can be determined. Alternatively, the position where the error is maximum can be estimated, and the maximum value of the error can be confirmed.

[0100] Figure 11 is a graph showing the error in the X direction and the error in the Y direction when the camera is moved in the X direction of the reference coordinate system. Figure 12 is a graph showing the error in the Y direction and the error in the X direction when the camera is moved in the Y direction of the reference coordinate system. Referring to Figures 11 and 12, in this example, the error in the same direction as the camera movement increases as the amount of movement increases. Furthermore, the error in the direction perpendicular to the direction of camera movement remains almost unchanged. In this case, it can be determined that the setting of the distance hz from the camera to the characteristic portion of the workpiece is incorrect.

[0101] 13 and 14 are graphs showing errors measured in different states from those shown in FIGS. 11 and 12 . FIG. 13 is a graph showing the error in the X direction and the error in the Y direction when the camera is moved in the X direction of the reference coordinate system. FIG. 14 is a graph showing the error in the Y direction and the error in the X direction when the camera is moved in the Y direction of the reference coordinate system. Referring to FIGS. 13 and 14 , in this example, the error in the direction in which the camera is moved and the error in the direction perpendicular to the direction in which the camera is moved are large. In this case, it can be determined that the setting of the camera coordinate system is incorrect. For example, it can be determined that the camera coordinate system has an error around the R axis. Therefore, it can be determined that the external parameters related to the setting of the camera coordinate system need to be corrected.

[0102] In this way, the display unit 28 displays the relationship between the amount of movement of the actual measured position and the error in the distance between the estimated positions relative to the amount of movement of the actual measured position, so that the operator can visually understand the magnitude of the error, i.e., can visually determine the accuracy of the camera parameters.

[0103] The operator or the evaluation unit 58 can determine whether or not to modify the camera parameters based on the graph displayed on the display unit 28. Furthermore, the operator or the selection unit 59 can select the parameters to be modified based on the error in the direction in which the camera is moved and the error in the direction perpendicular to the direction in which the camera is moved. In other words, it is possible to select the parameters that are the cause of the inability to accurately calculate the estimated position.

[0104] The image showing the relationship between the movement amount of the actual measured position and the error in the distance between the estimated positions relative to the movement amount of the actual measured position is not limited to a graph, and any image can be displayed. For example, the display unit may display a table showing the movement amount and error of the actual measured position.

[0105] 2, the image estimation unit 61 of the image processing unit 51 estimates an image of the characteristic part when the camera 6 is placed at the second position based on the image of the characteristic part captured when the camera 6 is placed at the first position, the camera parameters, and the amount of movement of the actual measured position. The display control unit 63 can display the image of the characteristic part estimated by the image estimation unit 61 superimposed on the image of the characteristic part captured when the camera 6 is placed at the second position.

[0106] 15 shows an image including an estimated image of the workpiece estimated when the workpiece is imaged at the second position. Image 89 displays image 31g of the workpiece 31 when the workpiece 31 is actually imaged at the second position, and estimated image 31r of the workpiece estimated by image estimation unit 61. Specific point 31p indicates the position of a characteristic part of the workpiece 31 in actual image 31g of the workpiece.

[0107] 6 and 15 , in an image 81 captured by the camera at the first position, the workpiece image 31g is positioned at the center of the image 81. The estimated position calculation unit 57 calculates a first estimated position of the workpiece 31 in the camera coordinate system 72. The measured position calculation unit 52 calculates a first measured position and a second measured position of the camera. The image estimation unit 61 estimates a second estimated position based on the first estimated position and the direction and amount of movement of the measured position from the first measured position to the second measured position. That is, the image estimation unit 61 estimates the position of the specific point when the camera is placed at the second position using the coordinate values ​​of the camera coordinate system 72. The image estimation unit 61 can estimate the coordinate values ​​of the specific point 31p in the image coordinate system using the coordinate values ​​of the camera coordinate system 72 and the camera parameters according to Equation (1). Then, the image estimation unit 61 can create an estimated image 31r of the workpiece based on the position of the specific point 31p of the workpiece 31 in the image 89. The display control unit 63 can display the estimated image 31r of the workpiece estimated by the image estimation unit 61.

[0108] Alternatively, the image estimation unit 61 can estimate the amount of movement and direction of movement of the specific point 31p in the image coordinate system 73 using equation (1) based on the amount of movement and direction of movement from the first measured position to the second measured position of the camera 6. Then, the position of the specific point 31p corresponding to the second position in the image can be calculated from the position of the specific point 31p corresponding to the first position. The image estimation unit 61 can generate an estimated image 31r of the workpiece based on the position of the specific point 31p in the image corresponding to the second measured position.

[0109] The display control unit 63 displays the image of the characteristic part estimated by the image estimation unit 61 superimposed on the image of the characteristic part captured when the camera 6 is placed at the second position. If the accuracy of the camera parameters 68 is excellent, the position of the workpiece image 31g in the image 89 after the camera is moved to the second position will approximately match the position of the estimated workpiece image 31r estimated by the image estimation unit 61. On the other hand, if the accuracy of the camera parameters 68 is poor, the estimated workpiece image 31r will be displayed shifted from the actual workpiece image 31g.

[0110] The worker can estimate the magnitude of the error from the image of the workpiece by looking at the image 89. The worker may determine whether or not to correct the camera parameters 68 based on this image 89.

[0111] Here, the position of the estimated image 31r estimated by the image estimation unit 61 and the position of the workpiece image 31g captured by the camera 6 may move in the same direction relative to the position of the workpiece captured at the first position, but the magnitude of the movement may differ. In this case, it can be determined that the setting of the distance hz between the camera 6 and the characteristic portion of the workpiece 31 is inaccurate. Alternatively, a portion of the estimated image 31r of the workpiece may accurately match the image 31g of the workpiece, but other portions of the estimated image 31r of the workpiece may not match the image 31g of the workpiece actually captured. In this case, it can be determined that the reference image of the workpiece is inaccurate. For example, if multiple surfaces with different heights are set as characteristic portions in the reference image, some of the estimated image of the workpiece estimated by the image estimation unit will no longer match the image of the workpiece captured by the camera 6. In this case, the operator can re-create the reference image.

[0112] In this way, the estimated image created by the image estimation unit 61 can identify inaccurate parameters among the multiple parameters included in the camera parameters 68. Alternatively, it can be identified that the reference image is inaccurate.

[0113] In the image 89 shown in FIG. 15, the workpiece image 31g is positioned at the edge of the image 89. A two-dimensional camera has the characteristic that the image distortion increases the further away from the optical axis of the camera. In this example, distortion occurs in the workpiece image 31g. On the other hand, the estimated workpiece image 31r is an image estimated without taking image distortion into consideration. In other words, the image estimation unit 61 generates the estimated workpiece image 31r based on equation (1). As a result, the shapes of the images of the plate-like portions 31a and 31b of the workpiece are perfect circles.

[0114] 16 shows an image including an estimated image of a workpiece that takes image distortion into consideration. The image estimation unit 61 of this embodiment can create an estimated image 31s of the workpiece that includes image distortion. In particular, the image estimation unit 61 can estimate an estimated image of a characteristic portion of the workpiece that includes image distortion. In image 90, the estimated image 31s of the workpiece is displayed superimposed on an image 31g of the workpiece that was actually captured when the camera 6 was positioned at the second position.

[0115] The image estimation unit 61 can calculate the contour of the workpiece 31 in the image 90 by using a method that is an extension of the method for calculating the position of a specific point in an image. For example, the image estimation unit 61 acquires a first estimated position of the workpiece 31 when the camera is placed at a first position. Then, the image estimation unit 61 sets a large number of points on the contour of the workpiece 31. The image estimation unit 61 calculates the coordinate values ​​of the points on the contour in the camera coordinate system 72.

[0116] The image estimation unit 61 estimates the positions of points on the contour of the workpiece in the camera coordinate system 72 when the camera 6 is placed at the second position, based on the direction and amount of movement from the first actual measurement position to the second actual measurement position. Then, the image estimation unit 61 estimates the positions of points on the contour of the workpiece in the image coordinate system 73 from the positions of points on the contour of the workpiece in the camera coordinate system 72, based on equations (2) to (9). The image estimation unit 61 can generate an estimated image 31s of the workpiece by connecting the positions of points on the contour of the workpiece in the image.

[0117] This control causes an image including image distortion to be displayed in image 90. It is possible to display an estimated image that is close to the image of the workpiece that was actually captured. By controlling to display an estimated image including image distortion, it becomes easier to determine which of the camera parameters, the distance hz from the camera to the workpiece, or the reference image has a problem.

[0118] That is, when an estimated image including image distortion is displayed, if the estimated image matches the image captured by the camera, it can be determined that the accuracy of the camera parameters is excellent. If an estimated image including image distortion is generated but the image position is completely different, it can be determined that the setting of the distance hz from the camera to the workpiece is inaccurate. Furthermore, if an image including image distortion is generated but parts of the image do not overlap, it can be determined that the reference image is inappropriate. Furthermore, if the worker checks the reference image and finds it to be appropriate, it can be determined that the camera parameters related to distortion are inaccurate.

[0119] FIG. 17 shows an image illustrating the scores calculated by the score calculation unit of this embodiment. In this embodiment, the edge 31c of the upper surface of the plate-shaped portion 31b of the workpiece 31 is designated as the characteristic portion. With reference to FIGS. 2 and 17, the score calculation unit 62 detects an overlapping portion between the characteristic portion of the estimated image 31s of the workpiece estimated by the image estimation unit 61 and the characteristic portion of the image 31g of the workpiece captured by the camera. In this embodiment, the overlapping portion is a line segment 31n where the outline of the characteristic portion of the estimated image 31s of the workpiece and the outline of the characteristic portion of the image 31g of the workpiece captured by the camera substantially overlap. In this embodiment, if the distance between the outline of the characteristic portion of the estimated image 31s of the workpiece and the outline of the characteristic portion of the image 31g of the workpiece is less than a predetermined determination value, it is determined that the outlines overlap with each other.

[0120] The display control unit 63 can display the image so that the overlapping portions between the characteristic portions of the image 31g of the workpiece captured by the camera 6 and the characteristic portions of the estimated image 31r of the workpiece estimated by the image estimation unit 61 can be seen. Here, a line segment 31n representing the overlapping portion between the characteristic portions is shown by a thick solid line.

[0121] The score calculation unit 62 calculates a score indicating the degree to which the image of the characteristic part estimated by the image estimation unit 61 overlaps with the image of the characteristic part captured when the camera 6 is placed at the second position. The score calculation unit 62 of this embodiment can calculate, as a score, the ratio of the length of the line segment 31n of the overlapping part to the length around the characteristic part of the image 31g of the workpiece captured by the camera 6. The closer this score is to 1, the better it can be determined that the estimated image matches the image of the workpiece captured by the camera.

[0122] Alternatively, the score calculation unit 62 can calculate a score by subtracting the length of the overlapping line segment from the perimeter of the characteristic portion of the workpiece image 31g captured by the camera 6. The closer this score is to zero, the better it can be determined that the estimated image matches the workpiece image captured by the camera. The display control unit 63 can display the score on the display unit 28.

[0123] This control allows the operator to determine that the higher the score, the more accurately the camera parameters have been set. Alternatively, the evaluation unit 58 may determine whether to modify the camera parameters based on the score calculated by the score calculation unit 62.

[0124] (Second Robot Device) Fig. 18 shows a schematic diagram of a second robot device in this embodiment. In the second robot device 4, the position of the camera 6 is fixed, and the position of the workpiece 31 is changed by the robot 1. That is, in the second robot device, one of the camera 6 and the workpiece 31 is the workpiece 31. The workpiece 31 is held by a hand 5 attached to the robot 1. The position of the camera 6, which is the other member, is fixed by a mount 36. The robot 1 is configured to change its position and orientation, thereby changing the three-dimensional position and orientation of the workpiece 31.

[0125] The characteristic portion of the workpiece 31 is a portion that can be imaged by the camera 6. In the second robot device 4, the bottom surface of the workpiece 31 is set as the characteristic portion of the workpiece 31. For example, the edge of the circle on the bottom surface of the plate-shaped portion 31b is set as the characteristic portion of the workpiece 31. The position of the center of the bottom surface of the plate-shaped portion 31b is set as the position of a specific point. A reference image of the bottom surface of the plate-shaped portion 31b is created in advance. The characteristic detection unit 56 detects the outline of the bottom surface of the plate-shaped portion 31b in the image of the workpiece 31.

[0126] The estimated position calculation unit 57 calculates the estimated position of the workpiece 31 in the camera coordinate system 72 based on the positions of characteristic parts in the image of the workpiece 31 and the camera parameters. The actual position calculation unit 52 calculates the actual position, which is the three-dimensional position of the workpiece 31, in the reference coordinate system 71 based on the position and posture of the robot 1 as the driving state of the moving device.

[0127] The control device 2 places the workpiece 31 at a first position P31a by changing the position and posture of the robot. The camera 6 captures an image of the bottom surface of the workpiece 31, which is a characteristic part of the workpiece 31. The control device 2 moves the workpiece 31 to a second position P31b, as indicated by arrow 99, by changing the position and posture of the robot. The distance hz from the camera 6 to the characteristic part of the workpiece 31 is determined in advance. The robot 1 translates the workpiece 31 so that the distance hz from the camera 6 to the characteristic part of the workpiece 31 and the posture of the workpiece 31 do not change.

[0128] The feature detection unit 56 detects feature portions in an image captured when the workpiece 31 is placed at the first position P31a and feature portions in an image captured when the workpiece 31 is placed at the second position P31b. The estimated position calculation unit 57 calculates a first estimated position when the workpiece 31 is placed at the first position and a second estimated position when the workpiece 31 is placed at the second position. The actual measured position calculation unit 52 calculates a first actual measured position when the workpiece 31 is placed at the first position and a second actual measured position when the workpiece 31 is placed at the second position.

[0129] The evaluation unit 58 evaluates the camera parameters based on the error between the amount of movement between the first measured position and the second measured position and the distance from the first estimated position to the second estimated position, and determines whether the camera parameters need to be corrected.

[0130] When the camera parameters need to be corrected, the corrector 60 corrects the camera parameters based on the amount of movement between the first measured position and the second measured position and the distance from the first estimated position to the second estimated position. In particular, the corrector 60 calculates the error between the amount of movement between the first measured position and the second measured position and the distance from the first estimated position to the second estimated position, and corrects the camera parameters so as to minimize the error.

[0131] In this way, the camera parameters can be evaluated and corrected in the second robotic device 4. The other configurations, actions, and effects are the same as those of the first robotic device 3, and therefore the description thereof will not be repeated here.

[0132] In this embodiment, a robot is used as an example of a moving device that moves a workpiece or a sensor, but the present invention is not limited to this. Any moving device that can change the relative position of the sensor with respect to the workpiece can be used. For example, a conveyor that transports a workpiece, an automated guided vehicle (AGV), or the like can be used as the moving device.

[0133] In this embodiment, one of the sensor and the workpiece is moved by a moving device, while the other is fixed in position, but this is not limited to this. Both the sensor and the workpiece may be configured to be moved by a moving device. For example, the moving device may include a robot and a conveyor, and the robot may move the sensor as one of the components, while the conveyor may move the workpiece as the other component. In this case, too, the correction unit can correct the sensor parameters based on the amount of movement based on the first and second actual measured positions of the one component and the distance from the first estimated position to the second estimated position.

[0134] More specifically, the actual position calculation unit calculates a first actual position of one member when it is placed at a first position and a second actual position of the one member when it is placed at a second position. The actual position calculation unit calculates a third actual position of the other member when it is placed at the first position and a fourth actual position of the other member when it is placed at the second position. The correction unit can calculate the relative movement amount of one member with respect to the other member based on the first actual position and the second actual position of the one member and the third actual position and the fourth actual position of the other member. The correction unit can calculate the movement amount of one member when the other member is assumed to be stationary. In other words, the correction unit can calculate the movement amount of one member as seen from the other member. The correction unit can then correct the sensor parameters based on the relative movement amount of one member with respect to the other member.

[0135] In this embodiment, a camera that captures a two-dimensional image is exemplified as the sensor, but the present invention is not limited to this. Any sensor that calculates a three-dimensional position in space corresponding to a specific position in an image of a workpiece captured by the sensor using sensor parameters can be used. For example, a three-dimensional sensor that detects a spatial position can be used as the sensor. If the three-dimensional sensor is fixed to the work tool, external parameters that convert coordinate values ​​in the sensor coordinate system to coordinate values ​​in the tool coordinate system can be evaluated or corrected.

[0136] In this embodiment, the operator mainly manually drives the robot and evaluates the camera parameters and images, but this is not limiting. The control device may automatically drive the moving device, capture images with the sensor, and evaluate the sensor parameters and images based on a predetermined program. Furthermore, the control device may automatically select parameters to be modified and modify the parameters.

[0137] At least one of the embodiments described above can provide a parameter correction device that can correct sensor parameters with excellent accuracy.

[0138] Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0139] The following supplementary notes are disclosed regarding the above-described embodiment and modifications.

[0140] (Supplementary Note 1) A parameter correction device 2 corrects sensor parameters for calculating a three-dimensional position corresponding to a specific position in an image 31g of a workpiece captured by a sensor 6, the parameter correction device 2 comprising: a feature detection unit 56 that detects a characteristic portion 31c in the image 31g of the workpiece; an estimated position calculation unit 57 that calculates an estimated position, which is the three-dimensional position of the workpiece 31, based on the position of the characteristic portion in the image 31g of the workpiece and the sensor parameters; an actual measured position calculation unit 52 that calculates an actual measured position, which is the three-dimensional position of at least one of the sensor 6 and the workpiece 31, based on a driving state of a moving device 1 that moves at least one of the members; and a correction unit 60 that corrects the sensor parameters, wherein the feature detection unit 56 detects a characteristic portion in an image 81 captured when the one member is located at a first position P6a, P31a and a characteristic portion in an image 82 captured when the one member is located at a second position P6b, P31b, a parameter correction device (2) in which an estimated position calculation unit (57) calculates a first estimated position when one of the members is placed at a first position (P6a, P31a) and a second estimated position when one of the members is placed at a second position; an actual measured position calculation unit (52) calculates a first actual measured position when one of the members is placed at the first position and a second actual measured position when one of the members is placed at the second position; and a correction unit (60) corrects sensor parameters based on a movement amount based on the first actual measured position and the second actual measured position and a distance from the first estimated position to the second estimated position.

[0141] (Supplementary Note 2) The parameter modification device according to Supplementary Note 1, wherein the modification unit 60 calculates an error between a movement amount based on the first measured position and the second measured position and a distance from the first estimated position to the second estimated position, and modifies the sensor parameters so as to minimize the error.

[0142] (Supplementary Note 3) The parameter modification device according to Supplementary Note 1, further comprising a selection unit 59 that selects a parameter to be modified from a plurality of parameters included in the sensor parameters, and a modification unit 60 that modifies the parameter selected by the selection unit based on images obtained by repeatedly capturing images while changing at least one of the first position and the second position a number of times corresponding to the number of parameters selected by the selection unit 59.

[0143] (Supplementary Note 4) The parameter correction device according to Supplementary Note 1, wherein the other of the sensor and the workpiece is fixed in position, and the correction unit corrects the sensor parameters based on the amount of movement between the first measured position and the second measured position.

[0144] (Supplementary Note 5) The parameter correction device according to any one of Supplementary Notes 1 to 4, further comprising an evaluation unit 58 that evaluates the sensor parameters, wherein the evaluation unit 58 evaluates the sensor parameters based on an error between a movement amount based on the first measured position and the second measured position and a distance from the first estimated position to the second estimated position.

[0145] (Supplementary Note 6) A parameter correction device according to any one of Supplementary Notes 1 to 4, comprising a display control unit 63 that controls an image to be displayed on the display unit 28, wherein the correction unit 60 calculates an error between an amount of movement based on the first measured position and the second measured position and a distance from the first estimated position to the second estimated position, and the display control unit 63 displays on the display unit 28 the relationship between the amount of movement based on the first measured position and the second measured position and the error.

[0146] (Supplementary Note 7) A parameter modification device according to any one of Supplementary Notes 1 to 4, comprising: a display control unit that controls an image to be displayed on a display unit; and an image estimation unit 61 that estimates an image of the characteristic part when one of the components is placed at a second position based on an image of the characteristic part captured when one of the components is placed at a first position, sensor parameters, and a movement amount based on an actual measured position, wherein the display control unit 63 displays the image of the characteristic part estimated by the image estimation unit superimposed on the image of the characteristic part captured when one of the components is placed at the second position.

[0147] (Supplementary Note 8) The parameter correction device according to Supplementary Note 7, wherein the sensor parameters include parameters related to image distortion, and the image estimation unit 61 estimates an image of a characteristic portion including the image distortion.

[0148] (Supplementary Note 9) A parameter modification device according to any one of Supplementary Notes 1 to 4, comprising: an image estimation unit 61 that estimates an image of the characteristic part when one of the components is placed at a second position based on an image of the characteristic part captured when one of the components is placed at a first position, sensor parameters, and a movement amount based on an actual measured position; and a score calculation unit 62 that calculates a score indicating the degree to which the image of the characteristic part estimated by the image estimation unit 61 overlaps with the image of the characteristic part captured when one of the components is placed at the second position.

[0149] (Appendix 10) A parameter correction device as described in Appendix 4, wherein the moving device is a robot 1, one member is a sensor 6, the other member is a workpiece 31, the sensor 6 is fixed to the robot 1 or a work tool 5 attached to the robot 1, the position and posture of the sensor 6 change when the robot 1 changes its position and posture, and the actual position calculation unit 52 calculates the actual position of the sensor 6 based on the position and posture of the robot 1.

[0150] (Appendix 11) A parameter correction device as described in Appendix 4, wherein the moving device is a robot 1, one member is a workpiece 31, the other member is a sensor 6, the workpiece 31 is grasped by a work tool 5 attached to the robot 1, the position and posture of the workpiece 31 change when the robot 1 changes its position and posture, and an actual position calculation unit 52 calculates the actual position of the workpiece 31 based on the position and posture of the robot 1.

[0151] (Supplementary Note 12) The parameter modification device according to any one of Supplementary Notes 1 to 4, wherein the sensor is a camera 6 that captures a two-dimensional image, and the sensor parameters are camera parameters including extrinsic parameters and intrinsic parameters.

[0152] 1 Robot 2 Control device 6 Camera P6a, P6b Position 23 Position detector 31 Work 31a, 31b Plate-shaped portion 31c Edge portion 31p Specific point 31g Image 31r, 31s Estimated image P31a, P31b Position 40 Control device main body 42 Memory unit 51 Image processing unit 52 Actual measured position calculation unit 56 Feature detection unit 57 Estimated position calculation unit 58 Evaluation unit 59 Selection unit 60 Correction unit 61 Image estimation unit 62 Score calculation unit 63 Display control unit 68 Camera parameters 81, 82, 83, 84, 85a, 85b, 86a, 86b, 87a, 87b, 88a, 88b, 89, 90, 91 Image

Claims

1. A parameter correction device that corrects sensor parameters for calculating a three-dimensional position corresponding to a specific position in an image of a workpiece captured by a sensor, comprising: a feature detection unit that detects a feature in the image of the workpiece; an estimated position calculation unit that calculates an estimated position, which is the three-dimensional position of the workpiece, based on the position of the feature in the image of the workpiece and the sensor parameters; an actual position calculation unit that calculates an actual position, which is the three-dimensional position of at least one of the sensor and the workpiece, based on a driving state of a moving device that moves at least one of the sensor and the workpiece; and a correction unit that corrects the sensor parameters, wherein the feature detection unit detects a feature in an image captured when the one member is disposed at a first position and a feature in an image captured when the one member is disposed at a second position, the estimated position calculation unit calculates a first estimated position when the one member is disposed at the first position and a second estimated position when the one member is disposed at the second position, and the actual position calculation unit calculates the first actual position when the one member is disposed at the first position and the second actual position when the one member is disposed at the second position, The parameter correction device, wherein the correction unit corrects the sensor parameters based on an amount of movement based on a first actual measured position and a second actual measured position, and a distance from a first estimated position to a second estimated position.

2. The parameter modification device of claim 1, wherein the modification unit calculates an error between an amount of movement based on the first measured position and the second measured position and a distance from the first estimated position to the second estimated position, and modifies the sensor parameters so as to minimize the error.

3. A parameter modification device as described in claim 1, further comprising a selection unit which selects a parameter to be modified from a plurality of parameters included in sensor parameters, and the modification unit modifies the parameter selected by the selection unit based on images obtained by repeatedly capturing images while changing at least one of a first position and a second position a number of times corresponding to the number of parameters selected by the selection unit.

4. The parameter correction device according to claim 1, wherein the other of the sensor and the workpiece is fixed in position, and the correction unit corrects the sensor parameters based on the amount of movement between the first actual measured position and the second actual measured position.

5. A parameter correction device as described in any one of claims 1 to 4, further comprising an evaluation unit that evaluates a sensor parameter, the evaluation unit evaluating the sensor parameter based on an error between an amount of movement based on a first actual measured position and a second actual measured position and a distance from the first estimated position to the second estimated position.

6. A parameter correction device as described in any one of claims 1 to 4, further comprising a display control unit that controls an image to be displayed on a display unit, wherein the correction unit calculates an error between an amount of movement based on a first actual measured position and a second actual measured position and a distance from a first estimated position to a second estimated position, and the display control unit displays on the display unit the relationship between the amount of movement based on the first actual measured position and the second actual measured position and the error.

7. A parameter modification device as described in any one of claims 1 to 4, comprising: a display control unit that controls an image to be displayed on a display unit; and an image estimation unit that estimates an image of the characteristic part when the one of the components is disposed at a second position based on an image of the characteristic part captured when the one of the components is disposed at a first position, sensor parameters, and an amount of movement based on an actual measured position, wherein the display control unit displays the image of the characteristic part estimated by the image estimation unit superimposed on the image of the characteristic part captured when the one of the components is disposed at the second position.

8. The parameter correction device according to claim 7, wherein the sensor parameters include parameters related to image distortion, and the image estimation unit estimates an image of a characteristic portion including image distortion.

9. A parameter modification device as claimed in any one of claims 1 to 4, comprising: an image estimation unit that estimates an image of the characteristic part when the one of the components is disposed at a second position based on an image of the characteristic part captured when the one of the components is disposed at a first position, sensor parameters, and an amount of movement based on an actual measured position; and a score calculation unit that calculates a score indicating the degree to which the image of the characteristic part estimated by the image estimation unit overlaps with the image of the characteristic part captured when the one of the components is disposed at the second position.

10. A parameter correction device as described in claim 4, wherein the moving device is a robot, the one member is a sensor, the other member is a workpiece, the sensor is fixed to the robot or a work tool attached to the robot, and the position and orientation of the sensor are changed when the robot changes its position and orientation, and the actual position calculation unit calculates the actual position of the sensor based on the position and orientation of the robot.

11. A parameter correction device as described in claim 4, wherein the moving device is a robot, the one member is a workpiece, the other member is a sensor, the workpiece is held by a work tool attached to the robot, and the position and posture of the workpiece change when the robot changes its position and posture, and the actual position calculation unit calculates the actual position of the workpiece based on the position and posture of the robot.

12. A parameter correction device according to any one of claims 1 to 4, wherein the sensor is a camera that captures a two-dimensional image, and the sensor parameters are camera parameters including external parameters and internal parameters.

Images