Method, system, and computer program for calibrating a camera for a robot.

Abstract

To provide a technique which can calibrate a camera without sticking a marker on a robot arm.SOLUTION: This method includes: (a) a process for estimating pixel coordinate values of a plurality of feature points preset to a robot arm from an image of the robot arm photographed by a camera using a first machine learning model; (b) a process for estimating first coordinate values of the plurality of feature points in a three-dimensional camera coordinate system from the pixel coordinate values of the plurality of feature points using a second machine learning model; (c) a process for calculating second coordinate values of the plurality of feature points in a three-dimensional robot coordinate system using an encoder value of the robot arm; and (d) a process for executing the processes (a) to (c) as for a plurality of postures of the robot arm and estimating a calibration parameter including an extrinsic parameter of the camera using the first coordinate values and the second coordinate values of the plurality of feature points in the plurality of postures.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a method for calibrating a camera for a robot, a system, and a computer program.

Background Art

[0002] When using a camera in robot work, calibration parameters of the camera are set by performing calibration in advance. The calibration parameters include internal parameters representing the performance of the lens and the relationship between the lens and pixels, and external parameters representing the relative position between the camera and an external device. Calibration of the camera is usually performed using a dedicated calibration board on which patterns such as dots or checkerboards are printed or processed. Such normal calibration processing requires dedicated equipment and takes a very long time for its adjustment. Also, calibration needs to be performed each time the positional relationship between the camera and the robot changes, which is one of the factors that degrades the usability of the robot.

[0003] In order to eliminate such a degradation in usability, Patent Document 1 proposes a method of attaching an arm marker to a robot arm and associating a known position on the robot arm with the arm marker.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the conventional technology described above requires attaching markers to the robot arm, which presents problems such as a cumbersome work process. Therefore, there is a need for a technology that can perform camera calibration without attaching markers to the robot arm. [Means for solving the problem]

[0006] A first embodiment of this disclosure provides a method for calibrating a camera for a robot. This method includes: (a) using a trained first machine learning model to estimate the pixel coordinate values ​​of a plurality of feature points pre-set on the robot arm from an image of the robot arm captured by the camera; (b) using a trained second machine learning model to estimate the first coordinate values ​​of the plurality of feature points in a three-dimensional camera coordinate system from the pixel coordinate values ​​of the plurality of feature points; (c) using encoder values ​​of the robot arm to calculate the second coordinate values ​​of the plurality of feature points in a three-dimensional robot coordinate system; and (d) performing steps (a) to (c) for a plurality of poses of the robot arm, and estimating calibration parameters, including external parameters of the camera, using the first and second coordinate values ​​of the plurality of feature points in the plurality of poses.

[0007] A second embodiment of the present disclosure provides a system for performing camera calibration processing for a robot. The system comprises a camera capable of photographing the robot arm of the robot, and a calibration processing unit that performs camera calibration processing using images captured by the camera. The calibration processing unit performs the following: (a) a process of estimating the pixel coordinate values ​​of a plurality of feature points pre-set on the robot arm from images of the robot arm captured by the camera using a first trained machine learning model; (b) a process of estimating the first coordinate values ​​of the plurality of feature points in a three-dimensional camera coordinate system from the pixel coordinate values ​​of the plurality of feature points using a second trained machine learning model; (c) a process of calculating the second coordinate values ​​of the plurality of feature points in a three-dimensional robot coordinate system using encoder values ​​of the robot arm; and (d) a process of performing the above processes (a) to (c) for a plurality of poses of the robot arm, and estimating calibration parameters including external parameters of the camera using the first and second coordinate values ​​of the plurality of feature points in the plurality of poses.

[0008] A third embodiment of this disclosure provides a computer program that causes a processor to perform a camera calibration process for a robot. This computer program causes the processor to perform the following: (a) using a trained first machine learning model to estimate the pixel coordinate values ​​of a plurality of feature points pre-set on the robot arm from an image of the robot arm taken by the camera; (b) using a trained second machine learning model to estimate the first coordinate values ​​of the plurality of feature points in a three-dimensional camera coordinate system from the pixel coordinate values ​​of the plurality of feature points; (c) using the encoder values ​​of the robot arm to calculate the second coordinate values ​​of the plurality of feature points in a three-dimensional robot coordinate system; and (d) performing the above processes (a) to (c) for a plurality of poses of the robot arm, and estimating calibration parameters including external parameters of the camera using the first and second coordinate values ​​of the plurality of feature points in the plurality of poses. [Brief explanation of the drawing]

[0009] [Figure 1] An explanatory diagram showing the configuration of the robot system. [Figure 2] A conceptual diagram showing the relationships between various coordinate systems. [Figure 3] Functional block diagram of an information processing device. [Figure 4] An explanatory diagram showing the functions of the first and second machine learning models. [Figure 5] A flowchart illustrating the procedure for the calibration process in the first embodiment. [Figure 6] A flowchart illustrating the calibration procedure in the second embodiment. [Figure 7] An explanatory diagram showing an example of mechanical constraints used in step S135. [Modes for carrying out the invention]

[0010] A. First Embodiment: Figure 1 is an explanatory diagram showing an example of a robot system in one embodiment. This robot system comprises a robot 100 as a camera-utilizing device, a robot controller 200 that controls the robot 100, an information processing device 300, and a camera 400. The information processing device 300 is, for example, a personal computer. The information processing device 300 transmits control commands to the robot controller 200. The information processing device 300 can also be called a "higher-level information processing device."

[0011] The robot 100 comprises a robot arm 110 and an end effector 120. The robot arm 110 has a first link 111 and a second link 112. The end effector 120 can be implemented as a gripper or suction pad capable of gripping a workpiece.

[0012] The robot arm 110 and the end effector 120 are connected sequentially at joints J1 to J3. However, the robot 100 can be any robot with a robot arm mechanism having multiple joints.

[0013] The camera 400 is used to photograph the work that is the object of the operation of the robot 100 and recognize the position and orientation of the work. Also, the camera 400 has a field of view large enough to photograph the robot arm 110. As the camera 400, a two-dimensional RGB camera can be used, or an RGBD camera or a monochrome camera may also be used. The RGBD camera is a camera having an RGB camera and a D camera (depth camera).

[0014] FIG. 2 is a conceptual diagram showing the relationships of various coordinate systems. The coordinate systems depicted in FIG. 2 are as follows. (1) Robot coordinate system Σr The robot coordinate system Σr is a three-dimensional orthogonal coordinate system with a predetermined position of the robot 100 as the coordinate origin. (2) Camera coordinate system Σc The camera coordinate system Σc is a three-dimensional orthogonal coordinate system with a predetermined position of the camera 400 as the coordinate origin. (3) Pixel coordinate system Σp The pixel coordinate system Σp is a two-dimensional orthogonal coordinate system of the image photographed by the camera 400.

[0015] The pixel coordinate values (u, v) of the pixel coordinate system Σp and the three-dimensional coordinate values (Xc, Yc, Zc) of the camera coordinate system Σc can be converted using the internal parameters of the camera 400 as shown in the following equation.

Equation

[0016] The three-dimensional coordinate values (Xc, Yc, Zc) of the camera coordinate system Σc and the three-dimensional coordinate values (Xr, Yr, Zr) of the robot coordinate system Σr can be converted using the coordinate transformation matrix [R|t]cr represented by the external parameters of the camera 400 as shown in the following equation.

Equation

[0017] Figure 2 further shows multiple feature points P1, P2, and P3 set on the robot arm 110. In this embodiment, feature points P1, P2, and P3 are set at the center positions of the three joints J1, J2, and J3 of the robot arm 110, respectively. Feature points P1 and P2 can be considered to indicate the positions of both endpoints of the first link 111, which has a length L1. Similarly, feature points P2 and P3 can be considered to indicate the positions of both endpoints of the second link 112, which has a length L2. It is possible to set any number of feature points, two or more, on the robot arm 110, and it is preferable to set three or more feature points. Furthermore, feature points can be set at any position on the robot arm 110, not just at the joint positions. However, setting feature points at the joint positions of the robot arm 110 is preferable because it allows for easy calculation of the position of the feature points from the encoder values ​​at multiple joints of the robot arm 110. Even when feature points are set at locations other than joints, the relative relationship between the feature point location and the joint location can be predetermined, allowing the feature point location to be calculated from the encoder value of the robot arm 110.

[0018] Figure 3 is a block diagram showing the functions of the information processing device 300. The information processing device 300 includes a processor 310, a memory 320, and an interface circuit 330. An input device 340 and a display device 350 are connected to the interface circuit 330, as well as a robot controller 200. The robot controller 200 is connected to a camera 400, and is also connected to a current sensor 140 that measures the motor current of each joint of the robot 100, and an encoder 150 that measures the displacement of each joint.

[0019] The processor 310 functions as a calibration execution unit 610 and a robot control execution unit 620. The calibration execution unit 610 performs a process to determine the calibration parameters of the camera 400 by performing calibration of the camera 400 relative to the robot 100. The calibration execution unit 610 includes a first machine learning model 611, a second machine learning model 612, a learning execution unit 614, and a calibration processing unit 616. The robot control execution unit 620 recognizes a workpiece from an image of the work area captured by the camera 400 and performs a process to cause the robot 100 to perform work using the recognized workpiece. The functions of the calibration execution unit 610 are realized by the processor 310 executing computer programs stored in the memory 320. However, some or all of the functions of the calibration execution unit 610 may be realized by hardware circuits.

[0020] Memory 320 stores training data LD used for training two machine learning models 611 and 612, robot attribute data RD, calibration parameters CP, and robot control program RP. Robot attribute data RD is data indicating attributes of the robot 100, such as its mechanical structure and range of motion. Calibration parameters CP include the internal and external parameters mentioned above. Robot control program RP consists of multiple instructions for operating the robot 100.

[0021] Figure 4 is an explanatory diagram showing the functions of the first machine learning model 611 and the second machine learning model 612. The first machine learning model 611 takes an image IM(u,v) of the robot arm 110 captured by the camera 400 as input and estimates the pixel coordinate values ​​Pj(u,v) of multiple feature points Pj. j is an ordinal number that distinguishes multiple feature points P1 to P3. The image IMp(u,v) shown in Figure 4 is a virtual image in which multiple feature points Pj are drawn on the original image IM(u,v). This image IMp(u,v) may be created by the calibration processing unit 616 during calibration execution and displayed on the display device 350, but it is not necessary to create the image IMp(u,v).

[0022] The second machine learning model 612 takes the pixel coordinate values ​​Pj(u,v) of multiple feature points Pj as input and estimates the 3D coordinate values ​​Pj(Xc,Yc,Zc) of the feature points Pj in the camera coordinate system Σc. These 3D coordinate values ​​Pj(Xc,Yc,Zc) are used by the calibration processing unit 616 when estimating the calibration parameters of the camera 400.

[0023] As the first machine learning model 611, various neural networks that infer structural feature points in an image can be used, for example, any of the following: (1a) DeeplabCUT (http: / / www.mackenziemathislab.org / deeplabcut) (1b) DeepPose (https: / / arxiv.org / abs / 1312.4659) The training data for the first machine learning model 611 is preferably training data that includes images of the robot arm 110 taken by the camera 400 and the pixel coordinate values ​​Pj(u,v) of multiple feature points Pj.

[0024] As the second machine learning model 612, various neural networks that infer SfM (Structure from Motion) or NRSfM (Non-Rigid Structure from Motion) can be used, for example, any of the following can be used. (2a) C3DPO (https: / / arxiv.org / abs / 1909.02533) (2b) RepNet (https: / / sites.google.com / view / repnet) The training data for the second machine learning model 612 is preferably training data that includes the pixel coordinate values ​​Pj(u,v) of multiple feature points Pj obtained from the image of the robot arm 110, and the 3D coordinate values ​​Pj(Xc,Yc,Zc) of multiple feature points Pj.

[0025] Figure 5 is a flowchart showing the calibration procedure in the first embodiment. Here, it is assumed that the two machine learning models 611 and 612 have been trained. It is also assumed that, prior to the calibration procedure, at least the external parameters of the camera 400's calibration parameters are unknown. The internal parameters of the camera 400 may be known or unknown. In other words, the calibration procedure in Figure 5 is a process for estimating the calibration parameters, including the external parameters of the camera 400.

[0026] In step S110, the calibration processing unit 616 uses the camera 400 to capture an image of the robot arm 110 and generate an image IM. In step S120, the calibration processing unit 616 uses the first machine learning model 611 to estimate the pixel coordinate values ​​Pj(u,v) of multiple feature points Pj from the image IM of the robot arm 110. In step S130, the calibration processing unit 616 uses the second machine learning model 612 to estimate the first coordinate value Pj(Xc,Yc,Zc) of the feature point Pj in the camera coordinate system Σc from the pixel coordinate value Pj(u,v) of the feature point Pj. Note that even if the intrinsic parameters of the camera 400 are known before the calibration process, the Z coordinate value Zc is necessary to calculate the coordinate value (Xc,Yc,Zc) in the camera coordinate system Σc from the pixel coordinate value (u,v) using equation (1) above. Therefore, the second machine learning model 612 is used in step S130 to obtain the Z coordinate value Zc.

[0027] In step S140, the calibration processing unit 616 calculates the second coordinate value Pj(Xr,Yr,Zr) of the feature point Pj in the robot coordinate system Σr using the encoder value of the robot 100. This calculation process is performed using forward kinematics, with the encoder values ​​of joints J1 and J2 and the lengths L1 and L2 of links 111 and 112.

[0028] In step S150, the calibration processing unit 616 estimates the calibration parameters of the camera 400 using the first coordinate value Pj(Xc,Yc,Zc) and the second coordinate value Pj(Xr,Yr,Zr) of the feature point Pj. In this embodiment, a Kalman filter is used for this estimation process. By using a Kalman filter, parameter estimation can be performed sequentially even when the positional relationship between the camera 400 and the robot 100 changes.

[0029] Generally, a Kalman filter consists of two processes: time update and observation update. By repeating these two processes, the estimation error is minimized. When estimating the calibration parameters of camera 400, there is no control input and no time change, so time update does not need to be considered. That is, by defining the state vector, observation matrix, and observation vector and performing observation updates, the calibration parameters of camera 400 can be estimated. For example, when estimating only the external parameters, the state vector x, observation matrix H, and observation vector Z are as follows.

number

[0030] This section describes a method for estimating only external parameters, but it is also possible to perform estimation that includes internal parameters. When estimating internal parameters along with external parameters, the internal parameters are added to the state vector x, and the observation matrix is ​​modified accordingly. Note that calibration parameters can be estimated using methods other than Kalman filters, such as successive least squares or particle filters.

[0031] In step S160, the calibration processing unit 616 determines whether the calibration parameter estimation process is complete. For example, if the processes in steps S110 to S150 are considered as one iteration, the calibration parameter estimation process can be determined to be complete when a predetermined number of iterations have been performed. Alternatively, the calibration parameter estimation process can be determined to be complete when the difference in estimated values ​​or the difference in error covariance between the previous iteration and the current iteration falls below a threshold.

[0032] If the calibration parameter estimation process is not complete, the process returns to step S110, and steps S110 to S150 described above are executed again. Note that in multiple iterations, the posture of the robot arm 110 is set in step S110 so that the robot arm 110 takes on a different posture. That is, each time an image IM of the robot arm 110 is taken by the camera 400 for the new posture of the robot arm 110, steps S120 to S150 are executed, and the calibration parameters are estimated by performing observation updates using a Kalman filter. As a result, a correspondence between the first coordinate value Pj(Xc,Yc,Zc) of the camera coordinate system Σc and the second coordinate value Pj(Xr,Yr,Zr) of the robot coordinate system Σr is obtained for multiple different postures of the robot arm 110, making it possible to correctly estimate the calibration parameters of the camera 400.

[0033] Once the calibration parameter estimation process is complete, the estimated calibration parameters CP are saved to memory 320, and the process shown in Figure 5 is finished. In the case of a stationary robot, the relative positions of camera 400 and robot 100 do not change significantly. Therefore, by saving the estimated calibration parameters CP along with estimation information including the error covariance of the Kalman filter, and reusing it as initial values ​​in the next calibration process, the convergence of the estimated values ​​can be accelerated.

[0034] The calibration process shown in Figure 5 above may be performed in parallel with the actual work using the robot 100. In this way, the calibration process of the camera 400 can be performed while the robot 100 is working on the workpiece, so no special processing time is required for the calibration process. Even when the calibration process is performed in parallel with the actual work, it is preferable to take an image of the robot arm 110 while the robot arm 110 is stopped in step S110.

[0035] As described above, in the first embodiment, for a plurality of feature points Pj pre-set on the robot arm 110, the first coordinate values ​​Pj(Xc,Yc,Zc) in the camera coordinate system Σc are estimated from the image IM captured by the camera 400, and the second coordinate values ​​Pj(Xr,Yr,Zr) in the robot coordinate system Σr are calculated from the encoder values ​​of the robot arm 110. The calibration parameter CP of the camera 400 is then estimated from these three-dimensional coordinate values, so that the calibration parameter CP of the camera 400 can be estimated without using a calibration board. Furthermore, the camera 400 can be calibrated without attaching markers to the robot arm 110.

[0036] B. Second Embodiment: Figure 6 is a flowchart showing the calibration procedure in the second embodiment. The only difference from the first embodiment is the addition of step S135 between steps S130 and S140; the other steps are the same as those in the calibration procedure of the first embodiment shown in Figure 5. The apparatus configuration is also the same as in the first embodiment.

[0037] In step S135, the calibration processing unit 616 inputs the estimated first coordinate value Pj(Xc,Yc,Zc) of the feature point Pj obtained in step S130 into the Kalman filter and updates the estimated first coordinate value of the feature point Pj probabilistically by adding constraints to the observation. Mechanical or structural constraints related to the robot arm 110 can be used as constraints. That is, in step S135, the estimated first coordinate value Pj(Xc,Yc,Zc) of the feature point Pj is updated by updating the observation using a Kalman filter that includes mechanical constraints related to the robot arm 110.

[0038] Figure 7 is an explanatory diagram showing an example of a constraint used in step S135. In this example, the length L2 of the second link 112 is used as a constraint. In this case, the state vector x, observation matrix H, and observation vector Z in the Kalman filter used in step S135 are as follows.

number

[0039] By performing observation updates using a Kalman filter that includes such mechanical constraints, it is possible to update the estimated first coordinate value of feature point Pj so as to satisfy the constraints given by equation (4) above.

[0040] Another example of a mechanical constraint is the constraint that multiple feature points Pj must lie on the same plane. For example, if three feature points P1 to P3 lie on the same plane, and we assume a 3D vector represented by the 3D coordinate values ​​of each feature point Pj, then the volume of the cube spanned by the three 3D vectors will be zero. This is equivalent to the determinant of the matrix formed by arranging the three 3D column vectors being zero. In this case, the state vector x, observation matrix H, and observation vector Z in the Kalman filter are as follows.

number

[0041] In addition to the examples mentioned above, other mechanical constraints can also be applied, such as the lengths of multiple links, or the requirement that three or more feature points Pj are aligned in the same straight line when multiple links extend linearly. Furthermore, by applying multiple constraints simultaneously, the first coordinate values ​​of multiple feature points Pj can be updated to more accurate values.

[0042] By using the mechanical constraints described above, it becomes possible to update the first coordinate value of the feature point Pj in accordance with the shape of the robot arm 110, thereby suppressing variations and jumps in the 3D position of the feature point Pj, and thus enabling stable calibration parameter estimation.

[0043] The second embodiment also has the same effects as the first embodiment. Furthermore, in the second embodiment, the estimated first coordinate values ​​of multiple feature points Pj in the camera coordinate system Σc are updated using a Kalman filter that includes mechanical constraints, thereby improving the accuracy of the estimation of the first coordinate values.

[0044] Other forms: This disclosure is not limited to the embodiments described above, and can be implemented in various forms without departing from its spirit. For example, this disclosure can also be implemented in the following forms (aspects). The technical features in the embodiments described above that correspond to the technical features in each of the forms described below can be replaced or combined as appropriate in order to solve some or all of the problems of this disclosure, or to achieve some or all of the effects of this disclosure. Furthermore, if such technical features are not described as essential in this specification, they can be deleted as appropriate.

[0045] (1) According to a first embodiment of the present disclosure, a method for calibrating a camera for a robot is provided. This method includes: (a) using a trained first machine learning model to estimate the pixel coordinate values ​​of a plurality of feature points pre-set on the robot arm from an image of the robot arm captured by the camera; (b) using a trained second machine learning model to estimate the first coordinate values ​​of the plurality of feature points in a three-dimensional camera coordinate system from the pixel coordinate values ​​of the plurality of feature points; (c) using encoder values ​​of the robot arm to calculate the second coordinate values ​​of the plurality of feature points in a three-dimensional robot coordinate system; and (d) performing steps (a) to (c) for a plurality of poses of the robot arm, and estimating calibration parameters, including external parameters of the camera, using the first and second coordinate values ​​of the plurality of feature points in the plurality of poses. This method estimates the first coordinate values ​​of the 3D camera coordinate system from images captured by a camera for multiple feature points pre-set on the robot arm, and calculates the second coordinate values ​​of the 3D robot coordinate system from the encoder values ​​of the robot arm. By estimating the camera calibration parameters from these 3D coordinate values, the camera calibration parameters can be estimated without using a calibration board. Furthermore, camera calibration can be performed without attaching markers to the robot arm.

[0046] (2) In the above method, the multiple feature points may be set at the center position of each of the multiple joints of the robot arm. This method allows for easy estimation of the first coordinate values ​​of multiple feature points using a first machine learning model, and also allows for easy calculation of the second coordinate values ​​in the robot coordinate system.

[0047] (3) In the above method, step (d) may include a step of estimating the calibration parameters by performing observation updates using a Kalman filter each time an image of the robot arm is taken with the camera for a new posture of the robot arm. This method uses a Kalman filter to estimate the calibration parameters, so the accuracy of the calibration parameters can be improved each time a new pose of the robot arm is captured by a camera and estimation is performed.

[0048] (4) In the above method, step (b) may include a step of updating the estimated values ​​of the first coordinate values ​​of the plurality of feature points by performing observation updates on the first coordinate values ​​obtained by the second machine learning model using a Kalman filter that includes mechanical constraints relating to the robot arm. This method improves the accuracy of estimating the first coordinate values ​​by updating the estimated first coordinate values ​​of multiple feature points in the 3D camera coordinate system using a Kalman filter that includes mechanical constraints.

[0049] (5) According to a second embodiment of the present disclosure, a system is provided for performing camera calibration processing for a robot. The system comprises a camera capable of photographing the robot arm of the robot, and a calibration processing unit that performs camera calibration processing using images captured by the camera. The calibration processing unit performs the following: (a) a process of estimating the pixel coordinate values ​​of a plurality of feature points pre-set on the robot arm from images of the robot arm captured by the camera using a first trained machine learning model; (b) a process of estimating the first coordinate values ​​of the plurality of feature points in a three-dimensional camera coordinate system from the pixel coordinate values ​​of the plurality of feature points using a second trained machine learning model; (c) a process of calculating the second coordinate values ​​of the plurality of feature points in a three-dimensional robot coordinate system using encoder values ​​of the robot arm; and (d) a process of performing the above processes (a) to (c) for a plurality of poses of the robot arm, and estimating calibration parameters including external parameters of the camera using the first and second coordinate values ​​of the plurality of feature points in the plurality of poses.

[0050] (6) According to a third embodiment of the present disclosure, a computer program is provided that causes a processor to perform a camera calibration process for a robot. This computer program causes the processor to perform the following: (a) using a trained first machine learning model to estimate the pixel coordinate values ​​of a plurality of feature points pre-set on the robot arm from an image of the robot arm taken by the camera; (b) using a trained second machine learning model to estimate the first coordinate values ​​of the plurality of feature points in a three-dimensional camera coordinate system from the pixel coordinate values ​​of the plurality of feature points; (c) using the encoder values ​​of the robot arm to calculate the second coordinate values ​​of the plurality of feature points in a three-dimensional robot coordinate system; and (d) performing the above processes (a) to (c) for a plurality of poses of the robot arm, and estimating calibration parameters including external parameters of the camera using the first and second coordinate values ​​of the plurality of feature points in the plurality of poses.

[0051] This disclosure can also be implemented in various forms other than those described above. For example, it can be implemented in the form of a robot system comprising a robot and a robot information processing device, a computer program for realizing the functions of the robot information processing device, and a non-transitory storage medium on which the computer program is recorded. [Explanation of Symbols]

[0052] 100...Robot, 110...Robot arm, 111...First link, 112...Second link, 120...End effector, 140...Current sensor, 150...Encoder, 200...Robot controller, 300...Information processing unit, 310...Processor, 320...Memory, 330...Interface circuit, 340...Input device, 350...Display device, 400...Camera, 610...Calibration execution unit, 611...First machine learning model, 612...Second machine learning model, 614...Learning execution unit, 616...Calibration processing unit, 620...Robot control execution unit

Claims

1. A method for calibrating a camera for a robot, (a) Using a pre-trained first machine learning model, estimate the pixel coordinate values ​​of a plurality of feature points pre-set on the robot arm from an image of the robot arm taken by the camera, (b) Using a trained second machine learning model, estimate the first coordinate values ​​of the feature points in the 3D camera coordinate system from the pixel coordinate values ​​of the feature points, (c) A step of calculating the second coordinate values ​​of the plurality of feature points in the three-dimensional robot coordinate system using the encoder values ​​of the robot arm, (d) A step of performing steps (a) to (c) for multiple postures of the robot arm, and estimating calibration parameters including external parameters of the camera using the first coordinate values ​​and second coordinate values ​​of the multiple feature points in the multiple postures, Methods that include...

2. The method according to claim 1, A method in which the aforementioned multiple feature points are set at the central position of each of the multiple joints of the robot arm.

3. The method according to claim 1, The method includes step (d) of estimating the calibration parameters by performing observation updates using a Kalman filter each time an image of the robot arm is taken with the camera for a new posture of the robot arm.

4. The method according to claim 1, The method includes step (b) of updating the estimated values ​​of the first coordinate values ​​of the plurality of feature points by performing observation updates on the first coordinate values ​​obtained by the second machine learning model using a Kalman filter that includes mechanical constraints relating to the robot arm.

5. A system for performing camera calibration processing on a robot, A camera capable of photographing the robotic arm of the aforementioned robot, A calibration processing unit that performs calibration processing on the camera using the image captured by the camera, Equipped with, The calibration processing unit, (a) Using a first machine learning model that has been trained, a process is performed to estimate the pixel coordinate values ​​of a plurality of feature points that have been set in advance on the robot arm from an image of the robot arm taken by the camera, (b) A process of estimating the first coordinate values ​​of the plurality of feature points in the 3D camera coordinate system from the pixel coordinate values ​​of the plurality of feature points using a trained second machine learning model, (c) A process to calculate the second coordinate values ​​of the plurality of feature points in the three-dimensional robot coordinate system using the encoder values ​​of the robot arm, (d) A process to perform the above processes (a) to (c) for multiple postures of the robot arm, and estimate calibration parameters including the external parameters of the camera using the first coordinate values ​​and second coordinate values ​​of the multiple feature points in the multiple postures, A system that executes this.

6. A computer program that causes a processor to perform camera calibration processing for a robot, (a) Using a pre-trained first machine learning model, a process is performed to estimate the pixel coordinate values ​​of a plurality of feature points pre-set on the robot arm from an image of the robot arm taken by the camera, (b) A process of estimating the first coordinate values ​​of the plurality of feature points in the 3D camera coordinate system from the pixel coordinate values ​​of the plurality of feature points using a trained second machine learning model, (c) A process to calculate the second coordinate values ​​of the plurality of feature points in the three-dimensional robot coordinate system using the encoder values ​​of the robot arm, (d) A process to perform the above processes (a) to (c) for multiple postures of the robot arm, and estimate calibration parameters including the external parameters of the camera using the first coordinate values ​​and second coordinate values ​​of the multiple feature points in the multiple postures, A computer program that causes the aforementioned processor to execute.

Images