Calibration method, system and storage medium for a camera of a robot

Abstract

Disclosed are a camera calibration method and system for a robot, and a storage medium, which can calibrate a camera without pasting a marker on a robot arm. The method includes: (a) using a first machine learning model to estimate pixel coordinate values of a plurality of feature points pre-set on the robot arm from an image of the robot arm captured by the camera; (b) using a second machine learning model to estimate first coordinate values of the plurality of feature points in a three-dimensional camera coordinate system from the pixel coordinate values of the plurality of feature points; (c) using an encoder value of the robot arm to calculate second coordinate values of the plurality of feature points in a three-dimensional robot coordinate system; and (d) performing steps (a) to (c) for a plurality of poses of the robot arm, and using the first coordinate values and the second coordinate values of the plurality of feature points in the plurality of poses to estimate calibration parameters containing external parameters of the camera.

Description

Technical Field

[0001] This invention relates to a method, system, and computer program for calibrating cameras for robots. Background Technology

[0002] When using cameras in robot operations, the camera's calibration parameters are set beforehand through correction (calibration). Calibration parameters include internal parameters representing lens performance or the relationship between the lens and pixels, and external parameters representing the relative position of the camera to external devices. Camera calibration is typically performed using specialized calibration plates printed or machined with patterns such as dots or checkerboards. This routine calibration process requires specialized equipment and is very time-consuming. Furthermore, calibration is required whenever the camera's positional relationship with the robot changes, becoming a major reason for reduced ease of use of the robot.

[0003] To eliminate this reduction in ease of use, Patent Document 1 proposes a method of attaching arm markers to the robotic arm so that known positions on the robotic arm correspond to the arm markers.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2017-124448

[0005] However, the aforementioned existing technologies require affixing markings to the robotic arm, which presents problems such as cumbersome operations. Therefore, a technology is desired that can calibrate cameras without affixing markings to the robotic arm. Summary of the Invention

[0006] According to a first aspect of the present invention, a method for calibrating a camera for a robot is provided. The method includes: (a) estimating pixel coordinate values ​​of a plurality of pre-defined feature points on the robot arm using a learned first machine learning model and based on an image of the robot arm captured by the camera; (b) estimating first coordinate values ​​of the plurality of feature points in a three-dimensional camera coordinate system using a learned second machine learning model and based on the pixel coordinate values ​​of the plurality of feature points; (c) calculating second coordinate values ​​of the plurality of feature points in a three-dimensional robot coordinate system using encoder values ​​of the robot arm; and (d) performing steps (a) to (c) for multiple poses of the robot arm, using the first and second coordinate values ​​of the plurality of feature points in the multiple poses to estimate calibration parameters including external parameters of the camera.

[0007] According to a second aspect of the present invention, a system for performing correction processing for a camera of a robot is provided. The system includes: a camera capable of capturing images of a robotic arm of the robot; and a correction processing unit that performs correction processing on the camera using images captured by the camera. The correction processing unit performs: (a) a process of estimating pixel coordinate values ​​of a plurality of pre-defined feature points on the robotic arm using a learned first machine learning model and based on images of the robotic arm captured by the camera; (b) a process of estimating first coordinate values ​​of the plurality of feature points in a three-dimensional camera coordinate system using a learned second machine learning model and based on the pixel coordinate values ​​of the plurality of feature points; (c) a process of calculating second coordinate values ​​of the plurality of feature points in a three-dimensional robot coordinate system using encoder values ​​of the robotic arm; and (d) a process of performing processes (a) to (c) for multiple poses of the robotic arm, using the first and second coordinate values ​​of the plurality of feature points in the multiple poses to estimate correction parameters including external parameters of the camera.

[0008] According to a third aspect of the present invention, a computer program is provided that enables a processor to perform correction processing for a camera on a robot. The computer program enables the processor to perform: (a) processing to estimate pixel coordinate values ​​of a plurality of pre-defined feature points on the robot arm using a learned first machine learning model and based on images of the robot arm captured by the camera; (b) processing to estimate first coordinate values ​​of the plurality of feature points in a three-dimensional camera coordinate system using a learned second machine learning model and based on the pixel coordinate values ​​of the plurality of feature points; (c) processing to calculate second coordinate values ​​of the plurality of feature points in a three-dimensional robot coordinate system using encoder values ​​of the robot arm; and (d) performing processes (a) to (c) for multiple poses of the robot arm, using the first and second coordinate values ​​of the plurality of feature points in the multiple poses to estimate correction parameters including external parameters of the camera. Attached Figure Description

[0009] Figure 1 This is an explanatory diagram showing the structure of a robot system.

[0010] Figure 2 It is a conceptual diagram representing the relationship between various coordinate systems.

[0011] Figure 3 This is a functional block diagram of an information processing device.

[0012] Figure 4 This is an explanatory diagram illustrating the functions of the first and second machine learning models.

[0013] Figure 5This is a flowchart illustrating the steps of the correction process in the first embodiment.

[0014] Figure 6 This is a flowchart illustrating the steps of the correction process in the second embodiment.

[0015] Figure 7 This is an explanatory diagram showing an example of the constraints on the machine used in step S135.

[0016] Explanation of reference numerals in the attached figures

[0017] 100…robot, 110…robotic arm, 111…first link, 112…second link, 120…end effector, 140…current sensor, 150…encoder, 200…robot controller, 300…information processing device, 310…processor, 320…memory, 330…interface circuit, 340…input device, 350…display device, 400…camera, 610…calibration actuator, 611…first machine learning model, 612…second machine learning model, 614…learning actuator, 616…correction processing unit, 620…robot control actuator. Detailed Implementation

[0018] A. First implementation method:

[0019] Figure 1 This is an explanatory diagram illustrating an example of a robot system according to one embodiment. The robot system includes a robot 100 that functions as a camera, a robot controller 200 that controls the robot 100, an information processing device 300, and a camera 400. The information processing device 300 is, for example, a personal computer. The information processing device 300 sends control commands to the robot controller 200. The information processing device 300 may also be referred to as a "host information processing device."

[0020] The robot 100 includes a robotic arm 110 and an end effector 120. The robotic arm 110 has a first link 111 and a second link 112. The end effector 120 can be implemented as a gripper or suction pad for holding a workpiece.

[0021] The robotic arm 110 and the end effector 120 are connected sequentially via joints J1 to J3. However, as the robot 100, a robot with an arbitrary robotic arm mechanism having multiple joints can be used.

[0022] Camera 400 is used to capture images of the workpiece that becomes the object of operation for robot 100, thereby identifying the position and orientation of the workpiece. Furthermore, camera 400 has a sufficiently large field of view to capture images of robotic arm 110. Camera 400 can be a two-dimensional RGB camera, or it can be an RGBD camera or a monochrome camera. An RGBD camera is a camera that combines an RGB camera and a D-camera (depth camera).

[0023] Figure 2 It is a conceptual diagram representing the relationship between various coordinate systems. Figure 2 The coordinate system depicted in the figure is as follows.

[0024] (1) Robot coordinate system Σr

[0025] The robot coordinate system Σr is an orthogonal three-dimensional coordinate system with the predetermined position of robot 100 as the origin.

[0026] (2) Camera coordinate system Σc

[0027] The camera coordinate system Σc is an orthogonal three-dimensional coordinate system with the predetermined position of the camera 400 as the origin.

[0028] (3) Pixel coordinate system Σp

[0029] The pixel coordinate system Σp is an orthogonal two-dimensional coordinate system of the image captured by camera 400.

[0030] The pixel coordinates (u, v) of the pixel coordinate system Σp and the three-dimensional coordinates (Xc, Yc, Zc) of the camera coordinate system Σc are shown in the following formula, which can be converted using the internal parameters of the camera 400.

[0031] Mathematical formula 1:

[0032]

[0033] Here, Kx and Ky are the distortion coefficients, Ox and Oy are the optical centers, and f is the focal length.

[0034] The three-dimensional coordinate values ​​(Xc, Yc, Zc) of the camera coordinate system Σc and the three-dimensional coordinate values ​​(Xr, Yr, Zr) of the robot coordinate system Σr are shown in the following formula. They can be transformed using the coordinate transformation matrix [R|t]cr, which is represented by the external parameters of the camera 400.

[0035] Mathematical formula 2:

[0036]

[0037] Figure 2The document also describes multiple feature points P1, P2, and P3 set on the robotic arm 110. In this embodiment, feature points P1, P2, and P3 are respectively set at the center positions of the three joints J1, J2, and J3 of the robotic arm 110. Feature points P1 and P2 can be considered to represent the positions of the two endpoints of the first link 111 with a length L1. Similarly, feature points P2 and P3 can be considered to represent the positions of the two endpoints of the second link 112 with a length L2. Furthermore, any number of feature points, more than two, can be set on the robotic arm 110, with three or more feature points being preferred. In addition, the feature points are not limited to the joint positions of the robotic arm 110 and can be set at any position on the robotic arm 110. However, if the feature points are set at the joint positions of the robotic arm 110, the positions of the feature points can be easily calculated based on the encoder values ​​on the multiple joints of the robotic arm 110, which is preferred. Even when the feature point is set outside the joint, the position of the feature point can be calculated based on the encoder value of the robotic arm 110 by pre-setting the relative relationship between the position of the feature point and the joint position.

[0038] Figure 3 This is a block diagram illustrating the functions of the information processing device 300. The information processing device 300 includes a processor 310, a memory 320, and an interface circuit 330. An input device 340 and a display device 350 are connected to the interface circuit 330, as well as a robot controller 200. The robot controller 200 is connected to a camera 400, and also to a current sensor 140 for measuring the motor current of each joint of the robot 100 and an encoder 150 for measuring the displacement of each joint.

[0039] The processor 310 functions as both a calibration execution unit 610 and a robot control execution unit 620. The calibration execution unit 610 performs processing to determine the correction parameters of the camera 400 by performing calibration on the camera 400 of the robot 100. The calibration execution unit 610 includes a first machine learning model 611, a second machine learning model 612, a learning execution unit 614, and a correction processing unit 616. The robot control execution unit 620 performs the following processing: identifying workpieces based on images of the work area captured by the camera 400, and instructing the robot 100 to perform operations using the identified workpieces. The functions of the calibration execution unit 610 are implemented by the processor 310 executing computer programs stored in the memory 320. However, some or all of the functions of the calibration execution unit 610 can also be implemented using hardware circuitry.

[0040] The memory 320 stores the learning data LD, robot attribute data RD, calibration parameters CP, and robot control program RP used in the learning of two machine learning models 611 and 612. The robot attribute data RD represents attributes such as the mechanical structure or range of motion of the robot 100. The calibration parameters CP include the aforementioned internal and external parameters. The robot control program RP consists of multiple commands that cause the robot 100 to move.

[0041] Figure 4 This is an explanatory diagram illustrating the functions of the first machine learning model 611 and the second machine learning model 612. The first machine learning model 611 takes the image IM(u, v) of the robotic arm 110 captured by the camera 400 as input and estimates the pixel coordinate values ​​Pj(u, v) of multiple feature points Pj. j is the ordinal number that distinguishes the multiple feature points P1 to P3. Figure 4 The image IMp(u,v) shown is a virtual image depicting multiple feature points Pj on the original image IM(u,v). This image IMp(u,v) can be created by the calibration processing unit 616 and displayed on the display device 350 during calibration, but it is also possible not to create the image IMp(u,v).

[0042] The second machine learning model 612 takes the pixel coordinates Pj(u, v) of multiple feature points Pj as input and estimates the three-dimensional coordinates Pj(Xc, Yc, Zc) of the feature points Pj in the camera coordinate system Σc. These three-dimensional coordinates Pj(Xc, Yc, Zc) are used by the correction processing unit 616 to estimate the correction parameters of the camera 400.

[0043] As the first machine learning model 611, various neural networks can be used to infer the feature points constructed within the image, such as any of the following.

[0044] (1a) DeeplabCUT (http: / / www.mackenziemathislab.org / deeplabcut) (1b) DeepPose (https: / / arxiv.org / abs / 1312.4659) The learning data of the first machine learning model 611 is preferably set as teaching data containing images of the robotic arm 110 captured by the camera 400 and the pixel coordinate values ​​Pj(u,v) of multiple feature points Pj.

[0045] As a second machine learning model 612, various neural networks of inference SfM (Structure from Motion) or NRSfM (Non-Rigid Structure from Motion) can be used, for example, any of the following.

[0046] (2a) C3DPO (https: / / arxiv.org / abs / 1909.02533)(2b) RepNet (https: / / sites.google.com / view / repnet) The learning data of the second machine learning model 612 is preferably set as teaching data containing the pixel coordinate values ​​Pj(u,v) of multiple feature points Pj obtained from the image of the robotic arm 110 and the three-dimensional coordinate values ​​Pj(Xc,Yc,Zc) of multiple feature points Pj.

[0047] Figure 5 This is a flowchart illustrating the steps of the correction process in the first embodiment. Here, it is assumed that the two machine learning models 611 and 612 are fully learned machine learning models. Furthermore, it is assumed that at the point in time before the correction process, at least the extrinsic parameters of the camera 400 are unknown. The intrinsic parameters of the camera 400 may be known or unknown. That is, Figure 5 The correction process is a process that presupposes correction parameters that include the external parameters of the camera 400.

[0048] In step S110, the correction processing unit 616 uses the camera 400 to capture an image of the robotic arm 110 and generates an image IM. In step S120, the correction processing unit 616 uses the first machine learning model 611 and estimates the pixel coordinate values ​​Pj(u, v) of multiple feature points Pj based on the image IM of the robotic arm 110. In step S130, the correction processing unit 616 uses the second machine learning model 612 and estimates the first coordinate value Pj(Xc, Yc, Zc) of the feature point Pj in the camera coordinate system Σc based on the pixel coordinate values ​​Pj(u, v) of the feature point Pj. Furthermore, since the internal parameters of the camera 400 are known before the correction processing, in order to calculate the coordinate values ​​(Xc, Yc, Zc) of the camera coordinate system Σc based on the pixel coordinate values ​​(u, v) using the above equation (1), the Z coordinate value Zc is required. Therefore, in order to obtain the Z coordinate value Zc, the second machine learning model 612 is used in step S130.

[0049] In step S140, the correction processing unit 616 uses the encoder values ​​of the robot 100 to calculate the second coordinate value Pj(Xr, Yr, Zr) of the feature point Pj in the robot coordinate system Σr. This calculation process is a calculation based on forward kinematics using the encoder values ​​of joints J1 and J2 and the lengths L1 and L2 of links 111 and 112.

[0050] In step S150, the correction processing unit 616 uses the first coordinate value Pj(Xc, Yc, Zc) and the second coordinate value Pj(Xr, Yr, Zr) of the feature point Pj to estimate the correction parameters of the camera 400. In this embodiment, a Kalman filter is used in this estimation process. If a Kalman filter is used, parameter estimation can be performed sequentially even if the positional relationship between the camera 400 and the robot 100 changes.

[0051] Generally, a Kalman filter consists of two processes: time update and observation update. It minimizes the estimation error by repeating these two processes. When estimating the calibration parameters of camera 400, there is no control input and no time variation; therefore, time update can be disregarded. That is, if we define a state vector, observation matrix, and observation vector for observation update, we can estimate the calibration parameters of camera 400. For example, in the case of estimating only the external parameters, the state vector x, observation matrix H, and observation vector Z are as follows.

[0052] Mathematical formula 3:

[0053] x = [r 11 r 12 r 12 r 21 r 22 r 23 r 31 r 32 r 33 t x t y t z ] T (3a)

[0054]

[0055] Z = [X] c Y c Z c ] T (3c)

[0056] The state vector x is an external parameter, the observation matrix H is the 3D coordinates of feature point Pj in the robot coordinate system ∑r, and the observation vector Z is the 3D coordinates of feature point Pj in the camera coordinate system ∑c. Given that the external parameters constituting the state vector x are correctly estimated, the observation vector Z becomes the result of multiplying the observation matrix H by the state vector x.

[0057] Here, the method of estimating only the extrinsic parameters has been described, but the estimation of intrinsic parameters can also be performed in the same way. When estimating intrinsic parameters together with extrinsic parameters, the intrinsic parameters are added to the state vector x, and the observation matrix is ​​changed accordingly. Furthermore, not limited to the Kalman filter, other methods such as successive least squares or particle filters can be used to estimate the correction parameters.

[0058] In step S160, the correction processing unit 616 determines whether the estimation process of the correction parameters has ended. For example, when the processing in steps S110 to S150 is set to one iteration, it can be determined that the estimation process of the correction parameters has ended after a predetermined number of iterations have been performed. Alternatively, it can be determined that the estimation process of the correction parameters has ended when the difference between the estimated value in the previous iteration and the current iteration or the difference in the error covariance is below a threshold.

[0059] If the estimation of the correction parameters is not completed, the process returns to step S110 and repeats steps S110 to S150. Furthermore, in multiple iterations, the robot arm 110's posture is set in step S110 to allow it to assume different poses. That is, for each new posture of the robot arm 110, whenever the camera 400 captures an image IM of the robot arm 110, steps S120 to S150 are executed to update the observations using a Kalman filter, thereby estimating the correction parameters. As a result, for multiple different postures of the robot arm 110, the correspondence between the first coordinate value Pj(Xc, Yc, Zc) of the camera coordinate system Σc and the second coordinate value Pj(Xr, Yr, Zr) of the robot coordinate system Σr is obtained, thereby correctly estimating the correction parameters of the camera 400.

[0060] When the estimation process of the correction parameters is completed, the estimated correction parameters CP are stored in memory 320, thus completing the process. Figure 5 In the case of a fixed robot, the positional relationship between the camera 400 and the robot 100 rarely changes significantly. Therefore, by saving the estimated information, including the error covariance of the Kalman filter, along with the estimated correction parameter CP, and reusing it as the initial value in the next correction process, the convergence of the estimated value can be accelerated.

[0061] The above Figure 5 The calibration process can also be performed in parallel with the actual operation using the robot 100. In this way, the calibration process of the camera 400 can be performed while the robot 100 is operating on the workpiece, so no special processing time is required for the calibration process. Furthermore, when the calibration process is performed in parallel with the actual operation, it is preferable to take an image of the robot arm 110 while the robot arm 110 is stopped in step S110.

[0062] As described above, in the first embodiment, for a plurality of feature points Pj pre-set on the robotic arm 110, the first coordinate values ​​Pj(Xc, Yc, Zc) of the camera coordinate system Σc are estimated based on the image IM captured by the camera 400. Furthermore, the second coordinate values ​​Pj(Xr, Yr, Zr) of the robot coordinate system Σr are calculated based on the encoder values ​​of the robotic arm 110. The calibration parameters CP of the camera 400 are then estimated based on these three-dimensional coordinate values. Therefore, the calibration parameters CP of the camera 400 can be estimated without using a calibration plate. Additionally, camera calibration can be performed without affixing markings to the robotic arm 110.

[0063] B. Second implementation method:

[0064] Figure 6 This is a flowchart illustrating the steps of the correction process in the second embodiment. The only difference from the first embodiment is the addition of step S135 between steps S130 and S140; the other steps are the same. Figure 5 The correction process is the same as in the first embodiment shown. Furthermore, the device structure is also the same as in the first embodiment.

[0065] In step S135, the correction processing unit 616 inputs the estimated value of the first coordinate value Pj(Xc, Yc, Zc) of the feature point Pj obtained in step S130 into the Kalman filter, and adds constraints to the observation, thereby probabilistically updating the estimated value of the first coordinate value of the feature point Pj. As constraints, mechanical or mechanism constraints related to the robotic arm 110 can be used. That is, in step S135, the estimated value of the first coordinate value Pj(Xc, Yc, Zc) of the feature point Pj is updated by performing an observation update using a Kalman filter that includes mechanical constraints related to the robotic arm 110.

[0066] Figure 7 This is an explanatory diagram illustrating an example of the constraint used in step S135. In this example, the length L2 of the second link 112 is used as the constraint. In this case, the state vector x, observation matrix H, and observation vector Z in the Kalman filter used in step S135 are as follows.

[0067] Mathematical formula 4:

[0068]

[0069]

[0070]

[0071] Z = r = L2(4d)

[0072] Here, the elements of the state vector x are the estimated values ​​of the correct first coordinates of the feature point Pj in the camera coordinate system Σc, and the elements of the observation matrix H are the estimated values ​​of the first coordinates of the feature point Pj in the camera coordinate system Σc obtained in step S130.

[0073] If an observation update is performed using a Kalman filter that includes such mechanical constraints, the estimated value of the first coordinate of feature point Pj can be updated in a manner that satisfies the constraints given by equation (4) above.

[0074] Another example of a mechanical constraint is the constraint that multiple feature points Pj exist on the same plane. For instance, if we consider three feature points P1 to P3 existing on the same plane, and imagine a three-dimensional vector represented by the three-dimensional coordinates of each feature point Pj, then the volume of the cube formed by the three-dimensional vectors is zero. This is synonymous with the fact that the determinant of three arranged three-dimensional column vectors is zero. The state vector x, observation matrix H, and observation vector Z in this case of the Kalman filter are as follows.

[0075] Mathematical formula 5:

[0076]

[0077]

[0078]

[0079] Z = D = 0 (5d)

[0080] In addition to the examples mentioned above, other constraints that can be applied as mechanical constraints include multiple link lengths or three or more feature points Pj arranged in the same straight line when multiple links extend in a straight line. Furthermore, applying multiple constraints simultaneously allows for updating the first coordinate values ​​of multiple feature points Pj to more accurate values.

[0081] By using the aforementioned mechanical constraints, the first coordinate value of the feature point Pj that conforms to the shape of the robotic arm 110 can be updated, and deviations or jumps in the three-dimensional position of the feature point Pj can be suppressed, thus enabling stable estimation of correction parameters.

[0082] The second embodiment also has the same effects as the first embodiment. Furthermore, in the second embodiment, since a Kalman filter incorporating mechanical constraints is used to update the estimated values ​​of the first coordinates of multiple feature points Pj in the camera coordinate system Σc, the estimation accuracy of the first coordinates can be improved.

[0083] Other methods:

[0084] This invention is not limited to the embodiments described above, and can be implemented in various forms without departing from the spirit of the invention. For example, the invention can be implemented in the following aspects. To solve part or all of the problems of the invention, or to achieve part or all of the effects of the invention, the technical features in the above embodiments corresponding to the technical features in the various aspects described below can be appropriately replaced or combined. In addition, if a technical feature is not required to be described in this specification, it can be appropriately omitted.

[0085] (1) According to a first aspect of the present invention, a method for calibrating a camera for a robot is provided. The method includes: (a) estimating pixel coordinate values ​​of a plurality of pre-defined feature points on the robot arm using a learned first machine learning model and based on an image of the robot arm captured by the camera; (b) estimating first coordinate values ​​of the plurality of feature points in a three-dimensional camera coordinate system using a learned second machine learning model and based on the pixel coordinate values ​​of the plurality of feature points; (c) calculating second coordinate values ​​of the plurality of feature points in a three-dimensional robot coordinate system using encoder values ​​of the robot arm; and (d) performing steps (a) to (c) for a plurality of poses of the robot arm, using the first and second coordinate values ​​of the plurality of feature points in the plurality of poses to estimate calibration parameters including external parameters of the camera.

[0086] According to this method, for multiple feature points pre-set on the robotic arm, the first coordinate value of the three-dimensional camera coordinate system is estimated based on images captured by the camera. Furthermore, the second coordinate value of the three-dimensional robot coordinate system is calculated based on the encoder values ​​of the robotic arm. The camera's calibration parameters are then estimated based on these three-dimensional coordinate values, thus enabling the estimation of camera calibration parameters without the use of a calibration plate. Additionally, camera calibration can be performed without affixing markings to the robotic arm.

[0087] (2) In the above method, the plurality of feature points can also be configured such that the plurality of feature points are set at the center position of each of the plurality of joints of the robotic arm.

[0088] According to this method, the first coordinate values ​​of multiple feature points can be easily estimated using a first machine learning model, and the second coordinate values ​​of the robot coordinate system can be easily calculated.

[0089] (3) In the above method, the step (d) may also be configured such that: whenever an image of the robotic arm is captured by the camera for a new posture of the robotic arm, the correction parameters are estimated by performing an observation update using a Kalman filter.

[0090] According to this method, since the correction parameters are estimated using a Kalman filter, the accuracy of the correction parameters can be improved whenever a new posture of the robotic arm is captured by a camera.

[0091] (4) In the above method, the step (b) may also be configured to include the following steps: for the first coordinate value obtained by the second machine learning model, an observation update using a Kalman filter containing mechanical constraints related to the robotic arm is performed, thereby updating the estimated value of the first coordinate value of the plurality of feature points.

[0092] According to this method, the estimation accuracy of the first coordinate value can be improved by using a Kalman filter containing mechanical constraints to update the estimated values ​​of the first coordinate values ​​of multiple feature points in the 3D camera coordinate system.

[0093] (5) According to a second aspect of the present invention, a system for performing correction processing for a camera of a robot is provided. The system includes: a camera capable of capturing images of a robotic arm of the robot; and a correction processing unit that performs correction processing on the camera using images captured by the camera. The correction processing unit performs: (a) processing to estimate pixel coordinate values ​​of a plurality of pre-defined feature points on the robotic arm using a learned first machine learning model and based on images of the robotic arm captured by the camera; (b) processing to estimate first coordinate values ​​of the plurality of feature points in a three-dimensional camera coordinate system using a learned second machine learning model and based on the pixel coordinate values ​​of the plurality of feature points; (c) processing to calculate second coordinate values ​​of the plurality of feature points in a three-dimensional robot coordinate system using encoder values ​​of the robotic arm; and (d) processing (a) to (c) for a plurality of poses of the robotic arm, and using the first and second coordinate values ​​of the plurality of feature points in the plurality of poses to estimate correction parameters including external parameters of the camera.

[0094] (6) According to a third aspect of the present invention, a computer program is provided that enables a processor to perform correction processing for a camera of a robot. The computer program enables the processor to perform: (a) a process of estimating pixel coordinate values ​​of a plurality of pre-defined feature points on the robot arm using a learned first machine learning model and based on an image of the robot arm captured by the camera; (b) a process of estimating first coordinate values ​​of the plurality of feature points in a three-dimensional camera coordinate system using a learned second machine learning model and based on the pixel coordinate values ​​of the plurality of feature points; (c) a process of calculating second coordinate values ​​of the plurality of feature points in a three-dimensional robot coordinate system using encoder values ​​of the robot arm; and (d) a process of performing the processes (a) to (c) for multiple poses of the robot arm, using the first and second coordinate values ​​of the plurality of feature points in the multiple poses to estimate correction parameters including external parameters of the camera.

[0095] This invention can be implemented in various ways other than those described above. For example, it can be implemented as a robot system having a robot and a robot information processing device, a computer program for implementing the functions of the robot information processing device, and a non-transitory storage medium on which the computer program is recorded.

Claims

1. A calibration method, characterized in that, This is a calibration method for cameras used on robots. The correction method includes: (a) The process of using a first machine learning model that has been learned and estimating the pixel coordinates of a plurality of pre-defined feature points on the robotic arm based on images of the robotic arm captured by the camera; (b) The step of using the learned second machine learning model and estimating the first coordinate values ​​of the plurality of feature points in the three-dimensional camera coordinate system based on the pixel coordinate values ​​of the plurality of feature points; (c) The step of using the encoder values ​​of the robotic arm to calculate the second coordinate values ​​of the plurality of feature points in the three-dimensional robot coordinate system; and (d) Performing steps (a) to (c) for multiple postures of the robotic arm, using the first coordinate value and the second coordinate value of the multiple feature points under the multiple postures to estimate the correction parameters including the external parameters of the camera.

2. The correction method according to claim 1, characterized in that, The plurality of feature points are set at the center positions of the plurality of joints of the robotic arm.

3. The correction method according to claim 1, characterized in that, The step (d) includes the following steps: whenever an image of the robotic arm is captured by the camera for a new posture of the robotic arm, the correction parameters are estimated by performing an observation update using a Kalman filter.

4. The correction method according to claim 1, characterized in that, The step (b) includes the following step: for the first coordinate value obtained by the second machine learning model, an observation update using a Kalman filter containing mechanical constraints related to the robotic arm is performed, thereby updating the estimated value of the first coordinate value of the plurality of feature points.

5. A correction system characterized by, It is a system that performs camera calibration processing for robots. The correction system includes: A camera capable of filming the robot's robotic arm; and The correction processing unit uses the images captured by the camera to perform correction processing on the camera. The correction processing unit performs: (a) Processing of estimating the pixel coordinates of a plurality of pre-defined feature points on the robotic arm using a first machine learning model that has been learned and based on images of the robotic arm captured by the camera; (b) Processing of using the learned second machine learning model and estimating the first coordinate values ​​of the plurality of feature points in the three-dimensional camera coordinate system based on the pixel coordinate values ​​of the plurality of feature points; (c) Processing of using the encoder values ​​of the robotic arm to calculate the second coordinate values ​​of the plurality of feature points in the three-dimensional robot coordinate system; as well as (d) Performing the processes (a) to (c) for multiple poses of the robotic arm, using the first coordinate value and the second coordinate value of the multiple feature points under the multiple poses to estimate the correction parameters including the external parameters of the camera.

6. A storage medium, characterized by The storage enables the processor to execute computer programs for camera correction processing for the robot. The computer program causes the processor to execute: (a) Processing of estimating the pixel coordinates of a plurality of pre-defined feature points on the robotic arm using a first machine learning model that has been learned and based on images of the robotic arm captured by the camera; (b) Processing of using the learned second machine learning model and estimating the first coordinate values ​​of the plurality of feature points in the three-dimensional camera coordinate system based on the pixel coordinate values ​​of the plurality of feature points; (c) Processing of using the encoder values ​​of the robotic arm to calculate the second coordinate values ​​of the plurality of feature points in the three-dimensional robot coordinate system; as well as (d) Performing the processes (a) to (c) for multiple poses of the robotic arm, using the first coordinate value and the second coordinate value of the multiple feature points under the multiple poses to estimate the correction parameters including the external parameters of the camera.

Images