A method and system for simultaneous eye-in-hand and eye-in-hand-off camera calibration

Abstract

The application provides a method and system for simultaneously performing eye-in-hand and eye-in-hand-out camera calibration. The method comprises the following steps: S1, for a fixed calibration board, simultaneously acquiring a picture set P1, a picture set P2, P1 comprising n pictures of the fixed calibration board taken by an eye-in-hand camera, P2 comprising n pictures of the fixed calibration board taken by an eye-in-hand-out camera, and a transformation matrix of a mechanical arm end to a mechanical arm base; S2, calibrating the eye-in-hand camera through P1 to obtain n groups of S3, obtaining n groups of through n pairs of pictures in P1 and P2; S4, obtaining n groups of according to n groups of and n groups of ; and S5, performing first processing on n groups of to obtain a transformation relationship of the eye-in-hand-out camera to the mechanical arm base. The scheme of the application simplifies the calibration process and improves the calibration efficiency.

Description

Technical Field

[0001] This application belongs to the field of image measurement technology, and in particular relates to a method and system for simultaneously calibrating an eye on the hand and an eye on an external camera. Background Technology

[0002] In image measurement and machine vision applications, to determine the 3D geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters are the camera parameters. Therefore, the purpose of camera calibration is to determine the camera's intrinsic and extrinsic parameters, as well as distortion parameters. In robot vision applications, hand-eye calibration is a very fundamental and crucial problem. Simply put, the purpose of hand-eye calibration is to obtain the relationship between the robot coordinate system and the camera coordinate system, and finally transfer the visual recognition results to the robot coordinate system. Hand-eye calibration has two forms. Depending on how the camera is fixed, if the camera is fixed to the robot's end effector, it is called "eye in hand"; if the camera is fixed to a base outside the robot, it is called "eye to hand," or global camera.

[0003] With the camera in the hand, the relationship between the robot base and the calibration plate remains unchanged during two movements. The solution is the pose relationship between the camera and the robot's end effector coordinate system, such as... Figure 1 As shown.

[0004] During calibration, the calibration board needs to be placed in a fixed position relative to the robot base and kept stationary. Move the robotic arm's end effector to take n images of the calibration board from different angles. The following relationship applies to each image:

[0005]

[0006] in, The extrinsic parameters for camera calibration can be directly obtained from the images of the calibration board. It can be obtained from the pose parameters of the robotic arm's end effector. Since the calibration plate is fixed in one position throughout the process, the transformation matrix is ​​the same for each set of images. According to equation (1), the following relationship exists for each image:

[0007]

[0008] With the eye outside the hand, the pose relationship between the robot's end effector and the calibration plate remains unchanged during two movements. The quantity to be solved is the pose relationship between the camera and the robot's base coordinate system, such as... Figure 2 As shown.

[0009] During the calibration process, the calibration plate needs to be fixed to the end of the robotic arm, and n (more than 3) images of the calibration plate are taken using a camera under different robotic arm postures. For each image, the following relationship applies:

[0010]

[0011] in, The extrinsic parameters for camera calibration can be directly obtained from the images of the calibration board. It can be obtained from the pose parameters of the robotic arm's end effector. Since the calibration plate is fixed at the end of the robotic arm, the transformation matrix is ​​the same for each set of images. According to equation (3), the following relationship exists for each image:

[0012]

[0013] For the n images taken during the calibration process, the case where the eye is on the hand has the following system of equations:

[0014]

[0015] For the n images taken during the calibration process, the following system of equations applies to the case where the eye is outside the hand:

[0016]

[0017] Whether the eye is outside the hand or inside the hand, a classic system of equations can be obtained: AX = XB. This system of equations contains n-1 equations, where n is the number of images taken during the calibration process. X is the transformation matrix that needs to be obtained.

[0018] In scenarios where both eye-to-hand and eye-to-eye camera calibration are required, the aforementioned method is often used to perform the calibration separately. However, when calibrating the eye-to-eye camera in the robotic arm, the calibration plate needs to be fixed to the end of the robotic arm. For calibration plates that are large or heavy, separate tooling needs to be designed and specific tools made to fix them to the end of the robotic arm, which is inconvenient and costly.

[0019] Chinese patent CN113635311B discloses a method for decomposing an eye-on-hand external calibration problem into an eye-on-hand calibration problem and a binocular calibration problem. First, a calibration plate is fixed in a certain position to complete the eye-on-hand camera calibration. Then, a robotic arm is fixed, forming a binocular camera system with the eye-on-hand external camera for binocular camera calibration. This method partially solves the problem of simultaneous calibration, but it has the following disadvantages:

[0020] a) To achieve high accuracy during dual-target calibration, the position and orientation of the calibration plate need to be constantly changed. Therefore, this patent does not keep the calibration plate fixed throughout the entire calibration process.

[0021] b) To achieve high accuracy in binocular calibration, the distance between the two cameras in the left and right fields of view of the binocular camera system cannot be too far. Otherwise, the accuracy will inevitably be very low, or even effective calibration will not be possible. Therefore, this patent requires that the distance between the cameras with the eye on the hand and the camera with the eye on the hand be relatively close in binocular calibration, which greatly limits the scope of practical application scenarios. If the distance between the two cameras is too far, effective high-precision binocular calibration cannot be achieved.

[0022] c) Eye-on-the-hand camera calibration must be performed first, followed by eye-on-the-hand external camera calibration. This cannot be done simultaneously, and the calibration cycle is relatively long. Summary of the Invention

[0023] The main objective of this invention is to provide a method and system for simultaneously calibrating eye-on-hand and eye-on-external cameras, in order to solve the problems in the prior art where simultaneously calibrating eye-on-external and eye-on-hand cameras is complex to operate and cannot simultaneously meet the requirements of application scenarios and accuracy.

[0024] To address the aforementioned problems, this invention provides a method for simultaneously performing eye-on-hand and eye-on-hand external camera calibration, the method comprising:

[0025] Step S1: For a fixed calibration board, simultaneously acquire image set P1, image set P2, and... P1 includes n photographs taken by the hand-held camera of the eye against a fixed calibration plate, and P2 includes n photographs taken by the hand-held external camera against the fixed calibration plate. Let n be the transformation matrix from the end effector of the robotic arm to the base of the robotic arm, where n is an integer greater than 1;

[0026] Step S2, through P1 and Calibrate the eye using a handheld camera and obtain n sets of data. The relationship between the robotic arm's end effector and the eye on the hand and the camera's pose transformation;

[0027] Step S3: Obtain n sets of images from the n pairs of images in the P1 and P2 image sets. The pose transformation relationship between the camera with the eye on the hand and the camera with the eye on the hand;

[0028] Step S4, according to n groups and n groups Obtain n groups The transformation matrix from the eye-to-hand external camera to the robotic arm base;

[0029] Step S5, for n groups The first processing step yields the transformation relationship between the eye and the external camera on the hand, and the robotic arm base.

[0030] Furthermore, step S2 above includes:

[0031] Step S21: Obtain the n photos from P1.

[0032] Step S22, through and We obtain n-1 AX = XB equations, where

[0033]

[0034] Step S23: Obtain n sets of equations based on AX = XB.

[0035] Furthermore, step S3 above includes:

[0036] Using n pairs of images from image sets P1 and P2, the Perspective-n-point method is used to determine n pose transformation relationships from the eye-outside-hand camera to the eye-outside-hand camera.

[0037] Furthermore, step S4 above includes:

[0038] n groups and n groups Multiply them one by one to get n sets.

[0039] Furthermore, the first processing step in step S5 above is an averaging process.

[0040] According to another aspect of this application, a system is provided for simultaneously calibrating an eye on the hand and an eye on the hand external camera. The system includes a fixed calibration plate, a robotic arm, a computer, an eye on the hand camera and an eye on the hand external camera. The eye on the hand camera is fixed to the end of the robotic arm, and the calibration plate is within the field of view of the eye on the hand camera and the eye on the hand external camera.

[0041] At the same time, n photos are taken by the hand-held camera towards the fixed calibration plate, resulting in P1; n photos are taken by the hand-held external camera towards the fixed calibration plate, resulting in P2. The computer calculates... Let n be the transformation matrix from the end effector of the robotic arm to the base of the robotic arm, where n is an integer greater than 1;

[0042] The computer uses P1 and Calibrate the eye using a handheld camera and obtain n sets of data. The relationship between the robotic arm's end effector and the eye on the hand and the camera's pose transformation;

[0043] The computer obtains n sets of images from n pairs of images in the image sets P1 and P2. The pose transformation relationship between the camera with the eye on the hand and the camera with the eye on the hand;

[0044] The computer uses n groups and n groups Obtain n groups The transformation matrix from the eye-to-hand external camera to the robotic arm base;

[0045] Computer on n groups The first processing step yields the transformation relationship between the eye and the external camera on the hand, and the robotic arm base.

[0046] Furthermore, the aforementioned computer obtains from the n photos in P1.

[0047] Computer through and We obtain n-1 AX = XB equations, where

[0048]

[0049] The computer obtains n sets of equations based on the equation AX = XB.

[0050] Furthermore, the computer uses the Perspective-n-point method to determine the pose transformation relationships from the eye-outside-hand camera to the eye-outside-hand camera using n pairs of images in the P1 and P2 image sets.

[0051] Furthermore, the aforementioned computer will group n... and n groups Multiply them one by one to get n sets.

[0052] Furthermore, the first treatment described above is an averaging treatment.

[0053] The solution of this invention achieves the following effects.

[0054] a) No binocular camera calibration is required, therefore the distance requirements for the eye-on-hand camera and the eye-on-outside camera are more lenient;

[0055] b) The calibration plate does not need to be moved during the entire process; only the position of the robotic arm's end effector needs to be changed, which is quite convenient.

[0056] c) The eye-on-hand camera calibration and the eye-on-hand external camera calibration simultaneously acquire images and are performed at the same time, which is relatively fast and allows for averaging of multiple sets of photos, greatly ensuring accuracy. Attached Figure Description

[0057] The following description, in conjunction with the accompanying drawings, will further illustrate the above-mentioned features, technical characteristics, advantages, and implementation methods of this application in a clear and understandable manner. The accompanying drawings are for illustrative and explanatory purposes only and do not limit the scope of this application. Wherein:

[0058] Figure 1 A schematic diagram of a handheld camera calibration system in the prior art is shown;

[0059] Figure 2 A schematic diagram of an eye-in-hand external camera calibration system in the prior art is shown;

[0060] Figure 3 This invention illustrates a flowchart of a method for simultaneously calibrating the eye on the hand and the eye on the hand external camera in one embodiment of the present application.

[0061] Figure 4 This illustration shows a schematic diagram of a system that simultaneously performs eye-on-hand and eye-on-hand external camera calibration in one embodiment of this application. Detailed Implementation

[0062] To provide a clearer understanding of the technical features, objectives, and effects of this application, specific embodiments of this application will now be described with reference to the accompanying drawings.

[0063] As described in the background section, existing technologies suffer from the problem that simultaneously calibrating both eye-on-hand and eye-on-external calibration cameras is complex to operate, and the application scope and accuracy cannot be simultaneously considered. To solve the above problems, according to a specific embodiment of this application, such as... Figure 3 As shown, a method for simultaneously performing eye-on-hand and eye-on-hand external camera calibration is provided. The method includes: Step S1, for a fixed calibration board, simultaneously acquiring image set P1, image set P2, and... P1 includes n photographs taken by the hand-held camera of the eye against a fixed calibration plate, and P2 includes n photographs taken by the hand-held external camera against the fixed calibration plate. Let n be the transformation matrix from the end effector of the robotic arm to the base of the robotic arm, where n is an integer greater than 1; Step S2, through P1 and Calibrate the eye using a handheld camera and obtain n sets of data. The pose transformation relationship from the robotic arm's end effector to the eye on the hand camera; Step S3, obtain n sets of images from n pairs of images in the P1 and P2 image sets. The pose transformation relationship from eye-on-hand external camera to eye-on-hand camera; Step S4, based on n sets and n groups Obtain n groups The transformation matrix from the eye to the external camera and then to the robotic arm base; Step S5, for n groups The first processing step yields the transformation relationship between the eye and the external camera on the hand, and the robotic arm base.

[0064] The present invention proposes a hand-eye camera calibration system that simultaneously performs eye-on-hand and eye-off-hand calibration using the calibration method described in this application. During the calibration process, the tooling required to fix the calibration plate to the end of the robotic arm is eliminated, and the position of the calibration plate does not need to be moved. While ensuring calibration accuracy, the system simplifies the calibration process, improves calibration efficiency, and saves calibration costs.

[0065] Specifically, a single calibration plate is used to conveniently and quickly calibrate two types of hand-eye cameras simultaneously. Images from both cameras are acquired synchronously as input. The hand-eye camera requires changes in its viewing angle based on the robotic arm's posture, while the external hand-eye camera simply needs to be fixed in a specific position. Furthermore, the calibration plate does not need to be fixed to the end of the robotic arm; it only needs to be placed in a fixed position within the field of view of all cameras. By calibrating the hand-eye camera on the hand using traditional methods and simultaneously and indirectly calibrating the external hand-eye camera, and by processing multiple images, calibration accuracy is improved. In addition, this device can connect to multiple hand-eye and external hand-eye cameras for simultaneous multi-camera calibration.

[0066] In one embodiment, step S2 above includes: step S21, obtaining from the n photos in P1 Step S22, through and We obtain n-1 AX = XB equations, where Step S23: Obtain n sets of equations based on AX = XB.

[0067] In one embodiment, step S3 includes: using the Perspective-n-point method to determine the pose transformation relationship between the eye-on-hand camera and the eye-on-hand-on-upper camera from n pairs of images in the P1 and P2 image sets.

[0068] In one embodiment, step S4 above includes: taking n groups and n groups Multiply them one by one to get n sets.

[0069] In one embodiment, in order to improve measurement accuracy, the first processing in step S5 above is an averaging process.

[0070] According to another embodiment of this application, such as Figure 4As shown, a system is provided for simultaneously calibrating an eye-on-hand and an eye-on-external camera. The system includes a fixed calibration plate, a robotic arm, a computer, an eye-on-hand camera, and an eye-on-external camera. The eye-on-hand camera is fixed to the end of the robotic arm, and the calibration plate is within the field of view of both the eye-on camera and the eye-on-external camera. Simultaneously, the eye-on camera takes n photos of the fixed calibration plate, resulting in P1; the eye-on-external camera takes n photos of the fixed calibration plate, resulting in P2; and the computer calculates... Let n be the transformation matrix from the end effector of the robotic arm to its base, where n is an integer greater than 1; the computer uses P1 and... Calibrate the eye using a handheld camera and obtain n sets of data. This describes the pose transformation relationship from the robotic arm's end effector to the camera on the hand; the computer obtains n sets of images from n pairs of images in image sets P1 and P2. This represents the pose transformation relationship from an external camera to an internal camera; the computer uses n sets of... and n groups Obtain n groups The transformation matrix from the eye to the external camera and then to the robotic arm base; the computer performs n sets of transformations. The first processing step yields the transformation relationship between the eye and the external camera on the hand, and the robotic arm base.

[0071] The calibration system described in this application allows for the convenient and rapid simultaneous calibration of two hand-eye cameras using a single calibration board. Images from both cameras are acquired synchronously as input. The hand-eye camera requires changes in viewing angle based on the robotic arm's posture, while the external hand-eye camera simply needs to be fixed in a specific position. Furthermore, the calibration board does not need to be fixed to the end of the robotic arm; it only needs to be placed in a fixed location within the field of view of all cameras. This method calibrates the hand-eye camera using traditional methods and simultaneously calibrates the external hand-eye camera indirectly, improving calibration accuracy by processing multiple images. In addition, this device can connect to multiple hand-eye and external hand-eye cameras for simultaneous multi-camera calibration.

[0072] In one embodiment, the computer obtains the n photos in P1. Computer through and We obtain n-1 AX = XB equations, where The computer obtains n sets of equations based on the equation AX = XB.

[0073] In one embodiment, the computer uses the Perspective-n-point method to determine the pose transformation relationship from the eye-outside-the-hand camera to the eye-outside-the-hands-on-the-upper camera using n pairs of images in the P1 and P2 image sets.

[0074] In one embodiment, the computer will use n groups and n groups Multiply them one by one to get n sets.

[0075] It should be understood that although this specification is described according to various embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0076] The above description is merely an illustrative embodiment of this application and is not intended to limit the scope of this application. Any equivalent changes, modifications, and combinations made by those skilled in the art without departing from the concept and principles of this application shall fall within the scope of protection of this application.

Claims

1. A method for simultaneously performing eye-on-hand and eye-on-hand external camera calibration, wherein, The method includes: Step S1: For a fixed calibration board, simultaneously acquire image set P1, image set P2, and... P1 includes n photographs taken by an eye-on hand camera of the fixed calibration plate, and P2 includes n photographs taken by an eye-on external camera of the fixed calibration plate. Let n be the transformation matrix from the end effector of the robotic arm to the base of the robotic arm, where n is an integer greater than 1; Step S2, through P1 and The handheld camera was calibrated to obtain n sets of data. The The pose transformation relationship between the end effector of the robotic arm and the camera on the hand; Step S3: Obtain n sets of images from the n pairs of images in the P1 and P2 image sets. The The pose transformation relationship between the external eye-on-hand camera and the internal eye-on-hand camera; Step S4, according to the n groups and the n groups , obtain n groups The The transformation matrix from the eye-on-hand external camera to the robotic arm base; Step S5, for the n groups The first process is performed to obtain the transformation relationship between the eye in the hand-mounted external camera and the robotic arm base.

2. The method for simultaneously calibrating the eye on the hand and the eye on the hand external camera according to claim 1, characterized in that, Step S2 includes: Step S21, obtain the n photos in P1. ; Step S22, through the and This yields n-1 equations AX=XB, where A= * X= B= * ; Step S23: Obtain the n sets of equations based on the given AX=XB equation. .

3. The method for simultaneously calibrating the eye on the hand and the eye on the hand external camera according to claim 1, characterized in that, Step S3 includes: Using n pairs of images from the P1 and P2 image sets, the Perspective-n-point method is used to determine n pose transformation relationships from the eye-on-hand camera to the eye-on-hand upper camera. .

4. The method for simultaneously calibrating the eye on the hand and the eye on the hand external camera according to claim 1, characterized in that, Step S4 includes: The n groups and the n groups Multiply them one by one to get n sets. .

5. The method for simultaneously calibrating the eye on the hand and the eye on the hand external camera according to claim 1, characterized in that, The first process in step S5 is to calculate the average.

6. A system for simultaneously performing eye-on-hand and eye-on-hand external camera calibration, characterized in that, The system includes a fixed calibration plate, a robotic arm, a computer, an eye-on-hand camera and an eye-on-external camera. The eye-on-hand camera is fixed to the end of the robotic arm, and the calibration plate is within the field of view of the eye-on-hand camera and the eye-on-external camera. At the same time, the eye takes n photos of the fixed calibration plate with the hand-held camera, resulting in P1; the eye takes n photos of the fixed calibration plate with the hand-held external camera, resulting in P2; and the computer calculates... The Let n be the transformation matrix from the end effector of the robotic arm to the base of the robotic arm, where n is an integer greater than 1; The computer uses P1 and The handheld camera was calibrated to obtain n sets of data. The The pose transformation relationship between the end effector of the robotic arm and the camera on the hand; The computer obtains n sets of images from n pairs of images in the P1 and P2 image sets. The The pose transformation relationship between the external eye-on-hand camera and the internal eye-on-hand camera; The computer uses the n groups and the n groups , obtain n groups The The transformation matrix from the eye-on-hand external camera to the robotic arm base; The computer processes the n groups The first process is performed to obtain the transformation relationship between the eye in the hand-mounted external camera and the robotic arm base.

7. The system for simultaneously calibrating the eye on the hand and the eye on the hand external camera according to claim 6, characterized in that, The computer obtains the n photos in P1. ; The computer via and This yields n-1 equations AX=XB, where A= * X= B= * ; The computer obtains the n sets of equations based on the equation AX=XB. .

8. The system for simultaneously calibrating the eye on the hand and the eye on the hand external camera according to claim 6, characterized in that, The computer uses the Perspective-n-point method to determine the pose transformation relationship between the eye-on-hand camera and the eye-on-hand upper camera from n pairs of images in the P1 and P2 image sets. .

9. The system for simultaneously calibrating the eye on the hand and the eye on the hand external camera according to claim 6, characterized in that, The computer will use the n groups and the n groups Multiply them one by one to get n sets. .

10. The system for simultaneously calibrating the eye on the hand and the eye on the hand external camera according to claim 6, characterized in that, The first process is to calculate the average.

Images