Hand-eye calibration method for rotating mechanism and binocular endoscope in narrow space

Abstract

The invention discloses a hand-eye calibration method for a rotating mechanism and a binocular endoscope in a narrow space. The hand-eye calibration method comprises the following steps: calibrating the binocular endoscope; the target ball is placed on the calibration plate, coordinates of the target ball are measured through the binocular endoscope and the laser tracker respectively, and the position relation between the binocular endoscope and the laser tracker is obtained through rigid body transformation calculation; a target ball is placed on a sleeve, the position where the sleeve is located serves as a rotation starting point, a rotating mechanism rotates at a fixed angle from the rotation starting point, and a laser tracker collects the center coordinates, the fitting rotation plane and the normal vector of the target ball after each rotation is stopped; establishing a local coordinate system of the rotating mechanism, and calculating a position relation between the laser tracker and the rotating mechanism through rigid body transformation; and by combining the position relationship between the binocular endoscope and the laser tracker, the unification of a global coordinate system is realized. According to the invention, hand-eye calibration of the rotating mechanism and the binocular endoscope in the narrow space can be realized, and a new way for global calibration of measurement of the rotating mechanism in the narrow space is provided.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular is a hand-eye calibration method for a rotating mechanism and a binocular endoscope in a narrow space, which enables the rotating mechanism to perform global calibration by rotating itself in a narrow space. Background Art

[0002] The rotary mechanism measurement platform usually consists of a rotary mechanism and a measuring instrument, and is used to measure parameters in a limited space. In actual operation, the positional relationship between measuring instruments at different positions is involved, and it is necessary to accurately establish the connection between them to ensure measurement accuracy and reliability. Traditional local calibration methods are often limited by space and time, and it is difficult to meet the calibration requirements of the rotary mechanism measurement platform in a small space. Therefore, a global calibration method is needed to solve this problem. The global calibration method sets multiple reference points or calibration plates within the entire measurement platform, and uses cameras or other sensors to identify and locate these reference points, thereby establishing an absolute position relationship between different positions. Through the global calibration method, the precise positioning of each measuring instrument in the rotary mechanism measurement platform in a small space can be achieved, avoiding the practical operability problems in the local calibration method, and improving the measurement accuracy and precision. This global calibration method can provide a more effective and convenient solution for the practical application of the rotary mechanism measurement platform in a small space. Summary of the invention

[0003] In view of the existing practical needs, the technical problem to be solved by the present invention is to solve the problem of the difficulty of hand-eye calibration of the rotating mechanism and the binocular endoscope in a narrow space. The present invention obtains the transformation matrix through the target ball coordinates measured by the laser tracker and the binocular endoscope, and obtains the transformation matrix through the three-point coordinates of the rotating mechanism and the laser tracker. The two matrices are multiplied to achieve the unification of the global coordinate system, that is, the calibration of the hand-eye relationship.

[0004] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a hand-eye calibration method of a rotating mechanism and a binocular endoscope in a narrow space, comprising the following steps:

[0005] S1) In a small space, a calibration plate is set at the processing station, and a binocular endoscope is used to simultaneously capture the image of the calibration plate to calibrate the binocular endoscope camera to establish a binocular fusion coordinate system;

[0006] S2) Place N target balls on the calibration plate, use the binocular endoscope and laser tracker to obtain the coordinates of the centers of the N target balls in their respective coordinate systems, and calculate the transformation matrix H between the binocular fusion coordinate system and the laser tracker coordinate system through rigid body transformation C-L ;

[0007] S3) N target balls are placed on the sleeve, and the rotating mechanism rotates around its central axis within a set angle range with a fixed step length, and during the rotation process, the coordinates of the centers of the N target balls after each rotation stops are obtained by a laser tracker, which are used to fit the rotation plane and the normal vector and establish a local coordinate system of the rotating mechanism;

[0008] S4) Record the coordinates of any three points on the rotating mechanism, collect the coordinates of the three points using a laser tracker, and calculate the transformation matrix H between the laser tracker and the local coordinate system of the rotating mechanism through rigid body transformation L-R ;

[0009] S5) Combining the transformation matrix among the binocular fusion coordinate system, the laser tracker coordinate system, and the local coordinate system of the rotating mechanism to realize the hand-eye calibration of the global coordinate system.

[0010] A calibration plate, a target ball, and a laser tracker are arranged in a small space, a sleeve is provided on a rotating mechanism, and a binocular endoscope camera is installed in the rotating mechanism; a calibration step is performed to achieve global calibration of the rotating mechanism through its own rotation in a small space.

[0011] The calibrated binocular endoscope comprises:

[0012] The RetinexNet algorithm is used to perform image enhancement processing on the binocular image, and the coordinates of the corner points to be matched in the two images are obtained respectively;

[0013] Combined with the actual corner coordinates in the world coordinate system, the internal parameters and spatial position relationship of the binocular endoscope camera are determined;

[0014] After fusion, the binocular fusion coordinate system is obtained.

[0015] The binocular endoscope and the laser tracker are used to obtain the center coordinates of the target ball in the scene respectively, by photographing the target ball image to obtain the surface point coordinates of the target ball, and fitting the center coordinates in the respective coordinate systems.

[0016] The transformation matrix H between the binocular fusion coordinate system and the laser tracker coordinate system is calculated by rigid body transformation. C-L ,include:

[0017] Define that for the N target balls placed on the calibration plate, the coordinate set of the ball center obtained in the binocular fusion coordinate system is K, and the coordinate set obtained in the laser tracker coordinate system is Q;

[0018] According to the rigid body transformation, the transformation relationship formula is as follows:

[0019] Q=H C-L *K

[0020] Among them, H C-L is the 4*3 transformation matrix from the binocular endoscope coordinate system to the laser tracker coordinate system, and

[0021]

[0022] 0 T =[0 0 0]

[0023] Matrix R C-L Yes H C-L The rotation components of the 3*3 rotation matrix, matrix t C-L Yes H C-L The translation components of the 3*1 translation matrix, where t x is the translation component in the x direction from the binocular fusion coordinate system to the laser tracker coordinate system, t y is the translation component in the y direction from the binocular fusion coordinate system to the laser tracker coordinate system, t z It is the translation component in the z direction from the binocular fusion coordinate system to the laser tracker coordinate system.

[0024] The rotating mechanism rotates with a fixed step length of 5°, and the rotation angle range is 180°.

[0025] When establishing the local coordinate system of the rotating mechanism, if the rotating mechanism rotates so that the laser tracker cannot scan the position of the target ball, it is necessary to set up more than 4 common points in the scene and move the laser tracker. The laser tracker scans the coordinates of the common points before and after the movement to establish the conversion relationship between the moving position points.

[0026] The local coordinate system of the rotating mechanism is obtained by the rotating plane and the normal vector.

[0027] The transformation matrix H of the laser tracker and the local coordinate system of the rotating mechanism is calculated by rigid body transformation. L-R ,include:

[0028] Define that for any three points on the rotating mechanism, the set of N target ball center coordinates obtained in the local coordinate system of the rotating mechanism is M, and the set of coordinates obtained in the laser tracker coordinate system is N;

[0029] According to the rigid body transformation, the transformation relationship formula is as follows:

[0030] M=H L-R *N

[0031] Among them, H L-R is the 4*3 transformation matrix from the laser tracker coordinate system to the rotating mechanism coordinate system, and

[0032]

[0033]

[0034] 0 T=[0 0 0]

[0035] Matrix R L-R Yes H L-R The rotation components of the 3*3 rotation matrix, matrix t L-R Yes H L-R The translation components of the 3*1 translation matrix; where t xx is the translation component in the x direction from the laser tracker coordinate system to the rotating mechanism coordinate system, t yy is the translation component in the y direction from the laser tracker coordinate system to the rotating mechanism coordinate system, t zz is the translation component from the laser tracker coordinate system to the rotation mechanism coordinate system in the z direction.

[0036] The hand-eye calibration of the global coordinate system is the unification of the binocular fusion coordinate system and the local coordinate system of the rotation mechanism. The transformation relationship formula is as follows:

[0037] P=H C-L *H L-R *P′

[0038] Where P′ is the point coordinate in the binocular internal fusion coordinate system, and P is the point coordinate in the rotation mechanism coordinate system.

[0039] The present invention has the following beneficial effects and advantages:

[0040] The method of the present invention uses a laser tracker as a conversion medium to calibrate the positional relationship between the rotating mechanism and the binocular endoscope, greatly improving the on-site calibration efficiency and being able to achieve global unification of the coordinates of the rotating mechanism measurement platform in a narrow space. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the method of the present invention;

[0042] Figure 2 Schematic diagram of the measurement platform of the method of the present invention;

[0043] Figure 3 It is a partially enlarged schematic diagram of the measurement platform of the method of the present invention;

[0044] Figure 4 This is a network diagram of the RetinexNet algorithm. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0047] The application scenario of the present invention is: in a small space, the rotating mechanism is used as an assembly robot, and a binocular endoscope camera is installed inside it as an "eye" to capture images of equipment on the processing station within the field of view. The rotating mechanism extends a support arm outward, and an upper sleeve is provided on the support arm. The end of the sleeve is used as a "hand" to clamp the screws and perform assembly tasks on the screw holes of the equipment on the processing station. Therefore, it is necessary to unify the coordinate systems of the binocular endoscope camera and the rotating mechanism to achieve hand-eye calibration.

[0048] To achieve the above purpose, the present invention deploys a laser tracker in a small space as a coordinate system conversion relay. A hand-eye calibration method for a rotating mechanism and a binocular endoscope in a small space of the present invention comprises the following steps:

[0049] Step 1: Set a calibration plate on the processing station, and use the binocular endoscope camera to realize binocular calibration on the calibration plate to obtain a binocular fusion coordinate system, so that the coordinates of the equipment on the processing station can be accurately obtained;

[0050] Step 2: Place the target ball in the four holes of the calibration plate on the processing station, use the binocular endoscope and laser tracker to measure the coordinates of the center of the target ball, and determine the transformation matrix H of the positional relationship between the binocular endoscope and the laser tracker by calculating the rigid body transformation. C-L ;

[0051] Step 3: Place the target ball on the sleeve, take the position of the sleeve as the rotation starting point, and use the rotating mechanism to rotate from the rotation starting point at a fixed angle, that is, rotate the rotating mechanism to drive the end of the sleeve to rotate 180° around the axis of the rotating mechanism. After each rotation stops, the laser tracker collects the coordinates of the center of the target ball, fits the rotation plane and normal vector after the rotation is completed, and establishes the local coordinate system of the rotating mechanism;

[0052] Step 4: Take any 3 points on the rotating mechanism and record the coordinates. Use the laser tracker to collect the coordinates of the 3 points and calculate the transformation matrix H of the positional relationship between the laser tracker and the rotating mechanism through rigid body transformation. L-R ;

[0053] Step 5: Through the positional relationship between the laser tracker and the rotating mechanism, combined with the positional relationship between the binocular endoscope and the laser tracker, the hand-eye calibration of the global coordinate system is realized.

[0054] The following are detailed descriptions of the above key steps:

[0055] Step 1. Calibrate the binocular endoscope, including:

[0056] Two endoscope cameras are used to simultaneously capture the calibration plate image in a common field of view, and the internal parameters and spatial position relationship of the two endoscopes are determined by extracting and matching feature points.

[0057] Specifically include:

[0058] 1) Perform image enhancement processing on the binocular image to obtain the coordinates of the corner points to be matched in the two images respectively;

[0059] 2) Determine the internal parameters and spatial position relationship of the binocular endoscope camera based on the actual corner point coordinates in the world coordinate system;

[0060] 3) After fusion, the binocular fusion coordinate system is obtained.

[0061] In a small space, the lighting capacity is very limited due to space constraints, which increases the noise of image acquisition. The RetinexNet algorithm is an image enhancement algorithm based on the Retinex theory. The algorithm combines deep learning and Retinex theory. It learns and simulates the human eye's perception of images by training neural networks to achieve image enhancement and defogging effects, making the image clearer and more realistic. The RetinexNet algorithm trains a neural network on the collected low-light data blocks. The entire network is divided into two parts: the Decom-Net network is used to achieve image decomposition, and the Enhance-Net network is used to achieve lighting adjustment. In the process of training Decom-Net, there is no GT. When learning the network, only the following key constraints need to be met, including the consistent reflectivity shared by paired low / normal light images and the consistency of lighting. Enhance-Net is used to achieve lighting enhancement.

[0062] Step 2. Place the target ball in the four holes of the calibration plate, use the binocular endoscope and the laser tracker to measure the coordinates of the center of the target ball, and determine the positional relationship between the binocular endoscope and the laser tracker through the calculation of rigid body transformation, including:

[0063] The target ball should be placed on the circular hole reserved in the calibration plate. The center coordinates of the four target balls are measured using a binocular endoscope and a laser tracker. The coordinate set of the center of the ball in the binocular endoscope coordinate system is K, and the coordinate set of the center of the ball in the laser tracker coordinate system is Q. The transformation formula for calculating the coordinate set K and the coordinate set Q is as follows:

[0064] Q=H C-L *K

[0065] Among them, H C-L is the 4*3 transformation matrix from the binocular endoscope coordinate system to the laser tracker coordinate system, and

[0066]

[0067] 0 T =[0 0 0]

[0068] Matrix R C-L Yes H C-L The rotation components of the 3*3 rotation matrix, matrix t C-L Yes H C-L The translation components of the 3*1 translation matrix, where t x is the translation component in the x direction from the binocular endoscope coordinate system to the laser tracker coordinate system, t y is the translation component in the y direction from the binocular endoscope coordinate system to the laser tracker coordinate system, t z is the translation component in the z direction from the binocular endoscope coordinate system to the laser tracker coordinate system.

[0069] Step 3. Place the target ball on the sleeve, use the sleeve position as the rotation starting point, and use the rotation mechanism to rotate from the rotation starting point at a fixed angle. After each rotation stops, the laser tracker collects the coordinates of the center of the target ball. After the rotation is completed, the rotation plane and normal vector are fitted, including:

[0070] The position of the sleeve is taken as the rotation starting point. The rotating mechanism starts from the rotation starting point and rotates with a fixed step length of 5° for a total rotation of 180°. The laser tracker collects the surface point coordinates of the target ball after each rotation stops, and fits the coordinates of the center of the ball. Finally, the least squares method is used to fit the rotation plane. The equation is expressed as Ax+By+Cz=0, and the normal vector is determined. Then the local coordinate system of the rotating mechanism is established.

[0071] Note: If the laser tracker cannot scan the location information, you need to perform a station transfer operation, that is, you need to set up more than 4 common points and record the coordinates of the common points before and after the laser tracker moves.

[0072] Step 4. Establish the local coordinate system of the rotating mechanism and calculate the positional relationship between the laser tracker and the rotating mechanism through rigid body transformation, including:

[0073] Take any three points on the rotating mechanism, and the coordinate set M in the rotating mechanism coordinate system. At the same time, get the coordinate set N under the corresponding laser tracker, perform rigid body transformation, and calculate the transformation relationship between the coordinate set M and the coordinate set N as follows:

[0074] M=H L-R *N

[0075] Among them, H L-R is the 4*3 transformation matrix from the laser tracker coordinate system to the rotating mechanism coordinate system, and

[0076]

[0077] 0 T =[0 0 0]

[0078] Matrix R L-R Yes H L-R The rotation components of the 3*3 rotation matrix, matrix t L-R Yes H L-R The translation components of the 3*1 translation matrix. xx is the translation component in the x direction from the laser tracker coordinate system to the rotating mechanism coordinate system, t yy is the translation component in the y direction from the laser tracker coordinate system to the rotating mechanism coordinate system, t zz is the translation component from the laser tracker coordinate system to the rotation mechanism coordinate system in the z direction.

[0079] Step 5. The global coordinate system is unified by combining the positional relationship between the laser tracker and the rotating mechanism with the positional relationship between the binocular endoscope and the laser tracker, including:

[0080] The unification of the global coordinate system is the unification of the binocular endoscope coordinate system and the rotation mechanism coordinate system, that is, the calibration of the hand-eye relationship. The transformation relationship formula is as follows:

[0081] P=H C-L *H L-R *P′

[0082] Where P′ is the point coordinate in the binocular endoscope coordinate system, and P is the point coordinate in the rotation mechanism coordinate system.

[0083] In summary, the method of the present invention provides a hand-eye calibration method for a rotating mechanism and a binocular endoscope in a narrow space, which has the characteristics of convenient operation and high efficiency, and is suitable for on-site calibration in a narrow space.

[0084] The embodiments described in the above description will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

Claims

1. A hand-eye calibration method for a rotating mechanism and a binocular endoscope in a narrow space, characterized in that: The following steps are involved: S1) In a small space, a calibration plate is set at the processing station, and a binocular endoscope is used to simultaneously capture the image of the calibration plate to calibrate the binocular endoscope camera to establish a binocular fusion coordinate system; S2) Place N target balls on the calibration plate, use the binocular endoscope and laser tracker to obtain the coordinates of the centers of the N target balls in their respective coordinate systems, and calculate the transformation matrix H between the binocular fusion coordinate system and the laser tracker coordinate system through rigid body transformation C-L ; S3) N target balls are placed on the sleeve, and the rotating mechanism rotates around its central axis within a set angle range with a fixed step length, and during the rotation process, the coordinates of the centers of the N target balls after each rotation stops are obtained by a laser tracker, which are used to fit the rotation plane and normal vector and establish a local coordinate system of the rotating mechanism; S4) Record the coordinates of any three points on the rotating mechanism, collect the coordinates of the three points using a laser tracker, and calculate the transformation matrix H between the laser tracker and the local coordinate system of the rotating mechanism through rigid body transformation L-R ; S5) Combining the transformation matrix among the binocular fusion coordinate system, the laser tracker coordinate system, and the local coordinate system of the rotating mechanism to realize the hand-eye calibration of the global coordinate system.

2. According to the method for hand-eye calibration of a rotating mechanism and a binocular endoscope in a narrow space in claim 1, it is characterized in that: A calibration plate, a target ball, and a laser tracker are arranged in a small space, a sleeve is provided on a rotating mechanism, and a binocular endoscope camera is installed in the rotating mechanism; a calibration step is performed to achieve global calibration of the rotating mechanism through its own rotation in a small space.

3. The hand-eye calibration method of a rotating mechanism and a binocular endoscope in a narrow space according to claim 1, characterized in that: The calibrated binocular endoscope comprises: The RetinexNet algorithm is used to perform image enhancement processing on the binocular image, and the coordinates of the corner points to be matched in the two images are obtained respectively; Combined with the actual corner coordinates in the world coordinate system, the internal parameters and spatial position relationship of the binocular endoscope camera are determined; After fusion, the binocular fusion coordinate system is obtained.

4. The hand-eye calibration method of a rotating mechanism and a binocular endoscope in a narrow space according to claim 1, characterized in that: The binocular endoscope and the laser tracker are used to obtain the coordinates of the center of the target ball in the scene respectively, which are obtained by photographing the target ball image to obtain the surface point coordinates of the target ball and fitting the coordinates of the center of the target ball in the respective coordinate systems.

5. The hand-eye calibration method of a rotating mechanism and a binocular endoscope in a narrow space according to claim 1 is characterized in that , the transformation matrix H between the binocular fusion coordinate system and the laser tracker coordinate system is calculated through rigid body transformation C-L ,include: Define that for the N target balls placed on the calibration plate, the coordinate set of the ball center obtained in the binocular fusion coordinate system is K, and the coordinate set obtained in the laser tracker coordinate system is Q; According to the rigid body transformation, the transformation relationship formula is as follows: Q=H C-L *K Among them, H C-L is the 4*3 transformation matrix from the binocular endoscope coordinate system to the laser tracker coordinate system, and 0 T ＝[0 0 0] Matrix R C-L Yes H C-L The rotation components of the 3*3 rotation matrix, matrix t C-L Yes H C-L The translation components of the 3*1 translation matrix, where t x is the translation component in the x direction from the binocular fusion coordinate system to the laser tracker coordinate system, t y is the translation component in the y direction from the binocular fusion coordinate system to the laser tracker coordinate system, t z It is the translation component in the z direction from the binocular fusion coordinate system to the laser tracker coordinate system.

6. The hand-eye calibration method of a rotating mechanism and a binocular endoscope in a narrow space according to claim 1 is characterized in that ,The rotating mechanism rotates with a fixed step of 5°, and the rotation angle range is 180°.

7. The hand-eye calibration method of a rotating mechanism and a binocular endoscope in a narrow space according to claim 1 is characterized in that ,When establishing the local coordinate system of the rotating mechanism, if the rotating mechanism rotates so that the laser tracker cannot scan the target ball position, it is necessary to set more than 4 common points in the scene and move the laser tracker position. The laser tracker scans the coordinates of the common points before and after the movement to establish the conversion relationship between the moving position points.

8. The hand-eye calibration method of a rotating mechanism and a binocular endoscope in a narrow space according to claim 1 is characterized in that ,The local coordinate system of the rotating mechanism is obtained by the rotation plane and the normal vector.

9. The hand-eye calibration method of a rotating mechanism and a binocular endoscope in a narrow space according to claim 1 is characterized in that , the transformation matrix H of the laser tracker and the local coordinate system of the rotating mechanism is calculated by rigid body transformation L-R ,include: Define that for any three points on the rotating mechanism, the set of N target ball center coordinates obtained in the local coordinate system of the rotating mechanism is M, and the set of coordinates obtained in the laser tracker coordinate system is N; According to the rigid body transformation, the transformation relationship formula is as follows: M=H L-R *N Among them, H L-R is the 4*3 transformation matrix from the laser tracker coordinate system to the rotating mechanism coordinate system, and 0 T ＝[0 0 0] Matrix R L-R Yes H L-R The rotation components of the 3*3 rotation matrix, matrix t L-R Yes H L-R The translation components of the 3*1 translation matrix; where t xx is the translation component in the x direction from the laser tracker coordinate system to the rotating mechanism coordinate system, t yy is the translation component in the y direction from the laser tracker coordinate system to the rotating mechanism coordinate system, t zz is the translation component from the laser tracker coordinate system to the rotation mechanism coordinate system in the z direction.

10. The hand-eye calibration method of a rotating mechanism and a binocular endoscope in a narrow space according to claim 1, characterized in that ,The hand-eye calibration of the global coordinate system is the unification of the binocular fusion coordinate system and the local coordinate system of the rotation mechanism. The transformation relationship formula is as follows: P=H C-L *H L-R *P′ Where P ′ is the point coordinate in the binocular internal fusion coordinate system, and P is the point coordinate in the rotation mechanism coordinate system.

Images