Telecentric Imaging Combined with Scham Projection Three-Dimensional Microscopic Measurement System and Joint Calibration Method

Abstract

The present invention discloses a telecentric imaging combined with Scheimpflug projection three-dimensional microscopic measurement system and a joint calibration method, which establish a Scheimpflug projection model, a Scheimpflug distortion model and a telecentric imaging model; capture and preprocess the target attitude images of the calibration plate, perform the phase unwrapping of the center of the telecentric camera and the center mapping of the projector, estimate the internal parameter intrinsic matrix, the external parameter matrix and the distortion coefficient of the projector, correct the projector distortion and the Scheimpflug angle; optimize the error of the external parameter matrix of the projector; align the three-dimensional coordinates of the center of the calibration plate to the coordinate system of the projector, and calibrate the projection matrix of the telecentric imaging model; perform the camera-projector joint calibration to realize the calibration of the three-dimensional microscopic measurement system with high precision. The present invention can solve the problem of limited depth of field under small field of view and high magnification, overcome the reconstruction error caused by defocus of the projection and imaging branches, and increase the dynamic measurement range and accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of precision manufacturing and semiconductor manufacturing microstructure imaging technology, and in particular to a three-dimensional microscopic measurement system combining telecentric imaging and Scham projection and a joint calibration method. Background Art

[0002] With the latest progress in precision manufacturing and semiconductor manufacturing, 3D imaging and metrology of complex microstructures have become increasingly important. Many non-contact optical methods have been studied to measure the morphology of small objects, such as confocal microscopes, white light interference microscopes, and fringe projection 3D microscopes. In contrast, fringe projection profilometry can be widely applied to complex industrial environments because it is insensitive to changes in background, contrast, and noise.

[0003] Existing fringe projection 3D microscopes usually have two system frameworks. One is to improve one channel of a stereomicroscope through different projection techniques. Due to the limitations of the objective lens aperture and imaging principle, both the field of view and the depth of field are severely limited to the sub-millimeter level. The other uses a non-telecentric lens with a long working distance. However, in such a system, the limited field of view usually results in a small DOF, which is insufficient to measure 3D objects with height variations of several millimeters. Telecentric lenses have been applied to fringe projection 3D microscopy systems due to their good characteristics, such as orthographic projection, low distortion, and constant magnification within a specific distance range. Almost all FP-3DM systems using telecentric lenses consist of a projection branch and an imaging branch with an angle between them. This will result in only a small part of the projected fringes being in the focal position, and the rest of the area remaining out of focus. In addition, the limited degrees of freedom of the telecentric lens will further significantly reduce the confocal region.

[0004] In addition, the calibration of a fringe projection 3D microscopy system using a telecentric lens is not simple, especially for the Z direction, because telecentricity makes the depth insensitive to changes along the optical axis. Currently, there is no mature calibration algorithm for the calibration of long-distance telecentric lens imaging in related research. The related calibration algorithms either require the use of a high-precision translation stage for system calibration, which is usually difficult to set up (for example, the moving direction is completely perpendicular to the Z axis) and costly, or due to strong constraint requirements (such as the orthogonality of the rotation matrix), it is difficult to achieve high-precision external parameter calibration with this method. In addition, since the magnification and external parameters are naturally coupled and difficult to separate, the calibration process is further complicated, increasing the uncertainty of the modeling accuracy. In addition, due to the image plane tilt of the Scham projection model, the offset generated by it cannot be coupled with the conventional distortion model, and there is currently no unified and perfect model description method. Moreover, due to the strong geometric constraints between the camera and the projector, the system parameter calibration process is complex and it is difficult to achieve high-precision system calibration of the system. Summary of the Invention

[0005] To solve the above-mentioned defects in the prior art, the purpose of the present invention is to provide a telecentric imaging Scheimpflug projection three-dimensional microscopic measurement system and a joint calibration method. By constructing the telecentric imaging Scheimpflug projection three-dimensional microscopic measurement system, this method solves the problem of small field of view, high magnification and depth of field, overcomes the reconstruction error caused by defocus of the projection and imaging branches, and increases the dynamic measurement range.

[0006] The present invention is realized through the following technical solutions.

[0007] On the one hand, the present invention provides a joint calibration method for a three-dimensional microscopic measurement system combining telecentric imaging and Scheimpflug projection, including:

[0008] Establish a Scheimpflug projection model, a Scheimpflug distortion model, and a telecentric imaging model;

[0009] Use a telecentric camera to capture and preprocess the target pose image of the calibration plate, and use the multi-frequency heterodyne method to perform phase unwrapping of the center of the telecentric camera;

[0010] According to the obtained center phase, perform projector center mapping through the local homography method;

[0011] According to the established Scheimpflug projection model and the mapped projector center, use the maximum likelihood estimation method to estimate the internal parameter intrinsic matrix, external parameter matrix, and distortion coefficient of the projector;

[0012] According to the established Scheimpflug distortion model and the obtained distortion coefficient, correct the projector distortion and Scheimpflug angle;

[0013] Optimize the error of the projector external parameter matrix obtained in different poses through the PnP method;

[0014] According to the projector external parameter matrix, align the three-dimensional coordinates of the center of the calibration plate to the coordinate system of the projector;

[0015] Use the direct linear transformation and singular value decomposition method to calibrate the projection matrix of the telecentric imaging model according to the aligned three-dimensional coordinates of the center;

[0016] According to the obtained projection matrix of the telecentric imaging model, the internal parameter intrinsic matrix, external parameter matrix, and distortion coefficient of the projector, use the nonlinear method for camera-projector joint calibration to achieve high-precision calibration of the three-dimensional microscopic measurement system.

[0017] Preferably, when establishing the Scheimpflug projection model and the distortion model, when the tilt angle between the imaging surface and the lens surface ≤ 6°, compensate for the tilt effect by adding additional aberration distortion parameters.

[0018] Preferably, each target pose image is captured and preprocessed, and the calibration plate target pose image is captured and preprocessed, including: capturing a set of images of each target pose, including white light projection, horizontal stripe pattern, and vertical stripe pattern; the white light projection is used to extract the center of the circle captured by the camera as a feature point, and the horizontal and vertical stripe patterns are used to solve the horizontal and vertical phases of the center of the circle.

[0019] Preferably, the projector center mapping is performed, including: using the local homography method, using the integer pixel phase values in the neighborhood of the camera center, solving the coordinate values of the corresponding projector, constructing the local mapping relationship from the camera coordinates to the projector coordinates, and solving the accurate camera-projector sub-pixel center mapping.

[0020] Preferably, the three-dimensional coordinates of the center of the calibration plate are aligned to the coordinate system of the projector. Using the external parameter matrix of the projector obtained under different poses of the calibration plate, the world coordinates of the center of the calibration plate are aligned with the projector coordinate system, and the projector external parameter matrix becomes the unit matrix and the zero vector matrix.

[0021] On the other hand, the present invention provides a telecentric imaging combined with Scheimpflug projection three-dimensional microscopic measurement reconstruction method of the joint calibration method, including:

[0022] Using the stereo vision algorithm to reconstruct the 3D topography of the target pose image and reconstruct the microscopic 3D target pose image:

[0023] By solving the phase of the projected stripes, the corresponding points of the telecentric imaging point and the projection point are obtained. Given the camera image coordinates and the corresponding projector horizontal coordinates, the 3D coordinates of the target pose image are calculated.

[0024] On the other hand, the present invention provides a telecentric imaging combined with Scheimpflug projection three-dimensional microscopic measurement system of the method, including a projection branch, an imaging branch, and a wedge member arranged at an angle to each other;

[0025] The projection branch includes a projection lens and a DMD digital projection chip, and the DMD digital projection chip generates a digital stripe pattern and projects it onto the object surface of the object to be measured;

[0026] The imaging branch includes a telecentric imaging lens and a high-resolution color camera. The high-resolution color camera captures the deformed stripe pattern reflected from the object surface through the telecentric imaging lens, and reconstructs the surface topography through the system model and the phase recovery algorithm;

[0027] The wedge member is arranged between the projection lens and the DMD digital projection chip and is used to shift the imaging object plane. The extended lines of the Scheimpflug shifted object plane, the projection lens principal plane, and the DMD digital projection chip plane intersect at the Scheimpflug coincidence point on the object plane.

[0028] Due to the above technical solutions adopted by the present invention, it has the following beneficial effects:

[0029] 1. The present invention adopts a telecentric imaging Scheimpflug projection three-dimensional microscopic measurement system, which consists of a camera with a telecentric lens and a small field of view pinhole projector using the Scheimpflug condition. It solves the problem of too small depth of field in the case of a small field of view and high magnification, overcomes the reconstruction error caused by defocusing of the projection and imaging branches, makes full use of the limited degrees of freedom of the telecentric lens, and increases the dynamic measurement range.

[0030] 2. The present invention adopts the design process and parameter determination criteria of the telecentric imaging Scheimpflug projection three-dimensional microscopic measurement system, which solves the problem of rapid construction and integration of the system.

[0031] 3. The present invention adopts a joint calibration method for the telecentric imaging Scheimpflug projection three-dimensional microscopic measurement system. This method constructs a new Scheimpflug projection mathematical model, and makes full use of the established Scheimpflug projection model and Scheimpflug distortion model to calibrate telecentric imaging. Further, the camera-projector joint calibration error is optimized non-linearly. This method solves the problem of high-precision calibration of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and do not constitute an improper limitation of the present invention. In the drawings:

[0033] Figure 1 is a schematic diagram of the system structure of the present invention;

[0034] Figure 2 is a structural design flow chart of the present invention;

[0035] Figure 3 is a Scheimpflug shift cross-sectional view of the present invention;

[0036] Figure 4 is a comparison diagram of the image quality of the Scheimpflug shift projection of the present invention;

[0037] Figure 5 is a flow chart of the joint calibration method of the present invention;

[0038] Figure 6 is a schematic diagram of the Scheimpflug projection optical path of the present invention.

[0039] Among them: 1 - telecentric imaging lens, 2 - high-resolution color camera, 3 - projection lens, 4 - DMD digital projection chip, 5 - plane of the DMD digital projection chip, 6 - principal plane of the projection lens, 7 - object plane, 8 - Scheimpflug coincidence point, 9 - object to be measured, 10 - multi-channel RGBW ring light source, 11 - ideal DMD projection plane, 12 - real DMD projection plane, 13 - principal point of the DMD chip. DETAILED DESCRIPTION OF THE INVENTION

[0040] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but do not limit the present invention.

[0041] A telecentric imaging combined with Scheimpflug projection three-dimensional microscopic measurement system provided by the present invention has a specific implementation method as follows.

[0042] Please refer to Figure 1 , in one specific embodiment, the entire system includes a telecentric imaging lens 1, a high-resolution color camera 2, a projection lens 3, and a DMD digital projection chip 4.

[0043] The system includes a projection branch composed of a projection lens 3 and a DMD digital projection chip 4 and an imaging branch composed of a telecentric imaging lens 1 and a high-resolution color camera 2, and there is an angle between them. In the projection branch, a digital fringe pattern is generated by the DMD digital projection chip 4 and projected onto the surface of the object 9 to be measured.

[0044] In the imaging branch, the high-resolution color camera 2 captures the deformed fringe pattern reflected from the object surface through the telecentric imaging lens 1. Then, the surface topography can be reconstructed using an appropriate system model and phase retrieval algorithm. The optical axis of the imaging branch is perpendicularly configured at a 90° angle relative to the imaging object surface, and the projection branch is tilted at a specific angle β after Scheimpflug condition design for projection.

[0045] The high-resolution color camera 2 and the telecentric imaging lens 1 are also used to collect the color information of the object to be measured. The multi-channel RGBW annular light source 10 emits lights of different wavelength bands in sequence. After being reflected by the surface of the object to be measured, they are collected by the imaging branch to obtain the color texture information of the object surface. According to the calibration information, the pixels of this image are matched with the three-dimensional point cloud, and color texture modulation is performed on the point cloud.

[0046] The axis of the multi-channel RGBW annular light source 10 coincides with the optical axis of the imaging branch to assist the imaging branch in collecting the color image of the object to be measured.

[0047] Furthermore, in the present invention, the projection branch adopts a Scheimpflug tilted image plane structure to solve the problem of too small depth of field in the case of a small field of view and high magnification, overcome the reconstruction error problem caused by defocus of the projection and imaging branches, make full use of the limited degrees of freedom of the telecentric lens, and increase the dynamic measurement range.

[0048] A wedge is arranged between the projection lens 3 and the DMD digital projection chip 4 to shift the imaging object surface.

[0049] Scheimpflug shift needs to follow that the extension lines of the object plane 7, the principal plane 6 of the projection lens, and the plane 5 of the DMD digital projection chip intersect at the Scheimpflug coincidence point 8 on the object plane.

[0050] For the design process and basis of the telecentric imaging Scheimpflug projection three-dimensional microscopic measurement system, please refer to Figure 2 , and the design process and basis are as follows:

[0051] 1. First, determine the working distance and field of view size of the telecentric imaging lens; 2. Then, based on the depth of field to be increased on the object plane and the depth of focus range on the image plane, determine the angle of shift projection required on the object plane and the projection working distance; 3. Secondly, determine the size of the projection format according to the Gaussian imaging formula and imaging geometric relationship; 4. Finally, determine the Scheimpflug angle of image plane shift according to formula (1).

[0052] θ = arctan(M * tan(β)) (1)

[0053] Refer to Figure 3 , where θ is the Scheimpflug angle of image plane shift, M = p i / p o is the nominal lateral magnification of the optical axis, p i and p o are the image distance and object distance of the nominal target respectively, and need to satisfy the Gaussian conjugate formula

[0054] Figure 4 shows the comparison of image quality between direct tilt projection and Scheimpflug projection. It can be seen that without Scheimpflug adjustment, only some pattern features can be accurately focused, and after performing Scheimpflug tilt adjustment, the entire object plane can be accurately focused.

[0055] The three-dimensional microscopic measurement system of the present invention uses a long-distance telecentric lens and Scheimpflug projection. The present invention also provides a joint calibration method for the telecentric imaging Scheimpflug projection three-dimensional microscopic measurement system. First, a new Scheimpflug projection mathematical model is constructed, and it is proposed to use the established Scheimpflug projection model and Scheimpflug distortion model to calibrate telecentric imaging. Further, the camera-projector joint calibration error is optimized non-linearly, and finally, high-precision system calibration is achieved.

[0056] As Figure 5 shown, the specific implementation method of the joint calibration method for the telecentric imaging combined with Scheimpflug projection three-dimensional microscopic measurement system provided by the present invention is as follows:

[0057] Step 1: Establish a Scheimpflug projection model and a distortion model

[0058] The projector model can be described by the pinhole imaging perspective imaging model. As Figure 6 shown, where 11 is the ideal DMD projection plane, 12 is the real DMD projection plane, and 13 is the principal point of the DMD chip.

[0059] From the 3D object point p w = (x w, y w , z w ) T to the 2D projector DMD point p p (u p , v p ) T The projection can be described as follows:

[0060]

[0061]

[0062] where: s is the scale factor, K p is the intrinsic internal parameter matrix of the projector, f x and f y are the effective focal lengths of the projector in the u and v directions respectively, (u p 0, v p 0) are the coordinates of the principal point; R p 3×3 = [r p ij and T p 3×1 = [t p x t p y t p z T are the rotation and translation matrices of the projection coordinate system (X p , Y p , Z p ) T relative to the world coordinate system (X w , Y w , Z w ) T , and is the projection matrix of the projector, m p ij are the projection matrix parameters.

[0063] Equations (2) and (3) describe the ideal linear projection process. However, the actual projection process is non-linear because lens distortion changes the direction of light. Ideally, the coordinates (x w , y w , z w ) T projected onto the normalized DMD plane are

[0064]

[0065] Lens distortion will change the ideal undistorted normalized DMD coordinates in the projector coordinate system ​ At the position, introducing second-order radial and tangential distortions, can be expressed as

[0066]

[0067] Wherein, is the radial distance, is the normalized DMD coordinate of the actual distortion, and k1, k2 and p1, p2 are the radial and tangential distortion coefficients respectively.

[0068] Under the Scheimpflug condition, the Scheimpflug adapter tilts the projection lens by rotation, resulting in perspective distortion, as shown in Figure 6 . The rotation can be divided into two rotations around the x p and y p axes, with angles of θ x and θ y .

[0069] In the Scheimpflug projection designed by the present invention, when the tilt angle between the imaging surface and the lens surface is small (≤6°), the tilt effect can be compensated by adding additional aberration distortion parameters, and the Scheimpflug camera calibration problem can be solved to a certain extent. The shift and tilt of the image plane will result in perspective distortion. This distortion can be modeled in the following way:

[0070]

[0071]

[0072] Wherein, p p is the DMD pixel coordinate of the projector, is the DMD coordinate after the Scheimpflug distortion, is the homography transformation between the ideal projection DMD plane and the Scheimpflug projection DMD plane. The homography matrix is determined by the Scheimpflug angles θ x and θ y . According to the projection geometry, the same object point and its corresponding projection points in the two DMD planes are on a line. Therefore, the homography has the following QR decomposition:

[0073]

[0074] Wherein, R h (θ x , θ y ) is the rotation matrix between the two DMD planes, and K h (θ x , θ y) is the coordinate scale transformation caused by the perspective projection between two DMD planes. Equation (8) shows that the homography from the ideal DMD plane to the Scheimpflug DMD plane can be decomposed into two matrix multiplications. One is an upper triangular matrix, and the other is an orthogonal matrix.

[0075] The present invention simultaneously introduces the tangential distortion coefficient and an additional Scheimpflug distortion coefficient [θ x , θ y , so the shift amount will be evenly distributed into the tangential distortion and the additional Scheimpflug distortion.

[0076] Step 2, establish the telecentric imaging model

[0077] The most important prerequisite for accurate calibration is the correct mathematical expression of the camera model. The telecentric lens only magnifies in the X and Y directions and is insensitive to the depth in the Z direction. By analyzing the light transmission matrix of this optical system, the object point q = (x c , y c , z c ) T in the camera coordinates, and its homogeneous image coordinates q c (u c , v c ) T can be described as follows:

[0078]

[0079] Where: K c is the intrinsic matrix of the telecentric camera, m x and m y are the effective magnification factors of the telecentric lens in the X and Y directions respectively, q c (u c , v c ) T are the homogeneous image coordinates, (u c 0, v c 0) is the principal point of the projection center, and x c , y c are the X coordinate and Y coordinate of the object point in the camera coordinate system respectively.

[0080] Because the telecentric lens performs parallel imaging, for the telecentric imaging in Equation (9), there is no principal point (u c 0, v c 0) and projection center.

[0081] Where: m x and m y are the effective magnification factors in the X and Y directions respectively, and for an ideal telecentric camera, they can be set to zero.

[0082] The world coordinate system q w=(x w , y w , z w ) T The relationship between and the camera coordinate system is as follows:

[0083]

[0084] Where: R c 3×3 =[r c ij is the rotation matrix, and T c 3×1 =[t c x t c y t c z T And the translation vector. Combining equations (9) and (10), the orthographic projection imaging formula of the telecentric lens from the 3D object point to the 2D camera image point is:

[0085]

[0086] Where is the telecentric imaging projection matrix, and m c ij are the projection matrix parameters.

[0087] Step 3: Image acquisition and preprocessing

[0088] The present invention uses an asymmetric black and white circular calibration plate as the calibration target. The calibration target is placed in different spatial directions. After telecentric imaging, a set of images of each target pose are captured by the telecentric camera, including white light projection, horizontal stripe pattern, and vertical stripe pattern. The white light projection is used to extract the center of the circle captured by the camera as the feature point, where i and j respectively represent the i-th spatial pose and the j-th feature point of the calibration target. The horizontal and vertical stripe patterns are used to solve the horizontal and vertical phases of the center of the circle.

[0089] In addition, the captured images usually have noise, so the captured images need to be preprocessed before solving the phase. In the present invention, Gaussian filtering is used to remove high-frequency and stray noise.

[0090] Step 4: Center phase unwrapping

[0091] Obtain the sub-pixel position coordinates of the center of the calibration plate circle. Using the multi-frequency heterodyne method, the horizontal and vertical absolute phase values of the integer pixels in the neighborhood near the center of the circle can be solved (n×n, where n is the pixel range of the selected neighborhood).

[0092] Step 5: Projector center mapping​

[0093] If the absolute phase value of the sub-pixel coordinates of the center of the circle is known, the unique projector center corresponding to the camera center can be obtained.

[0094]

[0095]

[0096] Where: and respectively represent the vertical and horizontal absolute phase values of the center of the circle, n v and n h respectively represent the number of fringes of the projected horizontal and vertical patterns, H p ×W p is the projector resolution.

[0097] However, the phase decoding accuracy can only be at the pixel level, and there may be errors in the decoding of a single pixel. To overcome these problems, the present invention adopts a local homography method. Specifically, first, the absolute phase values of the integer pixels within (n×n) near the center of the circle obtained in step 4 are used to obtain the corresponding projector coordinate values through formulas (12) and (13), and then a local mapping relationship from the camera coordinates to the projector coordinates is constructed to solve the accurate camera-projector sub-pixel center mapping.

[0098] Let be the integer pixel image coordinates near the center of the circle, be the corresponding projector DMD pixel at this point, and the minimum homography is found through the following formula:

[0099]

[0100] Where: ||·|| represents the least squares distance, and argmin() is a function used to calculate the minimum value of a certain function.

[0101] Formula (14) can be solved by a robust algorithm based on RANSAC.

[0102] Applying the local homography matrix, the sub-pixel coordinates of the camera center are converted to the sub-pixel coordinates of the projector DMD

[0103]

[0104] The local homography method can accurately obtain the corresponding coordinates of the projector DMD and reduce the calibration error of the projector. In addition, the local homography method allows for the non-linear error of the model caused by camera distortion.

[0105] Step 6: Estimate the intrinsic matrix, extrinsic parameter matrix, and distortion coefficients of the projector

[0106] Using the center points (u p , v p ) of the projector DMD at different poses extracted from the previous step, estimate the intrinsic matrix K p (i.e., f x , f y , u0, v0) of the projector using the camera calibration method of maximum likelihood estimation, and iteratively solve for the distortion coefficients and the Shearing angles (k1, k2, p1, p2, θ x , θ y ) through the Levenberg-Marquardt algorithm.

[0107]

[0108] Where: is the projection of the center feature point in image i according to formulas (2)-(8), is the vector of the projector distortion coefficients, n represents the number of different poses, m represents the number of feature points on the calibration target, k1, k2, p1, and p2 are set to zero, and θx and θy are set to the theoretical design values.

[0109] Thus, obtain the accurate intrinsic matrix K p of the projector and all the distortion coefficients of the projector, as well as the initial value of the projector extrinsic parameter matrix .

[0110] Step 7: Projector distortion and Shearing angle correction

[0111] Since the distortion and manufacturing error of the projection lens are greater than those of the telecentric lens, and the system configuration adopts Shearing projection. Therefore, it is necessary to correct the distortion and Shearing angles of the projector. Apply the distortion coefficients and Shearing angles obtained in Step 6 to formulas (4)-(7) to correct the center point of the projector and record it as

[0112] Step 8: Optimization of the projector extrinsic parameter matrix error

[0113] To obtain a more accurate projector extrinsic parameter matrix, use the PnP (Perspective-n-point) method to optimize the extrinsic parameters of the feature points at different poses. The optimization method iteratively minimizes the difference between the observed projection points and the calculated projection points, expressed as the following function:

[0114]

[0115] where ||·|| represents the least square distance, is the center coordinate of the projector after distortion correction. is the external parameter matrix of the projector, and the initial value is the result obtained in step 6. is the coordinate of the center in the world coordinate system.

[0116] Step 9: Align the three-dimensional coordinates of the center of the calibration board to the coordinate system of the projector

[0117] Using the external parameter matrix of the projector obtained in the previous step at different poses, align the world coordinates of the center of the calibration target with the coordinate system of the projector Then the external matrix of the projector becomes the identity matrix and the zero vector matrix [I 3×3 | 0 3×1 (I 3×3 is the identity matrix, 0 3×3 is the zero vector), and the measurement data is converted to the coordinate system of the projector:

[0118]

[0119] Step 10: Calculate the projection matrix of the telecentric imaging model

[0120] According to the three-dimensional coordinates of the center at each target pose in the projector coordinate system, through direct linear transformation (DLT), the orthographic projection imaging of the telecentric camera (Equation (11)) can be rewritten as

[0121]

[0122]

[0123]

[0124] Where: is the center captured by the camera in step 3. Substitute all the centers into Equation (19), and use the singular value decomposition (SVD) method to obtain the least squares solution of L 1×11 to achieve high-precision calibration of the telecentric imaging projection matrix using an accurately calibrated Scham projector. The solved telecentric imaging projection matrix contains the accurate pose relationship between the Scham projector and the telecentric camera.

[0125] Step 11: Camera-Projector Joint Calibration

[0126] The purpose of camera-projector joint calibration is to improve the calibration accuracy by simultaneously minimizing the reprojection errors of telecentric imaging and Scham projection. The joint calibration can be carried out in the following ways:

[0127]

[0128] Wherein: is the DMD sub-pixel coordinate of the projector, is the coordinate of the center feature point in image i calculated according to the Scheimpflug projection model, is the camera pixel coordinate of the center of the calibration plate in image i, is the image coordinate of the center point in image i calculated according to the telecentric imaging model, K p is the intrinsic matrix of the projector, R p 3×3 =[r p ij and T p 3×1 =[t p x t p y t p z T is the rotation and translation matrix of the projection coordinate system (X p , Y p , Z p ) T relative to the world coordinate system (X w , Y w , Z w ) T and, is the vector of the projector distortion coefficient, is the telecentric imaging projection matrix, is the image coordinate of the center point in image i calculated according to Equation (11). Equation (20) can be solved by the Levenberg-Marquardt nonlinear optimization algorithm, and all initial values have been obtained in the previous steps.

[0129] The present invention further provides a telecentric imaging Scheimpflug projection three-dimensional microscopic measurement and reconstruction method, and a specific implementation method thereof is as follows.

[0130] After the camera and the projector are calibrated for the Scheimpflug condition, the 3D shape of the object is reconstructed using the stereo vision algorithm. After calibrating the projection lens distortion and the Scheimpflug angle, the microscopic 3D image can be reconstructed through the following equation:

[0131] [u c v c 1] T = M c ·X (14)

[0132] [u p v p 1]= M p ·X (15) ​

[0133] Among them: M c 3×4 and M p 3×4 are respectively the projection matrices of the telecentric imaging and the projector. Formulas (14) and (15) can be rearranged as Q·X = 0, and Q is derived as:

[0134]

[0135] Among them: and are respectively and the i-th row of

[0136] By solving the phase of the projected fringe, the corresponding point pairs of the telecentric imaging point and the projection point are obtained. Given the camera image coordinates (u c , v c ) and the corresponding horizontal coordinate u p of the projector, the 3D coordinates of the object can be calculated by the following formula:

[0137]

[0138] Among them: [] -1 represents matrix inversion operation, f x is the effective focal length parameter of the intrinsic matrix of the projector, (x p , y p , z p ) T while the three-dimensional reconstruction coordinates in the projector coordinate system, (u c , v c ) T are the camera coordinates of the object to be measured, (u p , v p ) T are the corresponding projector coordinates, is is

[0139] It can be seen from formula (17) that during the reconstruction process, almost all the parameters of the telecentric projection matrix M c 3×4 = [m c ij are used, indicating that they are crucial for the reconstruction accuracy.

[0140] It can be seen from the above implementation manners that all the parameters of the telecentric projection matrix are involved in the three-dimensional reconstruction process, indicating that they are the key factors affecting the reconstruction accuracy. Compared with the traditional three-dimensional reconstruction imaging, the calculation of formula (17) is more concise and efficient.

[0141] Further analysis shows that the use of Scheimpflug projection can solve the reconstruction error caused by defocus of the projection and imaging branches and increase the dynamic measurement range.

[0142] Further analysis shows that calibrating the telecentric imaging with a precisely calibrated projector can avoid the model error caused by separate calibration of internal parameters; at the same time, by utilizing the characteristics of telecentric orthogonal imaging, high-precision and low-distortion three-dimensional measurement can be achieved.

[0143] The present invention is not limited to the above embodiments. Based on the technical solutions disclosed in the present invention, those skilled in the art can make some substitutions and deformations to some of the technical features without creative labor according to the disclosed technical content, and these substitutions and deformations are all within the protection scope of the present invention.

Claims

1. A combined calibration method for a telecentric imaging and Scheimpflug projection three-dimensional microscopic measurement system, characterized in that, Including: Establish a Scham projection model, a Scham distortion model, and a telecentric imaging model; Use a telecentric camera to capture and preprocess the target pose images of the calibration board, and use the multi-frequency heterodyne method to perform phase unwrapping of the center of the telecentric camera; According to the obtained center phase, perform projector center mapping through the local homography method; According to the established Scham projection model and the mapped projector center, use the maximum likelihood estimation method to estimate the internal parameter intrinsic matrix, external parameter matrix, and distortion coefficient of the projector; According to the established Scham distortion model and the obtained distortion coefficient, correct the projector distortion and Scham angle; Optimize the error of the projector external parameter matrix obtained at different poses through the PnP method; Including: Iteratively minimize the difference between the observed projection points and the calculated projection points, and this difference is expressed as: where ||·|| is the least square distance, is the center coordinate of the projector after distortion correction, is the extrinsic parameter matrix of the projector, is the coordinate of the center in the world coordinate system; According to the projector external parameter matrix, align the three-dimensional coordinates of the center of the calibration board to the coordinate system of the projector; Use the direct linear transformation and singular value decomposition method to calibrate the projection matrix of the telecentric imaging model according to the aligned three-dimensional coordinates of the center; According to the obtained projection matrix of the telecentric imaging model, the internal parameter intrinsic matrix, external parameter matrix, and distortion coefficient of the projector, use a non-linear method to perform camera-projector joint calibration to achieve high-precision calibration of the three-dimensional microscopic measurement system.

2. The combined calibration method according to claim 1, wherein Establish a Scham projection model and a distortion model. When the tilt angle between the imaging plane and the lens plane ≤ 6°, compensate for the tilt effect by adding additional aberration distortion parameters. The distortion model is: where p p is the DMD pixel coordinate of the projector, is the DMD coordinate after the Scheimpflug distortion, is the normalized DMD coordinate of the actual distortion, is the homography transformation between the ideal projection DMD plane and the Scheimpflug projection DMD plane.

3. The combined calibration method according to claim 1, characterized in that The telecentric imaging model is: Among them, K c is the intrinsic matrix of the telecentric camera, m x and m y are the effective magnification factors of the telecentric lens in the X and Y directions respectively, q c is the homogeneous image coordinate, u c 0, v c 0 is the principal point of the projection center, x c , y c are the X coordinate and Y coordinate of the object point in the camera coordinate system respectively.

4. The combined calibration method according to claim 1, wherein Capture and preprocess each target pose image, and capture and preprocess the target pose images of the calibration board, including: Capture a set of images for each target pose, including white light projection, horizontal stripe pattern, and vertical stripe pattern; the white light projection is used to extract the center of the circle captured by the camera as a feature point, and the horizontal and vertical stripe patterns are used to solve the horizontal and vertical phases of the center of the circle.

5. The combined calibration method according to claim 1, characterized in that Perform projector center mapping, including: adopting the local homography method, using the whole pixel phase values in the neighborhood of the camera center to solve the coordinate values of the corresponding projector, constructing a local mapping relationship from the camera coordinates to the projector coordinates, and solving the accurate camera-projector sub-pixel center mapping.

6. The combined calibration method according to claim 1, characterized in that Align the three-dimensional coordinates of the center of the calibration board to the coordinate system of the projector. Use the external parameter matrix of the projector obtained at different poses of the calibration board to align the world coordinates of the calibration board center with the projector coordinate system, and the projector external parameter matrix becomes the unit matrix and the zero vector matrix.

7. The combined calibration method according to claim 1, wherein Perform camera-projector joint calibration using the following formula: Wherein: is the DMD sub-pixel coordinate of the projector, is the coordinate of the center feature point in image i, is the camera pixel coordinate of the calibration plate center in image i, is the image coordinate of the center point in image i, K p is the internal reference intrinsic matrix of the projector.

8. The joint calibration method according to any one of claims 1-7, characterized in that, Including: Use the stereo vision algorithm to reconstruct the 3D topography of the target pose image and reconstruct the microscopic 3D target pose image: Solve the corresponding point pairs of the telecentric imaging point and the projection point by the phase of the projected fringe. Given the camera image coordinates (u c , v c ) and the corresponding horizontal coordinates u p of the projector, the 3D coordinates of the target pose image are calculated by the following formula: Wherein: -1 is the matrix inverse operation, f x is the effective focal length parameter of the intrinsic matrix of the projector, (x p , y p , z p ) T is the three-dimensional reconstruction coordinate in the projector coordinate system, (u c , v c ) T is the camera coordinate of the object to be measured, (u p , v p ) T is the corresponding projector coordinate, is the abscissa of the principal point of the projector, is the telecentric projection matrix parameter.

9. A telecentric imaging combined with Scheimpflug projection three-dimensional microscopic measurement system, characterized in that, Apply the joint calibration method according to any one of claims 1-7, including a projection branch, an imaging branch, and a wedge arranged at an angle to each other; The projection branch includes a projection lens and a DMD digital projection chip, and the DMD digital projection chip generates a digital stripe pattern and projects it onto the object surface of the object to be measured; The imaging branch includes a telecentric imaging lens and a high-resolution color camera. The high-resolution color camera captures the deformed fringe pattern reflected from the object surface through the telecentric imaging lens, and reconstructs the surface topography through the system model and the phase recovery algorithm; The wedge is arranged between the projection lens and the DMD digital projection chip and is used for shifting the imaging object plane. The extension lines of the Scheimpflug shifted object plane, the projection lens principal plane, and the DMD digital projection chip plane intersect at the Scheimpflug coincidence point on the object plane.

Images