Multi-angle moving-axis fringe projection three-dimensional measurement system and calibration method

Abstract

The application discloses a kind of multi-angle shift axis fringe projection three-dimensional measurement system and calibration method, system includes imaging branch, is equipped with high-resolution color camera, telecentric imaging lens and multichannel RGBW ring light source, high-resolution color camera is captured by telecentric imaging lens and is deformed fringe pattern reflected by object surface, reconstructs surface topography;Projection branch, respectively by projection lens and DMD digital projection chip Multi-angle projection branch is formed, and word fringe pattern is generated by DMD digital projection chip and is projected to object plane;By introducing shift axis angle, adjust projection focal plane and object plane are coincident.Method includes establishing shift axis projection model, distortion model and telecentric imaging model;Camera-projector system calibration parameter is optimized jointly;Global coordinate system is realized by using projector external parameter matrix, and global registration error is compensated.The application can expand measurement range, compensate the data loss caused by shading shadow, improve measurement accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of surface measurement of complex structures in precision manufacturing, and in particular to a multi-angle shift fringe projection three-dimensional measurement system and a calibration method. Background Art

[0002] With the rapid development of precision manufacturing and semiconductor manufacturing, 3D imaging and measurement of complex microstructures are becoming increasingly important. Most optical methods used for microscopic 3D microscopy are divided into two categories based on their principles: interferometry and triangulation. Phase-shift interferometry, white-light interferometry, and digital holographic microscopy are based on the principle of optical interference, while fringe projection and stereo vision measurement use spatial triangulation. Due to the characteristics of coherent measurement, interferometry has nanometer depth resolution, but the field of view (FOV) is very small. In contrast, triangulation can provide a field of view of millimeters to centimeters and micrometer-level depth resolution.

[0003] Compared to interferometry and stereo vision measurement, fringe projection profilometry (FPP) is insensitive to variations in background, contrast, and noise, making it widely applicable in complex industrial environments. Fringe projection 3D microscopy typically utilizes two different system frameworks. One involves modifying one channel of a stereo microscope using different projection technologies. Due to limitations in the objective lens aperture and imaging principles, the imaging field of view and depth of field are severely restricted to the micrometer or submillimeter scale. The other utilizes non-telecentric lenses with long working distances, but their depth of field decreases with increasing magnification, making it difficult to measure objects with millimeter-scale height variations. Telecentric lenses have been adopted for fringe projection 3D microscopy due to their superior properties, such as orthogonal projection, low distortion, and constant magnification. Since different parameters can be selected for projection and telecentric imaging, greater flexibility is provided in terms of magnification matching, working distance adjustment, and system design.

[0004] However, current fringe projection 3D microscopy still has some shortcomings that need to be further addressed. First, the measurement depth range is limited. Almost all fringe projection 3D microscopy systems using telecentric lenses consist of a projection branch and an imaging branch, with a certain angle between them. This will cause a small part of the projected fringe to be in focus, while the rest of the area will remain defocused. In addition, the limited depth of field of the telecentric lens will further significantly reduce the common focus area between the projector and the camera. The second problem is occlusion. Due to the principle of triangulation, when the camera or projector FOV is blocked, the shadow area in the image cannot be reconstructed. Third, fringe projection, as a typical directional lighting, produces more local specular reflections and local area scattering on the surface of the object. Therefore, a large number of overexposed pixels may appear in the captured image, resulting in artifacts in 3D reconstruction.

[0005] Furthermore, for the calibration of multi-angle tilt-shift structured light 3D microscopy systems, orthogonal imaging with telecentric lenses results in insensitivity to depth variations along the optical axis, and currently no unified and comprehensive calibration method exists. Tilt-shift projection also lacks a unified description model. Due to strong constraints (such as the orthogonality of the rotation matrix), existing calibration methods struggle to achieve high-precision calibration of the extrinsic matrix parameters for telecentric imaging. Furthermore, the intrinsic and extrinsic parameters are naturally coupled, further complicating the calibration process and increasing calibration uncertainty. Summary of the Invention

[0006] To address the aforementioned deficiencies in the prior art, one of the objectives of the present invention is to provide a multi-angle tilt-shift fringe projection 3D measurement system. The system utilizes tilt-shift imaging and multi-angle projection, and employs a high-resolution telecentric camera for imaging. Based on the principle of tilt-shift imaging, a special angle is introduced between the digital microlens (DMD) chip and the projection lens, expanding the shared focal area between them. By leveraging the characteristics of multi-angle projection, the measurement range can be expanded, data loss caused by occlusions can be compensated, and measurement accuracy can be improved.

[0007] Another object of the present invention is to provide a calibration method for a multi-angle shift fringe projection three-dimensional measurement system, which fully utilizes the established shift projection model to calibrate telecentric imaging and perform joint optimization.

[0008] The present invention is achieved through the following technical solutions.

[0009] According to one aspect of the present invention, a multi-angle shift fringe projection three-dimensional measurement system is provided, comprising:

[0010] The imaging branch is equipped with a high-resolution color camera, a telecentric lens, and a multi-channel RGBW ring light source. The high-resolution color camera captures the deformed fringe pattern reflected from the object surface through the telecentric imaging lens and reconstructs the surface topography.

[0011] The projection branch is composed of a projection lens and a DMD digital projection chip to form a multi-angle projection branch. The DMD digital projection chip generates a digital fringe pattern and projects it onto the object plane.

[0012] Multiple projection branches are respectively equipped with wedge-shaped adjusters, which can adjust the projection focal plane and the object plane to coincide with each other by introducing a shift angle.

[0013] According to an exemplary embodiment of the present invention, the optical axis of the imaging branch is arranged perpendicularly at an angle of 90° relative to the object plane, and the axis of the multi-channel RGBW ring light source coincides with the optical axis of the imaging branch.

[0014] The object plane, the main plane of the projection lens and the plane of the DMD digital projection chip intersect at the shift intersection line.

[0015] In the imaging branch, the telecentric imaging resolution is greater than the projector resolution; in the projection branch, the projector depth of field is greater than the telecentric imaging depth of field.

[0016] According to another aspect of the present invention, a calibration method for a multi-angle tilt-shift fringe projection three-dimensional measurement system is provided, comprising:

[0017] Establish tilt-shift projection model, distortion model and telecentric imaging model;

[0018] The calibration plate image and multi-frequency phase-shift fringe image are collected using a high-resolution color camera and filtered preprocessed.

[0019] The DMD digital projection chip is used to extract the phase of the center of the feature point of the calibration plate image after filtering, and the sub-pixel coordinates of the center of the DMD plane feature point are obtained using the local homography method;

[0020] According to the sub-pixel coordinates of the center of the DMD plane feature point and the tilt-shift projection model, the projector intrinsic parameter matrix is ​​calibrated to obtain the initial value of the projector extrinsic parameter matrix;

[0021] The distortion model is used to correct the distortion and shift angle of the projector, and the center coordinates of the corrected projector feature points are obtained;

[0022] Optimize the projector extrinsic matrix using the PnP method;

[0023] Calculate the three-dimensional coordinates of the center of the projector feature point according to the optimized external parameter matrix of the projector;

[0024] Substitute the three-dimensional coordinates of the center of the projector feature point into the telecentric imaging model to solve the camera telecentric projection matrix;

[0025] According to the camera telecentric projection matrix and the projector intrinsic and extrinsic parameter matrices, the camera-projector system calibration parameters are jointly optimized using the bundle adjustment method;

[0026] A multi-angle shift fringe projection 3D measurement model is established according to the system calibration parameters. The projector extrinsic matrix is ​​used to realize the global coordinate system 1 and compensate for the global registration error.

[0027] According to an exemplary embodiment of the present invention, when the tilt angle between the imaging plane and the lens plane is small, an additional aberration distortion parameter is added to compensate for the tilt effect, thereby constructing a tilt-shift projection model.

[0028] According to an exemplary embodiment of the present invention, phase extraction is performed on the center of the feature point of the calibration plate image after filtering, and the sub-pixel coordinates of the center of the feature point on the DMD plane are obtained using a local homography method, including:

[0029] For the filtered calibration plate fringe image, the absolute phase value of the feature point center is extracted using the multi-frequency heterodyne phase shift method, and the corresponding projector DMD center integer pixel coordinates are calculated;

[0030] The local homography method is used to calculate the coordinates of the DMD center, obtain the corresponding projector pixel coordinate values, and establish a local homography relationship from the camera to the DMD plane;

[0031] Solve the corresponding optimal local homography matrix for each circle center neighborhood:

[0032] The random sampling consensus algorithm is used to solve the optimal local homography matrix and calculate the sub-pixel coordinates of the center of the DMD plane feature point.

[0033] According to an exemplary embodiment of the present invention, calibrating the projector intrinsic parameter matrix to obtain the initial value of the projector extrinsic parameter matrix includes:

[0034] The precise DMD coordinates of the tilt-shift projector corresponding to the feature points are obtained by the local homography method;

[0035] Using the projector center at different poses, calculate the closed-form solution of the intrinsic parameter matrix of the mth projector;

[0036] The distortion coefficient and shift angle are iteratively solved using the Levenberg-Marquardt algorithm.

[0037] The present invention adopts the above technical solution, which has the following beneficial effects:

[0038] 1. The system of the present invention adopts the tilt-shift imaging condition, which can solve the problem of too small depth of field under the condition of small field of view and high magnification, so that the projection focal plane and the imaging focal plane remain coincident, overcoming the reconstruction error caused by the defocus of the projection and imaging branches, making full use of the limited degree of freedom of the telecentric lens, and increasing the dynamic measurement range.

[0039] 2. The system of the present invention consists of a camera with a telecentric lens and multiple projectors using tilt-shift lenses. Multi-angle tilt-shift projection effectively reduces shadows caused by occlusion, improving measurement integrity. It also overcomes artifacts caused by directional lighting in 3D reconstruction and expands the system's measurement depth of field.

[0040] 3. The method of the present invention constructs a new mathematical model of tilt-shift projection and makes full use of the established tilt-shift projection model and distortion model to calibrate telecentric imaging. It further reduces the camera-projector joint calibration error through nonlinear optimization, and ultimately achieves high-precision system calibration.

[0041] 4. The method of the present invention constructs and compensates for the registration error, thereby improving the accuracy of complete measurement of three-dimensional morphology.

[0042] 5. The present invention also designs a design process and parameter determination criteria for a multi-angle shift fringe projection three-dimensional measurement system, which can quickly realize the construction and integration of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute an improper limitation of the present invention. In the drawings:

[0044] Figure 1 It is a schematic diagram of the system structure of the present invention;

[0045] Figure 2 2. It is a schematic diagram of overlapping depth of field of the tilt-shift projection of the present invention;

[0046] Figure 3 This is a schematic diagram of the cross-sectional optical path of the tilt-shift projection of the present invention;

[0047] Figure 4 This is a comparison chart of the image quality of the tilt-shift projection of the present invention;

[0048] Figure 5 It is a flow chart of the system calibration of the present invention;

[0049] Figure 6 This is a schematic diagram of the optical path of the tilt-shift projection of the present invention;

[0050] Figure 7 Positional relationship and error diagram of a multi-angle tilt-shift projector.

[0051] In the figure: 1-high-resolution color camera, 2-telecentric imaging lens, 3-projection lens A, 4-projection lens B, 5-DMD digital projection chip A, 6-DMD digital projection chip B, 7-DMD digital projection chip plane A, 8-projection lens A principal plane, 9-DMD digital projection chip plane B, 10-projection lens B principal plane, 11-shift intersection line A, 12-shift intersection line B, 13-object plane, 14-measured object, 15-multi-channel RGB ring light source, 16-projection and imaging focus plane, 17-imaging branch depth of field, 18-projection branch depth of field, 19-common focus area, 20-ideal DMD plane, 21-DMD plane after shifting, 22-DMD chip principal point. DETAILED DESCRIPTION

[0052] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The exemplary embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0053] The present invention provides a multi-angle shift fringe projection three-dimensional measurement system, Figure 1This paper demonstrates a specific embodiment of the present invention. The entire system includes a high-resolution color camera 1, a telecentric imaging lens 2, projection lenses A3 and B4, a DMD digital projection chip A5, and a DMD digital projection chip B6. The system consists of an imaging branch and multiple projection branches.

[0054] The imaging branch includes a high-resolution color camera 1, a telecentric imaging lens 2, and a multi-channel RGBW ring light source 15, which are arranged in sequence. The multi-channel RGBW ring light source 15 is arranged in front of the telecentric imaging lens 2. The optical axis of the imaging branch is perpendicularly configured at a 90-degree angle relative to the object plane 13. The high-resolution color camera 1 captures the deformed fringe pattern reflected from the object surface through the telecentric imaging lens 2. Then, the surface topography can be reconstructed using an appropriate system model and phase recovery algorithm.

[0055] The high-resolution color camera 1 and telecentric imaging lens 2 are also used to collect color information of the object being measured. The axis of the multi-channel RGBW ring light source 15 coincides with the optical axis of the imaging branch to assist the imaging branch in collecting color images of the object being measured.

[0056] The multi-channel RGBW ring light source 15 emits light of different wavelengths in sequence, which is reflected by the surface of the object and collected by the imaging branch to obtain the color texture information of the object surface. The pixels of the image are matched with the three-dimensional point cloud according to the calibration information, and the point cloud is color-textured.

[0057] In the projection branch, the multi-angle projection branch is composed of projection lens and DMD digital projection chip. Figure 1 The figure shows two projection branches, one each consisting of a projection lens A3 and a DMD digital projection chip A5, and one consisting of a projection lens B4 and a DMD digital projection chip B6. However, the number of projection branches is not limited to two; multiple projection branches can be provided. The DMD digital projection chip generates and projects a digital fringe pattern onto object plane 13.

[0058] The projection branch adopts a tilt-shift structure to solve the problem of too small depth of field in the case of small field of view and high magnification, overcome the reconstruction error problem caused by defocus of the projection and imaging branches, make full use of the limited degree of freedom of the telecentric lens, and increase the dynamic measurement range.

[0059] After the multiple projection branches are designed under the shift imaging condition (i.e., when the object plane, the main plane of the projection lens and the extended plane of the DMD plane intersect in a line, a clear image of the entire field can be obtained), they are projected at a specific tilt angle β.

[0060] Wedge adjusters are respectively arranged between the projection lens A3, the projection lens B4 and the DMD digital projection chip A5, the DMD digital projection chip B6 for adjusting the shift angle.

[0061] The design of the shift angle needs to follow that the extension planes of the object plane 13, the main plane 8 of the projection lens A and the plane A7 of the DMD digital projection chip intersect at the shift intersection line A11, and the extension planes of the object plane 13, the main plane 10 of the projection lens B and the plane B9 of the DMD digital projection chip intersect at the shift intersection line B12.

[0062] Please refer to Figure 2 Based on the shift imaging principle, a special angle (i.e. the projection lens is inclined relative to the DMD plane) is introduced between the DMD chip and the projection lens, so that the projection focal plane and the imaging focal plane coincide, that is, the projection and imaging focusing plane 16.

[0063] The imaging branch depth of field 17 and the projection branch depth of field 18 can be adjusted to overlap to obtain a larger common focusing area 19, thereby allowing each projector to perform focused projection on the object surface in the entire FOV at the same time.

[0064] The present application deploys multiple shift projection branches to establish a multi-angle shift projection microscopic measurement system. The design of the multi-angle shift projection framework is based on the following considerations: first, multi-angle projection can effectively reduce the shadow area caused by occlusion and improve the integrity of the measurement by providing projection patterns in different directions; second, multi-angle projection is an effective method to reduce local reflection and scattering of the workpiece; in addition, by superimposing the depth of field of multiple projection branches, the measurable depth range of the system can be expanded.

[0065] On the basis of the above analysis, the present application gives the overall design criteria and basis of the multi-angle shift fringe projection three-dimensional measurement system.

[0066] 1. Determine the telecentric imaging FOV, the projection FOV and the working distance, and then determine the vertical magnification M of the projector.

[0067] 2. Determine the tilt angle β of the object plane according to the phase-height sensitivity.

[0068] 3. Determine the tilt angle θ of the projection lens according to formula (1).

[0069] The tilt angle β of the object plane and the tilt angle θ of the projection lens satisfy the following relationship.

[0070] θ = arctan (M * tan (β)) (1)

[0071] In the figure, θ is the shift angle, M = pi / po is the nominal vertical magnification of the optical axis, and needs to satisfy the Gaussian conjugate formula

[0072] 4. According to the three-dimensional imaging depth of field, use formula (2) and (3) to determine the telecentric imaging depth of field (Δ c ) and the projector depth of field (Δ p).

[0073] Cross-section under tilt-shift imaging conditions, such as Figure 3 As shown. The near limit and far limit (dashed line) of the depth of field intersect at the hinge line (H). Theoretically, the projection depth of field range under the tilt-shift condition is as follows Figure 3 As shown in the shaded area in the figure. Consider only the near and far limits of the depth of field without shifting ( and ), vertical offset limit (Δ s ) can be approximately calculated as:

[0074]

[0075] It should be noted that the actual vertical offset limit is greater than the calculated value. In addition, for the entire system, the vertical offset limit of the common focus area is also affected by the telecentric imaging depth of field (Δ c ) and field of view limitations.

[0076] Focus projection can be achieved throughout the common focus area, but the common focus area outside the perfect focus plane cannot meet the precise shift conditions. In addition, the telecentricity of telecentric imaging is limited (the incident light is not completely parallel to the optical axis within the depth of field), and the imaging resolution will be reduced outside the perfect focus plane.

[0077] Based on the above analysis, in order to ensure accurate three-dimensional imaging, two conditions are proposed for system design: (1) The telecentric imaging resolution is greater than the projector resolution. (2) The projector depth of field (Δ p ) is greater than the telecentric imaging depth of field (Δ c ), and satisfies the following empirical equation.

[0078] k1·Δ p cos(β)=k2·Δ c (3)

[0079] Among them, k1 and k2 are empirical parameters, and it is recommended that k1 be 0.3-0.5 and k2 be 0.5-0.8. This means that the 3D imaging depth of field only occupies the telecentric imaging depth of field (Δ c ) part, and the vertical offset limit (Δ s ) needs to be greater than the 3D imaging depth of field. In actual operation, the telecentric lens depth of field (Δ c ), the middle part is used as the depth of field of 3D imaging (i.e., parameter k2 is determined). Then, the vertical offset limit (Δ s ) (i.e., determine the parameter k1). Finally, calculate the required projector depth of field (Δ p ).

[0080] 5. Design relevant optical parameters of the projector (such as focal length, working distance, aperture, etc.).

[0081] 6. Design the direction and position of multi-angle projections to maximize the overlap of common focus areas.

[0082] The contrast effect of tilt-shift projection is as follows Figure 4 As shown in Figure 2. Under the same experimental conditions, the binary fringe patterns of forward projection and tilt-shift projection are compared. It can be seen that because the focal plane of forward projection is not perpendicular to the camera's optical axis, a uniformly clear image cannot be obtained, and the fringe edges become increasingly blurred. In contrast, the captured tilt-shift projection image has clear fringe edges, and the overall image contrast and quality are better than those of forward projection. As the vertical coordinate increases, the fringe edge profiles of the forward projection become more defocused. Clearly, the projection pattern quality of the tilt-shift optical projection system is significantly better than that of the traditional optical projection system.

[0083] The multi-angle shift fringe projection three-dimensional measurement system of the present invention, due to the use of a telecentric lens in the imaging branch, will result in insensitivity to depth changes along the optical axis. Currently, there is no mature calibration algorithm for the calibration of telecentric lens imaging. Related calibration algorithms either require the use of a high-precision translation stage for system calibration, which is usually difficult to set up (for example, the movement direction is completely perpendicular to the Z axis) and expensive, or due to strong constraint requirements (such as the orthogonality of the rotation matrix), it is difficult to achieve high-precision calibration of external parameters. In addition, because internal and external parameters are naturally coupled together and difficult to separate, the calibration process is further complicated, increasing the uncertainty of modeling accuracy.

[0084] In order to solve the limitations of the above-mentioned system calibration, the present invention also provides a joint calibration method for a multi-angle shift fringe projection three-dimensional measurement system.

[0085] The calibration flow chart is as follows Figure 5 As shown, a joint calibration method for a multi-angle tilt-shift fringe projection 3D measurement system is implemented as follows:

[0086] Step 1: Establish the tilt-shift projection model, distortion model, and telecentric imaging model.

[0087] The pinhole model can be used to describe the tilt-shift projection process, such as Figure 6 As shown. w =X w Y w Z w is the world coordinate system, is the projector coordinate system, the superscript represents the mth projector (m=1,2,3,4...), Is the projection center of the projector. The DMD plane is at z c =f. In zc = 1, and the virtual normalized DMD plane is parallel to the DMD plane. Assume that there is a point p on the DMD plane pm The three-dimensional coordinate projected into space is p w , considering the imaging distortion introduced by the tilt-shift angle, the following tilt-shift projection model is established:

[0088]

[0089] Where s is the scaling factor, represents the two-dimensional coordinates of a point on the DMD plane of the mth projector, p w Indicates the three-dimensional coordinates of the object point projected onto the surface of the object in the world coordinate system. is the intrinsic parameter matrix of the m-th projector, and Tilt-Shift Angle The introduced projection matrix and brick selection matrix, and is a 3×3 rotation matrix and a 3×1 translation vector.

[0090] Since lens distortion changes the direction of light, the actual projection process is nonlinear. This paper introduces second-order radial and tangential distortion and establishes the following distortion model:

[0091]

[0092] in, is the ideal distortion-free normalized DMD coordinate, is the normalized DMD coordinate after distortion, radial distance k1, k2 and p1, p2 are the radial and tangential distortion coefficients respectively.

[0093] Since lens distortion changes the direction of light, the actual projection process is nonlinear. This paper introduces second-order radial and tangential distortion and establishes the following distortion model:

[0094]

[0095] in, is the ideal distortion-free normalized DMD coordinate, is the normalized DMD coordinate after distortion, radial distance k1, k2 and p1, p2 are the radial and tangential distortion coefficients respectively.

[0096] The telecentric imaging model of a telecentric camera from 3D object points to 2D camera image coordinates can be described as follows:

[0097]

[0098] in, c and q w Represent the coordinates in the camera coordinate system and the coordinates in the world coordinate system, K c is the camera intrinsic parameter matrix, and is a 3×3 rotation matrix and a 3×1 translation vector, is the telecentric projection matrix, m c ij is the projection matrix parameter, which contains the intrinsic parameter matrix K of the telecentric camera c and the external parameter matrix m x and m y are the effective magnifications in the X and Y directions, is the camera principal point coordinate. It is worth noting that, unlike the pinhole imaging model, the telecentric projection matrix has only eight valid values.

[0099] Step 2: Use a high-resolution color camera to capture the calibration plate image and multi-frequency phase-shifted fringe image, and perform filtering preprocessing.

[0100] This paper uses a black and white circular calibration plate as a calibration target. The calibration target is placed in different spatial orientations, and a set of images is collected at each target position, including white light projection, horizontal stripe pattern, and vertical stripe pattern. White light projection is used for illumination, allowing the camera to capture the center of the circle as a feature point. (The superscript c indicates a telecentric camera, and the subscripts i and j indicate the i-th spatial position and j-th feature point of the calibration plate, respectively.) The horizontal and vertical fringe patterns are used to solve the horizontal and vertical absolute phase maps.

[0101] In addition, the captured image usually has noise, so the captured image needs to be pre-processed before solving the phase. In the present invention, Gaussian filtering is used to remove high frequency and stray noise.

[0102] Step 3: Use the DMD digital projection chip to extract the phase of the center of the feature point of the calibration plate image after filtering, and use the local homography method to obtain the sub-pixel coordinates of the center of the DMD plane feature point.

[0103] For the calibration plate fringe image filtered in step 2, the center of the feature point is extracted using the multi-frequency heterodyne phase shifting technique. The absolute phase value of the m-th projector DMD circle center integer pixel coordinates are further calculated by the following formula

[0104]

[0105] in and Represents the vertical and horizontal absolute phases of the center of the circle, n v and n h denote the number of horizontal and vertical pattern fringes of the projection, H p ×W p is the projector resolution.

[0106] In order to obtain more accurate sub-pixel coordinates, the present invention adopts a local homography method to calculate the DMD center coordinates. First, use the center The absolute phase of the integer pixels in the neighborhood (k×k, k is the pixel range of the selected neighborhood) is used to obtain the corresponding projector pixel coordinate value through equation (7), and then a local homography relationship is established from the camera to the DMD plane. The following equation is the optimization cost function to solve the corresponding optimal local homography matrix for each circle center neighborhood:

[0107]

[0108] Where, are the homogeneous coordinates of the integer pixels within the circle center area, is the pixel homogeneous coordinate of the mth projector corresponding to the point, argmin represents the variable value when the objective function takes the minimum value, and ||·|| represents the Euclidean distance of the vector.

[0109] The present invention uses the random sampling consensus algorithm (RANSAC) to solve the optimal local homography matrix The sub-pixel coordinates of the center of the DMD plane feature point are further calculated by the following formula:

[0110]

[0111] Step 4: Based on the sub-pixel coordinates of the center of the DMD plane feature point and the tilt-shift projection model, calibrate the distortion coefficient and tilt-shift angle of the projector intrinsic parameter matrix, and obtain the initial value of the projector extrinsic parameter matrix.

[0112] The local homography method in step 3 can be used to obtain the precise DMD coordinates of the tilt-shift projector corresponding to the feature points. Using the center of the projector at different postures, the intrinsic parameter matrix of the mth projector can be calculated: In the present invention, the distortion coefficient and the shift angle are iteratively solved for the following equation using the Levenberg-Marquardt algorithm.

[0113]

[0114] in is the coordinate of the center of the feature point on the DMD plane calculated according to equations (4) and (5), is the axis-shift projection distortion vector. n represents the number of different spatial positions, and m represents the number of feature points on the calibration plate. The initial values ​​of k1, k2, p1, and p2 can be set to zero. and The initial value of can be set to the theoretical design value. At the same time, the external parameter matrix of the projector is obtained The initial value of .

[0115] Step 5: Use the distortion model to correct the distortion and shift angle of the projector to obtain the corrected center coordinates of the projector feature points.

[0116] Apply the distortion coefficient and shift angle obtained in step 4 to equations (4) and (5) to obtain the corrected coordinates of the center of the projector feature point:

[0117] Step 6: Use the PnP method to optimize the projector extrinsic parameter matrix.

[0118] The n-point perspective method (Perspective-n-point, PnP) is used to iteratively calculate the minimum distance between the feature point coordinates and the reverse projection, and the external parameter matrix of the feature points at different spatial positions is calculated. In the present invention, the following formula is used to optimize the projector external parameter matrix:

[0119]

[0120] Among them, ||·|| represents the least squares distance, is the three-dimensional coordinate of the feature point in the world coordinate system.

[0121] Step 7: Calculate the three-dimensional coordinates of the center of the projector feature point based on the optimized projector's external parameter matrix.

[0122] Using the projector extrinsic matrix obtained for different positions in step 6, align the world coordinates of the feature point with the mth projector coordinate system, where the subscripts i and j represent the i-th spatial position and j-th feature point of the calibration plate, respectively. Then the projector extrinsic matrix becomes [I 3×3 ,0 3×1 ](I 3×3 is the identity matrix, 0 3×1 is a zero vector), the 3D coordinates of the feature points are transformed into the projector coordinate system. In essence, the optimization method iteratively minimizes the difference between the observed projection point and the calculated projection point. The 3D coordinates of the center of the feature point in the projector coordinate system are It can be calculated by the following formula:

[0123]

[0124] Step 8: Substitute the 3D coordinates of the projector feature point circle center into the telecentric imaging model to solve the camera telecentric projection matrix.

[0125] The 3D coordinates of each spatial position feature point are determined. Through direct linear transformation (DLT), the imaging model (formula (6)) of the telecentric camera in the mth projector coordinate system can be rewritten as:

[0126]

[0127] wherein, is the feature point coordinate captured by the camera, is an array matrix of 3D coordinates of the feature point, is the telecentric projection matrix parameter in the mth projector coordinate system.

[0128] In the present application, all feature points are substituted into equation (13), and the least squares solution is calculated by using the singular value decomposition (SVD) method, so as to obtain the telecentric projection matrix, which contains the accurate position relationship between the projector and the camera.

[0129] Step 9: According to the camera telecentric projection matrix and the internal and external parameter matrices of the projector, the bundle adjustment method is used to jointly optimize the camera-projector system calibration parameters.

[0130] Telecentric lens distortion is inevitable. Therefore, considering the lens distortion, the telecentric camera is first subjected to nonlinear optimization.

[0131]

[0132] wherein is the coordinate of the feature point on the camera imaging plane calculated according to formula (6), is the image coordinate of the feature point circle center on the camera imaging plane, and the distortion coefficient The initial value of the distortion coefficient

[0133] The purpose of camera-projector joint optimization is to improve the calibration accuracy by simultaneously minimizing the re-projection errors of telecentric imaging and off-axis projection. In the present application, the bundle adjustment method is used to jointly optimize the system calibration parameters, and the cost function of joint optimization can be expressed as follows, which can be solved by the Levenberg-Marquardt nonlinear optimization algorithm.

[0134]

[0135] wherein m represents the number of feature points on the calibration board, and n represents the number of different spatial positions, is the point on the i th spatial position and the j th feature point on the DMD plane, is the calculated coordinate of the feature point on the DMD plane.

[0136] Step 10: Establish a multi-angle shift fringe projection 3D measurement model based on the system calibration parameters, use the projector extrinsic matrix to realize the global coordinate system 1, and compensate for the global registration error.

[0137] After the system calibration is completed, a stereo vision algorithm is used for 3D reconstruction. In the present invention, the multi-angle shift fringe projection 3D measurement model is as follows:

[0138]

[0139] in, is the three-dimensional homogeneous coordinate in the m-th projector coordinate system, and are respectively the projection matrices of telecentric imaging and tilt-shift projection in the mth projector coordinate system, is the internal parameter matrix of the m-th projector, (u c ,v c ) and (u pm ,v pm ) are the camera image coordinates and projector DMD coordinates respectively.

[0140] According to the projector extrinsic matrix obtained in step 6 Align the 3D data in multiple projector coordinate systems to a certain projector coordinate system to achieve global coordinate system 1. For ease of representation and derivation, assume alignment with the first projector coordinate system. The coordinate system transformation is expressed as follows:

[0141]

[0142] in, and is the rotation matrix and translation vector that align the mth projector coordinate system with the first projector coordinate system, is the rotation matrix of the first projector, is the rotation matrix of the m-th projector, is the translation vector of the first projector, is the translation vector of the m-th projector.

[0143] Further, if Figure 7 As shown in the figure, a global registration error compensation model is established to compensate for the registration error. The rotation matrix with error and the translation vector are transformed into and After error correction, the coordinate relationship of the same object point in the 1st and mth projector coordinate systems can be expressed as:

[0144]

[0145] in, and are the rotation matrix and translation vector of the mth projector coordinate system relative to the first one, is the coordinate of the characteristic point of the calibration plate calculated according to formula (12) in the mth projector coordinate system, are the feature point alignment coordinates calculated directly using equation (19).

[0146] In the present invention, all feature points at all spatial positions are introduced, and the global registration error is calculated and compensated by formula (21). The error compensation model is as follows:

[0147]

[0148] in, and are the rotation matrix error and translation vector error of the mth projector coordinate system relative to the first one, is the coordinate of the feature point of the calibration plate calculated in the m-th projector coordinate system, are the feature point alignment coordinates.

[0149] The method of the present invention solves the problem that orthogonal imaging of a telecentric lens leads to insensitivity to depth changes along the optical axis, making it difficult to achieve high-precision calibration of the external matrix parameters of telecentric imaging. It improves the accuracy of complete three-dimensional topography measurement and can quickly realize system construction and integration.

[0150] The present invention is not limited to the above-mentioned embodiments. On the basis of the technical solutions disclosed in the present invention, those skilled in the art can make some substitutions and modifications to some of the technical features therein according to the disclosed technical content without creative labor, and these substitutions and modifications are all within the protection scope of the present invention.

Claims

1. A calibration method for a multi-angle tilt-shift fringe projection 3D measurement system, characterized in that: The system used in the method includes: The imaging branch is equipped with a high-resolution color camera, a telecentric imaging lens, and a multi-channel RGBW ring light source. The high-resolution color camera captures the deformed fringe pattern reflected from the object surface through the telecentric imaging lens and reconstructs the surface topography. The projection branch is composed of a projection lens and a DMD digital projection chip to form a multi-angle projection branch. The DMD digital projection chip generates a word stripe pattern and projects it onto the object plane. Multiple projection branches are respectively equipped with wedge-shaped adjusters, which can adjust the projection focal plane to coincide with the object plane by introducing a shift angle; The method comprises: Establish tilt-shift projection model, distortion model and telecentric imaging model; The calibration plate image and multi-frequency phase-shift fringe image are collected using a high-resolution color camera and filtered preprocessed. The DMD digital projection chip is used to extract the phase of the center of the feature point of the calibration plate image after filtering, and the sub-pixel coordinates of the center of the DMD plane feature point are obtained using the local homography method; According to the sub-pixel coordinates of the center of the DMD plane feature point and the tilt-shift projection model, the projector intrinsic parameter matrix is ​​calibrated to obtain the initial value of the projector extrinsic parameter matrix; The distortion model is used to correct the distortion and shift angle of the projector, and the center coordinates of the corrected projector feature points are obtained; Optimize the projector extrinsic matrix using the PnP method; Calculate the three-dimensional coordinates of the center of the projector feature point according to the optimized external parameter matrix of the projector; Substitute the three-dimensional coordinates of the center of the projector feature point into the telecentric imaging model to solve the camera telecentric projection matrix; According to the camera telecentric projection matrix and the projector intrinsic and extrinsic parameter matrices, the camera-projector system calibration parameters are jointly optimized using the bundle adjustment method; A multi-angle shift fringe projection 3D measurement model is established based on the system calibration parameters, and the projector extrinsic matrix is ​​used to realize the global coordinate system 1 to compensate for the global registration error. Align the 3D data in multiple projector coordinate systems to a certain projector coordinate system to achieve global coordinate system 1. The coordinate system conversion is as follows: in, and is the rotation matrix and translation vector that align the mth projector coordinate system with the first projector coordinate system, is the rotation matrix of the first projector, is the rotation matrix of the m-th projector, is the translation vector of the first projector, is the translation vector of the m-th projector; Establish a global registration error compensation model to compensate for the registration error: in and are the rotation matrix error and translation vector error of the mth projector coordinate system relative to the first one, is the coordinate of the feature point of the calibration plate calculated in the m-th projector coordinate system, are the feature point alignment coordinates.

2. The calibration method of a multi-angle tilt-shift fringe projection 3D measurement system according to claim 1, characterized in that: When the tilt angle between the imaging surface and the lens surface is small, the tilt effect is compensated by adding additional aberration distortion parameters. The tilt-shift projection model is as follows: Where s is the scaling factor, represents the two-dimensional coordinates of a point on the DMD plane of the mth projector, Indicates the three-dimensional coordinates of the object point projected onto the surface of the object in the world coordinate system. is the intrinsic parameter matrix of the m-th projector, and Tilt-Shift Angle The projection matrix and rotation matrix introduced, and is a 3×3 rotation matrix and a 3×1 translation vector; The distortion model is as follows: in, is the normalized DMD coordinate of the actual distortion, is the radial distance, , is the ideal distortion-free normalized DMD coordinate, k 1, k 2 and p 1, p 2 are radial and tangential distortion coefficients respectively; The telecentric imaging model is as follows; in, and Represent the coordinates in the camera coordinate system and the coordinates in the world coordinate system respectively. is the intrinsic parameter matrix of the telecentric camera, and is a 3×3 rotation matrix and a 3×1 translation vector, is the telecentric projection matrix, m c ij is the projection matrix parameter, which contains the intrinsic parameter matrix of the telecentric camera K c and the external parameter matrix , m x and m y are the effective magnifications in the X and Y directions, 、 are the camera principal point coordinates.

3. The calibration method of a multi-angle tilt-shift fringe projection 3D measurement system according to claim 1, characterized in that: The phase of the center of the feature point of the calibration plate image after filtering is extracted, and the sub-pixel coordinates of the center of the feature point on the DMD plane are obtained using the local homography method, including: For the filtered calibration plate fringe image, the absolute phase value of the feature point center is extracted using the multi-frequency heterodyne phase shift method, and the corresponding projector DMD center integer pixel coordinates are calculated; The local homography method is used to calculate the coordinates of the DMD center, obtain the corresponding projector pixel coordinate values, and establish a local homography relationship from the camera to the DMD plane; Solve the corresponding optimal local homography matrix for each circle center neighborhood: The random sampling consensus algorithm is used to solve the optimal local homography matrix and calculate the sub-pixel coordinates of the center of the DMD plane feature point.

4. The calibration method of a multi-angle tilt-shift fringe projection 3D measurement system according to claim 1, characterized in that: Calibrate the projector's intrinsic parameter matrix to obtain the initial value of the projector's extrinsic parameter matrix, including: The precise DMD coordinates of the tilt-shift projector corresponding to the feature points are obtained by the local homography method; Using the projector center at different poses, calculate the closed-form solution of the intrinsic parameter matrix of the mth projector; The distortion coefficient and axis angle are iteratively solved using the Levenberg-Marquardt algorithm.

5. The calibration method of a multi-angle tilt-shift fringe projection 3D measurement system according to claim 1, characterized in that: The bundle adjustment method is used to jointly optimize the calibration parameters of the camera-projector system, including: Perform nonlinear optimization on the telecentric camera, and the optimization formula is as follows: in, are the distortion coefficients, is the image coordinate of the center of the feature point on the camera imaging plane, is the coordinate of the calculated feature point on the camera imaging plane, i and j Respectively represent the first i spatial position and j feature points; The cost function of the joint optimization is solved by the Levenberg-Marquardt nonlinear optimization algorithm: in, m Indicates the number of feature points on the calibration plate, n represents the number of different spatial locations, DMD plane i spatial position and j points on the feature points, is the coordinate of the calculated feature point center on the DMD plane.

6. The calibration method of a multi-angle tilt-shift fringe projection 3D measurement system according to claim 1, characterized in that: The multi-angle tilt-shift fringe projection 3D measurement model is as follows: in, is the three-dimensional homogeneous coordinate in the m-th projector coordinate system, and are respectively the projection matrices of telecentric imaging and tilt-shift projection in the mth projector coordinate system, is the intrinsic parameter matrix of the m-th projector, ( u c , v c )and( u pm , v pm ) are the camera image coordinates and projector DMD coordinates respectively.

7. The calibration method of a multi-angle tilt-shift fringe projection 3D measurement system according to claim 1, characterized in that: The optical axis of the imaging branch is arranged perpendicularly at an angle of 90° relative to the object plane; the axis of the multi-channel RGBW ring light source coincides with the optical axis of the imaging branch.

8. The calibration method of a multi-angle tilt-shift fringe projection 3D measurement system according to claim 1, characterized in that: The object plane, the main plane of the projection lens and the plane of the DMD digital projection chip intersect at the shift intersection line.

9. The calibration method of a multi-angle tilt-shift fringe projection 3D measurement system according to claim 1, characterized in that: In the imaging branch, the telecentric imaging resolution is greater than the projector resolution; in the projection branch, the projector depth of field is greater than the telecentric imaging depth of field.

Images