Multi-angle tilt-shift fringe projection three-dimensional measurement system and calibration method

Abstract

The invention discloses a multi-angle tilt-shift fringe projection three-dimensional measurement system and a calibration method, the system comprises an imaging branch, the imaging branch is provided with a high-resolution color camera, a telecentric imaging lens and a multi-channel RGBW annular light source, and the high-resolution color camera captures a deformed fringe pattern reflected by the surface of an object through the telecentric imaging lens and reconstructs the surface topography; the projection branches respectively form a multi-angle projection branch by a projection lens and a DMD digital projection chip, and the DMD digital projection chip generates a word stripe pattern and projects the word stripe pattern to an object plane; by introducing a tilt angle, a projection focal plane is adjusted to coincide with an object plane. The method comprises the following steps: establishing a tilt-shift projection model, a distortion model and a telecentric imaging model; performing joint optimization on calibration parameters of the camera-projector system; and realizing the unification of a global coordinate system by using a projector extrinsic parameter matrix, and compensating a global registration error. According to the invention, the measurement range can be expanded, data missing caused by shadow shielding is compensated, and the measurement precision is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of surface measurement of complex structures in precision manufacturing, and in particular to a multi-angle axis-shift fringe projection three-dimensional measurement system and a calibration method. Background Art

[0002] With the rapid development of precision manufacturing and semiconductor manufacturing, 3D imaging and measurement of complex microstructures are becoming increasingly important. According to different principles, most optical methods for microscopic 3D microscopic measurement are divided into two categories: interferometry and triangulation. Phase shift interferometry, white light interferometry, and digital holographic microscopy are based on the principle of optical interference, while fringe projection and stereo vision measurement use spatial triangulation. Due to the characteristics of coherent measurement, interferometry has nanometer depth resolution, but the field of view (FOV) is very small. In contrast, triangulation can provide millimeter to centimeter fields of view and micrometer depth resolution.

[0003] Compared with interferometry and stereo vision measurement, fringe projection profilometry (FPP) is insensitive to changes in background, contrast, and noise, so it can be widely used in complex industrial environments. Fringe projection 3D microscopy usually uses two different system frameworks. One is to improve one channel of a stereo microscope by different projection technologies. Due to the limitations of the objective lens aperture and the imaging principle, the imaging field of view and depth of field are severely limited to the micrometer or sub-millimeter level. The other uses a non-telecentric lens with a long working distance, but its depth of field decreases with the increase of magnification, which is not conducive to measuring objects with millimeter-level height changes. Telecentric lenses have been used in fringe projection 3D microscopy measurement technology due to their excellent characteristics such as orthogonal projection, low distortion, and constant magnification. Since different parameters can be selected for projection and telecentric imaging, there is greater flexibility in magnification matching, working distance adjustment, and system design.

[0004] However, the current fringe projection 3D microscope still has some shortcomings that need to be further addressed. First, the measurement depth range is limited. Almost all fringe projection 3D microscope systems using telecentric lenses consist of a projection branch and an imaging branch with a certain angle between them. This will cause a small part of the projected fringe to be in focus, while the remaining area is still in a defocused state. In addition, the limited depth of field of the telecentric lens will further significantly reduce the common focus area between the projector and the camera. The second problem is occlusion. Due to the principle of triangulation, when the camera or projector FOV is blocked, the shadow area in the image cannot be reconstructed. Third, fringe projection, as a typical directional illumination, produces more local specular reflections and local area scattering on the surface of the object. Therefore, a large number of overexposed pixels may appear in the captured image, resulting in artifacts in 3D reconstruction.

[0005] In addition, for the calibration of multi-angle tilt-shift projection structured light 3D microscopy measurement systems, the orthogonal imaging of the telecentric lens will lead to insensitivity to depth changes along the optical axis, and there is currently no unified and complete calibration method. Tilt-shift projection also lacks a unified description model. Due to strong constraints (such as the orthogonality of the rotation matrix), it is difficult for existing calibration methods to achieve high-precision calibration of telecentric imaging external matrix parameters. In addition, intrinsic and extrinsic parameters are naturally coupled together, which further complicates the calibration process and increases the uncertainty of the calibration. Summary of the invention

[0006] In order to solve the above-mentioned defects existing in the prior art, one of the purposes of the present invention is to provide a multi-angle shift fringe projection three-dimensional measurement system. The system framework uses the shift imaging condition and multi-angle projection, and uses a high-resolution telecentric camera for imaging. Based on the principle of shift imaging, a special angle is introduced between the digital microlens (DMD) chip and the projection lens, which expands the common focal area of ​​the two. By making full use of the characteristics of multi-angle projection, the measurement range can be expanded, the data loss caused by occlusion and shadow can be compensated, and the measurement accuracy can be improved.

[0007] Another object of the present invention is to provide a calibration method for a multi-angle shift fringe projection three-dimensional measurement system, which method fully utilizes the established shift projection model to calibrate telecentric imaging and performs joint optimization.

[0008] The present invention is achieved through the following technical solutions.

[0009] According to one aspect of the present invention, a multi-angle shift fringe projection three-dimensional measurement system is provided, comprising:

[0010] The imaging branch is equipped with a high-resolution color camera, a telecentric lens, and a multi-channel RGBW ring light source. The high-resolution color camera captures the deformed fringe pattern reflected from the surface of the object through the telecentric imaging lens and reconstructs the surface topography.

[0011] The projection branch is composed of a projection lens and a DMD digital projection chip to form a multi-angle projection branch. The DMD digital projection chip generates a digital stripe pattern and projects it onto an object plane.

[0012] A plurality of projection branches are respectively provided with wedge-shaped adjusters, and the projection focal plane is adjusted to coincide with the object plane by introducing an axis shift angle.

[0013] According to an exemplary embodiment of the present invention, the optical axis of the imaging branch is arranged perpendicularly at an angle of 90° relative to the object plane, and the axis of the multi-channel RGBW ring light source coincides with the optical axis of the imaging branch.

[0014] The object plane, the projection lens principal plane and the DMD digital projection chip plane intersect at the axis-shift intersection line.

[0015] In the imaging branch, the telecentric imaging resolution is greater than the projector resolution; in the projection branch, the projector depth of field is greater than the telecentric imaging depth of field.

[0016] According to another aspect of the present invention, a calibration method for a multi-angle axis-shift fringe projection three-dimensional measurement system is provided, comprising:

[0017] Establish tilt-shift projection model, distortion model and telecentric imaging model;

[0018] The calibration plate image and multi-frequency phase-shift fringe image are collected using a high-resolution color camera and filtered for preprocessing.

[0019] The DMD digital projection chip is used to extract the phase of the center of the feature point of the calibration plate image after filtering, and the sub-pixel coordinates of the center of the feature point on the DMD plane are obtained by using the local homography method.

[0020] According to the sub-pixel coordinates of the center of the DMD plane feature point and the axis-shift projection model, the projector intrinsic parameter matrix is ​​calibrated to obtain the initial value of the projector extrinsic parameter matrix;

[0021] The distortion model is used to correct the distortion and shift angle of the projector, and the center coordinates of the corrected feature points of the projector are obtained;

[0022] Optimize the projector extrinsic matrix using the PnP method;

[0023] Calculate the three-dimensional coordinates of the center of the projector feature point according to the optimized external parameter matrix of the projector;

[0024] Substitute the three-dimensional coordinates of the center of the projector feature point into the telecentric imaging model to solve the camera telecentric projection matrix;

[0025] According to the camera telecentric projection matrix and the projector intrinsic and extrinsic parameter matrices, the camera-projector system calibration parameters are jointly optimized using the bundle adjustment method;

[0026] A multi-angle axis-shift fringe projection 3D measurement model is established according to the system calibration parameters, and the projector extrinsic matrix is ​​used to realize the global coordinate system 1 to compensate for the global registration error.

[0027] According to an exemplary embodiment of the present invention, when the tilt angle between the imaging plane and the lens plane is small, an additional aberration distortion parameter is added to compensate for the tilt effect, thereby constructing an axis-shift projection model.

[0028] According to an exemplary embodiment of the present invention, phase extraction is performed on the center of the feature point of the calibration plate image after filtering, and the sub-pixel coordinates of the center of the feature point of the DMD plane are obtained by using a local homography method, including:

[0029] For the filtered fringe image of the calibration plate, the absolute phase value of the center of the feature point is extracted by using the multi-frequency heterodyne phase shift method, and the corresponding integer pixel coordinates of the center of the projector DMD are calculated;

[0030] The local homography method is used to calculate the coordinates of the DMD center, obtain the corresponding projector pixel coordinate values, and establish a local homography relationship from the camera to the DMD plane;

[0031] Solve the corresponding optimal local homography matrix for each circle center neighborhood:

[0032] The random sampling consensus algorithm is used to solve the optimal local homography matrix and calculate the sub-pixel coordinates of the center of the feature point on the DMD plane.

[0033] According to an exemplary embodiment of the present invention, calibrating the projector internal parameter matrix to obtain the initial value of the projector external parameter matrix includes:

[0034] The accurate tilt-shift projector DMD coordinates corresponding to the feature points are obtained by the local homography method;

[0035] Using the projector center at different poses, calculate the closed form solution of the intrinsic parameter matrix of the mth projector;

[0036] The distortion coefficient and the shift angle are iteratively solved by the Levenberg-Marquardt algorithm.

[0037] The present invention adopts the above technical solution, which has the following beneficial effects:

[0038] 1. The system of the present invention adopts the axis-shift imaging condition, which can solve the problem of too small depth of field under the condition of small field of view and high magnification, so that the projection focal plane and the imaging focal plane remain coincident, overcome the reconstruction error caused by the defocus of the projection and imaging branches, make full use of the limited degree of freedom of the telecentric lens, and increase the dynamic measurement range.

[0039] 2. The system of the present invention consists of a camera with a telecentric lens and multiple projectors using a shift lens. The multi-angle shift projection can effectively reduce the shadows caused by occlusion, improve the measurement integrity, overcome the artifact problem caused by directional lighting in three-dimensional reconstruction, and expand the system measurement depth range.

[0040] 3. The method of the present invention constructs a new mathematical model of tilt-shift projection, and makes full use of the established tilt-shift projection model and distortion model to calibrate telecentric imaging, further reduces the camera-projector joint calibration error through nonlinear optimization, and finally achieves high-precision system calibration.

[0041] 4. The method of the present invention constructs and compensates for the registration error, thereby improving the accuracy of complete measurement of three-dimensional morphology.

[0042] 5. The present invention also designs a design process and parameter determination criteria for a multi-angle shift fringe projection three-dimensional measurement system, which can quickly realize the construction and integration of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute an improper limitation of the present invention. In the drawings:

[0044] Figure 1 It is a schematic diagram of the system structure of the present invention;

[0045] Figure 2 is a schematic diagram of overlapping depth of field of the tilt-shift projection of the present invention;

[0046] Figure 3 It is a schematic diagram of the cross-sectional optical path of the axis-shift projection of the present invention;

[0047] Figure 4 This is a comparison chart of the image quality of the tilt-shift projection of the present invention;

[0048] Figure 5 is a flow chart of the system calibration of the present invention;

[0049] Figure 6 It is a schematic diagram of the optical path of the axis-shift projection of the present invention;

[0050] Figure 7 Position relationship and error diagram of a multi-angle tilt-shift projector.

[0051] In the figure: 1-high-resolution color camera, 2-telecentric imaging lens, 3-projection lens A, 4-projection lens B, 5-DMD digital projection chip A, 6-DMD digital projection chip B, 7-DMD digital projection chip plane A, 8-projection lens A principal plane, 9-DMD digital projection chip plane B, 10-projection lens B principal plane, 11-axis shift intersection line A, 12-axis shift intersection line B, 13-object plane, 14-measured object, 15-multi-channel RGB ring light source, 16-projection and imaging focus plane, 17-imaging branch depth of field, 18-projection branch depth of field, 19-common focus area, 20-ideal DMD plane, 21-DMD plane after axis shift, 22-DMD chip principal point. DETAILED DESCRIPTION

[0052] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The exemplary embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0053] The present invention provides a multi-angle shift fringe projection three-dimensional measurement system. Figure 1A specific implementation of the present invention is shown, the whole system includes a high-resolution color camera 1, a telecentric imaging lens 2, a projection lens A3, a projection lens B4, a DMD digital projection chip A5 and a DMD digital projection chip B6. The system consists of an imaging branch and multiple projection branches.

[0054] In the imaging branch, a high-resolution color camera 1, a telecentric imaging lens 2 and a multi-channel RGBW ring light source 15 are arranged in sequence. The multi-channel RGBW ring light source 15 is arranged in front of the telecentric imaging lens 2. The optical axis of the imaging branch is vertically configured at an angle of 90° relative to the object plane 13. The high-resolution color camera 1 captures the deformed stripe pattern reflected from the surface of the object through the telecentric imaging lens 2. Then, the surface morphology can be reconstructed using an appropriate system model and phase recovery algorithm.

[0055] The high-resolution color camera 1 and the telecentric imaging lens 2 are also used to collect color information of the object being measured. The axis of the multi-channel RGBW ring light source 15 coincides with the optical axis of the imaging branch, so as to assist the imaging branch in collecting the color image of the object being measured.

[0056] The multi-channel RGBW ring light source 15 emits light of different wavelengths in sequence, which is reflected by the surface of the object and collected by the imaging branch to obtain the color texture information of the object surface. The pixels of the image are matched with the three-dimensional point cloud according to the calibration information, and the point cloud is modulated for color texture.

[0057] In the projection branch, the multi-angle projection branch is composed of projection lens and DMD digital projection chip. Figure 1 Two projection branches are shown, which are dual-angle projection branches composed of projection lens A3 and DMD digital projection chip A5, and projection lens B4 and DMD digital projection chip B6, respectively, but not limited to two, and multiple projection branches can be set. The digital stripe pattern is generated by the DMD digital projection chip and projected onto the object plane 13 to form a digital stripe pattern.

[0058] The projection branch adopts an axis-shift structure to solve the problem of too small depth of field in the case of small field of view and high magnification, overcome the reconstruction error problem caused by defocus of the projection and imaging branches, make full use of the limited degree of freedom of the telecentric lens, and increase the dynamic measurement range.

[0059] After the multiple projection branches are designed under the shift imaging condition (ie, when the object plane, the main plane of the projection lens and the extended plane of the DMD plane intersect in a line, a full-field clear image can be obtained), they are projected at a specific tilt angle β.

[0060] Wedge adjusters are respectively arranged between the projection lens A3, the projection lens B4 and the DMD digital projection chip A5, the DMD digital projection chip B6 for adjusting the shift angle.

[0061] The design of the shift angle needs to follow that the extended surfaces of the object plane 13, the projection lens A principal plane 8 and the DMD digital projection chip plane A7 intersect at the shift intersection line A11, and the extended surfaces of the object plane 13, the projection lens B principal plane 10 and the DMD digital projection chip plane B9 intersect at the shift intersection line B12.

[0062] Please refer to Figure 2 Based on the principle of axis-shift imaging, a special angle is introduced between the DMD chip and the projection lens (i.e., the projection lens is tilted relative to the DMD plane) so that the projection focal plane and the imaging focal plane coincide, i.e., the projection and imaging focal planes 16.

[0063] The imaging branch depth of field 17 and the projection branch depth of field 18 can be adjusted to overlap to obtain a larger common focus area 19, thereby allowing each projector to perform focused projection on the object surface simultaneously throughout the FOV.

[0064] The present invention deploys multiple shift projection branches to establish a multi-angle shift projection microscopic measurement system. The design of the multi-angle shift projection framework is based on the following considerations: first, multi-angle projection can effectively reduce the shadow area caused by occlusion and improve the integrity of measurement by providing projection patterns in different directions; second, multi-angle projection is an effective method to reduce local reflection and scattering of workpieces; in addition, by superimposing the depth of field of multiple projection branches, the depth range that the system can measure can be expanded.

[0065] On the basis of the above analysis, the present invention provides the overall design criteria and basis of the multi-angle axis-shift fringe projection three-dimensional measurement system.

[0066] 1. Determine the telecentric imaging FOV, projection FOV, and working distance, and then determine the vertical magnification M of the projector.

[0067] 2. Determine the tilt angle β of the object plane based on the phase-height sensitivity.

[0068] 3. Determine the tilt angle θ of the projection lens according to equation (1).

[0069] The inclination angle β of the object plane and the inclination angle θ of the projection lens satisfy the following relationship.

[0070] θ=arctan(M*tan(β)) (1)

[0071] In the figure, θ is the shift angle, M = pi / po is the nominal vertical axis magnification of the optical axis, and it needs to satisfy the Gaussian conjugate formula

[0072] 4. According to the 3D imaging depth of field, use equations (2) and (3) to determine the telecentric imaging depth of field (Δ c ) and the projector depth of field (Δ p).

[0073] Cross-section under tilt-shift imaging conditions, such as Figure 3 The near and far limits of the depth of field (dashed lines) intersect at the hinge line (H). Theoretically, the projection depth of field under the shift condition ranges as follows: Figure 3 As shown in the shaded area in the figure. Considering only the near and far limits of the depth of field without shifting ( and ), vertical offset limit (Δ s ) can be approximately calculated as:

[0074]

[0075] It should be noted that the actual vertical offset limit is greater than the calculated value. In addition, for the entire system, the vertical offset limit of the common focus area is also limited by the telecentric imaging depth of field (Δ c ) and field of view limitations.

[0076] Focus projection can be achieved in the entire common focus area, but the common focus area outside the perfect focus plane cannot meet the exact shift condition. In addition, the telecentricity of telecentric imaging is limited (the incident light is not completely parallel to the optical axis in the depth of field), and the imaging resolution will be reduced outside the perfect focus plane.

[0077] Based on the above analysis, in order to ensure accurate 3D imaging, two conditions are proposed for system design: (1) The telecentric imaging resolution is greater than the projector resolution. (2) The projector depth of field (Δ p ) is greater than the telecentric imaging depth of field (Δ c ), and satisfies the following empirical equation.

[0078] k 1 ·Δ p cos(β)=k 2 ·Δ c (3) k1 and k2 are empirical parameters, and it is recommended that k1 be 0.3-0.5 and k2 be 0.5-0.8. This means that the 3D imaging depth of field only occupies the telecentric imaging depth of field (Δ c ) and the vertical offset limit (Δ s ) needs to be greater than the 3D imaging depth of field. In actual operation, the telecentric lens depth of field (Δ c ), the middle part is used as the 3D imaging depth of field (i.e., determining the parameter k2). Then, the vertical offset limit (Δ s Finally, calculate the required projector depth of field (Δ p ).

[0079] 5. Design relevant optical parameters of the projector (such as focal length, working distance, aperture, etc.).

[0080] 6. Design the orientation and location of multi-angle projections to maximize overlap of common focus areas.

[0081] The contrast effect of tilt-shift projection is as follows Figure 4 As shown. Under the same experimental conditions, the binary fringe patterns of forward projection and tilt-shift projection are compared. It can be seen that since the focal plane of the forward projection is not perpendicular to the optical axis of the camera, a uniform and clear image cannot be obtained, and the fringe edges become increasingly blurred. In contrast, the captured tilt-shift projection image has clear fringe edges, and the overall contrast and quality of the image are better than those of the forward projection. As the vertical coordinate increases, the fringe edge profile of the forward projection becomes more defocused. Obviously, the projection pattern quality of the tilt-shift optical projection system is significantly better than that of the traditional optical projection system.

[0082] The multi-angle shift fringe projection three-dimensional measurement system of the present invention uses a telecentric lens in the imaging branch, which will cause the depth to be insensitive to changes along the optical axis. At present, there is no mature calibration algorithm for the calibration of telecentric lens imaging in related research. The relevant calibration algorithms either require the use of a high-precision translation stage for system calibration, which is usually difficult to set up (for example, the moving direction is completely perpendicular to the Z axis) and expensive, or due to strong constraint requirements (such as the orthogonality of the rotation matrix), it is difficult to achieve high-precision calibration of external parameters. In addition, since the internal parameters and external parameters are naturally coupled together and difficult to separate, the calibration process is further complicated, increasing the uncertainty of modeling accuracy.

[0083] In order to solve the limitations of the above system calibration, the present invention also provides a joint calibration method for a multi-angle axis-shift fringe projection three-dimensional measurement system.

[0084] The calibration flow chart is as follows: Figure 5 As shown, a joint calibration method for a multi-angle shift fringe projection 3D measurement system is specifically implemented as follows:

[0085] Step 1: Establish the tilt-shift projection model, distortion model and telecentric imaging model.

[0086] The pinhole model can be used to describe the tilt-shift projection process, such as Figure 6 As shown. w =X w Y w Z w is the world coordinate system, is the projector coordinate system, the superscript represents the mth projector (m=1,2,3,4...), is the projection center of the projector. The DMD plane is at z c=f. c = 1, and the virtual normalized DMD plane is parallel to the DMD plane. Assume that there is a point p on the DMD plane pm The three-dimensional coordinate projected into space is p w , considering the imaging distortion introduced by the shift angle, the following shift projection model is established:

[0087]

[0088] Where s is the scaling factor, represents the two-dimensional coordinates of a point on the DMD plane of the mth projector, p w It represents the three-dimensional coordinates of the object point projected onto the surface of the object in the world coordinate system. is the intrinsic parameter matrix of the m-th projector, and Axis-Shift Angle The introduced projection matrix and brick selection matrix, and is a 3×3 rotation matrix and a 3×1 translation vector.

[0089] Since lens distortion changes the direction of light, the actual projection process is nonlinear. The present invention introduces second-order radial and tangential distortion and establishes the following distortion model:

[0090]

[0091] in, is the ideal distortion-free normalized DMD coordinate, is the normalized DMD coordinate after distortion, radial distance k 1 , k 2 and p 1 , p 2 are the radial and tangential distortion coefficients, respectively.

[0092] Since lens distortion changes the direction of light, the actual projection process is nonlinear. The present invention introduces second-order radial and tangential distortion and establishes the following distortion model:

[0093]

[0094] in, is the ideal distortion-free normalized DMD coordinate, is the normalized DMD coordinate after distortion, radial distance k 1 , k 2 and p 1 , p 2 are the radial and tangential distortion coefficients, respectively.

[0095] The telecentric imaging model of a telecentric camera from 3D object points to 2D camera image coordinates can be described as follows:

[0096]

[0097] in, c and q w Represent the coordinates in the camera coordinate system and the coordinates in the world coordinate system, respectively. c is the camera intrinsic parameter matrix, and is a 3×3 rotation matrix and a 3×1 translation vector, is the telecentric projection matrix, m c ij is the projection matrix parameter, which contains the intrinsic parameter matrix K of the telecentric camera c and the external parameter matrix m x and m y are the effective magnifications in the X and Y directions, is the camera principal point coordinate. It is worth noting that, unlike the pinhole imaging model, the telecentric projection matrix has only eight valid values.

[0098] Step 2: Use a high-resolution color camera to capture the calibration plate image and multi-frequency phase-shift fringe image, and perform filtering preprocessing.

[0099] The present invention uses a black and white circle calibration plate as a calibration target. The calibration target is placed in different spatial directions and a set of images are collected at each target position, including white light projection, horizontal stripe pattern, and vertical stripe pattern. White light projection is used for illumination, so that the camera can capture the center of the circle as a feature point. (The superscript c indicates the telecentric camera, and the subscripts i and j indicate the i-th spatial position and j-th feature point of the calibration plate, respectively.) The horizontal and vertical fringe patterns are used to solve the horizontal and vertical absolute phase maps.

[0100] In addition, the captured image usually has noise, so the captured image needs to be preprocessed before solving the phase. In the present invention, Gaussian filtering is used to remove high frequency and stray noise.

[0101] Step 3: Use the DMD digital projection chip to extract the phase of the center of the feature point of the calibration plate image after filtering, and use the local homography method to obtain the sub-pixel coordinates of the center of the DMD plane feature point.

[0102] For the calibration plate fringe image filtered in step 2, the center of the feature point is extracted using multi-frequency heterodyne phase shifting technology. The absolute phase value of the m-th projector DMD circle center integer pixel coordinates are further calculated by the following formula

[0103]

[0104] in and Respectively represent the vertical and horizontal absolute phases of the center of the circle, n v and n h denote the number of fringes of the projected horizontal and vertical patterns, respectively, and H p ×W p is the projector resolution.

[0105] In order to obtain more accurate sub-pixel coordinates, the present invention adopts a local homography method to calculate the DMD center coordinates. First, use the center The absolute phase of the integer pixels in the neighborhood (k×k, k is the pixel range of the selected neighborhood), the corresponding projector pixel coordinate value is obtained by equation (7), and then the local homography relationship from the camera to the DMD plane is established. The following equation is the optimization cost function, and the corresponding optimal local homography matrix of each circle center neighborhood is solved:

[0106]

[0107] In the formula, are the homogeneous coordinates of the integer pixels in the center area of ​​the circle, is the pixel homogeneous coordinate of the mth projector corresponding to the point, argmin represents the variable value when the objective function takes the minimum value, and ||·|| represents the Euclidean distance of the vector.

[0108] The present invention uses the random sampling consensus algorithm (RANSAC) to solve the optimal local homography matrix The sub-pixel coordinates of the center of the DMD plane feature point are further calculated by the following formula:

[0109]

[0110] Step 4: According to the sub-pixel coordinates of the center of the DMD plane feature point and the axis-shift projection model, the distortion coefficient and the axis-shift angle of the projector intrinsic parameter matrix are calibrated, and the initial value of the projector extrinsic parameter matrix is ​​obtained.

[0111] Through the local homography method in step 3, the precise DMD coordinates of the tilt-shift projector corresponding to the feature points can be obtained. Using the center of the projector at different postures, the intrinsic parameter matrix of the mth projector can be calculated: In the present invention, the distortion coefficient and the shift angle are iteratively solved for the following equation by using the Levenberg-Marquardt algorithm.

[0112]

[0113] in is the coordinate of the center of the feature point on the DMD plane calculated according to equations (4) and (5), is the axis-shift projection distortion vector. n represents the number of different spatial positions, and m represents the number of feature points on the calibration plate. 1 , k 2 、p 1 and p 2 The initial value of can be set to zero, and The initial value of can be set to the theoretical design value. At the same time, the external parameter matrix of the projector is obtained The initial value of .

[0114] Step 5: Use the distortion model to correct the distortion and shift angle of the projector to obtain the corrected center coordinates of the projector feature points.

[0115] Apply the distortion coefficient and shift angle obtained in step 4 to equations (4) and (5) to obtain the corrected coordinates of the center of the projector feature point:

[0116] Step 6: Use the PnP method to optimize the projector extrinsic matrix.

[0117] The Perspective-n-point (PnP) method is used to iteratively calculate the minimum distance between the feature point coordinates and the reverse projection, and the external parameter matrix of the feature points at different spatial positions is In the present invention, the following formula is used to optimize the projector external parameter matrix:

[0118]

[0119] Among them, ||·|| represents the least squares distance, is the three-dimensional coordinate of the feature point in the world coordinate system.

[0120] Step 7: Calculate the three-dimensional coordinates of the center of the projector feature point based on the optimized projector's external parameter matrix.

[0121] Using the projector extrinsic matrix obtained for different positions in step 6, align the world coordinates of the feature point with the mth projector coordinate system, where the subscripts i and j represent the i-th spatial position and j-th feature point of the calibration plate, respectively. Then the projector extrinsic matrix becomes [I 3×3 ,0 3×1 ](I 3×3 is the identity matrix, 0 3×1is a zero vector), the 3D coordinates of the feature points are transformed to the projector coordinate system. In essence, the optimization method iteratively minimizes the difference between the observed projection point and the calculated projection point. The 3D coordinates of the center of the feature point in the projector coordinate system It can be calculated by the following formula:

[0122]

[0123] Step 8: Substitute the three-dimensional coordinates of the center of the projector feature point into the telecentric imaging model to solve the camera telecentric projection matrix.

[0124] The three-dimensional coordinates of each spatial position feature point are determined. Through direct linear transformation (DLT), the imaging model of the telecentric camera in the mth projector coordinate system (Equation (6)) can be rewritten as:

[0125]

[0126] in, are the coordinates of the feature points captured by the camera, is the array matrix of the three-dimensional coordinates of the feature points, are the telecentric projection matrix parameters in the m-th projector coordinate system.

[0127] In the present invention, all feature points are substituted into equation (13), and the least square solution is calculated by using the singular value decomposition (SVD) method to obtain the telecentric projection matrix, which contains the precise positional relationship between the projector and the camera.

[0128] Step 9: Based on the camera telecentric projection matrix and the projector intrinsic and extrinsic parameter matrices, the bundle adjustment method is used to jointly optimize the camera-projector system calibration parameters.

[0129] Telecentric lens distortion is inevitable. Therefore, considering the lens distortion, the telecentric camera is firstly optimized nonlinearly.

[0130]

[0131] in is the coordinate of the feature point on the camera imaging plane calculated according to formula (6), is the image coordinate of the center of the feature point on the camera imaging plane, the distortion coefficient The initial value of can be set to zero.

[0132] The purpose of camera-projector joint optimization is to improve the calibration accuracy by minimizing the reprojection errors of telecentric imaging and axis-shift projection at the same time. In the present invention, the bundle adjustment method is used to jointly optimize the system calibration parameters. The cost function of the joint optimization can be expressed as follows, which can be solved by the Levenberg-Marquardt nonlinear optimization algorithm.

[0133]

[0134] Where m represents the number of feature points on the calibration plate, n represents the number of different spatial positions, is the point at the i-th spatial position and j-th feature point on the DMD plane, are the coordinates of the calculated feature points on the DMD plane.

[0135] Step 10: Establish a multi-angle shift fringe projection 3D measurement model based on the system calibration parameters, use the projector extrinsic matrix to realize the global coordinate system 1, and compensate for the global registration error.

[0136] After the system calibration is completed, a stereoscopic vision algorithm is used for 3D reconstruction. In the present invention, the multi-angle shift fringe projection 3D measurement model is as follows:

[0137]

[0138] in, is the three-dimensional homogeneous coordinate in the m-th projector coordinate system, and are respectively the projection matrices of telecentric imaging and axis-shift projection in the mth projector coordinate system, is the internal parameter matrix of the m-th projector, (u c ,v c ) and (u pm ,v pm ) are the camera image coordinates and projector DMD coordinates respectively.

[0139] According to the projector extrinsic matrix obtained in step 6 Align the 3D data in multiple projector coordinate systems to a certain projector coordinate system to achieve global coordinate system 1. For ease of representation and derivation, assume alignment with the first projector coordinate system, and the coordinate system transformation is expressed as follows:

[0140]

[0141] in, and is the rotation matrix and translation vector that align the mth projector coordinate system with the first projector coordinate system, is the rotation matrix of the first projector, is the rotation matrix of the mth projector, is the translation vector of the first projector, is the translation vector of the mth projector.

[0142] Further, if Figure 7As shown in , a global registration error compensation model is established to compensate for the registration error. The rotation matrix with error and the translation vector are and After error correction, the coordinate relationship of the same object point in the 1st and mth projector coordinate systems can be expressed as:

[0143]

[0144] in, and are the rotation matrix and translation vector of the mth projector coordinate system relative to the first one, is the coordinate of the characteristic point of the calibration plate calculated according to formula (12) in the mth projector coordinate system, are the feature point alignment coordinates calculated directly using equation (19).

[0145] In the present invention, all feature points at all spatial positions are introduced, and the global registration error is calculated and compensated by formula (21). The error compensation model is as follows:

[0146]

[0147] in, and are the rotation matrix error and translation vector error of the mth projector coordinate system relative to the first one, is the coordinate of the feature point of the calibration plate calculated in the mth projector coordinate system, are the feature point alignment coordinates.

[0148] The method of the present invention solves the problem that orthogonal imaging of a telecentric lens leads to insensitivity to depth changes along the optical axis and it is difficult to achieve high-precision calibration of external matrix parameters of telecentric imaging, thereby improving the accuracy of complete measurement of three-dimensional morphology and enabling rapid construction and integration of the system.

[0149] The present invention is not limited to the above-mentioned embodiments. On the basis of the technical solution disclosed in the present invention, technicians in this field can make some substitutions and deformations to some technical features therein according to the disclosed technical content without creative labor, and these substitutions and deformations are all within the protection scope of the present invention.

Claims

1. A multi-angle shift fringe projection three-dimensional measurement system, characterized in that: include: The imaging branch is equipped with a high-resolution color camera, a telecentric imaging lens, and a multi-channel RGBW ring light source. The high-resolution color camera captures the deformed fringe pattern reflected from the surface of the object through the telecentric imaging lens and reconstructs the surface topography. The projection branch is composed of a projection lens and a DMD digital projection chip to form a multi-angle projection branch. The DMD digital projection chip generates a word stripe pattern and projects it onto the object plane. The plurality of projection branches are respectively provided with wedge-shaped adjusters, and the projection focal plane is adjusted to coincide with the object plane by introducing an axis shift angle.

2. The multi-angle shift fringe projection three-dimensional measurement system according to claim 1, characterized in that: The optical axis of the imaging branch is vertically arranged at an angle of 90° relative to the object plane; the axis of the multi-channel RGBW ring light source coincides with the optical axis of the imaging branch.

3. The multi-angle shift fringe projection three-dimensional measurement system according to claim 1, characterized in that: The object plane, the projection lens principal plane and the DMD digital projection chip plane intersect at the axis-shift intersection line.

4. The multi-angle shift fringe projection three-dimensional measurement system according to claim 1, characterized in that: In the imaging branch, the telecentric imaging resolution is greater than the projector resolution; in the projection branch, the projector depth of field is greater than the telecentric imaging depth of field.

5. A calibration method for a multi-angle shift fringe projection three-dimensional measurement system according to any one of claims 1 to 4, characterized in that: include: Establish tilt-shift projection model, distortion model and telecentric imaging model; The calibration plate image and multi-frequency phase-shift fringe image are collected using a high-resolution color camera and filtered for preprocessing. The DMD digital projection chip is used to extract the phase of the center of the feature point of the calibration plate image after filtering, and the sub-pixel coordinates of the center of the feature point on the DMD plane are obtained by using the local homography method. According to the sub-pixel coordinates of the center of the DMD plane feature point and the axis-shift projection model, the projector intrinsic parameter matrix is ​​calibrated to obtain the initial value of the projector extrinsic parameter matrix; The distortion model is used to correct the distortion and shift angle of the projector, and the center coordinates of the corrected feature points of the projector are obtained; The projector extrinsic matrix is ​​optimized using the PnP method; Calculate the three-dimensional coordinates of the center of the projector feature point according to the optimized external parameter matrix of the projector; Substitute the three-dimensional coordinates of the center of the projector feature point into the telecentric imaging model to solve the camera telecentric projection matrix; According to the camera telecentric projection matrix and the projector intrinsic and extrinsic parameter matrices, the camera-projector system calibration parameters are jointly optimized using the bundle adjustment method; A multi-angle axis-shift fringe projection 3D measurement model is established according to the system calibration parameters, and the projector extrinsic matrix is ​​used to realize the global coordinate system 1 to compensate for the global registration error.

6. The calibration method of a multi-angle shift fringe projection three-dimensional measurement system according to claim 5, characterized in that: When the tilt angle between the imaging surface and the lens surface is small, the tilt effect is compensated by adding additional aberration distortion parameters. The tilt-shift projection model is as follows: Where s is the scaling factor, represents the two-dimensional coordinates of a point on the DMD plane of the mth projector, p w It represents the three-dimensional coordinates of the object point projected onto the surface of the object in the world coordinate system. is the intrinsic parameter matrix of the mth projector, and Tilt-Shift Angle The introduced projection matrix and rotation matrix, and is a 3×3 rotation matrix and a 3×1 translation vector; The distortion model is as follows: in, is the normalized DMD coordinate of the actual distortion, r is the radial distance, are the ideal distortion-free normalized DMD coordinates, k1, k2 and p1, p2 are the radial and tangential distortion coefficients, respectively; The telecentric imaging model is as follows; Among them, q c and q w Represent the coordinates in the camera coordinate system and the coordinates in the world coordinate system, respectively. c is the intrinsic parameter matrix of the telecentric camera, and is a 3×3 rotation matrix and a 3×1 translation vector, is the telecentric projection matrix, m c ij is the projection matrix parameter, which contains the intrinsic parameter matrix K of the telecentric camera c and the external parameter matrix m x and m y are the effective magnifications in the X and Y directions, are the camera principal point coordinates.

7. The calibration method of a multi-angle shift fringe projection three-dimensional measurement system according to claim 5, characterized in that: The phase of the center of the feature point of the calibration plate image after filtering is extracted, and the sub-pixel coordinates of the center of the feature point on the DMD plane are obtained using the local homography method, including: For the filtered fringe image of the calibration plate, the absolute phase value of the center of the feature point is extracted by using the multi-frequency heterodyne phase shift method, and the corresponding integer pixel coordinates of the center of the projector DMD are calculated; The local homography method is used to calculate the coordinates of the DMD center, obtain the corresponding projector pixel coordinate values, and establish a local homography relationship from the camera to the DMD plane; Solve the corresponding optimal local homography matrix for each circle center neighborhood: The random sampling consensus algorithm is used to solve the optimal local homography matrix and calculate the sub-pixel coordinates of the center of the feature point on the DMD plane.

8. The calibration method of multi-angle shift fringe projection three-dimensional measurement system according to claim 5, characterized in that: Calibrate the projector internal parameter matrix to obtain the initial value of the projector external parameter matrix, including: The accurate tilt-shift projector DMD coordinates corresponding to the feature points are obtained by the local homography method; Using the projector center at different poses, calculate the closed form solution of the intrinsic parameter matrix of the mth projector; The distortion coefficient and axis angle are iteratively solved by the Levenberg-Marquardt algorithm.

9. The calibration method of a multi-angle shift fringe projection three-dimensional measurement system according to claim 5, characterized in that: The bundle adjustment method is used to jointly optimize the calibration parameters of the camera-projector system, including: The nonlinear optimization of the telecentric camera is performed, and the optimization formula is as follows: in, are the distortion coefficients, is the image coordinate of the center of the feature point on the camera imaging plane, is the coordinate of the calculated feature point on the camera imaging plane, i and j represent the i-th spatial position and j-th feature point of the calibration plate respectively; The cost function of joint optimization is solved by the Levenberg-Marquardt nonlinear optimization algorithm: Where m represents the number of feature points on the calibration plate, n represents the number of different spatial positions, is the point at the i-th spatial position and j-th feature point on the DMD plane, are the coordinates of the calculated feature point center on the DMD plane.

10. The calibration method of a multi-angle shift fringe projection three-dimensional measurement system according to claim 5, characterized in that: The multi-angle shift fringe projection 3D measurement model is as follows: in, is the three-dimensional homogeneous coordinate in the m-th projector coordinate system, and are respectively the projection matrices of telecentric imaging and axis-shift projection in the mth projector coordinate system, is the internal parameter matrix of the m-th projector, (u c ,v c ) and (u pm ,v pm ) are the camera image coordinates and the projector DMD coordinates respectively; Align the 3D data in multiple projector coordinate systems to a certain projector coordinate system to realize global coordinate system 1. The coordinate system conversion is as follows: in, and is the rotation matrix and translation vector that align the mth projector coordinate system with the first projector coordinate system, is the rotation matrix of the first projector, is the rotation matrix of the mth projector, is the translation vector of the first projector, is the translation vector of the mth projector; A global registration error compensation model is established to compensate for the registration error: in, and are the rotation matrix error and translation vector error of the mth projector coordinate system relative to the first one, is the coordinate of the feature point of the calibration plate calculated in the mth projector coordinate system, are the feature point alignment coordinates.

Images