Fringe projection trinocular vision calibration method based on phase reliability weighting

Abstract

The invention provides a fringe projection trinocular vision calibration method based on phase reliability weighting, and the method comprises the steps: constructing a three-dimensional point set for each feature circle by taking a pixel coordinate in a neighborhood of the feature circle as a plane coordinate and taking a corresponding phase value as a height coordinate; carrying out plane fitting on the point set and obtaining a normal vector; the phase normal vector difference of each point is obtained, a frequency statistical graph is generated, curve fitting is carried out, and a self-adaptive segmentation threshold value is obtained according to a preset exponential decay curve; dividing the error curve into three sections by utilizing a self-adaptive segmentation threshold value and endowing the three sections with weights; on the basis of accurate geometric constraint provided by a calibration plate, 'phase normal vector difference 'is introduced as a physical criterion of the quality of feature points, a weight model is constructed through a piecewise function, and adaptive weighting is performed on the feature points with different qualities in multi-view beam adjustment optimization, so that the multi-view beam adjustment optimization is realized. Therefore, the calibration precision of the projector in the trinocular system is obviously improved.

Description

A trinocular visual calibration method based on phase reliability weighting for fringe projection Technical Field

[0001] This invention relates to the fields of computer vision and optical 3D measurement, specifically to a trinocular vision calibration method based on phase reliability weighting of fringe projection. Background Technology

[0002] Structured light 3D measurement systems typically consist of a camera and a projector, where the system's calibration accuracy directly determines the final measurement accuracy. The projector is often considered a "camera" for inverse imaging, and its calibration usually relies on the mapping relationship between the image captured by the camera and the phase information of the phase-shifted fringes. Current calibration methods are suitable for monocular systems consisting of a single camera and projector, or dual-camera systems where the projector only provides stereo matching fringe textures. However, for trinocular fringe projection systems consisting of two cameras and a projector, the measurement principle is based on trinocular vision geometry. Due to feature point extraction errors and phase errors, the projector parameters calibrated using the phase information of the left and right cameras in such systems are inconsistent, thus affecting the final 3D reconstruction accuracy.

[0003] In recent years, numerous methods have been developed to improve the calibration accuracy of projectors in trinocular vision systems using fringe projection. Most of these methods involve projecting orthogonal sinusoidal fringe images onto a white plane, using phase information to determine the sub-pixel coordinates of feature points on the projector's image plane, and simultaneously using a binocular camera system to perform 3D reconstruction of corresponding point pairs to obtain their corresponding world coordinates, thus constructing 3D-2D corresponding point pairs for calibration. Some methods utilize these 3D-2D corresponding points to directly solve for the projector's internal parameters using the Zhang Zhengyou plane calibration method. Other methods use these 3D-2D corresponding points and employ an improved bundle adjustment method to achieve global parameter optimization of the trinocular vision system, simultaneously optimizing both the projector's internal parameters and the external parameter matrix between the camera and the projector. However, the feature point world coordinates of the above two methods are established on a blank whiteboard without any marked features, i.e., reference feature points without spatial measurement. Instead, they rely on the calculation quality of the phase and are easily affected by phase deviations between the left and right viewing angles, limiting the system's calibration accuracy.

[0004] For example, Chinese patent CN121191155A provides a three-dimensional visual measurement method and system for complex heterogeneous surfaces, which includes the following steps: (1) extracting the features of the part design model using semantic segmentation and intelligent recognition methods to establish a set of features to be measured; (2) using a three-dimensional measurement viewpoint planning method based on reinforcement learning to obtain the viewpoint sequence and motion path; (3) when the local measurement system reaches the preset viewpoint, for mutually reflective and semi-transparent surfaces, an adaptive single-pixel imaging feature matching method is used; for other surfaces, a parameter adaptive stripe projection three-dimensional measurement method is used to obtain a single-viewpoint three-dimensional point cloud; (4) converting the single-viewpoint three-dimensional point cloud to the global coordinate system to realize the stitching of the three-dimensional point cloud; (5) traversing each viewpoint of the viewpoint sequence and executing steps (3) and (4) to obtain a multi-viewpoint stitched three-dimensional point cloud; (6) fusing the multi-viewpoint three-dimensional point cloud based on point cloud error estimation.

[0005] This method is a fringe projection measurement method. Its calculation process reveals a high dependence on the quality of phase calculation. The solutions for viewing angle and phase errors are not clearly defined. Most projector calibration methods based on phase mapping suffer from these errors, resulting in non-coincident mapping of corresponding points from the left and right cameras to the projector, leading to differences in the calibrated projector parameters. Furthermore, due to the bias of its algorithm, the purpose and direction of model building for local feature point quality after image acquisition differ significantly from that of this invention. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide a trinocular visual calibration method for fringe projection based on phase reliability weighting, using feature points on a calibration target as the physical benchmark for spatial measurement. This method, relying on the precise geometric constraints provided by the calibration board, introduces "phase normal vector difference" as a physical criterion for feature point quality. A weighting model is constructed through a piecewise function, and adaptive weighting of feature points of different qualities is applied in multi-view beam adjustment optimization, thereby significantly improving the calibration accuracy of the projector.

[0007] The technical problem solved by this invention is achieved through the following technical solution:

[0008] A trinocular visual calibration method based on phase reliability weighting for fringe projection includes the following steps:

[0009] S1: Data Acquisition: Project white light and a two-way three-frequency four-step phase-shifting fringe pattern onto the circular feature point calibration plate, and control the left and right cameras to simultaneously acquire white light images and phase-shifting fringe images; and repeat the above projection and acquisition process by changing the pose of the calibration plate.

[0010] S2: Feature extraction: Perform image processing on the acquired white light image, extract the pixel coordinates of the center of the feature circle, and construct an N×N (e.g., 13×13) pixel neighborhood with each circle center as the center;

[0011] S3: Phase Calculation: The phase-shifted image obtained in S1 is calculated using the phase-shifting method and the multi-frequency heterodyne principle, and then unfolded into an absolute phase to obtain a bidirectional absolute phase unfolded image φ. v φ h Where v represents the vertical direction and h represents the horizontal direction;

[0012] S4: Construct phase space: Based on the center and neighboring pixel coordinates obtained in S2, index the corresponding absolute phase value in S3;

[0013] S5: Normal vector difference calculation: For each feature circle, construct a 3D point set using the pixel coordinates (u, v) in its neighborhood as the planar coordinates and the corresponding phase value as the height coordinate; perform plane fitting on the point set and obtain the normal vector; calculate the angle between the normal vectors of the corresponding feature circles in the left and right views, which is defined as the phase normal vector difference angle;

[0014] S6: Phase-Coordinate Mapping: Based on the camera pixel coordinates of the local neighborhood of the feature points obtained in S2 and the bidirectional absolute phase map obtained in S3, the corresponding set of projector pixel coordinates of the local neighborhood of the feature points is obtained using the phase-coordinate mapping formula.

[0015] S7: Homography Transform Solution: Utilizing the local neighborhood camera pixel coordinate set of feature points of each feature circle. and the corresponding feature point local neighborhood point projector pixel coordinate set Solve for the optimal homography matrix H between the camera and the projector. i ;

[0016] S8: Projector Feature Point Localization: The sub-pixel edges of feature circles in the camera image are extracted using an ellipse fitting algorithm. High-precision circle center coordinates are calculated, and these coordinates are mapped onto the projector image plane using the homography matrix to obtain the pixel coordinates of the feature circle center in the projector.

[0017] S9: Initial value calibration: Establish the correspondence between the physical coordinates of the calibration plate center and the pixel coordinates of the projector obtained in S8, use Zhang Zhengyou calibration method to calculate the initial values ​​of the projector's intrinsic and extrinsic parameters, and determine the structural parameters and intrinsic parameters of the binocular camera;

[0018] S10: Initial value reconstruction of 3D space: Using the calibrated left and right camera parameters, extract the pixel coordinates of the feature circles of the left and right cameras, and use the double-ray intersection triangulation method to perform 3D reconstruction to obtain the initial value of the 3D space coordinates of the feature points.

[0019] S11: Confidence-based adaptive segmented weighted model: Statistically analyze the phase normal vector difference of each point obtained in S5, generate a frequency statistics graph and perform curve fitting; after normalizing the fitted curve, calculate the adaptive segmentation threshold with the preset exponential decay curve; use the adaptive segmentation threshold to divide the error curve into three segments: the first segment is assigned a unit weight, and the second and third segments are assigned weight values ​​that conform to the exponential decay law according to the position of the adaptive segmentation threshold.

[0020] S12: Weighted bundle adjustment optimization: The bundle adjustment method is used to optimize the intrinsic parameters of the projector and the extrinsic parameters between the binocular camera and the projector. The intrinsic parameters of the two cameras are fixed, and a joint objective function is constructed that includes the reprojection error of the left camera, the reprojection error of the right camera and the reprojection error of the projector. The weight value of each feature point obtained in S11 is introduced into the objective function, and the Levenberg-Marquardt algorithm is used to solve for the optimal parameters of the system.

[0021] Furthermore, the specific steps of S5 are as follows:

[0022] Using S4, for the k-th feature point, a point set is constructed by taking the pixel coordinates (u, v) and phase value φ within its 13×13 neighborhood. A local plane Au+Bv+Cφ+D=0 is fitted, with its normal vector n=(A,B,C). The normal vector n of this feature point in the left camera view is then calculated. L and the normal vector n in the right camera view R Since the left and right cameras observe the same feature point neighborhood, theoretically their normal vectors should have geometric consistency. Calculate the angle θ between them. k :

[0023]

[0024] This refers to the difference in phase normal vectors. The larger this value, the lower the reliability of that point in the entire system.

[0025] Furthermore, the bidirectional three-frequency four-step phase-shifting stripe pattern adopts both vertical and horizontal directions in both directions.

[0026] Furthermore, in S1, the number of times the images of multiple feature circles are repeatedly projected and acquired is no less than 3 times.

[0027] Furthermore, the stripe pattern in S1 is expressed by the formula:

[0028] I K (x,y)=I'(x,y)+I″(x,y)cos[φ(x,y)+δ k ], k = 0, 1, 2, 3

[0029] Where I′(x,y) represents the average intensity, I″(x,y) represents the fringe modulation intensity, and φ(x,y) is the phase to be solved. k represents the k-th phase-shifted image, and the total number of phase-shifting steps is N;

[0030] The phase-shifting method described in S3 uses a four-step phase-shifting method, where N=4. For each frequency, the wrapping phase φ(x,y) is obtained using the following formula based on the principle of the phase-shifting method:

[0031]

[0032] The phase unfolding of the three frequency envelope phases is performed using the multi-frequency heterodyne principle. Specifically, the phase diagram φ1 of the first frequency and the phase diagram φ2 of the second frequency are combined to form an equivalent phase φ with a smaller frequency and a longer period. 12 Let φ1, φ2 and φ 12 The frequencies are λ1, λ2, and λ3 respectively. 12 The mathematical expression for frequency superposition is:

[0033]

[0034] The second frequency phase diagram φ2 and the third frequency phase diagram φ3 are combined to form an equivalent phase diagram φ. 23 φ 12 and φ 23 Synthetic equivalent phase diagram φ 123 Due to the influence of noise, a synthesized equivalent phase map φ needs to be used. 123 Perform reverse phase expansion, first reverse expansion to synthesize the phase diagram φ 12 The corresponding absolute phase diagram Φ is obtained. 12 Then, by reversing φ1, the corresponding absolute phase diagram Φ1 is obtained, with Φ 12 Taking φ1 as an example, the formula is:

[0035] Φ1=φ1+2kπ

[0036]

[0037] Here, "round" represents rounding, and the fringe order "k" is the core operation of phase unrolling, through which the bidirectional absolute phase map Φ is obtained. v Φ h .

[0038] Furthermore, in S4, the set of coordinates of the neighborhood pixels of the circle center obtained in S2 is... The bidirectional absolute phase map φ obtained from S3 v φ hThe coordinate set of the corresponding projector pixel points in the neighborhood of the circle is obtained by calculating using the following phase-coordinate mapping formula.

[0039]

[0040] in, For the set of camera pixel coordinates in the neighborhood of the circle The nth point, N v and N h These represent the number of fringe periods in the vertical and horizontal directions, respectively, while W and H represent the width and height of the projector resolution.

[0041] Further, in S12, a joint optimization objective function is constructed, incorporating constraints from the left camera, right camera, and projector. The objective function, minimizing the weighted total reprojection error, is optimized using the LM algorithm, as follows:

[0042]

[0043] In the formula, k represents the k-th feature point, and M is the total number of feature points; W n The weight value is calculated based on the difference in phase normal vectors in step S11; These are the actual observed pixel coordinates extracted from the left and right camera images; The coordinates of the observed pixels on the projector image plane are calculated using phase-coordinate mapping; the intrinsic parameters of the left and right cameras are A and B, respectively. l A r The projector's internal parameter is A. p The distortion coefficients of the left and right cameras are k respectively. l k r The distortion coefficient of the projector is k. p ω is the vector form of the rotation matrix R; T is the translation vector; P w The world coordinates of the feature point.

[0044] Further in S11, the angle difference in S5 is transformed into optimization weights. This invention adopts the following segmentation strategy:

[0045] 1. Calculate the difference in normal vectors {θ1, ..., θ2} for all M feature points. M}, plot the frequency distribution histogram and fit a smooth curve F(θ).

[0046] 2. Find the peak value of F(θ) and normalize the curve to F. norm (θ), where λ is set to the peak value of F(θ).

[0047] 3. Define the exponential decay function G(θ) = e-λθ Calculate F norm The adaptive segmentation threshold between G(θ) and G(θ) is denoted as θ cut1 and θ cut2 .

[0048] Construct a piecewise weighted model function W based on confidence level. n The first segment θ n ≤θ cut1 The feature points in this part are assigned full weight. The second segment θ cut1 <θ n ≤θ cut2 The weight is determined by the exponential curve value corresponding to the adaptive segmentation threshold position, i.e., W. n =G(θ) cut1 ). Third segment θ n >θ cut2 The weight is W n =G(θ) cut2 ).

[0049] A trinocular vision calibration device based on phase reliability weighting with stripe projection includes a projector, two cameras, a circular pattern calibration plate, and a computing module. The circular pattern calibration plate is placed in the common field of view of the projector and the cameras, and the projector and the cameras are respectively connected to the computing module.

[0050] The calculation module includes an acquisition unit, a circle center extraction unit, a phase calculation unit, a normal vector calculation unit, a mapping unit, an ellipse fitting unit, a homography estimation unit, an adaptive weight calculation unit, and a calibration unit.

[0051] The acquisition unit is used to receive images of multiple feature circles in various poses and phase-shifted images acquired by the camera from the calibration board;

[0052] The center extraction unit is used to process the acquired image and extract the camera pixel coordinates of the center of the circle;

[0053] The phase calculation unit is used to calculate the wrapping phase of the acquired phase-shifted image using the phase-shifting method and the multi-frequency heterodyne principle, and unfold it into an absolute phase to obtain a two-way absolute phase map;

[0054] The normal vector calculation unit is used to calculate the normal vectors corresponding to the feature point regions under the left and right camera views obtained by the phase solution unit, and to calculate the angle between the two normal vectors.

[0055] The mapping unit uses the acquired camera feature point neighborhood pixel coordinates and the bidirectional absolute phase map obtained by the calculation unit to calculate the corresponding projector pixel coordinates of the center neighborhood point through the bit-coordinate mapping formula.

[0056] The ellipse fitting unit uses an ellipse fitting algorithm to extract the sub-pixel edges of feature circles in the camera image and calculates high-precision circle center coordinates.

[0057] The homography estimation unit uses the camera pixel coordinate set of the local neighborhood of each feature circle and the corresponding projector pixel coordinate set of the local neighborhood of the feature point to solve for the optimal homography matrix between the camera and the projector. It then uses the fitted center coordinates of the circle and the homography matrix to calculate the center coordinates of the feature points on the projector.

[0058] The adaptive weight calculation unit generates a frequency statistics graph and performs curve fitting using the difference in the normal vector; after normalizing the fitted curve, it calculates an adaptive segmentation threshold with a preset exponential decay curve; the adaptive segmentation threshold is used to divide the error curve into three segments, and the three segments are assigned corresponding weights respectively.

[0059] The calibration unit optimizes the intrinsic parameters of the projector and the extrinsic parameters between the binocular camera and the projector using the bundle adjustment method. The intrinsic parameters of the two cameras are fixed, and a joint objective function is constructed that includes the reprojection error of the left camera, the reprojection error of the right camera, and the reprojection error of the projector. The weight of each point obtained in the adaptive weight calculation unit is used in the reprojection error function, and the Levenberg-Marquardt algorithm is used to solve for the optimal parameters of the system.

[0060] An electronic device includes: a memory and processors interconnected, the memory storing computer instructions, and the processors executing the computer instructions to perform the phase reliability weighted fringe projection trinocular vision calibration method described above.

[0061] A computer storage medium storing computer instructions for causing a computer to execute the phase reliability weighted fringe projection trinocular vision calibration method described above.

[0062] The advantages and positive effects of this invention are:

[0063] (1) This invention is applicable to the calibration of projectors in a trinocular structured light system using a calibration plate with a circular pattern. This calibration method innovatively constructs a three-dimensional point set of “pixel coordinates-absolute phase” in the local neighborhood of the feature point, and uses the consistency of the local phase normal vector as the criterion for the quality of the feature point. This invention does not rely on empirically fixed thresholds, but statistically models the distribution of normal vector differences of global feature points, adaptively determines the attenuation coefficient using the peak characteristics of the distribution curve, and automatically calculates the segmented threshold through the intersection of the distribution curve and the attenuation function.

[0064] (2) The present invention introduces the above-mentioned adaptive weights in the bundle adjustment to construct a three-viewpoint system including the left camera, the right camera, and the projector. Through this strongly coupled global optimization solution, the projector parameters are forced to maintain geometric consistency in the trinocular vision system, eliminating the systematic deviation caused by the independent calibration of the left camera and the projector, and the right camera and the projector, and significantly improving the 3D reconstruction accuracy of the fringe projection trinocular vision system. Attached Figure Description

[0065] Figure 1 is a flowchart of the stripe projection trinocular vision calibration in an embodiment of the present invention;

[0066] Figure 2 is a diagram showing the difference in phase normal vectors at the same point between the left and right views in an embodiment of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer and easier to understand, the present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit the scope of this application.

[0068] This invention provides a projector calibration method for a circular pattern feature calibration plate, and the overall calibration process is shown in Figure 1. The overall process of this method is as follows:

[0069] Step 1: Set up a trinocular fringe projection measurement system. Place a circular pattern calibration board within the common field of view of the projector and camera. Project white light and bidirectional three-frequency four-step (vertical and horizontal) phase-shifted images onto the calibration board, and the camera acquires the images. The white light image is used to extract the circular feature edge points of the calibration board, and the fringe image is used to calculate the absolute phase on the calibration board. Change the calibration board pose and repeat the projector projection and camera acquisition process.

[0070] Two-way three-frequency four-step phase-shifting images are measurement data images acquired based on fringe projection profilometry, enabling high-precision 3D measurements in fields such as optical measurement and 3D topography reconstruction. This technique involves vertical and horizontal two-way phase measurement, phase-shifting algorithms, and multi-frequency heterodyne phase unwrapping methods to improve the accuracy and stability of 3D measurements.

[0071] By projecting and acquiring vertical and horizontal stripes, phase information of the same object under test in two directions is obtained. In this embodiment, the phase information in both vertical and horizontal directions is acquired, and the corresponding projector pixel coordinates are uniquely determined using the phase-coordinate mapping formula. The phase-shifting method refers to acquiring multiple images at each frequency by changing the phase of the grating pattern, and then using the phase difference between these images to calculate the phase distribution of the object. The multi-frequency heterodyne method is a time-phase unfolding method that illuminates the object with stripe patterns of different frequencies, simultaneously acquiring a stripe pattern sequence modulated by the object's surface, and then independently unfolding the phase of each point along the time sequence. This method can theoretically avoid the propagation of errors.

[0072] Step 2: Perform image processing on the white light image obtained in Step 1 to extract the camera pixel coordinates of the center of the circle. Where c represents the pixel coordinates of the camera image, and i represents the number of feature circles on the calibration plate.

[0073] Step 3: Using the phase-shifting method and the multi-frequency heterodyne principle, calculate the wrapped phase and unfold the absolute phase of the phase-shifted image acquired by S1 to obtain the bidirectional absolute phase unfolded image φ. v φ h Where v represents the vertical direction and h represents the horizontal direction. A fringe pattern can be represented as:

[0074] I k (x,y)=I′(x,y)+I″(x,y)cos[φ(x,y)+δ k ], k = 0, 1, 2, 3.

[0075] Where I′(x,y) represents the average intensity, I″(x,y) represents the fringe modulation intensity, and φ(x,y) is the phase to be solved. k represents the k-th phase-shifted image, and the total number of phase-shifting steps is N.

[0076] The phase-shifting method obtains the position of a point on an object's surface on a projector image by projecting a series of phase-shifted fringe images. This method requires at least three phase-shifting steps to recover the three-dimensional shape of the object. Because it obtains the phase pixel-by-pixel, it has high spatial resolution. Furthermore, since the pixel-by-pixel phase calculation automatically cancels out the surface reflectivity at each point, the method is robust to changes in the surface reflectivity of the object. In addition, it automatically cancels out the DC component, thus theoretically being unaffected by ambient light.

[0077] This method uses a four-step phase shift, i.e., N=4, as an example. For the four-step phase shift fringe pattern at each frequency, the wrapping phase φ(x,y) can be obtained from the following formula according to the principle of the phase shift method:

[0078]

[0079] The phase unfolding of the three frequency envelope phases is performed using the multi-frequency heterodyne principle. Specifically, the phase diagram φ1 of the first frequency and the phase diagram φ2 of the second frequency are combined to form an equivalent phase φ with a smaller frequency and a longer period. 12 Let φ1, φ2, and φ 12 The frequencies are λ1, λ2, and λ3 respectively. 12 The mathematical expression for frequency superposition is:

[0080]

[0081] Similarly, the second frequency phase diagram φ2 and the third frequency phase diagram φ3 are combined to form an equivalent phase diagram φ. 23 φ 12 and φ 23 Synthetic equivalent phase diagram φ 123 Due to the influence of noise, a synthesized equivalent phase map φ is required. 123 Perform reverse phase expansion, first reverse expansion to synthesize the phase diagram φ 12 The corresponding absolute phase diagram Φ is obtained. 12 Then, by reversing φ1, the corresponding absolute phase diagram Φ1 is obtained. Using Φ 12 Taking φ1 as an example, the formula is:

[0082] Φ1=φ1+2kπ

[0083]

[0084] Here, "round" represents rounding, and the fringe order k is the core operation of phase unwrapping. Therefore, this step yields the bidirectional absolute phase map Φ. v Φ h .

[0085] Multi-frequency heterodyne is a time-phase unfolding method that illuminates an object with fringe patterns of different frequencies, simultaneously acquiring a sequence of fringe patterns modulated by the object's surface. The phase at each point is then independently unfolded along the time sequence, thus fundamentally preventing error propagation. Specifically, the three-frequency heterodyne principle involves superimposing three different frequency wrapper phases (i.e., subtracting the wrapper phases) to obtain an equivalent phase with a lower frequency and longer period. This equivalent phase is then used for inverse phase unfolding in image processing.

[0086] Step 4: Based on the center of the circle and the 13×13 neighboring pixel coordinates obtained in Step 2, index the corresponding absolute phase value in Step 3.

[0087] Step 5: For the k-th feature point, within its 13×13 neighborhood, take the pixel coordinates (u, v) and phase value φ to form a point set. Fit the local plane Au+Bv+Cφ+D=0, whose normal vector is n=(A,B,C). Calculate the normal vector n of this feature point in the left camera view. L and the normal vector n in the right camera view R The difference in the normal vectors is shown in Figure 2. Calculate the angle θ between them. k :

[0088]

[0089] Step 6: Based on the local neighborhood camera pixel coordinates of the feature points obtained in Step 2 And step 3 to obtain the bidirectional absolute phase map Φ v Φ h The corresponding local neighborhood point projector pixel coordinate set of the feature point can be calculated using the following phase-coordinate mapping formula.

[0090]

[0091] in, For the local neighborhood camera pixel coordinate set of feature points The nth point, N v and N h These represent the number of fringe periods in the vertical and horizontal directions, respectively, while W and H represent the width and height of the projector resolution.

[0092] Step 7: Utilize the local neighborhood camera pixel coordinate set of feature points for each feature circle and the corresponding feature point local neighborhood point projector pixel coordinate set Solve for the optimal homography matrix H between the camera and the projector. i :

[0093]

[0094] Using the obtained homography matrix H i Camera pixel coordinate set of local neighborhood points of feature points The calculation can obtain the local neighborhood point projector pixel coordinate set of the corresponding feature point.

[0095]

[0096] in, For the local neighborhood camera pixel coordinate set of feature points The coordinates of the nth point in the middle. The set of pixel coordinates of the local neighborhood projector for feature points obtained by homography matrix calculation. The corresponding pixel coordinates.

[0097] Step 8: Extract the sub-pixel edges of the feature circles in the camera image using an ellipse fitting algorithm, calculate the high-precision center coordinates of the circles, and map them onto the projector image plane using the homography matrix to obtain the pixel coordinates of the feature circle centers in the projector.

[0098] Step 9: Establish the correspondence between the physical coordinates of the calibration plate center and the pixel coordinates of the projector obtained in Step 8, and use Zhang Zhengyou's calibration method to calculate the initial values ​​of the projector's intrinsic and extrinsic parameters.

[0099] Zhang Zhengyou's calibration method is a commonly used method for camera calibration. This method uses a calibration board containing feature points, takes photos of the calibration board from multiple different angles and positions, and then processes the photos using computer vision technology to extract the pixel coordinates and world coordinates of the feature points, thereby solving for the camera's internal and external parameters.

[0100] Step 10: Using the calibrated left and right camera parameters, extract the pixel coordinates of the feature circles of the left and right cameras, and perform 3D reconstruction using the two-ray intersection method;

[0101] Step 11: Statistically analyze the phase normal vector difference of each point obtained in Step 5, generate a frequency statistics graph and perform curve fitting; normalize the fitted curve and calculate the adaptive segmentation threshold with the preset exponential decay curve; use the adaptive segmentation threshold to divide the error curve into three segments.

[0102] Calculate the difference values ​​of the normal vectors of all M feature points {θ1, ..., θ2}. M}, plot the frequency distribution histogram and fit a smooth curve F(θ).

[0103] Find the peak value of F(θ) F max The curve is normalized to obtain curve F. norm (θ)=F(θ) / F max At this point, the curve mode is reset to 1. The exponential decay function G(θ) = e^(-θ / θ) is set. -λθ Calculate F norm The adaptive segmentation threshold between G(θ) and G(θ) is denoted as θ cut1 and θ cut2 λ is F max The value of . Construct a piecewise weighted model function W based on confidence level. n The first segment θ n ≤θ cut1 The feature points in this part are assigned full weight. The second segment θ cut1 <θn ≤θ cut2 The weight is determined by the exponential curve value corresponding to the adaptive segmentation threshold position, i.e., W. n =G(θ) cut1 ). Third segment θ n >θ cut2 The weight is W n =G(θ) cut2 ).

[0104] Step 12: Weighted bundle adjustment optimization: Optimize the intrinsic parameters of the projector and the extrinsic parameters between the binocular camera and the projector using the bundle adjustment method; fix the intrinsic parameters of the two cameras and construct a joint objective function that includes the reprojection error of the left camera, the reprojection error of the right camera, and the reprojection error of the projector; introduce the weight value of each feature point obtained in Step 11 into the objective function, and use the Levenberg-Marquardt algorithm to solve for the optimal parameters of the system.

[0105]

[0106] The results of the trinocular visual reconstruction by stripe projection in this embodiment are shown in Table 1. The accuracy is improved compared with the traditional binocular method, which effectively proves the effectiveness of the present invention.

[0107] Table 1. Measurement error of standard ceramic balls (unit: μm)

[0108] Method: Ball 1 diameter error, Ball 2 diameter error, Ball center distance error; Traditional binocular: 40.00, 13.70, 53.61; This invention method: 23.50, 3.30, 51.99 surface

[0109] A trinocular vision calibration device based on phase reliability weighting for fringe projection includes a projector, two cameras, a circular pattern calibration plate, and a calculation module. The circular pattern calibration plate is placed in the common field of view of the projector and the cameras. The projector and the cameras are respectively connected to the calculation module. The calculation module includes an acquisition unit, a circle center extraction unit, a phase calculation unit, a normal vector calculation unit, a mapping unit, a homography estimation unit, an ellipse fitting unit, an adaptive weight calculation unit, and a calibration unit.

[0110] The acquisition unit is used to receive images of multiple feature circles in various poses and phase-shifted images acquired by the camera from the calibration board;

[0111] The center extraction unit is used to process the acquired image and extract the camera pixel coordinates of the center of the circle;

[0112] The phase calculation unit is used to calculate the wrapping phase of the acquired phase-shifted image using the phase-shifting method and the multi-frequency heterodyne principle, and unfold it into an absolute phase to obtain a two-way absolute phase map;

[0113] The normal vector calculation unit is used to calculate the neighborhood normal vectors of the feature points corresponding to the left and right camera views obtained by the phase calculation unit, and to calculate the angle between the two normal vectors.

[0114] The mapping unit uses the acquired camera pixel coordinates and the bidirectional absolute phase map obtained by the calculation unit to calculate the corresponding projector pixel coordinates of the center point using the bit-coordinate mapping formula.

[0115] The homography estimation unit uses the camera pixel coordinate set of the local neighborhood of each feature circle and the corresponding projector pixel coordinate set of the local neighborhood of each feature circle to solve for the optimal homography matrix between the camera and the projector; the camera pixel coordinate set of the local neighborhood of each feature circle is converted into the corresponding projector pixel coordinate set of the local neighborhood of each feature circle.

[0116] The ellipse fitting unit uses an ellipse fitting algorithm to extract the sub-pixel edges of feature circles in the camera image and calculates high-precision circle center coordinates.

[0117] The adaptive weight calculation unit generates a frequency statistics graph and performs curve fitting using the difference in the described normal vectors; after normalizing the fitted curve, it calculates an adaptive segmentation threshold with a preset exponential decay curve; the adaptive segmentation threshold is used to divide the error curve into three segments, and the three segments are assigned corresponding weights respectively.

[0118] The calibration unit optimizes the intrinsic parameters of the projector and the extrinsic parameters between the binocular camera and the projector using the bundle adjustment method. The intrinsic parameters of the two cameras are fixed, and a joint objective function is constructed that includes the reprojection error of the left camera, the reprojection error of the right camera, and the reprojection error of the projector. The weight of each point obtained in the adaptive weight calculation unit is used in the reprojection error function, and the Levenberg-Marquardt algorithm is used to solve for the optimal parameters of the system.

[0119] An electronic device includes: a memory and processors interconnected, the memory storing computer instructions, and the processors executing the computer instructions to perform the phase reliability weighted fringe projection trinocular vision calibration method described above.

[0120] A computer storage medium storing computer instructions for causing a computer to execute the phase reliability weighted fringe projection trinocular vision calibration method described above.

[0121] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A trinocular visual calibration method based on phase reliability weighting for fringe projection, characterized in that: The steps include: S1: Data Acquisition: Project white light and a two-way three-frequency four-step phase-shifting fringe pattern onto the circular feature point calibration plate, and control the left and right cameras to simultaneously acquire white light images and phase-shifting fringe images; and repeat the above projection and acquisition process by changing the pose of the calibration plate. S2: Feature extraction: Perform image processing on the acquired white light image, extract the pixel coordinates of the center of the feature circle, and construct an N×N pixel neighborhood with each circle center as the center; S3: Phase Calculation: The phase-shifted image obtained in S1 is calculated using the phase-shifting method and the multi-frequency heterodyne principle, and then unfolded into an absolute phase to obtain a bidirectional absolute phase unfolded image φ. v φ h Where v represents the vertical direction and h represents the horizontal direction; S4: Construct phase space: Based on the center and neighborhood pixel coordinates obtained in S2, index the corresponding absolute phase value in S3; S5: Calculate normal vector difference: For each feature circle, construct a 3D point set using the pixel coordinates (u, v) in its neighborhood as the plane coordinates and the corresponding phase value as the height coordinate; perform plane fitting on this point set and obtain the normal vector; calculate the angle between the normal vectors of the corresponding feature circles in the left and right views, defined as the phase normal vector difference angle; S6: Phase-coordinate mapping: Based on the camera pixel coordinates of the local neighborhood of the feature points obtained in S2 and the bidirectional absolute phase map obtained in S3, use the phase-coordinate mapping formula to obtain the corresponding feature point local neighborhood point projector pixel coordinate set. S7: Homography Transform Solution: Utilizing the local neighborhood camera pixel coordinate set of feature points of each feature circle. and the corresponding feature point local neighborhood point projector pixel coordinate set Solve for the optimal homography matrix H between the camera and the projector. i S8: Projector Feature Point Localization: The sub-pixel edges of feature circles in the camera image are extracted using an ellipse fitting algorithm. High-precision circle center coordinates are calculated, and these coordinates are mapped onto the projector image plane using the homography matrix to obtain the pixel coordinates of the feature circle center in the projector. S9: Initial Value Calibration: Establish the correspondence between the physical coordinates of the calibration plate center and the pixel coordinates of the projector obtained in S8. Calculate the initial values ​​of the projector's intrinsic and extrinsic parameters using Zhang Zhengyou's calibration method, and determine the structural parameters and intrinsic parameters of the binocular camera. S10: 3D Space Initial Value Reconstruction: Using the calibrated left and right camera parameters, extract the pixel coordinates of the feature circles of the left and right cameras. Perform 3D reconstruction using the dual-ray intersection triangulation method to obtain the initial values ​​of the 3D space coordinates of the feature points. S11: Confidence-Based Adaptive Piecewise Weighted Model: Statistically analyze the phase normal vector difference of each point obtained in S5, generate a frequency statistics graph, and perform curve fitting. After normalizing the fitted curve, compare it with the preset exponential decay... S12: Obtain the adaptive segmentation threshold by subtracting the curve; use the adaptive segmentation threshold to divide the error curve into three segments: the first segment is assigned a unit weight, and the second and third segments are assigned weight values ​​that conform to the exponential decay law according to the position of the adaptive segmentation threshold; S12: Weighted bundle adjustment optimization: use the bundle adjustment method to optimize the intrinsic parameters of the projector and the extrinsic parameters between the binocular camera and the projector; fix the intrinsic parameters of the two cameras, construct a joint objective function that includes the reprojection error of the left camera, the reprojection error of the right camera, and the reprojection error of the projector, introduce the weight value of each feature point obtained in S11 into the objective function, and use the Levenberg-Marquardt algorithm to solve for the optimal parameters of the system.

2. The trinocular visual calibration method based on phase reliability weighting for fringe projection according to claim 1, characterized in that: In step S1, the bidirectional three-frequency four-step phase-shifting fringe pattern employs both vertical and horizontal directions, and the images of multiple feature circles are repeatedly projected and acquired at least three times. The fringe pattern in step S1 is expressed by the formula: I K (x,y)=I'(x,y)+I″(x,y)cos[φ(x,y)+δ k [,k=0,1,2,3 where I′(x,y) represents the average intensity, I″(x,y) represents the fringe modulation intensity, and φ(x,y) is the phase to be solved. k represents the k-th phase-shifted image, and the total number of phase-shifting steps is N.

3. The trinocular visual calibration method based on phase reliability weighting for fringe projection according to claim 1, characterized in that: In S3, the phase shift method employs a four-step phase shift method, where N=4. For each frequency's four-step phase shift fringe pattern, the wrapping phase φ(x,y) is obtained using the following formula based on the phase shift method principle: The phase unfolding of the three frequency envelope phases is performed using the multi-frequency heterodyne principle. The phase diagram φ1 of the first frequency and the phase diagram φ2 of the second frequency are combined to form an equivalent phase φ with a smaller frequency and a longer period. 12 Let φ1, φ2 and φ 12 The frequencies are λ1, λ2, and λ3 respectively. 12 The mathematical expression for frequency superposition is: The second frequency phase diagram φ2 and the third frequency phase diagram φ3 are combined to form an equivalent phase diagram φ. 23 , φ 12 and φ 23 Synthetic equivalent phase diagram φ 123 Using the synthesized equivalent phase map φ 123 Perform reverse phase expansion, first reverse expansion to synthesize the phase diagram φ 12 The corresponding absolute phase diagram Φ is obtained. 12 Then, by reversing φ1, the corresponding absolute phase diagram Φ1 is obtained, with Φ 12 Taking φ1 as an example, the formula is: Φ1=φ1+2kπ Here, "round" represents rounding, and the fringe order "k" is the core operation of phase unrolling, through which the bidirectional absolute phase map Φ is obtained. v Φ h .

4. The trinocular visual calibration method based on phase reliability weighting for fringe projection according to claim 1, characterized in that: In step S4, the set of coordinates of the neighborhood pixels of the circle obtained in step S2 is... The bidirectional absolute phase map φ obtained from S3 v φ h The coordinate set of the corresponding projector pixel points in the neighborhood of the circle is obtained by calculating using the following phase-coordinate mapping formula. in, For the set of camera pixel coordinates in the neighborhood of the circle The nth point, N v and N h These represent the number of fringe periods in the vertical and horizontal directions, respectively, while W and H represent the width and height of the projector resolution.

5. The trinocular visual calibration method for fringe projection based on phase reliability weighting according to claim 1, characterized in that: In S12, a joint optimization objective function is constructed, incorporating constraints from the left camera, right camera, and projector. The goal is to minimize the weighted total reprojection error. The LM algorithm is used to optimize the reprojection error function. The objective function is as follows: In the formula, k represents the k-th feature point, and M is the total number of feature points; W n The weight value is calculated based on the difference in phase normal vectors in step S11; These are the actual observed pixel coordinates extracted from the left and right camera images; The coordinates of the observed pixels on the projector image plane are calculated using phase-coordinate mapping; the intrinsic parameters of the left and right cameras are A and B, respectively. l A r The projector's internal parameter is A. p The distortion coefficients of the left and right cameras are k respectively. l k r The distortion coefficient of the projector is k. p ω is the vector form of the rotation matrix R; T is the translation vector; P w The world coordinates of the feature point.

6. The trinocular visual calibration method for fringe projection based on phase reliability weighting according to claim 4, characterized in that: The specific steps of S5 are as follows: Using the pixel coordinates (u, v) and phase value φ within the 13×13 neighborhood of the k-th feature point from S4, fit a local plane Au+Bv+Cφ+D=0, whose normal vector is n=(A,B,C). Calculate the normal vector n of this feature point in the left camera view. L and the normal vector n in the right camera view R Calculate the angle θ between the two. k : This refers to the difference in phase normal vectors. The larger this value is, the lower the reliability of that point in the entire system.

7. The trinocular visual calibration method for fringe projection based on phase reliability weighting according to claim 6, characterized in that: In S11, the angle difference of S5 is transformed into optimization weights, and the following steps are taken: (1) Calculate the difference values ​​of the normal vectors of all M feature points {θ1, ..., θ2}. M }, plot the frequency distribution histogram and fit a smooth curve F(θ); (2) find the peak value of F(θ) and normalize the curve to F norm (θ), set λ to the peak value of F(θ); (3) set the exponential decay function G(θ) = e -λθ Calculate F norm The adaptive segmentation threshold between G(θ) and G(θ) is denoted as θ cut1 and θ cut2 (4) Construct a piecewise weighted model function W based on confidence level. n The first segment θ n ≤θ cut1 This part of the feature points is assigned full weight, and the second segment θ cut1 <θ n ≤θ cut2 The weight is determined by the exponential curve value corresponding to the adaptive segmentation threshold position, i.e., W. n =G(θ) cut1 ), the third segment θ n >θ cut2 The weight is W n =G(θ) cut2 ).

8. A trinocular vision calibration device based on phase reliability weighting for fringe projection, characterized in that: The system includes a projector, two cameras, a circular pattern calibration board, and a computing module. The circular pattern calibration board is placed within the common field of view of the projector and cameras. The projector and cameras are each communicatively connected to the computing module. The computing module includes an acquisition unit, a center extraction unit, a phase calculation unit, a normal vector calculation unit, a mapping unit, an ellipse fitting unit, a homography estimation unit, an adaptive weight calculation unit, and a calibration unit. The acquisition unit receives images of multiple feature circles in various poses and phase-shifted images acquired by the cameras from the calibration board. The center extraction unit processes the acquired images to extract the camera pixel coordinates of the center of each circle. The phase calculation unit calculates the wrapping phase of the acquired phase-shifted image using the phase-shift method and the multi-frequency heterodyne principle, and unfolds it into an absolute phase to obtain a bidirectional absolute phase map. The normal vector calculation unit calculates the normal vectors corresponding to the feature point regions obtained by the phase calculation unit in the left and right camera views, and calculates the angle between the two normal vectors. The mapping unit uses the obtained camera feature point neighborhood pixel coordinates and the bidirectional absolute phase map obtained by the calculation unit to calculate the corresponding circle center neighborhood point projector pixel coordinates through the bit-coordinate mapping formula; the ellipse fitting unit uses the ellipse fitting algorithm to extract the sub-pixel edges of the feature circles in the camera image and calculates the high-precision circle center coordinates; the homography estimation unit uses the camera pixel coordinate set of the local neighborhood of the feature points of each feature circle and the corresponding feature point local neighborhood point projector pixel coordinate set to solve the optimal homography matrix between the camera and the projector, and uses the fitted circle center coordinates and the homography matrix to calculate the circle center coordinates of the feature points on the projector; The adaptive weight calculation unit generates a frequency statistics graph and performs curve fitting using the difference in the normal vector; after normalizing the fitted curve, it calculates an adaptive segmentation threshold with a preset exponential decay curve; the adaptive segmentation threshold is used to divide the error curve into three segments, and the three segments are assigned corresponding weights respectively. The calibration unit optimizes the intrinsic parameters of the projector and the extrinsic parameters between the binocular camera and the projector using the bundle adjustment method. The intrinsic parameters of the two cameras are fixed, and a joint objective function is constructed that includes the reprojection error of the left camera, the reprojection error of the right camera, and the reprojection error of the projector. The weight of each point obtained in the adaptive weight calculation unit is used in the reprojection error function, and the Levenberg-Marquardt algorithm is used to solve for the optimal parameters of the system.

9. An electronic device, comprising: The processors are interconnected in the memory, which stores computer instructions. The processors execute the computer instructions, characterized in that they perform the phase reliability weighted fringe projection trinocular vision calibration method according to any one of claims 1 to 7.

10. A computer storage medium, wherein computer instructions are stored on the computer-readable storage medium, characterized in that: The computer instructions are used to cause the computer to execute the fringe projection trinocular vision calibration method based on phase reliability weighting as described in any one of claims 1 to 7.

Images