A 3D shape measurement method based on frequency-shifted fringe projection under complex illumination conditions

Abstract

The present invention discloses a three-dimensional shape measurement method based on frequency-shifted fringe projection under complex illumination conditions, comprising the following steps: firstly, introducing window function analysis, determining the highest fringe pattern sampling frequency and step length, generating a phase-shifted pattern and projecting it; then, obtaining the spectrum of the complex signal from the phase-shifted pattern by a four-step phase-shift method, and obtaining the Fourier spectrum diagram by discrete Fourier transform; then, using epipolar constraints, accurately locating the direct illumination position corresponding to the peak value of the Fourier domain spectrum response; finally, obtaining the same-name points between the projector and camera pixels by the direct illumination position, generating a depth map based on triangulation, and completing high-precision three-dimensional measurement. The present invention eliminates the interference of ambient light, constructs direct illumination and indirect illumination feature models, accurately obtains direct illumination information, significantly improves the imaging efficiency and robustness in surface measurement, and realizes high-precision, high-quality, and high-efficiency three-dimensional measurement in a strong mutual reflection complex illumination environment.

Description

Technical Field

[0001] The present invention relates to a three-dimensional shape measurement method based on frequency-shifted fringe projection under complex illumination conditions. Background Art

[0002] Due to its advantages such as non-contact, high precision, full-field measurement, and high efficiency of point cloud reconstruction, fringe projection technology has been widely used in the fields of precision measurement, biomedical engineering, digital modeling, etc., and is mainly divided into Fourier measurement profilometry and phase-shift measurement profilometry. An important prerequisite for three-dimensional imaging by fringe projection technology is that the light signal emitted by the projection device directly irradiates the target imaging point on the object and then reflects into the camera, that is, there is only a diffuse reflection light propagation phenomenon on the object surface. However, in actual industrial scenarios, the object to be measured often consists of complex concave-convex surfaces with specular reflection or surface materials with specular reflection (metal devices, ceramic products, gypsum bodies, etc.). The light received by the imaging point is a mixture of light signals emitted from different positions by the projection device and reflected once or multiple times and ambient light. The mixed light composed of light signals emitted from different positions of the projection device and reflected once or multiple times and ambient light is received by the object imaging point. The reflected light received by the camera not only includes the direct illumination formed by the projector pixels after one reflection, but also includes the indirect illumination brought by multiple reflections at other positions and ambient light, constituting a transmission model between the projector and the camera under complex illumination conditions. Schematic diagram of mutual reflection of structured light images. How to quickly and accurately obtain the 3D shape information of the target surface in the presence of strong mutual reflection is an important challenge in the current field of fringe projection three-dimensional imaging.

[0003] To eliminate the interference of strong mutual reflection on fringe projection three-dimensional imaging, relevant personnel have proposed a variety of methods. One is the method of explicitly separating illumination, which uses a high-frequency illumination mode to suppress the influence of strong mutual reflection and global illumination. However, on high-reflection surfaces, the illumination frequency of the projector will be difficult to meet the requirements, resulting in measurement failure. On this basis, pattern coding is performed using codes with only high-frequency modes or low-frequency modes to design elastic pattern modulation of global illumination interference, but it can only be used in a single light transmission scenario and cannot solve the three-dimensional reconstruction problems brought by specular reflection and strong mutual reflection. In addition, single-pixel imaging is used in the camera-projector system to separate direct illumination and indirect illumination. The imaging effect is excellent, but using single-pixel imaging requires tens of thousands or even millions of patterns, and the time cost of data acquisition and processing is too high.

[0004] Another method combining frequency analysis and environmental matting realizes the three-dimensional reconstruction of the surfaces of transparent and shiny objects. However, this method cannot perform three-dimensional measurement on objects with high curvature or highlight points. On this basis, the baseline frequency shift method (BFSM) of projecting frequency-shifted fringe patterns is proposed. By projecting multiple sine fringe patterns with different frequencies, the projector rows and columns are encoded using the fringe frequency, and it is proposed to use the discrete Fourier transform to separate the direct illumination and the indirect illumination in the frequency domain. However, in order to establish the correspondence between the projector and the camera in the frequency space, the frequency shift method needs to project a large number of structured light patterns, which is easily interfered by the background light, and there is a possibility of error in obtaining the direct illumination based on the method of local maximum of the spectrum. The technical solution of the present invention to solve the above problems is: a three-dimensional shape measurement method based on frequency-shifted fringe projection under complex illumination conditions, including the following steps:

[0005] Step 1: Introduce window function analysis, determine the highest fringe pattern sampling frequency and step size, generate phase-shifted patterns and project them.

[0006] Step 2: Through the four-step phase-shift method, perform differential operations on the fringe images collected under four phase-shift conditions to obtain the real part and the imaginary part of the complex signal respectively, and perform a discrete Fourier transform on the complex signal to obtain the frequency-domain signal.

[0007] Step 3: Use the epipolar constraint to accurately locate the position of the direct illumination corresponding to the peak of the Fourier-domain spectrum response.

[0008] Step 4: Obtain the corresponding points between the projector and the camera pixels through the position of the direct illumination, generate a depth map based on triangulation, and complete high-precision three-dimensional measurement.

[0009] In the above three-dimensional shape measurement method based on frequency-shifted fringe projection under complex illumination conditions, the specific process of the first step is as follows:

[0010] 1-1) Based on the Shannon-Nyquist sampling theorem, optimize the frequency distribution of the fringe pattern by adjusting the width of the initial window function to ensure that the highest frequency of the fringe is effectively controlled and reduce the total number of images required.

[0011] 1-2) Generate projection patterns, and the formula for the projected fringe pattern is as follows:

[0012]

[0013] where A is the background light intensity, B is the modulation intensity, C P is the column coordinate or row coordinate on the projected image, and f n is the fringe frequency of the nth fringe pattern.

[0014] 1-3) Multiple (usually three or four) fringe images with different phase offsets are generated by gradually changing the phase of the projected fringes. The phase shift can be achieved by moving the fringe pattern in the projector or adjusting the pattern itself. To accurately measure the geometry of the object surface, a periodic fringe pattern is generated by a computer.

[0015] Calculate the intensity change of each fringe at different phases according to the following formula.

[0016]

[0017] This process includes calculating the intensity of the fringes at four phases, and accordingly generating the corresponding fringe images.

[0018] 1-4) Burn the fringe images generated by the computer in the horizontal and vertical directions into the projector. By precisely controlling the projector, the fringe images evenly cover the object surface, and a planar array camera is used to collect the horizontal and vertical fringe images.

[0019] For the above three-dimensional shape measurement method based on frequency-shifted fringe projection under complex illumination conditions, the specific process of the second step is as follows:

[0020] 2-1) Process the collected fringe images based on the four-step phase-shifting method. Based on the intensity difference of the fringe images at different phases, a complex signal is constructed by calculating the difference between the real part and the imaginary part according to the following formula.

[0021] D(x,y) = I 0 (x,y) - I 2 (x,y) + j[I 1 (x,y) - I 3 (x,y)]

[0022] 2-2) Perform a discrete Fourier transform (DFT) on the complex signal obtained in the previous step to obtain the frequency-domain signal.

[0023] When the complex signal is Fourier-transformed, only the positive frequency spectrum part is retained, and the DC component and the negative frequency spectrum part are eliminated, that is, there is no background illumination interference. At the same time, the sampling frequency only needs to be greater than or equal to the projector resolution, breaking through the Nyquist theorem limit and reducing the highest frequency to half.

[0024] 2-4) According to the components of direct and indirect illumination analyzed by the spectrum, accurately distinguish the main light source and the reflected light that affect imaging. Conduct a detailed analysis of each peak of the frequency-domain signal to effectively eliminate the interference of background illumination and ensure the accuracy of spectrum analysis.

[0025] For the above three-dimensional shape measurement method based on frequency-shifted fringe projection under complex illumination conditions, the specific process of the third step is as follows:

[0026] 3-1) Load the calibration data of the projector and the camera, including their intrinsic matrix, rotation matrix, and translation vector.

[0027] 3-2) Utilize the loaded calibration parameters to construct the projection matrices of the camera and the projector. These matrices convert three-dimensional world coordinates into two-dimensional image coordinates of the corresponding devices. The key to this step is to combine the rotation matrix and the translation vector and apply the intrinsic matrix to form a complete projection matrix.

[0028] 3-3) Based on the projection matrix and the intrinsic parameters obtained in the above steps, calculate the position of the camera's optical center in the world coordinate system, and then map this point onto the projector's image plane through the projector's projection matrix to obtain the epipole. Next, for any given camera image coordinates, calculate the corresponding epipolar line on the projector's image plane through undistortion processing and coordinate transformation.

[0029] 3-4) In the frequency domain, analyze and identify all significant peaks. These peaks represent possible light sources, including direct light and indirect light generated by complex lighting conditions.

[0030] 3-5) Use a specific threshold to identify significant peaks in the spectrum. This step is achieved by enumerating each point of the frequency-domain signal and comparing its intensity with that of adjacent points. If the intensity of a point is not only higher than the threshold but also higher than that of its adjacent points, then this point is considered a peak. For closely adjacent peaks, they are merged by calculating the weighted average position to ensure the accuracy and reliability of the position of each peak.

[0031] 3-6) Based on the known geometric relationship between the camera and the projector, the light intensity received by the camera pixel (X, Y) includes direct illumination from the projector pixel (x 1 , y 1 ) and indirect illumination after strong mutual reflection of the projector pixel (x 2 , y 2 ). The projector pixels (x 1 , y 1 ) and (x 2 , y 2 ) are located on and outside the epipolar line respectively. Therefore, when searching for the pixel coordinates of the projector's direct light, it can be determined as the point closest to the epipolar line, and the direct light and indirect light are distinguished by the epipolar line.

[0032] 3-7) Through pixel-by-pixel precise positioning of the direct light of the spectral response peaks in the Fourier domain, accurately separate the direct light.

[0033] For the above three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions, the specific process of step four is as follows:

[0034] 4-1) The system creates the corresponding projection matrix by loading the calibrated camera and projector parameters.

[0035] 4-2) Based on the coordinates of the corresponding points and the known geometric relationship between the camera and the projector, calculate the three-dimensional coordinates, and convert the calculated depth value Z into a depth map.

[0036] 4-3) Extract the three-dimensional coordinates from the depth map to generate point cloud data.

[0037] The beneficial effects of the present invention are as follows:

[0038] 1. By applying the window function technology, the present invention reduces the highest fringe frequency, reduces the number of required projection patterns, and effectively reduces the computational complexity and resource consumption in the imaging process.

[0039] 2. Through the improved frequency shift method, the present invention can dynamically distinguish direct illumination and indirect illumination. By corresponding the intensity signal of the projector pixel to the peak value in the frequency domain, this method can accurately distinguish whether the signal is generated by direct illumination or reflected illumination, greatly improving the accuracy and reliability of imaging.

[0040] 3. By effectively processing strong mutual reflection and complex illumination environments, the present invention not only improves the imaging accuracy but also enhances the robustness of the system. Experimental results show that the improved frequency shift method shows better performance than the traditional method in three-dimensional reconstruction of different materials. Especially when dealing with metal objects in dark areas, the imaging integrity is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the main optical structure used in the present invention.

[0042] Figure 2 It is a detection flow chart of the present invention.

[0043] Figure 3 It is a schematic diagram of the Fourier transform of the rectangular window function of the present invention.

[0044] Figure 4 It is a fringe pattern generated by the present invention.

[0045] Figure 5 It is an image obtained by using an area array camera in the present invention.

[0046] Figure 6 It is the constraint relationship between the peak value and the epipolar line of the present invention.

[0047] Figure 7 It is a three-dimensional reconstruction diagram of the object to be measured in the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0048] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0049] 1. As Figure 1 shown, the optical structure of the present invention consists of three parts: a projector, a area array camera, and a telecentric lens. The resolution of the projector, which is the resolution of the input fringe pattern, affects the measurement accuracy. The higher the resolution of the projector used, the higher the accuracy. The frame rate of the projector affects the measurement speed of the system. The fewer the projected pictures, the higher the frame rate, and the higher the measurement speed of the entire system. The working distance needs to be considered in combination with the actual measurement object. At least it is necessary to ensure that the measured object is within the field of view of the projector. Considering that larger objects need to be measured, the projector needs to be kept about 0.5m - 1.0m away from the object to be measured.

[0050] 2. Area array camera: The requirement of the present invention for the camera resolution is at least greater than the resolution of the projector. The area array camera used has a high resolution of 2448×2048 to ensure that the complete projected fringes are captured.

[0051] 3. Telecentric lens: When an ordinary lens takes pictures of an object, if there is a height difference in the object itself, then the phenomenon of "near is large and far is small" will occur during imaging, and the number of pixels occupied by the same-sized object will be different. The telecentric lens selected in the present invention can overcome this phenomenon of "near is large and far is small", ensure that the same-sized object occupies the same number of pixels during imaging, and thus improve the detection accuracy.

[0052] As Figure 2 shown, a three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions includes the following steps:

[0053] Step 1: Introduce window function analysis, determine the highest fringe pattern sampling frequency and step size, generate a phase-shifted pattern and project it.

[0054] The specific process of Step 1 is as follows:

[0055] 1-1) As Figure 3 First, introduce a window function to optimize the frequency distribution of the fringe pattern. Select to use M = 121 different fringe patterns to ensure that the fringe pattern minimizes the computational burden while meeting the imaging requirements.

[0056] 1-2) As Figure 4 shown, adopt four-step phase shift, and a total of M * 4 = 484 pictures are processed. Burn the generated fringe images with different frequencies in the horizontal and vertical directions into the projector.

[0057] 1-3) As Figure 5 shown, obtain the original image through the area array camera. Among them, using a telecentric lens can overcome the phenomenon of "near is large and far is small" caused by the height difference, thereby reducing the imaging distortion.

[0058] Step 2: Through the four-step phase-shifting method, perform differential operations on the fringe images collected under four phase-shifting conditions to obtain the real and imaginary parts of the complex signal respectively, and perform a discrete Fourier transform on the complex signal to obtain the frequency-domain signal.

[0059] The specific process of Step 2 is as follows:

[0060] 2-1) Spectral response of the real signal obtained from the fringe pattern

[0061]

[0062] Among them, C DC is the spectral response generated by the DC component at position C P =0. The real signal will generate conjugate spectral responses at two positions, and the response intensity is where the negative spectral part is redundant.

[0063] 2-2) In order to remove the DC component and negative spectral response, perform four-step phase-shifting processing on the fringe pattern projected onto the target object. Specifically, by changing the phase of each image (0°, 90°, 180°, 270°), collect four fringe patterns with different phases. Let be substituted into the following formula. Let the four images be I 0 , I 1 , I 2 , I 3 .

[0064]

[0065] 2-3) Calculate the difference: Use these four images to calculate two key difference values:

[0066] D C =I 0 -I 2 : The difference calculated in this step is the difference of the cosine term of the phase-shifted image, providing the real part of the complex signal.

[0067] D S =I 1 -I 3 : The difference calculated in this step is the difference of the sine term of the phase-shifted image, providing the imaginary part of the complex signal.

[0068] 2-4) Through the real part D C and the imaginary part D S obtained from the above calculations, a complex signal can be synthesized:

[0069] D = D C +jD S

[0070] The obtained complex signal D is subjected to Fourier transform to obtain the spectral response C of the signal. In this process, focus on retaining the positive spectral part of the signal while removing the DC component and all negative spectral components, effectively eliminating the interference of background illumination and ensuring the accuracy of spectral analysis.

[0071] Step 3: Using the epipolar constraint, accurately locate the direct illumination position corresponding to the peak of the spectral response in the Fourier domain.

[0072] The specific process of Step 3 is as follows:

[0073] 3-1) Load the calibration data of the projector and the camera, including their intrinsic matrix, rotation matrix, and translation vector.

[0074] 3-2) Using the loaded calibration parameters, construct the projection matrices of the camera and the projector. The projection matrix is a 3x4 matrix composed of the rotation matrix and the translation vector, which can convert the three-dimensional world coordinates into the two-dimensional image coordinates of the corresponding device.

[0075] 3-3) Through the above steps to obtain the projection matrix and the intrinsics, calculate the position of the camera's optical center in the world coordinate system, and then map this point to the projector's image plane through the projector's projection matrix to obtain the epipole. For any given camera image coordinates, calculate its corresponding epipolar line on the projector's image plane through undistortion processing and coordinate transformation.

[0076] 3-4) Set a suitable threshold, thres = 1 to determine the valid and invalid data regions, regard the frequency-domain response below this threshold as background noise and exclude it, and only retain the peak responses above the threshold. These retained peaks represent the possible direct illumination positions.

[0077] 3-5) By calculating the modulus of each complex number, obtain the amplitude spectrum, and extract the amplitude value of each frequency point from the complex numbers. Search for the index of the maximum value at each position along the specified axis. This means that for each column of the image, it finds the position of the frequency point with the largest amplitude, and this position is the position of the peak.

[0078] 3-6) As Figure 6 shown, apply the epipolar constraint to the retained peaks, and the point closest to the epipolar line is the direct illumination point. Separate the direct illumination and the indirect illumination by calculating the depth information of each pixel point, and use the separated direct illumination information to obtain the disparity map.

[0079] Step 4: Obtain the corresponding points between the projector and the camera pixels through the direct illumination position, generate the depth map based on triangulation, and complete the high-precision three-dimensional measurement.

[0080] The specific process of Step 4 is as follows:

[0081] 4-1) Load the configuration data of the calibrated camera and projector, and construct a matrix for image projection.

[0082] 4-2) As Figure 7 shown, from the disparity map obtained in the above steps, use the coordinates of corresponding points and the known geometric relationship between the camera and the projector to calculate the three-dimensional coordinates, and convert the disparity map into point cloud data.

[0083] 4-3) Save the processed point cloud in a standard format for further analysis or visualization use, and complete high-precision three-dimensional measurement.

Claims

1. A three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions, characterized in that: The following steps are involved: Step 1: Introduce window function analysis to determine the highest fringe pattern sampling frequency and step size, generate a phase shift pattern and project it; Step 2: Through the four-step phase shift method, the fringe images collected under four phase shift conditions are differentially operated to obtain the real part and the imaginary part of the complex signal respectively, and the complex signal is subjected to discrete Fourier transform to obtain the frequency domain signal; Step 3: Use epipolar constraints to accurately locate the direct illumination position corresponding to the peak of the Fourier domain spectrum response; Step 4: Obtain the same-name points between projector and camera pixels through direct illumination positions, and generate a depth map based on triangulation.

2. The three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions according to claim 1, characterized in that: The specific process of step 1 is: 1-1) Based on the Shannon-Nyquist sampling theorem, the frequency distribution of the fringe pattern is finely adjusted by adjusting the width of the initial window function to reduce the number of projections; 1-2) Generate a periodic fringe pattern by computer, and calculate the intensity change of each fringe at different phases according to the following formula; This process involves calculating the fringes The intensity of the four phases generates the corresponding fringe images; 1-3) Burn the pre-generated horizontal and vertical stripe images into the projector and project them so that the images completely cover the object to be measured; 1-4) Use a high-resolution area array camera with a telecentric lens to capture images to reduce distortion caused by differences in imaging distance.

3. The three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions according to claim 2, characterized in that: The specific process of step 2 is: 2-1) Using the four-step phase shift method to process the fringe image generated above, according to the intensity difference of the fringe pattern at different phases, the difference between the real part and the imaginary part is constructed by phase difference calculation to construct a complex signal; D C =I0-I2: This step calculates the difference of the cosine terms of the phase-shifted image, providing the real part of the complex signal; D S =I1-I3: This step calculates the difference of the sinusoidal terms of the phase-shifted image, providing the imaginary part of the complex signal; Complex signal: D = D C +jD S 2-2) Perform discrete Fourier transform (DFT) on the complex signal obtained in the previous step to obtain a frequency domain signal. Only the positive spectrum part of the complex signal is retained, and the DC component and the negative spectrum part are eliminated. In addition, the sampling frequency of the system only needs to be greater than or equal to the resolution of the projector, breaking through the Nyquist theorem limitation and reducing the maximum frequency to half.

4. The three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions according to claim 3, characterized in that: The specific process of step three is: 3-1) Use the calibration parameters of the camera and projector to establish the projection matrix, calculate the position of the optical center of the camera in the world coordinate system, and map it to the image plane of the projector to determine the poles and corresponding epipolar lines; 3-2) Analyze each peak value of the frequency domain signal to eliminate the interference of background light and ensure the accuracy of spectrum analysis; 3-3) Analyze the response characteristics of different frequencies in the frequency domain, and by setting an appropriate threshold, only retain the peak response that exceeds the threshold. Utilize the polar line constraint to accurately locate the direct illumination of the spectral response peak in the Fourier domain pixel by pixel, and accurately separate the direct illumination source.

5. The three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions according to claim 4, characterized in that: The specific process of step 1 is: The same-name points between the projector and camera pixels are obtained through the direct illumination position, and a depth map is generated based on triangulation to complete high-precision three-dimensional measurement.

Images