Three-dimensional shape measurement method based on frequency shift fringe projection under complex illumination condition
Through the method based on frequency shift fringe projection, the window function and Fourier transform are used to separate direct and indirect light under complex lighting conditions, solving the accuracy of three-dimensional imaging under complex lighting conditions, and achieving high-precision three-dimensional shape measurement .
Patent Information
- Application Number
- CN202411023152.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-29
AI Technical Summary
Under complex lighting conditions, streak projection three-dimensional imaging is difficult to quickly and accurately obtain the 3D shape information of the target surface, especially in the presence of high reflection and specular reflection, and existing methods are difficult to effectively deal with the strong mutual reflection and complex lighting. interference.
Using the method based on frequency shift fringe projection, the Fourier domain spectrum response peak is accurately positioned by introducing window function analysis, four-step phase shift method and discrete Fourier transform, and the Fourier domain spectrum response peak is separated from direct and indirect illumination to generate depth maps for high accuracy. Three-dimensional measurement.
It improves the accuracy and robustness of imaging, reduces computational complexity and resource consumption, and significantly improves the accuracy and imaging completeness of three-dimensional reconstruction in complex lighting environments.
Smart Images

Figure CN118936361A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a three-dimensional shape measurement method based on frequency-shift fringe projection under complex illumination conditions. Background Art
[0002] Fringe projection technology is widely used in precision measurement, biomedical engineering, digital modeling and other fields due to its advantages of non-contact, high precision, full field measurement, and high efficiency of point cloud reconstruction. It is mainly divided into Fourier measurement profilometry and phase shift measurement profilometry. The important premise of fringe projection technology three-dimensional imaging is that the light signal emitted by the projection device directly illuminates the target imaging point on the object and then reflects into the camera, that is, there is only diffuse light propagation on the surface of the object. However, in actual industrial scenarios, the measured objects are often composed of complex concave and convex surfaces with glossy reflection or mirror reflection surface materials (metal devices, ceramic products, gypsum, etc.). The light received by the imaging point is a mixture of light signals emitted from different positions of the projection device and reflected once or multiple times and ambient light. The mixed light composed of light signals emitted from different positions of the projection device and reflected once or multiple times and ambient light is received by the imaging point of the object. The reflected light received by the camera not only includes the direct illumination formed by the projector pixel after one reflection, but also includes the indirect illumination and ambient light brought by multiple reflections at other positions, which constitutes the transmission model between the projector and the camera under complex lighting conditions. Schematic diagram of structured light image mutual reflection. How to quickly and accurately obtain the 3D shape information of the target surface in the presence of strong mutual reflection is an important challenge in the current field of fringe projection 3D imaging.
[0003] In order to eliminate the interference of strong mutual reflection on fringe projection three-dimensional imaging, relevant personnel have proposed a variety of methods. One is the method of explicit separation of illumination, which uses high-frequency illumination mode to suppress the influence of strong mutual reflection and global illumination. However, on highly reflective surfaces, the illumination frequency of the projector will be difficult to meet the demand and cause measurement failure. On this basis, patterns are encoded using codes with only high-frequency modes or low-frequency modes, and elastic patterns are designed to modulate global illumination interference, but they can only be used in a single light transmission scenario and cannot solve the three-dimensional reconstruction problems caused by mirror reflection and strong mutual reflection. In addition, single-pixel imaging is used in the camera-projector system to separate direct illumination and indirect illumination. The imaging effect is excellent, but the use of single-pixel imaging requires tens of thousands or even millions of patterns, and the time cost of data acquisition and processing is too high.
[0004] Another method based on the combination of frequency analysis and environmental cutout achieves three-dimensional reconstruction of the surface of transparent and glossy objects, but this method cannot perform three-dimensional measurement for objects with high curvature or high light points. On this basis, a baseline frequency shift method (BFSM) of projected frequency-shifted fringe patterns is proposed. By projecting multiple sinusoidal fringe patterns with different frequencies and encoding the projector rows and columns using the fringe frequencies, it is proposed to use discrete Fourier transform to separate direct illumination and indirect illumination in the frequency domain. However, in order to establish the corresponding relationship between the projector and the camera in the frequency space, the frequency shift method needs to project a large number of structured light patterns, which is easily affected by background illumination, and there is a possibility of error in obtaining direct illumination based on the method of obtaining the local maximum value of the spectrum. The technical solution of the present invention to solve the above problems is: a three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions, comprising the following steps:
[0005] Step 1: Introduce window function analysis to determine the highest fringe pattern sampling frequency and step size, generate a phase shift pattern and project it.
[0006] Step 2: Obtain the spectrum of the complex signal from the phase shift pattern using the four-step phase shift method, and then use discrete Fourier transform to obtain the Fourier spectrum diagram.
[0007] Step 3: Use epipolar constraints to accurately locate the direct illumination position corresponding to the peak of the Fourier domain spectrum response.
[0008] Step 4: Obtain the same-name points between the projector and camera pixels through the direct illumination position, generate a depth map based on triangulation, and complete high-precision 3D measurement.
[0009] The specific process of the above-mentioned three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions is as follows:
[0010] 1-1) Based on the Shannon-Nyquist sampling theorem, the frequency distribution of the fringe pattern is optimized by adjusting the width of the initial window function to ensure that the highest frequency of the fringe is effectively controlled and the number of images required overall is reduced.
[0011] 1-2) Generate projection pattern, the projection fringe pattern formula is as follows:
[0012]
[0013] Where A is the background light intensity, B is the modulation intensity, C P is the column coordinate or row coordinate on the projected image, f n is the fringe frequency of the nth fringe pattern.
[0014] 1-3) Generate multiple (usually three or four) fringe images with different phase shifts by gradually changing the phase of the projected fringe. Phase shift can be achieved by moving the fringe pattern in the projector or adjusting the pattern itself. In order to accurately measure the geometric shape of the object surface, the periodic fringe pattern is generated by computer.
[0015] The intensity change of each fringe at different phases is calculated according to the following formula.
[0016]
[0017] This process involves calculating the fringes The intensities of the four phases are used to generate corresponding fringe images.
[0018] 1-4) Burn the computer-generated horizontal and vertical stripe images into the projector. By precisely controlling the projector, the stripe image evenly covers the surface of the object, and the horizontal and vertical stripe images are collected using an area array camera.
[0019] The specific process of the above-mentioned three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions is as follows:
[0020] 2-1) The fringe image collected above is processed based on the four-step phase shift method. According to the intensity difference of the fringe pattern at different phases, the complex signal is constructed by calculating the difference between the real part and the imaginary part based on the following formula.
[0021] D(x,y)=I 0 (x,y)-I 2 (x,y)+i(I 1 (x,y)-I 3 (x,y)
[0022] 2-2) Perform discrete Fourier transform (DFT) on the complex signal obtained in the previous step to obtain a frequency domain signal.
[0023] 2-3) The complex signal is subjected to Fourier transform to retain only the positive spectrum part and eliminate the DC component and the negative spectrum part, that is, there is no background light interference. At the same time, the sampling frequency only needs to be greater than or equal to the projector resolution, breaking through the Nyquist theorem limitation and reducing the maximum frequency to half.
[0024] 2-4) According to the spectrum analysis, the direct and indirect light components are accurately distinguished from the main light source and reflected light that affect the imaging. Each peak of the frequency domain signal is carefully analyzed to effectively eliminate the interference of background light and ensure the accuracy of spectrum analysis.
[0025] In the above-mentioned three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions, the specific process of step three is as follows:
[0026] 3-1) Load the calibration data of the projector and camera, including their intrinsic parameter matrices, rotation matrices and translation vectors.
[0027] 3-2) Using the loaded calibration parameters, construct the projection matrices for the camera and projector. These matrices convert 3D world coordinates into 2D image coordinates for the corresponding device. The key to this step is to combine the rotation matrix and translation vector, apply the intrinsic matrix, and form a complete projection matrix.
[0028] 3-3) Using the projection matrix and intrinsic parameters obtained in the above steps, calculate the position of the optical center of the camera in the world coordinate system, and then map this point to the image plane of the projector through the projection matrix of the projector to obtain the pole. Next, for any given camera image coordinate, calculate its corresponding pole line on the projector image plane through dedistortion processing and coordinate transformation.
[0029] 3-4) In the frequency domain, analyze and identify all significant peaks. These peaks represent possible sources of illumination, including direct illumination and indirect illumination caused by complex lighting conditions.
[0030] 3-5) Use a specific threshold to identify significant peaks in the spectrum. This step is achieved by enumerating each point of the frequency domain signal and comparing its intensity with that of adjacent points. If the intensity of a point is not only higher than the threshold, but also higher than the intensity of its adjacent points, the point is considered to be a peak. For closely adjacent peaks, they are merged by calculating the weighted average position to ensure that the position of each peak is accurate and reliable.
[0031] 3-6) Based on the known geometric relationship between the camera and the projector, the light intensity received by the camera pixel (X, Y) includes the light intensity from the projector pixel (x 1 ,y 1 ) of direct illumination and projector pixels (x 2 ,y 2 ) Indirect illumination after strong mutual reflection, projector pixel (x 1 ,y 1 ) and (x 2 ,y 2 ) are located on and outside the epipolar line, respectively. Therefore, when looking for the coordinates of the projector's direct illumination pixels, we can determine them as the points closest to the epipolar line, and distinguish between direct illumination and indirect illumination through the epipolar line.
[0032] 3-7) The direct illumination is precisely located by pixel by pixel at the peak of the spectral response in the Fourier domain, and the direct illumination is precisely separated.
[0033] In the above-mentioned three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions, the specific process of step 4 is as follows:
[0034] 4-1) The system creates the corresponding projection matrix by loading the calibrated camera and projector parameters.
[0035] 4-2) Based on the coordinates of the same-name points and the known geometric relationship between the camera and the projector, the three-dimensional coordinates are calculated, and the calculated depth value Z is converted into a depth map.
[0036] 4-3) Extract three-dimensional coordinates from the depth map and generate point cloud data.
[0037] The beneficial effects of the present invention are:
[0038] 1. The present invention reduces the maximum fringe frequency by applying window function technology, thereby reducing the number of required projection patterns and effectively reducing the computational complexity and resource consumption in the imaging process.
[0039] 2. The present invention can dynamically distinguish direct illumination from indirect illumination through an improved frequency shift method. By matching the intensity signal of the projector pixel with the peak value in the frequency domain, this method can accurately distinguish whether the signal is generated by direct illumination or reflected illumination, greatly improving the accuracy and reliability of imaging.
[0040] 3. The present invention effectively handles strong interreflection and complex lighting environments, which not only improves the accuracy of imaging, but also enhances the robustness of the system. Experimental results show that the improved frequency shift method performs better than traditional methods in 3D reconstruction of different materials, especially when dealing with metal objects in dark areas, the imaging integrity is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 This is a schematic diagram of the main optical structure used in the present invention.
[0042] Figure 2 It is a detection flow chart of the present invention.
[0043] Figure 3 Schematic diagram of the Fourier transform of the rectangular window function of the present invention.
[0044] Figure 4 Fringe pattern generated for the present invention.
[0045] Figure 5 The present invention uses images acquired by an area array camera.
[0046] Figure 6 This is the constraint relationship between the peak value and the polar line of the present invention.
[0047] Figure 7 This is a three-dimensional reconstruction diagram of the measured object according to the present invention. DETAILED DESCRIPTION
[0048] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0049] 1. As Figure 1 As shown, the optical structure of the present invention consists of three parts: a projector, an array camera and a telecentric lens. The resolution of the projector, that is, the resolution of the input fringe pattern, affects the measurement accuracy. The higher the resolution of the projector used, the higher the accuracy. The frame rate of the projector affects the measurement speed of the system. The fewer the projected images, the higher the frame rate, and the higher the measurement speed of the entire system. The working distance needs to be considered in combination with the actual measurement object. At least it is necessary to ensure that the measured object is within the field of view of the projector. Considering the need to measure larger objects, the projector needs to be kept about 0.5m-1.0m away from the object to be measured.
[0050] 2. Area array camera: The present invention requires that the camera resolution must be at least greater than the resolution of the projector. The area array camera used has a high resolution of 2448×2048 to ensure that the complete projection fringes are captured.
[0051] 3. Telecentric lens: When ordinary lenses are used to shoot objects, if there is a height difference between the objects themselves, the image will appear larger when near and smaller when far, and the number of pixels occupied by objects of the same size will be different. The telecentric lens selected in the present invention can overcome this phenomenon of larger when near and smaller when far, ensuring that objects of the same size occupy the same number of pixels when imaging, thereby improving detection accuracy.
[0052] like Figure 2 As shown, a three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions comprises the following steps:
[0053] Step 1: Introduce window function analysis to determine the highest fringe pattern sampling frequency and step size, generate a phase shift pattern and project it.
[0054] The specific process of step one is:
[0055] 1-1) If Figure 3 Firstly, the window function is introduced to optimize the frequency distribution of the fringe pattern, and M=121 different fringe patterns are selected to ensure that the fringe pattern meets the imaging requirements while minimizing the computational burden.
[0056] 1-2) If Figure 4 As shown, a four-step phase shift is used to process M*4=484 images in total, and the generated horizontal and vertical stripe images of different frequencies are burned into the projector.
[0057] 1-3) If Figure 5 As shown, the original image is obtained by an area array camera, wherein a telecentric lens is used to overcome the phenomenon that near objects are larger and far objects are smaller due to height difference, thereby reducing imaging distortion.
[0058] Step 2: Obtain the spectrum of the complex signal from the phase shift pattern using the four-step phase shift method, and obtain the Fourier spectrum using discrete Fourier transform.
[0059] The specific process of step 2 is:
[0060] 2-1) Real signal spectrum response obtained from fringe pattern
[0061]
[0062] Among them, C DC is the DC component at position C P = 0, the spectrum response produced by the real signal will be A conjugate spectrum response will be generated at two locations, and the response intensity is The negative spectrum part is redundant.
[0063] 2-2) In order to remove the DC component and negative spectral response, the fringe pattern projected onto the target object is subjected to a four-step phase shift process. Specifically, by changing the phase of each image (0°, 90°, 180°, 270°), four fringe patterns with different phases are collected. Substitute into the following formula. Assume that these four images are I 0 ,I 1 ,I 2 ,I 3 .
[0064]
[0065] 2-3) Calculate the difference: Use these four images to calculate two key difference values:
[0066] D C =I 0 -I 2 : This step calculates the difference of the cosine terms of the phase-shifted image, providing the real part of the complex signal.
[0067] D S =I 1 -I 3 : This step calculates the difference of the sinusoidal terms of the phase-shifted image, providing the imaginary part of the complex signal.
[0068] 2-4) The real part D obtained by the above calculation C and the imaginary part D S , complex signals can be synthesized:
[0069] D=D C +jD S
[0070] The obtained complex signal D is Fourier transformed to obtain the signal's spectrum response C. In this process, we focus on retaining the positive spectrum part of the signal while removing the DC component and all negative spectrum components, effectively eliminating the interference of background illumination and ensuring the accuracy of spectrum analysis.
[0071] Step 3: Use epipolar constraints to accurately locate the direct illumination position corresponding to the peak of the Fourier domain spectrum response.
[0072] The specific process of step three is:
[0073] 3-1) Load the calibration data of the projector and camera, including their intrinsic parameter matrices, rotation matrices and translation vectors.
[0074] 3-2) Using the loaded calibration parameters, construct the projection matrix of the camera and projector. The projection matrix is a 3x4 matrix consisting of a rotation matrix and a translation vector, which can convert the three-dimensional world coordinates into the two-dimensional image coordinates of the corresponding device.
[0075] 3-3) Through the above steps, the projection matrix and internal parameters are obtained, the position of the optical center of the camera in the world coordinate system is calculated, and then this point is mapped to the image plane of the projector through the projection matrix of the projector to obtain the pole. For any given camera image coordinate, the corresponding pole line on the projector image plane is calculated through dedistortion processing and coordinate transformation.
[0076] 3-4) Set a suitable threshold, thres=1, to determine the valid and invalid data areas, and treat the frequency domain responses below this threshold as background noise and exclude them. Only the peak responses above the threshold are retained. These retained peaks represent possible direct illumination positions.
[0077] 3-5) By calculating the modulus of each complex number, the amplitude spectrum is obtained, and the amplitude value of each frequency point is extracted from the complex number. The index of the maximum value at each position is searched along the specified axis. This means that for each column of the image, it finds the position of the frequency point with the largest amplitude, which is the position of the peak.
[0078] 3-6) If Figure 6 As shown in Figure 1, the epipolar constraint is applied to the retained peak, and the point closest to the epipolar line is the direct illumination point. The direct illumination and indirect illumination are separated by calculating the depth information of each pixel, and the disparity map is obtained by using the separated direct illumination information.
[0079] Step 4: Obtain the same-name points between the projector and camera pixels through the direct illumination position, generate a depth map based on triangulation, and complete high-precision 3D measurement.
[0080] The specific process of step 4 is as follows:
[0081] 4-1) Load the calibrated camera and projector configuration data and build the matrix for image projection.
[0082] 4-2) If Figure 7 As shown, the disparity map obtained from the above steps uses the coordinates of the same-name points and the known geometric relationship between the camera and the projector to calculate the three-dimensional coordinates and convert the disparity map into point cloud data.
[0083] 4-3) Save the processed point cloud in a standard format for further analysis or visualization to complete high-precision 3D measurement.
Claims
1. A three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions. It is characterized in that The following steps are involved: Step 1: Introduce window function analysis to determine the highest fringe pattern sampling frequency and step size, generate a phase shift pattern and project it; Step 2: Obtain the spectrum of the complex signal from the phase shift pattern by a four-step phase shift method, and obtain the Fourier spectrum diagram by discrete Fourier transform; Step 3: Use epipolar constraints to accurately locate the direct illumination position corresponding to the peak of the Fourier domain spectrum response; Step 4: Obtain the same-name points between projector and camera pixels through direct illumination positions, and generate a depth map based on triangulation.
2. A three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions according to claim 1, It is characterized in that The specific process of step 1 is: 2-1) Based on the Shannon-Nyquist sampling theorem, the frequency distribution of the fringe pattern is fine-tuned by adjusting the width of the initial window function to reduce the number of projections. 2-2) Generate a periodic fringe pattern by computer and calculate the intensity change of each fringe at different phases according to the following formula. This process involves calculating the fringes The intensities of the four phases are used to generate corresponding fringe images. 2-3) Burn the pre-generated horizontal and vertical stripe images into the projector and project them so that the images completely cover the object to be measured. 2-4) Use a high-resolution area array camera with a telecentric lens to capture images to reduce distortion caused by differences in imaging distance.
3. A three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions according to claim 2, It is characterized in that The specific process of step 2 is: 3-1) Use the four-step phase shift method to process the fringe image generated above. According to the intensity difference of the fringe pattern at different phases, the difference between the real part and the imaginary part is constructed by phase difference calculation to construct a complex signal. D C =I 0 -I 2 : This step calculates the difference of the cosine terms of the phase-shifted image, providing the real part of the complex signal. D S =I 1 -I 3 : This step calculates the difference of the sinusoidal terms of the phase-shifted image, providing the imaginary part of the complex signal. Complex signal: D = D C +jD S 3-2) Perform discrete Fourier transform (DFT) on the complex signal obtained in the previous step to obtain a frequency domain signal. Only the positive spectrum part of the complex signal is retained, and the DC component and the negative spectrum part are eliminated. In addition, the sampling frequency of the system only needs to be greater than or equal to the resolution of the projector, breaking through the Nyquist theorem limitation and reducing the maximum frequency to half.
4. The three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions according to claim 3, It is characterized in that The specific process of step three is: 4-1) Use the calibration parameters of the camera and projector to establish the projection matrix, calculate the position of the camera's optical center in the world coordinate system, and map it to the image plane of the projector to determine the poles and corresponding epipolar lines. 4-2) Analyze each peak value of the frequency domain signal to eliminate the interference of background light and ensure the accuracy of spectrum analysis. 4-3) Analyze the response characteristics of different frequencies in the frequency domain, and by setting an appropriate threshold, only retain the peak response that exceeds the threshold. Utilize the polar line constraint to accurately locate the direct illumination of the spectral response peak in the Fourier domain pixel by pixel, and accurately separate the direct illumination source.
5. The three-dimensional shape measurement method based on frequency-shifted fringe projection under complex lighting conditions according to claim 4, It is characterized in that The specific process of step 1 is: The same-name points between the projector and camera pixels are obtained through the direct illumination position, and a depth map is generated based on triangulation to complete high-precision three-dimensional measurement.
Citation Information
Patent Citations
Scan error correction in low coherence scanning interferometry
CN102057269A
Micro projector mobile phone platform-based portable three-dimensional scanning system and method
CN102184566A
Multispectral three-dimensional morphology measurement method and device based on grating projection three-dimensional imaging
CN112097688A
Target three-dimensional information measurement method and system
CN115307577A
High-precision structured light time domain phase shift measurement method based on light intensity correction
CN116558444A
Cited By
Fringe projection three-dimensional shape measurement method based on frequency domain and time domain joint coding
CN121026011A
