A method for object error modeling and accurate 3D reconstruction under glare

Abstract

This invention discloses a method for modeling and accurately reconstructing objects under the influence of glare. The method includes: constructing an error model for 3D reconstruction using fringe projection profilometry under the influence of glare, concluding that periodic errors consistent with the projected fringe period in the reconstructed point cloud of non-highlighted areas of a metal object are caused by glare; introducing a two-dimensional glare transfer function to quantitatively represent the additional brightness generated by glare throughout the imaging process; simulating the effect of glare on the entire image by convolving this function with the actual brightness of the original fringe highlight areas; and, based on the characteristics of glare, proposing a method to address phase errors caused by glare, by selecting either a cross-polarization optical path or a single-frame windowed Fourier transform as needed. The analysis results and measurement accuracy were verified in various high-reflectivity scenarios, effectively verifying the accuracy of the established glare model and addressing the fringe errors in the reconstructed object point cloud caused by glare.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical measurement, and in particular relates to a method for error modeling and precise three-dimensional reconstruction of an object under the influence of glare. Background Art

[0002] Fringe projection profilometry, a core technology in non-contact optical 3D measurement, has demonstrated significant advantages in recent years in fields such as industrial inspection, biomedicine, and cultural heritage digitization. This technology projects a coded fringe pattern sequence onto the surface of the object being measured. A camera captures the deformed fringes, modulated by the object's topography, and then reconstructs the submicron 3D topography through phase resolution. Compared to traditional laser triangulation and stereo vision, FPP systems offer significant advantages in measurement speed, point cloud density, and environmental robustness.

[0003] However, when using FPP to measure certain metal workpieces or products, periodic fringe errors, identical to the period of the projected sinusoidal fringes, can appear in the final 3D point cloud. The source of these errors warrants investigation. Due to the high reflectivity of metal objects and the limitations of dynamic range, images can be overexposed, causing problems with 3D measurement in highlight areas. This problem is typically addressed using HDR technology. However, these periodic errors occur not in the highlight areas of the object, but rather in other, less-highlighted areas. This suggests that the errors are not directly related to overexposure, but rather are more likely to be influenced by other factors.

[0004] In reality, this periodic fringe-like error is caused by a global illumination phenomenon, resulting from the transmission of unwanted light within the optical system. This phenomenon, known as flare, can be understood as excessively intense light scattered and reflected within the camera lens and camera body, reaching pixels on the sensor that should not be illuminated. When imaging metallic objects, flare in the imaging system primarily comes from specular highlights reflected from saturated areas of the metal back to the camera. This glare causes noticeable image blur, reduces contrast, and creates bright spots in the image. This effectively adds an additional parasitic image layer to the original image, distorting it. During the phase-shifting fringe projection process, the specular highlights received by the camera vary, resulting in different parasitic images. This parasitic image interferes with the sinusoidal pattern of the FPP method used to capture surface reflections, affecting phase reconstruction and ultimately impacting the accuracy of 3D measurement results. For some metallic objects, glare can be avoided by adjusting the placement angle. However, for some fully curved objects with high reflectivity, specular reflection conditions will inevitably exist regardless of the relative angle between the projector, camera, and object. These locations will inevitably cause glare and phase errors. Therefore, for accurate 3D reconstruction, it is very necessary to eliminate the influence of image glare.

[0005] Currently, commonly used glare reduction methods fall into three main categories: optical enhancement, computational post-processing, and occlusion-based techniques. Optical enhancement methods typically use high-quality anti-glare lens coatings to reduce reflectivity. For example, they connect the CCD to the lens with a liquid to reduce interface reflections, or introduce an electronically controlled shutter array in front of the lens to block strong reflected light from specific directions. While effective, these methods rely on hardware modifications and are less versatile. Among existing computational methods, deconvolution algorithms occupy a central position. They estimate the glare diffusion function by fitting the light around highlight areas and thus restore the image. However, their requirement that the bright areas of the image must not be saturated severely limits their performance. Occlusion-based methods use structured illumination or high-frequency masks to separate direct and indirect illumination during the imaging phase. These methods exploit the separability of high-frequency components in ray space to effectively suppress glare interference. However, these methods are not universally applicable due to the complex optical path design and synchronization control involved.

[0006] While these diverse techniques are proven effective methods for removing image glare, they often require expensive hardware modifications and complex computational models designed for specific imaging systems. However, in 3D imaging systems, not only must the effect of glare on the phase of objects in the image be reduced, but the integrity of the sinusoidal fringes and the accuracy of their grayscale values ​​must also be maintained to correctly interpret the phase. This requires considering more suitable methods that can simultaneously remove glare and achieve accurate 3D reconstruction. Summary of the Invention

[0007] The purpose of the present invention is to provide a method for error modeling and accurate three-dimensional reconstruction of objects under the influence of glare.

[0008] The technical solution for achieving the purpose of the present invention is: a method for error modeling and accurate three-dimensional reconstruction of an object under the influence of glare, comprising the following steps:

[0009] Step 1: A projector projects a set of phase-shifted sinusoidal fringe patterns onto a metal object. The FPP algorithm analyzes the phase changes of the fringe patterns captured by the camera to reconstruct the object's 3D shape and surface information.

[0010] Step 2: Introduce the glare transfer function (GSF), quantitatively express the effect of glare on the entire image as the convolution of GSF and the ideal value of the image without glare, and approximate the scattering veil glare GSF that affects the entire image with a Gaussian function;

[0011] Step 3: Use a convolution formula to express the effect of glare on the intensity of fixed pixels in the image within a phase shift cycle. Establish an error model and deduce that the phase of the object in non-highlight areas and the streak-like error in the reconstructed point cloud are caused by the superposition of the additional intensity in the form of a sinusoidal function caused by glare in the N-step phase shift image and the original intensity of the fringe pattern.

[0012] Step 4: Use the cross-polarization method or single-frame windowed Fourier transform method to reconstruct the 3D point cloud;

[0013] The cross-polarization method involves placing a linear polarizer in front of the projector and camera, adjusting the two polarizers to the darkest viewing angle so that their polarization directions are orthogonal. A set of phase-shifted sinusoidal fringes is then projected, and three-dimensional point cloud reconstruction is achieved through FPP calculation.

[0014] The single-frame windowed Fourier transform method is specifically as follows: a high-frequency sinusoidal fringe pattern is projected onto an object through a projector and captured by a camera; based on the different components of glare and fringe information in the frequency domain image, the windowed Fourier transform method is used to resolve the glare effect and achieve three-dimensional point cloud reconstruction.

[0015] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0016] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0017] A computer program product comprises a computer program, which implements the steps of the above method when executed by a processor.

[0018] Compared with existing technologies, the present invention offers the following advantages: It addresses the periodic error problem of point clouds generated during 3D reconstruction of some metal objects by constructing an error model for 3D reconstruction using fringe projection profilometry (FPP) under the influence of glare. By combining GSF with FPP, the present invention provides a formula that expresses the effect of glare on the phase of the fringe image wrapping. Furthermore, it proposes a method for solving the phase using either the cross-polarization method or the single-frame windowed Fourier transform method, depending on the requirements for detail accuracy or acquisition efficiency, to eliminate the influence of glare and achieve accurate 3D reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the method for error modeling and accurate three-dimensional reconstruction of objects under the influence of glare of the present invention.

[0020] Figure 2 The difference between overexposure and glare of metal objects: A represents the affected area of ​​overexposure, and B represents the affected area of ​​glare;

[0021] Figure 3 The middle left image shows the light path diagram for normal imaging (including overexposure), and the right image shows the light path diagrams for different types of glare under strong light.

[0022] Figure 4 is the curve of GSF changing with distance;

[0023] Figure 5 is the effect of glare on the stripe image;

[0024] Figure 6 is the effect of glare on the intensity of other pixels in the image during the N-step phase shift (expressed as a sine function);

[0025] Figure 7 is the wrapping phase and overall image phase error caused by glare;

[0026] Figure 8 The effect of glare on phase and 3D reconstruction;

[0027] Figure 9 This is the principle diagram of the optical path of the cross-polarization method;

[0028] Figure 10 The experimental setup is as follows: the DLP projector projects the pattern onto the object being measured, and the Balser camera captures the light reflected from the object's surface.

[0029] Figure 11 These are experimental comparisons of the non-highlighted area of ​​the same object with and without glare. (a) is a stainless steel lunch box, and (b) is a house model with a mirror.

[0030] Figure 12 The wrapped phase contrast diagram with and without glare in the experiment and its overall phase error;

[0031] Figure 13 is the impact intensity of glare on other areas in the N-step phase shift in the experiment and the wrapping phase value caused by glare;

[0032] Figure 14 Take a picture of a standard flatbed scene with a reflector;

[0033] Figure 15 Quantitative comparison of the effects of (a) glare on a standard flat panel with a reflector, (b) glare removal using the cross-polarization method, and (c) glare removal using the single-frame windowed Fourier method.

[0034] Figure 16 This is a comparison chart of the effects of different metal objects under the influence of glare, cross-polarization de-glare method, and single-frame windowed Fourier method. DETAILED DESCRIPTION

[0035] This invention proposes a method for modeling object errors and accurately reconstructing 3D objects under the influence of glare. The method constructs an error model for 3D reconstruction using fringe projection profilometry (FPP) under the influence of glare. It deduces that periodic errors in the reconstructed point cloud of non-highlighted areas of metal objects, consistent with the projected fringe period, are caused by glare, an undesirable global illumination effect caused by multiple reflections and scattering of reflected light within the camera body and lens optical system. A two-dimensional glare spread function (GSF) is then introduced to quantify the additional brightness generated by glare during the entire imaging process. The effect of glare on the entire image is simulated by convolving this function with the actual brightness of the original fringe highlight areas. In a fringe image captured with N-step phase shifts, the additional intensity effect of glare on pixels in other areas due to the highlight area is expressed as a sinusoidal function with a roughly consistent phase. The intensity of each step of this function is superimposed on the original sine function intensity modulated by the phase shift, causing periodic fringe-like errors in the FPP phase solution, affecting the 3D result. Finally, based on the characteristics of glare, a method is proposed to address the phase error caused by glare, using either a cross-polarization optical path or a single-frame windowed Fourier transform method, depending on the requirements. This method primarily analyzes the causes of glare and the causes of phase streaking errors. Based on the requirements for detail accuracy or acquisition efficiency, either the cross-polarization method or the single-frame windowed Fourier transform method is used to solve the phase error, thereby removing the phase error caused by glare. The analysis results and measurement accuracy were verified in various high-reflectivity scenarios, effectively verifying the accuracy of the established glare model and addressing the streaking errors in the reconstructed object point cloud caused by glare.

[0036] This paper first explains the causes of glare when photographing certain metal objects, distinguishing between glare and overexposure, and introduces the glare propagation function (GSF) to quantitatively characterize glare in images. Based on this function, we develop an error model to describe how glare causes streak-like errors in 3D reconstruction using the FPP method. Furthermore, we propose two different approaches: establishing a cross-polarization optical path and using a single-frame windowed Fourier transform to correct for periodic streak-like errors in 3D object reconstruction caused by glare, meeting different requirements for reconstruction detail and efficiency.

[0037] Combine Figure 1 The present invention provides a method for error modeling and accurate three-dimensional reconstruction of an object under the influence of glare, comprising the following steps:

[0038] Step 1: A projector projects a set of phase-shifted sinusoidal fringe patterns onto a highly reflective metal object. The FPP algorithm analyzes the phase changes of the fringe patterns captured by the camera to reconstruct the object's 3D shape and surface information.

[0039] The reason why stripe-like errors consistent with the projection stripe period appear in the reconstruction of non-highlight areas of metal objects in highly reflective scenes is clarified. The glare generated by the object's mirror-reflected light passing through the lens reduces the overall image quality by introducing uneven brightness and contrast changes, affecting the accurate extraction of phase information.

[0040] Step 2: Introduce the glare transfer function GSF, and quantitatively express the impact of glare on the entire image as the convolution of GSF and the ideal value of the image without glare. The scattering veil glare GSF that affects the entire image is approximated by a Gaussian function.

[0041] Step 3: Use the convolution formula to express the effect of glare on the intensity of fixed pixels in the image within one phase shift cycle. Establish an error model and deduce that the phase of the object in the non-highlighted area and the stripe-like error in the reconstructed point cloud are caused by the superposition of the additional intensity in the form of a sine function caused by glare in the N-step phase shift image and the original intensity of the fringe pattern.

[0042] Step 4: Use cross-polarization to address glare. Place a linear polarizer in front of the projector and camera, respectively. Adjust the two polarizers to the darkest field of view so that their polarization directions are orthogonal. This filters out most of the light reflected by the mirrors, suppressing glare. Then, project a set of phase-shifted sinusoidal fringes. Through FPP calculations, highly accurate and detailed 3D point cloud reconstruction is achieved.

[0043] This method also employs another approach: projecting a high-frequency sinusoidal fringe pattern onto an object using a projector and capturing it with a camera. Based on the differences in the representation of glare and fringe information in the frequency domain image, a windowed Fourier transform algorithm is used to address the effects of glare, enabling fast and efficient 3D point cloud reconstruction.

[0044] Furthermore, in step 1, the sinusoidal fringe pattern captured by the camera for

[0045]

[0046] in is the background light intensity, is the modulation intensity, n is the image phase shift of the nth step, N is the total number of phase shift steps, is the image phase to be solved.

[0047] Furthermore, glare is an undesirable global illumination effect caused by multiple reflections and scattering of light within the camera body and lens optical system. This phenomenon degrades image quality by introducing uneven brightness and contrast variations, altering the grayscale values ​​of each fringe image and causing periodic errors in the calculated phase and reconstructed point cloud that are consistent with the projected fringe period.

[0048] There's a significant difference between image flare and overexposure. Overexposure occurs when the light intensity received by a local area of ​​the camera exceeds the dynamic range of the sensor, resulting in a loss of detail and "burned-out" highlights, resulting in a pure white or brightly lit image. This makes it impossible to determine the phase of fringes in these areas. Flare, the result of strong light scattering and reflecting within the lens optical system, can lead to an overall decrease in image quality and produce non-uniform additional light in non-bright areas, causing periodic errors in the phase of these areas.

[0049] Furthermore, the intensity of the glare experienced by the pixel in step 2 depends on the quality of the lens and its distance to the glare point pixel. This two-dimensional function describing the intensity is called the glare spread function (GSF) of the lens. In an image, glare can be approximated by using the spatially invariant convolution of the GSF in a traditional camera.

[0050]

[0051] in Indicates projection to coordinates The incident radiation on the pixel, represents GSF. GSF can be simulated by specific effects of scattering and reflection. Usually, the optical parameters of the lens are used to describe the characteristics of GSF. We use Represents the distance from the pixel to the glare point, that is , represents the coordinates of the glare point. GSF can be expressed as

[0052]

[0053] in, is the Dirac function. The first term is given by The second term models the spread of glare from the central point. It is an exponential decay function that reflects the effect of veil glare on the image, which is a generalized scattering phenomenon that occurs when high-intensity direct light passes through the system aperture.

[0054] Veil glare, as a subtype of scattered glare, affects the entire image and is the main factor affecting the fringe phase in FPP. In a typical lens, GSF can be simply expressed by a Gaussian function.

[0055]

[0056] It is a parameter that controls the diffusion degree of scattered glare. As can be seen from the formula, the diffusion degree of glare decreases with the increase of distance.

[0057] is a constant, which is represented by C. For image glare, we can get

[0058]

[0059] in, It is the actual brightness of the highlight area. In the simulation, its grayscale value is generally much larger than 255.

[0060] Furthermore, in step 3, for a pixel at a fixed distance from the glare point, the effect of the glare on it during the N-step phase shift is equivalent to the sinusoidal function of the intensity change multiplied by a fixed parameter, that is,

[0061]

[0062] in, is the actual light intensity of the glare point in the N-step phase shift, is the phase of the glare point, For N-step phase shift The additional light intensity caused by the glare point, It is the constant obtained by multiplying C by the glare point modulation degree.

[0063] For such a pixel, the additional light intensity caused by the glare is equal to the glare point The value of is closely related to . When the distance 𝑟 between the two is fixed, the intensity of the glare point within one phase cycle is proportional to the sinusoidal change of the glare point's own intensity. The intensity of this additional influence also changes in the form of a sine function within N phase shifts.

[0064] In the N-step phase shift of the FPP method, the n-step phase shift is used as the independent variable. For different glare points, since the number of phase shift steps is the same, the angular frequency of the sine function of the change is for For a fixed pixel, the final impact value it receives is the sum of the impacts of all glare points on it. The two sine functions have amplitudes P and Q and phases P and Q. and , the addition formula is

[0065]

[0066] In the N-step phase shift, the influence of all glare points in the area on the intensity change of fixed pixels is equivalent to an angular frequency A fixed sine function, where Since the glare affects all pixels in the image, each pixel will receive an additional intensity value that is expressed as a sinusoidal function in an N-step phase shift.

[0067] In one phase cycle, the effect of each glare point on the pixel is The distances between two adjacent glare points and a pixel are and , the phase effect on a certain pixel is

[0068]

[0069] During the N-step phase shift process, The additional phase in the sine function representing the impact of all glare points in the glare area on a given pixel is calculated sequentially using the above formula. The scattered glare affects the entire image, so the value of 𝜎 is very large. For pixels in non-highlight areas, the distance from the glare area affects the additional phase value. Therefore, the difference in the overall image phase value is For FPP, the method of solving the wrapped phase is based on the sine function of the pixel intensity value obtained during the N-step phase shift. For each pixel in the image, the effect of glare is equivalent to adding a sine function to the original sine function representing its intensity value. is a fixed-valued sine function added to the pixel. Therefore, the phase of each pixel deviates from the true value. Compared to the true phase, this phase error is consistent with the period of the fringes. This also explains why the error in the reconstructed point cloud exhibits a similar waveform.

[0070] Furthermore, the cross-polarization method is adopted in the step 4. During the FPP imaging process, a linear polarizer is placed in front of the projector and the camera to form a polarization pair. The polarization directions of the two polarizers are orthogonal by adjusting the two polarizers to make the camera field of view darkest. When the metal surface maintains specular reflection, most of the light can better maintain the linear polarization state of the incident light, and then be effectively filtered out by the analyzer. However, due to reasons such as surface roughness and scattering, a small part of the light will be depolarized. This method filters out most of the specular reflected light, leaving only a small amount of diffuse reflection retained due to depolarization and some specular reflection residues that are not completely eliminated, thereby suppressing the generation of glare, ensuring correct phase extraction, and achieving low efficiency but very accurate three-dimensional reconstruction.

[0071] Furthermore, glare typically appears as low-frequency components in a two-dimensional image, while the phase information in a fringe image is primarily concentrated in the high-frequency region. A camera captures a single frame of high-frequency fringe patterns projected onto an object by a projector. Using a single-frame windowed Fourier transform (WFT), the image can be decomposed into its frequency components. Spectral analysis using a local window extracts high-frequency fringe information while effectively filtering out low-frequency glare, achieving fast and efficient 3D reconstruction with slightly blurred details.

[0072] The specific implementation of the present invention will be described below in conjunction with the accompanying drawings and embodiments to more clearly and completely illustrate the technical solutions of the present invention.

[0073] Example

[0074] Combine Figure 1 A method for error modeling and accurate three-dimensional reconstruction of an object under glare includes the following steps:

[0075] A. Generation and classification of glare

[0076] Glare is an undesirable global illumination effect caused by multiple reflections and scattering of strong light within the camera body and lens optical system. This phenomenon degrades image quality by introducing uneven brightness and contrast variations, altering the grayscale values ​​of each fringe image and causing periodic errors in the desired phase and reconstructed point cloud that are consistent with the projected fringe period.

[0077] Although both are caused by reflected light from the bright areas of metal objects, there is a significant difference between image glare and image overexposure. Figure 2 As shown in the figure, A represents overexposed areas of the image, while B represents non-overexposed areas, also known as areas affected by flare. Overexposure occurs when the light intensity received by a local area of ​​the camera exceeds the dynamic range of the sensor, resulting in a loss of detail and "burned-out" highlights, resulting in a pure white or brightly lit appearance. This makes it impossible to determine the phase of fringes in the bright areas. Flare, the result of strong light scattered and reflected within the lens optical system, leads to an overall degradation of image quality and produces non-uniform additional light in non-highlight areas, causing periodic errors in the phase of the non-highlight areas. In short, overexposure affects the overexposed areas of the image, while flare affects the entire image.

[0078] For cameras, the lens flare that affects the image is mainly the result of scattered glare, reflected glare and body free glare. Scattered glare is produced by the diffusion effect of the lens surface, which is a scattering phenomenon that occurs when high-intensity direct light passes through the system aperture. Reflected glare is caused by a series of complex reflections of a strong light source on the lens surface. It appears in the form of parasitic images and appears as ghosts and flares. Body free glare is formed by the scattering of light from the last lens element and appears outside the light spot. It can be eliminated by using a smaller aperture. Therefore, it can be ignored in practice. Figure 3 It can be seen that the imaging light paths of these different types of glare under the influence of strong light.

[0079] The additional intensity of a pixel affected by glare is crucial in image processing and depends primarily on the quality of the lens and its distance to the pixel where the glare occurs. This two-dimensional function describing the intensity is called the glare spread function (GSF) of the lens. It can be approximated by using the spatially invariant convolution of the GSF in conventional cameras

[0080]

[0081] in Indicates projection to coordinates The incident radiation on the pixel, that is, the true brightness of the ideal image without glare, Denotes GSF. In order to improve computational efficiency, convolution operations are usually performed in the Fourier domain. It can be expressed as

[0082]

[0083] in, and is an expression and The Fourier transform form of is the frequency domain coordinate, that is, the frequency variable after two-dimensional Fourier transform.

[0084] In an image with significant glare, even a bright pixel in one corner will affect the pixels in the farthest corner. Therefore, in the modeling process, in order to more accurately simulate glare, the size of the convolution kernel needs to be twice that of the image.

[0085] In practical applications, GSF can be simulated by specific effects of scattering and reflection. Usually, the optical parameters of the lens are used to describe the characteristics of GSF. We use Represents the distance from the pixel to the glare point, that is , Represents the coordinates of the glare point, and then we can use the following formula to express GSF

[0086]

[0087] in is the Dirac function, and the first term is given by Modulated instantaneous glare point, represents the modulation. The second term is a model of the glare spreading outward from the center point. It is an exponential decay function, where is the scaling strength, It affects the attenuation rate. Based on distance Adjusting the nonlinear response reflects the effect of veil glare on the image, which is a generalized scattering phenomenon that occurs when high-intensity direct light passes through the system aperture. Figure 4 It shows the curve of GSF changing with distance.

[0088] Veil glare, as a subtype of scattered glare, affects the entire image and is the main factor affecting the fringe phase in FPP. In a typical lens, GSF can be simply expressed by a Gaussian function.

[0089]

[0090] in, are the coordinates of the glare point, are the coordinates of other points affected by glare, It is a parameter that controls the degree of glare diffusion and represents the scattering intensity. As can be seen from the formula, the degree of glare diffusion decreases with increasing distance.

[0091] is a constant, replace it with C. Therefore, for image glare, we can get

[0092]

[0093] in, It is the actual light intensity of the highlight area. In the simulation, its grayscale value is generally much larger than 255. Figure 5 It shows the effect of glare on the stripe image.

[0094] B. Error model of glare's effect on phase in FPP

[0095] When using the FPP algorithm, the sinusoidal fringe pattern captured by the camera is

[0096]

[0097] in is the background light intensity, is the modulation intensity, is the image phase to be solved.

[0098] In the FPP process, the intensity of the original value of the glare point during the N-step phase shift process shows a sinusoidal function change. Its initial value is greater than 255. Therefore, for the pixel point at a fixed distance from the glare point, the effect of the glare during the N-step phase shift is equivalent to the above-mentioned sinusoidal function of intensity change multiplied by a fixed parameter, that is,

[0099]

[0100] in, are the coordinates of the glare point, is the actual light intensity of the glare point in the N-step phase shift, is the phase of the glare point, For N-step phase shift The additional light intensity caused by the glare point, It is the constant obtained by multiplying C by the glare point modulation degree.

[0101] For such a pixel, the additional light intensity caused by the glare is equal to the glare point The value of is closely related. When the distance between the two When fixed, the intensity of the glare point within a phase cycle is proportional to the sinusoidal change of the glare point's own intensity. Therefore, the intensity of this additional influence also changes in the form of a sine function within the N-step phase shift. This conclusion is drawn under the assumption that there is only one glare point. In fact, the actual impact of the glare area on other pixels is the sum of all glare points in the area. In the N-step phase shift of the FPP method, the n-step phase shift is used as the independent variable. For different glare points, since the number of phase shift steps is the same, the angular frequency of the changing sine function is for For a fixed pixel, the final impact value it receives is the sum of the impacts of all glare points on it. The two sine functions have amplitudes P and Q and phases P and Q. and The formula for their addition is

[0102]

[0103] Therefore, in the N-step phase shift, the influence of all glare points in the area on the intensity change of fixed pixels is equivalent to an angular frequency A fixed sine function, where This additional sine function affects the intensity of the original sine function due to the phase shift of the fringes. Therefore, at each phase shift, the measured actual light intensity deviates from the true value. Since the glare affects all pixels in the image, each pixel will be affected by an additional intensity value, which is expressed as a sine function in the N-step phase shift. Figure 6 It shows the effect of glare on the intensity of other pixels in the image during the N-step phase shift process (expressed in the form of a sine function).

[0104] In one phase cycle, the effect of each glare point on the pixel is The distances between two adjacent glare points and a pixel are and , its influence on the phase of a pixel is

[0105]

[0106] During the N-step phase shift process, The additional phase in the sine function representing the impact of all glare points in the glare area on a given pixel is calculated sequentially from the above formula. Since the impact covers the entire image, the value of 𝜎 is very large. Therefore, for a specific pixel, its distance from the glare area has an impact on the additional phase value. This results in a difference in the overall additional phase value of the image. Very small, almost negligible. For FPP, the method of solving the wrapped phase is based on the sine function of the pixel intensity value obtained during the N-step phase shift. For each pixel in the image, the effect of glare is equivalent to adding a sine function to the original sine function representing its intensity value. is an additional sine function with a fixed value, the specific effect is Figure 7 This is reflected in the . Therefore, the phase of each pixel deviates from the true value. Compared with the true phase, this phase error is consistent with the period of the fringes. This also explains why the error in the reconstructed point cloud exhibits a similar waveform. Figure 8 The effects of glare on phase and 3D reconstruction are shown in detail.

[0107] Therefore, for each pixel whose phase is solved, the effect of glare is equivalent to adding a new N-step phase shift on the basis of the original N-step phase shift. Therefore, for the pixel affected by glare, the formula is

[0108]

[0109] in is the total intensity of the Nth-step phase shift map under the influence of glare, is the ideal phase, is the additional phase shift caused by glare, is the function modulation degree that represents the glare effect. Solve the phase using the least square method and express the original equation as

[0110] ;The phase difference of each phase shift By performing at least three phase shifts, it is possible to determine and The values ​​of the previous parameters are calculated. Finally, the arctan function is used to determine the phase value affected by glare. The formula shows that in each phase shift image, the true fringe pattern and the additional light caused by glare are mixed together, making them indistinguishable and difficult to compensate for at the formula level. Therefore, it is necessary to use other methods to eliminate errors and reconstruct a correct 3D point cloud model.

[0111] C. Cross-polarization method for phase deglare

[0112] During the FPP imaging process, a linear polarizer is placed in front of the projector and camera to form a polarization pair. A linear polarizer is an optical element that precisely controls the polarization state of light waves. It selectively allows light waves whose oscillation direction is aligned with its transmission axis to pass through, while blocking light waves oscillating in other directions, thereby achieving the purpose of adjusting and filtering the polarization characteristics of light; the optical path principle of the cross-polarization method is as follows Figure 9 shown.

[0113] The light emitted by the projection light source is modulated by the polarizer to form linearly polarized light with a specific polarization direction, ensuring that only light with the desired polarization direction is transmitted. On this basis, by precisely controlling the polarization direction of the projected light, the intensity of the reflected light can be significantly reduced. Most of the specular reflected light still maintains its original polarization state after reflecting on the metal surface, and is then effectively filtered out by the analyzer. After passing through the analyzer, the relationship between light intensity and polarization angle can be expressed by Malus's law.

[0114]

[0115] in, is the intensity of the incident light before entering the analyzer, is the intensity of the light after passing through the analyzer, is the angle between the polarization direction of the incident light and the axis of the analyzer. For orthogonal polarizers, Ideally, the specular reflection component should be completely blocked. However, since the object being photographed is a metal object, when the polarized light is incident on the metal surface, its reflection characteristics follow the Fresnel equation.

[0116] ;

[0117] ;

[0118] in, is the reflection coefficient of s-polarized light (perpendicular to the incident plane), is the reflection coefficient of p-polarized light (parallel to the incident plane), is the complex refractive index of the metal, which reflects the optical properties of the metal material. is the angle of incidence. ,in is the real part of the refractive index, is the extinction coefficient, which characterizes the light absorption characteristics of the material. The value causes light to change not only in amplitude but also in phase when it is reflected. To convert linear polarized light into elliptically polarized light, the orthogonal polarizer can remove most of the specular reflected light, but some components of the elliptically polarized light can still pass through, forming residual light intensity. The angle of incidence affects the Fresnel reflection coefficient. 、 The value of , thus changing the amplitude and phase difference. When the incident light is normal, and Therefore, while ensuring sufficient machine limits, we must minimize the distance between the projector and the camera, reduce the incident angle, and thus reduce the residual light intensity caused by the phase difference.

[0119] In addition, in the actual imaging process, due to reasons such as surface roughness, a small part of the light will be depolarized (non-polarized or partially randomized), and the depolarization effect caused by the surface micro-roughness will produce a diffuse reflection component.

[0120]

[0121] in, represents the intensity of the incident light, Indicates the intensity of the diffuse reflection component caused by depolarization, depolarization Quantitatively describe the degree of polarization state randomization, This part of the diffuse reflection is very important, as it allows the camera to clearly capture metal objects under cross-polarization while increasing the exposure.

[0122] In addition, there are some residual specular reflections caused by the quality of the polarizer itself and the error angle of the installation. To avoid such errors, the experiment selected linear polarizers with higher optical quality and larger extinction ratios and installed them on a rotation stage with a graduation accuracy of <0.1°.

[0123] In summary, by placing polarizing films in front of the projector and the camera respectively and adjusting them to be orthogonal to each other, most of the specular reflected light on the metal surface that maintains its polarization state is effectively blocked, leaving only a small amount of residual specular reflected light caused by phase delay, polarizing film quality and installation error angle, and diffuse reflection caused by depolarization. This significantly reduces the impact of glare and ensures the correct phase for 3D reconstruction.

[0124] D. Single frame windowed Fourier transform phase deglare

[0125] Using the WFT method for phase retrieval can effectively eliminate the effects of glare, enabling fast and efficient 3D reconstruction. This is because glare typically appears as a low-frequency component in a 2D image, while the phase information in a fringe image is primarily concentrated in the high-frequency region. Using the WFT, the image can be decomposed into distinct frequency components. A local window is used to analyze the spectrum to extract high-frequency fringe information while effectively filtering out low-frequency glare. Therefore, the final calculated phase information reflects only the fringe and is unaffected by glare. This ensures that the calculated phase primarily represents the actual variations in the fringe, significantly reducing the interference of glare on phase measurement and improving the accuracy of phase unwrapping.

[0126] First, the camera collects a single-frame fringe image projected by the projector and performs a windowed Fourier transform on it.

[0127] ;

[0128] ;

[0129] ;

[0130] For stripe images The spectrum after windowed Fourier transform, is the window function Frequency domain form, window function is a Gaussian function, and Represents the spatial offset in the x-axis and y-axis directions respectively, and is the standard deviation of the Gaussian function in the x and y directions, which controls the extent of the window function in the spatial domain. and is the frequency of the window function in the x and y directions.

[0131] Next, in order to effectively separate the frequencies of glare and stripes, the maximum value of the spectrum information amplitude obtained by multiplying the stripe image spectrum with the window functions of different frequencies is selected as the spectrum output result, that is, the most significant frequency components corresponding to different window functions are found in the spectrum domain.

[0132]

[0133] in, Represents the image at local points The frequency components along the horizontal and vertical directions, Indicates when The frequency corresponding to the maximum value and frequency ,at this time The value of , The value of Finally, the output spectrum value is subjected to a windowed inverse Fourier transform to restore the amplitude and phase information of the object to be measured, while effectively filtering out the frequencies affected by glare.

[0134]

[0135]

[0136]

[0137] in, is the restored fringe image after windowed Fourier transform, is the amplitude of the object to be measured, is the wrapped phase of the object to be measured. Based on this, phase unwrapping is performed to perform 3D reconstruction of the object.

[0138] Using a projector to project higher-frequency fringes can better eliminate the effects of glare on phase, as this facilitates the separation of high- and low-frequency components in the spectrum. When using a windowed Fourier transform for fringe phase extraction and 3D reconstruction, due to the local limitations of the window function, there is a trade-off between frequency resolution and spatial resolution. This results in insufficient capture of spatial frequency information in fine details, leading to a lack of clarity in the local structural details of the reconstructed object. Larger windows improve frequency resolution and phase extraction stability while better filtering out glare, but they also smooth out subtle variations in the image, reducing spatial detail. Smaller windows, while preserving more local detail, can lead to spectral leakage and phase instability. Combined with the influence of noise in actual measurements, this ultimately blurs local details in the reconstructed 3D object. Therefore, this method is not suitable when high reconstruction accuracy is required. However, a significant advantage of this method is that only a single fringe image is required to determine the phase, significantly improving measurement efficiency and making it more suitable for dynamic measurements.

[0139] E. Experiment

[0140] The 3D measurement system mainly consists of a DLP6500 projector with a resolution of 1920×1080, a Basler acA1920-40um CMOS camera with a resolution of 1920×1200, and a camera lens with a focal length of 35 mm. The object to be measured should be placed about 0.5 m away from the measurement system. For FPP, the fringe selection , , generating 12 phase shift patterns, each measurement takes 2 seconds. The experimental configuration is as follows Figure 10 shown.

[0141] First, we verified the conditions and form of glare's effect on non-highlight areas in the above formula. We projected a fringe pattern onto the same scene with highlight areas where glare occurs. The objects being measured were a stainless steel lunch box and a doll with a mirror. The difference was that the first projection covered the entire scene, while the second projection excluded the highlight areas that would otherwise cause glare. This comparative experiment eliminated a number of influencing factors, such as the material properties of the object, and allowed for a more intuitive investigation of the source of the periodic fringe-like error.

[0142] Figure 11 As can be seen, when the projection image is avoided on the highlighted areas of the object that would cause glare, the reconstruction of the remaining areas is error-free. However, when the projection image is projected onto the entire area, the glare caused by the specular reflection of the object's highlighted areas affects the grayscale value of each fringe pattern, further affecting the phase of the non-glare areas and causing fringe-like errors. This proves that the periodic errors in the phase and point cloud of non-highlight areas are caused by glare caused by strong light reflection from the object's highlighted areas.

[0143] To further verify this, we explored the effect of glare on the phase of a standard flat panel. Figure 12 The wrapped phase of the standard plate in the experiments with and without glare interference is shown in Figure 2. The phase difference between the two appears as stripes with a period that matches the projected stripes, consistent with theoretical derivation.

[0144] Only the stripes are projected onto the mirror, and the projection light intensity of the surrounding background is ensured to be zero. Due to the glare of the mirror highlight area during each phase shift projection, the background area captured by the camera still has a certain grayscale value. Solving its wrapped phase, the final result is consistent with the principle. Figure 13 In the image, the extra phase wrapping caused by glare has very small differences in the whole image. This extra phase leads to periodic streak-like errors in the final reconstructed 3D result.

[0145] In order to solve the influence of glare on the 3D reconstruction of objects, the cross-polarization method is used to effectively suppress glare for high-precision and detailed 3D reconstruction tasks; in scenarios where reconstruction speed and efficiency are required to be high, the single-frame windowed Fourier method is used to quickly remove glare and then perform 3D reconstruction. Figure 14 The following is a scene image. The reflector is the bright area that generates glare, and the light reflected from its mirror surface is directed into the lens. The effects of glare were tested and the effectiveness of the two methods in eliminating glare was verified. The experiment measured the 3D reconstruction results of a flat panel affected by glare and compared them with the results of the aforementioned methods. The plane fitting error of the measured flat panel was used to evaluate the measurement accuracy.

[0146] Figure 15As can be seen, the standard flat plate fitting error under the influence of glare is very large, with periodic fringe errors being very noticeable, and the standard error of the plane fitting is 0.25mm. However, both the cross-polarization method and the single-frame Fourier transform method effectively remove the effects of glare on the phase and 3D point cloud, with standard errors within 0.03mm, effectively eliminating the effects of glare. The single-frame windowed Fourier transform method produces a smoother flat plate surface, which is due to the characteristics of the algorithm.

[0147] Figure 16 Measurements were conducted using aluminum plates, stainless steel lunch boxes, steel rulers, and CDs as examples. These objects all have brightly lit areas and generate glare when projected. The targets were measured using traditional FPP and compared with the cross-polarization method and the single-frame windowed Fourier transform method. The experimental results show that both methods effectively address the effects of glare on metal objects, eliminating phase and periodic errors in the point cloud, including fringes. However, for the relatively smooth and flawless aluminum plates and stainless steel lunch boxes, both the cross-polarization and windowed Fourier transform methods achieve good reconstruction results. However, for the heavily scaled steel ruler and heavily scratched CD, the cross-polarization method still provides good reconstruction results, while the windowed Fourier transform method yields poorer results, with significant errors at the scale and scratches on the CD. This is due to the effects of the single-frame windowed Fourier transform algorithm. When filtering low-frequency glare components, using a larger window function is generally advantageous because it provides higher frequency domain resolution, allowing for more precise separation and suppression of low-frequency glare components. However, a larger window reduces spatial resolution, blurring local details and making it unsuitable for 3D reconstruction of objects with high detail and extremely high precision requirements. However, single-frame windowed Fourier transform (FFFT) offers the advantages of speed and efficiency, requiring only one frame to be acquired at an average frame rate of 200 frames per second, meaning an image can be acquired in approximately 0.05 seconds. However, the cross-polarization method, because it filters out most of the highly reflected light from specular surfaces, results in an extremely dark camera field of view. The average exposure time during acquisition must be extended to 3 million microseconds to produce a clear image. Capturing a complete FPP process takes approximately 2 minutes to fully acquire an image, resulting in very low acquisition efficiency. Therefore, the method for glare removal and accurate 3D reconstruction should be selected based on specific requirements. Experiments have shown that the cross-polarization method is suitable for glare removal when high-precision and detailed reconstruction is required, while the single-frame windowed Fourier transform (FFFT) method is preferred for glare removal when fast and efficient reconstruction is required.

[0148] The above specific embodiments merely describe preferred embodiments of the present invention and do not limit the scope of protection of the present invention. Without departing from the design concept and spirit of the present invention, various modifications, substitutions, and improvements made by those skilled in the art to the technical solution of the present invention based on the text description and drawings provided herein shall fall within the scope of protection of the present invention.

Claims

1. A method for error modeling and accurate 3D reconstruction of objects under glare, characterized in that: The following steps are involved: Step 1: A projector projects a set of phase-shifted sinusoidal fringe patterns onto a metal object. The FPP algorithm analyzes the phase changes of the fringe patterns captured by the camera to reconstruct the object's 3D shape and surface information. Step 2: Introduce the glare transfer function (GSF), quantitatively express the effect of glare on the entire image as the convolution of GSF and the ideal value of the image without glare, and approximate the scattering veil glare GSF that affects the entire image with a Gaussian function; The intensity of glare experienced by a pixel depends on the quality of the lens and its distance to the pixel where the glare occurs. The two-dimensional function describing the intensity is called the glare spread function (GSF) of the lens. In images, glare is approximated by using the spatially invariant convolution of the GSF in conventional cameras: ； in Indicates projection to coordinates The incident radiation on the pixel, Represents GSF; GSF is simulated by the effects of scattering and reflection; the optical parameters of the lens are used to describe the characteristics of GSF; Represents the distance from the pixel to the glare point, that is , represents the coordinates of the glare point; GSF is expressed as ； in, is the Dirac function; the first term is given by The modulated instantaneous glare point; the second term is a model of glare spreading outward from the central point, where is the scaling strength, It affects the attenuation rate. Based on distance Adjust nonlinear responses; In the lens, the GSF is represented by a Gaussian function: ； It is a parameter that controls the diffusion degree of scattered glare. As can be seen from the formula, the diffusion degree of glare decreases with the increase of distance. is a constant, which is represented by C; for image glare, we can get ； in, is the actual brightness of the highlight area; Step 3: Use a convolution formula to express the effect of glare on the intensity of fixed pixels in the image within a phase shift cycle. Establish an error model and deduce that the phase of the object in non-highlight areas and the streak-like error in the reconstructed point cloud are caused by the superposition of the additional intensity in the form of a sinusoidal function caused by glare in the N-step phase shift image and the original intensity of the fringe pattern. Step 4: Use the cross-polarization method or single-frame windowed Fourier transform method to reconstruct the 3D point cloud; The cross-polarization method involves placing a linear polarizer in front of the projector and camera, adjusting the two linear polarizers to the darkest field of view so that their polarization directions are orthogonal. A set of phase-shifted sinusoidal fringes is then projected, and three-dimensional point cloud reconstruction is achieved through FPP calculation. The single-frame windowed Fourier transform method is specifically as follows: a high-frequency sinusoidal fringe pattern is projected onto an object through a projector and captured by a camera; based on the different components of glare and fringe information in the frequency domain image, the windowed Fourier transform method is used to resolve the glare effect and achieve three-dimensional point cloud reconstruction.

2. The method for error modeling and accurate 3D reconstruction of objects under glare according to claim 1, characterized in that: In step 1, the camera captures the sinusoidal fringe pattern for: ； in is the background light intensity, is the modulation intensity, n is the image phase shift of the nth step, N is the total number of phase shift steps, is the image phase to be solved.

3. The method for error modeling and accurate 3D reconstruction of objects under glare according to claim 1, characterized in that: In step 3, for a pixel at a fixed distance from the glare point, the effect of glare on it during the N-step phase shift is equivalent to the sine function of intensity change multiplied by a fixed parameter, that is, ； in, is the actual light intensity of the glare point in the N-step phase shift, is the phase of the glare point, For N-step phase shift The additional light intensity caused by the glare point, is the constant obtained by multiplying C by the glare point modulation degree; In the N-step phase shift of the FPP method, the n-step phase shift is used as the independent variable; for different glare points, since the number of phase shift steps is the same, the angular frequency of the changing sine function is for ; For a fixed pixel, the final impact value it receives is the sum of the impacts of all glare points on it; It is known that for two sine functions with the same angular frequency, their amplitudes are P and Q respectively, and their phases are and , the addition formula is ； In the N-step phase shift, the influence of all glare points in the area on the intensity change of fixed pixels is equivalent to an angular frequency A fixed sine function, where ; In one phase cycle, the effect of each glare point on the pixel is The distances from two adjacent glare points to a pixel are and , the phase effect on a certain pixel is ； During the N-step phase shift process, The additional phase in the sine function that represents the impact of all glare points in the glare area on a given pixel.

4. The method for error modeling and accurate 3D reconstruction of objects under glare according to claim 3, characterized in that: The specific method of single-frame windowed Fourier transform is: The camera collects the single-frame fringe image projected by the projector and performs a windowed Fourier transform on it. ； ； ； For stripe images The spectrum after windowed Fourier transform, is the window function Frequency domain form, window function is a Gaussian function, and is the standard deviation of the Gaussian function in the x and y directions, which controls the expansion of the window function in the spatial domain; and is the frequency of the window function in the x and y directions; The maximum value of the spectrum information amplitude obtained by multiplying the fringe image spectrum with the window function of different frequencies is selected as the spectrum output result, and the most significant frequency components corresponding to different window functions are found in the spectrum domain. ； in, Represents the image at local points The frequency components along the horizontal and vertical directions, Indicates when The frequency corresponding to the maximum value and frequency ,at this time The value of , The value of ; Performing a windowed inverse Fourier transform on the output spectrum value can restore the amplitude and phase information of the object to be measured, while filtering out the frequencies affected by glare. ； ； ； in, is the amplitude of the object to be measured, is the wrapped phase of the object to be measured; finally, phase unwrapping is performed to complete the three-dimensional reconstruction of the object.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images