Object error modeling and accurate three-dimensional reconstruction method under glare influence

Abstract

The invention discloses an object error modeling and accurate three-dimensional reconstruction method under the influence of glare, and the method comprises the steps: building an error model during the three-dimensional reconstruction of a fringe projection profilometry under the influence of glare, and obtaining that a periodic error, consistent with a projection fringe period, of a metal object at a reconstruction point cloud of a non-highlight region is caused by glare; a two-dimensional function glare transfer function is introduced, and extra brightness generated by glare in the whole imaging process is quantitatively expressed; the influence of glare on the whole image is simulated through the convolution of the function and the actual brightness of the highlight area of the original stripe; aiming at the characteristics of glare, the invention provides a method for solving the phase error caused by the glare by selecting and building a cross polarized light path or a single-frame windowing Fourier method according to requirements. The analysis result and the measurement precision are verified in different high-reflectivity scenes, the accuracy of the established glare model is effectively verified, and the stripe error of the reconstructed object point cloud caused by glare is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical measurement, and particularly relates to a method for object error modeling and precise three-dimensional reconstruction under the influence of glare. Background Art

[0002] As a core technology in the field of non-contact optical three-dimensional measurement, fringe projection profilometry has shown significant advantages in recent years in fields such as industrial inspection, biomedicine, and digitalization of cultural heritage. This technology projects a sequence of encoded fringe patterns onto the surface of the object to be measured, captures the deformed fringes modulated by the object morphology using a camera, and then realizes sub-micron three-dimensional morphology reconstruction through phase analysis. Compared with traditional laser triangulation and stereo vision, the FPP system has significant advantages in terms of measurement speed, point cloud density, and environmental anti-interference.

[0003] However, when using FPP to measure certain metal workpieces or products, periodic stripe-like errors that are the same as the period of the projected sine fringes will appear in the final 3D point cloud results, and their origin is worth exploring. Due to the high reflectivity of metal objects and the limitation of the dynamic range, overexposure of the image will occur, resulting in problems in 3D measurement of the high-brightness area. This problem is usually solved using HDR technology. However, the above-mentioned periodic errors do not appear in the high-brightness area of the object, but in other non-high-brightness areas of the object. This shows that the error has no direct relationship with overexposure and is more affected by other factors.

[0004] In fact, this periodic stripe-like error is caused by a global illumination phenomenon, which is generated due to the transmission of unwanted light in the optical system. This phenomenon is called glare, which can be understood as too strong light reaching pixels on the sensor that should not be illuminated through scattering and reflection inside the camera lens and body. When photographing metal objects, the glare in the imaging system mainly comes from the specular reflection highlights reflected from the metal saturation area to the camera. Glare will cause obvious image blurring, reduce contrast, and form light spots in the image. This is equivalent to adding an additional parasitic image layer to the original image, thus distorting the original image. During the process of projecting phase-shifted fringes, the specular highlights received by the camera are different, and the parasitic images formed by glare are also different. This parasitic image will interfere with the sine pattern of the FPP method used to capture the reflection of the object surface, thereby affecting phase reconstruction and ultimately the accuracy of the 3D measurement result. For some metal objects, glare can be avoided by adjusting the placement angle, but for some fully curved objects with relatively high reflectivity, no matter what the relative angle between the projector, camera, and object is, there will inevitably be positions that meet the specular reflection conditions, and glare will occur at these positions, causing phase errors. Therefore, for precise three-dimensional reconstruction, it is very necessary to eliminate the influence of image glare.

[0005] The currently commonly used anti-glare methods mainly include three categories: optical enhancement, computing and post-processing, and occlusion-based techniques. Optical enhancement methods usually use high-quality anti-glare lens coatings to reduce the reflectivity. For example, a liquid is used to connect the CCD and the lens to reduce interface reflection, or an electronically controlled shutter array is introduced in front of the lens to block strong reflected light in a specific direction. Although effective, they rely on hardware modification and have poor generality. Among the existing computing methods, the deconvolution algorithm occupies a core position. It can estimate the glare diffusion function by fitting the light around the highlighted area and then restore the image. However, its requirement that the bright part of the image cannot be saturated severely limits its performance. Occlusion-based methods separate direct radiation and indirect illumination during the imaging stage through structured illumination or high-frequency masks, and utilize the separability of high-frequency components in the ray space to effectively suppress glare interference. However, they are not generally applicable due to the complex optical path design and synchronization control involved.

[0006] These different categories of techniques are all proven effective methods for removing image glare. They usually require expensive hardware modifications and complex computational models designed for specific imaging systems. However, in a three-dimensional imaging system, not only is it necessary to reduce the impact of glare on the phase of objects in the image, but it is also necessary to ensure the integrity of the sine stripes and the accuracy of the gray values to correctly solve for the phase. This requires us to consider a more suitable method that can achieve accurate three-dimensional reconstruction while removing glare. Summary of the Invention

[0007] The object of the present invention is to provide a method for object error modeling and precise three-dimensional reconstruction under the influence of glare.

[0008] The technical solution for achieving the object of the present invention is: A method for object error modeling and precise three-dimensional reconstruction under the influence of glare, comprising the following steps:

[0009] Step 1: Project a set of phase-shifted sine stripe patterns onto a metal object through a projector; analyze the phase change of the stripes captured by the camera through the FPP algorithm to reconstruct the 3D shape and surface information of the object;

[0010] Step 2: Introduce the glare transfer function GSF, quantitatively represent the influence of glare on the entire image as the convolution of GSF and the ideal value when the image has no glare, and approximate the scattered veil glare GSF affecting the entire image with a Gaussian function;

[0011] Step 3: Represent the intensity influence of glare on a fixed pixel point in the image within a phase shift period through the convolution formula, establish an error model, and deduce that the phase of the object in the non-highlight area and the striped error of the reconstructed point cloud are due to the superposition of the additional intensity in the form of a sine function caused by glare in the N-step phase shift image and the original intensity of the stripe pattern;

[0012] Step 4: Implement 3D point cloud reconstruction using the cross-polarization method or the single-frame windowed Fourier transform method;

[0013] Among them, the cross-polarization method is specifically as follows: Place a linear polarizer in front of the projector and the camera respectively, adjust the two polarizers to make the field of view darkest so that their polarization directions are orthogonal, and then project a set of phase-shifted sine fringes. Through the calculation of FPP, 3D point cloud reconstruction is achieved;

[0014] The single-frame windowed Fourier transform method is specifically as follows: Project a high-frequency sine fringe pattern onto the object through the projector and capture it through the camera; According to the different components of glare and fringe information in the frequency-domain image, solve the influence of glare through the windowed Fourier method to achieve 3D point cloud reconstruction.

[0015] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the above method are implemented.

[0016] A computer-readable storage medium stores a computer program, and when the program is executed by a processor, the steps of the above method are implemented.

[0017] A computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: Aiming at the periodic error problem of point clouds generated during the 3D reconstruction of some metal objects, the present invention constructs an error model for 3D reconstruction using fringe projection profilometry (FPP) under the influence of glare, combines GSF and FPP, and shows the influence of glare on the wrapped phase of the fringe image through a formula. And aiming at this error, it is proposed to select the cross-polarization method or the single-frame windowed Fourier method to solve the phase according to the requirements of detail accuracy or acquisition efficiency, remove the influence of glare, and achieve accurate 3D reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of the method for object error modeling and accurate 3D reconstruction under the influence of glare of the present invention.

[0020] Figure 2 It is the difference between overexposure and glare of a metal object: A represents the affected area of overexposure, and B represents the affected area of glare;

[0021] Figure 3 The left figure in is the optical path diagram of normal imaging (including overexposure), and the right figure is the optical path diagram of the formation of different types of glare under strong light irradiation;

[0022] Figure 4 It is the change curve of GSF with distance;

[0023] Figure 5 is the influence effect of glare on the fringe image;

[0024] Figure 6 is the influence of glare on the intensity of other pixel points of the image during the N-step phase shift process (presented in the form of a sine function);

[0025] Figure 7 is the wrapped phase and the overall phase error of the image caused by glare;

[0026] Figure 8 is the influence of glare on phase and three-dimensional reconstruction;

[0027] Figure 9 is the optical path schematic diagram of the cross-polarization method;

[0028] Figure 10 is the experimental setup; the DLP projector projects the pattern onto the object to be measured, and the Balser camera captures the light reflected from the surface of the object;

[0029] Figure 11 is the experimental comparison diagram of the non-highlight area of the same object with and without the influence of glare, where (a) is a stainless steel lunch box and (b) is a house model with a mirror;

[0030] Figure 12 is the comparison diagram of the wrapped phase with and without the influence of glare in the experiment and its overall phase error;

[0031] Figure 13 is the influence intensity of glare on other areas during the N-step phase shift in the experiment and the wrapped phase value caused by glare;

[0032] Figure 14 is the photographed image of the standard flat plate scene with a mirror;

[0033] Figure 15 is the quantitative comparison diagram of the standard flat plate with a mirror under the influence of (a) glare, (b) glare removal by the cross-polarization method, and (c) glare removal by the single-frame windowed Fourier method;

[0034] Figure 16 is the effect comparison diagram of different metal objects under the influence of glare, glare removal by the cross-polarization method, and glare removal by the single-frame windowed Fourier method. Specific implementation manner

[0035] The present invention proposes a method for object error modeling and precise three-dimensional reconstruction under the influence of glare, including constructing an error model for three-dimensional reconstruction using fringe projection profilometry (FPP) under the influence of glare, and obtaining that the periodic error consistent with the projection fringe period appears in the reconstructed point cloud of the metal object in the non-highlight area, which is an unwanted global illumination effect caused by the multiple reflections and scattering of the reflected strong light in the camera body and the lens optical system, that is, glare. Then, a two-dimensional function, the glare spread function (GSF), is introduced to quantitatively represent the additional brightness generated by glare during the entire imaging process. The influence of glare on the entire image is simulated by the convolution of this function and the actual brightness of the original fringe highlight area. In the fringe pattern taken by N-step phase-shift, the influence of the additional intensity of glare on the point pixels in other areas caused by the highlight area is in the form of a sine function with basically the same phase. The intensity of each step of this function is superimposed on the intensity of the sine function originally modulated by phase shift, resulting in a periodic fringe-like error in the phase solved by FPP, affecting the three-dimensional result. Finally, according to the characteristics of glare, a method of selecting and building a cross-polarization optical path or single-frame windowed Fourier transform according to requirements is proposed to solve the phase error caused by glare. The present invention mainly analyzes the causes of glare and the reasons for the phase fringe-like error, and then selects the cross-polarization method or single-frame windowed Fourier method to solve the phase according to the requirements of detail accuracy or acquisition efficiency, and removes the phase error caused by glare. The analysis results and measurement accuracy are verified in different high-reflectivity scenarios, and the results effectively verify the accuracy of the established glare model and solve the fringe-like error of the reconstructed object point cloud caused by glare.

[0036] The present invention first explains the cause of glare when photographing certain metal objects, distinguishes glare from overexposure, and introduces the glare spread function (GSF) to quantitatively characterize the glare in the image. Based on the above function, we establish an error model to show how glare causes fringe-like errors in three-dimensional reconstruction using the FPP method. On this basis, two different methods of building a cross-polarization optical path and using single-frame windowed Fourier transform are proposed to correct the periodic fringe-like error in three-dimensional object reconstruction caused by glare under different requirements for reconstruction details and efficiency.

[0037] Combined with Figure 1 , a method for object error modeling and precise three-dimensional reconstruction under the influence of glare according to the present invention includes the following steps:

[0038] Step 1: Project a set of phase-shifted sine fringe patterns onto a highly reflective metal object through a projector. Analyze the phase change of the fringes captured by the camera through the FPP algorithm, and reconstruct the 3D shape and surface information of the object.

[0039] Clarify the reason for the striped error consistent with the projection stripe period in the reconstruction of the non-highlight area of metal objects in a high-reflection scene. The glare generated by the specular reflection light of the object passing through the lens reduces the overall image quality by introducing uneven brightness and contrast changes, affecting the accurate extraction of phase information.

[0040] Step 2: Introduce the glare transfer function GSF, quantitatively represent the influence of glare on the entire image as the convolution of GSF and the ideal value when the image has no glare, and approximate the scattered veil glare GSF affecting the entire image with a Gaussian function.

[0041] Step 3: Represent the intensity influence of glare on a fixed pixel point in the image within one phase shift period through the convolution formula, establish an error model, and deduce that the phase in the non-highlight area of the object and the striped error in the reconstructed point cloud are due to the superposition of the additional intensity in the form of a sine function caused by glare in the N-step phase shift image and the original intensity of the fringe pattern.

[0042] Step 4: Solve the influence of glare through the cross-polarization method. Place a linear polarizer in front of the projector and the camera respectively, adjust the two polarizers to make the field of view darkest so that their polarization directions are orthogonal. Most of the specularly reflected light is filtered out and the generation of glare is suppressed. Then project a set of phase shift sine fringes, and through the calculation of FPP, achieve high-precision and multi-detail three-dimensional point cloud reconstruction.

[0043] This method also has another scheme. Project a high-frequency sine fringe pattern onto the object through the projector and capture it through the camera. According to the different components of glare and fringe information in the frequency domain image, solve the influence of glare through the windowed Fourier method to achieve fast and efficient three-dimensional point cloud reconstruction.

[0044] Furthermore, in the above-mentioned step 1, the sine fringe pattern captured by the camera is

[0045]

[0046] where is the background light intensity, is the modulation intensity, n is the image phase shift of the nth step, N is the total number of phase shifts, is the image phase to be solved.

[0047] Furthermore, glare is an unwanted global illumination effect caused by multiple reflections and scattering of light within the camera body and lens optical system. This phenomenon reduces the image quality by introducing uneven brightness and contrast changes, changing the gray value of each fringe image, and causing periodic errors in the phase and reconstructed point cloud to be consistent with the projection stripe period.

[0048] There is a significant difference between image glare and image overexposure. Overexposure occurs when the light intensity received by a local area of the camera exceeds the dynamic range of its sensor, resulting in the loss of details, the "burning" of bright part information, presenting pure white or high-brightness parts, and making it impossible to solve the phase of the stripes in the high-brightness area. Glare is the result of the scattering and reflection of strong light in the lens optical system. It causes an overall degradation of image quality and generates non-uniform additional light in non-bright areas, resulting in periodic errors in the phase of the non-bright areas obtained by solving.

[0049] Furthermore, the glare intensity received by the pixels in step 2 depends on the quality of the lens and its distance to the glare point pixel. This two-dimensional function describing the intensity is called the glare spread function (GSF) of the lens. In an image, glare can be approximated by using the spatially invariant convolution of the GSF in a traditional camera.

[0050]

[0051] where represents the incident radiation projected onto the coordinate pixel, and represents the GSF. The GSF can be simulated by specific effects of scattering and reflection. Generally, the optical parameters of the lens are used to describe the characteristics of the GSF. We use to represent the distance from the pixel to the glare point, that is , represents the coordinates of the glare point. The GSF can be expressed as

[0052]

[0053] where, is the Dirac function. The first term is the instantaneous glare point modulated by . The second term is a model of the spread of glare from the center point outward. It is an exponential decay function, reflecting the impact of veiling glare on the image, which is a generalized scattering phenomenon that occurs when high-intensity direct light passes through the system aperture.

[0054] Veiling glare, as a subclass of scattering glare, affects the entire image and is also the main factor affecting the fringe phase in FPP. In a typical lens, the GSF can be simply expressed by a Gaussian function

[0055]

[0056] is a parameter to control the degree of spread of scattering glare. It can be seen from the formula that the degree of glare spread decays with the increase of distance.

[0057] is a constant, which is represented by C. For image glare, we can obtain

[0058]

[0059] Among them, is the actual brightness of the highlighted area, and its grayscale value is generally much greater than 255 in the simulation.

[0060] Furthermore, in step 3, for the pixel points at a fixed distance from the glare point, the influence of glare on them during N-step phase shift is equivalent to the sine function of the above-mentioned intensity change multiplied by a fixed parameter, that is

[0061]

[0062] Among them, is the actual light intensity of the glare point during N-step phase shift, is the phase of the glare point, is during N-step phase shift the additional light intensity generated by the influence of the glare point at this position, is the constant obtained by multiplying C by the modulation degree of the glare point.

[0063] For such a pixel, the additional light intensity generated by its being affected by glare is closely related to the value of the glare point. When the distance 𝑟 between the two is fixed, the intensity influence of the glare point on it within a phase period is proportional to the sine change of the intensity of the glare point itself. The change of the intensity of this additional influence within N-step phase shift also shows the form of a sine function.

[0064] In the N-step phase shift of the FPP method, the n-th step phase shift is used as the independent variable. For different glare points, due to the same number of phase shift steps, the angular frequency of its changing sine function is . And for a fixed pixel, the final influence value it receives is the sum of the influences of all glare points on it. It is known that for two sine functions with the same , their amplitudes are P and Q respectively, and their phases are and , and the formula for their addition is

[0065]

[0066] In the N-step phase shift, the influence of all glare points in this area on the light intensity change of the fixed pixel is equivalent to a sine function with a fixed angular frequency , where . Since the glare influence range covers all pixels in the image, each pixel will receive an additional additional intensity value, and this intensity value shows the form of a sine function in the N-step phase shift.

[0067] During one phase cycle, the influence of each glare point on a pixel is proportional to The distances from two adjacent glare points to a certain pixel are respectively and The phase influence on a certain pixel is

[0068]

[0069] During the N-step phase shift process, represents the additional phase in the sine function of the influence values of all glare points in the glare area on a given pixel, which is calculated sequentially by the above formula. The scattered glare effect covers the entire image, so the value of 𝜎 is very large. For the pixels in the non-highlight area, the influence of their distance from the glare area on the additional phase value is extremely small. Therefore, the difference in the overall additional phase value of the image can be almost ignored. For FPP, the method of solving the wrapped phase is based on the sine function formed by the pixel intensity values obtained during the N-step phase shift process. For each pixel in the image, the effect of glare is equivalent to adding an additional sine function with a fixed value to the original sine function representing its intensity value. Therefore, the phase of each pixel obtained deviates from the true value. Compared with the true phase, this phase error shows the same period as the stripe. This also explains why the error in the reconstructed point cloud shows a similar waveform.

[0070] Furthermore, the cross-polarization method is adopted in step 4. During the FPP imaging process, a linear polarizer is placed in front of the projector and the camera respectively to form a polarization pair. By adjusting the two polarizers to make the camera field of view darkest, the polarization directions of the two polarizers are orthogonal. When the metal surface maintains specular reflection, most of the light can maintain the linear polarization state of the incident light well, and then be effectively filtered out by the analyzer. However, due to reasons such as surface roughness and scattering, a small part of the light will be depolarized. This method filters out most of the specular reflection light, only leaving a small amount of diffuse reflection generated by depolarization and some remaining specular reflection residues that are not completely removed, thereby suppressing the generation of glare, ensuring correct phase extraction, and achieving three-dimensional reconstruction with low efficiency but very high precision.

[0071] Furthermore, glare usually appears as low-frequency components in the two-dimensional image, while the phase information in the stripe image is mainly concentrated in the high-frequency region. The camera captures a single-frame high-frequency stripe projected by the projector onto the object. Using the single-frame windowed Fourier transform (WFT), the image can be decomposed into different frequency components. Analyzing the spectrum using a local window can extract the high-frequency stripe information and effectively filter out the low-frequency glare, achieving three-dimensional reconstruction with slightly blurred details but fast and efficient.

[0072] The specific implementation manners of the present invention will be described below in conjunction with the accompanying drawings and embodiments, so as to more clearly and completely elaborate the technical solutions of the present invention.

[0073] Embodiment

[0074] In combination with Figure 1 , a method for object error modeling and precise three-dimensional reconstruction under the influence of glare includes the following steps:

[0075] A. Generation and classification of glare

[0076] Glare is an unwanted global illumination effect caused by multiple reflections and scattering of strong light within the camera body and lens optical system. This phenomenon reduces the image quality by introducing uneven brightness and contrast changes, alters the gray values of each fringe image, and causes periodic errors consistent with the projection fringe period to appear at the sought phase and reconstructed point cloud.

[0077] Although both are caused by the reflected light from the high-brightness areas of metal objects, there are significant differences between image glare and image overexposure. As Figure 2 shown, area A represents the overexposed area of the image, and area B represents the non-overexposed area of the image, that is, the area affected by glare. Overexposure occurs when the light intensity received by a local area of the camera exceeds the dynamic range of its sensor, resulting in the loss of details, the "burning" of the bright part information, presenting as pure white or high-brightness parts, and making it impossible to solve the phase of the fringes in the high-brightness area. Glare is the result of the scattering and reflection of strong light in the lens optical system. It causes an overall decrease in image quality and generates non-uniform additional light in the non-high-brightness area, resulting in periodic errors in the phase of the non-high-brightness area obtained by solving. In short, overexposure targets the overexposed parts in the image, while glare targets all areas of the entire image.

[0078] For a camera, the lens glare that affects the image is mainly the combined result of scattered glare, reflected glare, and body free glare. Scattered glare is generated by the diffusion effect on the lens surface. It is a scattering phenomenon that occurs when high-intensity direct light passes through the system aperture. Reflected glare is caused by a series of complex reflections of strong light sources on the lens surface. It appears in the form of parasitic images, presenting as ghosts and flares. Body free glare is formed by the light scattering of the last lens element and appears outside the spot. It can be eliminated by using a smaller aperture. Therefore, it can be ignored in practice. Figure 3 It can be seen the imaging optical path diagrams of these different glares under the influence of strong light.

[0079] The additional intensity generated by the pixels affected by glare is crucial in image processing, which mainly depends on the quality of the lens and its distance to the pixel of the glare point. This two-dimensional function describing the intensity is called the glare spread function (GSF) of the lens. Glare is the additional radiation projected onto the sensor can be approximated by using the spatially invariant convolution of the GSF in a traditional camera

[0080]

[0081] where represents the incident radiation projected onto the coordinate pixel, that is, the true brightness of the ideal image without glare, represents the GSF. To improve the computational efficiency, the convolution operation is usually carried out in the Fourier domain. It can be expressed as

[0082]

[0083] where and are the Fourier transform forms of the expressions and , is the frequency domain coordinate, that is, the frequency variable after two-dimensional Fourier transform.

[0084] In an image with obvious glare, even a bright pixel in one corner will affect the pixels in the farthest corner. Therefore, in the modeling process, to more accurately simulate glare, the size of the convolution kernel needs to be twice that of the image.

[0085] In practical applications, the GSF can be simulated by specific effects of scattering and reflection. Usually, the optical parameters of the lens are used to describe the characteristics of the GSF. We use to represent the distance from the pixel to the glare point, that is , represents the coordinates of the glare point, and then we can represent the GSF with the following formula

[0086]

[0087] where is the Dirac function, the first term is the instantaneous glare point modulated by , represents the modulation degree. The second term is the model of the glare spreading outward from the center point. It is an exponential decay function, where is the scaling intensity, is the influence decay rate, is according to the distance Adjusting the non - linear response reflects the impact of veiling glare on the image, which is a generalized scattering phenomenon that occurs when high - intensity direct light passes through the system aperture. Figure 4 It reflects the variation curve of GSF with distance.

[0088] As a subclass of scattering glare, veiling glare affects the entire image and is also a major factor affecting the fringe phase in FPP. In a typical lens, GSF can be simply represented by a Gaussian function

[0089]

[0090] Among them, are the coordinates of the glare point, are the coordinates of other points affected by glare, is a parameter controlling the degree of glare diffusion, representing the scattering intensity. It can be seen from the formula that the degree of glare diffusion decays with the increase of distance.

[0091] is a constant, replace it with C. Therefore, for image glare, we can get

[0092]

[0093] Among them, is the actual light intensity of the highlighted area, and its gray value is generally much larger than 255 in the simulation. Figure 5 It reflects the influence effect of glare on the fringe image.

[0094] B. Error model of the influence of glare on phase in FPP

[0095] When using the FPP algorithm, the sinusoidal fringe pattern captured by the camera is

[0096]

[0097] Among them is the background light intensity, is the modulation intensity, is the image phase to be solved.

[0098] During the FPP process, the original value of the glare point shows a sinusoidal - function - form change in intensity during the N - step phase - shift process. Its initial value is greater than 255. Then, for the pixel points at a fixed distance from the glare point, the influence of glare on them during the N - step phase - shift is equivalent to multiplying the above - mentioned sinusoidal function of intensity change by a fixed parameter, that is

[0099]

[0100] Among them, are the coordinates of the glare point, is the actual light intensity of the glare point in N-step phase shift, is the phase of the glare point, in N-step phase shift is the additional light intensity generated at due to the influence of the glare point, and

[0101] is a constant obtained by multiplying C by the modulation degree of the glare point. For such a pixel, the additional light intensity generated by the glare influence is closely related to the value of the glare point. When the distance between the two is fixed, the intensity influence of the glare point on it within a phase period is proportional to the sine change of the intensity of the glare point itself. Therefore, the change of this additional influence intensity within N-step phase shift also shows the form of a sine function. This conclusion is obtained under the assumption of only one glare point. In fact, the actual influence of the glare area on other pixel points is the sum of all glare points in this area. In the N-step phase shift of the FPP method, the nth step phase shift is taken as the independent variable. For different glare points, due to the same number of phase shift steps, the angular frequency of the sine function of its change is . And for a fixed pixel, the final influence value it receives is the sum of the influences of all glare points on it. It is known that for two sine functions with the same and , their amplitudes are P and Q respectively, and their phases are

[0102]

[0103] Therefore, in the N-step phase shift, the influence of all glare points in this area on the light intensity change of a fixed pixel is equivalent to a sine function with a fixed angular frequency , where . This additional sine function affects the original sine function intensity generated by the fringe phase shift. Therefore, in each step of phase shift, the actually measured light intensity deviates from the true value. Since the glare influence range covers all pixels in the image, each pixel will receive an additional additional intensity value, and this intensity value shows the form of a sine function in the N-step phase shift. Figure 6 shows the influence of glare on the intensity of other pixel points in the image (showing the form of a sine function) during the N-step phase shift.

[0104] Within a phase period, the influence of each glare point on a pixel is proportional to the change of and are the distances from two adjacent glare points to a certain pixel respectively, and their phase influence on a certain pixel is

[0105]

[0106] During the N-step phase shift process, The additional phase in the sine function representing the influence value of all glare points in the glare area on a given pixel is calculated sequentially by the above formula. Since the influence covers the entire image, the value of 𝜎 is very large. Therefore, for a specific pixel, the influence of its distance from the glare area on the additional phase value is extremely small. This results in a very small difference in the overall additional phase value of the image, which is almost negligible. For FPP, the method of solving the wrapped phase is based on the sine function formed by the pixel intensity values obtained during the N-step phase shift process. For each pixel in the image, the effect of glare is equivalent to adding an additional sine function with a fixed value to the original sine function representing its intensity value, and the specific effect is reflected in Figure 7 . Therefore, the phase of each pixel obtained deviates from the true value. Compared with the true phase, this phase error shows the same period as the fringe. This also explains why the error in the reconstructed point cloud presents a similar waveform. Figure 8 Details show the influence of glare on phase and 3D reconstruction.

[0107] Therefore, for each pixel point whose phase is solved, the influence of glare is equivalent to adding a new N-step phase shift on the basis of the original N-step phase shift. Therefore, for the pixel points affected by glare, its formula is

[0108]

[0109] where is the total intensity of the N-step phase shift map under the influence of glare, is the ideal phase, is the additional phase shift caused by glare, is the modulation degree of the function representing the influence of glare. Solving the phase by the least squares method, the original equation is expressed as

[0110] ; the phase difference of each step of phase shift . By performing at least three steps of phase shift, and the values of the previous parameters can be determined. Finally, the phase value affected by glare is obtained through the arctan function. It can be seen from the formula that in the images taken at each step of phase shift, the true fringe pattern and the extra light caused by glare are mixed together, indistinguishable, and it is also difficult to compensate at the formula level. Therefore, it is necessary to use some other methods to eliminate the error and reconstruct the correct 3D point cloud model.

[0111] C. Cross-polarization method for phase unwrapping and glare reduction

[0112] During FPP imaging, a linear polarizer is placed in front of the projector and the camera respectively to form a polarization pair. A linear polarizer is an optical element that precisely controls the polarization state of light waves. It regulates and filters the polarization characteristics of light by selectively allowing light waves with oscillation directions aligned with its transmission axis to pass through while blocking light waves oscillating in other directions. The optical path principle of the cross-polarization method is as shown in Figure 9 shown below.

[0113] The light emitted by the projection light source is modulated by the polarizer to form linearly polarized light with a specific polarization direction, ensuring that only light with the desired polarization direction is transmitted. On this basis, by precisely controlling the polarization direction of the projected light, the intensity of the reflected light can be significantly reduced. Most specularly reflected light remains in its original polarization state after reflection from the metal surface and is then effectively filtered out by the analyzer. After passing through the analyzer, the relationship formula between the light intensity and the polarization angle can be expressed by Malus' law

[0114]

[0115] where is the intensity of the incident light before entering the analyzer, is the intensity of the transmitted light after passing through the analyzer, is the angle between the polarization direction of the incident light and the axis of the analyzer. For orthogonally placed polarizers, , ideally, the specular reflection component should be completely blocked. However, since a metal object is being photographed, when this polarized light is incident on the metal surface, its reflection characteristics follow the Fresnel equations

[0116] ;

[0117] ;

[0118] where is the reflection coefficient of s-polarized light (perpendicular to the plane of incidence), is the reflection coefficient of p-polarized light (parallel to the plane of incidence), is the complex refractive index of the metal, which reflects the optical properties of the metal material, is the angle of incidence. , where is the real part of the refractive index, is the extinction coefficient, which characterizes the light absorption properties of the material. The high value of the metal results in not only an amplitude change but also a phase change when light is reflected. The phase difference Convert linearly polarized light into elliptically polarized light. When polarizers are orthogonal, most specular reflection light can be removed, but there are still some components in the elliptically polarized light that can pass through, forming residual light intensity. The incident angle affects the Fresnel reflection coefficient , values, and thus changes the amplitude and phase difference. When the incidence is normal, and , so we should try to reduce the distance between the projector and the camera as much as possible while ensuring sufficient limit, reduce the incident angle, and thus reduce the residual light intensity caused by the phase difference.

[0119] In addition, during the actual imaging process, due to reasons such as surface roughness, a small part of the light will be depolarized (unpolarized or partially randomized), and the depolarization effect caused by the surface micro-roughness generates a diffuse reflection component of

[0120]

[0121] wherein, represents the intensity of the incident light, represents the intensity of the diffuse reflection component caused by depolarization, and the depolarization degree quantitatively describes the degree of polarization state randomization, is the diffuse reflectance. This part of the diffuse reflection is very important, which enables the camera to clearly capture the metal object under cross-polarization with increased exposure.

[0122] In addition, there is also some residual specular reflection caused by the quality of the polarizer itself and the misalignment error angle, etc. To reduce such errors, linear polarizers with higher optical quality and larger extinction ratios are selected in the experiment, and a rotary table with a graduation accuracy of <0.1° is used for its installation.

[0123] Generally speaking, by placing polarizers in front of the projector and the camera respectively and adjusting them to be orthogonal to each other, most of the specular reflection light that maintains the polarized state on the metal surface is effectively blocked, leaving only a small amount of residual specular reflection light caused by phase delay, polarizer quality and misalignment error angle and the diffuse reflection generated by depolarization, thus significantly reducing the influence of glare and ensuring the correct phase for three-dimensional reconstruction.

[0124] D. Single-frame windowed Fourier phase unwrapping to remove glare

[0125] Using the WFT method for phase retrieval can effectively eliminate the influence of glare and achieve fast and efficient 3D reconstruction. This is because glare usually appears as low-frequency components in 2D images, while the phase information in fringe images is mainly concentrated in the high-frequency region. Using WFT, the image can be decomposed into different frequency components. By using local window analysis of the spectrum to extract high-frequency fringe information, the low-frequency glare can be effectively filtered out. Therefore, the finally calculated phase information only reflects the fringes and is not affected by glare. This ensures that the calculated phase mainly represents the true changes of the fringes, greatly reducing the interference of glare on phase measurement and improving the accuracy of phase unwrapping.

[0126] First, the camera captures a single-frame fringe image projected by the projector and performs a windowed Fourier transform on it.

[0127] ;

[0128] ;

[0129] ;

[0130] is the fringe image. is the spectrum after the windowed Fourier transform of the fringe image. is the frequency-domain form of the window function. The window function is a Gaussian function. and represent the spatial offsets in the x-axis and y-axis directions respectively. and are the standard deviations of the Gaussian function in the x-direction and y-direction, controlling the extent of the window function's expansion in the spatial domain. and are the frequencies of the window function in the x-direction and y-direction.

[0131] Then, to effectively separate the frequencies of glare and fringes, the maximum value of the amplitude of the spectrum information obtained by multiplying the spectrum of the fringe image by window functions of different frequencies is taken as the spectrum output result, that is, the most significant frequency components corresponding to different window functions are found in the frequency domain.

[0132]

[0133] Among them, respectively represent the frequency components of the image in the horizontal and vertical directions at the local point . represents the frequency corresponding to the maximum value when and the frequency . At this time, the value of is . The value of . Finally, perform the inverse window Fourier transform on the output spectral values to restore the amplitude and phase information of the object to be measured, while effectively filtering out the frequencies affected by glare.

[0134]

[0135]

[0136]

[0137] Among them, is the fringe image restored after the windowed Fourier transform, is the amplitude of the object to be measured, is the wrapped phase of the object to be measured. Based on this, phase unwrapping is performed to carry out the three-dimensional reconstruction of the object.

[0138] Projecting higher-frequency fringes with a projector can better eliminate the influence of glare on the phase, because this is more conducive to the separation of high-frequency and low-frequency components in the spectrum. During the process of using the windowed Fourier transform for fringe phase calculation and three-dimensional reconstruction, due to the locality limitation of the window function, there is a certain compromise between the frequency resolution and the spatial resolution, resulting in the failure to fully capture the spatial frequency information of the details, thus making the local structural details of the reconstructed object not clear enough. A larger window helps to improve the frequency resolution and the stability of phase extraction, while better filtering out the influence of glare, but it will smooth out the subtle changes in the image, thus reducing the spatial details; while a smaller window can retain more local details, but may lead to spectral leakage and phase instability. Coupled with the influence of noise in actual measurement, finally, the reconstructed three-dimensional object has blurred local details. Therefore, when our requirement for the reconstruction accuracy of the object is relatively high, this method is not applicable. However, a significant advantage of this method is that only one fringe image is required to solve the phase, greatly improving the measurement efficiency and being more suitable for dynamic measurement.

[0139] E. Experiment

[0140] This three-dimensional measurement system mainly includes a DLP6500 projector with a resolution of 1920×1080, a Basler acA1920-40um CMOS camera with a resolution of 1920×1200, and a camera lens with a focal length of 35 mm. The object to be measured should be placed about 0.5 m away from the measurement system. For FPP, the fringe selection , , generating 12 phase-shifted patterns, and each measurement takes 2 seconds. The experimental configuration is as Figure 10 shown.

[0141] First, verify the conditions and forms of the influence of glare on non-highlight areas in the above formula. Project the stripe pattern onto the same scene where glare occurs in the area with highlight. The objects to be measured are a stainless steel lunch box and a doll with a mirror. The difference is that the first projection covers the entire scene, while the second projection excludes the highlight area where glare will occur. Such a comparative experiment eliminates a series of influencing factors, such as the material properties of the objects, etc., and can intuitively explore the source of periodic stripe-like errors.

[0142] Figure 11 It can be seen that when the projection pattern is avoided from being projected onto the highlight area of the object that will produce glare, the reconstruction effect of other parts has no errors. When we project onto the entire area, the glare generated by the specular reflection of the highlight area of the object will affect the gray value of each stripe pattern, further affecting the phase of the non-glare area and resulting in stripe-like errors. This proves that the periodic errors of the phase and point cloud in the non-highlight area are caused by the glare generated by the strong light reflection in the highlight area of the object.

[0143] To further verify, we explore the influence effect of glare on the phase of the standard flat plate. Figure 12 The wrapped phases of the standard flat plate in the experiments with and without glare interference are shown. The phase difference between the two shows as stripe-like, with a period matching the projection stripes, which is consistent with the theoretical derivation conclusion.

[0144] Only project the stripes onto the mirror and ensure that the projection light intensity of the surrounding background is zero. Due to the influence of glare in the highlight area of the mirror during each phase-shift projection, there is still a certain gray value in the background area captured by the camera. Solve its wrapped phase, and the final result is consistent with the principle. In Figure 13 it, the additional wrapped phase values caused by glare vary very little throughout the image. This additional phase results in periodic stripe-like errors in the final reconstructed 3D result.

[0145] To solve the influence of glare on the three-dimensional reconstruction of objects, for high-precision and detailed three-dimensional reconstruction tasks, the cross-polarization method is used to effectively suppress glare; while in scenarios with higher requirements for reconstruction speed and efficiency, the single-frame windowed Fourier method is used for fast glare removal and then three-dimensional reconstruction is carried out. Measure a standard flat plate with a mirror, Figure 14 which is the scene picture. The mirror is the highlight area that generates glare, and its specular reflected light directly enters the lens. Test the influencing way of glare generation and verify the effects of these two methods in eliminating the influence of glare. The three-dimensional reconstruction results of the flat plate affected by glare are experimentally measured and compared with the results of the above methods. The plane fitting error of the measured flat plate is used to evaluate the measurement accuracy.

[0146] Figure 15It can be seen that the fitting error of the standard flat plate under the influence of glare is very large, and the periodic stripe error is very obvious. The standard error of its plane fitting is 0.25 mm. Both the cross-polarization method and the single-frame Fourier method can effectively remove the influence of glare on the phase and three-dimensional point cloud. The standard error of their plane fitting is within 0.03 mm, and both can very effectively remove the influence of glare. The flat plate surface obtained by the single-frame windowed Fourier method is smoother, which is caused by the characteristics of the algorithm.

[0147] Figure 16 Taking an aluminum plate, a stainless steel lunch box, a steel ruler and a CD as examples for measurement, they are all objects with high-brightness areas and glare under projection. The traditional FPP is used to measure the target and compared with the cross-polarization method and the single-frame windowed Fourier method. It can be seen from the experimental results that both methods can effectively solve the glare influence on metal objects and eliminate the periodic errors of the phase and point cloud stripes. The difference is that for the aluminum plate and the stainless steel lunch box with relatively smooth surfaces and few defects, both the cross-polarization method and the windowed Fourier method can obtain good reconstruction results. However, for the steel ruler with more scales and the CD with more scratches, the reconstruction effect of the cross-polarization method is still good, but the result obtained by the windowed Fourier method is poor, and very large errors occur at the scales of the ruler and the scratches of the CD, which is due to the influence of the single-frame windowed Fourier algorithm. In the application of filtering out low-frequency glare components, it is usually more beneficial to use a larger window function, because a larger window can provide higher frequency domain resolution, so as to more accurately separate and suppress low-frequency glare components. However, a larger window will reduce the spatial resolution, resulting in the blurring of local details, and is not suitable for the three-dimensional reconstruction of objects with more details and extremely high precision requirements. However, the single-frame windowed Fourier has the advantages of fast speed and high efficiency. Only one frame needs to be collected, and its average acquisition frame rate is 200 frames per second, and it takes about 0.05 seconds to collect the image. Due to the fact that the cross-polarization method filters out most of the specular high-reflection light, the camera field of view is extremely dark, and the average exposure time during acquisition needs to be extended to 3 million microseconds to take a clearer image. To shoot a complete FPP process, it takes about 2 minutes to completely collect the image, and its acquisition efficiency is very low. Therefore, it is necessary to select a method for accurate three-dimensional reconstruction of glare removal according to specific requirements. The experiment shows that the cross-polarization method is used to remove glare for high-precision and multi-detail reconstruction, while the single-frame windowed Fourier method is used to remove glare for fast and efficient reconstruction.

[0148] The above specific implementation manners only describe the preferred implementation manners of the present invention, and do not limit the protection scope of the present invention. Without departing from the design concept and spirit scope of the present invention, various deformations, substitutions and improvements made by those of ordinary skill in the art to the technical solutions of the present invention according to the text description and drawings provided by the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for object error modeling and precise three-dimensional reconstruction under the influence of glare, characterized in that Including the following steps: Step 1: Project a set of phase-shifted sinusoidal fringe patterns onto a metal object through a projector; analyze the phase change of the fringes captured by the camera through the FPP algorithm to reconstruct the 3D shape and surface information of the object; Step 2: Introduce the glare transfer function GSF, quantitatively represent the influence of glare on the entire image as the convolution of GSF and the ideal value when the image has no glare, and approximate the scattering veil glare GSF that affects the entire image with a Gaussian function; Step 3: Represent the influence of glare on the intensity of a fixed pixel point in the image within a phase shift period through the convolution formula, establish an error model, and deduce that the phase in the non-highlight area of the object and the appearance of striped errors in the reconstructed point cloud are due to the superposition of the additional intensity in the form of a sine function caused by glare in the N-step phase shift image and the original intensity of the fringe pattern; Step 4: Implement 3D point cloud reconstruction by using the cross-polarization method or the single-frame windowed Fourier transform method; Among them, the cross-polarization method is specifically as follows: Place a linear polarizer in front of the projector and the camera respectively, adjust the two linear polarizers to the darkest field of view so that the polarization directions of the two linear polarizers are orthogonal, and then project a set of phase-shifted sinusoidal fringes. Through the calculation of FPP, 3D point cloud reconstruction is realized; The single-frame windowed Fourier transform method is specifically as follows: Project a high-frequency sinusoidal fringe pattern onto the object through a projector and capture it through the camera; according to the different components of glare and fringe information shown in the frequency-domain image, solve the influence of glare through the windowed Fourier method to realize 3D point cloud reconstruction.

2. The method for object error modeling and precise three-dimensional reconstruction under the influence of glare according to claim 1, wherein In the said step 1, the sine stripe pattern captured by the camera is as follows: ； wherein is the background light intensity, is the modulation intensity, n is the image phase shift at the n-th step, N is the total number of phase shift steps, is the image phase to be solved.

3. The method for object error modeling and precise three-dimensional reconstruction under the influence of glare according to claim 2, characterized in that, The glare intensity received by a pixel depends on the quality of the lens and its distance to the pixel of the glare point. The two-dimensional function describing the intensity is called the glare diffusion function GSF of the lens; in the image, glare is approximated by using the spatially invariant convolution of GSF in a traditional camera: ； Among them represents the incident radiation projected onto the coordinate pixels, represents GSF; GSF is simulated through specific effects of scattering and reflection; the optical parameters of the lens are used to describe the characteristics of GSF; use represents the distance from the pixel to the glare point, that is , represents the glare point coordinates; GSF is expressed as ； Among them, is the Dirac function; the first term is the instantaneous glare point modulated by ; the second term is a model of the glare spreading outward from the center point, where is the scaling intensity, is the influence attenuation rate, is to adjust the non-linear response according to the distance ; In the lens, GSF is represented by a Gaussian function: ； is a parameter for controlling the diffusion degree of scattered glare; it can be seen from the formula that the diffusion degree of glare attenuates with the increase of distance; is a constant, denoted by C; for image glare, we can obtain ； Among them, is the actual brightness of the highlighted area.

4. The method for object error modeling and precise three-dimensional reconstruction under the influence of glare according to claim 3, characterized in that In the said Step 3, for the pixel points at a fixed distance from the glare point, the influence of glare on them in the N-step phase shift is equivalent to a sine function of intensity change multiplied by a fixed parameter, that is ； Among them, is the actual light intensity of the glare point in the N-step phase shift, is the phase of the glare point, in the N-step phase shift is the additional light intensity generated at the position affected by the glare point, is the constant obtained by multiplying C by the modulation degree of the glare point; In the N-step phase shift of the FPP method, the n-th step phase shift is taken as the independent variable; for different glare points, since the number of phase shift steps is the same, the angular frequency of the changing sine function is ; for a fixed pixel, the final influence value it receives is the sum of the influences of all glare points on it; it is known that for two sine functions with the same angular frequency, their amplitudes are P and Q respectively, and their phases are and , and the formula for their addition is ； In the N-step phase shift, the influence of all the glare points in this area on the light intensity change of the fixed pixel is equivalent to a sine function with a fixed angular frequency where ; Within a phase period, the influence of each glare point on a pixel is proportional to the change in ; the distances from two adjacent glare points to a certain pixel are respectively and , and the phase influence on a certain pixel is ； During the N-step phase shift process, The additional phase in the sine function representing the influence value of all glare points in the glare area on a given pixel.

5. The method for object error modeling and precise three-dimensional reconstruction under the influence of glare according to claim 4, characterized in that The single-frame windowed Fourier transform method is specifically as follows: The camera collects a single-frame fringe image projected by the projector and performs a windowed Fourier transform on it ； ； ； is a striped image is the spectrum after windowed Fourier transform, is the window function is the frequency domain form of the window function, and the window function is a Gaussian function, and are the standard deviations of the Gaussian function in the x and y directions, controlling the extent of the window function's spread in the spatial domain; and are the frequencies of the window function in the x and y directions; Select the maximum value of the amplitude of the spectral information obtained by multiplying the window functions of different frequencies with the spectrum of the fringe image as the spectral output result, and find the most significant frequency components corresponding to different window functions in the frequency domain ； Among them, respectively represent the frequency components of the image in the local point along the horizontal and vertical directions, represents when takes the maximum value, the corresponding frequency and the frequency , at this time takes the value of , takes the value of ; Performing an inverse window Fourier transform on the output spectral values can restore the amplitude and phase information of the object to be measured, and at the same time filter out the frequencies affected by glare; ； ； ； Among them, is the amplitude of the object to be measured, is the wrapped phase of the object to be measured; finally, phase unwrapping is performed to complete the three-dimensional reconstruction of the object.

6. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the said program, it realizes the steps of the method described in any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the said program is executed by the processor, it realizes the steps of the method described in any one of claims 1-5.

8. A computer program product, comprising a computer program, characterized in that, When the said computer program is executed by the processor, it realizes the steps of the method described in any one of claims 1-5.

Images