A complex-amplitude-based multi-luminance fringe image fusion method

Abstract

The application discloses a kind of multi-luminance fringe image fusion methods based on complex amplitude, utilize the complex form of phase, combine inverse variance weighting principle, all unsaturated effective pixels in measurement process are used in fringe fusion process, maximum limit uses all original measurement data, improves the measurement accuracy of high dynamic range surface object and the anti-random noise ability of structured light measurement system. The weight of each unsaturated pixel is set as the inverse of variance for weighted fusion, introduce complex amplitude in the calculation process to simplify calculation, convert the calculation of variance into the calculation of modulation, improve the speed of phase solution;Compared with the existing fringe fusion algorithm, combined with the inverse variance weighting principle, all the unsaturated pixels captured are used for fringe fusion, the optimal fringe fusion pattern with the smallest variance in theory is obtained, the utilization rate of scanning original data is improved, and the structured light three-dimensional measurement accuracy is improved;The inverse of variance of all effective pixels is used as the weight of weighted fusion, which greatly improves the anti-random noise ability of environmental light in the measurement process.

Description

Technical Field

[0001] This invention belongs to the field of high dynamic range surface scanning technology and relates to a method for multi-brightness stripe image fusion based on complex amplitude. Background Technology

[0002] Research on 3D scanning of high dynamic range (HDR) objects has made significant progress in recent years. It can be mainly divided into five methods: (1) multiple exposure method; (2) adjusting projection pattern intensity method; (3) polarizing filter method; (4) color invariant method; and (5) photometric stereo technique. Multiple exposure technique refers to fusing images acquired at different exposure times into a single image to avoid image saturation and achieve a higher signal-to-noise ratio (SNR). Zhang et al. from Harvard University first proposed the "high dynamic range scanning" technique using multiple exposures. A series of images captured at different exposure times are merged into a set of HDR images, i.e., phase-shifted images are obtained by selecting the brightest unsaturated intensity at each pixel. Liu et al. from Southwest University of Science and Technology proposed a technique combining binocular and monocular structured light to acquire the 3D morphology of highly reflective surfaces and fill in missing data in highlight and diffuse dark areas, i.e., operating two cameras and a projector from different perspectives. This method introduces image segmentation technology to distinguish which monocular system is suitable for accurately reconstructing missing points.

[0003] Zhang et al. from Huazhong University of Science and Technology proposed a weighted fusion method for multiple exposures. This method calculates the proportion of saturated pixels in the captured pattern, the magnitude of cross-reflection error, and the gradient smoothing information of the phase edges. By setting different powers of these three factors, a score is calculated, normalized, and used as the fusion weight for each exposure. However, this method lacks quantitative mathematical proof. The powers of the three evaluation indicators rely on subjective inference based on experience, and different powers significantly affect the accuracy of the measurement results. Furthermore, the computational load is enormous, increasing the system's complexity, and the measurement accuracy is only average.

[0004] Jiang et al. from Beihang University proposed a method to generate composite fringe images by adjusting camera exposure time and projected fringe light intensity. Their proposed maximum modulation index (MPI) fringe fusion strategy uses a multi-exposure method to acquire a series of fringe patterns with different brightness levels. When calculating the phase of structured light reflected from the object under test, the brightness value of the pixel with the highest modulation index in each brightness level pattern is selected pixel-by-pixel as the final calculation criterion. This method uses the highest modulation intensity, rather than the brightest unsaturated intensity, as the criterion for selecting pixels from the original fringe image, thereby achieving a higher surface-to-noise ratio (SNR) and minimizing ambient light effects.

[0005] Existing multi-exposure fringe fusion strategies primarily rely on modulation index as the determining factor. For each pixel in a fringe pattern across multiple brightness levels, the modulation index is calculated and compared, and the brightness value of the pixel with the highest modulation index is selected as the brightness value of the fused fringe pattern. This method is based on the assumption that high-modulation points have a high signal-to-noise ratio, but it does not utilize information from other brightness levels of the fringe pattern. Furthermore, this method is highly sensitive to changes in ambient light during measurement; if the ambient light noise around the object being measured is significant, the measurement accuracy will be greatly reduced.

[0006] Definitions: HDR (High Dynamic Range): In this invention, it mainly refers to a surface with a high dynamic range, that is, a surface whose reflectivity varies over a large range, making it difficult to obtain all surface texture information using a set of exposure parameters.

[0007] SNR (Signal-to-Noise Ratio): A metric used to measure the relative strength of a signal and noise. A higher SNR means the signal is stronger relative to the noise, which can improve signal quality and reliability. A lower SNR means the noise is stronger relative to the signal, which may lead to signal distortion or unreliable decoding.

[0008] Stripe fusion: In the scanning process of high dynamic range surface objects, multiple exposures are usually required to obtain all surface information of the object. Multiple exposures will produce multiple stripe patterns. Fusing these multiple patterns into a single stripe pattern is called stripe fusion. Stripe fusion is the basis for a series of subsequent steps in structured light scanning, such as phase de-phase.

[0009] Modulation degree: This refers to a measure of the magnitude or contrast of the brightness variation in a structured light pattern. A higher modulation degree means that the brightness variation in the structured light pattern is more pronounced, while a lower modulation degree means that the brightness variation is more subtle. A higher modulation degree can provide greater signal difference, allowing the camera to capture the details of the structured light pattern more accurately. This helps improve the accuracy and reliability of depth measurements and reduces measurement errors caused by noise or other interference factors.

[0010] Complex amplitude: A complex number representation used to describe the amplitude and phase of a wave or oscillation. The complex number consists of a real part representing the amplitude and an imaginary part representing the phase. Complex amplitude provides a more comprehensive description, capturing both the amplitude and phase information of the wave. Its advantage lies in its ability to handle phase differences and variations. Phase is crucial in waves and oscillations, determining the position and temporal shift of the waveform. Summary of the Invention

[0011] To overcome existing problems, this invention provides a multi-brightness stripe image fusion method based on complex amplitude, which achieves the theoretical minimum variance of the fused stripe pattern. This method has strong resistance to random noise and improves scanning accuracy.

[0012] To achieve the above objectives, the present invention employs the following technical solution: A multi-brightness stripe image fusion method based on complex amplitude includes the following steps: Step 1: The projector projects multiple stripe patterns of different brightness levels onto the object being tested. The lowest brightness level should ensure that there are no saturated pixels on the entire object being tested. Step 2: The camera captures the stripe pattern reflected from the object being measured and transmits it to the computer to obtain a phase-shifted stripe image; The light intensity In collected at a pixel (x, y) on the striped image is represented as: (1) in, It is the average light intensity of the sinusoidal fringes. It represents the modulation degree of the sinusoidal fringes, where N is the number of phase shift steps. It is a phase shift index. It is the phase of the sine stripes; Sine stripe modulation Calculated by the following formula: (2) Sine stripe phase Calculated by the following formula: (3) in, It is the arctangent function, and its range is ; Step 3: The computer traverses all phase-shifted stripe images pixel by pixel, sets the saturation gray threshold to 250, and discards any images that exceed the saturation threshold. All valid pixels are phase-fused according to the inverse variance weighting method. Using formula (6) and the complex amplitude principle, the equivalent calculation formula for inverse variance weighting is obtained: the sum of the square of the modulation index of each stripe image and the product of the real and imaginary parts of the phase is calculated and compared. Finally, the phase is expanded by arctanc and the wrapped phase is calculated. (6) Step 4: After traversing all pixels, obtain the wrapped phase, and then use the multi-frequency heterodyne method to obtain the unwrapped phase, thus obtaining a one-to-one correspondence between the phase and the coordinates; Step 5: Use stereo vision to find matching points and reconstruct the 3D point cloud data of the scanned object.

[0013] Furthermore, in Step 2, the captured stripe pattern is adaptively generated into a sequence of projection brightness and exposure time combinations based on the dynamic range of the reflected light intensity from the object's surface. This sequence is then used to scan the object, resulting in multiple sets of phase-shifted stripe images. The standard deviation of each phase image is calculated using the following formula: (4) Where σ represents the standard deviation of ambient light, which is usually considered to be Gaussian noise with a mean of 0 and a standard deviation of σ.

[0014] Furthermore, in Step 3, after the camera captures the multi-brightness stripe pattern reflected by the projector onto the object surface, the strategy for stripe fusion is to first discard invalid pixels with gray values ​​exceeding the threshold; then perform weighted fusion of gray values ​​in all valid pixels, with the weight being the inverse of the variance of each pixel; and then obtain the equivalent calculation formula (6) for the fusion phase based on the complex amplitude formula.

[0015] This invention primarily addresses the problem of fusing multiple fringe patterns using multiple exposure methods when scanning high dynamic range (HDR) surfaces. It utilizes the complex form of phase and combines it with the inverse variance weighting principle to incorporate all unsaturated effective pixels from the measurement process into the fringe fusion, maximizing the use of all original measurement data. The weight of each unsaturated pixel is set to the reciprocal of its variance for weighted fusion. Complex amplitude is introduced during the calculation to simplify the calculation, converting variance calculation into modulation calculation, thus improving the phase resolution speed.

[0016] The fused fringe pattern calculated by this invention achieves a variance that reaches the theoretical minimum. Experimental verification shows a significant improvement in the final phase resolution and 3D reconstruction results compared to the maximum modulation fusion algorithm. Furthermore, it can obtain accurate measurement results even when introducing varying ambient light noise during the measurement process, demonstrating strong resistance to random noise.

[0017] Compared to existing stripe fusion algorithms, this invention combines the principle of inverse variance weighting, using all captured unsaturated pixels for stripe fusion to obtain the optimal stripe fusion pattern with the smallest variance in theory, thereby improving the utilization rate of the original scan data and enhancing the accuracy of structured light 3D measurement; by using the inverse variance of all effective pixels as the weight for weighted fusion, the measurement process is greatly improved in terms of resistance to ambient light random noise.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1) Compared with the traditional stripe fusion strategy, the obtained fused stripe pattern has the smallest variance and the smallest random noise. It is more accurate in the phase unwinding stage and reduces the occurrence of erroneous phase results in the phase unwinding process. Experiments have verified that the accuracy error of the model before and after repeated scanning is significantly reduced, that is, the scanning accuracy is improved. 2) Compared with the traditional maximum modulation fusion strategy, weighted fusion of patterns of all brightness levels improves data utilization and greatly reduces the abnormal phase resolution results caused by noise interference of a single pixel, which greatly improves the ability to resist random ambient light noise. 3) By using Euler's formula, complex amplitude is introduced into the calculation process. Simulation confirms that the phase weighted fusion is equal to the phase real part and imaginary part weighted fusion respectively, and then the arctangent value is calculated. Using this formula reduces the number of division operations when calculating the phase, which greatly improves the accuracy of the phase calculation. Attached Figure Description

[0019] Figure 1 Simulation results of ambient light interference Figure 2 Measurement accuracy experiment results diagram Figure 3 Simulation results of two phase calculation methods Detailed Implementation The present invention will be further described in detail below with reference to specific embodiments. These descriptions are for explanation purposes only and are not intended to limit the scope of the invention.

[0020] High-level reflection can cause localized saturation in camera images, affecting phase calculation in phase-shifting fringe projection and leading to significant phase errors. For line structured light, high-level saturation reduces the accuracy of fringe center extraction. Therefore, high-level reflection affects the accuracy and completeness of shape data. The acquisition of highly reflective surface shapes based on complex amplitude fusion utilizes the principle of complex amplitude combined with inverse variance weighting, using the square of the modulation intensity of each fringe pattern as a weight for fringe fusion. This yields the theoretically optimal fused fringe pattern, thereby improving the accuracy of measuring highly reflective surfaces. Phase-shifting fringe projection projects a phase-shifting sinusoidal fringe pattern onto the object using a structured light projector. The camera acquires the fringe image, and the light intensity In at a pixel (x, y) in the image can be expressed as: (1) in, It is the average light intensity of the sinusoidal fringes. It represents the modulation degree of the sinusoidal fringes, where N is the number of phase shift steps. It is a phase shift index. It is the phase of the sine stripe.

[0021] Adjustment system It can be calculated using the following formula: (2) phase It can be calculated using the following formula: (3) in, It is the arctangent function, and its range is Therefore, the calculated phase Wrapped in Therefore, it is necessary to unwrap the phase to obtain the absolute phase, and then calculate the three-dimensional coordinates using the trigonometric principle. If a fixed brightness and exposure sequence is used, it cannot adapt well to various surface materials. Therefore, it is necessary to adaptively generate a sequence of projected brightness and exposure time combinations based on the dynamic range of the reflected light intensity of the object's surface to scan the object, obtaining multiple sets of phase-shifted fringe images. The standard deviation of each phase image can be calculated by the following formula: (4) Where σ represents the standard deviation of ambient light, which is typically considered to be Gaussian noise with a mean of 0 and a standard deviation of σ. According to the inverse variance weighting principle, choosing the reciprocal of the variance as the weight ratio yields the fused fringe pattern with the minimum variance, i.e., the weights: (5) Through Matlab simulation experiments, the final phase obtained by weighted averaging of the phases of multiple brightness levels and then taking the weighted average of the real and imaginary parts of the phase before arctan analysis can be compared. The results show that the error range between the two methods is maintained within 10. -15 -10 -32 Between orders of magnitude, such as Figure 3 As shown, the two images on the left, a and b, are the wrap-around phase maps obtained by the two methods, and the image on the right, c, is a pie chart of the error distribution for both methods. This verifies that the phase-weighted average of multiple fringe patterns is equal to the weighted average of the real and imaginary parts of the phase, then taken as an arctanine, i.e.: (6) The cumulative sum can be calculated by multiplying the square of the modulation index of each fringe pattern by the product of the real and imaginary parts of the phase, and then compared. Finally, arctanc can be used to obtain the final unfolded phase, which improves the accuracy and speed of 3D scanning of highly reflective objects.

[0022] The specific steps in the actual measurement process are as follows: Step 1: The projector projects multiple stripe patterns of different brightness levels onto the object being tested. The lowest brightness level should ensure that there are no saturated pixels on the entire object being tested. Step 2: The camera captures the stripe pattern reflected from the object being measured and transmits it to the computer; Step 3: The computer traverses all phase-shifted stripe images pixel by pixel, sets the saturation gray threshold to 250, and discards images if they exceed the saturation threshold; otherwise, it calculates the wrapping phase using formula (6). Step 4: After traversing all pixels, obtain the wrapped phase, and then use the multi-frequency heterodyne method to obtain the unwrapped phase, thus obtaining the one-to-one correspondence between the phase and the coordinates; Step 5: Use stereo vision to find matching points and reconstruct the 3D point cloud data of the scanned object.

[0023] like Figure 1 As shown, through ambient light noise simulation, the present invention demonstrates better resistance to random noise compared to the maximum modulation method. The first row represents different types of ambient light, simulating linear sinusoidal ambient light interference with varying amplitudes and frequencies; the second row represents the phase result obtained by the maximum modulation fusion strategy, with the number in the upper right corner indicating the percentage of error compared to the correct phase result; the third row represents the phase result obtained by the present invention. It can be seen that the present invention has stronger resistance to ambient light random noise.

[0024] like Figure 2 As shown, the measurement accuracy of the algorithm is determined by comparing the final 3D point clouds obtained from two sets of reflective fringe patterns obtained through continuous imaging. Column a represents the measured object, selecting two types of objects with large variations in surface reflectivity and those exhibiting high-frequency mutual reflection; column b represents a captured fringe pattern; column c represents the reconstructed 3D point cloud model; column d compares the differences between the two results calculated using the maximum modulation fusion algorithm; and column e compares the differences between the two results calculated using this invention. It can be seen that compared to column d, the two measurement results in column e are closer, with a smaller measurement error.

[0025] like Figure 3 As shown, simulations were used to verify the error range of the two phase calculation methods, and the error results were both within 10. -15 Up to 10 -32 Within the order of magnitude, it can be verified that the weighted average of the phases is equal to the weighted average of the real and imaginary parts of the phases, compared and then arctaned. Through this formula transformation, the present invention can reduce the number of division operations in phase calculation, significantly improve the accuracy of phase calculation, and thus improve the accuracy of three-dimensional measurement results.

Claims

1. A multi-brightness stripe image fusion method based on complex amplitude, characterized in that... Includes the following steps: Step 1: The projector projects multiple stripe patterns of different brightness levels onto the object being tested. The lowest brightness level should ensure that there are no saturated pixels on the entire object being tested. Step 2: The camera captures the stripe pattern reflected from the object being measured and transmits it to the computer to obtain a phase-shifted stripe image; The light intensity In collected at a pixel (x, y) on the striped image is represented as: (1) in, It is the average light intensity of the sinusoidal fringes. It represents the modulation degree of the sinusoidal fringes, where N is the number of phase shift steps. It is a phase shift index. It is the phase of the sine stripes; Sine stripe modulation Calculated by the following formula: (2) Sine stripe phase Calculated by the following formula: (3) in, It is the arctangent function, and its range is ; Step 3: The computer iterates through all phase-shifted fringe images pixel by pixel, sets the saturation grayscale threshold to 250, and discards any pixels exceeding the saturation threshold. All valid pixels are then phase-fused using the inverse variance weighting method. Using formula (6) and the complex amplitude principle, the equivalent calculation formula for inverse variance weighting is obtained: the sum of the squared modulation of each fringe image and the product of the real and imaginary parts of the phase is calculated and compared. Finally, the phase is calculated using arctanc expansion, i.e.: (6) Step 4: After traversing all pixels, obtain the wrapped phase, and then use the multi-frequency heterodyne method to obtain the unwrapped phase, thus obtaining a one-to-one correspondence between the phase and the coordinates; Step 5: Use stereo vision to find matching points and reconstruct the 3D point cloud data of the scanned object.

2. The multi-brightness stripe image fusion method based on complex amplitude as described in claim 1, characterized in that: In Step 2, the captured stripe pattern is adaptively generated into a sequence of projection brightness and exposure time combinations based on the dynamic range of the reflected light intensity from the object's surface. This sequence is then used to scan the object, resulting in multiple sets of phase-shifted stripe images. The standard deviation of each phase image is calculated using the following formula: (4) Where σ represents the standard deviation of ambient light, and the ambient light is assumed to be Gaussian noise with a mean of 0 and a standard deviation of σ.

3. The multi-brightness stripe image fusion method based on complex amplitude as described in claim 1, characterized in that: In Step 3, after the camera captures the multi-brightness stripe pattern reflected from the object surface by the projector, the strategy for stripe fusion is to first discard invalid pixels with gray values ​​exceeding the threshold; then, weighted fusion of gray values ​​is performed on all valid pixels, and the weight is the inverse of the variance of each pixel. Then, based on the complex amplitude formula, the equivalent calculation formula for the fused phase (6) is obtained.

Images