Polarization-based high dynamic range surface three-dimensional reconstruction method

Abstract

For a high dynamic range surface, the invention provides a simple and effective three-dimensional reconstruction method, which combines the characteristics of non-overexposure areas in a polarization image and a non-polarization image, and aims to improve the reconstruction quality of a high dynamic range surface object. A non-overexposure area in a non-polarization picture generally contains complete stripe information and is less influenced by highlight interference. The optimal stripe regions of different polarization picture groups are extracted, then the incomplete and unclear stripe information in the non-polarization image is gradually replaced by the optimal stripe information in different polarization directions, and finally, the stripes in the overexposure region in the non-polarization image are replaced by the complete stripe information. In this way, global complete stripe information can be obtained. Experimental results show that when three-dimensional reconstruction is carried out on an object with a strong reflection surface, the negative influence of an overexposure area on absolute phase calculation and three-dimensional reconstruction can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of optical three-dimensional measurement, and in particular to the recovery of fringe information in an overexposed area during three-dimensional reconstruction of a surface object with a high dynamic range. Background Art

[0002] Given the advantages of high speed, non-contact, and simple hardware configuration, visual 3D shape measurement is widely used in different fields, such as defect detection, industrial parts quality control, and reverse engineering. Existing structured light methods are generally suitable for the topography measurement of diffuse reflective surfaces. When measuring high dynamic range (HDR) surfaces, it is difficult to balance the high dynamic range and low response areas in a single exposure due to the limitation of the camera's intensity response range. When reconstructing the 3D shape of objects on high dynamic range surfaces, the processing of overexposed areas has always been a challenge. Overexposure usually occurs in high dynamic range (HDR) scenes, resulting in the loss of details in certain areas of the image, which seriously affects the accuracy of the reconstruction results. Traditional reconstruction techniques often cannot effectively restore the geometric features of these areas, thereby limiting their applicability in practical applications. It is urgent to develop a set of methods for 3D reconstruction of objects on high dynamic range surfaces. Summary of the invention

[0003] The present invention proposes a simple method for reconstructing surface objects in a high dynamic range, which combines the features of non-highlight areas in polarized images and non-polarized images to overcome the problems of absolute phase calculation errors and point cloud missing caused by overexposed areas.

[0004] In order to achieve the above object, the technical solution of the present invention is implemented as follows: a priori guided restoration method for strong highlights in a fringe projection profilometry image comprises the following steps: S 1: Take a non-polarized image and several sets of polarized images using a calibrated system; S 2: Dephase the non-polarized image and identify the areas where dephase was unsuccessful; S 3: The brighter polarized image further identifies the unsuccessful dephased area based on the unsuccessful dephased area of ​​the non-polarized image; S 4: extracting the phase information of the darker polarized image in the region where the brighter polarized image is not successfully dephased. If the brighter polarized image is successfully dephased in the entire region where the non-polarized image is not successfully dephased, directly performing phase fusion; S 5: Fusion of phase information of the successfully dephased regions of the different polarization images into the overexposed regions of the non-polarization images; S 6: A method for calculating the absolute phase of the fused phase information and converting it into a three-dimensional point cloud.

[0005] A commonly used method is to use multiple exposures to obtain the stripe information of the overexposed area, while the present invention adds a polarizing plate in front of the camera and the projector, and obtains the stripe information of the overexposed area by rotating the polarizing plate. The method of obtaining images in the present invention is much more convenient than the method of obtaining images by multiple exposures.

[0006] The present invention only needs to determine the projection light intensity and camera exposure time once before acquiring a picture of an object, and can be implemented without complicated calculations and cumbersome operation steps.

[0007] The accuracy of the method proposed in the present invention is related to the accuracy of the system used. The higher the system accuracy, the higher the accuracy of the three-dimensional reconstruction result of the method proposed in the present invention. This method will not affect the accuracy of the original system.

[0008] Traditional multi-exposure fusion technology needs to rely on sophisticated and complex algorithms to fuse related phase information together, while the method proposed in this invention only needs to use 1∼2 order substitution functions to restore the phase information of the overexposed area.

[0009] Through the experimental process and results, it can be obtained that the total number of pictures required by the present invention is less than the number of pictures required by the popular multi-exposure fusion method. Although the method proposed in the present invention uses fewer pictures, the absolute phase map and three-dimensional point cloud map obtained by the method of the present invention show that this method can also restore the geometric features of the overexposed area. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flow chart of the prior-guided repair method for strong highlights in fringe projection profilometry images described in the present invention.

[0011] Figure 2 The non-polarized image has overexposed areas and the polarized image has non-overexposed parts of its area.

[0012] Figure 3 Four-step phase shift diagram.

[0013] Figure 4 Unpolarized image with fringe information and its calculated absolute phase map and 3D point cloud.

[0014] Figure 5 The general process of phase information fusion.

[0015] Figure 6 The absolute phase map and three-dimensional point cloud map obtained by the method proposed in the present invention.

[0016] Figure 7 The traditional method reconstructs the fusion area of ​​the present invention at different shooting angles and compares them. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] like Figure 1 As shown, a simple method for high dynamic range surface object reconstruction includes the following steps: S 1: Take a non-polarized image and several sets of polarized images using a calibrated system. The specific method is as follows; The object to be measured and the system calibrated by the present invention are determined to have corresponding positions on the test bench. After the positions are fixed, a sinusoidal fringe pattern is projected by a projector. The method for determining the light intensity is to project a blue fringe pattern onto the surface of the object to be measured. Under the premise that the camera exposure time remains unchanged (t=20000us), the criterion for judging the light intensity is that the fringe information of the non-overexposed area of ​​the object can be accurately and completely acquired by the camera. After determining the light intensity, a group of non-polarized fringe images are taken. Subsequently, under the premise that the camera exposure time, projection brightness and the positions of other objects remain unchanged, a polarizing plate is placed in front of the camera and the projector. When shooting a brighter group of polarized images, two polarization angles need to be 0. When shooting a darker group of polarized images, it is only necessary to rotate the polarizing plate carrier in front of the camera. The number of polarized image groups taken is determined by the fact that only the polarized group images are needed to complete the fringe information of the overexposed area of ​​the non-polarized image, such as Figure 2 As shown, the non-overexposed area is , and the overexposed area is domain.

[0019] S 2: Dephase the non-polarized image and identify the area where dephase was unsuccessful: S2.1: The present invention adopts a three-frequency four-step phase shift method for three-dimensional reconstruction, and the intensity of the projection pattern is shown in formula (1): (1) In the formula is the background intensity of the fringe image, is the modulation amplitude, is the phase value, n is the phase shift number, N is the number of phase shift steps, is the position of the projected image pixel. In the four-step phase shift, n=0, 1, 2, 3, and N=4, then the phase shift of the four fringe images is 0. , , ,like Figure 3 Its strength is shown in formula (2): (2) Based on the fringe projection image obtained by the camera, the following formula can be used to solve its related value, as shown in formula (3): (3) From formula (4) and formula (5), we can get , as shown in formula (4): (4) Solving the above formula, we can get The value range is arrive , unwrapping the wrapped phase can obtain the global unambiguous absolute phase. The phase unwrapping algorithm used in the present invention is the multi-frequency phase unwrapping method. The phase information of the three frequencies is obtained from the above formula: , , , that is, calculate the phase difference between the first group and the second group, as shown in formula (5): (5) Similarly, the phase difference between the second group and the third group is as shown in formula (6): (6) Combining formulas (5) and (6), the final phase difference can be obtained, as shown in formula (9): (7) The phase unwrapping algorithm increases the measurement range by reducing its signal-to-noise ratio, so the synthesized phase map Only used as a reference phase to assist phase unwrapping and synthesize wavelength With smaller wavelength A scaling factor is defined, which is applied to the phase at the beat frequency to determine the fringe order and finally obtain the final absolute phase , as shown in formula (8): (8) After calculation, the absolute phase image can be obtained. The metal plate of the present invention is tested and its non-polarized image is as follows: Figure 4 (a) shows the absolute phase diagram. Figure 4 As shown in (b), the final 3D point cloud is as follows Figure 4 As shown in (c), the absolute phase diagram can be used to determine the range of phase resolution failure in the overexposed area.

[0020] S 3: The brighter polarized image further identifies the unsuccessful phase separation area based on the unsuccessful phase separation area of ​​the non-polarized image, and the specific method is as follows; Similar to step S2, the brighter polarized image is obtained to further identify the unsuccessful dephased area based on the unsuccessful dephased area of ​​the non-polarized image. Because the overexposed area will become smaller after the addition of the polarizer, this process can obtain correct and complete fringe information of a part of the overexposed area of ​​the non-polarized image.

[0021] S 4: extract the phase information of the darker polarized image in the area where the brighter polarized image is not successfully dephased. If the brighter polarized image is successfully dephased in the entire area where the non-polarized image is not successfully dephased, directly perform phase fusion. The specific method is as follows; Similar to step S2, the fringe information of the unsuccessful phase-decoding area of ​​the brighter polarized image is obtained from the darker polarized image, and then the phase information of the successfully phase-decoded area is fused together. If the phase information of the overexposed area of ​​the non-polarized image has been completed in the brighter polarized image, phase fusion is directly performed. The specific fusion method will be specifically described in S5.

[0022] S 5: The phase information of the successfully dephased area of ​​the different polarization images is fused into the overexposed area of ​​the non-polarization image. The specific method is as follows; The image captured by the camera cannot intuitively judge the range of its accurate overexposed area and cannot directly determine whether the stripe information of the captured image is missing. The overexposed areas of images captured at different frequencies and different phase shifts will be slightly different. Therefore, the best method is to determine the unsuccessful phase resolution area from the final phase resolution result. This method is also used in the brighter group of polarized images, but the judgment area is the unsuccessful phase resolution area of ​​the image taken without a polarizer. The remaining darker polarized images select smaller areas with successful phase resolution in a larger area based on the brighter group of polarized images, until there is no unsuccessful phase resolution area in the object area under test. Finally, all the successfully phase resolved areas are fused into a group of images. Phase fusion is the process of replacing the phase information of the overexposed area step by step. For a better understanding, it is assumed that the final phase incorrect area of ​​a non-polarized image group of an object is , the final phase incorrect area of ​​the brighter polarization image group is , the darker polarization image group needs to be in the region The final phase information is correct, and the darker polarization image group area The phase information of the brighter polarization image group is replaced by the phase information of the brighter polarization image group. Finally, the brighter polarization image group area is fused. The phase information of the non-polarized image group replaces the phase information of this area, so that a global complete phase information can be obtained. Indicates that the first image is at position The phase information of represents the pixel value of the second image at the same position, then the pixel information of an area is replaced by a formula, as shown in formula 11: (9) in, is a repair model, the area Indicates an area within the internal image, so that the phase information of the first image pixel can replace the phase information of the second image pixel. The logic of this formula used in the present invention is to apply this formula twice to replace the phase information of the overexposed area with the correct phase information. The general process is as follows Figure 5 shown.

[0023] S 6: Convert the final phase information of the global phase solution successfully obtained by fusion into a three-dimensional point cloud.

[0024] The fused phase information is passed through S2 steps to obtain the absolute phase of the global phase solution successfully as follows: Figure 6 As shown in (a), the absolute phase is obtained by the three-frequency four-step method, from which the phase value of each pixel can be obtained. Then, the depth value of each pixel is calculated by the known camera parameters. Then, the pixel coordinates are converted into three-dimensional world coordinates according to the intrinsic parameters of the camera. Finally, the three-dimensional point cloud image is obtained by the correct absolute phase information, as shown in Figure 6 As shown in (b), in order to verify that the method proposed in the present invention does not affect the accuracy of the original system, the present invention uses the traditional method to reconstruct the fusion area of ​​the present invention at different shooting angles and compare them, as shown in FIG. Figure 7 As shown, Figure 7 (a) is a point cloud image obtained by reconstructing the fusion area of ​​the present invention using the traditional method, Figure 7 (b) is a point cloud image obtained by the method of the present invention, Figure 7 (c) highlights the area without stripe information, which corresponds to the area fused by the present invention, Figure 7 (d) is a comparison of the same area of ​​the two point cloud images. Due to the different shooting angles, the details are slightly different, but the reconstruction results of the more obvious scratches on the metal plate show that the method proposed in the present invention has almost no impact on the accuracy of the original system.

Claims

1. A high dynamic range surface three-dimensional reconstruction method based on polarization, characterized in that: The following steps are involved: S 1: Take a non-polarized image and several sets of polarized images using a calibrated system; S 2: Dephase the non-polarized image and identify the areas where dephase was unsuccessful; S 3: The brighter polarized image further identifies the unsuccessful dephased area based on the unsuccessful dephased area of ​​the non-polarized image; S 4: extracting the phase information of the darker polarized image in the region where the brighter polarized image is not successfully dephased. If the brighter polarized image is successfully dephased in the entire region where the non-polarized image is not successfully dephased, directly performing phase fusion; S 5: Fusion of phase information of the successfully dephased regions of the different polarization images into the overexposed regions of the non-polarization images; S 6: A method for calculating the absolute phase of the fused phase information and converting it into a three-dimensional point cloud.

2. The method for high dynamic range surface three-dimensional reconstruction based on polarization according to claim 1, characterized in that: The method of taking non-polarized pictures and several groups of polarized pictures by a calibrated system is: The object to be measured and the system calibrated by the present invention are determined to have corresponding positions on the test bench. After the positions are fixed, a sinusoidal fringe pattern is projected by a projector. The method for determining the light intensity is to project a blue fringe pattern onto the surface of the object to be measured. Under the premise that the camera exposure time remains unchanged, the criterion for judging the light intensity is that the fringe information of the non-overexposed area of ​​the object can be accurately and completely acquired by the camera. After determining the light intensity, a group of non-polarized fringe images are taken. Subsequently, under the premise that the camera exposure time, projection brightness and the positions of other objects remain unchanged, a polarizing plate is placed in front of the camera and the projector. When shooting a brighter group of polarized images, two polarization angles need to be 0. When shooting a darker group of polarized images, it is only necessary to rotate the polarizing plate carrier in front of the camera. The number of polarized image groups taken is determined by the fact that only the polarized group images are needed to complete the fringe information of the overexposed area of ​​the non-polarized image. Generally, the total number of image groups is 2-3 groups.

3. The method for high dynamic range surface three-dimensional reconstruction based on polarization according to claim 1, characterized in that: The method for performing phase separation on the non-polarization image and identifying the region where phase separation is unsuccessful is as follows: (1) The present invention adopts a three-frequency four-step phase shift method for three-dimensional reconstruction. In the fringe projection profile measurement system, the generated image containing sinusoidal fringe information is imported into the projector, and then the projector grating pattern is projected onto the surface of the measured object and the fringe information on the surface of the measured object is obtained through the camera. Finally, the data is processed through the MATLAB program to obtain the phase information; (2) After obtaining the phase information, the wrapped phase is unwrapped so that the global unambiguous absolute phase can be obtained. The phase unwrapping algorithm adopted by the present invention is a multi-frequency phase unwrapping method, and the phase information of the three frequencies is obtained as,,,, that is, the phase difference between the first group and the second group and the phase difference between the second group and the third group are calculated. Then, the final phase difference can be obtained from these two phase differences. The phase unwrapping algorithm increases the measurement range by reducing its signal-to-noise ratio, so the synthesized final phase difference is only used as a reference phase to assist phase unwrapping. The synthetic wavelength and the smaller wavelength define a proportional factor, which is applied to the phase at the beat frequency to determine the fringe order, and finally the final absolute phase is obtained; (3) When analyzing the phase deconvolution results of non-polarized images, overexposed areas usually lead to loss or distortion of phase information, thus affecting the correctness of phase deconvolution. In these areas, since the image brightness exceeds the range that the sensor can capture, the phase value may not be accurately obtained, which will eventually lead to errors in its absolute phase. The absolute phase map can be used to determine the areas in the image where phase deconvolution is unsuccessful.

4. The method for high dynamic range surface three-dimensional reconstruction based on polarization according to claim 1, characterized in that: The method for further identifying the unsuccessful phase separation area of ​​the brighter polarized image based on the unsuccessful phase separation area of ​​the non-polarized image is: Similar to step 3, the brighter polarized image is obtained to further identify the unsuccessful dephased area based on the unsuccessful dephased area of ​​the non-polarized image. Because the overexposed area will become smaller after the addition of the polarizer, this process can obtain correct and complete fringe information of a part of the overexposed area of ​​the non-polarized image.

5. The method for high dynamic range surface three-dimensional reconstruction based on polarization according to claim 1, characterized in that: The method of extracting the phase information of the darker polarized image in the region where the brighter polarized image is not successfully dephased, and directly merging the phases if the brighter polarized image is successfully dephased in the entire region where the non-polarized image is not successfully dephased is: Similar to step 3, the fringe information of the unsuccessful phase-decoding area of ​​the brighter polarized image is obtained from the darker polarized image, and then the phase information of the successfully phase-decoded area is fused together. If the phase information of the overexposed area of ​​the non-polarized image has been completed in the brighter polarized image, phase fusion is directly performed. The specific fusion method will be described in detail in step 6.

6. The method for high dynamic range surface three-dimensional reconstruction based on polarization according to claim 1, characterized in that: The method for fusing the phase information of the successfully dephased area of ​​the different polarization images into the overexposed area of ​​the non-polarization image is: The method proposed in the present invention can be realized by simply replacing the phase information of the corresponding area of ​​the non-polarized image with the phase information of the corresponding area of ​​the polarized image when performing phase fusion. Indicates that the first image is at position The phase information of Represents the pixel value of the second image at the same position, then the pixel information of an area is replaced by a formula, as shown in Formula 1: （1） in, is a repair model, the area Represents an area within the image, so that the phase information of the first image pixel can replace the phase information of the second image pixel. The logic of this formula used in the present invention is to apply this formula twice to replace the phase information of the overexposed area with the correct phase information, thereby obtaining complete phase information.

7. The method for high dynamic range surface three-dimensional reconstruction based on polarization according to claim 1, characterized in that: The method of calculating the absolute phase of the fused phase information and converting it into a three-dimensional point cloud is: The method for solving the absolute phase is the same as step 3. The absolute phase is obtained by the three-frequency four-step method, from which the phase value of each pixel can be obtained, and then the depth value of each pixel is calculated by the known camera parameters, and then the pixel coordinates are converted into three-dimensional world coordinates according to the intrinsic parameters of the camera, and finally a three-dimensional point cloud map is generated. In order to verify that the method proposed in the present invention will not affect the accuracy of the original system, the present invention uses traditional methods to reconstruct the fusion area of ​​the present invention at different shooting angles and compares them.

Images