3D reconstruction method of high dynamic range reflective surfaces based on domain generalization

Abstract

The present invention belongs to the field of image processing technology and relates to a method for three-dimensional reconstruction of high dynamic range reflective surfaces based on domain generalization. The method comprises: determining overexposed and underexposed regions in the original image; extracting background intensity feature information from the long-exposure image using a UNet network, and obtaining dark and bright features through brightness estimation; generating a pseudo-brightness feature map based on the dark and bright features; estimating the darkness offset between the dark features of the original image and the pseudo-brightness feature map, and the brightness offset between the bright features and the pseudo-brightness feature map using extended deformable convolution; fusing the darkness offset and brightness offset with a fringe image to obtain a corrected fringe image, and performing three-dimensional reconstruction using the corrected fringe image. The present invention significantly improves the modulation distribution of the fringe image, significantly improves the quality of the fringe image under HDR conditions, and provides high-quality input for subsequent three-dimensional reconstruction.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a three-dimensional reconstruction method of a high dynamic range reflective surface based on domain generalization. Background Art

[0002] Fringe Projection Profilometry (FPP), a highly efficient optical 3D measurement technology, is widely used in reconstructing complex surface morphologies. By projecting a sinusoidal fringe pattern and extracting phase information, it accurately reconstructs the 3D shape of an object. FPP plays a vital role in numerous fields, including industrial inspection, reverse engineering, and biomedicine.

[0003] However, FPP faces significant challenges in High Dynamic Range (HDR) reflective surface scenes. Strong surface reflections can cause bright areas to be overexposed, resulting in a loss of sinusoidal information, while dark areas with insufficient illumination are underexposed, leading to a loss of detail. These issues severely impact the accuracy of the phase solution, ultimately resulting in incomplete 3D reconstructions that fail to meet the demands of practical applications.

[0004] Traditional solutions using developer are not only complex and time-consuming, but can also damage the surface, making them unsuitable for applications requiring high surface integrity. While multi-exposure methods can improve image quality to a certain extent, they require capturing multiple sets of fringe images under different exposure conditions. This cumbersome data preparation process, coupled with lengthy image fusion processing and low efficiency, makes them difficult to meet the demands of both high efficiency and high precision in practical applications.

[0005] In recent years, deep learning technology has rapidly developed and been introduced into all stages of FPP 3D reconstruction, achieving numerous breakthroughs from streak image enhancement to end-to-end reconstruction. However, existing deep learning methods still have many shortcomings when processing HDR streak images. For example, they struggle to accurately distinguish dark streak regions from shadowed regions, and cannot effectively restore the original characteristics of sinusoidal streaks when repairing overexposed areas, leading to periodic fluctuations in the reconstruction results. Furthermore, the availability and quality of training data also limit model performance. Generating realistic training data is difficult, and data generated by a single digital simulation system often differs from the real system, resulting in poor performance on real-world test datasets.

[0006] In summary, traditional physics-driven algorithms are gradually exposing their limitations in HDR reflective surface measurement, and existing deep learning methods need further optimization and improvement. Therefore, a new method is urgently needed to address these issues and achieve efficient and accurate 3D reconstruction of high dynamic range surfaces. Summary of the Invention

[0007] In order to solve the above technical problems, the present invention provides a three-dimensional reconstruction method of a high dynamic range reflective surface based on domain generalization, comprising:

[0008] Setting an overexposure threshold and an underexposure threshold, and determining an overexposed area and an underexposed area in the original image according to the overexposure threshold and the underexposure threshold;

[0009] The UNet network is used to extract background intensity feature information of overexposed and underexposed areas from long-exposure images, and dark and bright features are obtained through brightness estimation.

[0010] A pseudo-feature generator is used to generate a pseudo-brightness feature sequence based on dark features and bright features;

[0011] The extended deformable convolution is used to estimate the darkness offset between the dark features of the original image and the pseudo brightness feature map, as well as the brightness offset between the bright features and the pseudo brightness feature map;

[0012] The darkness offset and the brightness offset are fused with the fringe image to obtain a corrected fringe image, which is then used for three-dimensional reconstruction.

[0013] On the basis of the above technical solution, the present invention can also be improved as follows.

[0014] Furthermore, the UNet network is used to extract background intensity feature information of overexposed and underexposed areas from the long-exposure image, and the dark and bright features are obtained through brightness estimation, including:

[0015] Set the exposure time threshold, the long exposure time is greater than the exposure time threshold, and the short exposure time is less than the exposure time threshold. Use the four-step phase-shifted image sequences taken under long exposure time and short exposure time as the input of the UNet network;

[0016] Extract background intensity feature information of each four-step phase-shifted image sequence;

[0017] According to the background intensity characteristic information, the bright modulation feature sequence corresponding to the four-step phase-shift image sequence shot under long exposure time is calculated to obtain the dark feature, and the dark modulation feature sequence corresponding to the four-step phase-shift image sequence shot under short exposure time is calculated to obtain the bright feature.

[0018] Further, suppose is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, is a set of real numbers, and the four-step phase-shift image sequence taken under long exposure time is , , the four-step phase-shift image sequence captured under short exposure time is , , The corresponding background intensity feature information is , , The corresponding background intensity feature information is , , The corresponding bright modulation characteristic sequence is , , The corresponding dark modulation characteristic sequence is , , Represents the background intensity feature information extraction function in the UNet network, then:

[0019] ;

[0020] .

[0021] Furthermore, a pseudo-feature generator is used to generate a pseudo-brightness feature sequence based on the dark features and bright features, including: inputting a four-step phase-shift image sequence taken under long exposure time and short exposure time, dark features and bright features into the pseudo-feature generator, and using the preliminary constraints of the original image to guide the generation of the pseudo-brightness feature sequence.

[0022] Further, suppose is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, is a set of real numbers, and the four-step phase-shift image sequence taken under long exposure time is , , the four-step phase-shift image sequence captured under short exposure time is , , The corresponding bright modulation characteristic sequence is , , The corresponding dark modulation characteristic sequence is , , the pseudo brightness feature sequence is , , represents the pseudo brightness feature map, then the pseudo brightness feature sequence is expressed as:

[0023] .

[0024] Furthermore, the four-step phase-shifted image sequences, dark features, and bright features taken under long exposure time and short exposure time are spliced ​​in the channel dimension and input into the mapping network. The weights of the four-step phase-shifted image sequences, dark features, and bright features taken under long exposure time and short exposure time are calculated, and the weight of each channel is extracted from the output weight map. The weight of each channel is multiplied by the input corresponding to the channel, and the final pseudo-brightness feature sequence is obtained by weighted summation.

[0025] Further, suppose is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, is a set of real numbers. The bright modulation feature sequence corresponding to the four-step phase-shift image sequence taken under long exposure time is , , the dark modulation characteristic sequence corresponding to the four-step phase-shift image sequence taken under short exposure time is , , the pseudo brightness feature sequence is , , the first modulation distribution offset is , , the second modulation distribution offset is , , represents the modulation offset estimation mapping, and the calculation formulas for the first modulation distribution offset and the second modulation distribution offset are:

[0026] ;

[0027] .

[0028] Furthermore, when using extended deformable convolution to estimate dark offset and bright offset, let is a The coordinate set of the grid, express The coordinate index in , is the input feature of the convolution operation, is the original coordinate center of the convolution kernel, The convolution kernel The original coordinates of the sampling points, Indicates the The spatial offset of the sampling points, Indicates the The modulation offset of the sampling points is is the learning parameter of the convolution kernel, is the deformable convolution output of each pixel in the input image, then the deformable convolution calculation process is expressed as:

[0029] .

[0030] Furthermore, the darkness offset and the brightness offset are fused with the fringe image to obtain a modified fringe image, including: inputting the first modulation distribution offset and the second modulation distribution offset into the modulation intensity fusion module, the modulation intensity fusion module uses two independent convolution kernels to extract the spatial offset and the modulation offset, and obtains a modulation enhanced fringe image sequence; assuming that the pseudo brightness feature sequence is , is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, is the set of real numbers, , the first modulation distribution offset is , , the second modulation distribution offset is , , represents the modulation intensity fusion map, and the corrected fringe image sequence is ,but:

[0031] .

[0032] The beneficial effects of the present invention are as follows: the present invention utilizes a Unet network to extract intensity feature maps of overexposed and underexposed areas from input long-exposure and short-exposure images, and utilizes a pseudo-feature generator to generate a pseudo-luminance feature map based on these extracted dark features and bright features; designs a modulation offset estimation module and adopts an extended deformable convolution to estimate and correct the brightness offset and darkness offset between the bright feature map, the dark feature map, and the pseudo-luminance feature map, and generates an enhanced stripe image through feature fusion. The present invention significantly improves the modulation distribution of the stripe image, significantly improves the quality of the stripe image under HDR conditions, and provides high-quality input for subsequent three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the three-dimensional reconstruction method of a high dynamic range reflective surface based on domain generalization provided by the present invention;

[0034] Figure 2 This is the principle block diagram of modulation offset estimation and feature fusion;

[0035] Figure 3 This is the principle block diagram of the feature fusion module. DETAILED DESCRIPTION

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0037] As an example, Figure 1 As shown, to solve the above technical problems, this embodiment provides a three-dimensional reconstruction method of a high dynamic range reflective surface based on domain generalization, including:

[0038] Setting an overexposure threshold and an underexposure threshold, and determining an overexposed area and an underexposed area in the original image according to the overexposure threshold and the underexposure threshold;

[0039] The UNet network is used to extract background intensity feature information of overexposed and underexposed areas from long-exposure images, and dark and bright features are obtained through brightness estimation.

[0040] A pseudo-feature generator is used to generate a pseudo-brightness feature sequence based on dark features and bright features;

[0041] The extended deformable convolution is used to estimate the darkness offset between the dark features of the original image and the pseudo brightness feature map, as well as the brightness offset between the bright features and the pseudo brightness feature map;

[0042] The darkness offset and the brightness offset are fused with the fringe image to obtain a corrected fringe image, which is then used for three-dimensional reconstruction.

[0043] This method improves HDR stripe images, enabling efficient 3D reconstruction of high-dynamic-range surfaces. First, a Unet network is used to extract intensity feature maps of overexposed and underexposed regions from the input long- and short-exposure images. Subsequently, a pseudo-feature generator is used to generate a pseudo-luminance feature map based on these extracted dark and bright features. To correct for modulation offsets, a Modulation Offset Estimation Module (MOE) is designed, employing dilated deformable convolution to estimate and correct for the brightness and darkness offsets between the bright and dark feature maps, as well as the pseudo-luminance feature map. Furthermore, an innovative Modulation Intensity Fusion Module (MIF) is proposed to generate an enhanced stripe image through feature fusion. This method significantly improves the modulation distribution of the stripe image and provides high-quality input for subsequent 3D reconstruction. Experimental results show that this method achieves reconstruction results comparable to those of multi-exposure fusion methods with only a 0.0685s increase in computational time compared to single-exposure methods, significantly improving the quality of stripe images under HDR conditions.

[0044] As attached Figure 2 As shown in the network structure diagram, it can be seen that UNet is first used to extract feature information of overexposed and underexposed areas from the input long-exposure and short-exposure images respectively; then, the pseudo-feature generator generates a pseudo-luminance feature map based on these features; in order to correct the distribution offset between the pseudo-luminance feature map and the real stripe image, the present invention designs a modulation offset estimation module, which uses extended deformable convolution to estimate the offset between the bright feature map and the dark feature map as well as the pseudo-luminance feature map; finally, a fusion module is used to correct the distribution of the pseudo-luminance feature map in combination with the offset calculated above, thereby generating the final stripe image. Through these steps, the present invention significantly improves the modulation distribution and dynamic range of the stripe image, and provides a high-quality stripe image for three-dimensional reconstruction.

[0045] Optionally, use the UNet network to extract background intensity feature information of overexposed and underexposed areas from the long-exposure image, and obtain dark and bright features through brightness estimation, including:

[0046] Set the exposure time threshold, the long exposure time is greater than the exposure time threshold, and the short exposure time is less than the exposure time threshold. Use the four-step phase-shifted image sequences taken under long exposure time and short exposure time as the input of the UNet network;

[0047] Extract background intensity feature information of each four-step phase-shifted image sequence;

[0048] According to the background intensity characteristic information, the bright modulation feature sequence corresponding to the four-step phase-shift image sequence shot under long exposure time is calculated to obtain the dark feature, and the dark modulation feature sequence corresponding to the four-step phase-shift image sequence shot under short exposure time is calculated to obtain the bright feature.

[0049] The four-step phase-shift image sequences captured at short and long exposure times carry information on high-reflective and low-reflective surface areas, respectively.

[0050] Optional, set is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, is a set of real numbers, and the four-step phase-shift image sequence taken under long exposure time is , , the four-step phase-shift image sequence captured under short exposure time is , , The corresponding background intensity feature information is , , The corresponding background intensity feature information is , , The corresponding bright modulation characteristic sequence is , , The corresponding dark modulation characteristic sequence is , , Represents the background intensity feature information extraction function in the UNet network, then:

[0051] ;

[0052] .

[0053] Optionally, a pseudo-feature generator is used to generate a pseudo-brightness feature sequence based on dark features and bright features, including: inputting a four-step phase-shift image sequence, dark features, and bright features taken under long exposure time and short exposure time into the pseudo-feature generator, and using preliminary constraints of the original image to guide the generation of the pseudo-brightness feature sequence.

[0054] Inputting the four-step phase-shifted image sequence, dark features and bright features into the pseudo-feature generator can make the distribution of the enhanced fringe image as close as possible to the distribution of the real fringe image.

[0055] Optional, set is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, is a set of real numbers, and the four-step phase-shift image sequence taken under long exposure time is , , the four-step phase-shift image sequence captured under short exposure time is , , The corresponding bright modulation characteristic sequence is , , The corresponding dark modulation characteristic sequence is , , the pseudo brightness feature sequence is , , represents the pseudo brightness feature map, then the pseudo brightness feature sequence is expressed as:

[0056] .

[0057] Optionally, the four-step phase-shift image sequence, dark features, and bright features taken under long exposure time and short exposure time are spliced ​​in the channel dimension and input into the mapping network, the weights of the four-step phase-shift image sequence, dark features, and bright features taken under long exposure time and short exposure time are calculated, the weight of each channel is extracted from the output weight map, the weight of each channel is multiplied by the input corresponding to the channel, and the final pseudo-brightness feature sequence is obtained by weighted summation.

[0058] Optional, set is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, is a set of real numbers. The bright modulation feature sequence corresponding to the four-step phase-shift image sequence taken under long exposure time is , , the dark modulation characteristic sequence corresponding to the four-step phase-shift image sequence taken under short exposure time is , , the pseudo brightness feature sequence is , , the first modulation distribution offset is , , the second modulation distribution offset is , , represents the modulation offset estimation mapping, and the calculation formulas for the first modulation distribution offset and the second modulation distribution offset are:

[0059] ;

[0060] .

[0061] Because standard convolution uses a fixed-shape kernel, and most studies apply deformable convolution only in the pixel spatial domain, it is difficult to flexibly address local spatial variations in modulation information and variations in modulation intensity in stripe images. The modulation offset estimation module in this paper employs a deformable convolution extended to the modulation intensity space to more accurately model the modulation offset in the image. Specifically, this modulation offset estimation module introduces spatial offset and modulation offset into standard convolution, enabling the convolution kernel to adaptively adjust its position within the pixel region and the amplitude of the modulation intensity, effectively capturing the complex variations in modulation offset in stripe images.

[0062] As attached Figure 3 As shown in the figure, the modulation offset estimation module first connects the pseudo modulation intensity enhanced image sequence with the light modulation feature sequence and the dark modulation feature sequence and inputs them into The convolution kernel, then, uses a separate The convolution kernel extracts the spatial offset using a separate The convolution kernel extracts a modulation offset.

[0063] Optionally, when using extended deformable convolution to estimate dark offset and bright offset, set is a The coordinate set of the grid, express The coordinate index in , is the input feature of the convolution operation, is the original coordinate center of the convolution kernel, The convolution kernel The original coordinates of the sampling points, Indicates the The spatial offset of the sampling points, Indicates the The modulation offset of the sampling points is is the learning parameter of the convolution kernel, is the deformable convolution output of each pixel in the input image, then the deformable convolution calculation process is expressed as:

[0064] .

[0065] The modulation offset estimation module is used to extract modulation offset features in the stripe image caused by underexposure or overexposure. However, these features behave differently in different regions, making the fusion of global information crucial for generating high-quality enhanced stripe images. The feature fusion module combines these modulation offset features with the HDR stripe image to effectively correct the offset in these regions.

[0066] Optionally, the darkness offset and the brightness offset are fused with the fringe image to obtain a modified fringe image, including: inputting the first modulation distribution offset and the second modulation distribution offset into a modulation intensity fusion module, the modulation intensity fusion module uses two independent convolution kernels to extract the spatial offset and the modulation offset to obtain a modulation enhanced fringe image sequence; assuming that the pseudo brightness feature sequence is , is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, is the set of real numbers, , the first modulation distribution offset is , , the second modulation distribution offset is , , represents the modulation intensity fusion map, and the corrected fringe image sequence is ,but:

[0067] .

[0068] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A three-dimensional reconstruction method for high dynamic range reflective surfaces based on domain generalization, characterized in that: include: Setting an overexposure threshold and an underexposure threshold, and determining an overexposed area and an underexposed area in the original image according to the overexposure threshold and the underexposure threshold; The UNet network is used to extract background intensity feature information of overexposed and underexposed areas from long-exposure images, and dark and bright features are obtained through brightness estimation. A pseudo-feature generator is used to generate a pseudo-brightness feature sequence based on dark features and bright features; Using extended deformable convolution to estimate the darkness offset between the dark features of the original image and the pseudo brightness feature map, as well as the brightness offset between the bright features and the pseudo brightness feature map, includes: using the pseudo brightness feature sequence as reference information, using a modulation offset estimation module to calculate a first modulation distribution offset between the bright modulation feature sequence and the pseudo brightness feature sequence, and using a second modulation distribution offset between another dark modulation feature sequence and the pseudo brightness feature sequence; set up is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, is a set of real numbers. The bright modulation feature sequence corresponding to the four-step phase-shift image sequence taken under long exposure time is , , the dark modulation characteristic sequence corresponding to the four-step phase-shift image sequence taken under short exposure time is , , the pseudo brightness feature sequence is , , the first modulation distribution offset is , , the second modulation distribution offset is , , represents the modulation offset estimation mapping, and the calculation formulas for the first modulation distribution offset and the second modulation distribution offset are: ; ; When using extended deformable convolution to estimate dark offset and bright offset, set is a The coordinate set of the grid, express The coordinate index in , is the input feature of the convolution operation, is the original coordinate center of the convolution kernel, The convolution kernel The original coordinates of the sampling points, Indicates the The spatial offset of the sampling points, Indicates the The modulation offset of the sampling points is is the learning parameter of the convolution kernel, is the deformable convolution output of each pixel in the input image, then the deformable convolution calculation process is expressed as: ; The darkness offset and the brightness offset are fused with the fringe image to obtain a corrected fringe image, which is then used for three-dimensional reconstruction.

2. The method for 3D reconstruction of high dynamic range reflective surfaces based on domain generalization according to claim 1, characterized in that: The UNet network is used to extract background intensity feature information of overexposed and underexposed areas from long-exposure images, and dark and bright features are obtained through brightness estimation, including: Set the exposure time threshold, the long exposure time is greater than the exposure time threshold, and the short exposure time is less than the exposure time threshold. Use the four-step phase-shifted image sequences taken under long exposure time and short exposure time as the input of the UNet network; Extract background intensity feature information of each four-step phase-shifted image sequence; According to the background intensity characteristic information, the bright modulation feature sequence corresponding to the four-step phase-shift image sequence shot under long exposure time is calculated to obtain the dark feature, and the dark modulation feature sequence corresponding to the four-step phase-shift image sequence shot under short exposure time is calculated to obtain the bright feature.

3. The method for 3D reconstruction of high dynamic range reflective surfaces based on domain generalization according to claim 2, characterized in that: set up is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, is a set of real numbers, and the four-step phase-shift image sequence taken under long exposure time is , , the four-step phase-shift image sequence captured under short exposure time is , , The corresponding background intensity feature information is , , The corresponding background intensity feature information is , , The corresponding bright modulation characteristic sequence is , , The corresponding dark modulation characteristic sequence is , , Represents the background intensity feature information extraction function in the UNet network, then: ； 。 4. The method for 3D reconstruction of high dynamic range reflective surfaces based on domain generalization according to claim 1, characterized in that: A pseudo-feature generator is used to generate a pseudo-brightness feature sequence based on dark features and bright features, including: inputting a four-step phase-shifted image sequence taken under long exposure time and short exposure time, dark features and bright features into the pseudo-feature generator, and using the preliminary constraints of the original image to guide the generation of the pseudo-brightness feature sequence.

5. The method for 3D reconstruction of high dynamic range reflective surfaces based on domain generalization according to claim 1, characterized in that: set up is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, is a set of real numbers, and the four-step phase-shift image sequence taken under long exposure time is , , the four-step phase-shift image sequence captured under short exposure time is , , The corresponding bright modulation characteristic sequence is , , The corresponding dark modulation characteristic sequence is , , the pseudo brightness feature sequence is , , represents the pseudo brightness feature map, then the pseudo brightness feature sequence is expressed as: 。 6. The method for 3D reconstruction of high dynamic range reflective surfaces based on domain generalization according to claim 4, characterized in that: The four-step phase-shifted image sequences, dark features, and bright features taken under long exposure time and short exposure time are spliced ​​in the channel dimension and input into the mapping network. The weights of the four-step phase-shifted image sequences, dark features, and bright features taken under long exposure time and short exposure time are calculated. The weight of each channel is extracted from the output weight map, and the weight of each channel is multiplied by the input corresponding to the channel. The final pseudo-brightness feature sequence is obtained by weighted summation.

7. The method for 3D reconstruction of high dynamic range reflective surfaces based on domain generalization according to claim 1, characterized in that: The darkness offset and the brightness offset are fused with the fringe image to obtain a modified fringe image, including: inputting the first modulation distribution offset and the second modulation distribution offset into the modulation intensity fusion module, the modulation intensity fusion module uses two independent convolution kernels to extract the spatial offset and the modulation offset, and obtains a modulation enhanced fringe image sequence; assuming that the pseudo brightness feature sequence is , is the height of the four-step phase-shifted image, is the width of the four-step phase-shifted image, is the set of real numbers, , the first modulation distribution offset is , , the second modulation distribution offset is , , represents the modulation intensity fusion map, and the corrected fringe image sequence is ,but: 。

Images