A three-dimensional reconstruction method based on multi-dimensional information fusion and convolutional neural network

Abstract

The application provides a three-dimensional reconstruction method based on multi-dimensional information fusion and a convolutional neural network, and comprises the following steps: acquiring polarized sub-images of a measured object under four transmission polarization directions and including multiple detection spectral bands through a multispectral polarization camera; performing image registration and joint denoising on all images; calculating polarization parameters of the measured object, combining spectral information and polarization information to calculate an azimuth angle and a zenith angle of the measured object; and acquiring surface normal vector component three-dimensional data of the measured object through a surface normal reconstruction model constructed based on a convolutional neural network and with the aid of physical prior knowledge. The method provided by the application can effectively restore the texture details of the surface of the measured object, can realize fast three-dimensional information acquisition of a measured object with a surface simultaneously having a diffuse reflection and a specular reflection characteristic in a complex indoor or outdoor scene under illumination conditions, and is helpful to expand the application range of the polarization three-dimensional reconstruction technology.

Description

Technical Field

[0001] This invention relates to a three-dimensional reconstruction method based on multi-dimensional information fusion and convolutional neural networks, specifically a three-dimensional reconstruction method based on monocular polarization three-dimensional reconstruction, and assisted by spectral information analysis, image processing and convolutional neural networks. Background Technology

[0002] Multispectral imaging, developed in the 1960s, is a detection technique used to acquire spatial image information of a measured object in different spectral bands. Polarization imaging, on the other hand, is an emerging approach for acquiring the intrinsic polarization information of an object's surface. Compared to traditional RGB cameras, which can only acquire basic, rough spatial information such as the object's size, shape, and relative position, multispectral imaging can obtain much richer information about the object: the spectral information acquired by multispectral imaging can be used to infer the object's surface composition and content; polarization information can be used to infer finer information such as the object's texture, roughness, and refractive index. Because this information is often indispensable in many fields, multispectral imaging has been widely applied in military, remote sensing, and industrial manufacturing. This type of detection technology is also widely used in 3D reconstruction, i.e., recovering the 3D information of a measured object from a 2D image to construct a 3D model.

[0003] Polarization is an important inherent characteristic of light waves, referring to the asymmetry of the vibration direction of the light wave vector relative to its propagation direction during light propagation. When light waves are reflected or refracted at the surface of an object, their polarization characteristics often change. This change is generally determined by a combination of factors such as the refractive index and morphological features of the object's surface material. Therefore, polarization information actually provides an effective detection dimension for 3D reconstruction, that is, the task of restoring the surface information of the measured object. Specifically, the surface morphology of the object can be calculated by reverse engineering the polarization characteristics of reflected light. This helps to improve the speed and accuracy of 3D reconstruction compared to traditional methods, while reducing the requirements for the number of devices and scene setup.

[0004] In the 21st century, researchers developed a systematic polarization 3D reconstruction scheme based on the aforementioned theories. Most current polarization 3D reconstruction techniques determine the surface morphology of an object by analyzing the state of polarization degree and polarization angle. Generally, polarization 3D reconstruction technology has low requirements for imaging equipment and shooting scenes. A conventional polarization 3D reconstruction process only requires a single real-time polarization imaging camera to perform a single image of the object being measured, obtaining four polariton images of the same object. The surface information of the object reconstructed through polarization 3D reconstruction often contains relatively complete detailed texture features, resulting in better reconstruction effects for objects with rich surface details or fine structures.

[0005] To obtain an accurate model of the surface of the object being measured, polarization-based 3D reconstruction technology first acquires the normal direction vectors at key points on the surface of the object, and then integrates these vectors to form a 3D model. The process of acquiring the normal direction vectors involves two dimensions: the reconstruction of the azimuth angle of the normal vectors and the reconstruction of the zenith angle of the normal vectors; each of these faces different challenges in its respective reconstruction process.

[0006] Currently, there is a relatively fixed scheme for reconstructing the azimuth angle of the normal vector of a measured object. Generally, the reconstruction of the normal azimuth angle is based on Malus's law, that is, by fitting the polarization angle-light intensity curve through light intensity information captured from multiple different polarization directions, the azimuth angle can be recovered. However, the principle of this method determines that the azimuth angle obtained by this method will inevitably produce a 180° ambiguity. Therefore, it is necessary to rely on data in dimensions other than polarization obtained from images or other sources to correct the azimuth angle. Most existing polarization 3D reconstruction schemes introduce traditional 3D reconstruction methods to assist in correcting the azimuth angle; some studies also achieve this goal through global least squares optimization or by introducing convolutional neural networks for processing.

[0007] In the reconstruction of the zenith angle of the normal to a measured object, most current studies choose to reconstruct only the diffuse reflection region of the object's surface. Since the relationship between the zenith angle of the normal and the degree of polarization of the outgoing light is a one-to-one mapping function on a diffuse reflection surface, zenith angle reconstruction on a diffuse reflection surface is relatively simple, given that the polarization angles have already been calculated from images acquired in four polarization directions. However, this is not the case for zenith angle reconstruction on specular reflection surfaces. In the visible spectrum, the degree of polarization of outgoing light after a specular reflection often corresponds to two possible zenith angles, making zenith angle reconstruction for specular reflection regions very difficult without introducing other information. To address this problem, some studies choose to normalize and fill a small number of specular reflection surfaces in the image as diffuse reflection surfaces to make it an effective method for reconstructing the normal to diffuse reflection surfaces; while other studies avoid specular reflection regions on the object surface by constructing a specialized measurement environment. However, if more general detection scenarios are considered, the object surface to be detected often has both specular and diffuse reflection regions, exhibiting complex surface reflection characteristics. Therefore, most of the above-mentioned solutions are ineffective in such detection scenarios.

[0008] To address this problem, this invention proposes a three-dimensional reconstruction method based on multi-dimensional information fusion and convolutional neural networks. This method is applicable to the three-dimensional reconstruction of small and medium-sized objects with complex surface reflection characteristics and can achieve three-dimensional reconstruction of the surface morphology of the measured object under various different scenarios, including outdoor natural light environment and indoor light source illumination. Summary of the Invention

[0009] The problem addressed by this invention is that traditional 3D reconstruction methods for small to medium-sized objects often require sophisticated measurement environments and advanced equipment, while polarization-based 3D reconstruction methods are largely ineffective in handling large specular reflection areas on the surface of the object in such detection scenarios. This invention combines the robustness of polarization-based 3D reconstruction technology to scene setup and detection environments, and introduces spectral dimension information to solve the aforementioned problem of zenith angle reconstruction in specular reflection areas, thus expanding the measurement range of polarization-based 3D reconstruction technology.

[0010] To achieve the above-mentioned objectives, the present invention employs the following technical solution:

[0011] Step 1: Using a multispectral polarization imaging detection system, extract polariton images of the object under test in four polarization directions (0°, 45°, 90°, and 135°) under multiple detection spectral bands.

[0012] Step 2.1: To address the small-scale translational shift caused by filter switching and minor camera displacement during shooting, a phase correlation registration method is adopted. This method involves using an image with a transmission polarization direction of 0° across the entire spectrum as the registration reference. The cross-power spectrum between each image and the reference image is calculated, and then an inverse Fourier transform is performed on the cross-power spectrum. The peak value of the resulting pulse function is the translational shift.

[0013] Step 2.2: To remove noise from the polarization image while preserving the structural consistency between each polarization channel, a joint guided filtering algorithm is used to remove noise. That is, for each spectral dimension, an image with a transmission polarization direction of 0° is used as a guide image, and a guided filter is applied to all polarization sub-images, including the image itself, to remove noise.

[0014] Step 3: Under each detection spectrum, calculate the degree of polarization and polarization angle of the surface of the object under test by using polariton images under four polarization directions of 0°, 45°, 90° and 135° respectively.

[0015] Step 4.1: Assuming that only diffuse reflection occurs on the surface of the object being measured, under this assumption, the zenith angle of the normal to the surface of the object can be obtained by substituting the average degree of linear polarization into the formula.

[0016]

[0017] The calculation yields the following result: In the formula, DoLP is the degree of polarization of the surface of the object being measured, n is the refractive index of the surface material of the object being measured, and θ is the zenith angle of the normal to the surface of the object being measured, and the same applies below.

[0018] Step 4.2, assuming that only specular reflection occurs on the surface of the object being measured, under this assumption, based on the wavelength-polarization data from multiple detection spectrum bands calculated in Step 2, the slope k of the fitted straight line corresponding to the wavelength-polarization data of each pixel in the image can be calculated.

[0019] Step 4.3: Using the wavelength-polarization data of any spectral band, the formula can be used to...

[0020]

[0021] Calculate two possible values ​​for the zenith angle of the normal to the surface of the object being measured: θ1 < θ2. If the slope k obtained in step 3.2 is greater than 0, then θ2 can be determined as the true zenith angle of the normal; otherwise, θ1 can be determined as the true zenith angle of the normal.

[0022] Step 4.4: Using the polarization images obtained in Step 1 at the four polarization angles, fit the polarization angle-light intensity curve of the measured object surface according to Malus's law, and then use the formula...

[0023]

[0024] Calculate the azimuth angle of the normal to the surface of the object being measured; where, I max with I min These represent the maximum and minimum light intensity values ​​in the curves above. The azimuth angle is the angle between the direction of vibration of the incident linearly polarized light and the transmission axis of the polarizer; the azimuth angle here has a 180° ambiguity.

[0025] Step 5: Encode the full-spectrum polariton image obtained in Step 1 and the two sets of surface normals of the measured object calculated in Step 3 into input data. Input this data into a surface normal reconstruction model based on a convolutional neural network and aided by prior physical knowledge. Its main modules include:

[0026] (1) A Res2Net encoding module based on hierarchical residual links;

[0027] (2) Spatially-Adaptive Normalization (SPADE) module;

[0028] The model outputs the predicted surface normal vector components of the measured object, which serve as the final 3D data.

[0029] The 3D reconstruction scheme for small and medium-sized objects with diffuse and specular reflection features provided by this invention has the following advantages compared with other existing methods:

[0030] 1. By introducing spectral dimension information, the difficulty of reconstructing the zenith angle of the normal to the specular reflection region of the measured object's surface is effectively solved by traditional polarization-based 3D reconstruction schemes. This enables the detection scheme of the present invention to achieve 3D reconstruction of the surface morphology of the measured object under various scenarios, including outdoor natural light environments and indoor light source illumination.

[0031] 2. This invention reconstructs the surface normal parameters of the measured object using a convolutional neural network constructed with physical priors. Compared to traditional convolutional neural networks used for 3D reconstruction, this network structure can effectively enhance the model's ability to acquire multi-scale features under different receptive fields, improve the model's accuracy in solving the azimuth binarity problem, and ensure that the spatial location information contained in the original image is not lost during the reconstruction process.

[0032] 3. By introducing polarization information as an important dimension for reconstruction, the 3D model reconstructed using this 3D reconstruction scheme has a good reconstruction effect on the detailed areas of the measured object, and can effectively restore the texture and uneven areas of the surface of the measured object.

[0033] 4. When performing 3D reconstruction using the method proposed in this invention, the requirements for photographic equipment are lower than those for traditional 3D reconstruction methods, and there is no need to set up a special measurement environment for this reconstruction work. Attached Figure Description

[0034] Appendix Figure 1 This is a data processing flowchart of the present invention.

[0035] Appendix Figure 2 This is a flowchart for reconstructing the zenith angle of the surface normal of the measured object. Appendix Figure 3 The present invention proposes a network structure for a polarization three-dimensional reconstruction model. Detailed Implementation

[0036] To more clearly demonstrate the specific technical approach of the three-dimensional reconstruction scheme mentioned in this invention, a more detailed explanation is provided for the parts not fully described in the invention description. The specific method is as follows:

[0037] 1. The data processing flow of the entire method is as follows: Figure 1As shown in the diagram, the polarization-spectrum-plane integrated data shown in the process is acquired by a multispectral polarization imager. Using a focal plane polarization imaging system, the aforementioned multispectral polarization imager can simultaneously acquire polariton images of the object under test at four different transmission polarization directions: 0°, 45°, 90°, and 135°, during a single imaging process. Furthermore, by rotating a rotating filter wheel equipped with filters, polarization data of the object under test in different incident spectral bands can be acquired. The imager involved in this invention can acquire corresponding polarization data in seven spectral bands within the visible light (441nm) to near-infrared (980nm) range, while also reserving a filter wheel slot without a filter for acquiring full-spectrum polarization data, which is sufficient for practical three-dimensional reconstruction work.

[0038] 2. In the image preprocessing stage, the first step is to perform translation registration on 8×4=32 input images. The main purpose of this registration process is to eliminate small-scale translational shifts caused by filter switching and minor camera displacements during shooting. In this scenario, the phase correlation method is used, with the image having a transmission polarization direction of 0° across the entire spectrum as the registration reference, and translation registration is performed on all images. That is, their cross-power spectrum is calculated:

[0039]

[0040] In the formula, F1 and F2 are the power spectra obtained after Fourier transform of the registration reference image and the image to be registered, respectively. Then, the peak position (dx, dy) of the impulse function after performing an inverse Fourier transform on the calculated result is the distance the image to be registered needs to be shifted.

[0041] After translation and registration, considering the linear relationship between the grayscale data of the four polarization images in each band, a guided filtering algorithm is used. The image with a transmission polarization direction of 0° in each band is used as the guide image. Guided filtering is applied to the four images, including the guide image itself, which involves calculating the optimal coefficients that minimize the cost function.

[0042]

[0043] The final output image result is

[0044]

[0045] In the formula, , , These represent the brightness values ​​at pixel i in the guide image, the image to be filtered, and the output image, respectively. Regularization terms to control the smoothness This represents the total number of pixels within the filtering window. , These are the mean and variance of the guiding image within the window, respectively. This represents the mean value of the image to be filtered within the filtering window.

[0046] 3. Based on a set of polariton images captured in each spectral band, a simplified Stokes vector can be calculated for each pixel. :

[0047]

[0048] From the Stokes vectors above, the degree of linear polarization (DoLP) and the angle of polarization (AoP) in the image can be calculated:

[0049]

[0050]

[0051] In this context, the degree of linear polarization is the ratio of the intensity of linearly polarized light to the total intensity, and the polarization angle represents the vibration direction of the linearly polarized portion of the light. Based on these obtained polarization parameters, the zenith angle and azimuth angle of the normal to the surface of the object under test can be reconstructed.

[0052] 4. Calculate the zenith angle of the normal based on polarization parameters. This invention first assumes that the reflection of the surface of the object being measured falls into two categories: diffuse reflection and specular reflection. Based on these assumptions, the zenith angle of the normal to the surface of the object is calculated. The overall data processing flow for this stage is as follows: Figure 2 As shown.

[0053] When light undergoes specular reflection at the surface of the object being measured, the degree of linear polarization of the outgoing light is:

[0054]

[0055] At this point, given the refractive index of the object being measured and the angle of incidence, a single polarization value can be uniquely determined. However, if only polarization data is obtained, attempting to reconstruct the zenith angle from the captured polarization data will result in two possible incident zenith angles.

[0056] When light undergoes diffuse reflection at the surface of the object being measured, the degree of linear polarization of the outgoing light is:

[0057]

[0058] At this point, on the diffuse reflection surface, there is a one-to-one correspondence between the degree of polarization and the incident zenith angle. Therefore, the zenith angle of the surface normal of the measured object can be uniquely determined by the degree of polarization.

[0059] To address the zenith angle ambiguity under the specular reflection assumption, and considering the variation in refractive index of the object's surface material across different incident spectral bands, this invention utilizes multiple sets of polarization degree data from different spectral bands to disambiguate the zenith angle of the object's surface normal under the specular reflection assumption.

[0060] The polarization data obtained in each spectral band are calculated to obtain the corresponding linear polarization degree, forming a set of wavelength-polarization degree data. Then, two different zenith angle values ​​θ1 < θ2 are calculated from the polarization degree value of any spectral band. Finally, the slope of the wavelength-polarization degree fitting curve is determined. If the slope obtained by fitting is k > 0, then θ2 is the true zenith angle value; otherwise, θ1 is selected as the true zenith angle value.

[0061] 5. Calculate the azimuth angle of the normal based on the polarization parameters. This invention employs a common method based on Malus's law:

[0062]

[0063] In the formula, Let be the initial light intensity of the beam before it passes through the polarizer. Let be the angle between the direction of vibration of the incident linearly polarized light and the transmission axis of the polarizer. Considering that during the change of the polarization direction of the polarizer, the intensity of the polarized light is equal to the difference between the maximum and minimum light intensities captured by the camera during this process, therefore, we have...

[0064]

[0065] In the formula, This is the azimuth angle of the normal that we are looking for. At this point, considering...

[0066]

[0067] If the azimuth angle is recovered using the light intensity information under each polarization direction, the resulting azimuth angle will have a 180° ambiguity, so this problem needs to be solved in subsequent steps.

[0068] 6. After reconstructing the zenith angle and azimuth angle of the surface normal of the measured object using physical methods, for each pixel in the image, this invention reconstructs the three-dimensional information of the surface of the measured object using a surface normal reconstruction model based on a convolutional neural network and aided by prior physical knowledge. The overall structure of this model is as follows: Figure 3 As shown.

[0069] The model takes four polariton images across the full spectrum and two sets of surface normal angle parameters of the measured object calculated in steps 4 and 5 as input, and outputs 3-dimensional surface normal vector component data. This model is based on the classic U-Net convolutional neural network architecture, specifically including:

[0070] (1) A module that segments and reassembles the original image. Upon input, this module crops the original 2048×2448 image into sub-images with sides of 256 pixels, and introduces an additional 32-pixel context region in each side direction. These sub-images are then integrated and packaged before being input into a convolutional neural network. Upon output, this module weights the context regions of the output sub-image according to their distance from the original sub-image region, and then combines this weighted weighted sum with the original pixels to which the context region belongs, resulting in a complete output.

[0071] (2) Hierarchical Residual Link Downsampling Module (Res2NetDownSample). This module contains a 1×1 convolutional block that outputs a feature map, three 3×3 convolutional blocks that extract features from the grouped feature data and corresponding feature fusion modules, a 1×1 convolutional module that integrates the feature map and restores the original image channel count, and an average pooling-1×1 convolutional module for constructing residual links, which enhances the model's ability to acquire multi-scale features under different receptive fields.

[0072] (3) A Spatially-Adaptive Normalization (SPADE) and Skip Connection upsampling module (SpecialUpSample) is introduced. Skip Connection is implemented by concatenating the feature map generated after the downsampling stage (with the same size as the input image) with the input feature map, and then performing a single convolutional upsampling operation with kernel_size=2 and stride=2. The feature map after skip connection is normalized by the SPADE module, then features are extracted through a single 3×3 convolutional block, and the residuals are normalized again, followed by a 3×3 convolution and concatenation. Here, the SPADE module extracts the scaling parameter α and bias parameter β from the spatial information of the original image by downsampling the original polariton image and then performing three 3×3 convolutions, and then using the formula...

[0073]

[0074] Correct the spatial information of the input module image.

[0075] (4) Normal Direction Component Restoration Module (FinalLayer). This module extracts features and normalizes them through a convolution with kernel_size=3 and stride=2, and then uses a 1×1 convolution to restore the 64-channel feature information into the normal direction component information. Finally, a cross-channel normalization is performed to standardize the output data into the standard form of 3D surface normal vector component data.

[0076] The surface normal reconstruction model described above can obtain accurate normal vector data of the surface of the object being measured while correcting the azimuth data.

[0077] Compared to traditional deep learning approaches, physics-based deep learning solutions introduce prior physical data as an aid. This allows the constructed surface normal reconstruction model to acquire 3D information of the object from the raw data while maintaining the general physical laws governing the polarization information and surface normal vector direction parameters during the reconstruction process—the relevant formulas involved in steps 4 and 5. This approach not only addresses the azimuth binarity problem but also calculates the mixing of different types of reflected light on the object's surface during the learning process, thus resolving the issue of mixed reflective regions. The introduction of the Res2Net module enhances the model's ability to acquire multi-scale features within different receptive fields, effectively improving the model's accuracy in solving the azimuth binarity problem. The SpecialUpSample module effectively mitigates the gradient vanishing problem that may occur in high-depth convolutional networks, addressing the issue that traditional normalization might wash away spatial and positional information in the input image.

[0078] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention. Contents not described in detail in this specification are prior art known to those skilled in the art.

Claims

1. A three-dimensional reconstruction method based on multi-dimensional information fusion and convolutional neural networks, characterized in that: Step 1: Using a multispectral polarization imaging detection system, extract polariton images of the object under test in four polarization directions (0°, 45°, 90°, and 135°) in the full spectrum and multiple detection spectrum bands. Step 2: Perform image preprocessing on the acquired images, namely, perform image registration and joint denoising on each detection spectral band separately; Step 3: Under each detection spectrum, calculate the degree of polarization and polarization angle of the surface of the object under test by using polariton images under four polarization directions of 0°, 45°, 90° and 135° respectively. Step 4: Assuming that the surface of the object being measured only undergoes diffuse reflection or only specular reflection, the spectral information is used as important information to determine the zenith angle of the object under the specular reflection assumption from the surface polarization information. The azimuth angle and zenith angle of the surface normal of the object being measured are calculated using the degree of polarization and polarization angle obtained in Step 3. Step 5: Encode the full-spectrum polariton image obtained in Step 1 and the two sets of surface normals of the measured object calculated in Step 3 into input data. Input this data into a surface normal reconstruction model based on a convolutional neural network and aided by prior physical knowledge. Its main modules include: (1) A Res2Net encoding module based on hierarchical residual links; (2) Spatially-Adaptive Normalization (SPADE) module; The model outputs the predicted surface normal vector components of the measured object, which serve as the final 3D data.

2. The three-dimensional reconstruction method based on multi-dimensional information fusion and convolutional neural networks according to claim 1, characterized in that, The specific steps described in step 2 are as follows: Step 2.1: To address the small-scale translational shift caused by filter switching and minor camera displacement during shooting, a phase correlation registration method is adopted. This method involves using an image with a transmission polarization direction of 0° across the entire spectrum as the registration reference. The cross-power spectrum between each image and the reference image is calculated, and then an inverse Fourier transform is performed on the cross-power spectrum. The peak value of the resulting pulse function is the translational shift. Step 2.2: To remove noise from the polarization image while preserving the structural consistency between each polarization channel, a joint guided filtering algorithm is used to remove noise. That is, for each spectral dimension, an image with a transmission polarization direction of 0° is used as a guide image, and a guided filter is applied to all polarization sub-images, including the image itself, to remove noise.

3. The three-dimensional reconstruction method based on multi-dimensional information fusion and convolutional neural networks according to claim 1, characterized in that, The specific steps described in step 3 are as follows: Step 3.1: Assuming that only diffuse reflection occurs on the surface of the object being measured, under this assumption, the zenith angle of the normal to the surface of the object can be obtained by substituting the degree of linear polarization into the formula. The calculation yields the following result: In the formula, DoLP is the degree of polarization of the surface of the object being measured, n is the refractive index of the surface material of the object being measured, and θ is the zenith angle of the normal to the surface of the object being measured, and the same applies below. Step 3.2, assuming that only specular reflection occurs on the surface of the object being measured, under this assumption, based on the wavelength-polarization data from multiple detection spectrum bands calculated in Step 2, the slope k of the fitted straight line corresponding to the wavelength-polarization data of each pixel in the image can be calculated. Step 3.3: Using the wavelength-polarization data of any spectral band, the formula can be used to... Calculate two possible values ​​for the zenith angle of the normal to the surface of the object being measured: θ1 < θ2. If the slope k obtained in step 3.2 is greater than 0, then θ2 can be determined as the true zenith angle of the normal; otherwise, θ1 can be determined as the true zenith angle of the normal. Step 3.4: Using the polarization images obtained in Step 1 at the four polarization angles, fit the polarization angle-light intensity curve of the measured object surface according to Malus's law, and then apply the formula... Calculate the azimuth angle of the normal to the surface of the object being measured; where, I max with I min These represent the maximum and minimum light intensity values ​​in the curves above. The azimuth angle is the angle between the direction of vibration of the incident linearly polarized light and the transmission axis of the polarizer; the azimuth angle here has a 180° ambiguity.

Images