Three-dimensional reconstruction method based on fusion of spectral information and polarization information

Abstract

The invention provides a three-dimensional reconstruction method based on fusion of spectral information and polarization information. The three-dimensional reconstruction method comprises the following steps: acquiring polarization sub-images of a detected object in a plurality of detection spectrum sections and four transmission polarization directions through a multispectral polarization camera; calculating a polarization parameter of the measured object by using the acquired image, and further calculating a surface normal azimuth angle and a zenith angle of the measured object when only diffuse reflection or specular reflection occurs; the method comprises the following steps: calculating an accurate normal of a target surface by using a surface normal reconstruction model constructed with assistance of physical priori knowledge based on a convolutional neural network; and restoring the relative depth of the to-be-measured object pixel by pixel through surface normal gradient integration. According to the method provided by the invention, the measured object comprising the mirror reflection surface can be accurately reconstructed, rapid reconstruction without additionally establishing a measurement environment can be realized in an outdoor scene with a complex illumination condition, and the specific application range of the polarization three-dimensional reconstruction technology can be expanded.

Description

Technical Field

[0001] The present invention relates to a three-dimensional reconstruction method based on the fusion of spectral information and polarization information, specifically a three-dimensional reconstruction method based on polarization three-dimensional reconstruction, assisted by spectral information and deep learning. Background Art

[0002] Multispectral imaging technology emerged in the 1960s and is a detection technology for obtaining spatial image information of an object under different spectral bands; polarization imaging is a new scheme for obtaining the natural polarization information of an object's surface. Compared with traditional RGB cameras, RGB cameras can only obtain rough external spatial information such as the basic size, shape, and relative position of the object to be measured, while multi-dimensional detection can obtain richer information: the spectral information obtained by multispectral imaging can be used to calculate internal information such as the material composition and content of the object's surface; polarization information can be used to calculate fine information such as the texture, roughness, and surface refractive index of the object's surface. The detection of multi-dimensional information is widely used in fields such as military, remote sensing, and industrial manufacturing.

[0003] Polarization is an important inherent property of light waves, referring to the asymmetry of the vibration direction of the light wave vector relative to its propagation direction during the propagation of light. When light waves are reflected or refracted on the surface of an object, the polarization characteristics of the light waves often change, and this change is generally determined by factors such as the surface material and morphological characteristics of the object. Therefore, polarization information actually provides an effective detection dimension for three-dimensional reconstruction, that is, the surface morphology of the object is inversely calculated through the polarization characteristics of the reflected light, which helps to improve the speed and accuracy of three-dimensional reconstruction on the basis of traditional three-dimensional reconstruction and reduce the requirements for the number of devices and scene construction for the task.

[0004] After the 21st century, researchers developed a systematic polarization three-dimensional reconstruction scheme based on the above scheme. Most current polarization three-dimensional reconstruction technologies determine the surface morphology of an object by analyzing the states of the degree of polarization and the polarization angle. Generally speaking, the polarization three-dimensional reconstruction technology has low requirements for imaging devices and shooting scenes. In a conventional polarization three-dimensional reconstruction process, only a single real-time polarization imaging camera is needed to image the object to be measured once, and four polarization sub-images of the same object to be measured can be obtained. The surface information of the object to be measured restored by polarization three-dimensional reconstruction often contains relatively complete detailed texture features, and the reconstruction effect is better for objects to be measured with rich surface details or fine structures.

[0005] To obtain an accurate model of the surface of the object under test, the polarization three-dimensional reconstruction technology first obtains the normal direction vectors at the key points on the surface of the object under test, and on this basis, gradientizes and integrates the vectors to form a three-dimensional model composed of relative depth information. The process of obtaining the normal direction vectors includes two stages, namely, the reconstruction of the azimuth angle of the normal vector and the reconstruction of the zenith angle of the normal vector; both face different problems in their respective reconstruction processes.

[0006] Currently, there is a relatively fixed scheme for reconstructing the azimuth angle of the normal vector of the object under test. Generally speaking, the reconstruction of the azimuth angle of the normal is based on Malus' law, that is, by fitting the polarization angle-intensity curve from the light intensity information captured in multiple different polarization directions, and then realizing the recovery of the azimuth angle. However, the implementation principle of this method determines that the azimuth angle obtained by this method will have an ambiguity of 180°, so it is necessary to rely on data from dimensions other than polarization obtained from images or other sources to correct the azimuth angle. Most of the existing polarization three-dimensional reconstruction schemes introduce traditional three-dimensional reconstruction methods to assist in correcting the azimuth angle; there are also some studies that achieve this purpose through total least squares optimization or by introducing deep learning assistance.

[0007] In the process of reconstructing the zenith angle of the normal of the object under test, most current studies choose to reconstruct only the diffuse reflection area on the surface of the object under test. Since the relationship between the zenith angle of the normal and the polarization degree of the outgoing light is a one-to-one mapping function on the diffuse reflection surface, the reconstruction of the zenith angle on the diffuse reflection surface is relatively simple in the case where the polarization angle has been calculated from the images obtained in four polarization directions. However, the reconstruction of the zenith angle for the specular reflection surface is not the case. In the visible spectral range, the polarization degree of an outgoing light after specular reflection often corresponds to two possible zenith angles, which makes it very difficult to reconstruct the zenith angle for the specular reflection area without introducing other information. To solve this problem, some studies choose to normalize and fill a small amount of specular reflection surfaces in the image as diffuse reflection surfaces so as to apply the reconstruction method for the normal of the diffuse reflection surface; while another part of the studies avoid the appearance of specular reflection areas on the object surface by building a special measurement environment. However, considering a more general detection scenario, the object surfaces to be detected often have both specular reflection and diffuse reflection areas at the same time. Therefore, most of the above-mentioned schemes have poor effects in such detection scenarios.

[0008] To address this difficult problem, the present invention proposes a three-dimensional reconstruction scheme for medium and small-sized objects with diffuse reflection characteristics and specular reflection characteristics based on the comprehensive detection of spectral dimension information and polarization information, which can achieve the three-dimensional reconstruction of the surface of the object under test in a more general scenario without constructing a complex detection scenario. Summary of the Invention

[0009] The problems solved by the technology of the present invention are as follows: When reconstructing medium and small objects in general detection scenarios, conventional 3D reconstruction schemes often require high requirements for measurement environment setup and equipment. However, polarization-based 3D reconstruction schemes are mostly ineffective for large specular reflection regions that appear on the surface of the object to be measured in such detection scenarios. The present invention combines the advantages of simple requirements for equipment environment of polarization-based 3D reconstruction technology, and introduces spectral dimension information, solves the above-mentioned problem of zenith angle reconstruction in the specular reflection region, and expands the application scope of polarization-based 3D reconstruction technology.

[0010] To achieve the above invention purposes, the present invention is implemented by the following technical methods:

[0011] Step 1: Use a multi-spectral polarization imaging detection system to separately extract polarized sub-images of the object to be measured at four transmission polarization directions of 0°, 45°, 90°, and 135° in multiple detection spectral bands.

[0012] Step 2: In each detection spectral band, calculate the degree of polarization and polarization angle of the surface of the object to be measured respectively through the polarized sub-images at four transmission polarization directions of 0°, 45°, 90°, and 135°.

[0013] Step 3.1: Assume that only diffuse reflection occurs on the surface of the object to be measured. Under this assumption, the zenith angle of the normal line on the surface of the object to be measured can be obtained by substituting the average degree of linear polarization into the formula

[0014]

[0015] for calculation; in the formula, DoLP is the degree of polarization of the surface of the object to be measured, n is the refractive index of the material on the surface of the object to be measured, θ is the zenith angle of the normal line on the surface of the object to be measured, and the same below;

[0016] Step 3.2: Assume that only specular reflection occurs on the surface of the object to be measured. Under this assumption, according to the wavelength-degree of polarization data from multiple detection spectral bands calculated in Step 2, the fitting line slope k corresponding to the wavelength-degree of polarization data can be calculated for each pixel in the picture.

[0017] Step 3.3: Through the wavelength-degree of polarization data of any spectral band, the two possible values of the zenith angle of the normal line on the surface of the object to be measured, θ

[0018]

[0019] can be calculated; at this time, if the slope k obtained in Step 3.2 > 0, then θ 1 <θ 2 can be determined as the true zenith angle of the normal line; otherwise, θ 2 is determined as the true zenith angle of the normal line; 1

[0020] Step 3.4, from the polarization images at the four polarization angles obtained in Step 1, fit the polarization angle - light intensity curve of the surface of the object to be measured according to Malus' law, and according to the formula

[0021]

[0022] calculate the azimuth angle of the normal of the surface of the object to be measured; in the formula, I max and I min are respectively the maximum and minimum light intensities in the above curve, θ is the angle between the vibration direction of the incident linearly polarized light and the transmission axis of the polarizer, is the azimuth angle of the normal of the surface of the object to be measured, with an ambiguity of 180°;

[0023] Step 4, for each pixel, encode the brightness data in the multiple polarization sub - images obtained in Step 1 and the two sets of normal directions of the surface of the object to be measured calculated in Step 3 into a corresponding set of input data, and input it into the surface normal reconstruction model constructed by using physical prior knowledge and based on a convolutional neural network; while correcting the azimuth angle and determining the reflection type of each pixel on the surface of the object to be measured, calculate the accurate normal of the surface of the object to be measured;

[0024] Step 5, convert the normal of the surface of the object to be measured corresponding to each pixel obtained in Step 4 into gradient information and integrate it to reconstruct the relative depth of the position where each pixel is located.

[0025] Compared with other existing methods, the three - dimensional reconstruction solution for small and medium - sized objects with diffuse reflection and specular reflection characteristics provided by the present invention has the following advantages:

[0026] (1) By introducing spectral dimension information, the problem of difficult reconstruction of the zenith angle of the normal of the specular reflection area on the surface of the object to be measured in the traditional polarization three - dimensional reconstruction solution is effectively solved. This enables the detection solution of the present invention to be applicable to three - dimensional reconstruction work under more general lighting conditions.

[0027] (2) The present invention uses a deep - learning method combined with physical prior knowledge to judge the reflection type of each area of the object to be measured and correct the azimuth angle of the surface normal of the corresponding area. This solution does not require obtaining additional information outside the image, nor does it require using equipment other than a polarization spectroscopy camera to obtain or add additional features to the object to be measured.

[0028] (3) When using the method proposed by the present invention for three - dimensional reconstruction, the requirements for equipment are lower than those of the traditional three - dimensional reconstruction solution, and there is no need to build an additional measurement environment for this reconstruction process. Description of the Drawings

[0029] Figure 1This is the data processing flow chart of the present invention.

[0030] Figure 2 This is the reconstruction flow chart of the zenith angle of the surface normal of the object to be measured. Specific implementation manner

[0031] In order to more clearly show the specific technical route of the three-dimensional reconstruction scheme mentioned in the present invention, a more detailed description is provided for the part that is not fully described in the invention content. The specific method is as follows:

[0032] 1. The data processing flow of the whole set of methods is as Figure 1 shown. The multi-polarization angle polarization images and multi-spectral data shown in the flow are all obtained by a multi-spectral polarization imager. With the help of a split focal plane polarization imaging system, the above multi-spectral polarization imager can simultaneously obtain the polarization sub-images of the object to be measured at four different transmission polarization directions of 0°, 45°, 90°, and 135° in a single imaging process; in addition, by rotating the rotating filter wheel equipped with filter sheets, the polarization data of the object to be measured under different incident spectral bands can be obtained. The multi-spectral polarization imager involved in the present invention can obtain the corresponding polarization data at a total of 8 spectral bands in the range from visible light (441 nm) to near-infrared (980 nm), which is sufficient to meet the actual needs of three-dimensional reconstruction work.

[0033] 2. According to a set of polarization sub-images taken in each spectral band, a set of simplified Stokes vectors [I, Q, U] can be calculated for each pixel T :

[0034]

[0035] From the above Stokes vectors, the degree of linear polarization (DoLP) and the angle of polarization (AoP) in the image can be calculated:

[0036] Linear Polarization, DoLP) and the angle of polarization (Angle of Polarization, AoP):

[0037]

[0038] Among them, the degree of linear polarization is the ratio of the intensity of linearly polarized light to the total light intensity, and the angle of polarization represents the vibration direction of the linearly polarized light part. Based on these obtained polarization parameters, the zenith angle and azimuth angle of the surface normal of the object to be measured can be reconstructed.

[0039] 3. Calculate the zenith angle of the normal according to the polarization parameters. The present invention first assumes that the reflection conditions on the surface of the object to be measured belong to two cases of diffuse reflection and specular reflection, and calculates the zenith angle of the surface normal of the object to be measured on the basis of the corresponding assumptions. The overall data processing flow of this stage is asFigure 2 as shown

[0040] When specular reflection occurs on the surface of the object to be measured, the degree of linear polarization of the outgoing light is:

[0041]

[0042] At this time, given the refractive index of the surface of the object to be measured and the angle of incidence, a unique degree of polarization value can be determined. However, when only the degree of polarization data is obtained, if an attempt is made to restore the zenith angle from the captured degree of polarization data, two possible incident zenith angles will be calculated.

[0043] When diffuse reflection occurs on the surface of the object to be measured, the degree of linear polarization of the outgoing light is:

[0044]

[0045] At this time, on the diffuse reflection surface, there is a one-to-one correspondence between the degree of polarization and the incident zenith angle. Therefore, the zenith angle of the normal line of the surface of the object to be measured can be uniquely determined by the degree of polarization.

[0046] To solve the problem of zenith angle ambiguity under the specular reflection assumption, considering the characteristic that the refractive index of the surface material of the object to be measured changes when the incident spectral band is different, the present invention selects to use multiple sets of degree of polarization data from different spectral bands to disambiguate the ambiguity of the zenith angle of the normal line of the surface of the object to be measured under the specular reflection assumption, that is:

[0047] (1) For each pixel in the image, calculate its degree of linear polarization in each spectral band, thereby constructing a set of wavelength-degree of polarization data;

[0048] (2) Calculate two different zenith angle values θ 1 <θ 2 ;

[0049] (3) Determine the slope of the wavelength-degree of polarization fitting curve. If the obtained slope k > 0, then θ 2 can be determined as the true normal zenith angle; otherwise, θ 1 is determined as the true normal zenith angle.

[0050] 4. Calculate the normal azimuth angle according to the polarization parameters. The present invention adopts the usual method based on Malus' law:

[0051] I = I p ·cos 2 θ

[0052] where I p$I_0$ is the initial light intensity before the light beam passes through the polarizer, and $\theta$ is the angle between the vibration direction of the incident linearly polarized light and the transmission axis of the polarizer. Considering that during the process of the transmission polarization direction of the polarizer changing, the light intensity of the polarized light is equal to the difference between the maximum light intensity and the minimum light intensity collected by the camera during this process, thus there is

[0053]

[0054] In the formula, is the azimuth angle of the normal line to be determined. At this time, considering that

[0055]

[0056] If the light intensity information in each transmission polarization direction is used to restore the azimuth angle, the obtained azimuth angle will have an ambiguity of 180°, so this problem needs to be solved in the subsequent steps.

[0057] 5. After physically restoring the zenith angle and azimuth angle of the normal line on the surface of the object to be measured, for each pixel in the image, the present invention encodes the brightness data of four polarized sub-images from multiple spectral bands and the calculated two sets of normal line directions on the surface of the object to be measured into a corresponding set of input data, and inputs it into the surface normal reconstruction model constructed based on a convolutional neural network and assisted by physical prior knowledge; while correcting the azimuth angle and determining the reflection type of each pixel on the surface of the object to be measured, calculate the accurate normal vector of the surface of the object to be measured.

[0058] Compared with the traditional deep learning scheme, the deep learning scheme based on physical knowledge additionally introduces physical prior data as an assistance, which can enable the constructed surface normal reconstruction model to not only obtain the three-dimensional information of the object to be measured from the original data, but also adhere to the general physical laws from polarization information to surface normal vector parameters as much as possible during the process of reconstructing the surface normal, that is, the relevant formulas involved in steps 3 and 4. This scheme can solve the azimuth angle ambiguity problem while determining the reflection type of the surface of the object to be measured during the learning process to solve the problem of mixing of different types of reflection regions.

[0059] 6. After calculating the accurate direction data of the normal line on the surface of the object to be measured, convert the normal line on the surface of the object to be measured corresponding to each pixel into gradient information and then integrate it, and the relative depth of the position where each pixel is located can be reconstructed, and then the true three-dimensional model of the object to be measured can be obtained.

[0060] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention. The content not detailedly described in the specification of the present invention belongs to the prior art well-known to those skilled in the art.

Claims

1. A three-dimensional reconstruction method based on the fusion of spectral information and polarization information, characterized in that: Step 1, using a multi-spectral polarization imaging detection system, respectively extracting polarization sub-images of the object under test at four transmission polarization directions of 0°, 45°, 90°, and 135° under multiple detection spectrum bands; Step 2, in each detection spectrum, respectively calculate the polarization degree and polarization angle of the surface of the object under test through the polarization sub-images in four transmission polarization directions of 0°, 45°, 90°, and 135°; Step 3, assuming that only diffuse reflection or only specular reflection occurs on the surface of the object under test, respectively, and using the polarization degree and polarization angle obtained in step 2 to calculate the azimuth angle and zenith angle of the surface normal of the object under test; under the assumption of diffuse reflection, the above polarization information is obtained by calculating the average of the brightness in the transmission direction of each spectral band, and under the assumption of specular reflection, additional spectral information will be used to eliminate the ambiguity of the zenith angle; Step 4: For each pixel, the brightness data in the multiple polarization sub-images obtained in step 1 and the two groups of surface normal directions of the measured object calculated in step 3 are encoded into a corresponding set of input data, and input into a surface normal reconstruction model based on a convolutional neural network and assisted by physical prior knowledge; while correcting the azimuth angle and determining the reflection type of each pixel on the surface of the measured object, the accurate normal of the surface of the measured object is calculated; Step 5: Convert the surface normal of the object under test corresponding to each pixel obtained in step 4 into gradient information and integrate it to reconstruct the relative depth of each pixel.

2. According to claim 1, a three-dimensional reconstruction method based on the fusion of spectral information and polarization information is characterized in that: The specific method for obtaining the surface information of the measured object in step 3 is as follows: Step 3.1, assuming that only diffuse reflection occurs on the surface of the object being measured. Under this assumption, the zenith angle of the normal line of the surface of the object being measured can be obtained by substituting the average linear polarization degree into the formula Calculated; where DoLP is the degree of polarization of the surface of the object being measured, n is the refractive index of the material on the surface of the object being measured, and θ is the zenith angle of the normal line of the surface of the object being measured, the same below; Step 3.2, assuming that only specular reflection occurs on the surface of the object being measured, under this assumption, based on the wavelength-polarization data from multiple detection spectrum bands calculated in step 2, the slope k of the fitting line corresponding to the wavelength-polarization data of each pixel in the image can be calculated; Step 3.3, through the wavelength-polarization data of any spectral band, the formula Calculate two possible values ​​of the normal zenith angle of the surface of the object to be measured, θ1<θ2; at this time, if the slope k obtained in step 3.2>0, θ2 can be determined as the true normal zenith angle; otherwise, θ1 is determined as the true normal zenith angle; Step 3.4, using the polarization images at the four polarization angles obtained in step 1, the polarization angle-intensity curve of the surface of the object to be measured is fitted according to Malus's law, and according to the formula Calculate the normal azimuth of the surface of the object being measured; where I max with I min are the maximum and minimum values ​​of light intensity in the above curves, respectively; θ is the angle between the vibration direction of the incident linear polarized light and the transmission axis of the polarizer, It is the azimuth of the normal line of the surface of the object being measured, and has an ambiguity of 180°.

Images