Multispectral photometric stereo three-dimensional reconstruction method based on Gaussian process model

Abstract

The invention relates to a multispectral luminosity three-dimensional reconstruction method based on a Gaussian process model. The method comprises the following steps: constructing a multispectral luminosity three-dimensional imaging system; the light field camera shoots a picture; after channel separation is carried out on the picture, the picture is used as input for depth estimation, and a reconstructed normal vector and a multispectral luminosity three-dimensional reconstruction depth map are obtained; performing three-dimensional reconstruction on the light field camera to obtain a light field camera depth map; and fusing the depth maps obtained in the two modes through a Gaussian process model mode to obtain a final depth map. Compared with a photometric stereo method, a light field camera comprising a micro-lens array has a higher utilization rate of light field information, the method combines light field camera reconstruction with the photometric stereo method based on the idea of Gaussian model hybrid reconstruction, gives full play to the respective advantages of the light field camera reconstruction and the photometric stereo method, improves the utilization rate of the light field information through the light field camera with the micro-lens array, and improves the reconstruction efficiency of the light field camera. And then an algorithm is designed to process the shot picture, so that a good low-frequency reconstruction effect is obtained, and higher precision is achieved.

Description

Technical Field

[0001] The invention relates to a multi-spectral photometric stereo three-dimensional reconstruction method based on a Gaussian process model, and belongs to the technical field of three-dimensional reconstruction. Background Art

[0002] With the rapid development of digital technology, 3D reconstruction technology has gradually become a research hotspot in the fields of computer vision and medical imaging. As a non-contact 3D reconstruction method, photometric stereo technology has the characteristics of high resolution and high precision; multispectral imaging technology can obtain the reflectance spectrum information of objects in different bands, thereby providing richer color and material information. By combining multispectral imaging with photometric stereo, it is possible to capture complex surface textures and details more accurately, further improving the accuracy and realism of 3D reconstruction. However, due to the principle limitations of photometric stereo, various forms of cumulative errors will be generated during reconstruction, which is reflected in the results as low-frequency distortion in some areas, which cannot meet the requirements for most scenes.

[0003] Light field cameras are an advanced imaging technology that can capture additional information about the direction and intensity of light. Compared to traditional cameras that only record the position and color of light passing through the lens, light field cameras can record the direction and intensity of light at these positions, thus providing richer and more comprehensive light information. However, due to the limitations of the microlens array, the resolution is reduced, affecting some high-frequency detail effects.

[0004] The Gaussian process (GP) model is a non-parametric Bayesian statistical inference model that models various complex surfaces through mean and covariance functions, incorporates measurement errors into the modeling process, and assigns credibility to the constructed model. Summary of the invention

[0005] The present invention proposes a multispectral photometric stereo 3D reconstruction method based on a Gaussian process model. Its purpose is to combine light field camera reconstruction with photometric stereo through a Gaussian process model, give full play to their respective advantages, improve the utilization rate of light field information through a light field camera with a microlens array, and then design an algorithm to establish a Gaussian process model to process the image data set obtained by shooting, so as to obtain a good low-frequency reconstruction effect and achieve higher accuracy.

[0006] The technical solution of the present invention:

[0007] A multi-spectral photometric stereo 3D reconstruction method based on a Gaussian process model comprises the following steps:

[0008] 1) Construction of multi-spectral photometric stereo imaging system;

[0009] 2) Light field camera takes pictures;

[0010] 3) After the image is channel-separated, it is used as input for depth estimation to obtain the reconstructed normal vector and multi-spectral photometric stereo depth map;

[0011] 4) 3D reconstruction of the light field camera to obtain the depth map of the light field camera;

[0012] 5) Based on a Gaussian process model, the multispectral photometric stereo depth map and the light field camera depth map are fused.

[0013] Furthermore, the multispectral photometric stereo imaging system includes a light shield, a slide rail, a light source, and a light field camera; the slide rail is arranged on the light shield; the light source is an RGB three-color light source, which is installed on the slide rail; and the acquisition camera is installed on the top of the light shield.

[0014] Furthermore, the light field camera sets a microlens array between the imaging lens and the sensor, and the microlens array is placed on the imaging plane of the main lens of the light field camera; the microlens array is arranged parallel to the sensor, and the spacing is equal to the focal length of the microlens array; the microlens array adopts a hexagonal arrangement, contains 15*15 microlenses, and can capture and demodulate 225 perspective images at a time.

[0015] Furthermore, after the light field camera takes the picture in step 2), image decoding is performed, the refocusable picture is exported and demodulated, and step 3) multispectral photometric stereo reconstruction and step 4) light field camera three-dimensional reconstruction are performed respectively.

[0016] Furthermore, the specific steps of reconstructing the normal vector in step 3) are:

[0017] Input the central viewing angle image, assume the object to be measured is a Lambertian body, and assume the light source is parallel light. According to the Lambertian body reflection model, the following formula is obtained:

[0018] m=EL·ρn

[0019] Wherein, m is the pixel value, E is the light source intensity, L is the incident direction of the light source, ρ is the reflectivity of the object surface, and n is the normal vector of the object surface; the incident direction of the light source is obtained by calibration;

[0020] Ignoring the unknown scaling and assuming a unit intensity light source, we get a simplified Lambertian imaging model:

[0021] ρn=L -1 ·m

[0022] Among them, n has unit length and norm of 1, which can estimate the surface normal and albedo at the same time; when T light sources are used, the light source direction matrix is ​​no longer a square matrix, and the linear equations to be solved become overdetermined; the normal vector n of the object surface can be approximated by the pseudo-inverse of L in the least squares sense:

[0023] n=(L T L) -1 L T m.

[0024] Furthermore, the method for estimating the multispectral photometric stereoscopic reconstruction depth map in step 3) is specifically to assume that the normal vector n of the object surface is orthogonal to v1 and v2, and the surface is corresponded to the pixel at the image plane (i, j), and the following relationship is obtained:

[0025] v1=[i+1 jz i+1,j ] T -[ijz i，j ] T

[0026] v2=[i j+1 z i,j+1 ] T -[ijz i，j ] T

[0027] n·v1=0

[0028] n·v2=0

[0029] After sorting, we can get:

[0030]

[0031] Write it in the form of a sparse matrix and solve it to get the depth value z corresponding to each pixel.

[0032] Furthermore, the three-dimensional reconstruction of the light field camera in step 4) includes the following steps: S41: performing camera calibration by processing the files provided by the light field camera; S42: performing distortion correction on the sub-aperture image analysis; S43: estimating the depth map by using a sub-pixel displacement method based on phase shift, setting a cost function for optimization, and performing subsequent visual optimization and enhancement to obtain a corresponding depth map.

[0033] Furthermore, the Gaussian process in step 5) includes the following steps: S51: establishing a "substitution model" formula, selecting appropriate mean function and covariance function, and setting substitution initial hyperparameters according to the kernel function; S52: inputting the multispectral photometric stereo depth map, and according to its objective function, using the gradient method to obtain the optimal hyperparameters, and determining the posterior model; S53: establishing a "correction model" formula, selecting appropriate scaling factors and movement factors, and setting correction initial hyperparameters; S54: inputting the light field camera depth map, and calculating the correction initial hyperparameters; S55: inputting the position to be predicted, and obtaining the final prediction value.

[0034] Beneficial effects of the present invention:

[0035] 1) The device of the present invention is simple and low-cost, and at the same time introduces the advantages of light field into the field of photometric stereo to make up for its shortcomings;

[0036] 2) Based on the idea of ​​hybrid reconstruction, the utilization rate of light field information is improved, and the captured images are processed by algorithms to obtain good low-frequency reconstruction effects, providing a highly robust solution to the low-frequency distortion problem of photometric stereo 3D reconstruction technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Attached Figure 1 It is a schematic flow chart of the three-dimensional reconstruction method of the present invention.

[0038] Attached Figure 2 It is a structural schematic diagram of a multispectral photometric stereo imaging system.

[0039] Attached Figure 3 It is a schematic diagram of the layout of the light field camera microlens array.

[0040] Attached Figure 4 Schematic diagram of the correspondence between multi-spectral photometric stereo normal vectors and image plane pixels. DETAILED DESCRIPTION

[0041] The technical solution of the present invention is further explained below in conjunction with the accompanying drawings.

[0042] Compare with Figure 1 A multispectral photometric stereo 3D reconstruction method based on a Gaussian process model specifically includes the following steps: building a multispectral photometric stereo imaging system; after a light field camera takes a picture, decoding the image, exporting and demodulating the refocusable picture; after channel separation of the picture, using it as input for depth estimation to obtain a reconstructed normal vector and a multispectral photometric stereo reconstruction depth map; three-dimensional reconstruction with a light field camera to obtain a light field camera depth map; fusing the multispectral photometric stereo depth map with the light field camera depth map to output a final depth map.

[0043] As attached Figure 2 As shown, the multispectral photometric stereo imaging system includes a light shield, a slide rail, a light source, and a light field camera; the slide rail is set on the light shield; the light source is an RGB three-color light source, which is installed on the slide rail; and the acquisition camera is installed on the top of the light shield.

[0044] As attached Figure 3 As shown, the light field camera sets a microlens array between the imaging lens and the sensor. The microlens array is placed on the imaging plane of the main lens of the light field camera. The microlens array is arranged parallel to the sensor, and the spacing is equal to the focal length of the microlens array. The microlens array adopts a hexagonal arrangement, contains 15*15 microlenses, and can capture and demodulate 225 viewing angle images at a time.

[0045] The normal vector n of the estimated surface of a multispectral photometric stereo object can be approximated by the pseudo-inverse of L to obtain the least squares approximate solution:

[0046] n=(L T L) -1 L T m

[0047] It can be seen that the more light sources there are, the more robust the results we can get. Figure 4 To estimate the depth: the normal vector n of the object surface patch must be orthogonal to v1 and v2. The patch corresponds to the pixel at (i, j) in the image plane, and the following relationship can be obtained:

[0048] v1=[i+1 jz i+1，j ] T -[ijz i，j ] T

[0049] v2=[i j+1 z i，j+1 ] T -[ijz i，j ] T

[0050] n·v1=0

[0051] n·v2=0

[0052] After sorting, we can get:

[0053]

[0054] Write it in the form of a sparse matrix and solve it to get the depth value z corresponding to each pixel.

[0055] The light field camera used is that of Lytro, and the image decoding after shooting is performed using Lytro's software. The specific steps of the light field camera 3D reconstruction are: calibrating the camera by processing the files that come with the camera, such as white images; analyzing the sub-aperture image for distortion correction; estimating the depth map using the phase-shift-based sub-pixel displacement method, setting the cost function for optimization, and performing subsequent visual optimization and enhancement to obtain the corresponding depth map.

[0056] The specific steps of Gaussian process modeling are:

[0057] S51: Establish a "substitution model" formula, select appropriate mean function and covariance function, set substitution initial hyperparameters according to the kernel function; establish a model from a multispectral photometric stereo depth map:

[0058] zPS (x i ,y i ) = f PS (x i ,y i )+ε PS

[0059] In the formula, f PS (x i ,y i ) represents the ideal value of the measured surface under the ideal condition of perfect fitting, ε PS represents the measurement error. The Gaussian process model is used to represent f PS (x i ,y i ):

[0060]

[0061] Among them, v i =(x i ,y i ), v j =(x j ,y j ); is the mean function; is the covariance function; the kernel function is the Matern kernel function, and we get:

[0062]

[0063]

[0064] Among them, ‖v i -v j ‖ represents point v on the (x,y) plane i and v j The Euclidean distance between represents the constant variance of the Gaussian model, λ is the smoothing coefficient, when λ=0.5, the Matern kernel function is an exponential kernel, and when λ→∞, it is a Gaussian kernel; l PS is the length parameter of the kernel function; Γ is the Gamma function, K is the second kind of modified Bassel function;

[0065] S52: Input the multispectral photometric stereo depth map, and according to its objective function, use the gradient method to obtain the optimal hyperparameters and determine the posterior model; the replacement initial hyperparameters are obtained by the maximum likelihood method, and the conditional probability negative logarithm of the multispectral photometric stereo depth map is used as the objective function. The partial derivative of the hyperparameters is obtained and maximized by the conjugate gradient method to obtain the optimal solution for the replacement initial hyperparameters.

[0066] S53: Establish a "correction model" formula, select appropriate scaling factors and shift factors, and set initial hyperparameters for correction; the "correction model" expression is:

[0067] z LF (x i ,y i )=ρ(x i ,y i )z′ PS (x i ,y i )+δ(x i ,y i )+ε LF

[0068] In the formula, i=1,2,…,nLF,ρ(x i ,y i ) and δ(x i ,y i ) represent the scaling factor and the shift factor, ε LF Indicates error;

[0069] ρ(x i ,y i )=ρ0+ρ1x i +ρ2y i

[0070] δ(x i ,y i )~GP(δ0,k δ (v i , v j ))

[0071] S54: Input the light field camera depth map and calculate and correct initial hyperparameters;

[0072] S55: Enter the position to be predicted and obtain the final predicted value. You can get the position v at any position on the (x, y) plane. i =(x i ,y i ) can be expressed as:

[0073] z′ LF (x i ,y i )=ρ(x i ,y i )z′ PS (x i ,y i )+δ(x i ,y i ).

[0074] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multispectral photometric stereo 3D reconstruction method based on a Gaussian process model, characterized in that: The specific steps include: 1) Construction of multi-spectral photometric stereo imaging system; 2) Light field camera takes pictures; 3) After the image is channel-separated, it is used as input for depth estimation to obtain the reconstructed normal vector and multi-spectral photometric stereo depth map; 4) 3D reconstruction of the light field camera to obtain the depth map of the light field camera; 5) Based on a Gaussian process model, the multispectral photometric stereo depth map and the light field camera depth map are fused.

2. The method for multi-spectral photometric stereo 3D reconstruction based on Gaussian process model according to claim 1, characterized in that The multi-spectral photometric stereo imaging system includes a light shield, a slide rail, a light source, and a light field camera; the slide rail is arranged on the light shield; the light source is an RGB three-color light source, which is installed on the slide rail; the acquisition camera is installed on the top of the light shield, and the light field camera is provided with a microlens array between the imaging lens and the image sensor.

3. The method for multi-spectral photometric stereo 3D reconstruction based on Gaussian process model according to claim 2, characterized in that The microlens array is placed on the imaging plane of the main lens of the light field camera; The microlens array is arranged in parallel with the image sensor, and the spacing is equal to the focal length of the microlens array; The microlens array is arranged in a hexagonal shape, includes 15*15 microlenses, and can capture and demodulate 225 viewing angle images at one time.

4. The method for multi-spectral photometric stereo 3D reconstruction based on Gaussian process model according to claim 1, characterized in that In step 2), after the light field camera takes the picture, the image is decoded, the refocusable picture is exported and demodulated, and multispectral photometric stereo reconstruction and light field camera 3D reconstruction are performed respectively.

5. The multi-spectral photometric stereo 3D reconstruction method based on Gaussian process model according to claim 1, characterized in that The specific steps of reconstructing the normal vector in step 3) are: Input the central viewing angle image, assume the object to be measured is a Lambertian body, and assume the light source is parallel light. According to the Lambertian body reflection model, the following formula is obtained: m=EL·ρn Wherein, m is the pixel value, E is the light source intensity, L is the incident direction of the light source, ρ is the reflectivity of the object surface, and n is the normal vector of the object surface; the incident direction of the light source is obtained by calibration; Ignoring the unknown scaling and assuming a unit intensity light source, we get a simplified Lambertian imaging model: ρn=L -1 ·m Among them, n has unit length and norm of 1, which can estimate the surface normal and albedo at the same time; when T light sources are used, the light source direction matrix is ​​no longer a square matrix, and the linear equations to be solved become overdetermined; the normal vector n of the object surface can be approximated by the pseudo-inverse of L in the least squares sense: n=(L T L) -1 L T m。 6. The method for multi-spectral photometric stereo 3D reconstruction based on Gaussian process model according to claim 5, characterized in that The method for estimating the multispectral photometric stereo depth map in step 3) is specifically to assume that the normal vector n of the object surface is orthogonal to v1 and v2, and correspond the surface to the pixel at the image plane (i, j), and obtain the following relationship: v1=[i+1 jz i+1,j ] T -[ijz i,j ] T <h2 style=";text-align:left;direction:ltr">v2=[i j+1 z<h2 style=";text-align:left;direction:ltr"> i,j+1 <h2 style=";text-align:left;direction:ltr"> ]<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> -[ijz<h2 style=";text-align:left;direction:ltr"> i,j <h2 style=";text-align:left;direction:ltr"> ]<h2 style=";text-align:left;direction:ltr"> T n·v1=0 n·v2=0 After sorting, we can get: Write it in the form of a sparse matrix and solve it to get the depth value z corresponding to each pixel.

7. The method for multi-spectral photometric stereo 3D reconstruction based on Gaussian process model according to claim 1, characterized in that Step 4) The light field camera 3D reconstruction comprises the following steps: S41: Performing camera calibration by processing the files provided by the light field camera; S42: Analyze the sub-aperture image and perform distortion correction; S43: using a phase-shift-based sub-pixel displacement method to estimate the depth map, setting a cost function for optimization, and performing subsequent inspection optimization and enhancement to obtain a corresponding depth map.

8. The method for multi-spectral photometric stereo 3D reconstruction based on Gaussian process model according to claim 1, characterized in that The specific steps of Gaussian process modeling in step 5) are: S51: Establish the "surrogate model" formula, select the appropriate mean function and covariance function, and set the substitute initial hyperparameters according to the kernel function; S52: Input the multispectral photometric stereo depth map, and according to its objective function, use the gradient method to obtain the optimal hyperparameters and determine the posterior model; S53: Establish a "correction model" formula, select appropriate scaling factors and shift factors, and set initial hyperparameters for correction; S54: Input the light field camera depth map and calculate and correct initial hyperparameters; S55: Input the location to be predicted and obtain the final predicted value.

9. The method for multi-spectral photometric stereo 3D reconstruction based on Gaussian process model according to claim 8, characterized in that The model is built from the multispectral photometric stereo depth map: z PS ( x i ,y i )=f PS ( x i ,y i )+ε PS In the formula, f PS (x i ,y i ) represents the ideal value of the measured surface under the ideal condition of perfect fitting, ε PS represents the measurement error. The Gaussian process model is used to represent f PS (x i ,y i ): Among them, v i =(x i ,y i ), v j =(x j ,y j ); is the mean function; is the covariance function; the kernel function is the Matern kernel function, and we get: Among them, ‖v i -v j ‖ represents point v on the (x,y) plane i and v j The Euclidean distance between represents the constant variance of the Gaussian model, λ is the smoothing coefficient, when λ=0.5, the Matern kernel function is an exponential kernel, and when λ→∞, it is a Gaussian kernel; l PS is the length parameter of the kernel function; Γ is the Gamma function, K is the second kind of modified Bassel function; The alternative initial hyperparameters are obtained by the maximum likelihood method, and the conditional probability negative logarithm of the multispectral photometric stereo depth map is used as the objective function. The partial derivative of the hyperparameters is obtained and maximized by the conjugate gradient method to obtain the optimal solution of the alternative initial hyperparameters.

10. The method for multi-spectral photometric stereo 3D reconstruction based on Gaussian process model according to claim 8, characterized in that The "correction model" expression is: z LF ( x i ,y i )=ρ(x i ,y i )z′ PS ( x i ,y i )+δ(x i ,y i )+ε LF In the formula, i=1,2,…,n LF ,ρ(x i ,y i ) and δ(x i ,y i ) represent the scaling factor and the shift factor, ε LF Indicates error; p(x i ,y i )=ρ0+ρ1x i +p2y i δ(x i ,and i )~GP(δ0,k δ (v i ,v j )) You can get any position v on the (x,y) plane i =(x i ,y i ) can be expressed as: z′ LF ( x i ,y i )=ρ(x i ,y i )z′ PS ( x i ,y i )+δ(x i ,y i )。

Images