Three-dimensional reconstruction method, device and storage medium based on light field and photometric stereo

Abstract

The present invention provides a three-dimensional reconstruction method, device and storage medium based on light field and photometric stereo, which relates to the field of three-dimensional reconstruction technology, including: isolating ambient light, illuminating the object to be measured through multiple light sources; collecting color pictures of the object to be measured through a light field camera; performing photometric stereo reconstruction based on the color pictures to obtain a photometric stereo depth map; estimating the depth map based on the color pictures to obtain a light field camera depth map; using Fourier transform to perform frequency domain mapping, setting a threshold, and fusing the low-frequency part of the light field camera depth map with the high-frequency part of the photometric stereo depth map to obtain a three-dimensional reconstructed depth map. The present invention uses a simple and low-cost method to simultaneously introduce the advantages of light field into the field of photometric stereo, thereby improving the accuracy of three-dimensional reconstruction.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method, device and storage medium based on light field and photometric stereo. Background Art

[0002] In recent years, with the rapid development of digital technology, three-dimensional reconstruction technology has gradually become a research hotspot in the fields of computer vision and medical imaging. As a non-contact three-dimensional reconstruction method, photometric stereo technology has the characteristics of high resolution and high precision, and is widely used in the fields of object shape reconstruction and virtual reality. However, traditional photometric stereo technology has certain limitations when processing low-frequency information and complex surface materials, and it is difficult to accurately capture the texture and subtle structure of objects. In order to overcome the limitations of traditional photometric stereo technology, multispectral imaging technology has also been introduced into photometric stereo reconstruction. Multispectral imaging technology can obtain the reflectance spectrum information of objects in different bands, thereby providing richer color and material information. By combining multispectral imaging with photometric stereo, researchers hope to achieve more accurate capture of complex surface textures and details, and further improve the accuracy and realism of three-dimensional reconstruction. Against this background, multispectral photometric stereo technology came into being, aiming to make full use of the advantages of multispectral imaging and photometric stereo to achieve more refined and realistic three-dimensional reconstruction of objects. This technology is not only expected to promote the development of the field of three-dimensional reconstruction, but also has broad application prospects, including cultural relics protection, medical diagnosis and other aspects. Therefore, in-depth research on multispectral photometric stereo technology has important theoretical and practical significance. However, due to the principle limitations of photometric stereo, various forms of cumulative errors will occur in the reconstruction, which are reflected in the low-frequency area in the results, specifically manifested as distortion in some areas, which cannot meet the requirements for most scenes.

[0003] Therefore, how to improve the accuracy and realism of 3D reconstruction becomes a technical problem that needs to be solved. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art or related technology, and discloses a three-dimensional reconstruction method, device and storage medium based on light field and photometric stereo, which combines light field camera reconstruction with photometric stereo, gives full play to their respective advantages, realizes complementary effects and achieves higher accuracy.

[0005] The first aspect of the present invention discloses a three-dimensional reconstruction method based on light field and photometric stereo, comprising: isolating ambient light and illuminating an object to be measured through multiple light sources; collecting a color picture of the object to be measured through a light field camera; performing photometric stereo reconstruction based on the color picture to obtain a photometric stereo depth map; estimating the depth map based on the color picture to obtain a light field camera depth map; performing frequency domain mapping using Fourier transform, setting a threshold, and fusing the low-frequency part of the light field camera depth map with the high-frequency part of the photometric stereo depth map to obtain a three-dimensional reconstructed depth map.

[0006] In this technical solution, the light field camera can capture additional information about the direction and intensity of light. Compared with the ability of traditional cameras to only record the position and color of light passing through the lens, the light field camera can record the direction and intensity of light at these positions, thus providing richer and more comprehensive light information. The development of deep learning technology has also brought new possibilities for the processing and analysis of light field images. Using deep learning algorithms, the rich information contained in light field images can be better understood and utilized, such as achieving more accurate three-dimensional reconstruction, more accurate depth estimation, and a deeper understanding of light propagation and reflection. However, due to the limitations of the microlens array, the resolution may be reduced, affecting some high-frequency detail effects. In order to solve this problem, the present invention combines light field camera reconstruction with photometric stereo, giving full play to their respective advantages, achieving complementary effects, and achieving higher accuracy. In addition, in the restoration part of photometric stereo, a neural network model can also be used to assist in improving the restoration effect. There are also many different reconstruction algorithms for the light field camera restoration method, and there will be different filtering methods when high and low frequencies are fused, such as wavelet transform.

[0007] According to the three-dimensional reconstruction method based on light field and photometric stereo disclosed in the present invention, preferably, the step of estimating the depth map based on the color picture to obtain the depth map of the light field camera specifically includes: using the microlens array of the light field camera to collect pictures of the object to be measured from multiple perspectives at a single time; analyzing the sub-aperture image to perform distortion correction; estimating the depth map of the array of pictures of the object to be measured based on the sub-pixel displacement method of phase shift, setting the cost function for optimization, and performing subsequent inspection optimization and enhancement to obtain the depth map of the light field camera.

[0008] According to the three-dimensional reconstruction method based on light field and photometric stereo disclosed in the present invention, preferably, the specific steps of photometric stereo reconstruction include: extracting a central perspective image, performing channel separation on the color image to obtain a single-channel grayscale image corresponding to each light source; performing photometric stereo reconstruction based on the single-channel grayscale image to obtain a photometric stereo depth map.

[0009] According to the three-dimensional reconstruction method based on light field and photometric stereo disclosed in the present invention, preferably, the fusion method is filtering fusion: the photometric stereo depth map and the light field camera depth map are frequency-domain mapped by Fourier transform to obtain a frequency-domain image; the threshold is adaptively set by an algorithm for different objects, and after filtering the above-mentioned image with a frequency-domain filter, the low-frequency part of the light field camera depth map and the high-frequency part of the photometric stereo depth map are fused according to the weight coefficient, and the weight coefficient is adaptively adjusted according to the error after fusion to obtain a fused and corrected frequency-domain image; the final depth map is obtained by inverse Fourier transform. The threshold is adaptively set, specifically by identifying the proportion of high-frequency details on the frequency-domain image, and the threshold of the multi-spectral photometric stereo depth map is automatically adjusted; the weight coefficient is adaptively adjusted, specifically according to the composition of high- and low-frequency details in the input image.

[0010] The second aspect of the present invention discloses a three-dimensional reconstruction device based on light field and photometric stereo, comprising: a light shield, covering the outside of the object to be measured, for isolating ambient light; a lighting light source, multiple lighting light sources are arranged at multiple positions inside the light shield, for illuminating the object to be measured from multiple angles; a light field camera, arranged on the top of the light shield; the light field camera includes an image sensor, an imaging lens, and a microlens array; the image sensor is arranged on the top of the light shield, and generates a color image and an image containing light field information according to the received light signal; the imaging lens is arranged above the object to be measured inside the light shield, for collecting the reflected light of the object to be measured; the microlens array is arranged between the imaging lens and the image sensor, for obtaining the reflected light of the object to be measured at multiple angles.

[0011] According to the three-dimensional reconstruction device based on light field and photometric stereo disclosed in the present invention, preferably, three slide rails are provided on the light shield for installing the lighting light source, and the multiple slide rails are located in the same horizontal plane.

[0012] According to the three-dimensional reconstruction device based on light field and photometric stereo disclosed in the present invention, preferably, the image sensor is arranged in parallel with the microlens array, and the distance between the image sensor and the microlens array is equal to the focal length of the microlens array.

[0013] According to the three-dimensional reconstruction device based on light field and photometric stereo disclosed in the present invention, preferably, it also includes: a memory for storing program instructions; a processor for calling the program instructions stored in the memory to implement the three-dimensional reconstruction method based on light field and photometric stereo as any of the above technical solutions.

[0014] A third aspect of the present invention discloses a computer-readable storage medium storing a program code, wherein the program code is used to implement a three-dimensional reconstruction method based on light field and photometric stereo as described in any of the above technical solutions.

[0015] The beneficial effects of the present invention include at least: introducing a light field camera, using photometric stereo to fuse the results restored by the light field camera, solving the low-frequency distortion caused by the cumulative error of photometric stereo, and compared with the photometric stereo method, the light field camera containing a microlens array has a higher utilization rate of light field information, and the algorithm is used to process the captured pictures to obtain a good low-frequency reconstruction effect, and the low-frequency result is fused with the high frequency of photometric stereo to obtain the final three-dimensional reconstruction result, which improves the accuracy of three-dimensional reconstruction and can be widely used in the reconstruction of various scenes and objects. The present invention adopts a simple and low-cost method for improvement, and at the same time introduces the advantages of light field into the field of photometric stereo to make up for its shortcomings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic structural diagram of a three-dimensional reconstruction device based on light field and photometric stereo according to an embodiment of the present invention is shown.

[0017] Figure 2 A schematic diagram of a calculation process of photometric stereo reconstruction according to an embodiment of the present invention is shown.

[0018] Figure 3 A light field camera imaging principle diagram according to an embodiment of the present invention is shown.

[0019] Figure 4 A schematic diagram of a depth map filtering and fusion process according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0020] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and therefore, the present invention is not limited to the limitations of the specific embodiments disclosed below.

[0021] like Figure 1 As shown, the three-dimensional reconstruction device based on light field and photometric stereo according to the present invention includes: a light shield, covering the outside of the object to be measured (surface to be measured) and used to isolate ambient light; a lighting light source, multiple lighting light sources (red LED, green LED, blue LED) are arranged at multiple positions inside the light shield, and are used to illuminate the object to be measured from multiple angles; an image sensor is arranged on the top of the light shield, and generates a color image and an image containing light field information according to the received light signal; an imaging lens is arranged above the object to be measured inside the light shield, has an adjustable field of view, and is used to collect reflected light of the object to be measured; a microlens array is arranged between the imaging lens and the image sensor, and is used to obtain reflected light of the object to be measured at multiple angles.

[0022] In this embodiment, based on the basic RGB three-color multi-spectral photometric stereo method, a three-color light source is set for lighting, a color picture is taken by a camera, and then separated into three pictures by channel for photometric stereo reconstruction. The image sensor, the microlens array and the imaging lens form a light field camera, which can capture the light direction and intensity information of each pixel in the scene, thereby realizing applications such as depth estimation, depth of field adjustment and three-dimensional reconstruction. The microlens array can obtain an array of pictures of different perspectives on the surface of an object so that it can be reconstructed through a light field reconstruction algorithm.

[0023] According to the above embodiment, preferably, three slide rails are provided on the light shield for installing the lighting source, and the multiple slide rails are located on the same horizontal plane. The slide rails are embedded in the inner surface of the light shield, and the position of installing the light source can be selected according to the needs.

[0024] According to the above embodiment, preferably, the image sensor is arranged in parallel with the microlens array, and the distance between the image sensor and the microlens array is equal to the focal length of the microlens array.

[0025] According to the above embodiment, preferably, it also includes: a memory for storing program instructions; a processor connected to the memory and the image sensor, for calling the program instructions stored in the memory to implement a three-dimensional reconstruction method based on light field and photometric stereo: a color picture of the object to be measured is collected by a light field camera; photometric stereo reconstruction is performed based on the color picture to obtain a photometric stereo depth map; depth map estimation is performed based on the image array collected by the light field camera to obtain a light field camera depth map; frequency domain mapping is performed using Fourier transform, a threshold is set, and the low-frequency part of the light field camera depth map is merged with the high-frequency part of the photometric stereo depth map to obtain a three-dimensional reconstructed depth map. Since the restoration result is the relative depth of the surface, its true depth can also be obtained later by standard parts calibration.

[0026] like Figure 3 As shown, the working principle of the light field camera in the above embodiment includes: using a microlens array to collect the incident direction and intensity information of the light. The present invention adopts a light field camera of Lytro. The Lytro light field camera adopts a hexagonally arranged microlens array, which is placed on the main lens imaging plane of a traditional camera. At the same time, the photosensitive device is arranged in parallel with the microlens array, and the distance between them is exactly equal to the focal length of the microlens array. Finally, through the mapping effect of the microlens array, the light is projected onto the corresponding area to achieve image capture.

[0027] After obtaining the picture taken by the light field camera, perform the following steps for processing:

[0028] (1) Use Lytro's software to decode the image and export the refocusable image to a computer.

[0029] (2) Image demodulation: The light field camera used in the present invention contains 15*15 microlenses, which can demodulate images of 225 viewing angles. At the same time, the image of the central viewing angle is used as the input of photometric stereo for reconstruction.

[0030] (3) Calibrate the camera by processing the camera's built-in files, such as white images.

[0031] (4) Analyze sub-aperture images and perform distortion correction

[0032] (5) A sub-pixel displacement method based on phase shift is used to estimate the depth map. A cost function is set for optimization, and subsequent visual optimization and enhancement are performed to obtain the corresponding depth map.

[0033] According to another embodiment of the present invention, a three-dimensional reconstruction method based on light field and photometric stereo is also disclosed, including: isolating ambient light and illuminating the object to be measured through multiple light sources; collecting a color picture of the object to be measured through a light field camera; performing photometric stereo reconstruction based on the color picture to obtain a photometric stereo depth map; estimating the depth map based on the color picture to obtain a light field camera depth map; using Fourier transform to perform frequency domain mapping, setting a threshold, and fusing the low-frequency part of the light field camera depth map with the high-frequency part of the photometric stereo depth map to obtain a three-dimensional reconstructed depth map.

[0034] like Figure 4 As shown, according to the above embodiment, the step of fusing the low-frequency part of the light field camera depth map with the high-frequency part of the photometric stereo depth map specifically includes: performing frequency domain mapping on the photometric stereo depth map and the light field camera depth map through Fourier transform to obtain a frequency domain image; adaptively setting thresholds for different objects using an algorithm, filtering the above image using a frequency domain filter, fusing the low-frequency part of the light field camera depth map with the high-frequency part of the photometric stereo depth map according to a weight coefficient, and adaptively adjusting the weight coefficient according to the fused error to obtain a fused and corrected frequency domain image; and obtaining the final depth map through inverse Fourier transform. Adaptively setting the threshold is specifically to automatically adjust the threshold of the multi-spectral photometric stereo depth map by identifying the proportion of the high-frequency detail part on the frequency domain image; adaptively adjusting the weight coefficient is specifically to adaptively adjust according to the composition of high- and low-frequency details in the input image.

[0035] like Figure 2As shown, according to the above embodiment, preferably, the process of photometric stereo reconstruction specifically includes: the present invention also follows the assumption of traditional photometric stereo, that is, the incident light source is parallel light and the object is a Lambertian body. According to the principle of photometric stereo, no less than three pictures are required as input for calculation to perform three-dimensional reconstruction, while multispectral photometric stereo can avoid the tedious step of multiple lighting, and can also restore and reconstruct objects in motion, which has obvious advantages. The basic principles of photometric stereo are as follows:

[0036] Photometric stereo is a single-view measurement method that does not require the use of special conditions such as lasers or structured light, but only requires the use of ordinary light sources for illumination. As a 3D reconstruction technology based on image brightness information, the basic principle of photometric stereo is to infer the depth information of objects in the scene by comparing the image brightness information at different viewing angles. In the classic photometric stereo hypothesis, the object to be measured is assumed to be a Lambertian body, and the light source is assumed to be incident as parallel light. Here, it is assumed that the image brightness is I, the intensity of the light source is E, the incident direction of the light source is L, and the reflectivity of the object surface is , n represents the normal vector of the object surface, and m represents the intensity of the measured pixel (pixel value). According to the Lambertian body reflection formula, the following formula can be obtained:

[0037]

[0038] The light source direction can be obtained by calibration, such as a highlight ball. Three non-coplanar light source angles can form a 3*3 non-singular matrix that can be inverted. As a constant, As a whole N, , after ignoring the unknown scaling and assuming a unit intensity light source, a simplified Lambertian imaging model can be obtained:

[0039]

[0040]

[0041] Since n has unit length, the surface normal (vector direction) and albedo (vector length) can be estimated simultaneously, and the norm of n should be 1, so the following formula is available:

[0042]

[0043]

[0044] If more than three light sources are used in the experiment, the corresponding light source direction matrix is ​​no longer a square matrix, and the linear equations to be solved also become overdetermined:

[0045]

[0046] Using the pseudo-inverse of L, we can get an approximate solution in the least squares sense:

[0047]

[0048] The more light sources there are, the more robust the result will be. After that, the depth is estimated:

[0049] The normal vector n of the surface patch of the object in the figure must be orthogonal to v1 and v2. The patch corresponds to the pixel at (i, j) in the image plane, and the following relationship can be obtained:

[0050]

[0051] Finally, write it into a sparse matrix form to solve it:

[0052]

[0053] Finally, the depth value z corresponding to each pixel is obtained. Multispectral photometric stereo is to obtain three pictures corresponding to three light sources through a single shot of the top camera, and then use them as input for reconstruction. The present invention uses a light field camera instead of an ordinary camera, and uses the picture corresponding to the central microlens of the microlens array as the initial picture for subsequent operations, which simplifies the device and operation steps. According to the above steps, the normal vector and depth map reconstructed by multispectral photometric stereo can be obtained.

[0054] According to yet another embodiment of the present invention, a computer-readable storage medium is disclosed. The computer-readable storage medium stores program codes, and the program codes are used to implement the three-dimensional reconstruction method based on light field and photometric stereo as in the above embodiment.

[0055] All or part of the steps in the various methods of the above embodiments can be completed by controlling the relevant hardware through a program, and the program can be stored in a readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other readable medium that can be used to carry or store data.

[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method based on light field and photometric stereo, characterized in that: include: Isolate ambient light and illuminate the object to be tested through multiple light sources; Collecting a color picture of the object to be measured by a light field camera; Perform photometric stereo reconstruction based on the color image to obtain a photometric stereo depth map; Performing depth map estimation based on the color image to obtain a light field camera depth map; Fourier transform is used to perform frequency domain mapping, a threshold is set, and a low-frequency part of the light field camera depth map is fused with a high-frequency part of the photometric stereo depth map to obtain a three-dimensional reconstructed depth map, wherein the fusion is a filtering fusion: the photometric stereo depth map and the light field camera depth map are frequency-domain mapped by Fourier transform to obtain a corresponding frequency domain image; an algorithm is used to adaptively set a threshold for different objects, and after filtering the frequency domain image using a frequency domain filter, the low-frequency part of the light field camera depth map and the high-frequency part of the photometric stereo depth map are fused according to a weight coefficient, and the weight coefficient is adaptively adjusted according to the error after fusion to obtain a fused and corrected frequency domain image; finally, a final depth map is obtained by inverse Fourier transform; In the classic photometric stereo hypothesis, the object to be measured is assumed to be a Lambertian body, the light source is assumed to be incident with parallel light; the intensity of the light source is E, the incident direction of the light source is L, and the reflectivity of the object surface is ρ , n represents the normal vector of the object surface, and m represents the intensity of the measured pixel; according to the Lambertian body reflection formula, the following formula is obtained: m = EL · ρ n。 2. The three-dimensional reconstruction method based on light field and photometric stereo according to claim 1, characterized in that: The step of estimating the depth map according to the color image to obtain the light field camera depth map specifically includes: The microlens array of the light field camera is used to collect images of the object to be measured at multiple viewing angles at a time; the sub-aperture image is analyzed and distortion correction is performed; and the depth map of the array of images of the object to be measured is estimated based on a sub-pixel displacement method of phase shift to obtain the depth map of the light field camera.

3. The 3D reconstruction method based on light field and photometric stereo according to claim 1, characterized in that: The specific steps of the photometric stereo reconstruction include: Extracting a central viewing angle image, performing channel separation on the color image, and obtaining a single-channel grayscale image corresponding to each light source; Photometric stereo reconstruction is performed according to the single-channel grayscale image to obtain the photometric stereo depth map.

4. A three-dimensional reconstruction device based on light field and photometric stereo, characterized in that: include: A light shield, covering the outside of the object to be tested, is used to isolate ambient light; A lighting light source, wherein a plurality of the lighting light sources are arranged at a plurality of positions inside the light shield, and are used to illuminate the object to be measured from a plurality of angles; A light field camera, arranged on top of the light shield; The light field camera includes an image sensor, an imaging lens, and a microlens array; The image sensor is arranged on the top of the light shield and generates a color image and an image containing light field information according to the received light signal; The imaging lens is disposed above the object to be measured inside the light shield and is used to collect reflected light from the object to be measured; The microlens array is disposed between the imaging lens and the image sensor, and is used to obtain reflected light of the object to be measured at multiple angles; A memory for storing program instructions; A processor is connected to the memory and the image sensor, and is used to call the program instructions stored in the memory to implement the three-dimensional reconstruction method based on light field and photometric stereo according to any one of claims 1 to 3.

5. The 3D reconstruction device based on light field and photometric stereo according to claim 4, characterized in that: The light shield is provided with three slide rails for mounting the lighting source, and the plurality of slide rails are located on the same horizontal plane.

6. The 3D reconstruction device based on light field and photometric stereo according to claim 4, characterized in that: The image sensor is arranged in parallel with the microlens array, and the distance between the image sensor and the microlens array is equal to the focal length of the microlens array.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code, and the program code is used to implement the three-dimensional reconstruction method based on light field and photometric stereo according to any one of claims 1 to 3.

Images