Active light field three-dimensional imaging method based on phase guidance

Abstract

The invention belongs to the technical field of three-dimensional imaging, and particularly relates to an active light field three-dimensional imaging method based on phase guidance. The phase information is introduced into the light field stereo matching process, and the spatial resolution loss of the light field camera is compensated through phase guidance measurement, so that the bottleneck problem that the existing light field camera is limited by the sub-aperture resolution to cause the low resolution of the depth map is solved. Based on effective fusion of the light field and the structured light, three-dimensional reconstruction with high pixel resolution can be realized in single shooting, the robustness and precision of depth estimation can be ensured, and the method has wide application potential in the fields of precision measurement, industrial detection, three-dimensional modeling and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional imaging, and in particular relates to an active light field three-dimensional imaging method based on phase guidance. Background Art

[0002] Light field cameras can record the spatial and angular information of a scene in a single exposure, thereby achieving three-dimensional imaging. They have a wide range of applications in industrial inspection, cultural heritage digitization, and robot navigation. However, due to the need to trade off spatial resolution when obtaining angular information, existing light field cameras usually suffer from low spatial resolution and limited depth accuracy. Existing methods have improved the reconstruction algorithm of light field images or introduced deep learning methods to improve the quality of depth maps, but this passive light field imaging technology still finds it difficult to simultaneously improve spatial resolution and depth accuracy. Active optical imaging technologies such as structured light projection can obtain high-precision depth information in traditional three-dimensional imaging, but these methods still face the difficulties of accurately calibrating the complex imaging model of the light field camera and the difficulty of using the phase information of the projected fringes to guide light field parallax matching in the application of light field cameras. Summary of the Invention

[0003] In order to solve the above problems, the present invention proposes an active light field three-dimensional imaging method based on phase guidance.

[0004] The technical solution of the present invention is:

[0005] An active light field three-dimensional imaging method based on phase guidance is characterized by comprising the following steps:

[0006] S1. Use a light field camera and a projector to acquire images, and determine the external parameter relationship between the light field camera and the projector through stereo calibration. Specifically, the main lens of the light field camera is regarded as a virtual camera, and a standard checkerboard calibration plate is used as a stereo calibration target. By obtaining multiple sets of calibration plate images observed by the projector and the light field camera at the same time, the relative posture and projection matrix between the projector and the virtual main camera of the light field camera, that is, the intrinsic and extrinsic parameters M of the projector, are calculated. p And the internal and external parameters M of the virtual camera c ;

[0007] S2. Calibrate the light field camera using the deformable cone model. The model is defined as:

[0008]

[0009] in, is the imaging distance in the light field camera, v is the virtual depth, a is the calibration parameter vector, and p is the incident angle feature vector, where θ x and θ yis the incident angle of the principal ray in the x and y directions, k is the depth-dependent distortion gain coefficient, and d is the axial aberration compensation distance, which are obtained by least squares fitting.

[0010] To calibrate the parameter a, the calibration plate is scanned M times within the required depth range. T microlens pixels are selected as target points from each scan, and the calibration parameter is solved by the nonlinear least squares method:

[0011]

[0012] Where z is a vector containing the true depth values ​​of all calibration points, is the axial aberration compensation distance, θ x,m,t and θ y,m,t is the component of the incident angle of the light corresponding to the t-th microlens in the x-direction and y-direction in the m-th scan. m,t is the virtual depth corresponding to the t-th microlens in the m-th scan;

[0013] After obtaining the calibration parameters, you can determine Substitute the imaging model to calibrate the light field camera:

[0014]

[0015] Where Z is the measured depth, that is, the actual physical distance of the object from the optical center of the main lens, D is the parallax between two adjacent microlens subviews in adjacent microlens images, and D μ is the diameter of the microlens in pixels, d μ is the distance between the sensor plane and the microlens array plane, f L is the focal length of the main lens;

[0016] S3, fringe frequency selection, that is, according to the expected measurement depth range [Z min ,Z max ] and the microlens parameter D μ and d μ , calculate the corresponding disparity range [D min ,D max In order to ensure that the phase is single-valued and easy to unfold within the parallax range, the frequency of the high-frequency sinusoidal stripes is selected as:

[0017]

[0018] Where N is the number of phase shift steps.

[0019] S4. Micro-lens stereo matching based on phase guidance, specifically:

[0020] In the phase-guided microlens stereo matching process, the phase shift algorithm is first applied to each microlens sub-image of the light field data corrected by the deformable cone model to obtain the wrapped phase φ wrp (x, y). After that, a set of adjacent microlens sub-images (circled by red solid lines) are used as units to perform phase-driven stereo matching on the same spatial point at their corresponding pixel positions. Specifically, for each microlens center pixel In the depth range [Z min ,Z max ]The parallax range obtained by reverse calculation [D min ,D max ] and calculate the matching cost in the w×w neighborhood E of the center pixel:

[0021]

[0022] Where ε is The set of all legal pixels in the area of ​​w×w as the center, N ε is the number of pixels in the set ε, W(s c ) is a pre-set weighted window function used to give different weights to adjacent viewpoints, Δφ(s c ,D) represents the phase difference of the corresponding pixel when the disparity is assumed to be D, and τ2 is the truncation threshold, which is used to limit the maximum value of the cost to reduce the impact of noise and phase discontinuity.

[0023] S5. Calculate the initial depth map: Based on the matching results obtained in S4, calculate the effective matching point distance D to obtain the initial depth map Z r ;

[0024] S6. Calculate the initial point cloud: Use the obtained initial depth map Z r Generate corresponding three-dimensional point cloud;

[0025] S7, generating a reference depth map: reprojecting, interpolating, and filtering the initial point cloud obtained in S6 to generate a reference depth map;

[0026] S8, phase unwrapping: in S7, the reference depth value Z corresponding to the reprojected image pixel is obtained. r (z,y) and the wrapped phase image φ in the same view wrp (x, y), the reference depth information is used to determine the phase unwrapping order of each pixel, thereby eliminating the 2π discontinuity caused by the periodicity of high-frequency fringes. Specifically, first, according to the reprojected three-dimensional coordinates (X r ,Y r ,Z r ) and the mapping coefficients in the calibration matrix, the theoretical number of phase cycles corresponding to each pixel can be calculated:

[0027]

[0028] The numerator represents the product of the continuous phase obtained by calibration mapping and the projection frequency f, and the denominator H p is the phase quantization constant of the projection system. Then, the wrapped phase is unfolded using the phase order k0 to obtain the absolute phase of the pixel:

[0029]

[0030] S9, final point cloud computing: After completing S8, the absolute phase Φ(x, y) and the corresponding initial depth Z of each pixel are obtained r First, the unwrapped phase is mapped to the corrected axial distance using the triangulation formula based on the camera intrinsic parameters and pixel coordinates (u,v).

[0031]

[0032] And further through

[0033]

[0034] Restore the true depth. Then, project the depth Z9x,y) into the camera coordinate system and calculate the three-dimensional coordinates of each pixel

[0035]

[0036] where f x and f y Indicates the effective focal length of the camera in the direction of the two imaging axes x and y, (c x ,c y ) represents the intersection of the ideal optical axis (chief ray) and the pixel grid projected on the two-dimensional image plane, in pixels.

[0037] The beneficial effects of this invention are as follows: it introduces phase information into the light field stereo matching process, compensating for the spatial resolution loss of light field cameras through phase-guided measurements, thereby overcoming the bottleneck problem of existing light field cameras, which are limited by sub-aperture resolution and result in low depth map resolution. The effective fusion of light field and structured light not only enables high-pixel resolution 3D reconstruction in a single shot, but also ensures the robustness and accuracy of depth estimation, with potential for broad application in precision measurement, industrial inspection, and 3D modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be described in detail below with reference to the accompanying drawings.

[0040] This paper combines structured light projection technology with a light field camera, effectively improving the spatial resolution and depth accuracy of three-dimensional imaging through phase guidance. It mainly includes the following three core modules: 1) Structured light field system calibration, which adopts a light field camera calibration method based on a deformable cone model. By correcting the axial aberration of the light field camera's main lens, high-precision geometric parameter estimation is achieved, effectively improving the imaging consistency of the light field camera at different depth planes; 2) Phase-guided stereo matching, which uses multi-frequency phase-shifted stripes to obtain the wrapped phase and designs a phase-guided sum of absolute difference matching cost. By combining the phase difference with the geometric consistency between spatial neighborhood pixels, fine stereo matching of light field microlens images is achieved, significantly reducing the matching error in occluded areas and depth discontinuity areas; 3) Reprojection and depth refinement, based on the initially generated three-dimensional point cloud, through reprojection correction and phase refinement of the depth map, the sub-aperture images obtained by the light field camera are refocused to a unified depth plane. Combined with the phase unfolding of high-frequency stripes, global optimization of the three-dimensional reconstruction results and high-precision detail recovery are achieved.

[0041] like Figure 1 As shown, the present invention mainly includes the following steps:

[0042] 1. Image preparation:

[0043] 1. Stereo calibration

[0044] Stereo calibration aims to determine the external parameter relationship between the light field camera and the projector. In the SLF system of the present invention, the present invention uses the traditional binocular stereo calibration method, regards the main lens of the light field camera as a virtual camera, and forms a pair of stereo systems with the projector. The present invention uses a standard checkerboard calibration plate as a stereo calibration target. By obtaining multiple sets of calibration plate images observed simultaneously by the projector and the light field camera, the relative posture and projection matrix between the projector and the virtual main camera of the light field camera, that is, the intrinsic and extrinsic parameters M of the projector, can be calculated. p And the internal and external parameters M of the virtual camera c The result of stereo calibration provides the mapping relationship between the projector coordinate system and the camera coordinate system for subsequent depth calculation.

[0045] 2. Light Field Calibration

[0046] An ideal light field camera follows the following linear imaging model:

[0047]

[0048] Where z is the imaging distance of the target three-dimensional point (X, Y, Z), v is the virtual depth, D is the distance between the corresponding points in the adjacent microlens images, and Dμ is the diameter of the microlens in pixels, d μ is the distance between the sensor plane and the microlens array plane, f L =The focal length of the main lens. However, due to the presence of aberrations in the main lens (especially field curvature and astigmatism), the actual optical imaging process of the light field camera will experience depth distortion, which significantly reduces the accuracy of light field 3D imaging. This phenomenon is mainly caused by axial aberrations, including field curvature and astigmatism. Therefore, assuming that field curvature and astigmatism are the incident angle of the main ray (θ x ,θ y ), θ x and θ y The incident angle of the principal ray in the x and y directions, and the imaging distance (equivalent to the virtual depth v) is also a function of the optical axis aberration. The present invention proposes a deformed cone model to characterize the imaging distance in the light field camera The model expression is as follows:

[0049]

[0050] Among them, a is the calibration parameter vector, p is the incident angle characteristic vector, and the specific expression is

[0051]

[0052] To calibrate parameter a, the present invention scans the calibration plate M times within the desired depth range. Then, T microlens pixels are selected as target points from each scan. Therefore, the present invention can solve the calibration parameter a using a nonlinear least squares method:

[0053]

[0054] in is the axial aberration compensation distance. Through the above calibration process, the axial aberration in light field 3D imaging can be effectively compensated, thereby significantly improving the accuracy of depth reconstruction. By replacing z in formula (1), the depth distortion caused by axial aberration can be eliminated.

[0055] 3. Stripe frequency selection

[0056] That is, according to the expected measurement depth range [Z min ,Z max ] and the microlens parameter D μ and d μ , calculate the corresponding disparity range [D min ,D max In order to ensure that the phase is single-valued and easy to unfold within the parallax range, the frequency of the high-frequency sinusoidal fringe is selected as

[0057]

[0058] Where N is the number of phase shift steps.

[0059] 2. Calculation of initial depth map and point cloud:

[0060] 1. Microlens Stereo Matching Based on Phase Guidance

[0061] Phase is considered a robust feature and is used for stereo matching between adjacent microlens images. Before performing phase-guided stereo matching, to avoid invalid calculations, the present invention selects six microlens images encircled by red solid lines for phase-guided stereo matching. Due to the directional nature of phase, the present invention excludes microlenses that meet the following conditions:

[0062]

[0063] Where g=[G x ,G y ] is the phase gradient, G x and G y are the median of the horizontal and vertical gradients in a 7×7 window, respectively, and u is the unit vector from the template point to the target microlens. This condition ensures that the phase gradient has sufficient components in the epipolar direction to avoid invalid matching.

[0064] 2. Phase-guided stereo matching method

[0065] Unlike traditional grayscale image stereo matching, the phase matching algorithm focuses on the absolute phase value rather than the gradient or relative difference. Therefore, in phase matching, the sum of absolute differences is used as the matching cost, which is suitable for block-based matching processes. However, in microlens images, when there are occluders, the occluded areas will significantly interfere with the matching of unoccluded areas, thereby causing depth discontinuity errors. In order to suppress depth discontinuity errors, the present invention proposes a stereo matching algorithm based on the sum of phase-guided absolute differences. The algorithm is based on two key assumptions - phase adjacency and pixel adjacency, that is, in the wrapped phase φ, when the phase difference of adjacent pixel points is close to 0 or 2π, or points that are close in space are more likely to be adjacent in three-dimensional space. Therefore, the cost function is weighted by spatial adjacency, and the template pixels are weighted by the spatial adjacency. The phase surface of is modeled as:

[0066]

[0067] where s c =(x c ,y c ), the index of any 2D pixel in the phase image, g j Template pixels The final phase-guided stereo matching cost is:

[0068]

[0069] Where ε is The set of all legal pixels in the area of ​​w×w as the center, N ε is the number of pixels in the set ε, W(s c ) is a pre-set weighted window function used to give different weights to adjacent viewpoints, Δφ(s c ,D) represents the phase difference of the corresponding pixel when the parallax is D, τ2 is the cutoff threshold, which is used to limit the maximum value of the cost to reduce the impact of noise and phase discontinuity. By minimizing The disparity D corresponding to each pixel can be calculated.

[0070] In calculation When , the present invention accumulates the matching costs of adjacent viewpoints for each possible disparity D, and takes the smaller value between the weighted phase difference in the adjacent viewpoints and the threshold τ1 for accumulation. This strategy ensures that when the phase difference is very large (for example, due to occlusion or noise), the contribution of the viewpoint to the overall cost is limited and does not excessively affect the matching result. Ultimately, by finding the minimum cost within a certain disparity search range, the present invention determines the optimal disparity estimate of the central viewpoint pixel. Due to the introduction of phase information and truncation mechanism, the phase-guided stereo matching cost shows higher robustness in depth discontinuous areas on the object surface, which can effectively avoid traditional methods from making incorrect matches in these areas. For each adjacent pixel point in the microlens image, after performing stereo matching, the effective matching point distance D is calculated to obtain the initial depth map Z r .

[0071] 3. Initial 3D point cloud computing

[0072] The initial depth map calculated above can be used to generate a corresponding 3D point cloud. Based on the parameters obtained from stereo calibration, each pixel on the depth map is back-projected into 3D space through beam triangulation. Specifically, the pixel coordinates (u, v) and their corresponding depth Z on the depth map are converted into 3D coordinates (X, Y, Z) in the camera coordinate system using the intrinsic parameters of the light field camera. Because the depth map resolution reaches the pixel level of the sub-aperture image, the initial 3D point cloud obtained by this method also has high density and high precision.

[0073] 3. Reprojection and Refined Imaging:

[0074] 1. Generate a reference depth map

[0075] In order to reconstruct an accurate 3D point cloud, this paper proposes a reprojection and refinement strategy. First, the initial sparse point cloud (X r (i),Y r (i),Z r (i)) Reprojects onto the projector pixel plane to generate an aligned but sparse depth map. For the i-th 3D point, the projector matrix M is used p have:

[0076]

[0077] Among them (s c ,t c ) is the reprojected position of the point in the projector pixel coordinate system. Subsequently, in order to fill the holes in the reprojected depth map, the present invention calculates the depth of each pixel to be interpolated (s c ,t c ) in the neighborhood window U, the edge-preserving interpolation strategy is adopted to convert the depth value Z of the hole into r (s c ) is calculated as

[0078]

[0079] The neighborhood window U is defined as

[0080]

[0081] Where Z0 is the distance (s c ,t c ) the nearest valid depth value, σ z is the standard deviation constant of the depth interpolation weight, which is 3 here to quickly attenuate the interpolation weight at the depth mutation point to preserve edge details. After 3×3 median filtering and multiple 11×11 side window frame filtering, the reference depth value Z is obtained. r .

[0082] 2. Phase unwrapping

[0083] Secondly, based on the stereo matching results, the captured fringe image is refocused and the wrapped phase is calculated. For each pixel point (s c ,t c ), the angular coordinates of the homologous points in the original light field image can be calculated by the following formula:

[0084]

[0085] Where k is the ratio coefficient between the original image and the refocused image, σ c is the pixel size of the light field camera, d is the object distance (the distance from the object to the lens), v is the virtual depth, dμ is the distance from the microlens to the sensor, f x and f y is the focal length of the light field camera in the x and y directions, α c and β c are the pixel center coordinates in the refocused image.

[0086] Then, by utilizing the analytical relationship between depth and distance between corresponding points, the reference depth map Z r Convert it into a densely aligned corresponding point distance map, thereby refocusing the captured fringe image. Retrieve the high-frequency phase that needs to be unwrapped from the refocused image. The phase unwrapping order k0 determined by the reference depth map is:

[0087]

[0088] The absolute phase Φ is obtained as follows:

[0089]

[0090] 3. Obtain the absolute phase Φ(x,y) and corresponding reference depth Z of each pixel r First, according to the camera intrinsic parameters and pixel coordinates (u, v), the unwrapped phase is mapped to the corrected axial distance using the triangulation formula

[0091]

[0092] And further through

[0093]

[0094] Restore the true depth. Then, project the depth Z(x,y) into the camera coordinate system and calculate the three-dimensional coordinates of each pixel.

[0095]

[0096] where f x and f y Indicates the effective focal length of the camera in the direction of the two imaging axes x and y, (c x ,c y ) represents the intersection of the ideal optical axis (chief ray) and the pixel grid projected on the two-dimensional image plane, in pixels.

Claims

1. A phase-guided active light field three-dimensional imaging method, characterized in that: The following steps are involved: S1. Use a light field camera and a projector to acquire images, and determine the external parameter relationship between the light field camera and the projector through stereo calibration. Specifically, the main lens of the light field camera is regarded as a virtual camera, and a standard checkerboard calibration plate is used as a stereo calibration target. By obtaining multiple sets of calibration plate images observed by the projector and the light field camera at the same time, the relative posture and projection matrix between the projector and the virtual main camera of the light field camera, that is, the intrinsic and extrinsic parameters M of the projector, are calculated. p And the internal and external parameters M of the virtual camera c ; S2. Calibrate the light field camera using the deformable cone model. The model is defined as: in, is the imaging distance in the light field camera, v is the virtual depth, a is the calibration parameter vector, and p is the incident angle feature vector, where θ x and θ y is the incident angle of the principal ray in the x and y directions, k is the depth-dependent distortion gain coefficient, and d is the axial aberration compensation distance, which are obtained by least squares fitting; To calibrate the parameter a, the calibration plate is scanned M times within the required depth range. T microlens pixels are selected as target points from each scan, and the calibration parameter is solved by the nonlinear least squares method: Where z is a vector containing the true depth values ​​of all calibration points, is the axial aberration compensation distance, θ x,m,t and θ y,m,t is the component of the incident angle of the light corresponding to the t-th microlens in the x-direction and y-direction in the m-th scan, v m,t is the virtual depth corresponding to the t-th microlens in the m-th scan; After obtaining the calibration parameters, you can determine Substitute the imaging model to calibrate the light field camera: Where Z is the measured depth, that is, the actual physical distance of the object from the optical center of the main lens, D is the parallax between two adjacent microlens subviews in adjacent microlens images, and D μ is the diameter of the microlens in pixels, d μ is the distance between the sensor plane and the microlens array plane, f L is the focal length of the main lens; S3, fringe frequency selection, that is, according to the expected measurement depth range [Z min ,Z max ] and the microlens parameter D μ and d μ , calculate the corresponding disparity range [D min ,D max ], in order to ensure that the phase is single-valued and easy to unfold within the parallax range, the frequency of the high-frequency sinusoidal fringe is selected as: Where N is the number of phase shift steps; S4. Micro-lens stereo matching based on phase guidance, specifically: In the phase-guided microlens stereo matching process, the phase shift algorithm is first applied to each microlens sub-image of the light field data corrected by the deformable cone model to obtain the wrapped phase φ wrp (x, y); thereafter, a phase-driven stereo matching is performed on the same spatial point at their corresponding pixel positions, taking a group of adjacent microlens sub-images as units. Specifically, for each microlens center pixel In the depth range [Z min ,Z max ]The parallax range obtained by reverse calculation [D min ,D max ] and calculate the matching cost in the w×w neighborhood E of the center pixel: Where ε is The set of all legal pixels in the area with a size of w×w and a center, N ε is the number of pixels in the set ε, W(s c ) is a pre-set weighted window function used to give different weights to adjacent viewpoints, Δφ(s c ,D) represents the phase difference of the corresponding pixel when the disparity is assumed to be D, τ2 is the truncation threshold, which is used to limit the maximum value of the cost to reduce the impact of noise and phase discontinuity; S5. Calculate the initial depth map: Based on the matching results obtained in S4, calculate the effective matching point distance D to obtain the initial depth map Z r ; S6. Calculate the initial point cloud: Use the obtained initial depth map Z r Generate corresponding three-dimensional point cloud; S7, generating a reference depth map: reprojecting, interpolating, and filtering the initial point cloud obtained in S6 to generate a reference depth map; S8, phase unwrapping: in S7, the reference depth value Z corresponding to the reprojected image pixel is obtained. r (z,y) and the wrapped phase image φ in the same view wrp (x, y), the reference depth information is used to determine the phase unwrapping order of each pixel, thereby eliminating the 2π discontinuity caused by the periodicity of high-frequency fringes; specifically, first according to the reprojected three-dimensional coordinates (X r ,Y r ,Z r ) and the mapping coefficients in the calibration matrix to calculate the theoretical phase cycle number corresponding to each pixel: The numerator represents the product of the continuous phase obtained by calibration mapping and the projection frequency f, and the denominator H p is the phase quantization constant of the projection system; the wrapped phase is then unfolded using the phase order k0 to obtain the absolute phase of the pixel: S9, final point cloud computing: After completing S8, the absolute phase Φ(x, y) and the corresponding initial depth Z of each pixel are obtained r (x, y); Based on the camera intrinsic parameters and pixel coordinates (u, v), the unwrapped phase is mapped to the corrected axial distance using the triangulation formula And further through Restore the true depth; then, project the depth Z(x,y) into the camera coordinate system to calculate the three-dimensional coordinates of each pixel where f x and f y Indicates the effective focal length of the camera in the direction of the two imaging axes x and y, (c x ,c y ) represents the intersection of the ideal optical axis and the pixel grid projected on the two-dimensional image plane, in pixels.

Images