A gradient field integration depth reconstruction method based on divergence compensation optimization

Abstract

The present invention discloses a gradient field integral depth reconstruction method based on divergence compensation optimization, including: determining the format of the gradient difference operator; solving the point spread function of the single pixel divergence of the gradient field; using a polarization vision sensor to obtain a polarization image, and calculating the target surface gradient data and the target mask; calculating the iterative initial value of the reconstructed depth field and its gradient field; using the error between the reconstructed gradient field and the target surface gradient data to determine the divergence compensation amount; using the point spread function of the single pixel divergence and the divergence compensation amount to compensate the reconstructed depth field and gradient field; iteratively optimizing the reconstructed depth field and gradient field. The present invention effectively improves the reconstruction accuracy through iteration, meets the accuracy requirements in practical applications, and can use fast Fourier transform to accelerate during the integration and convolution operations, effectively improving the rapidity.

Description

Technical Field

[0001] The invention relates to the technical field of polarization vision three-dimensional imaging detection, and in particular to a gradient field integral depth reconstruction method based on divergence compensation optimization. Background Art

[0002] Polarization 3D imaging uses the polarization state of the light emitted from the surface of an object to analyze the direction of the normal vector of the object surface, and restores the surface shape of the object through the normal vector field. Different from conventional vision-based 3D measurement technology, polarization 3D imaging can obtain more effective information when measuring highly reflective, weakly textured, and transparent objects, thereby completing the measurement work when conventional vision 3D measurement technology fails. Restoring the depth field from the normal vector field can be converted into a gradient field integration problem. At present, due to errors caused by discrete sampling, signal noise, etc., the gradient field obtained through polarization information usually has the problem of non-integrability. Therefore, it is worth studying to overcome the non-integrability of the gradient field and complete the reconstruction of the depth field of the target object from the normal vector field.

[0003] Gradient field integration algorithms can be roughly divided into four categories: path-based algorithms, Fourier transform-based algorithms, direct solution-based algorithms, and iteration-based methods. Path-based gradient field integration algorithms usually select a suitable path in the gradient map to start integration from a common pixel point. For example, Wu et al. proposed a line-by-line integration method in the article "A Line-integrationBased Method for Depth Recovery from Surface Normals", and Fraile et al. proposed a minimum spanning tree path method in the article "Combinatorial Surface Integration". These methods usually have a high running speed, but are very sensitive to noise and discontinuity. Robles-Kelly et al. proposed an improved method in the article "Surface Height Recovery from Surface Normals Using Manifold Embedding", averaging the integration results on different paths between each pair of pixels to alleviate the impact of noise, but the accuracy of this method is limited and has a high complexity. Fourier transform-based gradient field integration algorithms usually implement the integrability enhancement and integration of the gradient field in the frequency domain. Frankot et al. proposed the Frankot-Chellappa gradient field integration algorithm in the article "A Method for EnforcingIntegrability in Shape from Shading Algorithms" to solve the problem of non-integrable gradient fields, realize the least squares fitting from non-integrable gradient fields to integrable gradient fields and reconstruct the depth. This method uses the two-dimensional Fourier transform to transfer the least squares to the frequency domain for processing, and uses an implicit strategy to achieve integration, with a fast operation speed. This method has been widely used in the field of polarization vision three-dimensional imaging. On the basis of the Frankot-Chellappa algorithm, the accuracy can be further improved by improving the differential sampling method, such as the improved differential sampling method proposed by Sun Zhuang et al. in the article "High-precision gradient field integration method used in polarization reconstruction methods". The Fourier transform-based method usually spreads local non-integrable errors and noise errors to a larger area, which is easy to cause a large range of distortion, and it has a periodic requirement for the reconstructed depth map.The Shapelets algorithm proposed by Kovesi in the article "Shapelets Correlated with Surface Normals Produce Surfaces" is similar to the Frankot-Chellappa algorithm. The algorithm estimates the decomposition of a set of multi-scale Gaussian basis functions from the gradient field, uses correlation operations to obtain the component sizes of different basis functions, and thus reconstructs the depth map. Based on the direct solution method, the gradient integral problem is simplified to solving a large-scale linear equation system, and the depth value of each pixel is solved as an independent variable. In the article "Chellappa R, What is the Range of Surface Reconstructions from a Gradient Field?", Agrawal et al. transformed the gradient integral problem into solving the discrete Poisson equation and reconstructed the depth field from the divergence of the gradient field. Compared with the Fourier transform-based method, this method can add additional weight information to each pixel to overcome the shortcomings of the Fourier transform, but for large-sized images, solving the equation system requires quite high computing resources. The iterative method starts from an initial depth map and uses constraints to perform step-by-step optimization. For example, the iterative optimization algorithm proposed by Horn et al. in the article "The Variational Approach to Shape from Shading" combines the Laplacian operator with the divergence of the gradient field to construct an iterative formula for the depth map, which can converge to the best result. However, there is no easy way to obtain suitable boundary conditions and initial solutions, which can easily lead to slow convergence. Horovitz et al. improved the algorithm of Horn et al. in the article "Depth from Gradient Fields and Control Points: Bias Correction in Photometric Stereo", using a coarse-to-fine multi-grid method to speed up the iterative convergence speed, and used the known additional depth information to guide the reconstructed depth map in the form of sparse control points. In addition, Agrawal et al. in the article “An Algebraic Approach to Surface Reconstruction from Gradient Fields” attempted to use graph theory to utilize the information contained in the curl in the gradient field to enhance the integrability of the gradient field. This method divides the gradient map into two regions according to the curl value, and then estimates the residual gradient field in the high curl region to compensate the original gradient map.

[0004] Analyzing the current gradient field integral algorithm, if we pursue the rapidity of the gradient field integral, the accuracy requirement is difficult to meet; if we use a more complex solution method, we will encounter the problem of taking too long while obtaining better results. Therefore, how to ensure the high accuracy of the gradient field integral algorithm while having good rapidity is a problem worth studying. Summary of the invention

[0005] Based on the above background, the main purpose of the present invention is to provide a gradient field integral depth reconstruction method based on divergence compensation optimization, which can ensure high accuracy while having a certain degree of rapidity.

[0006] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] A gradient field integral depth reconstruction method based on divergence compensation optimization, the method comprising:

[0008] Step a: Under the premise of determining the gradient difference operator format, solving the point spread function of the single pixel divergence of the gradient field, the point spread function includes the depth field and the gradient field generated by the single pixel divergence field;

[0009] Step b, using the target surface gradient data to solve the initial value of the reconstructed depth field and the gradient field of the reconstructed depth field;

[0010] Step c, calculating the divergence compensation amount using the error between the reconstructed gradient field and the target surface gradient data;

[0011] Step d, compensating the reconstructed depth field and its gradient field using the divergence compensation amount;

[0012] Step e: repeating step c and step d, obtaining a compensated and optimized depth field after multiple iterations, and completing the reconstruction of the depth field.

[0013] Furthermore, the gradient difference operator used in step a is represented in the frequency domain; the point spread function includes the depth field and gradient field generated by the single pixel divergence field, which are obtained by solving the two-dimensional discrete Poisson equation.

[0014] Furthermore, in step b, the Frankot-Chellappa algorithm is used to solve the initial value of the depth field and the gradient field of the depth field, wherein the gradient difference operator is consistent with the format in step a.

[0015] Furthermore, the divergence compensation amount calculated in step c is obtained by convolving the error of the gradient field with the point spread function of the single pixel divergence.

[0016] Furthermore, the divergence compensation in step d acts on the point spread function of the single pixel divergence, and the result is directly added to the reconstructed depth field and its gradient field as compensation. The basis for determining whether the iteration is terminated in step e is the error of the reconstructed gradient field and the number of iterations.

[0017] The advantages of the present invention compared with the prior art are:

[0018] The present invention proposes a novel gradient field integral depth reconstruction algorithm based on divergence compensation optimization. First, the format of the gradient difference operator in the frequency domain is determined, and the point spread function of the single-pixel divergence field is established; secondly, the error of the reconstructed gradient field obtained by the Frankot-Chellappa algorithm is convolved with the single-pixel divergence field to obtain the compensated divergence field; then the compensated divergence field is applied to the point spread function of the single-pixel divergence field to compensate for the reconstructed depth field and its gradient field; finally, the result is optimized through multiple iterations. The present invention effectively improves the reconstruction accuracy through iteration, meets the accuracy requirements in practical applications, and can use fast Fourier transform to accelerate during the integration and convolution operations, effectively improving the speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the gradient field integral depth reconstruction algorithm based on divergence compensation optimization of the present invention;

[0020] Figure 2 Schematic diagram for obtaining polarization images, target masks, and gradient data. DETAILED DESCRIPTION

[0021] The basic idea of ​​the present invention is: under a given gradient difference operator format, the reconstructed depth field and the reconstructed gradient field generated by the single-pixel divergence field are solved as a point spread function. The gradient data and the target mask of the target surface are obtained from the polarized image, and the iterative initial values ​​of the reconstructed depth field and the reconstructed gradient field are calculated by the Frankot-Chellappa algorithm. The error is calculated using the reconstructed gradient field and the target surface gradient data, and the compensation amount of the divergence field is calculated by convolution operation. The reconstructed depth field and the reconstructed gradient field are compensated, and the optimization result is repeatedly iterated.

[0022] The present invention is further described in detail below by taking a polarization vision sensor composed of a polarization camera as an example.

[0023] like Figure 1 As shown, the gradient field integral depth reconstruction method based on divergence compensation optimization of the present invention mainly includes the following steps:

[0024] Step a: Under the premise of determining the gradient difference operator format, solving the point spread function of the single pixel divergence of the gradient field; specifically including the following steps: Step 11: Determine the gradient difference operator format in the frequency domain.

[0025] Here we determine the format of the gradient difference operator of the two-dimensional scalar field (or image), specifically using the representation in the frequency domain as the basis for the subsequent steps. Suppose the size of the two-dimensional scalar field is ,in, is the image width, is the image height, and the format of the gradient difference operator in the frequency domain is:

[0026]

[0027] Among them, parameter x is the transverse component of the two-dimensional scalar field, y is the longitudinal component of the two-dimensional scalar field, u is the transverse component of the Fourier transform, v is the longitudinal component of the Fourier transform, and j is the imaginary unit;

[0028] So for a known two-dimensional scalar field , let its two-dimensional discrete Fourier transform be , so the two-dimensional discrete Fourier transform of its gradient is calculated by a given gradient difference operator:

[0029] D x F ( u , v ) = [ exp ( j π u / N x ) − exp ( − j π u / N x ) ] F ( u , v ) D y F ( u , v ) = [ exp ( j π v / N y ) − exp ( − j π v / N y ) ] F ( u , v )

[0030] Step 12: Solve the point spread function of the single pixel divergence of the gradient field.

[0031] Here, under the premise of knowing the gradient difference operator, we determine the contribution of a single pixel to the integral result in the divergence of the gradient field. Let the single pixel divergence field be:

[0032]

[0033] Calculate its 2D discrete Fourier transform , the point spread function generated by the single pixel divergence field is calculated by solving the two-dimensional Poisson equation and the point spread function of the gradient field , :

[0034]

[0035]

[0036]

[0037] Step b, using the target surface gradient data to solve the initial value of the depth field and the gradient field of the depth field; specifically includes the following steps:

[0038] Step 13: Obtain polarization image, gradient data, and target mask.

[0039] like Figure 2 As shown, a polarization camera is used to obtain a polarization image containing the target object and solve the normal vector field of the target surface. and target mask , and use the normal vector field to calculate the gradient data :

[0040]

[0041] According to the above formula, the gradient data Only in target mask There are non-zero values ​​in the region.

[0042] Step 14: Calculate the initial values ​​of the reconstructed depth field and the reconstructed gradient field.

[0043] Here, the gradient data is used to calculate the initial values ​​of the reconstructed depth field and the reconstructed gradient field for subsequent iterative optimization. The two-dimensional discrete Fourier transform of The initial value of the reconstructed depth field is solved by the Frankot-Chellappa algorithm. and the initial value of the reconstructed gradient field , :

[0044]

[0045]

[0046]

[0047] in , Respectively , The complex conjugate of .

[0048] Step c: using the error between the reconstructed gradient field and the target surface gradient data to calculate the divergence compensation amount; the specific implementation method is:

[0049] Step 15: Calculate the error between the target surface gradient data and the reconstructed gradient field, and calculate the divergence compensation through convolution operation.

[0050] Here, the divergence field compensation amount is calculated based on the difference between the target surface gradient data and the reconstructed gradient field as the change amount of one iteration.

[0051] In the target mask The reconstructed gradient field is calculated in the region The error:

[0052] err x ( x , y ) = [ z x ( x , y ) − p ( x , y ) ] m ( x , y ) err y ( x , y ) = [ z y ( x , y ) − q ( x , y ) ] m ( x , y )

[0053] By convolving with the point spread function obtained in step 12 and constraining In the region of , calculate the divergence field compensation:

[0054] Δ r ( x , y ) = s [ err x ( x , y ) ⊗ z x , pixel ( x , y ) + err y ( x , y ) ⊗ z y , pixel ( x , y ) ] ( 1 − m ( x , y ) )

[0055] In the formula is the preset scale factor, Represents a circular convolution operation.

[0056] Step d: using the divergence compensation to compensate for the reconstructed depth field and its gradient field; the specific implementation method is:

[0057] Step 16: Calculate the compensated reconstructed depth field and reconstructed gradient field.

[0058] Here, the error compensation of the reconstructed depth field and the reconstructed gradient field is performed according to the point spread function obtained in step 12 and the divergence field compensation amount obtained in step 15. The calculation method is as follows:

[0059]

[0060]

[0061]

[0062] In the formula represents the updated reconstructed depth field, , represents the updated reconstructed gradient field.

[0063] Step e: Repeat step c and step d, obtain the compensated and optimized depth field after multiple iterations, and complete the reconstruction of the depth field. The specific implementation direction is:

[0064] Step 17: Determine whether the iteration termination condition is met. If so, output the final result; otherwise, jump to step 15.

[0065] Here, the iteration termination condition is set by error and number of iterations. , , Use , , Replace and count the number of iterations n. If n reaches the preset maximum number of iterations, output As the result of the final depth reconstruction; if n has not reached the maximum number of iterations, the error of the reconstructed gradient field is calculated:

[0066] err = ∑ x , y { [ z x ( x , y ) − p ( x , y ) ] 2 + [ z y ( x , y ) − q ( x , y ) ] 2 } m ( x , y )

[0067] like If the error is less than the preset maximum allowable error, the output as the result of the final depth reconstruction; otherwise, jump to step 15.

[0068] like Figure 2 As shown in FIG. 1 , it is a schematic diagram of obtaining polarization images, target masks and gradient data. Figure 2 (a) is a schematic diagram of obtaining a polarization image. Figure 2 (b) is the acquired polarization image. Figure 2 (c) is the target normal vector field, Figure 2 (d) is the target mask, Figure 2 (e) is the gradient data.

[0069] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A gradient field integral depth reconstruction method based on divergence compensation optimization, characterized in that: The method comprises the following steps: Step a, under the premise of determining the gradient difference operator format, solving the point spread function of the single pixel divergence of the gradient field, the point spread function includes the point spread function generated by the single pixel divergence field and the point spread function of the gradient field; Step b, using the target surface gradient data to solve the initial value of the depth field and the initial value of the gradient field; Step c, using the error between the initial value of the gradient field and the target surface gradient data and the point spread function to calculate the divergence compensation amount; Step d, using the divergence compensation amount to compensate the depth field and the gradient field; Step e: repeating step c and step d, obtaining a compensated and optimized depth field after multiple iterations, and completing the reconstruction of the depth field.

2. The gradient field integral depth reconstruction method based on divergence compensation optimization according to claim 1, characterized in that: Step a includes: Step 11: Determine the gradient difference operator format in the frequency domain; Assume the size of the two-dimensional scalar field is ,in, is the image width, is the image height, and the format of the gradient difference operator in the frequency domain is: Among them, parameter x is the transverse component of the two-dimensional scalar field, y is the longitudinal component of the two-dimensional scalar field, u is the transverse component of the Fourier transform, v is the longitudinal component of the Fourier transform, and j is the imaginary unit; For a known two-dimensional scalar field , let its two-dimensional discrete Fourier transform be , therefore, the two-dimensional discrete Fourier transform of the gradient calculated by the given gradient difference operator is: Step 12: Solve the point spread function of the single pixel divergence of the gradient field; Assume the single pixel divergence field is: Calculate its 2D discrete Fourier transform , using the two-dimensional Poisson equation to calculate the point spread function generated by the single pixel divergence field and the point spread function of the gradient field , : 。 3. The gradient field integral depth reconstruction method based on divergence compensation optimization according to claim 2, characterized in that: Step b includes: Step 13: Obtain polarization image, gradient data and target mask; use polarization camera to obtain polarization image containing target object and solve the normal vector field of target surface and target mask , and use the normal vector field to calculate the gradient data : According to the above formula, the gradient data Only in target mask There are non-zero values ​​in the region of ; Step 14: Calculate the initial values ​​of the depth field and the gradient field; Calculate gradient data The two-dimensional discrete Fourier transform of , solve the initial value of the depth field through the Frankot-Chellappa algorithm and the initial value of the gradient field , : in , Respectively , The complex conjugate of .

4. The gradient field integral depth reconstruction method based on divergence compensation optimization according to claim 3 is characterized in that: Step c includes: Step 15: Calculate the error between the target surface gradient data and the initial value of the gradient field, and calculate the divergence compensation amount through convolution operation; Calculate the divergence field compensation amount according to the difference between the target surface gradient data and the initial value of the gradient field as the change amount of one iteration; In the target mask Calculate the gradient field in the region The error: By convolving with the point spread function of the gradient field obtained in step 12, and constraining In the region of , calculate the divergence field compensation: In the formula is the preset scale factor, Represents a circular convolution operation.

5. The gradient field integral depth reconstruction method based on divergence compensation optimization according to claim 4 is characterized in that: Step d includes: Step 16: Calculate the compensated depth field and gradient field; perform error compensation on the depth field and gradient field according to the point spread function obtained in step 12 and the divergence field compensation amount obtained in step 15. The calculation method is as follows: In the formula represents the updated depth field, , represents the updated gradient field.

6. The gradient field integral depth reconstruction method based on divergence compensation optimization according to claim 5, characterized in that: Step e includes: Step 17: Determine whether the iteration termination condition is met. If so, output the final result; otherwise, jump to step 15; The specific method for judging whether the iteration termination condition is met is as follows: First, , , Use , , Replace and count the number of iterations n. If n reaches the preset maximum number of iterations, output As the result of the final depth reconstruction; if n has not reached the maximum number of iterations, the error of the gradient field is calculated: like If the error is less than the preset maximum allowable error, the output as the result of the final depth reconstruction; otherwise, jump to step 15.

Images