A liquid crystal lens depth estimation method without polarizer plate

Abstract

The application discloses a depth estimation method of a liquid crystal lens without a polarizer, which comprises the following steps: step 1, taking two images of an object at different focal lengths by using a liquid crystal lens imaging system without a polarizer; step 2, establishing a blur degradation model without a polarizer; step 3, respectively calculating blur degradation functions of different candidate depths of the target, and establishing a filter bank; calculating an energy function and a confidence level, and finally obtaining a depth estimation value. The depth estimation method of the liquid crystal lens without the polarizer can directly estimate the depth by using the liquid crystal lens image without the polarizer, can reduce system components, and can improve the accuracy of the depth estimation.

Description

Technical Field

[0001] This invention belongs to the field of optical technology and relates to monocular depth estimation, specifically to a depth estimation method for a polarizer-free liquid crystal lens. Background Technology

[0002] Monocular depth estimation schemes have advantages in cost and ease of installation that are unmatched by binocular and lidar systems, and have received increasing attention in recent years. Depth estimation based on fuzzy cues utilizes zoom image sequences and establishes a depth estimation model by taking into account the different degrees of fuzzy degradation at different depths.

[0003] Liquid crystal lenses (LCDs) can achieve image zoom of the same size and have the advantage of being purely electronically controlled without mechanical components. However, the zoom of an LCD can only adjust for unusual light. In LCD imaging systems, polarizers are often needed to filter out ordinary light, which brings two problems. One is that it directly reduces the amount of light entering the imaging system by half, reducing image quality and affecting the accuracy of depth estimation. The other problem is that it increases the number of components in the system, increasing the system cost.

[0004] Depth estimation is the distance between the object being photographed and the optical center of the main lens. The liquid crystal lens has the effect of deflecting the light path of unusual light, which can control the focal length f of the entire optical system (including the liquid crystal lens and the main lens) for unusual light. At the same time, the light path deflection effect of the liquid crystal lens on unusual light varies with the voltage applied to the liquid crystal lens, which can realize electrical focusing. Compared with the traditional solution of using motors and mechanical structures to achieve zoom, it has the advantages of long service life, low maintenance cost, and small size.

[0005] However, traditional optical solutions that use liquid crystal lenses for imaging often incorporate a polarizer at the front of the lens to filter out ordinary light, since liquid crystal lenses only work on extraordinary light. In natural light, ordinary and extraordinary light typically account for 50% and 50% respectively. Adding a polarizer increases system complexity and assembly costs, while also reducing the amount of light entering the lens, thus affecting image quality. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention discloses a method for estimating the depth of a polarizer-free liquid crystal lens.

[0007] The depth estimation method for a polarizer-free liquid crystal lens according to the present invention is characterized by comprising the following steps:

[0008] Step 1. Use a liquid crystal lens imaging system without polarizer to take two images i1 and i2 of the object at different focal lengths;

[0009] Furthermore, the optical powers P1 and P2 of the two images should satisfy:

[0010] ;

[0011] Where Ra is the aperture radius, S is the pixel unit size, u is the object distance, and v0 is the initial image distance;

[0012] Step 2. Establish a blur degradation model k without polarizer. np for:

[0013] k np =A E *k e + A O * k o ---(2)

[0014] k e and k o Let be the point spread functions for extraordinary and ordinary light, respectively.

[0015] A e and A o These represent the proportions of unusual and ordinary light in the imaging ray, respectively. e +A o =1;

[0016] Step 3. For images i1 and i2 with different optical powers, calculate the blur degradation function k1(d) and k2(d) of the object point with depth d according to formula (2), and establish filter groups g1(d) and g2(d) for the two images respectively.

[0017] ;

[0018] Where F and F -1 These represent the Fourier transform and the inverse Fourier transform, respectively.

[0019] For a point in the image with coordinates p and depth d, calculate the deviation e. rr (p,d) are as follows:

[0020]

[0021] The energy function Q is calculated as follows:

[0022] (7)

[0023] Let q be any point in the neighborhood Ω(p) of point p, and σ i The variance of the noise;

[0024] The depth estimate is:

[0025] (9)

[0026] The argmin function is used to find the depth d that minimizes Q.

[0027] Preferably, step 4, depth completion, is also included.

[0028] Calculate the confidence level P according to equation (8). p

[0029] (8)

[0030] σ i The variance of the noise;

[0031] Set a confidence threshold P0, remove points with confidence scores below the threshold, and then perform depth completion on the remaining points. Specifically:

[0032] The Pointrend algorithm is used for instance segmentation to obtain the instance segmentation results;

[0033] Calculate the Laplacian matrix L using the instance segmentation results, calculate the reliable sequence matrix C using the confidence score of the depth estimation, and generate the vector Y using the result after removing erroneous depth estimates. The completed depth is then...

[0034] D=(L+C) -1 *Y.

[0035] Preferably, k in step 2 e and k o It can be obtained from equation (1);

[0036] ---(1)

[0037] R is the diameter of the blurred spot corresponding to unusual or ordinary light, and x and y are pixel coordinates.

[0038] Preferably, the calculation process for R in equation (1) is as follows:

[0039] The radii of the blurred spots corresponding to unusual and ordinary light are as follows:

[0040] R n =R a v 01 (P g -1 / u-1 / v0) (3)

[0041] R ej =R a v 01 (P g + P lc (j) -1 / u-1 / v0),j=1,2 (4)

[0042] The diameter R of the blurred spot corresponding to ordinary light and unusual light is equal to twice the radius of the blurred spot calculated by equations (3) and (4), respectively;

[0043] The optical focal length modulated by the liquid crystal lens in the two photographs were P. lc (1) P lc (2) ;p g v0 is the optical power of the main lens, v0 is the initial image distance, and u is the distance from object O to the optical system.

[0044] Preferred, A e =A o =0.5.

[0045] Preferably, in step 3, a depth estimation range is preset, and multiple points are uniformly taken within the depth estimation range, corresponding to different depths d. Steps 2 to 3 are repeated, and finally, according to formula (9)...

[0046] (9)

[0047] The depth value corresponding to the point with the highest confidence level is taken as the depth estimate.

[0048] The depth estimation method for a polarizer-free liquid crystal lens described in this invention directly estimates depth using an image of the polarizer-free liquid crystal lens, which reduces system components and improves the accuracy of depth estimation. Attached Figure Description

[0049] Figure 1 This is a schematic diagram illustrating the imaging principle of a liquid crystal lens without polarizer.

[0050] Figure 2 This is an image of a liquid crystal lens without a polarizer at an optical power of -1.0m-1 in a specific embodiment;

[0051] Figure 3 This is an image of a liquid crystal lens without a polarizer at an optical power of 1.86 m⁻¹ in a specific embodiment.

[0052] Figure 4 This is a grayscale image of depth data in a specific embodiment; the brighter the image, the farther the distance.

[0053] Figure 5 This is the result of instance segmentation in a specific embodiment; different brightness values ​​represent different instances (targets).

[0054] Figure 6 This is a diagram showing the depth estimation effect without a polarizer in a specific embodiment;

[0055] Figure 7This is a diagram showing the depth estimation effect after adding a polarizer in a specific embodiment;

[0056] 1-CMOS imaging sensor, 2-Main lens, 3-Liquid crystal lens, 4-First unusual light, 5-Second unusual light, 6-Ordinary light (emitted from point O), 7-Mixed light. Detailed Implementation

[0057] The specific embodiments of the present invention will be described in further detail below.

[0058] Because liquid crystal lenses only modulate unusual light, the specific parameters of the blur degradation models for ordinary and unusual light are different. Even objects at the same depth will produce different sizes of blurred spots after being captured by a liquid crystal lens imaging system due to the difference between ordinary and unusual light emitted. This invention establishes blur degradation models for both ordinary and unusual light, unifying them into a single blur degradation model for liquid crystal lens imaging under natural light. This model is then used to model the relationship between depth and blur, yielding a depth estimate.

[0059] The present invention as described above Figure 1 As shown, this method directly performs depth estimation on the image sequence acquired by the polarizer-free liquid crystal lens imaging system. The specific steps are as follows:

[0060] Step 1: Using a liquid crystal lens imaging system without a polarizer, capture two images at different focal lengths under natural light. The liquid crystal lens is located at the aperture stop of the optical system to ensure that the near and far images are of equal size. The optical system is typically composed of various optical elements such as lenses, mirrors, prisms, and aperture stops arranged in a specific order, used for imaging, such as the lens in a camera. The aperture stop refers to the physical element in the optical system that limits the beam of light.

[0061] To achieve the best depth estimation results, the requirements for capturing two images at different focal lengths are as follows: the blur radius of the images within the measured depth range u should not exceed 3 pixels; and the difference in blur diameter between the two images should be controlled within 1 to 2 pixels.

[0062] Given an optical power of P, for a fixed initial image distance v0, the blur radius R of an object with an object distance u can be obtained by R = Ra * V0(P - 1 / u - 1 / V0).

[0063] According to the above criteria, the optical powers P1 and P2 of the two images should satisfy the following:

[0064]

[0065] Where Ra is the aperture radius, S is the pixel unit size, u is the object distance, and v0 is the initial image distance.

[0066] Step 2: Establish a blur degradation model for ordinary and unusual light under liquid crystal lens focusing, and then establish a blur degradation model for liquid crystal lens imaging without polarizer.

[0067] The optical path of the non-polarized liquid crystal lens based on blurred image depth estimation is shown in Figure 1. The radius of the aperture is Ra. The mixed light 7 reflected by the object located at point O0 is clearly imaged at a position with an image distance of v0 after passing through the main lens. The image sensor is located at this position. The distance u0 from O0 and the optical power p of the main lens are used to determine the image depth. g The Gaussian imaging formula can be used to deduce v0 = 1 / (p g -1 / u0).

[0068] The object O is at a distance u from the optical system. The ordinary ray emitted from point O passes through the principal lens and is imaged at v. n Place.

[0069] The unusual ray emitted from point O is imaged twice by the liquid crystal lens (the depth estimation algorithm based on fuzziness requires acquiring two images), and then projected onto v. e1 v e2 The optical power modulated by the liquid crystal lens in the two photographs were P and P, respectively. lc (1) P lc (2) The optical power of the primary lens is p. g The unusual light (e-light) is modulated by the liquid crystal lens. The radii of the blurred light spots formed by the first unusual light 4 and the second unusual light 5 in the two images are R and R, respectively. e1 R e2 The first unusual light 4 is emitted from point O, and the liquid crystal lens is adjusted to the first optical power value. The second unusual light 5 is emitted from point O, and the liquid crystal lens is adjusted to the second optical power value.

[0070] Without a polarizer, the blurred light spot is formed by superimposing the blurred light spot of ordinary light that has not been modulated with the light spot of extraordinary light that has been modulated.

[0071] The blur degradation model for imaging is as follows:

[0072] (1)

[0073] Where R is the diameter of the blurred spot, and x, y are the pixel coordinates. Based on this, the blur degradation model without a polarizer is obtained as follows:

[0074] k np =A E *k e + A O * ko --- (2)

[0075] Taking natural light as an example, since natural light contains approximately 50% polarized light and 50% unpolarized light, while a liquid crystal lens only modulates half of the polarized light, its blurring degradation function is k. e For unpolarized light, the liquid crystal lens has no effect, and its blurring degradation function is k. o Therefore, under natural light without a polarizer, the blur degradation function can be written as:

[0076] k np =0.5*k e + 0.5 * k o That is, A. e =A o =0.5.

[0077] k e and k o Let be the point spread functions of the extraordinary and ordinary light rays, respectively. Substituting these into the diameter of the corresponding blurred spot, the radii of the blurred spot are directly calculated using equation (1):

[0078] R n =R a v 01 (P g -1 / u-1 / v0) (3)

[0079] R ej =R a v 01 (P g + P lc (j) -1 / u-1 / v0),j=1,2 (4)

[0080] The diameters of the blurred spots corresponding to ordinary light and unusual light are twice the radii of the blurred spots calculated by equations (3) and (4), respectively.

[0081] Step 3: Based on the fuzzy degradation model without polarizers, establish a depth estimation algorithm model and perform initial depth estimation:

[0082] There are various depth estimation algorithms based on fuzzy degradation models, including rational filter banks and unbiased fuzzy equalization. Here, we will focus on the unbiased fuzzy equalization algorithm. Based on the fuzzy degradation model obtained in step two without a polarizer, we establish fuzzy degradation functions k1(d) and k2(d) for object points with depth d in the two images i1 and i2 acquired in step one. On this basis, we construct an unbiased equalization filter bank:

[0083] (5)

[0084] Where F and F -1 These represent the Fourier transform and the inverse Fourier transform, respectively.

[0085] The bias image e is obtained by convolving the filter bank and the degraded image separately and then subtracting them. rr :

[0086]

[0087] e rr It has the same distribution as white noise, if the variance σ of the noise is known. i Then, the L2 norm Q of the deviation within the neighborhood window Ω(p) can be used as the energy function to calculate the likelihood of depth d. By calculating the likelihood, we can know that the depth d that maximizes the probability is the estimated result.

[0088] The likelihood of depth d is calculated using the confidence level in equation (8).

[0089] Calculate the confidence level P according to equation (8). p ,

[0090] (8)

[0091] σ i Let Q be the variance of the noise. As can be seen from Equation 8, the smaller Q is, the greater the confidence level.

[0092] For point A with coordinates p and depth d, the energy function Q is as follows:

[0093] (7)

[0094] Right now

[0095] The neighborhood window Ω(p) is the set of all points whose distance from point p is less than a certain value. For example, the neighborhood window Ω(p) can be defined as the set of all points whose distance from point p is less than 1.

[0096] Let q be any point in the neighborhood Ω(p) of point p, and σ i Let be the variance of the noise. Since the depth d is related to the blur radius R, it consequently affects the blur degradation function k. e k O k np To obtain a correct depth estimate, multiple points can be uniformly selected within the depth range. Following the algorithm model described in equations 1-8, α is used to calculate the confidence level of each point as the true depth estimate. The value with the highest confidence level is taken as the final depth estimate. According to equation 7, the maximum confidence level corresponds to the minimum energy function Q. Therefore, the depth estimate can be calculated according to equation (9):

[0097] (9)

[0098] The argmin function is used to find the depth d that minimizes Q.

[0099] Step 4.

[0100] To improve the accuracy of the estimation, we set a confidence threshold P0 and calculate the confidence level P according to equation (8). p ,

[0101] (8)

[0102] σ i Let V be the variance of the noise.

[0103] Elimination of confidence level P p Points below the confidence threshold P0 are identified, and image completion is performed on the removed images.

[0104] Instance segmentation was performed using the Pointrend algorithm (see Kirillov A, Wu Y, He K, et al. Pointrend: Image segmentation as rendering[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 9799-9808.), followed by completion using the Laplacian matting algorithm (see Tseng C, Wang S J. Maximum-a-posteriori estimation for global spatial coherence recovery based on matting Laplacian[C] / / 2012 19th IEEE International Conference on Image Processing. IEEE, 2012: 293-296.).

[0105] The specific method involves calculating the Laplacian matrix L based on the instance segmentation results, calculating the reliable sequence matrix C using the confidence level of the depth estimation, and generating a vector Y using the result after removing erroneous depth estimates. The calculations of L, C, and Y are all existing techniques in this field. The completed depth D is then...

[0106] D=(L+C) -1 *Y (10). Specific Implementation

[0107] according to Figure 1 The optical system shown includes a main lens, an imaging sensor located behind the main lens, and a liquid crystal lens located in front of the main lens at the aperture stop and coaxial with the main lens. The object is located within the field of view of the imaging system, and the aperture radius R is... a =0.001m, lens optical power P g =40m -1 The initial focusing object distance u0 = 2.8m, and the optical power of the liquid crystal lens in the two photographs are P respectively. lc (1) =-1.0m -1 :P lc (2) =1.86m -1 The corresponding result can be obtained.

[0108] v0=1 / (p g -1 / u0)=0.0252m,

[0109] Images captured such as Figure 2 and Figure 3 As shown, its depth ground truth grayscale image is as follows Figure 4 As shown.

[0110] We select 33 points with candidate depth values ​​ranging from 0.2m to 2.5m, with a step size of (2.5-0.2) / 32 = 0.072m. Then the i-th candidate depth is...

[0111] d i =0.2+0.072i,i=0,2,4,...32;

[0112] Calculate the depth d for each candidate separately. i Corresponding blur radii for ordinary and unusual light:

[0113] The optical power of the liquid crystal lens is P lc (1) =-1.0m -1 The blur radius of the unusual e-ray is:

[0114]

[0115] The optical power of the liquid crystal lens is P lc (2) = 1.86m -1 Below, the blur radius of the e-ray is:

[0116]

[0117] The blur radius of ordinary light O-ray is:

[0118]

[0119] Will Substituting into equations (1) and (2) respectively, we get

[0120]

[0121] Substituting the above equation into equation (5) yields And the candidate depth d at point p is calculated using equation (7). i The corresponding energy function Q(p, d) i ), calculate the depth value at point p according to equation (9):

[0122] ;

[0123] Then, the confidence level of its depth estimate is calculated according to equation (8).

[0124] ;

[0125] Where σ is taken i =0.005, based on the calculated confidence level P P Only the top 50% of depth estimates with the highest confidence levels are retained to obtain Y and D. Specifically, for an input image of 640*480, Y is a vector of 307200*1, and D is a sparse diagonal matrix of 307200*307200.

[0126] If the confidence level at point p is P P If it is greater than 1, then Otherwise, D(p,p)=Y(p)=0;

[0127] The Pointrend algorithm is used to perform instance segmentation on the original image to obtain... Figure 5 The instance segmentation results are shown.

[0128] Using this result, the Laplacian matrix L is calculated (see Levin A, Lischinski D, Weiss YA closed-form solution to natural image matting[J]. IEEE transactions on pattern analysis and machine intelligence, 2007, 30(2): 228-242.). Substituting L, C, and Y into Equation 10, the completed depth map is obtained, as shown below. Figure 6 As shown. Comparison Figure 7 The diagram showing the effect of adding a polarizer demonstrates that the depth estimation algorithm proposed in this invention significantly improves the accuracy of depth estimation.

[0129] In another specific embodiment, in step 2, a depth estimation range can be preset, and multiple points can be uniformly selected within the depth estimation range, corresponding to different depths d. Steps 2 to 3 are repeated, and finally the depth value corresponding to the point with the highest confidence is selected as the depth estimation value. Since, according to Equation 8, the highest confidence corresponds to the minimum energy function Q, the depth d that minimizes the energy function Q is selected.

[0130] The foregoing descriptions are preferred embodiments of the present invention. Unless there is a clear contradiction between the preferred embodiments or a prerequisite for a particular preferred embodiment, the preferred embodiments can be arbitrarily combined and used. The embodiments and specific parameters described are only for clearly illustrating the inventor's invention verification process and are not intended to limit the scope of patent protection of the present invention. The scope of patent protection of the present invention shall still be determined by its claims. Similarly, any equivalent structural changes made based on the description and drawings of the present invention shall also be included within the scope of protection of the present invention.

Claims

1. A method for depth estimation using a polarizer-free liquid crystal lens, characterized in that... This includes the following steps: Step 1. Use a liquid crystal lens imaging system without polarizer to take two images i1 and i2 of the object at different focal lengths; Furthermore, the optical powers P1 and P2 of the two images should satisfy: ； Where Ra is the aperture radius, S is the pixel unit size, u is the object distance, and v0 is the initial image distance; Step 2. Establish a blur degradation model k without polarizer. np for: k np =A E *k e + A O * k o ---(2) k e and k o The point spread functions A and A' are respectively for extraordinary and ordinary light. e and A o These represent the proportions of unusual and ordinary light in the imaging ray, respectively. e +A o =1; Step 3. For images i1 and i2 with different optical powers, calculate the blur degradation function k1(d) and k2(d) of the object point with depth d according to formula (2), and establish filter groups g1(d) and g2(d) for the two images respectively. ； Where F and F -1 These represent the Fourier transform and the inverse Fourier transform, respectively. For a point in the image with coordinates p and depth d, calculate the deviation e. rr (p,d) are as follows: ； The energy function Q is calculated as follows: ； Let q be any point in the neighborhood Ω(p) of point p, and σ i The variance of the noise; The depth estimate is: ； The argmin function is used to find the depth d that minimizes Q.

2. The depth estimation method for a polarizer-free liquid crystal lens as described in claim 1, characterized in that, k in step 2 e and k o It can be obtained from equation (1); ； R is the diameter of the blurred spot corresponding to unusual or ordinary light, and x and y are pixel coordinates.

3. The depth estimation method for a polarizer-free liquid crystal lens as described in claim 2, characterized in that, The calculation process for R in equation (1) is as follows: The radii of the blurred spots corresponding to unusual and ordinary light are as follows: ； The diameter R of the blurred spot corresponding to ordinary light and unusual light is equal to twice the radius of the blurred spot calculated by equations (3) and (4), respectively; The optical focal length modulated by the liquid crystal lens in the two photographs were P. lc (1) P lc (2) ;p g v0 is the optical power of the main lens, v0 is the initial image distance, and u is the distance from object O to the optical system.

4. The depth estimation method for a polarizer-free liquid crystal lens as described in claim 1, characterized in that, A e =A o =0.5。 5. The depth estimation method for a polarizer-free liquid crystal lens as described in claim 1, characterized in that, In step 3, a depth estimation range is preset, and multiple points are uniformly taken within the depth estimation range, corresponding to different depths d. Steps 2 to 3 are repeated, and finally, according to formula (9). ； The depth value corresponding to the point with the highest confidence level is taken as the depth estimate.

Images