Light field camera based on superlens and its algorithm

Abstract

The present disclosure relates to a light field camera based on a superlens, comprising a main lens, a superlens array and an imaging unit; the main lens is used to obtain radiation from a target; the superlens array is arranged at the focal plane of the main lens; the imaging unit is arranged at the focal plane of the superlens array; the superlens array includes a plurality of superlens units, the superlens units include a substrate, and structural units arrayed on the surface of the substrate, the structural units are composed of periodically arranged nanostructures. The present disclosure also relates to a refocusing method for the light field camera, comprising inputting a light field image and refocusing parameters; interpolating the original light field image to obtain a light field; calculating pixel values of the refocusing plane; repeating the steps of calculating pixel values until a complete refocused image is obtained. The light field camera and algorithm can improve the distortion problem caused by the difficulty in accurately controlling parallelism in the prior art.

Description

Technical Field

[0001] The present application belongs to the field of optical technology, and specifically relates to a light field camera based on a superlens and its algorithm. Background Art

[0002] A light field is defined as light in all directions and positions. Light field imaging is a single-shot 3D measurement technology. Due to its unique optical structure, it can capture light from multiple directions of a target scene in a single imaging process, simultaneously collecting both propagation direction and position information.

[0003] In existing light field cameras, the parallelism of the lenses and sensor planes is difficult to precisely control. Non-parallelism between optical components leads to tangential errors, while inherent radial errors and optical component errors contribute to lens distortion. Light field cameras with traditional lenses must account for and correct for this distortion, resulting in complex algorithms. Summary of the Invention

[0004] In view of the above-mentioned defects in the prior art, the present disclosure provides a light field camera to solve the distortion problem, and also provides a refocusing calculation method adapted for the above-mentioned light field camera.

[0005] The above-mentioned light field camera based on superlens includes: a main lens assembly, a superlens array and an imaging unit;

[0006] The main lens assembly is used to obtain radiation from the target; the super lens array is arranged on the focal plane of the main lens assembly; and the imaging unit is arranged on the focal plane of the super lens array.

[0007] Among them, the main lens assembly includes multiple lenses, at least one of which is a super lens; the super lens array includes multiple super lens units arranged in an array; the main lens assembly and the super lens array are constructed in the form of wafer-level packaging.

[0008] Preferably, the metalens in the main lens assembly includes a substrate, and structural units arrayed on the surface of the substrate, and the structural units are composed of periodically arranged nanostructures.

[0009] Preferably, the super lens units are arranged in different areas on the same substrate surface, and

[0010] The super lens unit includes structural units arranged in an array, and the structural units are composed of periodically arranged nanostructures;

[0011] Based on the phase distribution of the nanostructure, each superlens unit has the same focal length.

[0012] Preferably, the structural unit is a regular hexagon, and at least one nanostructure is provided at each vertex and / or center of the regular hexagon.

[0013] Preferably, the structural unit is a square, and at least one nanostructure is provided at each vertex and / or center of the square.

[0014] Preferably, the nanostructure includes one or more combinations of circular columns, square columns, fin-shaped columns or elliptical columns.

[0015] Preferably, the metalens and the metalens unit are configured with different optical properties based on the phase distribution of the nanostructure.

[0016] Preferably, the main lens assembly further includes a refractive lens.

[0017] Preferably, the main lens assembly, super lens array and imaging unit are constructed in the form of wafer-level packaging.

[0018] A method for performing image refocusing using a light field camera as described above, the method comprising the following steps:

[0019] Input light field image and refocusing parameters;

[0020] Interpolate the original light field image to obtain the light field;

[0021] Calculate the pixel value of the refocusing plane;

[0022] The pixel values of the refocused plane are repeatedly calculated until a complete refocused image is obtained.

[0023] Preferably, it also includes:

[0024] Based on different refocusing parameters, refocused images with different focal depths are repeatedly obtained, and clear images with different focal depths superimposed are output.

[0025] Preferably, in the calculating of the pixel value of the refocusing plane, the pixel value of the refocusing plane is calculated based on the following formula:

[0026]

[0027] Where α is the refocusing position control parameter, (u, v) is the plane coordinate of the main optical device, (x, y) is the plane coordinate of the super lens array, and (x′, y′) is the pixel coordinate of the refocusing plane. The light field formed by the main optical device and the super lens array is expressed as L(u, v, x, y).

[0028] The above technical solution can at least achieve the following advantages or effects: the metalens array can be wafer-level packaged with the main lens composed of the metalens, and further can be wafer-level packaged together with the imaging unit. The parallelism is easy to control, and the lens distortion caused by the tangential error, radial error and error of the main lens is avoided. The overall size can be smaller and lighter, which is more promising when used as a handheld or micro system. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A structural diagram of a light field camera described in the present disclosure;

[0030] Figure 2 is a light path diagram of the light field camera described in this disclosure at a specific viewing angle;

[0031] Figure 3 Schematic diagram of the optical path from multiple perspectives;

[0032] Figure 4 The process of light screening of a light field camera based on a superlens is shown;

[0033] Figure 5 A diagram showing the refocusing principle of the light field camera described in the present disclosure;

[0034] Figure 6 This is a flow chart of the light field camera algorithm based on the superlens in the present disclosure;

[0035] Figure 7 An example diagram of a structural unit in a superlens array;

[0036] Figure 8 This is an example of a nanostructure.

[0037] Note in the figure:

[0038] 1 Main lens; 2 Super lens array; 3 Imaging unit.

[0039] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION

[0040] The present disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which various embodiments are shown. However, the present disclosure can be implemented in many different ways and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be exhaustive and complete and will fully convey the scope of the present disclosure to those skilled in the art. Throughout, the same reference numerals represent the same components. Furthermore, in the drawings, the thicknesses, ratios, and sizes of components are exaggerated for clarity.

[0041] The terms used herein are for the purpose of describing specific embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, as used herein, "a," "an," "the," and "at least one" do not indicate a limitation on quantity, but are intended to include both the singular and the plural. For example, "a component" has the same meaning as "at least one component" unless the context clearly indicates otherwise. "At least one" should not be interpreted as limited to the quantity "one." "Or" means "and / or." The term "and / or" is used interchangeably with "at least one" in the context of the present invention.

[0042] "Or" includes any and all combinations of one or more of the associated listed items.

[0043] Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meanings as those commonly understood by those skilled in the art. Terms defined in commonly used dictionaries should be interpreted as having the same meanings as in the relevant technical context, and unless expressly defined in the specification, these terms should not be interpreted as having formal meanings in an idealized or overly formal sense.

[0044] The meaning of “include” or “comprising” specifies properties, quantities, steps, operations, components, parts or a combination thereof, but does not exclude other properties, quantities, steps, operations, components, parts or a combination thereof.

[0045] Embodiments are described herein with reference to cross-sectional illustrations that are idealized embodiments. Thus, variations from the shapes shown as a result of, for example, manufacturing techniques and / or tolerances, are anticipated. Thus, the embodiments described herein should not be construed as limited to the specific shapes of the regions as illustrated herein, but are to include deviations in shape that result from, for example, manufacturing. For example, a region illustrated or described as flat may typically have rough and / or nonlinear features. Furthermore, sharp angles that are illustrated may be rounded. Therefore, the regions illustrated in the figures are schematic in nature, and their shapes are not intended to illustrate the precise shape of a region and are not intended to limit the scope of the claims.

[0046] In existing technology, precise control of the parallelism of the main lens, microlens plane, and CCD plane in light field cameras is difficult. In particular, maintaining high alignment accuracy and parallelism between traditional lenses with varying thickness and spherical microlens arrays is difficult. This results in tangential and radial errors, which in turn cause lens distortion. Given this, light field cameras with traditional lenses must account for the effects of distortion on imaging, resulting in complex algorithms.

[0047] In view of the defects in the prior art, the present disclosure proposes a light field camera using a super lens array. For example, the structure includes a main lens assembly, a super lens array and an imaging unit, such as Figure 1As shown, the main lens assembly 1 is used to obtain optical radiation from the target; the super lens array 2 is arranged at the focal plane of the main lens assembly 1; and the imaging unit 3 is arranged at the focal plane of the super lens array 2;

[0048] The super lens array 2 includes an array of multiple super lens units. The super lens unit 2 includes a substrate and structural units arrayed on the surface of the substrate. The structural units are composed of periodically arranged nanostructures.

[0049] The supplementary explanations for the above embodiment are as follows:

[0050] The main lens assembly 1 may include a single metalens, a metalens array, a multi-metalens group, or a metalens and a traditional lens group. Each metalens unit in the metalens array 2 and the imaging unit covered by it constitute a micro camera system. The imaging unit can be CCD or CMOS, and can be an array composed of multiple imaging units. The focal length f of each metalens in the metalens array is array The same, the focal length f of the main lens is much larger than f array .

[0051] For the light field camera in the above-described embodiment, the direction of the light within it can be determined. After passing through the main lens assembly 1, the light strikes the metalens array 2 and forms a new image. The metalens array 2 and imaging unit 3 effectively record all light rays passing through the main lens assembly 1. During post-processing, the light rays are re-traced to complete the refocusing process. In short, the light field camera directly records the four-dimensional light field, and images of different focal depths are two-dimensional integrals under different circumstances.

[0052] The specific imaging process is as follows:

[0053] The main lens plane (u, v), the super lens array plane (x, y), the light field can be expressed as L(u, v, x, y). When (u, v) is fixed, all (x, y) are integrated, that is, all units in the super lens array, to form an image of a specific perspective, such as Figure 2 shown.

[0054] The total amount of information collected by the light field camera based on the super lens is equal to the spatial information multiplied by the angle information. The size of the light field image at a single specific viewing angle is equal to the number of super lenses in the super lens array. The more super lenses in the super lens array, the higher the corresponding imaging clarity. Repeating different (u, v) forms a multi-view image, such as Figure 3 shown.

[0055] For the above embodiments, the supplementary description of the structural units and nanostructures is as follows:

[0056] The super lens array 2 in the embodiment utilizes the technical characteristics of the metasurface. The metasurface is a sub-wavelength artificial nanostructure film that can modulate the incident light according to the metasurface structural units thereon. The metasurface structural units include all-dielectric or plasma nanoantennas that can directly control the phase, amplitude, polarization and other characteristics of the light. In this example, the nanostructure is an all-dielectric structural unit with high transmittance in the target band. Optional materials include: titanium oxide, silicon nitride, fused quartz, aluminum oxide, gallium nitride, gallium phosphide, amorphous silicon, crystalline silicon and hydrogenated amorphous silicon. The space between the nanostructures can be filled with air or other materials that are transparent or translucent in the working band. It should be noted that the absolute value of the difference between the refractive index of this material and the refractive index of the nanostructure must be greater than or equal to 0.5.

[0057] The structural unit can be a topological structure such as hexagon, square and fan composed of nanostructures.

[0058] Nanostructures can be polarization-dependent structures, such as nanofins and nanoelliptical cylinders, which impose a geometric phase on the incident light; nanostructures can also be polarization-independent structures, such as nanocylinders and nanosquare cylinders, which impose a propagation phase on the incident light.

[0059] Because the processing of metasurfaces is compatible with semiconductor processing technology, compared with the microlens arrays used in existing technologies, superlens arrays are easier to process, smaller in size, and lower in cost. Moreover, they can be packaged at the wafer level with CMOS or CCD, which are also made on the wafer surface, to solve the distortion problem in traditional light field cameras.

[0060] In a preferred embodiment, the structural unit is a regular hexagon, and at least one nanostructure is provided at each vertex and center of the regular hexagon. Alternatively, the structural unit is a square, and at least one nanostructure is provided at each vertex and center of the square. Ideally, the structural unit should be a nanostructure arranged at a hexagonal point and center, or a nanostructure arranged at a square point and center. It should be understood that the actual product may lack nanostructures at the edge of the metalens due to the limitations of the metalens shape, so that it does not meet the requirements of a complete hexagon / square. Specifically, Figure 7 As shown, the structural units are formed by regularly arranging nanostructures, and several structural units are arranged in an array to form a metasurface structure.

[0061] like Figure 7 An embodiment shown on the left side includes a central nanostructure surrounded by six peripheral nanostructures at equal distances from the central nanostructure. The peripheral nanostructures are evenly distributed around the circumference to form a regular hexagon, which can also be understood as a combination of regular triangles composed of multiple nanostructures.

[0062] like Figure 7An embodiment shown on the right side is a central nanostructure surrounded by four peripheral nanostructures at equal distances from the central nanostructure, forming a square.

[0063] In a preferred embodiment, the array of metalens units is formed by etching an array of structural units and the nanostructures that comprise the structural units onto the surface of a single substrate using processes such as photolithography. That is, the metalens array 2 comprises a single substrate, with the structural units disposed in different regions of the substrate. Based on the phase distribution of the nanostructures, each metalens unit in the metalens array has the same focal length. The metalens array can be formed from a single, integral wafer, facilitating wafer-level packaging with the primary lens or imaging unit.

[0064] In a preferred embodiment, the nanostructure comprises a combination of one or more of a cylinder, a square cylinder, a fin-shaped cylinder or an elliptical cylinder. According to the phase required by the nanostructure at different wavelengths, a nanostructure with the closest phase is searched in the nanostructure database.

[0065]

[0066] An optimization algorithm that minimizes weighted errors can be used to find nanostructures. The principle can be expressed by the following formula:

[0067] Where Δ(x,y) is the total error at the hypersurface coordinate (x,y), is the wavelength λ i The theoretical phase under is the wavelength λ of the j-th structure in the database i The actual phase c i The weight coefficient for this wavelength is generally 1. By searching the entire database, the structure that minimizes the total error Δ is found and placed at the (x, y) position on the metasurface.

[0068] In a preferred embodiment, the primary lens is a lens assembly comprising lenses and / or metalenses. It should be understood that the primary lens can be a single conventional lens or a single metalenses; it can also be a lens assembly consisting of multiple conventional lenses or multiple metalenses; or it can be a lens assembly consisting of a combination of conventional lenses and metalenses. It should be understood that the lens assembly can include optical components required by the optical system, such as filters and polarizers.

[0069] In a preferred embodiment, when the main lens is a metalens, it is packaged with the metalens array at the wafer level to achieve smaller errors and improve the distortion problem in the prior art.

[0070] In a preferred embodiment, at least two of the primary lens, metalens array, and imaging unit are constructed in wafer-level packaging. This includes, but is not limited to, wafer-level packaging of the primary lens and metalens array, wafer-level packaging of the metalens array and imaging unit, and wafer-level packaging of the primary lens and metalens array and imaging unit. Compared to existing technologies, this eliminates the need to consider the distortion introduced by the primary lens in traditional light field cameras, significantly reducing algorithmic complexity.

[0071] The present disclosure also relates to a calculation method adapted for the above-mentioned light field camera. Since the direction of the light inside the light field camera can be determined, the light of different image points can be refocused by light screening to achieve clear imaging of the image points. That is, the light field camera performs calculation imaging on the collected information through the refocusing process. The process of light screening by the light field camera based on the metalens and its array is as follows: Figure 4 shown.

[0072] Mathematically, the principle diagram of refocusing is as follows Figure 5 As shown in the figure, x is the coordinate on the detector plane, l is the distance from the main lens to the detector, x' is the coordinate on the refocused image plane, and l' is the distance from the plane to the main lens. The difference between the two is a multiple, l'=αl, where α is the refocusing position control parameter, corresponding to the refocused images at different optical axis positions.

[0073] like Figure 5 As shown, according to the triangle similarity relationship,

[0074]

[0075]

[0076] After collecting L(u,v,x,y), the light intensity can be expressed as

[0077] I(x,y)=∫∫L(u,v,x,y)dudv Eq-3

[0078] Substituting Eq-1 and Eq-2 into Eq-3, we can get the integral relationship of light field refocusing:

[0079]

[0080] Transforming α results in images of refocused planes at different depths.

[0081] For example, Figure 6 , the algorithm includes the following steps:

[0082] S1. Input light field image and refocusing parameters;

[0083] S2. interpolating the original light field image to obtain a light field;

[0084] S3. Calculate the pixel value of the refocusing plane;

[0085] S4. Repeat step S3 until a complete refocused image is obtained.

[0086] In a preferred embodiment, it also includes:

[0087] S5. Based on different refocusing parameters, repeat step S4 to output clear images with different focal depths superimposed.

[0088] In a preferred embodiment, in step S3, the pixel value of the refocusing plane is calculated based on the following formula:

[0089]

[0090] Where α is the refocusing position control parameter, (u, v) is the plane coordinate of the main optical device, (x, y) is the plane coordinate of the super lens array, and (x′, y′) is the pixel coordinate of the refocusing plane. The light field formed by the main optical device and the super lens array is expressed as L(u, v, x, y).

[0091] The aforementioned light field camera based on a metalens and its array, the metalens, metalens array, and detection elements can be packaged at the wafer level, eliminating the need to consider the distortion introduced by the main lens in traditional light field cameras, significantly reducing algorithmic complexity. Light field cameras based on a metalens and its array can be smaller and lighter, making them more promising for use as handheld devices or micro-systems.

[0092] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A light field camera based on a superlens, characterized in that: include: Main lens assembly, metalens array, and imaging unit; The main lens assembly is used to obtain radiation from the target; the super lens array is arranged on the focal plane of the main lens assembly; the imaging unit is arranged on the focal plane of the super lens array; after the light passes through the main lens assembly, it is illuminated by the super lens array and is imaged again; The main lens assembly includes a plurality of lenses, at least one of which is a metalens; the metalens array includes a plurality of metalens units arranged in an array; the main lens assembly and the metalens array are constructed in the form of a wafer-level package; Each of the metalens units in the metalens array and the imaging unit covered thereby constitute a miniature camera system; for a fixed position on the plane of the main lens assembly, light information of the fixed position is collected by all the miniature camera systems to obtain an image of a certain perspective, and light field information is obtained by images of different perspectives obtained based on light information of different positions on the plane of the main lens assembly collected by the miniature camera system.

2. The light field camera based on a superlens according to claim 1, characterized in that The metalens in the main lens assembly includes a substrate and structural units arrayed on the surface of the substrate, wherein the structural units are composed of periodically arranged nanostructures.

3. The light field camera based on a superlens according to claim 1, characterized in that The super lens units are arranged in different areas on the same substrate surface, and The super lens unit includes structural units arranged in an array, and the structural units are composed of periodically arranged nanostructures; Based on the phase distribution of the nanostructure, each superlens unit has the same focal length.

4. The light field camera based on a superlens according to claim 2 or 3, characterized in that The structural unit is a regular hexagon, and at least one nanostructure is provided at each vertex and / or center of the regular hexagon.

5. The light field camera based on a superlens according to claim 2 or 3, characterized in that: The structural unit is a square, and at least one nanostructure is arranged at each vertex and / or center of the square.

6. The light field camera based on a superlens according to claim 2 or 3, characterized in that The nanostructure includes one or more combinations of circular columns, square columns, fin-shaped columns or elliptical columns.

7. The light field camera based on a superlens according to claim 2 or 3, characterized in that Based on the phase distribution of the nanostructures, the metalens and the metalens units are configured with different optical properties.

8. The light field camera based on a superlens according to claim 1, characterized in that The main lens assembly also includes a refractive lens.

9. The light field camera based on a superlens according to claim 1, characterized in that The main lens assembly, super lens array and imaging unit are constructed in the form of wafer-level packaging.

10. Method for image refocusing by means of a light field camera according to any one of claims 1 to 9, characterized in that include: Input light field image and refocusing parameters; Interpolate the original light field image to obtain the light field; Calculate the pixel value of the refocusing plane; The pixel values of the refocused plane are repeatedly calculated until a complete refocused image is obtained.

11. The method according to claim 10, characterized in that Also includes: Based on different refocusing parameters, refocused images with different focal depths are repeatedly obtained, and clear images with different focal depths superimposed are output.

12. The method according to claim 10, characterized in that In the step of calculating the pixel value of the refocusing plane, the pixel value of the refocusing plane is calculated based on the following formula: ； in, is the refocusing position control parameter, is the coordinate of the main optical device plane, is the plane coordinate of the superlens array, is the pixel coordinate of the refocusing plane; the light field formed by the main optical device and the super lens array is expressed as .

Images