Visible light full-color wide-angle superlens camera combined with deep neural network

Abstract

This invention provides a visible light full-color wide-angle superlens camera incorporating a deep neural network, comprising a wide-angle superlens, a CMOS image sensor, an assembly mechanical component, and a deep neural network image recovery module. The wide-angle superlens and the CMOS image sensor are sequentially arranged inside the assembly mechanical component from the object side to the image side along the light propagation direction to generate a visible light full-color image to be recovered. The assembly mechanical component is used to fix, seal, adjust, and protect the wide-angle superlens and the CMOS image sensor. The deep neural network image recovery module is electrically connected to the CMOS image sensor to recover a high-quality visible light full-color image. This invention has the advantages of small camera size, wide field of view, and high image quality, effectively improving the quality and effect of visible light imaging. It also provides a new method for camera miniaturization and has broad application prospects in handheld or wearable optical systems such as miniature endoscopes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of micro-nano optics, deep neural network image processing and visible light imaging, and more particularly to a visible light full-color wide-angle superlens camera combined with a deep neural network. BACKGROUND

[0002] Cameras usually use lenses to obtain high-quality imaging of distant or close scenes, and use charge-coupled device (CCD) image sensors or complementary metal-oxide-semiconductor (CMOS) image sensors to capture images or videos. Existing visible light cameras and their lenses are increasingly applied to fields such as vehicle-mounted, mobile, medical endoscopy, etc. which are sensitive to the weight, volume and cost of devices. Portability, lightweight, integration and low cost are gradually becoming the future development trend of visible light cameras. On the other hand, visible light cameras are also pursuing a larger field of view and higher quality image output, which puts higher requirements on camera lenses.

[0003] Traditional camera lenses usually need to be combined with complex spherical and aspherical optical lenses to correct various aberrations under a large field of view and achieve high-quality image output. This not only results in a large volume and high weight of the lens, but also requires high precision and difficulty in assembling the lenses and the sensor, thereby increasing the processing and assembly costs. Therefore, it is very difficult for traditional optical elements to simultaneously achieve miniaturization, a large field of view and high-quality cameras.

[0004] A superlens is a kind of planarized subwavelength structure unit arranged periodically to focus light beams. The superlens can achieve subwavelength resolution imaging, has the advantages of small size and volume, and has great design freedom, which is crucial for the development of lightweight and miniaturized imaging devices. In recent years, the imaging research of superlenses has made rapid progress, including field of view expansion, aberration correction, and image resolution improvement.

[0005] The prior art currently discloses a super-wide field of view planar optical device, which corrects third-order Seidel aberration and realizes wide-angle planar imaging of at least 120° field of view; however, this superlens is limited to realizing high-resolution wide-angle imaging at a single wavelength, and has distortion problems, which limits its wide-field imaging application in a wide spectral range, and it is difficult to realize full-color imaging of super-wide-angle visible light; on the other hand, although the superlens in the prior art can achieve focusing close to the diffraction limit, the imaging of the superlens still has problems such as low contrast and background noise, and the imaging quality of the superlens needs to be improved.

[0006] With the rise of deep neural network research, a series of deep neural network models represented by U-Net have shown strong image processing capabilities in denoising, defogging, deraining and super-resolution, etc. bottom visual tasks, so deep neural networks are expected to solve the problems of distortion, chromatic aberration, stray light and low contrast of wide-angle superlenses. SUMMARY

[0007] The present application is to overcome the defects of the prior art superlens, such as narrow spectral range, low contrast, background noise and low imaging quality, and provides a visible light full-color wide-angle superlens camera combined with a deep neural network, which has the advantages of small camera volume, wide field of view and high image quality, effectively improving the quality and effect of visible light imaging.

[0008] To solve the above technical problems, the technical scheme of the present application is as follows:

[0009] A visible light full-color wide-angle superlens camera combined with a deep neural network, comprising: a wide-angle superlens, a CMOS image sensor, an assembly mechanical component and a deep neural network image restoration module;

[0010] The wide-angle superlens and the CMOS image sensor are arranged in the assembly mechanical component from the object side to the image side along the light propagation direction, and are used to generate a visible light full-color image to be restored;

[0011] The assembly mechanical component is used to fix, seal, adjust and protect the wide-angle superlens and the CMOS image sensor;

[0012] The deep neural network image restoration module is electrically connected with the CMOS image sensor, and is used to restore the visible light full-color image to be restored to a visible light full-color high-quality image.

[0013] Preferably, the wide-angle superlens comprises, from the object side to the image side along the light propagation direction, an aperture protection layer, an aperture layer, a medium layer, a super surface micro-nano structure layer and a micro-nano structure protection coating in sequence;

[0014] The CMOS image sensor comprises, from the object side to the image side along the light propagation direction, a protective glass layer and a sensing unit in sequence;

[0015] The sensing unit is electrically connected with the deep neural network image restoration module;

[0016] The assembly mechanical component comprises a light shielding member, a lens barrel and a base;

[0017] The light shielding member has an adjustable light transmission area and is arranged on the object side of the aperture protection layer;

[0018] The wide-angle superlens is arranged in the lens barrel, and the axis of the wide-angle superlens coincides with the axis of the lens barrel.

[0019] The CMOS image sensor is arranged inside the base, and the axis of the CMOS image sensor coincides with the axis of the base;

[0020] The lens barrel and the base are threadedly connected.

[0021] Preferably, the metasurface micro-nano structure layer includes no less than one million micro-nano structures, each micro-nano structure is arranged in a phase rule, and the arrangement mode includes a square lattice and a hexagonal lattice;

[0022] The transmission efficiency of the micro-nano structure is greater than 90%, and the phase modulation range covers 0 to 2π;

[0023] The material of the metasurface micro-nano structure layer includes any one or more of silicon nitride, silicon-rich silicon nitride, titanium dioxide, silicon dioxide, gallium nitride, crystalline silicon, polycrystalline silicon, and single crystal silicon;

[0024] The shape of the metasurface micro-nano structure layer includes one or more combinations of a cylinder, a square column, an elliptical column, a fin-shaped column, and a rectangular column, and the combination form includes any one of a surrounding combination, an inner and outer shape wrapping combination, and a multi-layer stacking.

[0025] Preferably, the visible light transmittance of the diaphragm protection layer, the dielectric layer, and the micro-nano structure protection coating is greater than 95%;

[0026] The material of the diaphragm layer is specifically a material that reflects or absorbs visible light, including any one or more of Au, PMMA, SU-8, and BN32;

[0027] The material of the assembly mechanical member is specifically a visible light absorbing material, or a material that is blackened by sanding the surface.

[0028] Preferably, the maximum axial distance between the wide-angle superlens and the CMOS image sensor is less than 2mm.

[0029] Preferably, any one of a ray tracing optimization algorithm, a particle swarm optimization algorithm, and a topological optimization algorithm is used to obtain the focal length, aperture, and phase distribution of the wide-angle superlens.

[0030] Preferably, a deep neural network model is arranged in the deep neural network image restoration module;

[0031] The deep neural network model includes an encoder layer, an intermediate bottleneck layer, and a decoder layer connected in sequence;

[0032] The encoder includes three attention sub-models connected in sequence; the intermediate bottleneck layer includes one attention sub-model; and the decoder includes three attention sub-models connected in sequence.

[0033] All attention sub-models have the same structure, and the data dimensions of the input and output are different;

[0034] Each attention sub-model comprises an attention layer, a first normalization layer, a feedforward layer and a second normalization layer connected in sequence, the input end of the attention layer also forms a residual connection with the first normalization layer, and the input end of the feedforward layer also forms a residual connection with the second normalization layer.

[0035] Preferably, the specific method for restoring the visible light full-color image to be restored into a visible light full-color high-quality image by using the deep neural network image restoration module is as follows:

[0036] S1: acquiring prior knowledge of a wide-angle superlens and a first original image set;

[0037] S2: performing simulation processing on the first original image set by using the prior knowledge of the wide-angle superlens to obtain a simulation image set, and combining the first original image set to initialize and train a deep neural network model to obtain an initialized and trained deep neural network model;

[0038] S3: acquiring a second original image set, and collecting the second original image set by using the wide-angle superlens and a CMOS image sensor to obtain a real collection image set;

[0039] S3: acquiring a second original image set, and collecting the second original image set by using the wide-angle superlens and a CMOS image sensor to obtain a real collection image set;

[0040] S4: inputting the real collection image set and the photographed image set into the initialized and trained deep neural network model for secondary training to obtain an optimal deep neural network model;

[0041] S5: acquiring a visible light full-color image to be restored by using the wide-angle superlens and the CMOS image sensor and inputting the visible light full-color image into the optimal deep neural network model for image restoration to obtain a visible light full-color high-quality image.

[0042] Preferably, in the step S1, the prior knowledge of the wide-angle superlens comprises a point spread function of the wide-angle superlens at a specific wavelength in a visible light band under all fields of view.

[0043] Preferably, in the steps S2 and S4, the loss functions of the initialization training and the secondary training are the same; when the loss function reaches a minimum value during the initialization training or the secondary training, the initialized and trained deep neural network model or the optimal deep neural network model is obtained.

[0044] Compared with the prior art, the technical scheme of the present application has the following beneficial effects:

[0045] The application provides a visible light full-color wide-angle superlens camera combined with a deep neural network, which comprises a wide-angle superlens, a CMOS image sensor, an assembly mechanical component and a deep neural network image recovery module; the wide-angle superlens and the CMOS image sensor are sequentially arranged in the assembly mechanical component from the object side to the image side along the light propagation direction, and are used for generating a visible light full-color image to be recovered; the assembly mechanical component is used for fixing, sealing, adjusting and protecting the wide-angle superlens and the CMOS image sensor; and the deep neural network image recovery module is electrically connected with the CMOS image sensor, and is used for recovering the visible light full-color image to be recovered into a visible light full-color high-quality image.

[0046] The application combines the wide-angle superlens and assembly engineering technology, the planar structure of the wide-angle superlens makes it thin and easy to assemble, various tilt errors in the assembly process are avoided, the overall size of the wide-angle superlens camera can be smaller and lighter, and the structure is simple, easy to realize and has strong universality; meanwhile, the deep neural network image recovery module provided by the application shows a standardized process for wide-angle superlens image recovery, corrects the problems of picture distortion, chromatic aberration, central bright spot and low contrast of the wide-angle superlens camera, and realizes efficient visible light full-color high-quality image recovery of the wide-angle superlens camera; in addition, the wide-angle superlens camera provided by the application has the advantages of small camera volume, wide field of view and high image quality, effectively improves the quality and effect of visible light imaging, provides a new method for miniaturization of the camera, and has application prospects in handheld or wearable optical systems such as miniature endoscopes. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 A visible light full-color wide-angle superlens camera combined with a deep neural network provided in embodiment 1.

[0048] Figure 2 A light ray tracing diagram of wide-angle superlens imaging provided in embodiment 2.

[0049] Figure 3 A point array diagram of the wide-angle superlens provided in embodiment 2.

[0050] Figure 4 A modulation transfer function of the wide-angle superlens provided in embodiment 2.

[0051] Figure 5 An optical micrograph and a scanning electron micrograph of the micro-nano structure of the wide-angle superlens provided in embodiment 2.

[0052] Figure 6 Measurement projection images of the digital "7" of the wide-angle superlens provided in embodiment 2 under different field angles.

[0053] Figure 7A working schematic diagram of the deep neural network image restoration module provided in Embodiment 2.

[0054] Figure 8 A comparison diagram of shooting effects of full-color simple cartoon pictures provided in Embodiment 2. DETAILED DESCRIPTION

[0055] The accompanying drawings are only used for illustrative purposes and cannot be understood as a limitation to the patent;

[0056] In order to better illustrate the embodiments, some components in the drawings may be omitted, enlarged or reduced, and do not represent the actual size of the product;

[0057] It is understandable for those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0058] The technical solutions of the present application will be further described below in combination with the drawings and embodiments.

[0059] Embodiment 1

[0060] As shown in Figure 1 , the present embodiment provides a visible light full-color wide-angle superlens camera combined with a deep neural network, comprising: a wide-angle superlens 1, a CMOS image sensor 2, an assembly mechanical member 3 and a deep neural network image restoration module 4.

[0061] The wide-angle superlens 1 and the CMOS image sensor 2 are sequentially arranged inside the assembly mechanical member 3 from the object side to the image side along the light propagation direction, for generating a visible light full-color image to be restored.

[0062] The assembly mechanical member 3 is used for fixing, sealing, adjusting and protecting the wide-angle superlens 1 and the CMOS image sensor 2.

[0063] The deep neural network image restoration module 4 is electrically connected with the CMOS image sensor 2, for restoring the visible light full-color image to be restored into a visible light full-color high-quality image.

[0064] In the specific implementation process, the wide-angle superlens 1 and the CMOS image sensor 2 are sequentially arranged inside the assembly mechanical member 3 from the object side to the image side along the light propagation direction, and the assembly mechanical member 3 is used for fixing, sealing, adjusting and protecting the wide-angle superlens 1 and the CMOS image sensor 2.

[0065] The light of the object is received by the CMOS image sensor 2 after passing through the wide-angle superlens 1.

[0066] The CMOS image sensor 2 is electrically connected with the deep neural network image restoration module 4, and converts the received optical signal into an electrical signal and transmits the electrical signal to the deep neural network image restoration module 4, and finally restores the visible light full-color image of the wide-angle super-lens camera;

[0067] In the embodiment, the field of view of the wide-angle super-lens camera is 60°-180°, and the size of the wide-angle super-lens camera is less than 3*3*2mm 3 ;

[0068] The camera in the embodiment combines the wide-angle super-lens and assembly engineering technology, the planar structure of the wide-angle super-lens makes it thin and easy to assemble, avoids various tilt errors in the assembly process, and the overall size of the wide-angle super-lens camera can be smaller and lighter, and the structure is simple and easy to realize, and has strong universality; at the same time, the deep neural network image restoration module proposed in the embodiment shows a standardized process for wide-angle super-lens image restoration, which corrects the problems of picture distortion, chromatic aberration, central bright spot and low contrast of the wide-angle super-lens camera, and realizes high-efficiency visible light full-color high-quality image restoration of the wide-angle super-lens camera; in addition, the visible light full-color wide-angle super-lens camera combined with the deep neural network proposed in the embodiment has the advantages of small camera volume, wide field of view and high image quality, effectively improves the quality and effect of visible light imaging, and provides a new method for miniaturization of the camera, which has application prospects in handheld or wearable optical systems such as miniature endoscopes.

[0069] Embodiment 2

[0070] The embodiment provides a visible light full-color wide-angle super-lens camera combined with a deep neural network, which comprises a wide-angle super-lens 1, a CMOS image sensor 2, an assembly mechanical member 3 and a deep neural network image restoration module 4.

[0071] The wide-angle super-lens 1 and the CMOS image sensor 2 are sequentially arranged in the assembly mechanical member 3 along the light propagation direction from the object side to the image side, and are used to generate a visible light full-color image to be restored;

[0072] The assembly mechanical member 3 is used for fixing, sealing, adjusting and protecting the wide-angle super-lens 1 and the CMOS image sensor 2;

[0073] The deep neural network image restoration module 4 is electrically connected with the CMOS image sensor 2, and is used for restoring the visible light full-color image to be restored into a visible light full-color high-quality image;

[0074] The wide-angle super-lens 1 sequentially comprises an aperture protection layer 11, an aperture layer 12, a medium layer 13, a super-surface micro-nano structure layer 14 and a micro-nano structure protection coating 15 from the object side to the image side along the light propagation direction;

[0075] The CMOS image sensor 2 includes, in order from the object side to the image side along the light propagation direction, a protective glass layer 21 and a sensing unit 22;

[0076] The sensing unit 22 is electrically connected with a deep neural network image restoration module 4;

[0077] The assembly mechanical member 3 includes a light shielding member 31, a lens barrel 32, and a base 33;

[0078] The light shielding member 31 is provided on the object side of the diaphragm protective layer 11, and the light transmission area of the light shielding member 31 is adjustable;

[0079] The wide-angle superlens 1 is arranged inside the lens barrel 32, and the axis of the wide-angle superlens 1 coincides with the axis of the lens barrel 32;

[0080] The CMOS image sensor 2 is arranged inside the base 33, and the axis of the CMOS image sensor 2 coincides with the axis of the base 33;

[0081] The lens barrel 32 and the base 33 are screw-connected;

[0082] The super surface micro-nano structure layer 14 includes not less than one million micro-nano structures, each micro-nano structure is arranged according to a phase rule, and the arrangement mode includes a square lattice and a hexagonal lattice;

[0083] The transmission efficiency of the micro-nano structure is greater than 90%, and the phase modulation range covers 0 to 2π;

[0084] The material of the super surface micro-nano structure layer 14 includes any one or more of silicon nitride, silicon-rich silicon nitride, titanium dioxide, silicon dioxide, gallium nitride, crystalline silicon, polycrystalline silicon, and single crystal silicon;

[0085] The shape of the super surface micro-nano structure layer 14 includes one or more combinations of a cylinder, a square column, an elliptical column, a fin-shaped column, and a rectangular column, and the combination form includes any one of a four-around combination, an inner-outer shape wrapping combination, and a multi-layer stacking;

[0086] The visible light transmittance of the diaphragm protective layer 11, the dielectric layer 13, and the micro-nano structure protective coating 15 is greater than 95%;

[0087] The material of the diaphragm layer 12 is specifically a material that reflects or absorbs visible light, and includes any one or more of Au, PMMA, SU-8, and BN32;

[0088] The material of the assembly mechanical member 3 is specifically a visible light absorbing material, or a material that is blackened by sanding treatment of the surface;

[0089] The maximum axial distance between the wide-angle superlens 1 and the CMOS image sensor 2 is less than 2 mm.

[0090] acquire the focal length, aperture and phase distribution of the wide-angle superlens 1 by using any one of a ray tracing optimization algorithm, a particle swarm optimization algorithm and a topology optimization algorithm;

[0091] The deep neural network image restoration module 4 is provided with a deep neural network model;

[0092] The deep neural network model comprises an encoder layer, an intermediate bottleneck layer and a decoder layer connected in sequence;

[0093] The encoder comprises three attention sub-models connected in sequence; the intermediate bottleneck layer comprises one attention sub-model; and the decoder comprises three attention sub-models connected in sequence;

[0094] All the attention sub-models have the same structure and different data dimensions of input and output;

[0095] Each attention sub-model comprises an attention layer, a first normalization layer, a feedforward layer and a second normalization layer connected in sequence, the input end of the attention layer further forms a residual connection with the first normalization layer, and the input end of the feedforward layer further forms a residual connection with the second normalization layer;

[0096] The specific method for restoring the visible light full-color image to be restored into a visible light full-color high-quality image by using the deep neural network image restoration module 4 is as follows:

[0097] S1: acquire the prior knowledge of the wide-angle superlens 1 and a first original image set;

[0098] S2: simulate the first original image set by using the prior knowledge of the wide-angle superlens 1 to acquire a simulated image set, and initialize and train the deep neural network model in combination with the first original image set to obtain an initialized and trained deep neural network model;

[0099] S3: acquire a second original image set, and acquire a real acquisition image set by using the wide-angle superlens 1 and the CMOS image sensor 2 to collect the second original image set;

[0100] acquire a photographed image set by using a traditional commercial camera to photograph the second original image set;

[0101] S4: input the real acquisition image set and the photographed image set into the initialized and trained deep neural network model for secondary training to acquire an optimal deep neural network model;

[0102] S5: acquire the visible light full-color image to be restored by using the wide-angle superlens 1 and the CMOS image sensor 2 and input the visible light full-color image into the optimal deep neural network model for image restoration to acquire a visible light full-color high-quality image;

[0103] In the step S1, the prior knowledge of the wide-angle superlens 1 includes a point spread function of the wide-angle superlens 1 at a specific wavelength in a visible light band under all fields of view;

[0104] In the steps S2 and S4, the loss functions of the initialization training and the secondary training are the same; when the loss function reaches the minimum value during the initialization training or the secondary training, the deep neural network model after the initialization training or the optimal deep neural network model is obtained.

[0105] In the specific implementation process, the field of view angle of the wide-angle superlens camera is 60°-180°, and the size of the wide-angle superlens camera is less than 3x3x2mm 3 ;

[0106] In the embodiment, the specific parameters of the visible light full-color wide-angle superlens camera combined with the deep neural network are as follows: the full field of view angle is 140°, the image space numerical aperture is 0.176, the thickness of the wide-angle superlens is less than 0.8mm, the total thickness of the camera is less than 1.6mm, and the lens diameter is less than 3mm;

[0107] The wide-angle superlens 1 and the CMOS image sensor 2 are sequentially arranged in the assembly mechanical member 3 from the object side to the image side along the light propagation direction, and the assembly mechanical member 3 is used for fixing, sealing, adjusting and protecting the wide-angle superlens 1 and the CMOS image sensor 2;

[0108] The light of the object is received by the CMOS image sensor 2 after passing through the wide-angle superlens 1;

[0109] The CMOS image sensor 2 is electrically connected with the deep neural network image restoration module 4, and converts the received optical signal into an electrical signal and sends it to the deep neural network image restoration module 4, so as to finally restore the visible light full-color image of the wide-angle superlens camera;

[0110] Figure 2 The light ray tracing diagram of the wide-angle superlens imaging is shown in FIG. 1, wherein the wide-angle superlens 1 sequentially includes an aperture protection layer 11, an aperture layer 12, a medium layer 13, a super surface micro-nano structure layer 14 and a micro-nano structure protection coating layer 15 from the object side to the image side along the light propagation direction; Figure 2 The CMOS image sensor 2 sequentially includes a protection glass layer 21 and a sensing unit 22 from the object side to the image side along the light propagation direction;

[0111] The sensing unit 22 is electrically connected with the deep neural network image restoration module 4;

[0112]

[0113] ​In this embodiment, the aperture diameter of the aperture layer 12 is 220 μm, the thickness of the dielectric layer 13 is 700 μm, and the effective optical diameter of the metasurface micro / nano structure layer 14 is 1.53 mm.

[0114] In this embodiment, the thickness of the aperture layer 12 is greater than 200 nm, and the thickness of the aperture protection layer 11 and the micro-nano structure protection layer 15 is greater than 1 μm.

[0115] In this embodiment, the materials selected for the aperture protection layer 11 and the micro-nano structure protection layer 15 are SiO2, the material of the metasurface micro-nano structure layer 14 is polycrystalline silicon, the shape is cylindrical, the periodic arrangement of the micro-nano structure is a hexagonal lattice, and the material of the dielectric layer 13 is fused silica.

[0116] The assembly mechanical component 3 includes a light-shielding component 31, a lens barrel 32, and a base 33;

[0117] The light-transmitting area of ​​the light-shielding component 31 is adjustable. It is set on the object side of the aperture protection layer 11 and also on the inner side of one end of the lens barrel 32. It is used to eliminate stray light and block light outside the field of view of the wide-angle superlens 1.

[0118] The wide-angle super lens 1 is disposed inside the lens barrel 32, which is used to fix, seal and protect the wide-angle super lens 1;

[0119] The CMOS image sensor 2 is disposed inside the base 33, which is used to fix, seal and protect the CMOS image sensor 2.

[0120] The lens barrel 32 and the base 33 are connected by a threaded structure to adjust the axial distance between the wide-angle super lens 1 and the CMOS image sensor 2.

[0121] In this embodiment, the maximum axial distance between the wide-angle super lens 1 and the CMOS image sensor 2 is less than 2 mm, the medium between the wide-angle super lens 1 and the CMOS image sensor 2 is air, and the thickness of the protective glass layer 21 of the selected CMOS image sensor 2 is 0.4 mm.

[0122] The material used for assembling mechanical component 3 is a visible light absorbing material, and the processing technology includes frosting and surface blackening, which is used to absorb stray light.

[0123] To meet the requirements of a wide-angle imaging system, this embodiment selects a principal ray wavelength of 532nm and uses ray tracing to optimize the phase distribution of the wide-angle superlens 1.

[0124] The specific phase formula used is as follows:

[0125]

[0126] Where R is the radius of the metasurface micro / nanostructure layer 14, ρ is the radial coordinate position of the nanopillar, and a n These are the coefficients for each item;

[0127] The coefficients a of the phase distribution of the wide-angle superlens 1 obtained through the least squares algorithm and iterative optimization are... n Where n is 12; the optimized coefficients are shown in Table 1:

[0128] [a1] [a2] [a3] [a4] [a5] [a6] -5.29e+3 -1.09e+3 1.38e+4 -6.26e+4 -1.50e+5 -2.05e+5 [a7] [a8] [a9] a 10 ]]> a 11 ]]> a 12 ]]> 1.55e+5 -5.31e+4 -5.57e+2 3.64e+3 8.74e-02 4.45e-01

[0129] Table 1. Coefficients of the optimized phase distribution of wide-angle superlens 1

[0130] Figure 3 This is a point diagram of the wide-angle superlens 1 provided in this embodiment, by... Figure 3 It can be seen that the spot radius of the wide-angle superlens 1 is smaller than the Airy disk radius across the entire field of view;

[0131] Figure 4 The modulation transfer function of the wide-angle superlens 1 provided in this embodiment is used to characterize the contrast transmission capability of the optical system at different spatial frequencies. The horizontal axis represents the spatial frequency and the vertical axis represents the modulation transfer factor (MTF). Figure 4 The modulation transfer function curves of the optical system in each field of view are close to the diffraction limit. Therefore, the scheme in this embodiment greatly improves the image quality and achieves complete elimination of third-order Seidel aberration within the preset imaging range.

[0132] In this embodiment, the periodic lattice constant of the micro-nano structure and the height of the columnar structure in the wide-angle superlens 1 are 220nm and 260nm, respectively. Based on the phase modulation corresponding to each columnar structure, a set of columnar unit structures with a transmission efficiency of over 90% is finally obtained, while satisfying the phase modulation range covering 0 to 2π.

[0133] This embodiment also utilizes processes such as electron beam lithography and plasma etching to fabricate a wide-angle superlens 1 sample for experimental verification, such as... Figure 5 As shown, Figure 5 (a) is an optical micrograph of the wide-angle superlens 1 provided in this embodiment. Figure 5 (b) is a scanning electron microscope image of the micro / nano structure of the wide-angle superlens 1. Figure 5 The exhibit is a hexagonal lattice nanopillar structure based on the principle of transmission phase. The optical eccentricity of the aperture layer 12 and the metasurface micro / nano structure layer 14 in the wide-angle superlens 1 is within 1 μm.

[0134] Figure 6 These are measured projection images of the digit "7" from the wide-angle superlens 1 at different field of view angles. Figure 6This further demonstrates the imaging performance of the wide-angle superlens 1 in this embodiment. Figure 6 There are six sub-images, corresponding to 0°, 20°, 40°, 50°, 60°, and 65° respectively, with a light source of 532nm center wavelength and 10nm bandwidth; from Figure 6 As can be seen, when using only wide-angle superlens 1 for imaging, the digit "7" will be distorted as the field of view increases;

[0135] In this embodiment, a CMOS image sensor 2 is selected for matching based on the performance parameters of the wide-angle superlens 1: the imaging surface diameter of the wide-angle superlens 1 is 1.26mm, and a CMOS image sensor 2 with a single pixel size of 2μm and a diagonal diameter of 1.26mm is selected for adaptation, while ensuring that the size of the wide-angle superlens camera is less than 3×3×2mm. 3 ;

[0136] The preset wavelength of the main ray of the wide-angle super lens 1 is 532nm, which does not eliminate chromatic aberration in the visible light band. In order to further eliminate aberration, this embodiment also provides a deep neural network image restoration module 4 to restore the visible light full-color image of the wide-angle super lens camera.

[0137] Figure 7 This is a schematic diagram of the operation of the deep neural network image restoration module 4 provided in this embodiment, as shown below. Figure 7 As shown, the deep neural network image restoration module 4 is based on a two-stage construction paradigm to train the deep neural network.

[0138] The first stage utilizes the prior knowledge of the wide-angle superlens, uses the design parameters of the wide-angle superlens and the first original image set A to calculate and obtain the simulation image set A', combines the simulation image set A' and the first original image set A into a data pair, and initializes and trains the deep neural network model in a prior-supervised manner based on the loss function and gradient descent. When the preset loss function is minimized, the initialized and trained deep neural network model is obtained.

[0139] The second stage involves acquiring the second original image set B using a wide-angle superlens 1 and a CMOS image sensor 2 to obtain the real acquired image set B'; then taking pictures of the second original image set B using a traditional commercial camera to obtain the captured image set B”; finally, inputting the real acquired image set B' and the captured image set B” into the initialized and trained deep neural network model for secondary training, and obtaining the optimal deep neural network model when the preset loss function is minimized.

[0140] Finally, the image to be restored is acquired using the wide-angle super lens 1 and CMOS image sensor 2 and input into the optimal deep neural network model for image restoration, thereby obtaining a full-color visible light image from the wide-angle super lens camera;

[0141] To further verify the shooting performance of the visible light full-color wide-angle super lens camera combined with deep neural network, this embodiment uses an LCD screen to display images and uses the wide-angle super lens camera of this embodiment to test the image set. This embodiment shows a comparison of the results of two typical full-color images.

[0142] like Figure 8 The image shown is a comparison of the image to be restored, the restored image, and the original image taken from a full-color simple cartoon. Compared to the original image, the restored image significantly improves the distortion, color difference, bright spots in the center, and poor contrast in the image to be restored. At the same time, this embodiment also performs well in capturing full-color complex scene images. Compared to the original image, the restored image not only improves the above problems but also restores the details of objects with blurred edges.

[0143] In summary, the visible light full-color wide-angle superlens camera combined with a deep neural network in this embodiment includes only a compact wide-angle superlens optical element. The planar structure reduces assembly difficulty and also reduces size, making the camera lightweight and easy to carry. In addition, the deep neural network image restoration module implemented in this invention enables the wide-angle superlens camera to present high-contrast wide-angle visible light full-color images, which is highly likely to be applied to devices with lightweight and miniaturized requirements such as electronic endoscope systems, drones, and VR / AR.

[0144] The camera in this embodiment combines a wide-angle superlens with assembly engineering technology. The planar structure of the wide-angle superlens makes it thin and easy to assemble, avoiding various tilting errors during assembly. The overall size of the wide-angle superlens camera can be smaller and lighter, and its simple structure is easy to implement and highly versatile. At the same time, the deep neural network image restoration module proposed in this embodiment demonstrates a standardized process for wide-angle superlens image restoration, correcting problems such as distortion, chromatic aberration, central bright spots, and low contrast in images captured by the wide-angle superlens camera, achieving high-efficiency, full-color, high-quality image restoration of the wide-angle superlens camera. In addition, the visible light full-color wide-angle superlens camera proposed in this embodiment, which combines deep neural networks, has the advantages of small camera size, wide field of view, and high image quality, effectively improving the quality and effect of visible light imaging. It provides a new method for camera miniaturization and has application prospects in handheld or wearable optical systems such as miniature endoscopes.

[0145] The same or similar labels correspond to the same or similar parts;

[0146] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0147] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A visible light full-color wide-angle superlens camera combined with a deep neural network, characterized in that, The application relates to a wide-angle superlens (1), a CMOS image sensor (2), an assembly mechanical component (3) and a deep neural network image recovery module (4). The wide-angle superlens (1) and the CMOS image sensor (2) are sequentially arranged in the assembly mechanical component (3) from the object side to the image side along the light propagation direction, and are used for generating a visible light full-color image to be recovered. The assembly mechanical component (3) is used for fixing, sealing, adjusting and protecting the wide-angle superlens (1) and the CMOS image sensor (2). The deep neural network image recovery module (4) is electrically connected with the CMOS image sensor (2) and is used for recovering the visible light full-color image to be recovered into a visible light full-color high-quality image. The wide-angle superlens (1) sequentially comprises an aperture protection layer (11), an aperture layer (12), a medium layer (13), an ultrathin surface micro-nano structure layer (14) and a micro-nano structure protection coating (15) from the object side to the image side along the light propagation direction. The CMOS image sensor (2) sequentially comprises a protection glass layer (21) and a sensing unit (22) from the object side to the image side along the light propagation direction. The sensing unit (22) is electrically connected with the deep neural network image recovery module (4). The assembly mechanical component (3) comprises a light shielding piece (31), a lens barrel (32) and a base (33). The light shielding piece (31) is arranged on the object side of the aperture protection layer (11) and has an adjustable light transmission area. The wide-angle superlens (1) is arranged in the lens barrel (32), and the axis of the wide-angle superlens (1) is coincident with the axis of the lens barrel (32). The CMOS image sensor (2) is arranged in the base (33), and the axis of the CMOS image sensor (2) is coincident with the axis of the base (33). The lens barrel (32) and the base (33) are screw-connected. The maximum axial distance between the wide-angle superlens (1) and the CMOS image sensor (2) is less than 2 mm. The deep neural network image recovery module (4) is provided with a deep neural network model. The deep neural network model comprises an encoder layer, an intermediate bottleneck layer and a decoder layer which are sequentially connected. The encoder comprises three attention submodels which are sequentially connected; the intermediate bottleneck layer comprises one attention submodel; and the decoder comprises three attention submodels which are sequentially connected. All the attention submodels have the same structure and different data dimensions of input and output. Each attention submodel comprises an attention layer, a first normalization layer, a feedforward layer and a second normalization layer which are sequentially connected, the input end of the attention layer is further connected with the first normalization layer in a residual connection mode, and the input end of the feedforward layer is further connected with the second normalization layer in a residual connection mode. The specific method for recovering the visible light full-color image to be recovered into a visible light full-color high-quality image by the deep neural network image recovery module (4) is as follows: S1: acquiring prior knowledge of the wide-angle superlens (1) and a first original image set; ​ S2: Simulate the first original image set using the prior knowledge of the wide-angle superlens (1) to obtain a simulated image set, and combine the first original image set to initialize and train the deep neural network model to obtain an initialized and trained deep neural network model; S3: Obtain a second original image set, and use the wide-angle superlens (1) and the CMOS image sensor (2) to collect the second original image set to obtain a real collected image set; Use a traditional commercial camera to take the second original image set to obtain a photographed image set; S4: Input the real collected image set and the photographed image set into the initialized and trained deep neural network model for secondary training to obtain an optimal deep neural network model; S5: Use the wide-angle superlens (1) and the CMOS image sensor (2) to obtain a visible light full-color image to be restored and input it into the optimal deep neural network model for image restoration to obtain a visible light full-color high-quality image.

2. The visible light full-color wide-angle superlens camera combined with deep neural network according to claim 1, wherein, The super surface micro-nano structure layer (14) includes not less than one million micro-nano structures, each micro-nano structure is arranged according to a phase rule, and the arrangement mode includes a square lattice and a hexagonal lattice; The transmission efficiency of the micro-nano structure is greater than 90%, and the phase modulation range covers 0 to ; The material of the super surface micro-nano structure layer (14) includes any one or more of silicon nitride, silicon-rich silicon nitride, titanium dioxide, silicon dioxide, gallium nitride, crystalline silicon, polycrystalline silicon, and single crystal silicon; The shape of the super surface micro-nano structure layer (14) includes one or more combinations of a cylinder, a square column, an elliptical column, a fin-shaped column, and a rectangular column, and the combination form includes any one of a four-around combination, an inner-outer shape wrapping combination, and a multi-layer stacking. 3.The visible light full-color wide-angle superlens camera combined with deep neural network according to claim 1, wherein, The visible light transmittance of the diaphragm protection layer (11), the medium layer (13), and the micro-nano structure protection coating (15) is greater than 95%; The material of the diaphragm layer (12) is specifically a material that reflects or absorbs visible light, including any one or more of Au, PMMA, SU-8, and BN32; The material of the assembly mechanical member (3) is specifically a visible light absorbing material, or a material that is blackened by sanding the surface.

4. The visible light full-color wide-angle superlens camera combined with deep neural network of claim 1, wherein, Any one of a ray tracing optimization algorithm, a particle swarm optimization algorithm, and a topological optimization algorithm is used to obtain the focal length, aperture, and phase distribution of the wide-angle superlens (1).

5. The visible light full-color wide-angle superlens camera combined with deep neural network according to claim 1, wherein, In step S1, the prior knowledge of the wide-angle superlens (1) includes a point spread function of the wide-angle superlens (1) at all field of views and a specific wavelength in the visible light band.

6. The visible light full-color wide-angle superlens camera combined with deep neural network according to claim 1, wherein, In steps S2 and S4, the loss functions of the initialization training and the secondary training are the same; when the loss function reaches a minimum value during the initialization training or the secondary training, the initialized and trained deep neural network model or the optimal deep neural network model is obtained.

Images