End integrated imaging method based on deep learning, electronic equipment and medium

Abstract

The invention belongs to the technical field of image processing, and provides a terminal integrated imaging method based on deep learning, electronic equipment and a medium, and the method comprises the steps: obtaining an image in a real scene through employing a metal imaging system comprising a metal super lens, and inputting the image into an image restoration architecture, so as to obtain a restored output image. The image restoration framework at least comprises an image restoration module and a discriminator; the image restoration module is configured to reconstruct an input image through an image restoration model to obtain a reconstructed image; the discriminator is configured to perform adversarial learning on frequency domain images obtained after fast Fourier transform is performed on the reconstructed image and the real image. The method has the advantages that the metal imaging system and the image recovery framework are combined, the metal imaging system adopts a hyperplane lens to show higher performance, and the image recovery framework adopts an end-to-end recovery framework, so that the defect of position information loss caused by a traditional method can be overcome.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a terminal integrated imaging method, electronic device and medium based on deep learning. Background Art

[0002] Metasurface lens, also known as Metalenses, is a flat lens structure based on metasurface technology, which has great potential in compact imaging, photography, light detection and ranging (LiDAR), and virtual reality / augmented reality (VR / AR) applications. It can replace multiple components in a traditional lens group to achieve the same optical performance while greatly reducing the size and weight of the system.

[0003] The relentless pursuit of miniaturization and performance enhancement of optical imaging systems has prompted researchers in the field of optics to continuously explore breakthroughs in traditional geometric lens technology. A high-throughput, large-aperture metal lens produced by combining deep ultraviolet immersion lithography with wafer-level nanoimprint lithography can break through the limitations of traditional lenses and usher in an era of compact, efficient imaging systems. Although metal lenses have achieved remarkable results in focusing efficiency, lens diameter, and spectral bandwidth, they are significantly affected by chromatic aberration due to the inherent role dispersion in the meta-atom design. And for the most ideal metal lens, due to the existence of physical upper limits, the lens cannot simultaneously meet broadband operation and large diameter.

[0004] With the continuous development of modern optical technology, researchers can design Pancharatnam-Berry (PB) phase-based metal lenses composed of nanostructure arrays with arbitrary rotation angles, which can achieve diffraction-limited focusing, but are susceptible to angular aberrations. With the continuous development of computer technology and machine learning, computers have been able to enhance non-ideal images. Achieving higher resolution through linear deconvolution is the most classic traditional image restoration method. Compared with traditional image restoration methods, DNN-based image restoration models have excellent performance in image restoration tasks. However, since they cannot learn image degradation with position changes, using randomly cropped patches to train the model will result in the loss of position-related information. Summary of the invention

[0005] The purpose of the present invention is to provide a terminal integrated imaging method, electronic device and medium based on deep learning to solve the above problems.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is: A terminal integrated imaging method based on deep learning comprises the steps of: using a metal imaging system including a metal superlens to acquire an image in a real scene, and inputting the image into an image restoration architecture to obtain a restored output image; The image restoration architecture includes at least an image restoration module and a discriminator; the image restoration module is configured to reconstruct the input image through the image restoration model to obtain a reconstructed image; the discriminator is configured to perform adversarial learning on the frequency domain image obtained after fast Fourier transform of the reconstructed image and the real image respectively, so as to obtain the adversarial loss between the discriminator and the image restoration model; the adversarial loss is fed back to the image restoration model to train it.

[0007] Furthermore, the metal imaging system includes a left-handed circular polarizer, a metal superlens, a right-handed circular polarizer, and an image sensor arranged in sequence; the left-handed circular polarizer and the right-handed circular polarizer are represented to only generate left-handed circularly polarized light or right-handed circularly polarized light from the light entering the metal imaging system to be transmitted to the metal superlens.

[0008] Further, the left-handed circular polarizer and the right-handed circular polarizer are represented as a stack of a linear polarizer and a quarter wave plate, respectively.

[0009] Furthermore, the metallic superlens uses a nanoplate with an arbitrary rotation angle to make it a superlens based on the PB phase principle.

[0010] Furthermore, the metallic superlens was fabricated by nanoimprint lithography and atomic layer deposition.

[0011] Furthermore, the point spread function of the metal superlens is measured to obtain the modulation transfer function, which is expressed as: ; Where x and y are the horizontal and vertical positions on the image sensor, is the measured point spread function; and are the spatial frequencies along the x and y axes respectively.

[0012] Furthermore, the point spread function is measured by capturing images of red, green and blue collimated beams using a metal imaging system.

[0013] Furthermore, when training the image restoration model, the output image of the metal imaging system is randomly cropped and positionally embedded and then input into the image restoration model, and the peak signal-to-noise ratio loss between the reconstructed image and the real image is calculated to pass it to the image restoration model.

[0014] The present invention also provides an electronic device, which includes a processor and a storage medium communicatively connected to the processor, the storage medium is suitable for storing multiple instructions, and the processor is suitable for calling the instructions in the storage medium to execute the steps of implementing the above-mentioned deep learning-based terminal integrated imaging method.

[0015] The present invention also provides a storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the steps of the above-mentioned deep learning-based terminal integrated imaging method.

[0016] Compared with the prior art, the present invention has at least the following beneficial effects: (1) The traditional imaging system is reorganized into a deep learning-driven imaging method, which combines a metal imaging system and an image restoration architecture. The metal imaging system is responsible for acquiring images, while the image restoration architecture is responsible for restoring the captured images. The metal imaging system uses a hyperplane lens to achieve stronger performance, while the image restoration architecture uses an end-to-end image restoration framework to address the shortcomings of the traditional DNN method that causes the loss of position information; (2) Use the randomly cropped image as the input of the restoration model, and use the loss between the reconstructed image and the real image to represent the image fidelity loss. This loss is fed back to the image restoration model for adjustment to reduce the difference between the output of the restoration model and the real image. (3) Fast Fourier transform of the reconstructed image and the real image is input into the auxiliary discriminator for adversarial learning; at the same time, the adversarial loss of the discriminator and the image restoration model is fed back to the image restoration model using the hinge loss GAN training scheme. By training the image restoration model, the most reasonable loss function can be found, which can significantly improve the image quality produced by the metal imaging system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 is an overall implementation flow chart of the imaging method in the embodiment provided by the present invention; Figure 2 is a schematic diagram of a metal imaging system according to an embodiment of the present invention; Figure 3 is a schematic diagram of an optical device for point spread function (PSF) measurement in an embodiment provided by the present invention; Figure 4 is a schematic diagram of an image restoration architecture in an embodiment provided by the present invention; Figure 5 It is a schematic diagram for comparing a real image, an image captured by a metal lens, and a restored image in an embodiment provided by the present invention. DETAILED DESCRIPTION

[0019] It should be noted that in the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0020] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0021] The following are specific embodiments of the present invention, and the technical solutions of the present invention are further described in conjunction with the accompanying drawings, but the present invention is not limited to these embodiments.

[0022] like Figure 1 As shown, this embodiment provides a terminal integrated imaging method based on deep learning, comprising the steps of: S1. Acquire images of real scenes using a metal imaging system including a metal superlens; S2. Input the image into the image restoration architecture to obtain a restored output image.

[0023] In this embodiment, the traditional imaging system is reorganized into a deep learning driven imaging system, which combines a metal imaging system and an image restoration architecture, wherein the metal imaging system is responsible for acquiring images, and the image restoration architecture is responsible for restoring the captured images. The metal imaging system uses a hyperplane lens to show stronger performance, and the image restoration architecture uses an end-to-end image restoration framework to address the shortcomings of the traditional DNN method that causes the loss of position information.

[0024] like Figure 2 As shown, in the metal imaging system, it includes a left-handed circular polarizer, a metal superlens, a right-handed circular polarizer and an image sensor arranged in sequence.

[0025] Among them, the polarizer can be represented as a stack of a linear polarizer (LP) and a quarter-wave plate (QWP). The light entering the metal imaging system first passes through a left-hand circular polarizer, which transmits only left-hand circular polarization (LCP) to the metal superlens; the metal lens focuses the light with the polarization-dependent transmittance of the meta-atom and transmits it to the right-hand circular polarizer; finally, the linearly polarized light transmitted from the right-hand circular polarizer is focused on the image sensor, and the right-hand circular polarizer only transmits right-hand circular polarization (RCP) and blocks the LCP light to remove the unfocused light.

[0026] The metal superlens is a new type of super-plane lens. It uses a nanoplate with an arbitrary rotation angle to make it a superlens based on the PB phase principle. By measuring its point spread function (PSF), the modulation transfer function (MTF) passing through the metal superlens is calculated.

[0027] Furthermore, the metal superlens used in the metal imaging system is produced by applying a conventional imprint resin (MINS-311RM) to a specific mold and covering it with a glass wafer. After curing the imprint resin under a predetermined pressure and ultraviolet irradiation, a metal pattern is formed on the glass wafer after the mold is removed. Finally, a layer of high-refractive-index titanium dioxide film is coated on the imprinted metal using atomic layer deposition to obtain the metal superlens required for the metal imaging system.

[0028] like Figure 3 As shown, when measuring its point spread function, the red light, green light or blue light emitted by the light-emitting diode with peak intensities of 640nm, 525nm and 457nm is spatially filtered and collimated by a 0.5-inch aspherical lens, a 20-micron pinhole and a 1-inch spherical lens arranged in sequence. When the optical axis of the spatial filter and the metal lens imaging system are matched, the PSF can be measured, which is expressed as .

[0029] The modulation transfer function describes the image quality in terms of resolution and contrast. The MTF can be calculated from the measured PSF: ; Where x and y are the horizontal and vertical positions on the image sensor, is the measured point spread function; and are the spatial frequencies along the x and y axes respectively.

[0030] After using the metal imaging system, the metal superlens is affected by chromatic aberration and angular aberration, and the manufacturing process is affected by surface defects, which causes aberration problems. Therefore, this embodiment adopts a deep learning-based image restoration architecture to solve the problem of aberration in images captured by the metal imaging system.

[0031] like Figure 4 As shown, the image restoration architecture includes at least an image restoration module and a discriminator. The image restoration module is configured to reconstruct the input image through the image restoration model to obtain a reconstructed image. The discriminator is configured to perform adversarial learning on the frequency domain image obtained by performing fast Fourier transform (FFT) on the reconstructed image and the real image, respectively, to obtain the adversarial loss between the discriminator and the image restoration model, and the adversarial loss is fed back to the image restoration model to train it. By training the image restoration architecture, the most reasonable loss function can be found, so that the image quality generated by the metal imaging system can be significantly improved.

[0032] In this embodiment, the width of the initial layer of the image restoration model network is set to 32, and the width of each consecutive block doubles as the network goes deeper. The reconstructed image is calculated based on the output of the restoration model. , compared with the ground truth image The difference between the two is used to calculate the peak signal-to-noise ratio (PSNR) loss of the recovery model. , and finally the loss Passed to the image restoration model in the image restoration architecture to reduce the difference between the output of the restoration model and the true image.

[0033] At the same time, the output image of the metal imaging system is cropped, and the cropped coordinate information is used to apply random cropping and position embedding to the input data so that the model network is trained by randomly cropping patches from the full-resolution image.

[0034] In the image restoration stage, the reconstructed image Compared with the ground truth image The difference between the peak signal-to-noise ratio loss is calculated for: ; in, , and R represent the reconstructed image, the real image, and the maximum value of the real image, respectively.

[0035] Then, when training the image restoration model, after expressing the reconstructed image and the real image in the spatial frequency domain through fast Fourier transform (FFT), the discriminator is used to perform adversarial learning to enhance the ability of the image restoration architecture to recover the lost information.

[0036] In this embodiment, the spatial data from the reconstructed image and the real image channels are converted into the frequency domain space by fast Fourier transform (FFT), which can effectively solve the occurrence of pattern artifacts while restoring high-frequency information.

[0037] In the process of adversarial learning with the discriminator, the width of the discriminator is set to 64, and all layers are set to the same width. The FFT output is used as the input of the discriminator for adversarial training. Since the data transformed by FFT is very complex, including real and imaginary parts, it can be represented as a two-dimensional vector, which is then used as the input of the discriminator. At the same time, spectral normalization is applied to improve the stability of training. In the adversarial learning stage, the adversarial loss between the discriminator (D) and the image restoration model (G) is : ; ; Where F refers to FFT, and are operators representing the computation of the average of the true image value and the reconstructed image in a given mini-batch, respectively.

[0038] In order to verify the effectiveness of this embodiment, the method provided by this embodiment was tested under the five target conditions of PSNR, SSIM, LPIPS, MAE and CS, and compared with the four most advanced image restoration models MIRNetv2, SFNet, HINet and NAFNet using the same image test set (n=70). The experimental simulation is as follows: The number of iterations during the training of the deep learning image restoration architecture was set to 300,000. In the image restoration model, AdamW was used as the optimizer and the learning rate was initially set to 3×10 -4 , and gradually decreases to 10 according to the cosine annealing schedule -7 ; In the image restoration model, AdamW is used as the optimizer and the learning rate is initially set to 3×10 -4 , and then gradually decreases to 10 after cosine annealing -7 , the coefficient of β is [0.9, 0.9]. For the discriminative optimizer, the learning rate of Adam is set to 3×10 -4 , the same model as recovery, but with the β coefficient set to [0.0, 0.9].

[0039] like Figure 5As shown in Figure 3, the image captured by Metal is clearly corrupted by chromatic aberration, with a noticeable difference in the sharpness of the green compared to the red and blue components, resulting in significant blurring. In contrast, the image reconstructed by our proposed framework exhibits significant fidelity to the ground truth, demonstrating the framework's proficiency in recovering details obliterated by chromatic aberration.

[0040] The following table shows a qualitative comparison of the four mainstream models and the above methods in the comparative test of this embodiment. Our framework reduces LPIPS by 35.6%, indicating that the perceptual similarity of the reconstructed image to its original counterpart is significantly enhanced. At the same time, it also far exceeds the current state-of-the-art models in terms of PSNR, SSIM, MAE and CS;

[0041] This embodiment also provides an electronic device, which includes a processor and a storage medium communicatively connected to the processor, the storage medium is suitable for storing multiple instructions, and the processor is suitable for calling the instructions in the storage medium to execute the method provided by this embodiment.

[0042] This embodiment also provides a storage medium, which stores a plurality of instructions. The instructions are suitable for being loaded and executed by a processor to enable the above-mentioned electronic device to execute the method provided by this embodiment.

[0043] The computer programs for implementing the methods of the embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer programs are executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer programs may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0044] In the context of the present embodiment, the storage medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0045] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A terminal integrated imaging method based on deep learning, characterized in that: The method comprises the steps of: using a metal imaging system including a metal superlens to acquire an image in a real scene, and inputting the image into an image restoration architecture to obtain a restored output image; The image restoration architecture includes at least an image restoration module and a discriminator; the image restoration module is configured to reconstruct the input image through the image restoration model to obtain a reconstructed image; the discriminator is configured to perform adversarial learning on the frequency domain image obtained after fast Fourier transform of the reconstructed image and the real image respectively, so as to obtain an adversarial loss between the discriminator and the image restoration model; the adversarial loss is fed back to the image restoration model to train it.

2. The terminal integrated imaging method based on deep learning according to claim 1, characterized in that: The metal imaging system includes a left-handed circular polarizer, a metal superlens, a right-handed circular polarizer, and an image sensor arranged in sequence; the left-handed circular polarizer and the right-handed circular polarizer are represented to only generate left-handed circularly polarized light or right-handed circularly polarized light entering the metal imaging system for transmission to the metal superlens.

3. The terminal integrated imaging method based on deep learning according to claim 2, characterized in that: The left-handed circular polarizer and the right-handed circular polarizer are represented as a stack of a linear polarizer and a quarter wave plate, respectively.

4. The method for terminal integrated imaging based on deep learning according to claim 2, characterized in that: The metal superlens adopts a nanoplate with an arbitrary rotation angle to make it a superlens based on the PB phase principle.

5. The terminal integrated imaging method based on deep learning according to claim 4, characterized in that: The metal superlens is manufactured by nanoimprint lithography and atomic layer deposition.

6. The method for terminal integrated imaging based on deep learning according to claim 2, characterized in that: The modulation transfer function of the metal superlens is obtained by measuring its point spread function, and the modulation transfer function is expressed as: ； Where x and y are the horizontal and vertical positions on the image sensor, is the measured point spread function; and are the spatial frequencies along the x and y axes respectively.

7. The terminal integrated imaging method based on deep learning according to claim 6, characterized in that: The point spread function is measured by capturing images of red, green and blue collimated light beams by the metal imaging system.

8. The terminal integrated imaging method based on deep learning according to claim 1, characterized in that: When training the image restoration model, the output image of the metal imaging system is randomly cropped and positionally embedded and then input into the image restoration model, and the peak signal-to-noise ratio loss between the reconstructed image and the real image is calculated to be transferred to the image restoration model.

9. An electronic device, characterized in that: The electronic device includes a processor and a storage medium communicatively connected to the processor, the storage medium is suitable for storing multiple instructions, and the processor is suitable for calling the instructions in the storage medium to execute the steps of implementing the deep learning-based terminal integrated imaging method described in any one of claims 1-8.

10. A storage medium, characterized in that: The storage medium stores a plurality of instructions, which are suitable for being loaded by a processor and executing the steps of the deep learning-based terminal integrated imaging method described in any one of claims 1 to 8.

Images