A super-resolution optical imaging method and system based on a diffraction neural network

Abstract

The application discloses a kind of super-resolution optical imaging method and system based on diffraction neural network, it is related to optical imaging and information processing field, comprising: S1, by diffraction neural network construction includes multiple phase modulation layers of diffraction modulation component;S2, utilize electronic convolutional neural network to carry out phase compression coding to high-resolution image, obtain corresponding low-resolution phase encoding graph;S3, under the irradiation of light source module, load low-resolution phase encoding graph to spatial light modulator to obtain the input light field of optical system;S4, input light field sequentially through each phase layer of diffraction modulation component, and it is transmitted according to free space propagation law between each layer, again through the superposition interference and diffraction effect in transmission, complete the information processing of neural network-like, obtain corresponding output light field;S5, by setting detector at the end of diffraction neural network, to construct the detection surface of capture output light field, to complete super-resolution optical imaging.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of optical imaging and information processing. More specifically, the present application relates to a super-resolution optical imaging method and system based on a diffraction neural network. BACKGROUND

[0002] In recent years, with the wide application of optical imaging in frontier fields such as biomedical, astronomical observation, national defense monitoring, AR / VR wearable display, etc., it has become a technical bottleneck to be broken through to realize the synchronous improvement of high-resolution imaging and large field of view (FOV). However, due to the limitation of the space-bandwidth product (SBP) of traditional optical elements (such as spatial light modulators), high-resolution imaging often faces the problem of resolution limitation of the modulator itself. Specifically, when using a large pixel size and low-resolution modulator, the spatial details of the projected or collected image are limited, and it is difficult to restore a high-resolution image. Although the modulation precision and pixel density can be improved by manufacturing high-precision hardware, this solution is usually costly, power-consuming, difficult to mass-produce and maintain, and difficult to meet the application requirements of lightweight and high speed. Secondly, there are uncertain disturbances in the imaging path. In real imaging systems, light waves often encounter random phase disturbances in the propagation path, such as random phase scattering media formed by non-ideal media, scattering tissues, and atmospheric turbulence. These disturbances can seriously damage the phase front of the light wave, resulting in the loss of spatial information of the imaging target, and traditional optical systems are difficult to effectively invert and reconstruct. To overcome the above problems, researchers have proposed various solutions, such as adaptive optical technology, which uses wavefront sensors and deformable mirrors for real-time feedback control to compensate for wavefront distortion caused by scattering. However, this method relies on complex feedback mechanisms and control circuits, has a long response time, and has a complex system structure, which is not suitable for small or real-time imaging scenarios. Another solution is the single-pixel reconstruction method based on computational imaging, which encodes the target scene by using different modulation patterns multiple times and then reconstructs the target image using an inversion algorithm. Although this method can alleviate some scattering problems, it is slow and power-consuming due to the serial nature of the imaging process, making it difficult to meet the real-time imaging requirements.

[0003] In summary, current imaging technology still faces significant challenges in the face of low-resolution modulators and random phase disturbances, mainly in the following two aspects:

[0004] 1. Unable to modulate and reconstruct high-resolution images under low space-bandwidth conditions;

[0005] 2. Relies on a computing platform, unable to achieve high-speed and low-power target reconstruction. SUMMARY

[0006] An object of the present application is to solve at least the above problems and / or drawbacks and to provide at least the advantages as will appear later.

[0007] To achieve these objects and other advantages and in view of its purposes, the present application provides a super-resolution optical imaging method based on a diffraction neural network, comprising:

[0008] S1, constructing a diffraction modulation component comprising a plurality of phase modulation layers by a diffraction neural network, each phase modulation layer being loaded with a phase distribution obtained through deep learning training;

[0009] S2, phase compression encoding a high-resolution image by an electronic convolutional neural network to obtain a corresponding low-resolution phase encoding image;

[0010] S3, loading the low-resolution phase encoding image onto a spatial light modulator under the irradiation of a light source module to obtain an input light field of an optical system;

[0011] S4, the input light field sequentially passes through each phase layer of the diffraction modulation component and is transmitted between layers according to the law of free space propagation, and then the information processing of the neural network is completed through the superposition interference and diffraction effect in the transmission to obtain a corresponding output light field;

[0012] S5, setting a detector at the end of the diffraction neural network to construct a detection surface to capture the output light field, thereby completing the super-resolution optical imaging.

[0013] Preferably, in S1, each phase modulation layer comprises hundreds of thousands of neurons with adjustable phase, and each neuron compresses the phase value to the range of 0 to 2π through a compression function Sigmoid.

[0014] The physical deployment of each phase modulation layer is realized by a liquid crystal spatial light modulator, and the adjacent phase modulation layers have a predetermined axial spacing to realize the optical interconnection between the neurons.

[0015] Preferably, in S1, in the deep learning training stage, the network parameters are optimized by an error back propagation algorithm, and the loss function is a combination of MSE and a negative correlation coefficient.

[0016] Preferably, in S2, the architecture of the electronic convolutional neural network is:

[0017] a convolutional block I for extracting low-level features;

[0018] a convolutional block II for extracting middle-level features;

[0019] a convolutional block III for extracting high-level features;

[0020] a convolutional block IV for extracting abstract features;

[0021] a flattening and fully connected layer for flattening and mapping the multi-dimensional feature maps output by each convolutional block;

[0022] Each of the convolutional blocks comprises three convolutional layers, and each of the convolutional layers is followed by an activation function Leaky ReLU.

[0023] Preferably, in S3, the spatial light modulator is a phase-type modulator, wherein, after the phase-type modulator is loaded with a low-resolution two-dimensional phase map, uniformly incident coherent laser is modulated by a phase surface on the phase-type modulator to obtain a modulated complex amplitude light field as an input surface.

[0024] An optical super-resolution imaging system applied to a diffraction neural network-based optical super-resolution imaging method, comprising:

[0025] A light source module for providing stable, monochromatic illumination light field, the light source module adopts a coherent laser in continuous wave or pulse mode;

[0026] A collimation and spatial filtering assembly for collimating and spatially cleaning the laser transmitted by the light source module;

[0027] A beam splitter for splitting the light output by the spatial filtering assembly;

[0028] A spatial light modulator and a diffraction modulation assembly corresponding to the spatial side of the beam splitter;

[0029] The diffraction modulation assembly is provided downstream with a detector, and a random phase scattering medium is arranged between the diffraction modulation assembly and the beam splitter.

[0030] The spatial light modulator is loaded with a low-resolution phase encoding map generated by an electronic convolutional neural network.

[0031] The present application at least includes the following beneficial effects:

[0032] Firstly, the present application provides an imaging system, which can break through the physical limit of spatial modulation resolution, has robust image reconstruction capability for penetrating scattering media, and has the characteristics of no external computing platform, real-time response, low power consumption, good generalization ability, and is suitable for different target types and complex imaging environments, and is widely applicable to biological imaging, AR / VR display and other high-resolution real-time imaging tasks.

[0033] Secondly, in the super-resolution optical imaging method of this invention, a combination of low-resolution electronic coding and all-optical diffraction decoding is adopted. That is, image decoding is performed by constructing a diffraction neural network system based on deep learning training. This stage does not require any electronic computing resources and has zero-latency and low-power inference capabilities. At the same time, through low-resolution electronic coding under the condition of low spatial bandwidth product modulator, high-quality super-resolution image reconstruction is achieved in a breakthrough, with a resolution improvement of up to 4 times and a spatial bandwidth product improvement of up to 16 times. Furthermore, the robust training strategy significantly enhances the system's tolerance to manufacturing and alignment errors, realizing super-resolution all-optical image reconstruction under the condition of random scattering medium.

[0034] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0035] Figure 1 This is a block diagram of the super-resolution optical imaging system of the present invention;

[0036] Among them, the light source module-1, collimation and spatial filtering component-2, beam splitter-3, spatial light modulator-4, low-resolution phase coding pattern-5, random phase scattering medium-6, diffraction modulation component-7, and detector-8 are all included. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings, so that those skilled in the art can implement it based on the description.

[0038] This invention provides a super-resolution optical imaging method based on a diffraction neural network. This method utilizes an imaging system with an electron-optical hybrid network architecture. A high-resolution image is encoded into a low-resolution phase map using an electron convolutional neural network (CNN), and then super-resolution reconstruction in the optical domain is achieved through a trained multi-layer diffraction modulation network. The system requires no additional computing equipment and exhibits good robustness to randomly scattering media. This scheme achieves a 4x resolution improvement and a 16x spatial bandwidth product extension. Specifically, it includes the following steps:

[0039] Step 1: This step uses an electronic convolutional neural network to generate a low-resolution phase encoding map for a high-resolution image.

[0040] The input to the electronic convolutional neural network is a high-resolution grayscale image (H), such as an image with a size of 96×96 pixels (a custom handwritten image dataset can also be used).

[0041] The output of the electronic convolutional neural network is a low-resolution phase-encoding pattern, for example, an encoding representation with a size of 24x24 pixels. This pattern is loaded onto a modulator with a larger pixel size (e.g., 32 pm pixels) for subsequent optical transmission.

[0042] The core role of the electronic convolutional neural network is to "encode" the spatial information of a high-resolution (HR) image into a low-resolution encoding representation through feature extraction and dimension compression, ensuring that sufficient details are retained for the HR image to be reconstructed by a diffractive decoder even after transmission through a random diffuser.

[0043] The electronic convolutional neural network is trained in a joint optimization manner with the optical diffractive network, using a data-driven deep learning method (e.g., error backpropagation) and a dataset including custom handwritten images (based on an MNIST extension) and random phase diffusers.

[0044] The electrical network structure of the electronic convolutional neural network is shown in Table 1:

[0045] Table 1

[0046]

[0047] The original input is a high-resolution grayscale image with a size of 96x96 pixels, which contains rich spatial detail information. The image features are extracted by a convolutional neural network (CNN) and encoded into a low-resolution phase distribution encoding representation with a size of 24x24. This encoding pattern is a two-dimensional phase map with a large number of pixels but contains the main structure and content information of the original image. The CNN model is composed of multiple layers of convolutional structures combined with activation functions (e.g., Leaky ReLU) and fully connected layers, forming an end-to-end compression-encoding channel. This encoding pattern can be loaded into the optical path through a spatial light modulator and used as the initial phase plane for subsequent diffractive calculations, realizing the bridging from electronic high-dimensional input to optical low-dimensional modulation. This process is jointly optimized with the optical diffractive network during training, enabling the encoding pattern to be recoverable and resistant to interference;

[0048] Step two, the low-resolution phase-encoding pattern after encoding is loaded onto a spatial light modulator under the irradiation of light, serving as the input plane of the entire optical system. The spatial light modulator is a phase-type modulator with a large pixel size (e.g., 32 pm) and limited spatial resolution but good phase response capability. When the encoding pattern is loaded onto the modulator, a coherent laser is incident uniformly through the phase plane, forming a modulated complex amplitude light field. Since the encoding pattern has compressed the information of the original image, the modulator, although with a low space-bandwidth product, can still carry most of the features of the original image.

[0049] Step three, in this step, the low-resolution coded representation corresponding to the input target is transmitted into the diffractive network under the illumination of coherent light. The diffractive network is composed of multiple layers (such as 3-5 layers) of phase modulation layers, each layer containing hundreds of thousands of adjustable phase neurons (usually in a 400x400 structure). Each neuron only modulates the phase, and the phase value of each adjustable neuron is compressed to the range of 0 to 2π by using a Sigmoid activation function to meet the physical implementation requirements. Light propagates between layers according to the law of free space propagation (based on the Rayleigh-Sommerfeld model), and superimposes interference and diffraction effects during propagation, thereby realizing information processing of a neural network-like.

[0050] In the training phase, the error backpropagation algorithm is used to iteratively optimize the parameters of the diffractive neural network. The loss function used L total ( X , Y ) is composed of a mean square error (MSE) term L percep ( X , Y ) and a negative correlation coefficient term L NPCC ( X , Y ), where the mean square error is used to measure the pixel-level difference between the output result and the target image, and the negative correlation coefficient is used to constrain the consistency of the output and the target in overall correlation, thereby improving the reconstruction accuracy and stability of the network. Specifically:

[0051]

[0052] wherein, L percep ( X , Y ) is used to measure the absolute error between the reconstruction result and the target, i.e., to evaluate global perception, and L percep ( X , Y ) is characterized by the following formula:

[0053]

[0054] In the above formula, N represents the number of samples (or the number of pixels), α represents the scaling coefficient, which is used to adjust the amplitude of the prediction value, X represents the prediction result, and Y represents the target label.

[0055] L NPCC ( X , Y) for measuring the similarity of the predicted image and the target image in the overall structure, i.e. for measuring the difference pixel by pixel, and L NPCC ( X , Y ) is characterized by the following formula:

[0056]

[0057] In the above formula, n represents the number of pixels, X i represents the value of the predicted image at the i-th pixel, represents the mean value of the predicted image, Y i represents the value of the target image at the i-th pixel, represents the mean value of the target image.

[0058] After training, all diffraction layer phase values are fixed, forming an optical inference system that does not require any computing unit to work. The final output plane forms a spatial light intensity distribution, which is the super-resolution reconstruction image of the input image;

[0059] Step five, a detection plane is set at the end of the diffraction neural network, which is used to capture the final light intensity distribution image. The image size is consistent with the input encoded image, but the spatial information density is significantly improved, achieving 4 times the spatial resolution improvement (such as from 24x24 to 96x96 effective pixels), and 16 times the spatial bandwidth product expansion.

[0060] A super-resolution optical imaging system for realizing super-resolution imaging in a random scattering medium, suitable for biological imaging, augmented reality / virtual reality (AR / VR), head-mounted display devices and other application scenarios that require enhanced spatial bandwidth product (SBP), comprising:

[0061] A light source module 1 for providing stable, monochromatic illumination light field, which uses a coherent laser in continuous wave or pulse mode, and the wavelength of the coherent laser can be selected as a near-infrared band (such as 1064 nm) to balance the system resolution and experimental realizability;

[0062] Collimate and spatially clean the laser transmitted by the light source module to output uniform plane wave illumination, a collimation and spatial filtering assembly 2, which can be composed of a collimation lens and a pinhole filter, to ensure that the incident wavefront has good spatial coherence and light intensity uniformity;

[0063] A beam splitter 3 that splits the light output by the spatial filtering assembly into two paths, one of which enters the modulation path, and the other of which can be used for system alignment or reference purposes. In this embodiment, the beam splitter can be a 50 / 50 non-polarized cube beam splitter prism;

[0064] The spatial light modulator 4 (also referred to as a programmable spatial light modulator) corresponding to the light splitting side of the beam splitter, the diffractive modulation component 7, and the programmable spatial light modulator (SLM) as a phase type wavefront loader are used to load the low-resolution phase encoding diagram 5 after convolutional neural network coding. Because the size of the low-resolution phase encoding diagram 5 is small (such as 24x24 pixels), but it contains the spatial structure information of the original high-resolution image, the SLM needs to support a 2π phase modulation range;

[0065] The downstream of the diffractive modulation component is provided with a detector 8, which is used to capture the reconstructed image output by the diffractive network. The output image is a high-resolution light intensity distribution diagram. The detector 8 can be selected from a high-sensitivity CCD or CMOS chip. The imaging area size is consistent with the SLM output surface (such as 96x96 pixel resolution). Further, based on the high-resolution light intensity distribution diagram, the computer connected with the detector 8 performs subsequent image reconstruction and / or classification;

[0066] The diffractive modulation component 7 and the beam splitter are provided with a random phase scattering medium 6, which is arranged between the spatial light modulator 4 and the diffractive modulation component 7 (when the spatial light modulator 4 loads the low-resolution phase encoding diagram 5, the light is reversely transmitted to the beam splitter, refracted and output to the random phase scattering medium 6, and then transmitted to the diffractive modulation component 7. Because the beam splitter only plays a refractive role here, it can be said that the random phase scattering medium 6 is arranged between the spatial light modulator 4 and the diffractive modulation component 7). It is used to simulate the disturbance medium (such as biological tissue, atmospheric turbulence, turbid layer, etc.) existing in the real environment. The random phase scattering medium 6 is realized by loading a random phase surface. The typical model is a Gaussian correlation phase screen, and the correlation length is adjustable.

[0067] The diffractive modulation component 7 is provided with four layers, which are composed of four phase type liquid crystal spatial light modulators. Each liquid crystal spatial light modulator loads a phase distribution obtained by deep learning training. The liquid crystal spatial light modulators maintain a certain axial spacing (such as 10 cm) between them, support multi-stage wavefront modulation and information reconstruction. The number of modulation units in each layer is as high as hundreds of thousands, and the unit size can be 8 μm, which is used to realize high-density diffraction interconnection.

[0068] The spatial light modulator 4 loads a low-resolution phase encoding diagram 5 generated by an electronic convolutional neural network. The low-resolution phase encoding diagram 5 is generated by a front-end computer, which converts the high-dimensional information of the encoded target image into a low-dimensional phase diagram and loads it into the optical path through a phase type SLM. The low-resolution phase encoding diagram 5 is the information source of the system, which embodies the interface of electronic encoding and optical decoding.

[0069] The operation flow of the super-resolution optical imaging system is as follows: the plane wave emitted by the optical module is collimated by the collimation and spatial filtering assembly, irradiates on the spatial light modulator, the spatial light modulator loads the low-resolution phase coding pattern generated by the CNN coding and adjusts, the modulated light field carries the target image information and continues to propagate to the beam splitter for refraction, and after passing through the random phase scattering medium, it is output to the diffraction modulation assembly 7 for layer-by-layer correction and decoding to obtain the output light field for the detector to capture, so as to reconstruct a clear super-resolution image at the detector. In the actual light transmission process, the super-resolution optical imaging process is completely completed by relying on the propagation and interference of light, without involving any electronic calculation, the image reconstruction speed is equal to the speed of light, and the system has the advantages of significant energy efficiency and compactness. Because the output light of the random phase scattering medium will cause distortion of the wave front, resulting in serious loss of information, the trained diffraction modulation assembly 7 can correct and decode the distorted wave front layer by layer to reconstruct a clear super-resolution image.

[0070] The above solution is only a description of a preferred example, but is not limited thereto. In the implementation of the present application, appropriate replacement and / or modification can be made according to the user's needs.

[0071] Although the embodiments of the present application have been disclosed as above, it is not limited to the application listed in the specification and the embodiments. It can be fully applied to various fields suitable for the present application. Those skilled in the art can easily make other modifications. Therefore, the present application is not limited to specific details and the figures shown and described herein without departing from the general concept defined by the claims and the equivalent scope.

Claims

1. A method of super-resolution optical imaging based on a diffractive neural network, characterized in that, Comprise: S1, constructing a diffraction modulation component comprising a plurality of phase modulation layers by a diffraction neural network, each phase modulation layer is loaded with a phase distribution obtained by deep learning training; S2, using an electronic convolutional neural network to perform phase compression coding on a high-resolution image to obtain a corresponding low-resolution phase coding image; S3, under the irradiation of a light source module, loading the low-resolution phase coding image onto a spatial light modulator to obtain an input light field of an optical system; S4, the input light field sequentially passes through each phase layer of the diffraction modulation component and transmits between each layer according to the free space propagation law, and then completes the information processing of the neural network through the superposition interference and diffraction effect in the transmission to obtain a corresponding output light field; S5, by setting a detector at the end of the diffraction neural network, a detection surface capturing the output light field is constructed, thereby completing the super-resolution optical imaging.

2. The diffraction-based neural network super-resolution optical imaging method of claim 1, wherein, In S1, each phase modulation layer contains hundreds of thousands of neurons with adjustable phase, and each neuron compresses the phase value to the range of 0 to 2π through the compression function Sigmoid; The physical deployment of each phase modulation layer is realized by using a liquid crystal spatial light modulator, and the adjacent phase modulation layers have a predetermined axial spacing to realize the optical interconnection between the neurons. 3.The diffraction-based neural network super-resolution optical imaging method of claim 1, wherein, In S1, in the deep learning training stage, the network parameters are optimized by the error back propagation algorithm, and the loss function is the combination of MSE and negative correlation coefficient. 4.The diffraction-based neural network based super-resolution optical imaging method of claim 1, wherein, In S2, the architecture of the electronic convolutional neural network is: Convolution block I for extracting low-level features; Convolution block II for extracting middle-level features; Convolution block III for extracting high-level features; Convolution block IV for extracting abstract features; Flattening and fully connected layer for flattening and mapping the multi-dimensional feature maps output by each convolution block; Wherein, each convolution block includes three convolution layers, and each convolution layer is followed by an activation function Leaky ReLU.

5. The diffraction-based neural network super-resolution optical imaging method of claim 1, wherein, In S3, the spatial light modulator is a phase type modulator, wherein the phase type modulator loads a low-resolution two-dimensional phase image, and the uniformly incident coherent laser passes through the phase surface on the phase type modulator to obtain the modulated complex amplitude light field as the input surface.

6. An ultrresolution optical imaging system applied to the method of claim 1-5, characterized in that, Comprise: A light source module for providing stable, monochromatic illumination light field, the light source module uses a coherent laser in continuous wave or pulse mode; A collimation and spatial filtering component for collimating and spatially cleaning the laser transmitted by the light source module; A beamsplitter that splits the light output by the spatial filtering component in half; A spatial light modulator and a diffraction modulation component corresponding to the beamsplitter on the spatial side of the beamsplitter; Wherein, a detector is arranged downstream of the diffraction modulation component, and a random phase scattering medium is arranged between the diffraction modulation component and the beamsplitter; The spatial light modulator is loaded with a low-resolution phase coding image generated by an electronic convolutional neural network.

Images