Fourier lamination microscopy high-resolution reconstruction system based on WSTFPM

Abstract

The invention discloses a Fourier lamination microscopy high-resolution reconstruction system based on a WSTFPM (Wavelet Transform Template Fabry-Perot Microscopy). A traditional FPM reconstruction method has the defects of low time resolution, poor robustness under system aberration and the like. The method comprises the following steps: firstly, carrying out one-time FPM iteration on a collected low-resolution image to obtain an initial amplitude diagram and an initial phase diagram, taking the two initial diagrams as the input of a WSTFPM network, and outputting the two initial diagrams as a high-resolution amplitude diagram and a high-resolution phase diagram; the WSTFPM network decomposes an initial amplitude diagram and an initial phase diagram into different frequency components by using wavelet transform, extracts local features and global features in combination with CNN and Transform, and finally reconstructs a high-resolution amplitude diagram and a high-resolution phase diagram. Experimental results show that compared with a traditional FPM reconstruction method based on iteration, WSTFPM consumes much less time while obtaining a good effect, and meanwhile, a good reconstruction effect can still be achieved under the condition that low-resolution images are few or system errors exist.

Description

(1) Technical Field

[0001] The present invention belongs to the field of Fourier ptychographic microscopy, and particularly relates to a high-resolution reconstruction system for Fourier ptychographic microscopy based on WSTFPM. (2) Background Art

[0002] Fourier ptychographic microscopy is an emerging microscopic technique for large-field, high-resolution, and quantitative phase imaging. It combines technical ideas such as phase retrieval, ptychography, and synthetic aperture, and realizes the reconstruction of high-resolution amplitude images and phase images while maintaining the original field of view of the objective lens. Since its inception, FPM has been applied in multiple biomedical fields. To improve Fourier ptychography, various improvements have also been made in system settings and reconstruction methods.

[0003] Similar to ptychography, FPM can correct many system errors, such as objective lens aberration, LED position deviation, and defocus aberration, etc. In terms of improving the acquisition speed of low-resolution images, strategies such as multiplexing are introduced in FPM; in terms of improving the reconstruction speed and robustness, optimization theories such as the Gauss-Newton method, Wirtinger flow, and convex relaxation are proposed. In addition, innovative system designs of FPM have emerged, such as lensless systems, reflection systems, fluorescence systems, and macroscopic systems, etc. These improvements show the great application prospects of Fourier ptychographic microscopy in biomedical field observation and clinical diagnosis. However, traditional FPM reconstruction methods still have defects such as low temporal resolution and poor robustness under system errors.

[0004] Deep neural networks perform well in reconstructing high-resolution images, but they cannot recover phase images or focus on a single type of sample. Although the FPNN neural network model can reconstruct high-resolution amplitude and phase, it does not have good robustness in the face of system errors such as noise. (3) Summary of the Invention

[0005] The present invention proposes a high-resolution reconstruction system for Fourier ptychographic microscopy based on WSTFPM, aiming to solve the problem of high-resolution reconstruction of Fourier ptychographic microscopy.

[0006] To achieve the above object, the present invention provides a high-resolution reconstruction system for Fourier ptychographic microscopy based on WSTFPM, and the method includes the following steps:

[0007] S1: Set up a Fourier ptychographic microscopy optical path to sample the sample to be measured, and record the low-resolution image of the measured sample.

[0008] S2: Simulate the sampling process, and use the real high-resolution image to simulate a series of low-resolution images.

[0009] S3: Perform one FPM iteration on the simulated low-resolution image to obtain the initial amplitude map and phase map.

[0010] S4: Use the initial amplitude map and phase map as the input of the WSTFPM network, and the real high-resolution image as the output to train the network.

[0011] S5: Pass the collected low-resolution image through the S3 process and feed it into the trained WSTFPM network to reconstruct the high-resolution amplitude map and phase map.

[0012] In the above S1, the Fourier ptychography optical path is a 4f system composed of an array of LED light sources, a microscope objective, a tube lens, and an imaging camera. A set of low-resolution images are obtained by illuminating with LEDs at different positions.

[0013] In the above S2, randomly select two from the real high-resolution images as the amplitude and phase, and the formula for simulating the low-resolution image is:

[0014] I m (x,y) = ‖F -1 {F{o m (x m ,y m )}P(k x ,k y )}‖ 2

[0015]

[0016]

[0017] o(x,y) = amp * exp(1j * phase)

[0018]

[0019]

[0020]

[0021] Among them, I m (x,y) is the intensity of the low-resolution image, F{} is the Fourier transform, F -1 {} is the inverse Fourier transform, P(k x ,k y ) is the pupil function, o(x,y) is the object function, is the illumination wave vector, n is the refractive index of the medium in the illumination path, which is 1 in air, λ is the wavelength of the illumination light, (x center ,y center ) represents the LED center coordinates, (x m ,ym ) represents the coordinates of the m-th LED, and L represents the distance from the LED array to the sample.

[0022] In S3, one FPM iteration includes: First, use the image obtained by upsampling and interpolating the low-resolution intensity image under normal incidence by K times as the initial intensity guess of the object. The initial phase guess of the object is initialized to 0 to form the object function o(x, y). The pupil function is guessed according to the ideal aberration-free pupil P(k x , k y ). Under the illumination of the m-th LED, the Fourier transform of the light wave field emitted by the object passes through the low-pass of the pupil function, and the low-pass light wave field passes through an inverse Fourier transform once to reach the imaging plane, obtaining the corresponding estimated light field g m (x, y); keeping the phase information of the low-resolution estimated light field unchanged, use the corresponding low-resolution intensity image to update its amplitude information to obtain the updated low-resolution estimated light field g′ m (x, y). Fourier transform the updated low-resolution estimated light field into the frequency domain, update the object function spectrum in the sub-aperture, and keep other regions unchanged; repeat the previous steps until all low-resolution images are updated, which can be regarded as completing one iteration process. The formula is expressed as follows:

[0023]

[0024]

[0025] O(k x , k y ) = F{g′ m (x, y)}P(k x , k y ) + F{g m (k x , k y )}[1 - P(k x , k y )]

[0026] Among them, F -1 {} is the inverse Fourier transform, and F{} is the Fourier transform.

[0027] In S4, the WSTFPM is adopted in this experiment for Fourier ptychography reconstruction. The WSTFPM network includes three modules: an encoder module, a WSWT module, and a decoder module. The input initial phase and amplitude are first encoded with underlying features by the encoder. The feature encoder contains a convolutional layer and two stages of residual blocks (RBs), and each RB consists of three convolutional layers and two ReLu layers. The WSWT module decomposes the underlying features into a low-frequency subband and three high-frequency subbands, where the low-frequency subband contains detailed information and the high-frequency subbands contain structural information. To establish the correspondence between different frequency subbands, we use LF-WSA and HF-SWSA to extract global features and capture long-range dependencies. The HF-SWSA consists of a layer normalization (LN) layer, a window-based self-attention unit (WSA) based on multi-heads, a sliding window-based self-attention unit (SWSA) based on multi-heads, and two layers of MLP with a GELU non-linear activation function. The LF-WSA module is obtained by replacing one SWSA unit with one WSA unit on the basis of the HF-SWSA. The extracted multi-frequency features are integrated through IDWT to accurately reconstruct the original features from the wavelet subbands. The decoder module uses two stages of RBs and a convolutional layer for feature reconstruction. Corresponding to the encoder module, the high-resolution amplitude and phase are finally reconstructed. The loss function adopts the Charbonnier loss function to better handle outliers and obtain improved performance.

[0028] In S5, after one FPM iteration and WSTFPM model reconstruction of the collected low-resolution images, the high-resolution amplitude and phase are obtained.

[0029] The beneficial effects of the present invention are as follows:

[0030] The present invention designs a Fourier ptychographic microscopy high-resolution reconstruction system based on WSTFPM. The present invention performs one FPM iteration on the collected low-resolution images to obtain the initial amplitude map and phase map, and uses these two initial maps as the input of the WSTFPM network, and the output is the high-resolution amplitude map and phase map. The WSTFPM network decomposes the initial amplitude map and phase map into different frequency components by using wavelet transform, combines CNN and Transformer to extract local features and global features, and finally reconstructs the high-resolution amplitude map and phase map. Experimental results show that compared with the traditional iterative FPM reconstruction method, WSTFPM can achieve better results with much less time consumption. At the same time, when there are fewer low-resolution images or there are systematic errors, it can still achieve good reconstruction results. (IV) Description of the Drawings

[0031] Figure 1It is a schematic flow diagram of a Fourier ptychographic microscopy high-resolution reconstruction system based on WSTFPM of the present invention;

[0032] Figure 2 It is the network structure diagram of WSTFPM of the present invention.

[0033] Figure 3 It is the schematic diagram of the reconstruction result of WSTFPM of the present invention. (V) Specific implementation manners

[0034] Next, in combination with the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. The examples of the embodiments are shown in the drawings, in which the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

[0035] A Fourier ptychographic microscopy high-resolution reconstruction system based on WSTFPM includes the following steps:

[0036] S1: Build a Fourier ptychographic microscopy optical path to sample the sample to be measured, and record the low-resolution image of the measured sample.

[0037] S2: Simulate the sampling process, and simulate a series of low-resolution images by using a real high-resolution image.

[0038] S3: Perform one FPM iteration on the simulated low-resolution images to obtain the initial amplitude map and phase map.

[0039] S4: Use the initial amplitude map and phase map as the input of the WSTFPM network, and use the real high-resolution image as the output to train the network.

[0040] S5: Pass the collected low-resolution image through the S3 process and send it into the trained WSTFPM network to reconstruct the high-resolution amplitude map and phase map.

[0041] In the S1, the Fourier ptychographic microscopy optical path is a 4f system composed of an array-type LED light source, a microscope objective lens, a tube lens, and an imaging camera. A set of low-resolution images are obtained by illuminating with the LED at different positions.

[0042] In the S2, randomly select two from the real high-resolution images as the amplitude and phase, and the formula for simulating the low-resolution image is:

[0043] I m (x,y) = ‖F -1 {F{o m (x m ,y m)}P(k x ,k y )}‖ 2

[0044]

[0045]

[0046] o(x,y) = amp * exp(1j * phase)

[0047]

[0048]

[0049]

[0050] where I m (x,y) is the intensity of the low - resolution image, F{} is the Fourier transform, F -1 {} is the inverse Fourier transform, P(k x ,k y ) is the pupil function, o(x,y) is the object function, is the illumination wave vector, n is the refractive index of the medium of the illumination path, which is 1 in air, λ is the wavelength of the illumination light, (x center ,y center ) represents the central coordinates of the LED, (x m ,y m ) represents the coordinates of the m - th LED, and L represents the distance from the LED array to the sample.

[0051] In the above - mentioned S3, one FPM iteration includes: First, use the image obtained by up - sampling and interpolating the low - resolution intensity image under normal incidence by K times as the initial intensity guess of the object, initialize the initial phase guess of the object to 0, and form the object function o(x,y). The pupil function is guessed according to the ideal aberration - free pupil P(k x ,k y ). Under the illumination of the m - th LED, the Fourier transform of the light wave field emitted by the object passes through the low - pass of the pupil function, and the low - passed light wave field then passes through an inverse Fourier transform once to reach the imaging plane, obtaining the corresponding estimated light field g m (x,y). Keeping the phase information of the low - resolution estimated light field unchanged, use the corresponding low - resolution intensity image to update its amplitude information, obtaining the updated low - resolution estimated light field g′ m (x,y). Fourier - transform the updated low - resolution estimated light field into the frequency domain, update the spectrum of the object function in the sub - aperture, and keep other regions unchanged. Repeat the above steps until all low - resolution images are updated, which can be regarded as completing one iteration process. The formula is as follows:

[0052]

[0053]

[0054] O(k x ,k y ) = F{g′ m (x,y)}P(k x ,k y ) + F{g m (k x ,k y )}[1 - P(k x ,k y )]

[0055] where F -1 {} is the inverse Fourier transform and F{} is the Fourier transform.

[0056] In S4, the WSTFPM is adopted for Fourier ptychography reconstruction in this experiment. The WSTFPM network includes three modules: an encoder module, a WSWT module, and a decoder module. The input initial phase and amplitude are first encoded with underlying features by the encoder. The feature encoder contains a convolutional layer and two stages of residual blocks (RBs), and each RB consists of three convolutional layers and two ReLu layers. The WSWT module decomposes the underlying features into a low-frequency subband and three high-frequency subbands, where the low-frequency subband contains detailed information while the high-frequency subbands contain structural information. To establish the correspondence between different frequency subbands, we use LF-WSA and HF-SWSA to extract global features and capture long-range dependencies. HF-SWSA consists of a layer normalization (LN) layer, a multi-head window-based self-attention unit (WSA), a multi-head sliding window-based self-attention unit (SWSA), and two layers of MLP with GELU non-linear activation functions. The LF-WSA module is obtained by replacing one SWSA unit with one WSA unit based on HF-SWSA. The extracted multi-frequency features are integrated through IDWT to accurately reconstruct the original features from the wavelet subbands. The decoder module uses two stages of RBs and a convolutional layer for feature reconstruction, corresponding to the encoder module, and finally reconstructs the high-resolution amplitude and phase. The loss function uses the Charbonnier loss function to better handle outliers and obtain improved performance.

[0057] In S5, after one FPM iteration and WSTFPM model reconstruction of the collected low-resolution images, high-resolution amplitude and phase are obtained.

[0058] The above-disclosed is only a preferred embodiment of the present invention. Of course, the scope of the rights of the present invention cannot be limited thereby. Those of ordinary skill in the art can understand all or part of the processes of implementing the above embodiments, and the equivalent changes made according to the claims of the present invention still fall within the scope covered by the invention.

Claims

1. A Fourier ptychography high-resolution reconstruction system based on WSTFPM, characterized in that, The method includes the following steps: S1: Set up a Fourier ptychography microscopy optical path to sample the sample to be measured, and record the low-resolution image of the measured sample. S2: Simulate the sampling process, and use the real high-resolution image to simulate a series of low-resolution images. S3: Perform one FPM iteration on the simulated low-resolution images to obtain the initial amplitude map and phase map. S4: Use the initial amplitude map and phase map as the input of the WSTFPM network, and the real high-resolution image as the output to train the network. S5: Pass the collected low-resolution images through the S3 process and send them into the trained WSTFPM network to reconstruct the high-resolution amplitude map and phase map.

2. The Fourier ptychographic microscopy high-resolution reconstruction system based on WSTFPM according to claim 1, wherein: In the above S1, the Fourier ptychography microscopy optical path is a 4f system composed of an array LED light source, a microscope objective lens, a tube lens, and an imaging camera.

3. A Fourier ptychographic microscopy high-resolution reconstruction system based on WSTFPM according to claim 1, characterized in that: In the above S1, a set of low-resolution images are obtained by illuminating the LED at different positions.

4. The Fourier ptychographic microscopy high-resolution reconstruction system based on WSTFPM according to claim 1, characterized in that: In the above S2, randomly select two high-resolution images as amplitude and phase, and the formula for simulating low-resolution images is: I m (x,y) = ‖F -1 {F{o m (x m ,y m )}P(k x ,k y )}‖ 2 o(x,y) = amp * exp(1j * phase) where I m (x, y) is the intensity of the low-resolution image, F{} is the Fourier transform, F -1 {} is the inverse Fourier transform, P(k x , k y ) is the pupil function, o(x, y) is the object function, is the illumination wave vector, n is the refractive index of the medium of the illumination path, which is 1 in air, λ is the wavelength of the illumination light (x center , y center ) represents the center coordinates of the LED, (x m , y m ) represents the coordinates of the m-th LED, and L represents the distance from the LED array to the sample.

5. The Fourier ptychographic microscopy high-resolution reconstruction system based on WSTFPM according to claim 1, wherein: In S3, one FPM iteration includes: First, the image obtained by upsampling and interpolating the low-resolution intensity image under normal-incidence light by K times is used as the initial intensity guess of the object, and the initial phase guess of the object is initialized to 0 to form the object function o(x, y). The pupil function is guessed according to the ideal aberration-free pupil p(k x ,k y ); Under the illumination of the m-th LED, after the Fourier transform of the object-emitted light wave field, it passes through the low-pass of the pupil function, and the low-pass light wave field then undergoes an inverse Fourier transform once to reach the imaging plane, obtaining the corresponding estimated light field g m (x, y); Keeping the phase information of the low-resolution estimated light field unchanged, using the corresponding low-resolution intensity image to update its amplitude information, obtaining the updated low-resolution estimated light field g′ m (x, y), Fourier-transforming the updated low-resolution estimated light field into the frequency domain, updating the object function spectrum in the sub-aperture, and keeping other regions unchanged; Repeating the previous steps until all low-resolution images are updated, which can be regarded as completing one iteration process. The formula is as follows: O(k x ,k y ) = F{g′ m (x,y)}P(k x ,k y ) + F{g m (k x ,k y )}[1 - P(k x ,k y )] where, F -1 {} is the inverse Fourier transform, and F{} is the Fourier transform.

6. A Fourier ptychographic microscopy high-resolution reconstruction system based on WSTFPM according to claim 1, characterized in that: In the above S4, WSTFPM decomposes the input amplitude map and phase map into subbands of different frequencies through wavelet transform, combines CNN and Transformer to extract and integrate image features of different frequencies, and finally obtains the high-resolution amplitude map and phase map through wavelet reconstruction.

7. A Fourier ptychographic microscopy high-resolution reconstruction system based on WSTFPM according to claim 1, characterized in that: In the above S5, by combining the FPM and WSTFPM models, the high-resolution amplitude map and phase map can be reconstructed in a shorter time, and it is robust to system errors.

Images