Image reconstruction method and system based on fourier ptychographic microscopy

Abstract

The application discloses an image reconstruction method and system based on a Fourier ptychographic imaging technology, and the method comprises the following steps: acquiring a low-resolution image through a Fourier ptychographic imaging system; constructing an improved phase recovery algorithm based on the Fourier ptychographic imaging technology; iteratively reconstructing the low-resolution image through the improved phase recovery algorithm until an algorithm iteration termination condition is met, and obtaining a reconstructed image. The module comprises an acquisition module, a construction module and a reconstruction module. By using the application, better image reconstruction results can be obtained by having a better inhibitory effect on optical aberration. The application can be widely applied to the field of computational imaging technology.

Description

Technical Field

[0001] This invention relates to the field of computational imaging technology, and in particular to an image reconstruction method and system based on Fourier layered microscopy. Background Technology

[0002] High resolution (HR), large field of view (FOV), and quantitative phase imaging have long been the goals pursued in the field of microscopy. However, these goals are difficult to achieve simultaneously in traditional microscopy imaging techniques. Fourier transform microscopy (FPM) is a novel computational imaging technique that breaks through the diffraction limit of the objective lens through computational reconstruction, and at the same time achieves large field of view, high resolution, and quantitative phase imaging. It has wide applications in biomedicine such as cytology and digital pathology. However, existing FPM inevitably introduces various aberrations during the imaging process, including: defocus aberrations due to sample unevenness and inaccurate focusing, potential coma, spherical aberration, and astigmatism of the microscope objective lens, and a series of mixed aberrations that will significantly reduce the reconstruction quality of FPM. At present, there are also optimizations to the FPM reconstruction algorithm. The mainstream correction method is to update the pupil function in the reconstruction algorithm to correct the aberrations of the optical system. However, due to its own optimization strategy, when the imaging system has different degrees of mixed aberrations, the imaging performance and efficiency of the above aberration correction method are low. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention aims to provide an image reconstruction method and system based on Fourier layered microscopy imaging technology, which can achieve better image reconstruction results by having a better suppression effect on optical aberrations.

[0004] The first technical solution adopted in this invention is: an image reconstruction method based on Fourier layered microscopy imaging technology, comprising the following steps:

[0005] Low-resolution images are acquired using a Fourier layered microscopy system.

[0006] An improved phase retrieval algorithm was constructed based on Fourier layered microscopy imaging technology;

[0007] The low-resolution image is reconstructed iteratively using an improved phase retrieval algorithm until the algorithm's iteration termination condition is met, resulting in the reconstructed image.

[0008] Furthermore, the Fourier inlay microscopy system includes a programmable LED array and a low numerical aperture objective lens. The step of acquiring a low-resolution image using the Fourier inlay microscopy system specifically includes:

[0009] By sequentially illuminating the LEDs in the programmable LED array on the Fourier stacked microscopy system, plane wave illumination image samples with varying angles are obtained.

[0010] Illuminating image samples with plane waves of varying angles is equivalent to shifting the different spatial frequency information of the samples into the observation range of a low numerical aperture objective lens;

[0011] A series of low-resolution images of the sample are acquired by the camera, wherein the low-resolution images are low-resolution sub-images corresponding to different LED units.

[0012] Furthermore, the step of constructing an improved phase retrieval algorithm based on Fourier layered microscopy specifically includes:

[0013] Initialize the sample spectrum and pupil function;

[0014] The initial sample spectrum is obtained by upsampling and Fourier transforming the image corresponding to the center LED in the low-resolution image, and using the result of the transformation as the initial sample spectrum. The initial pupil function is obtained by initializing the pupil function through the coherent transfer function.

[0015] The image received by the camera is obtained by performing frequency domain constraints on the sample spectrum and pupil function based on the illumination wave vector, wherein the illumination wave vector is provided by each lit LED;

[0016] The low-resolution image is denoised based on the image received from the camera and used as an intensity constraint to replace the sample spectrum, thus obtaining the replaced sample spectrum.

[0017] An adaptive adjustment factor is introduced to iteratively update the spectrum and pupil function of the replaced samples until all low-resolution images have been updated, resulting in an improved phase retrieval algorithm.

[0018] Furthermore, the specific expression for the pupil function is as follows:

[0019] ;

[0020] In the above formula, This represents the pupil function of the imaging system. The numerical aperture of the objective lens. This represents the amplitude of the pupil function. Represents two-dimensional Fourier domain spatial coordinates. This indicates the wavelength of light illuminating the air.

[0021] Furthermore, the expression for the image received by the camera is as follows:

[0022] ;

[0023] In the above formula, This refers to the image received by the camera. Represented as LR image index, This indicates the sub-spectral region of the sample intercepted by the pupil function. This represents the inverse Fourier transform.

[0024] Furthermore, the step of performing noise reduction processing on the low-resolution image based on the image received from the camera and replacing the sample spectrum to obtain the replaced sample spectrum specifically includes:

[0025] A preset target image is generated, and the average intensity difference between the image received by the camera and the preset target image is calculated.

[0026] The low-resolution image is denoised based on the average intensity difference to obtain the denoised low-resolution image.

[0027] The preset target image is subjected to amplitude replacement processing based on the denoised low-resolution image to obtain the replaced target image;

[0028] The replaced target image is subjected to Fourier transform processing to obtain the spectrum of the replaced sample.

[0029] Furthermore, the expression for the adaptive control factor is as follows:

[0030] ;

[0031] In the above formula, Indicates the adaptive control factor. This represents the current value of the pupil function. This represents the maximum value of the pupil function. This represents the average arithmetic threshold for a low-resolution image dataset.

[0032] Furthermore, the formula for updating the spectrum of the replaced sample is as follows:

[0033] ;

[0034] In the above formula, This represents the updated sample spectrum. This represents the sample spectrum to be updated. This represents the ratio of the current value to the maximum value of the pupil function. This represents the complex conjugate operation of the current value of the pupil function. Represents the regularization parameter. , This represents the complex amplitude information of the target sample before and after strength constraint.

[0035] Furthermore, the formula for updating the pupil function is as follows:

[0036] ;

[0037] In the above formula, This represents the updated pupil function. This indicates the pupil function to be updated. Indicates the adaptive control factor. The complex conjugate operation represents the current value of the sample spectrum function. This represents the regularization parameter.

[0038] The second technical solution adopted in this invention is: an image reconstruction system based on Fourier layered microscopy imaging technology, comprising:

[0039] The acquisition module is used to acquire low-resolution images through a Fourier in-plane microscopy system;

[0040] The module is built upon Fourier layered microscopy imaging technology to construct an improved phase retrieval algorithm;

[0041] The reconstruction module is used to iteratively reconstruct low-resolution images using an improved phase retrieval algorithm until the algorithm's iteration termination condition is met, thus obtaining the reconstructed image.

[0042] The beneficial effects of the method and system of this invention are as follows: This invention acquires low-resolution images, further obtains the spectral function of the sample and the pupil function of the imaging system based on the low-resolution images, performs noise reduction processing on the images to replace the amplitude of the target complex amplitude information, and introduces an adaptive control factor to update the spectral function and pupil function. This results in better suppression of optical aberrations and thus better recovery results. In the pathological tissue sections in the images, the lines of the intensity reconstruction results are clearer and more distinct, the details are richer, and the phase reconstruction results have higher contrast. Attached Figure Description

[0043] Figure 1 This is a flowchart of the image reconstruction method based on Fourier layered microscopy imaging technology of the present invention;

[0044] Figure 2 This is a structural block diagram of the image reconstruction system based on Fourier layered microscopy imaging technology of the present invention;

[0045] Figure 3 This is a schematic diagram of an FPM device model;

[0046] Figure 4 This is a schematic diagram of an FPM device model performing image acquisition;

[0047] Figure 5 This is a flowchart of the steps of the AA-P algorithm of this invention;

[0048] Figure 6This is a schematic diagram comparing the reconstruction results of the AA-P algorithm of this invention with those of other algorithms on an open-source biological sample dataset. Figure 6 (a) represents the original LR image. Figure 6 (b) represents the reconstruction result of the existing algorithm AP. Figure 6 (c) represents the reconstruction result of the existing algorithm AA. Figure 6 (d) represents the reconstruction result of the existing algorithm EPRY-FPM. Figure 6 (e) represents the reconstruction result of AA-P by the method of the present invention. Detailed Implementation

[0049] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0050] This invention proposes an FPM aberration correction and reconstruction algorithm (AA-P) based on an improved phase recovery strategy. This method improves the update strategy of the sample spectrum function and pupil function during the reconstruction process, which can effectively solve the problems of low robustness of existing aberration correction methods to mixed aberrations and low algorithm convergence performance. It improves the quality of iterative reconstruction, reduces the strict requirements of FPM on optical components and clinical application conditions, and provides a foundation for promoting high-quality FPM.

[0051] Experimental device model such as Figure 3 and Figure 4 As shown, the model mainly consists of a programmable LED array and a low numerical aperture (NA) objective lens (the present invention takes a 4× / 0.1NA objective lens as an example) in a conventional Fourier stacked microscopy system. The imaging process of FPM includes image acquisition and image reconstruction. During the image acquisition process, the LED array is lit sequentially to provide plane wave illumination samples with varying angles. The sample spectral information that exceeds the objective lens bandwidth is moved to within the objective lens NA and positioned on the back focal plane of the objective lens. At the same time, the corresponding low resolution (LR) image is captured at the camera port. Then, the phase retrieval algorithm is used to iteratively reconstruct the acquired LR image dataset.

[0052] Reference Figure 1 This invention provides an image reconstruction method based on Fourier layered microscopy imaging technology, which includes the following steps:

[0053] S1, Capture image;

[0054] Specifically, with the first Taking an LED as an example, each lit LED can provide a wave vector that varies with the angle. Illuminated sample, This can be represented as follows:

[0055] ;

[0056] In the above formula, Indicates the wavelength of light illuminating the air. Indicates the first Each LED incident angle;

[0057] Therefore, the intensity image captured by the camera can be represented as:

[0058] ;

[0059] In the above formula, Represents the two-dimensional spatial coordinates, indicating the position of each LED unit in the LED array. Represents two-dimensional Fourier domain spatial coordinates. This is represented as the sample in the corresponding number. The spectrum of LED lighting and These represent the Fourier transform and the inverse transform, respectively. The pupil function of the imaging system is considered as the coherence transfer function (CTF) in the FPM system, and was previously considered as an aberration-free ideal circular domain function in FPM algorithms. Specifically, it can be expressed as:

[0060] ;

[0061] In the above formula, This represents the pupil function of the imaging system. The numerical aperture of the objective lens. This represents the amplitude of the pupil function. Represents two-dimensional Fourier domain spatial coordinates. This indicates the wavelength of light illuminating the air.

[0062] S2. Initialize the spectrum of the sample to be reconstructed. With pupil function ;

[0063] Specifically, refer to Figure 5 Initialize the spectrum of the sample to be reconstructed With pupil function The Fourier transform of the LR image corresponding to the center of the LED array at normal incidence is taken as the initial HR spectrum, but the samples and the spectrum function are updated after each iteration. and For the first The sample spectrum and pupil function of the next iteration.

[0064] S3, Frequency Domain Constraints;

[0065] Specifically, the frequency domain constraint is the process of inverting the images captured by the camera when the sample is illuminated by LED units at different angles, as detailed below;

[0066] Using the pupil function The HR spectrum of the sample after the previous iteration is extracted. The extracted sub-spectral region corresponds to the frequency domain of the LR image obtained by the LED unit illuminating the sample, and can be represented as:

[0067] ;

[0068] In the above formula, Represented as an LR image index;

[0069] Performing an inverse Fourier transform on it yields the image acquired by the digital detector:

[0070] ;

[0071] In the above formula, This refers to the image received by the camera. Represented as LR image index, This indicates the sub-spectral region of the sample intercepted by the pupil function. This represents the inverse Fourier transform.

[0072] S4. Image Denoising and Intensity Constraints;

[0073] Specifically, by calculating the actual acquired images With target image The average intensity difference is used to determine the threshold: Then, this threshold is used to denoise the LR image, as shown in the following formula:

[0074] ;

[0075] In the above formula, This represents the image after noise reduction;

[0076] The denoised LR image dataset is used for amplitude replacement of the target complex amplitude information, and the replacement expression is as follows:

[0077] ;

[0078] Then, a Fourier transform was performed to obtain the spectrum of the target complex amplitude image after amplitude replacement, and its specific expression is shown below:

[0079] ;

[0080] In the above formula, This represents the Fourier transform.

[0081] S5. Update the target spectrum function and pupil function;

[0082] Specifically, an adaptive control factor is introduced. Optimize the update of the target spectrum function. The calculation formula is as follows:

[0083] ;

[0084] ;

[0085] In the above formula, This indicates the total number of LR images acquired. Represents the LR image index. This represents the average arithmetic threshold of the LR image dataset;

[0086] In addition, a ratio of the current value to the maximum value of the pupil function is set to prevent the impact of sudden changes in the pupil function and to more accurately estimate aberrations. The expression for the ratio of the current value to the maximum value of the pupil function is as follows:

[0087] ;

[0088] The formula for updating the spectrum function is as follows:

[0089] ;

[0090] In the above formula, This represents the updated sample spectrum. This represents the sample spectrum to be updated. This represents the ratio of the current value to the maximum value of the pupil function. This represents the complex conjugate operation of the current value of the pupil function. Represents the regularization parameter. , This represents the complex amplitude information of the target sample before and after strength constraint;

[0091] constant The spectral update process was optimized, resulting in better reconstruction quality and convergence speed. Considering the aberrations of the imaging system, the pupil function was updated in a similar manner.

[0092] ;

[0093] ;

[0094] In the above formula, This represents the updated pupil function. This indicates the pupil function to be updated. Indicates the adaptive control factor. The complex conjugate operation represents the current value of the sample spectrum function. Represents the regularization parameter;

[0095] in, Settings and Consistent, constant To optimize the update of the pupil function, in this invention, let This can achieve better correction results.

[0096] S6. Update and iterate on all LR images.

[0097] Specifically, all LR images are updated iteratively, repeating steps S3 to S5 until the pupil function and sample spectrum information in all sub-apertures are updated. At this point, one iteration is completed. Steps S3 to S6 are repeated to continue subsequent iterations until the algorithm's termination iteration condition is met. The iteration termination condition is that the difference between the target image and the real image tends to converge during the iteration process, which is the algorithm's iteration termination condition.

[0098] Reference Figure 6 The image shows a comparison of the reconstruction results of the method (AA-P algorithm) of this invention with those of other algorithms on an open-source biological sample dataset (HE pathological tissue sections);

[0099] Wherein, (a) represents the original LR image; (bd) represents the reconstruction results of other existing algorithms; and (e) represents the AA-P reconstruction result of the present invention.

[0100] Figure 6 To compare the reconstruction results of different existing FPM algorithms in open-source biological sample experiments, all four algorithms were used to recover LR datasets under conditions where aberrations existed in the unknown imaging system. This invention uses LR images captured by central LED array illumination as shown in the example. Figure 6 (a) Upsampling was used as the initial HR spectrum, and the reconstructed intensity images and phase images of four FPM reconstruction algorithms (AP algorithm, AA algorithm, EPRY-FPM algorithm and the AA-P algorithm proposed in this invention) were compared respectively.

[0101] Furthermore, such as Figure 6 As shown in (b1)-(b3), the reconstruction quality using the AP algorithm is the worst, with blurred image information in the reconstructed intensity image. Figure 6 (c1)-(c3) are the HR images restored by the AA algorithm. Compared with the AP algorithm, the AA algorithm significantly improves the sharpness of the intensity-reconstructed images, and the line contours are more distinct. However, some blurring still exists, and there is some crosstalk in the recovery of phase information. Figure 6(d1)-(d3) represent the reconstruction results of the EPRY-FPM algorithm. As a mainstream aberration correction algorithm, it significantly improves the quality of the reconstructed phase image compared to the previous two algorithms. However, due to the optimization strategy of the algorithm itself, the contrast of the phase image is still not high enough, and the restored intensity image still has local blurring problems. In contrast, the AA-P algorithm proposed in this invention can identify more details in the reconstructed HR pathological tissue sections, with clear and distinct tissue lines and the highest overall contrast. Figure 6 As shown in (e1)-(e3), in addition, Figure 6 (d4) and (e4) are the phase components of the pupil function recovered by two algorithms with aberration correction capabilities.

[0102] Reference Figure 2 An image reconstruction system based on Fourier layered microscopy includes:

[0103] The acquisition module is used to acquire low-resolution images through a Fourier in-plane microscopy system;

[0104] The module is built upon Fourier layered microscopy imaging technology to construct an improved phase retrieval algorithm;

[0105] The reconstruction module is used to iteratively reconstruct low-resolution images using an improved phase retrieval algorithm until the algorithm's iteration termination condition is met, thus obtaining the reconstructed image.

[0106] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0107] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. An image reconstruction method based on Fourier layered microscopy imaging technology, characterized in that, Includes the following steps: Low-resolution images are acquired using a Fourier layered microscopy system. Initialize the sample spectrum and pupil function; The initial sample spectrum is obtained by upsampling and Fourier transforming the image corresponding to the center LED in the low-resolution image, and using the result of the transformation as the initial sample spectrum. The initial pupil function is obtained by initializing the pupil function through the coherent transfer function. The image received by the camera is obtained by performing frequency domain constraints on the sample spectrum and pupil function based on the illumination wave vector, wherein the illumination wave vector is provided by each lit LED; The low-resolution image is denoised based on the image received from the camera and used as an intensity constraint to replace the sample spectrum, thus obtaining the replaced sample spectrum. An adaptive adjustment factor is introduced to iteratively update the spectrum and pupil function of the replaced samples until all low-resolution images have been updated, resulting in an improved phase retrieval algorithm. The formula for updating the pupil function is shown below: ； In the above formula, This represents the updated pupil function. This indicates the pupil function to be updated. This indicates the update used to optimize the pupil function. The complex conjugate operation represents the current value of the sample spectrum function. Represents the regularization parameter. , This represents the complex amplitude information of the target sample before and after strength constraint; The low-resolution image is reconstructed iteratively using an improved phase retrieval algorithm until the algorithm's iteration termination condition is met, resulting in the reconstructed image.

2. The image reconstruction method based on Fourier layered microscopy imaging technology according to claim 1, characterized in that, The Fourier inlay microscopy system includes a programmable LED array and a low numerical aperture objective lens. The step of acquiring a low-resolution image using the Fourier inlay microscopy system specifically includes: By sequentially illuminating the LEDs in the programmable LED array on the Fourier stacked microscopy system, plane wave illumination image samples with varying angles are obtained. Illuminating image samples with plane waves of varying angles is equivalent to shifting the different spatial frequency information of the samples into the observation range of a low numerical aperture objective lens; A series of low-resolution images of the sample are acquired by the camera, wherein the low-resolution images are low-resolution sub-images corresponding to different LED units.

3. The image reconstruction method based on Fourier layered microscopy imaging technology according to claim 2, characterized in that, The expression for the pupil function is as follows: ； In the above formula, This represents the pupil function of the imaging system. The numerical aperture of the objective lens. This represents the amplitude of the pupil function. Represents two-dimensional Fourier domain spatial coordinates. This indicates the wavelength of light illuminating the air.

4. The image reconstruction method based on Fourier layered microscopy imaging technology according to claim 3, characterized in that, The expression of the image received by the camera is as follows: ； In the above formula, This refers to the image received by the camera. Represented as LR image index, This indicates the sub-spectral region of the sample intercepted by the pupil function. This represents the inverse Fourier transform.

5. The image reconstruction method based on Fourier layered microscopy imaging technology according to claim 4, characterized in that, The step of performing noise reduction processing on the low-resolution image based on the image received from the camera and replacing the sample spectrum to obtain the replaced sample spectrum specifically includes: A preset target image is generated, and the average intensity difference between the image received by the camera and the preset target image is calculated. The low-resolution image is denoised based on the average intensity difference to obtain the denoised low-resolution image. The preset target image is subjected to amplitude replacement processing based on the denoised low-resolution image to obtain the replaced target image; The replaced target image is subjected to Fourier transform processing to obtain the spectrum of the replaced sample.

6. The image reconstruction method based on Fourier layered microscopy imaging technology according to claim 5, characterized in that, The expression for the adaptive control factor is as follows: ； In the above formula, Indicates the adaptive control factor. This represents the current value of the pupil function. This represents the maximum value of the pupil function. This represents the average arithmetic threshold for a low-resolution image dataset.

7. The image reconstruction method based on Fourier layered microscopy imaging technology according to claim 6, characterized in that, The formula for updating the spectrum of the replaced sample is shown below: ； In the above formula, This represents the updated sample spectrum. This represents the sample spectrum to be updated. This represents the ratio of the current value to the maximum value of the pupil function. This represents the complex conjugate operation of the current value of the pupil function. Represents the regularization parameter. , This represents the complex amplitude information of the target sample before and after strength constraints.

8. An image reconstruction system based on Fourier layered microscopy imaging technology, characterized in that, Includes the following modules: The acquisition module is used to acquire low-resolution images through a Fourier in-plane microscopy system; Build the module and initialize the sample spectrum and pupil function; The initial sample spectrum is obtained by upsampling and Fourier transforming the image corresponding to the center LED in the low-resolution image, and using the result of the transformation as the initial sample spectrum. The initial pupil function is obtained by initializing the pupil function through the coherent transfer function. The image received by the camera is obtained by performing frequency domain constraints on the sample spectrum and pupil function based on the illumination wave vector, wherein the illumination wave vector is provided by each lit LED; The low-resolution image is denoised based on the image received from the camera and used as an intensity constraint to replace the sample spectrum, thus obtaining the replaced sample spectrum. An adaptive adjustment factor is introduced to iteratively update the spectrum and pupil function of the replaced samples until all low-resolution images have been updated, resulting in an improved phase retrieval algorithm. The formula for updating the pupil function is shown below: ； In the above formula, This represents the updated pupil function. This indicates the pupil function to be updated. This indicates the update used to optimize the pupil function. The complex conjugate operation represents the current value of the sample spectrum function. Represents the regularization parameter. , This represents the complex amplitude information of the target sample before and after strength constraint; The reconstruction module is used to iteratively reconstruct low-resolution images using an improved phase retrieval algorithm until the algorithm's iteration termination condition is met, thus obtaining the reconstructed image.

Images