An iterative reconstruction method based on adaptive moment estimation FPM

Abstract

The present invention provides an iterative reconstruction method based on adaptive moment estimation FPM, including S1: Under a low-magnification objective lens, using an LED array as a light source, illuminating the sample by sequentially lighting the LED units in the LED array, and collecting a series of low-resolution intensity images of the sample corresponding to the LED positions; S2: Using each low-resolution intensity image to iteratively update the sub-aperture spectrum information corresponding to the LED positions in the support constraint domain in the matching frequency domain, and completing the high-resolution image restoration based on the adaptive moment estimation-based ptychographic reconstruction framework. The present invention realizes faster and more stable image iterative reconstruction of high-quality adaptive FP in a noise environment by using adaptive moment estimation.

Description

Technical Field

[0001] The present invention relates to the technical field of computational microscopy imaging, and particularly relates to an iterative reconstruction method based on adaptive moment estimation FPM. Background Art

[0002] Quantitative phase imaging (QPI) has become a general computational imaging technology widely used in the fields of cell and tissue observation due to its robustness and label-free imaging advantages. With the development of QPI, as one of its implementation technologies, Fourier ptychographic microscopy (FPM) technology has received continuous attention from scholars. FPM uses a programmable LED array as a light source in cooperation with a low-magnification objective lens to achieve aperture scanning and aperture synthesis in the frequency domain, effectively broadening the spatial gain bandwidth product (SBP) and equivalent numerical aperture (NA) of the system, and achieving high-resolution microscopy imaging effects under a large field of view with a low-magnification objective lens.

[0003] As the reconstruction core of FPM, the design of its ptychographic iterative engine (PIE) directly affects the performance of the reconstruction algorithm. In 2013, Zheng et al. designed the basic FP algorithm based on the GS framework, effectively recovering the intensity and phase information of the target in the field of view. However, the theoretical implementation of the basic FP is based on some premise assumptions, such as ignoring the thickness of the sample, partially coherent light, assuming that the low-magnification lens is an ideal low-pass filter, and ignoring noise, etc. These objectively existing influencing factors greatly limit the use effect of FP in practical applications. Subsequently, the ePIE and EPRY algorithms were proposed, improving the robustness of the PIE and further relaxing the requirements of the actual hardware system for FP. The designs of ePIE and EPRY treat the aperture function and the sample spectrum equally, and achieve collaborative recovery under the condition of no system aberration prior through the alternating projection algorithm, becoming the two most widely used algorithms in the current FP field. However, these two algorithms still have some defects. First, the designs of ePIE and EPRY are based on the traditional gradient descent algorithm, which means that the algorithm is easily trapped in a local optimal solution. Second, to ensure the convergence of the algorithm, the step size of the algorithm uses the maximum value as the denominator term, slowing down the speed of the algorithm. Finally, the algorithm is susceptible to system environmental noise.

[0004] To further improve the performance of the PIE algorithm, rPIE and mPIE have been successively proposed. rPIE redesigned PIE and ePIE in the form of a convex combination, enhancing the optimization ability of the algorithm to a certain extent. Based on rPIE, mPIE first introduced the concept of machine learning and added a momentum acceleration module, enabling the algorithm to have the ability to cross local optima and becoming the currently optimal algorithm framework. Compared with other PIE algorithms, mPIE has achieved great breakthroughs in the algorithm convergence speed and reconstruction performance. However, the above advantages are obtained at the cost of seven additional introduced tuning hyperparameters and a cumbersome parameter tuning process. Moreover, the hyperparameter settings of mPIE are not applicable to different microscopic imaging scenarios, making it difficult for users or researchers to find a set of parameter settings suitable for their own needs. During the tuning process, some overly large parameter settings are likely to cause mPIE to fail in early reconstruction.

[0005] The traditional FP iterative framework has significant defects such as slow reconstruction speed and easy entry into local optima. Although the currently optimal FP design framework introduced the concept of machine learning and overcame the defects of the traditional FP algorithm, it brought new problems such as too many tuning hyperparameters, complex parameter tuning, and low algorithm applicability, greatly limiting the practical application and subsequent development of the algorithm. Summary of the Invention

[0006] To solve the above problems, the present invention provides an iterative reconstruction method based on adaptive moment estimation FPM. Aiming at the problems in the currently optimal Fourier ptychography iterative framework, such as too many tuning hyperparameters and heavy parameter tuning tasks, which greatly limit practical applications, the present invention uses adaptive moment estimation to achieve high-quality iterative reconstruction of adaptive FP in a noise environment. It not only overcomes the problem of easy entry into local minima commonly existing in traditional FP iterative reconstruction algorithms, but also completes high-quality, faster, and more stable image reconstruction with the introduction of very few tuning parameters, having the advantages of wide parameter adaptability and high algorithm robustness.

[0007] The present invention provides the following specific technical solutions:

[0008] S1: Under a low-magnification objective lens, an LED array is used as a light source, and the sample is illuminated by sequentially lighting the LED units in the LED array, and a series of low-resolution intensity images corresponding to the LED positions of the sample are collected.

[0009] S2: Using each of the low-resolution intensity images to iteratively update the sub-aperture spectrum information corresponding to the LED positions in the support constraint domain in the matching frequency domain.

[0010] Starting from an initial spectral estimate value O0 and an initial pupil function P, based on the ptychographic reconstruction framework of adaptive moment estimation, the high-resolution image recovery is completed to restore the low-resolution intensity image to a high-resolution image.

[0011] Further, step S1 is specifically as follows:

[0012] S101: Select an LED array with M×N LED units and place it on a horizontal plane at a set vertical distance directly below the sample;

[0013] S102: Light any LED unit in the LED array to irradiate the sample. The sample modulates the illumination wave vector emitted by the LED unit and then emits it to the objective lens. After being modulated by the objective lens, it propagates to the camera image plane to obtain a low-resolution intensity image;

[0014] S103: Based on the LED array, light the LED units in the array one by one to irradiate the sample from different angles, and obtain a series of low-resolution intensity images of the sample corresponding to the LED positions.

[0015] Further, the vertical distance is much greater than the thickness of the sample.

[0016] Further, step S2 is specifically as follows:

[0017] S201: Convert the sub-aperture information at the current updated position into complex amplitude information in the spatial domain;

[0018] S202: Use the collected low-resolution intensity image as the intensity-limited replacement amplitude part to obtain updated complex amplitude information and convert it into frequency domain information;

[0019] S203: Based on the FP background, design a gradient update term for the sample spectrum O;

[0020] S204: According to the gradient update term, construct the first-order moment estimate and second-order moment estimate of the sample;

[0021] S205: Use the first-order moment estimate and second-order moment estimate of the sample for iterative update of the pupil function and the corresponding sub-spectrum at the corresponding position.

[0022] Further, in step S203, the gradient update term is specifically as follows:

[0023]

[0024] where α is a tuning control hyperparameter in the original rPIE algorithm, which is used here to control the linear direction component in the gradient update term and is a fixed value; m and n represent the row and column of the LED unit in the LED array; i represents the current iteration number; P (m,n)-1 (k + k m,n ) represents the pupil function of the objective lens in the frequency domain at the previous moment at position (m, n); |P (m,n)-1 (k + k m,n )| maxRepresents the maximum value of the modulus of the pupil function defined with P; represents the sub-spectrum after replacing the amplitude at the sub-aperture position (m,n) in the i-th iteration; represents the initial sub-spectrum at the sub-aperture position (m,n) in the i-th iteration.

[0025] Furthermore, in step S204, the first-order moment of the sample is estimated as:

[0026] m obj,0 =0

[0027] m obj,t =β1m obj,t-1 +(1-β1)g obj,t ,

[0028] Among them, m obj represents the first-order moment estimate of the sample, t and t-1 represent two adjacent positions; represents the first-order moment attenuation coefficient;

[0029] The second moment estimate of the sample is:

[0030] v obj,0 =0

[0031]

[0032] Among them, v obj represents the second-order moment estimation of the sample, β1 and β2 are fixed universal hyperparameters, representing the first-order moment attenuation coefficient and the second-order moment attenuation coefficient, respectively, and g obj,(m,n) represents the gradient update term.

[0033] Furthermore, in step S204, it also includes performing adaptive bias correction on the first-order moment and the second-order moment during the iteration period, as follows:

[0034]

[0035]

[0036] in, and represent the corrected first-order moment and second-order moment respectively.

[0037] Furthermore, in step S205, the sub-spectrum based on the first-order moment and the second-order moment of the position number (m, n) is updated as follows:

[0038]

[0039] Among them, eps represents a very small fixed value, and the * operator represents the complex conjugate operation;

[0040] The pupil function P is updated as follows:

[0041]

[0042] Among them, and respectively represent the first - order moment estimation and the second - order moment estimation of the pupil function.

[0043] Furthermore, in step S205, an adaptive step - size attenuation function of the sample spectrum and the pupil function is introduced to participate in the iterative update, specifically as follows:

[0044]

[0045]

[0046] Among them, and respectively represent the adaptive step - size attenuation function of the sample spectrum and the pupil function.

[0047] The beneficial effects of the present invention are as follows:

[0048] The present invention proposes an FPM image reconstruction method combined with adaptive moment estimation. By introducing the first - order moment estimation and the second - order moment estimation corresponding to FP for iterative update, high - resolution image restoration is completed. In the design of adaptive moment estimation, the hyperparameters included are fixed values and no further tuning is required, realizing adaptive high - quality iterative reconstruction of FP in a noisy environment, making the image reconstruction faster and more stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a schematic diagram of the overall process flow of the method of the present invention;

[0050] Figure 2 is a schematic diagram of the low - resolution intensity image acquisition process of the present invention;

[0051] Figure 3 is a schematic diagram of the image reconstruction process based on adaptive moment estimation of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0052] In the following description, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0053] Embodiment 1

[0054] Embodiment 1 of the present invention discloses that, as Figure 1 shown, the specific step - by - step process is as follows:

[0055] S1: Under the low-magnification objective lens, illuminate the sample by sequentially lighting the LED units in the LED array, and collect a series of low-resolution intensity digital images of the sample corresponding to the LED positions.

[0056] Use the low-magnification objective lens to provide a large field of view for microscopic imaging. Provide different oblique illumination conditions by sequentially lighting the LED units in the LED array, capture a series of low-resolution intensity digital images of the sample corresponding to the LED positions on the camera imaging plane, and save them to the system hard disk for subsequent reconstruction.

[0057] The low magnification mentioned here is a relative concept. In practice, generally, 1x, 2x, 4x, etc. are used to reconstruct images of 10x, 20x, 40x or higher magnification objectives; even 10x images are used to reconstruct 50x, etc.; there is no specific limitation in the field of microscopic imaging.

[0058] Combine Figure 2 As shown in

[0059] S101: Select an LED array with M×N LED units and place it on a horizontal plane at a set vertical distance directly below the sample;

[0060] During the image acquisition process of this embodiment, select an LED array board with M×N units as the light source and place it on the horizontal plane directly below the sample to provide different angles of oblique illumination conditions.

[0061] Assume that the vertical distance between the light source and the sample is h. Among them, the vertical distance h is much larger than the thickness of the sample, that is, the sample thickness can be ignored for h. Then, in the LED array, the illumination wave vector k m,n = (k x,m,n , k y,m,n ) of a single LED with a central wavelength of λ located in the m-th row and n-th column can be expressed as:

[0062]

[0063]

[0064] Among them, (x m,n , y m,n ) and (x c , y c ) respectively represent the position coordinates of the currently lit LED and the central LED.

[0065] For simplicity of description, normalize the initial light field amplitude part to 1, and record the coordinates (x, y) as the vector r; when the sample is illuminated by the above plane wave, then the spectrum O m,n can be expressed as:

[0066] O m,n = F{e m,n (r)} = F{o(r)exp(ik m,n r)} = O(k - k m,n ),

[0067] where F represents the two-dimensional Fourier transform, e m,n (r) represents the complex amplitude exiting from the sample, o(r) represents the spatial amplitude part, and O(k) represents the sample spectrum under vertical illumination conditions.

[0068] S102: Light any LED unit in the LED array to irradiate the sample. After the sample modulates the illumination wave vector emitted by the LED unit, it is transmitted to the objective lens. After being modulated by the objective lens, it propagates to the camera image plane to obtain a low-resolution intensity image;

[0069] The light field is modulated by the objective lens and finally propagates to the camera image plane to form an intensity image. The description of this process is as follows:

[0070] I m,n = |F -1 {O m,n P(k)}| 2 = |F -1 {O(k - k m,n )P(k)}| 2

[0071] where I m,n represents the captured low-resolution intensity map, F -1 represents the two-dimensional inverse Fourier transform, and P(k) represents a low-pass filtering function of the objective lens in the frequency domain.

[0072] S103: Based on the LED array, light the LED units in the array one by one to irradiate the sample from different angles, and obtain a series of low-resolution intensity images of the sample corresponding to the LED positions.

[0073] Specifically, light all the LED units in the array one by one, repeat the above process, and finally obtain a set of low-resolution intensity images of the sample, which is expressed as follows:

[0074] I m,n (1 ≤ m ≤ M, 1 ≤ n ≤ N)

[0075] where m and n respectively represent the row and column of the current LED in the array, and M represents the row and column of the LED array in this embodiment.

[0076] S2: Use each of the low-resolution intensity images to iteratively update the sub-aperture spectrum information of the corresponding LED position in the matching frequency domain support restricted region;

[0077] Starting from an initial spectral estimate O0 and an initialized pupil function P, based on the adaptive moment estimation-based ptychographic reconstruction framework, the high-resolution image restoration is completed by restoring the low-resolution intensity image to a high-resolution image.

[0078] In this embodiment, the concept of adaptive moment estimation is used to design the first-order moment estimation and second-order moment estimation update formulas corresponding to FP;

[0079] Among them, the first-order moment estimation is designed to introduce a new optimization direction and provide the algorithm with the ability to cross local minima, and the second-order moment estimation is designed to ensure the convergence of the algorithm and balance the overall update scale;

[0080] In this embodiment, in the adaptive moment estimation design, the hyperparameters included are fixed values and no further tuning is required.

[0081] Combined with Figure 3 As shown, the specific steps are as follows:

[0082] S201: Convert the sub-aperture information at the current update position to the complex amplitude information in the spatial domain;

[0083] S202: Use the collected low-resolution intensity image as the intensity-limited replacement amplitude part to obtain the updated complex amplitude information and convert it to the frequency-domain information;

[0084] S203: Based on the FP background, design the gradient update term for the sample spectrum O;

[0085] S204: Construct the first-order moment estimation and second-order moment estimation of the sample according to the gradient update term;

[0086] S205: Use the first-order moment estimation and second-order moment estimation of the sample for iterative update of the pupil function and the sub-spectrum at the corresponding position.

[0087] In this embodiment, the above steps are as follows:

[0088] Taking a certain iteration process as an example, when updating the sub-aperture information at the position (m, n), the sub-aperture information at this position will be picked up and converted to the complex amplitude information in the spatial domain

[0089] Specifically as follows:

[0090]

[0091] Among them, i represents the current iteration number, represents the sample spectrum at the previous moment before updating the current position (m, n).

[0092] The collected intensity image I m,nAs the intensity limit replaces the amplitude part of the above formula, specifically as follows:

[0093]

[0094] where φ represents the updated complex amplitude of the sample.

[0095] Convert the updated complex amplitude to frequency-domain information Specifically as follows:

[0096]

[0097] Combined with the FP background, for the sample spectrum O, design the gradient update term g in the machine learning concept o b j,(m,n) :

[0098]

[0099] where α is a tuning control hyperparameter in the original rPIE algorithm, which is used here to control the linear direction component in the gradient update term and is a fixed value.

[0100] Based on the above process, construct the first-order moment estimate and the second-order moment estimate of the sample;

[0101] Starting from position t = 0, the first-order moment guides the update direction of the algorithm and provides the ability for the algorithm to cross local optima; specifically as follows:

[0102] m obj,0 = 0

[0103] m obj,t = β1m obj,t-1 +(1 - β1)g obj,t ,

[0104] where m obj represents the first-order moment estimate of the sample, t and t - 1 represent two adjacent positions, and β1 is a fixed general introduced hyperparameter, representing the first-order moment decay coefficient, which is used to control the update component of the first-order moment and does not need to be tuned for different usage environments.

[0105] The second-order moment update design ensures the convergence of the algorithm and balances the overall scale; specifically as follows:

[0106] v obj,0 = 0

[0107]

[0108] where v obj represents the second-order moment estimate of the sample, and β2 is a fixed general introduced hyperparameter, representing the second-order moment decay coefficient, which is used to control the update component of the second-order moment and does not need to be tuned for different usage environments.

[0109] In this embodiment, in order to prevent the moment estimation from being biased towards 0 in the early stage of updating, an adaptive bias correction of the above moments is performed with respect to the iteration period; specifically as follows:

[0110]

[0111]

[0112] The first-order moment and the second-order moment at this position are used for the sub-spectrum update at the corresponding position;

[0113] Specifically as follows:

[0114]

[0115] In the formula, eps represents a very small fixed value to prevent the denominator from being 0 and the * operator represents the complex conjugate operation, P (m,n)-1 (k + k m,n ) represents the pupil function of the objective lens in the frequency domain at the previous moment at the position (m, n); |P (m,n)-1 (k + k m,n )| max represents the maximum value of the modulus of the pupil function defined by P; represents the sub-spectrum after replacing the amplitude at the sub-aperture position (m, n) in the i-th iteration; represents the initial sub-spectrum at the sub-aperture position (m, n) in the i-th iteration.

[0116] Similar to the design of the sample spectrum O, the pupil function P is designed and updated in the same mode;

[0117] Finally, the update of P is specifically as follows:

[0118]

[0119] In the formula, and respectively represent the first-order moment estimation and the second-order moment estimation of the pupil function;

[0120] Similarly, and are calculated in the same way as the calculation processes of the first-order moment estimation and the second-order moment estimation of the above samples.

[0121] Due to the existence of noise in practice, when the first-order moment and the second-order moment are accumulated and updated, the noise is also continuously accumulated and gradually affects the reconstruction result after each overall iteration; when the target information gradually flattens out and converges, the proportion of target information update gradually decreases, while the noise accumulation will dominate the update, ultimately resulting in a decline in image quality or even serious degradation; considering that FP always shows the characteristics of fast update in the early stage and slow update in the later stage, in this embodiment, an adaptive step-size decay function is introduced to solve the influence of noise on the reconstructed image.

[0122] Its function in the update of the sample spectrum O and the pupil function P is as follows:

[0123]

[0124]

[0125] Among them, and respectively represent the adaptive step-size decay functions of the introduced sample spectrum and pupil function; they have a unified form, and the specific design is as follows:

[0126]

[0127] Among them, η represents the decay factor, which is optimized and selected according to the actual sample reconstruction difficulty and the system noise level; [] represents the Gaussian rounding function.

[0128] Based on the above process, when cycling through and updating the sub-aperture spectrum information corresponding to the positions of all LED units in the array until the set maximum number of iterations is reached or the error convergence condition is satisfied, the algorithm can be determined to converge, and high-resolution spectrum reconstruction is completed.

[0129] In this embodiment, the error convergence function used is as follows:

[0130]

[0131] Among them, E i obtains the minimum value or (E i - E i-1 ) / E i-1 is less than a certain set threshold, and it is considered that the error convergence condition is satisfied.

[0132] The present invention is not limited to the foregoing specific embodiments. The present invention extends to any new feature or any new combination disclosed in this specification, as well as any new method or process step or any new combination disclosed.

Claims

1. An iterative reconstruction method based on adaptive moment estimation FPM, characterized in that, Including: S1: Under the objective lens, using an LED array as the light source, illuminating the sample by sequentially lighting the LED units in the LED array, and collecting a series of low-resolution intensity images of the sample corresponding to the LED positions. S2: Using each of the low-resolution intensity images to iteratively update the sub-aperture spectrum information corresponding to the LED positions in the matching frequency domain. With an initial spectral estimate value and an initialized pupil function As a starting point, based on the adaptive moment estimation-based ptychographic reconstruction framework, complete the high-resolution image restoration to restore the low-resolution intensity image to a high-resolution image; The specific process is as follows: S201: Converting the sub-aperture information at the current updated position into complex amplitude information in the spatial domain. S202: Using the collected low-resolution intensity image as the intensity-limited replacement amplitude part to obtain updated complex amplitude information, and converting it into frequency domain information. S203: Based on the FP background, for the sample spectrum , design the gradient update term as follows: Among them, is a tuning control hyperparameter in the original rPIE algorithm and is a fixed value; m and n represent the row and column of the LED unit in the LED array; i represents the current iteration number; represents the pupil function of the objective lens in the frequency domain at the previous moment at the position (m, n); represents the maximum value of the modulus of the pupil function defined; represents the sub-spectrum after replacing the amplitude at the sub-aperture position (m, n) in the represents the initial sub-spectrum at the sub-aperture position (m, n) in the S204: Constructing the first-order moment estimate and second-order moment estimate of the sample according to the gradient update term. S205: Using the first-order moment estimate and second-order moment estimate of the sample for iterative update of the pupil function and the sub-spectrum at the corresponding position.

2. The iterative reconstruction method according to claim 1, wherein For step S1, the specific process is as follows: S101: Select an LED array with LED units and place it on a horizontal plane at a set vertical distance directly below the sample; S102: Lighting any one LED unit in the LED array to irradiate the sample. After the sample modulates the illumination wave vector emitted by the LED unit, it is emitted to the objective lens, and after being modulated by the objective lens, it propagates to the camera image plane to obtain a low-resolution intensity image. S103: Based on the LED array, sequentially lighting the LED units in the array to irradiate the sample from different angles, and obtaining a series of low-resolution intensity images of the sample corresponding to the LED positions.

3. The iterative reconstruction method according to claim 2, wherein The vertical distance is much larger than the thickness of the sample.

4. The iterative reconstruction method according to claim 1, wherein In step S204, the first-order moment estimate of the sample is: Among them, represents the first-order moment estimate of the sample, and represent two adjacent positions; represents the first-order moment decay coefficient; The second-order moment estimate of the sample is: Among them, represents the second moment estimate of the sample, and are fixed general introduced hyperparameters, representing the first moment decay coefficient and the second moment decay coefficient respectively, represents the gradient update term.

5. The iterative reconstruction method according to claim 1, wherein In step S204, it also includes adaptive bias correction for the first-order moment and second-order moment during the iteration period, specifically as follows: Among them, and respectively represent the corrected first-order moment and second-order moment.

6. The iterative reconstruction method according to claim 5, wherein Step S205, update the sub-spectrum of the first moment and the second moment based on the position sequence number as follows: Among them, represents an extremely small fixed value, The operator represents the complex conjugate operation; Pupil function Updated as follows: Among them, and respectively represent the first-order moment estimation and the second-order moment estimation of the pupil function.

7. The iterative reconstruction method according to claim 6, wherein In step S205, an adaptive step size attenuation function of the sample spectrum and the pupil function is introduced to participate in the iterative update, specifically as follows: Among them, and respectively represent the adaptive step attenuation functions of the sample spectrum and the pupil function.

Images