A quantitative phase imaging device, computer program and method thereof

Abstract

A quantitative phase imaging device comprising processing circuitry configured to: receive a stack of holograms; perform initialization to obtain initial estimates of latent optimization variables; perform shallow complex residual encoding; estimate latent optimization variables of a latent complex transmission distribution by performing iterations in an unrolled network, perform shallow complex residual decoding; and reconstruct the latent complex transmission distribution.

Description

[0001]EU-IPD: SYP354473WO01 DYC Ref: P129332PCT A QUANTITATIVE PHASE IMAGING DEVICE, COMPUTER PROGRAM AND METHOD THEREOF BACKGROUND Field of the Disclosure The present technique relates to a quantitative phase imaging device, computer program and method thereof. Description of the Related Art The “background” description provided herein is for the purpose of generally presenting the context ofthe disclosure. Work of the presently named inventors, to the extent it is described in the backgroundsection, as well as aspects of the description which may not otherwise qualify as prior art at the time of filing, are neither expressly or impliedly admitted as prior art against the present technique.Holography enables intriguing microscopic imaging modalities, particularly through Quantitative PhaseImaging (QPI), which utilizes the phase of coherent light as a way to reveal the contrast in transparent andthin microscopic specimens. Despite the limitation of image sensors, which detect only light intensity,phase information can still be recorded as a two-dimensional interference pattern between two distinctlight waves. Numerical reconstruction is needed to retrieve the amplitude and phase distributions fromsuch holographic measurements. A known imaging technique referred to as lensless inline holographic imaging or digital in-line holographic microscopy has been developed to provide small and low cost microscopy devices foranalyzing biological samples or specimen without the need for dedicated optical components in the setup.Lensless inline holographic imaging is particular suitable for situations such as point-of-care diseasediagnosis where advanced laboratory facilities are not available.In view of this, there is expected to be a desire for a lensless in-line holographic imaging system thatprovides spatial super-resolution resolution capability such that images having higher resolutions thanthose restricted by the diffraction limit can be obtained with a more efficient and less complex model. It is an aim of the disclosure to at least address this issue. SUMMARYAccording to one aspect of the disclosure, there is provided a quantitative phase imaging devicecomprising processing circuitry configured to: receive a stack of holograms; perform initialization toobtain initial estimates of latent optimization variables; perform shallow complex residual encoding;estimate latent optimization variables of a latent complex transmission distribution by performing EU-IPD: SYP354473WO01 DYC Ref: P129332PCTiterations in an unrolled network, perform shallow complex residual decoding; and reconstruct the latentcomplex transmission distribution. The foregoing paragraphs have been provided by way of general introduction, and are not intended tolimit the scope of the following claims. The described embodiments, together with further advantages,will be best understood by reference to the following detailed description taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS A more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:Figure 1 describes a quantitative phase imaging device according to embodiments of the disclosure;Figure 2 describes the model architecture of an interpretable, learning-based method for in-lineholographic image reconstruction in the quantitative phase imaging device of Figure 1 according toembodiments of the disclosure; Figure 3 describes ADMM iterations in an unrolled network architecture in the model architecture of Figure 2 according to embodiments of the disclosure; and Figure 4 shows a flow chart describing a process in the quantitative phase imaging device for reconstructing complex transmission distribution according to embodiments of the disclosure. DESCRIPTION OF THE EMBODIMENTSReferring now to the drawings, wherein like reference numerals designate identical or correspondingparts throughout the several views. Numerous modifications and variations of the present disclosure are possible in light of the aboveteachings. It is therefore to be understood that within the scope of the appended claims, the disclosuremay be practiced otherwise than as specifically described herein. 1 OverviewThe present disclosure provides interpretable and learning-based methods for in-line holographic imagereconstruction. The methods according to embodiments of the present disclosure enhance imagingcapability with spatial super-resolution, offer a versatile framework that accommodates multipleillumination wavelengths and support extensive refocusing range with a precision of up to 10 μm. Themethods according to embodiments of the present disclosure demonstrate effective adaptation to realholographic data obtained from a Digital In-line Holographic Microscope (DIHM). The present disclosure EU-IPD: SYP354473WO01 DYC Ref: P129332PCT not only advances holographic imaging techniques but also broadens the potential for detailed microscopic analysis.In this section, an in-line holographic image formation model will be described together with the problemformulation and a model of end-to-end learning for QPI according to embodiments of the presentdisclosure. 2.1 Image Formation ModelFigure 1 describes a quantitative phase imaging device according to embodiments of the disclosure. Theschematics in Figure 1 illustrate the basic setup of Gabor’s in-line lensless holographic imaging system100. In some embodiments, the in-line lensless holographic imaging system 100 may comprise anillumination device 110, an object plane 120, multiple detector planes 130, processing circuitry 140 andstorage 150. The illumination device 110 may provide incident wave such as white-light illumination, orillumination with a single wavelength, or illumination with multiple single wavelengths. The illuminationmay be coherent or partially coherent illumination provided by laser projection devices. Specifically, theuse of broadband illumination allows enhancement of the spatial sensitivity of QPI. The incident wave ofthe illumination device 110 is projected to the object located on the object plane 120. The object may be abiological sample or specimen, and particularly, one or more transparent or translucent samples whichcannot be detected with brightfield microscopic technologies. After passing through the object on theobject plane 120, the incident wave becomes a scattered wave and is detected by a stack of detectorplanes 130. Although Figure 1 shows only four detector planes, any number of detector planes isenvisaged.Additionally, the detector planes 130 are separated from the object plane 120 by different distances, z =[z1, ..., zN]. Each of the detector planes 130 contains an array of sensors in the (u, v) plane for capturingthe scattered wave as an individual hologram. The hologram refers to the recorded interference patterngenerated when light waves interact with the specimen. When illumination passes through the specimen,its phase changes according to the refractive index of the specimen at each point. These phase changes are captured as variations in intensity in the recorded hologram. The multiple holograms obtained by the stack of detector planes 130 are transmitted to the processingcircuitry 140 in the form of a hologram stack, H = [h1, ..., hN]. The processing circuitry 140 is embodiedas circuitry and may be any solid state circuitry such as circuitry controlled by software or an application specific integrated circuit. The processing circuitry 140 is connected to the detector planes 130. This connection may be over a wired or wireless connection. The processing circuitry 140 is also connected to storage 150. In embodiments, the storage 150 is solid-state storage, but is not so limited and may be optically readable storage or the like. Moreover, the storage 150 may be located remote to the in-linelensless holographic imaging system 100. In embodiments, the storage 150 contains computer readable EU-IPD: SYP354473WO01 DYC Ref: P129332PCT instructions which, when loaded onto the processing circuitry140, configures the in-line lensless holographic imaging system 100 to perform a method or methods according to embodiments of the disclosure.In the following, bold lower-case letters will be used to represent vectors, upper-case letters to representmatrices, and lowercase letters to represent functions. Given a latent complex transmission field at theobject plane ^ ∈ ℂℎ^ and in high-resolution space with spatial dimensions h × w, for each height zi, i =1, ..., N a hologram can be simulated by: where ^: ℂℎ^ ↦ ℝℎ′^′ is the forward in-line holographic image formation model, the latent field x ispropagated to the detector planes 130 using the complex near-field Fresnel propagation kernel ^^^ . Thereal valued hologram is obtained by applying the square of the modulus of ^^^^ which is then warpedusing the matrix Wτ simulating spatial shifts in the (u, v) plane with sub-pixel accuracy defined by the set^^ = (^ ^^ , ^ ^^ ), and Ds is a down-sampling matrix which reduces the size of an image by a factor s whereh' = h / s and w' = w / s. The lateral complex transmission distribution refers to the underlying spatialdistribution of complex transmission coefficients within the specimen, and characterizes how theamplitude and phase varies as the illumination propagates through different parts of the specimen. Thelatent field x is capable of representing various physical properties of the biological sample or specimensuch as the refractive index distribution, thickness variations, or variations of other optical features withinthe specimen. On the other hand, spatial shift refers to the displacement of the foregoing optical featureswithin the specimen relative to a reference position, for example, a reference hologram. Spatial shifts maybe caused by changes in the position of the specimen, vibrations, or drift in the imaging system over time.Accurate estimation of spatial shifts is necessary in order to enable image super-resolution capability.Notice that the image formation model as described here is non-linear, more general, and physicallyaccurate where the object is presumed to have both absorption and phase shift properties even if theformer is in some cases weak when imaging thin and transparent micro-organisms. Additionally, sensorread and shot noise sources are also simulated using the noise model: where σ is the pixel-dependent standard deviation of the overall noise at a pixel i, while α and γ arerespectively the variances of shot and read noises, and y(i) is the input pixel value. Further information onthe structure and functioning of the noise model can be found in [1]. 2.2 Problem Formulation EU-IPD: SYP354473WO01 DYC Ref: P129332PCT Figure 2 shows the model architecture 200 of an interpretable, learning-based method for in-lineholographic image reconstruction in the quantitative phase imaging device of Figure 1 according toembodiments of the present disclosure. The model architecture 200 comprises an initialization block 220,a shallow complex residual encoder 230, an unrolled network 240, and a shallow complex decoder 250. The initialization block 220 acquires hologram stack 210 from the detector planes 130, and performsinitialization to obtain initial estimates of latent optimization variables. Latent optimization involvesfinding the best parameters or model that describes the phase information recorded in the hologram stack.This optimization is typically performed by the unrolled network 240 which adjusts parameters tominimize the difference between the measured holographic data and the predicted data based on the current model.The initial estimates are then fed into the shallow complex residual encoder 230 such that the subsequentADMM iterations in the unrolled network 240 is operated in the complex domain to obtain the fieldestimate. The unrolled network 240 will be described later with reference to Figure 3. Finally, the shallowcomplex residual decoder 250 converts the field estimate obtained by the ADMM iterations fromcomplex domain to spatial domain. In some embodiments, the foregoing components of the modelarchitecture 200 may be implemented by processing circuitry 140.One of the objectives of the embodiments of the present disclosure is to reconstruct a spatially super-resolved complex transmission distribution from a given stack of noisy low-resolution input holograms H= [h1, ..., hN] captured at N different heights z = [z1, ..., zN] with spatial shifts ^ = In some embodiments, this problem can be reformulated as a regularized least squares minimization:(^, ^)̃ = arg where Ψ is a regularizer that constrains the possible solution space on the distribution of the latent field x,and β is a discrepancy parameter controlling the strength of such regularization. A regularizer is used ininverse problem solving to further constraint the solution space, and it is a form of prior knowledgeimposed on the desired solution, such as smoothness or sparse edges. Every one of those constraints canbe modelled by a function Ψ. By minimizing (or in some cases maximizing) such function, the solution isforced to obey the prior knowledge modelled by Ψ. On the other hand, β is a parameter that controls theeffect of Ψ, as too much regularization will result in over-smoothed image or other undesirable artefacts.Therefore, a suitable value for β may be chosen by taking into account of the trade-off between datafidelity and regularization.Accurate prediction of z is important in order to reconstruct x as the field at the detector plane can beback-propagated to the object plane using the conjugate of the Fresnel propagation kernel ^∗^ , z can beaccurately estimated using a known computational refocusing technique, such as the technique described EU-IPD: SYP354473WO01 DYC Ref: P129332PCTin [6]. Spatial shifts τ need to be estimated with sub-pixel accuracy in order to enable multi-frame imageregistration and thus spatial image super-resolution capability.According to embodiments of the present disclosure, the optimization framework as expressed in Eq.3 isnon-linear in all optimization variables (^, ^)̃ and is solved by alternating between the two latentoptimization variables, iteratively optimizing for a single variable at a time while keeping the other onefixed: (^) ^ = arg (^) ^̃ = arg min^^^ − ^^,^, (^)^^ (5) ^ ^^ ^2.3 End-to-end Learning for QPISolving sub-problem (A): Eq. 4 is a regularized non-linear least squares with no closed-form solution for^, a good approximation can be obtained iteratively by choosing a proper image prior such as TotalVariation or other natural prior statistics. Further details of the Total Variation prior statistics can befound in [5]. In accordance with embodiments of the present disclosure, a variable splitting technique isused to solve Eq. 4, namely the Alternating Direction Method of Multipliers (ADMM) where an auxiliaryvariable v is introduced such that:(^, ^) = arg Further information on the ADMM method can be found in [2]. Since Eq. 6 is now a constrained versionof the previous formulation in Eq. 4 where the data-fidelity and prior terms are no longer coupled andconsequently they can be solved separately. To this end, Eq. 6 can be further split into multiple sub-problems by first retrieving its augmented Lagrangian: where u is the scaled Lagrange multiplier and ρ is a penalty term that enforces the constraint in Eq. 6 sothat the final estimate v would be as close as possible to the true solution x. According to embodiments ofthe present disclosure, in order to minimize Eq. 6, the saddle point of Eq. 7 is found by iterativelyupdating the following three sub-problems: ^← arg minℒ^^^^^ (^, ^, ^) = arg^ ^← arg minℒ^^^^^ (^, ^, ^) = arg min^‖^ + ^ − ^‖ ^^ + ^^(^) (9)^ ^ ^ ^← ^ + ^ − ^ (10) EU-IPD: SYP354473WO01 DYC Ref: P129332PCTIn each ADMM iteration x, 310, 320 in Eq.8 is updated, in accordance with embodiments of the presentdisclosure, using multiple steps of plain gradient descent 311, 321 or other known gradient basedapproach like, for example, conjugate gradient. Since x is complex, gradients are calculated usingWirtinger derivatives with both the real as well as the imaginary parts of x updated via: where α is a learning rate and n is a ADMM iteration. Further information on the Wirtinger derivativescan be found in [4].According to embodiments of the present disclosure, the gradient calculation requires defining thebackward model of ^^^,^,^which is refer to as ^^,^,^ , it is approximated by first up-sampling all N lowresolution input holograms, back-propagating each hologram in the stack using ^∗^^ , i = 1, ...,N back to theobject plane, and aligning the resulting complex images, the final output xb is obtained by averaging all Naligned complex fields. One can see that Eq. 9 presents a simple denoising problem with the identitymatrix I as the forward model and with a noisy input x + u, the target is to estimate a clean complex fieldv. In principle any known plug-and-play denoiser should be suitable to solve Eq. 9. However, sinceembodiments of the present disclosure deals with an end-to-end learning approach, the denoiser 312, 322in this case is a trainable Convolutional Neural Network (CNN) where the weights are learned in an end-to-end fashion. Finally, the update step of u, 313, 323 is performed.Figure 3 describes ADMM iterations 310, 320 in an unrolled network architecture in the modelarchitecture of Figure 2 according to embodiments of the present disclosure. In particular, Figure 3 showsa detailed implementation of the update rule, according to embodiments of the disclosure, for any givenxn where L steps of gradient descent 311, 321 are first performed to solve Eq. 8 followed by a denoisingstep 312, 322 to solve Eq. 9. Notice that the weights of such CNN may be shared among two or moreADMM iterations which are unrolled to form the overall network, the weights pf the CNN as well as allother trainable parameter like ρ and β are learned in a supervised end-to-end manner in accordance withembodiments of the present disclosure. In some embodiments, the weights of the CNN may be sharedamong all ADMM iterations.Solving sub-problem (B): According to embodiments of the present disclosure, any known imageregistration technique can be used to estimate spatial shifts ^ = {(^ ^ ^^, ^^), … , (^ ^ ^^ , ^^ )} between areference frame (which in embodiments is the hologram closest to the object plane, though any hologramin the stack may be a reference frame) and all the other frames. Notice that the registration step is notperformed on the raw input holograms because diffraction patterns are different due to field propagation EU-IPD: SYP354473WO01 DYC Ref: P129332PCTsince the distance between the object and detector planes changes with each capture, the registration iscarried out on the phase and amplitude distributions of the back-propagated stack of N holograms instead.Referring back to Figure 2, the implementation of the initialization block 220 and the complex encoder 230, and complex decoder 250 is further described below. Initialization: According to embodiments of the present disclosure, initialization is done by first up-sampling all input holograms by the factor s then back-propagated to the object plane using z = [z1, ..., zN]in order to get multiple complex images of the same object but shifted with respect to each other fromwhich complex feature maps are extracted and aligning using the estimated spatial shifts, aligned compleximages are then averaged in order to get x0, u0 and v0 are initialized to zero.Complex encoder: According to embodiments of the present disclosure, initial estimates (x0 ∈ ℂ) are fedinto a shallow complex CNN with a residual connection between the input and output distribution: acomplex CNN has multiple convolution layers with complex kernels in ℂ thus the learned weights arecomplex in nature. Further details of the shallow complex CNN is described in [3].Complex decoder: According to embodiments of the present disclosure, the decoder 250 has the samearchitecture as the encoder 230 and it has also complex convolution layers. By operating in the complexdomain, close correlations between the real and imaginary parts of the field can be effectively learnedwithout the need to decouple such distributions and treating them as two different channels.Figure 4 illustrates a flow chart 400 describing a process in the quantitative phase imaging device 100 forreconstructing complex transmission distribution according to embodiments of the disclosure. Process400 may begin at step 405, where a stack of holograms is received, for example from a plurality ofdetector planes. At step 410, initialization is performed to obtain initial estimates of latent optimizationvariables. According to some embodiments of the disclosure, the initialization may include up-samplinginput the stack of holograms; back-propagating the up-sampled holograms to the object plane to obtaincomplex images of an object; estimating spatial shifts between the complex images; and aligning compleximages using the estimated spatial shifts. At step 415, shallow complex residual encoding is performed to convert the initial estimates to complexdomain. At step, 420, latent optimization variables of a latent complex transmission distribution isestimated by performing iterations in the unrolled network 240. For example, the latent optimizationvariables may include latent field and spatial shift with respect to the stack of holograms captured by thedetector planes at different heights. In some embodiments, the latent optimization variables may beestimated by alternating between the latent optimization variables, so as to iteratively optimize a latentoptimization variable at a time while keeping the other latent optimization variable fixed. In someembodiments, the latent field may be estimated by variable splitting, and more particularly, byAlternating Direction Method of Multipliers, ADMM, iterations. According to some embodiments of the EU-IPD: SYP354473WO01 DYC Ref: P129332PCT present disclosure, each of the ADMM iterations may include performing gradient descent; applying adenoiser to estimate a clean complex field; and updating a scaled Lagrange multiplier. In some examples,the denoiser may be a trained convolutional neural network, CNN. Furthermore, the weights of the CNNmay be shared among two or more ADMM iterations and may be learned in a supervised end-to-endmanner. In some embodiments, the weights of the CNN may be shared among all ADMM iterations.According to embodiments of the present disclosure, spatial shifts may be estimated between a reference frame of the hologram stack and other frames of the hologram stack by performing image registration onthe phase and amplitude distributions of the back-propagated stack of the holograms.The process then moves to step 425, where shallow complex residual decoding is performed to convertthe field estimate obtained by the unrolled network from complex domain to spatial domain. Finally, atstep 430, the latent complex transmission distribution is reconstructed from the field estimate. Throughthe implementation of the above process in a quantitative phase imaging device according toembodiments of the present disclosure, spatial super-resolution resolution capability can be achieved witha more efficient and less complex model. Numerous modifications and variations of the present disclosure are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the disclosure may be practiced otherwise than as specifically described herein. In so far as embodiments of the disclosure have been described as being implemented, at least in part, by software-controlled data processing apparatus, it will be appreciated that a non-transitory machine- readable medium carrying such software, such as an optical disk, a magnetic disk, semiconductor memory or the like, is also considered to represent embodiments of the present disclosure. It will be appreciated that the above description for clarity has described embodiments with reference to different functional units, circuitry and / or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, circuitry and / or processors may be used without detracting from the embodiments. Described embodiments may be implemented in any suitable form including hardware, software, firmware or any combination of these. Described embodiments may optionally be implemented at least partly as computer software running on one or more data processors and / or digital signal processors. The elements and components of any embodiment may be physically, functionally and logically implemented in any suitable way. Indeed the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the disclosed embodiments may be implemented in a single unit or may be physically and functionally distributed between different units, circuitry and / or processors.The different units, circuitry and / or processors may be physically distributed across any distance. The EU-IPD: SYP354473WO01 DYC Ref: P129332PCT disclosed embodiments may also be implemented based on processing and storage units located in a remote system, such as in the Cloud. Although the present disclosure has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. Additionally, although a feature may appear to be described in connection with particular embodiments, one skilled in the art would recognize that various features of the described embodiments may be combined in any manner suitable to implement the technique. Embodiments of the present technique can generally be described by the following numbered clauses:1. A quantitative phase imaging device comprising processing circuitry configured to:receive a stack of holograms; perform initialization to obtain initial estimates of latent optimization variables; perform shallow complex residual encoding; estimate latent optimization variables of a latent complex transmission distribution by performingiterations in an unrolled network, perform shallow complex residual decoding; and reconstruct the latent complex transmission distribution.2. The quantitative phase imaging device according to clause 1, wherein the latent optimizationvariables include latent field and spatial shift with respect to the stack of holograms captured at differentheights; and wherein the processing circuitry is configured to estimate the latent optimization variables byalternating between the latent optimization variables, so as to iteratively optimize a latent optimization variable at a time while keeping the other latent optimization variable fixed.3. The quantitative phase imaging device according to clause 2, wherein the estimated latent fieldwith respect to the stack of holograms captured at different heights is expressed as ^= arg where is a forward in-line holographic image formation model for simulating holograms, i= 1, …, N,N is number of heights, His the stack of holograms captured at each height zi,Ds is a down-sampling matrix reducing size of an image by a factor s, ^^^is a matrix simulating spatial shifts, EU-IPD: SYP354473WO01 DYC Ref: P129332PCT ^^^is complex near-field Fresnel propagation kernel, βis a discrepancy parameter,Ψ(x) is a regularizer.4. The quantitative phase imaging device according to any one of clauses 2 to 3, wherein theestimated spatial shift with respect to the stack of holograms captured at different heights is expressed as in-line holographic image formation model for simulating holograms, N is number of heights, His the stack of holograms captured at each height zi,Ds is a down-sampling matrix reducing size of an image by a factor s, ^^^is a matrix simulating spatial shifts, ^^^is complex near-field Fresnel propagation kernel.5. The quantitative phase imaging device according to clause 2, wherein the processing circuitry isconfigured to estimate the latent field by variable splitting.6. The quantitative phase imaging device according to clause 2, wherein the processing circuitry isconfigured to estimate the latent field by Alternating Direction Method of Multipliers, ADMM, iterations.7. The quantitative phase imaging device according to clause 6, wherein the processing circuitry isconfigured such that each of the ADMM iterations comprises: performing gradient descent; applying a denoiser to estimate a clean complex field; and updating a scaled Lagrange multiplier.8. The quantitative phase imaging device according to clause 7, wherein the processing circuitry isconfigured such that the denoiser is a trained convolutional neural network, CNN, and the weights of the CNN are shared among two or more of the ADMM iterations. 9. The quantitative phase imaging device according to clause 8, wherein the weights of the CNN are learned in a supervised end-to-end manner. EU-IPD: SYP354473WO01 DYC Ref: P129332PCT10. The quantitative phase imaging device according to clause 6, wherein the processing circuitry isconfigured to estimate the latent field and an auxiliary variable by the expression: (^, ^) = arg where vis the auxiliary variable.11. The quantitative phase imaging device according to clause 10, wherein the processing circuitry isconfigured to estimate the latent field and the auxiliary variable by the expressions: ^← arg minℒ^^^^^ (^, ^, ^) = arg min1 ^^ − ^^ ^ ^,^,^(^)^^ ^ +‖^ + ^ − ^‖^^ ^ 2 2 ^← arg minℒ^^^^^ (^, ^, ^) = arg min^ ‖‖ ^ ( )2^ + ^ − ^ ^ + ^^ ^^ ^ ^← ^ + ^ − ^and where uis a scaled Lagrange multiplier,ρ is a penalty term.12. The quantitative phase imaging device according to any one of clauses 10 to 11, wherein theprocessing circuitry is configured to update the latent field in each ADMM iteration estimate by theexpressions: where α is a learning rate, nis a ADMM iteration.13. The quantitative phase imaging device according to clause 1, wherein the processing circuitry isconfigured such that the initialization is performed by: up-sampling the stack of holograms; back-propagating the up-sampled holograms to the object plane to obtain complex images of an object; estimating spatial shifts between the complex images; and EU-IPD: SYP354473WO01 DYC Ref: P129332PCT aligning complex images using the estimated spatial shifts.14. The quantitative phase imaging device according to clause 1, wherein the processing circuitry isconfigured to perform complex encoding and complex decoding based on the same architecture comprising multiple complex convolution layers.15. The quantitative phase imaging device according to clause 1, wherein the processing circuitry isconfigured to: back-propagate the stack of holograms; and estimate spatial shifts between a reference frame of the hologram stack and other frames of the hologram stack by performing image registration on the phase and amplitude distributions of the back- propagated stack of holograms. 16. A method performed in a quantitative phase imaging device, the method comprising: receiving a stack of holograms; performing initialization to obtain initial estimates of latent optimization variables; performing shallow complex residual encoding; estimating latent optimization variables of a latent complex transmission distribution by performing iterations in an unrolled network, performing shallow complex residual decoding; and reconstructing the latent complex transmission distribution.17. The method according to clause 16, wherein the latent optimization variables include latent fieldand spatial shift with respect to the stack of holograms captured at different heights; and wherein the method further comprising estimating the latent optimization variables by alternating between the latentoptimization variables, so as to iteratively optimize a latent optimization variable at a time while keepingthe other latent optimization variable fixed.18. The method according to clause 17, wherein the estimated latent field with respect to the stack ofholograms captured at different heights is expressed as ^= arg where is a forward in-line holographic image formation model for simulating holograms, i= 1, …, N,N is number of heights, EU-IPD: SYP354473WO01 DYC Ref: P129332PCT His the stack of holograms captured at each height zi,Ds is a down-sampling matrix reducing size of an image by a factor s, ^^^is a matrix simulating spatial shifts, ^^^is complex near-field Fresnel propagation kernel, βis a discrepancy parameter,Ψ(x) is a regularizer.19. The method according to any one of clauses 17 to 18, wherein the estimated spatial shift withrespect to the stack of holograms captured at different heights is expressed as in-line holographic image formation model for simulating holograms, N is number of heights, His the stack of holograms captured at each height zi,Ds is a down-sampling matrix reducing size of an image by a factor s, ^^^is a matrix simulating spatial shifts, ^^^is complex near-field Fresnel propagation kernel.20. The method according to clause 17, further comprising estimating the latent field by variablesplitting.21. The method according to clause 17, further comprising estimating the latent field by AlternatingDirection Method of Multipliers, ADMM, iterations.22. The method according to clause 21, wherein each of the ADMM iterations comprises:performing gradient descent; applying a denoiser to estimate a clean complex field; and updating a scaled Lagrange multiplier.23. The method according to clause 22, wherein the denoiser is a trained convolutional neuralnetwork, CNN, and the weights of the CNN are shared among two or more of the ADMM iterations. 24. The method according to clause 23, wherein the weights of the CNN are learned in a supervised end-to-end manner. EU-IPD: SYP354473WO01 DYC Ref: P129332PCT 25. The method according to clause 21, further comprising estimating the latent field and an auxiliary variable by the expression: (^, ^) = arg where vis the auxiliary variable.26. The method according to clause 25, further comprising estimating the latent field and theauxiliary variable by the expressions: ^← arg minℒ^^^^^ (^, ^, ^) = arg min1 ^^ − ^^,^,^(^)^^ ^ +‖2^ ^ + ^ − ^‖^ ^ ^ ^ 2 ^← arg minℒ^^^^^ (^, ^, ^) = arg min^ ‖‖ ^ ( )2^ + ^ − ^ ^ + ^^ ^^ ^ ^← ^ + ^ − ^and where uis a scaled Lagrange multiplier,ρ is a penalty term.27. The method according to any one of clauses 25 to 26, further comprising updating the latent fieldin each ADMM iteration estimate by the expressions: where α is a learning rate, nis a ADMM iteration.28. The method according to clause 16, wherein the initialization is performed by:up-sampling the stack of holograms; back-propagating the up-sampled holograms to the object plane to obtain complex images of an object; estimating spatial shifts between the complex images; andaligning complex images using the estimated spatial shifts. EU-IPD: SYP354473WO01 DYC Ref: P129332PCT29. The method according to clause 16, further comprising: performing complex encoding andcomplex decoding based on the same architecture comprising multiple complex convolution layers.30. The method according to clause 16, further comprising:back-propagating the stack of holograms; and estimating spatial shifts between a reference frame of the hologram stack and other frames of the hologram stack by performing image registration on the phase and amplitude distributions of the back- propagated stack of holograms.31. A computer program product comprising computer readable instructions which, when loadedonto a computer, configures the computer to perform a method according to clauses 16 to 31.References1. Foi, A., Trimeche, M., Katkovnik, V., Egiazarian, K.: Practical poissonian-gaussian noise modelingand fitting for single-image raw-data. IEEE transactions on image processing 17(10), 1737–1754 (2008)2. Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finiteelement approximation. Computers & mathematics with applications 2(1), 17–40 (1976)3. Guberman, N.: On complex valued convolutional neural networks. arXiv preprint arXiv:1602.09046(2016)4. Mshimba, A.S.A., Tutschke, W.: Functional analytic methods in complex analysis and applications topartial differential equations. World Scientific (1995)5. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D:nonlinear phenomena 60(1-4), 259–268 (1992)6. Tamamitsu, M., Zhang, Y., Wang, H., Wu, Y., Ozcan, A.: Comparison of gini index and tamuracoefficient for holographic autofocusing based on the edge sparsity of the complex optical wavefront.arXiv preprint arXiv:1708.08055 (2017)

Claims

EU-IPD: SYP354473WO01 DYC Ref: P129332PCT CLAIMS1. A quantitative phase imaging device comprising processing circuitry configured to:receive a stack of holograms;perform initialization to obtain initial estimates of latent optimization variables;perform shallow complex residual encoding; estimate latent optimization variables of a latent complex transmission distribution by performing iterations in an unrolled network, perform shallow complex residual decoding; and reconstruct the latent complex transmission distribution.

2. The quantitative phase imaging device according to claim 1, wherein the latent optimizationvariables include latent field and spatial shift with respect to the stack of holograms captured at differentheights; and wherein the processing circuitry is configured to estimate the latent optimization variables byalternating between the latent optimization variables, so as to iteratively optimize a latent optimization variable at a time while keeping the other latent optimization variable fixed.

3. The quantitative phase imaging device according to claim 2, wherein the estimated latent fieldwith respect to the stack of holograms captured at different heights is expressed as ^= argwhereis a forward in-line holographic image formation model for simulating holograms, i= 1, …, N,N is number of heights, His the stack of holograms captured at each height zi,Ds is a down-sampling matrix reducing size of an image by a factor s, ^^^is a matrix simulating spatial shifts, ^^^is complex near-field Fresnel propagation kernel, βis a discrepancy parameter,Ψ(x) is a regularizer.

4. The quantitative phase imaging device according to claim 2, wherein the estimated spatial shiftwith respect to the stack of holograms captured at different heights is expressed as ^̃ = arg min1 ^ 2EU-IPD: SYP354473WO01 DYC Ref: P129332PCT where is a forward in-line holographic image formation model for simulatingN is number of heights, His the stack of holograms captured at each height zi,Dsis a down-sampling matrix reducing size of an image by a factor s, ^^^is a matrix simulating spatial shifts, ^^^is complex near-field Fresnel propagation kernel.

5. The quantitative phase imaging device according to claim 2, wherein the processing circuitry isconfigured to estimate the latent field by variable splitting.

6. The quantitative phase imaging device according to claim 2, wherein the processing circuitry isconfigured to estimate the latent field by Alternating Direction Method of Multipliers, ADMM, iterations.

7. The quantitative phase imaging device according to claim 6, wherein the processing circuitry isconfigured such that each of the ADMM iterations comprises: performing gradient descent; applying a denoiser to estimate a clean complex field; and updating a scaled Lagrange multiplier.

8. The quantitative phase imaging device according to claim 7, wherein the processing circuitry isconfigured such that the denoiser is a trained convolutional neural network, CNN, and the weights of theCNN are shared among two or more of the ADMM iterations.

9. The quantitative phase imaging device according to claim 8, wherein the weights of the CNN are learned in a supervised end-to-end manner.

10. The quantitative phase imaging device according to claim 6, wherein the processing circuitry isconfigured to estimate the latent field and an auxiliary variable by the expression: (^, ^) = argwhere vis the auxiliary variable.EU-IPD: SYP354473WO01 DYC Ref: P129332PCT11. The quantitative phase imaging device according to claim 10, wherein the processing circuitry isconfigured to estimate the latent field and the auxiliary variable by the expressions:^ ← arg minℒ^^^^^ (^, ^, ^) = arg min1 ^^ − ^^,^,^(^)^^ ^ ^ +‖^ + ^ − ^‖^ ^ ^ ^ 2 2 ^← arg minℒ^^^^^ (^, ^, ^) = arg min^ ‖‖ ^ ( )2^ + ^ − ^ ^ + ^^ ^^ ^ ^← ^ + ^ − ^andwhere uis a scaled Lagrange multiplier,ρ is a penalty term.

12. The quantitative phase imaging device according to claim 11, wherein the processing circuitry isconfigured to update the latent field in each ADMM iteration estimate by the expressions:where α is a learning rate, nis a ADMM iteration.

13. The quantitative phase imaging device according to claim 1, wherein the processing circuitry isconfigured such that the initialization is performed by: up-sampling the stack of holograms; back-propagating the up-sampled holograms to the object plane to obtain complex images of an object; estimating spatial shifts between the complex images; andaligning complex images using the estimated spatial shifts.

14. The quantitative phase imaging device according to claim 1, wherein the processing circuitry isconfigured to perform complex encoding and complex decoding based on the same architecture comprising multiple complex convolution layers.

15. The quantitative phase imaging device according to claim 1, wherein the processing circuitry isconfigured to:EU-IPD: SYP354473WO01 DYC Ref: P129332PCT back-propagate the stack of holograms; and estimate spatial shifts between a reference frame of the hologram stack and other frames of thehologram stack by performing image registration on the phase and amplitude distributions of the back-propagated stack of holograms.

16. A method performed in a quantitative phase imaging device, the method comprising: receiving a stack of holograms; performing initialization to obtain initial estimates of latent optimization variables; performing shallow complex residual encoding; estimating latent optimization variables of a latent complex transmission distribution by performing iterations in an unrolled network, performing shallow complex residual decoding; and reconstructing the latent complex transmission distribution.

17. The method according to claim 16, wherein the latent optimization variables include latent fieldand spatial shift with respect to the stack of holograms captured at different heights; and wherein the method further comprising estimating the latent optimization variables by alternating between the latent optimization variables, so as to iteratively optimize a latent optimization variable at a time while keeping the other latent optimization variable fixed.

18. The method according to claim 17, wherein the estimated latent field with respect to the stack ofholograms captured at different heights is expressed as ^= argwhereis a forward in-line holographic image formation model for simulatingholograms, i= 1, …, N,N is number of heights, His the stack of holograms captured at each height zi,Ds is a down-sampling matrix reducing size of an image by a factor s, ^^^is a matrix simulating spatial shifts, ^^^is complex near-field Fresnel propagation kernel, βis a discrepancy parameter,Ψ(x) is a regularizer.EU-IPD: SYP354473WO01 DYC Ref: P129332PCT19. The method according to claim 17, wherein the estimated spatial shift with respect to the stack ofholograms captured at different heights is expressed asin-line holographic image formation model for simulating holograms, N is number of heights, His the stack of holograms captured at each height zi,Dsis a down-sampling matrix reducing size of an image by a factor s, ^^^is a matrix simulating spatial shifts, ^^^is complex near-field Fresnel propagation kernel.

20. The method according to claim 17, further comprising estimating the latent field by variablesplitting.

21. The method according to claim 17, further comprising estimating the latent field by AlternatingDirection Method of Multipliers, ADMM, iterations.

22. The method according to claim 21, wherein each of the ADMM iterations comprises:performing gradient descent; applying a denoiser to estimate a clean complex field; and updating a scaled Lagrange multiplier.

23. The method according to claim 22, wherein the denoiser is a trained convolutional neuralnetwork, CNN, and the weights of the CNN are shared among two or more of the ADMM iterations.

24. The method according to claim 23, wherein the weights of the CNN are learned in a supervised end-to-end manner.

25. The method according to claim 21, further comprising estimating the latent field and an auxiliaryvariable by the expression: (^, ^) = argwhere vis the auxiliary variable.EU-IPD: SYP354473WO01 DYC Ref: P129332PCT26. The method according to claim 25, further comprising estimating the latent field and the auxiliaryvariable by the expressions: ^← arg minℒ^^^^^ (^, ^, ^) = arg min1 ^^ − ^^,^,^(^)^^ ^ ^ +‖^ + ^ − ^‖^ ^ ^ ^ 2 2 ^← arg minℒ^^^^^ (^, ^, ^) = arg min^ ‖^ + ^ − ^‖ ^^ + ^^(^)^ ^ 2 ^← ^ + ^ − ^andwhere uis a scaled Lagrange multiplier,ρ is a penalty term.

27. The method according to claim 26, further comprising updating the latent field in each ADMMiteration estimate by the expressions:where α is a learning rate, nis a ADMM iteration.

28. The method according to claim 16, wherein the initialization is performed by:up-sampling the stack of holograms; back-propagating the up-sampled holograms to the object plane to obtain complex images of an object; estimating spatial shifts between the complex images; andaligning complex images using the estimated spatial shifts.

29. The method according to claim 16, further comprising: performing complex encoding and complex decoding based on the same architecture comprising multiple complex convolution layers.

30. The method according to claim 16, further comprising: back-propagating the stack of holograms; andEU-IPD: SYP354473WO01 DYC Ref: P129332PCT estimating spatial shifts between a reference frame of the hologram stack and other frames of the hologram stack by performing image registration on the phase and amplitude distributions of the back- propagated stack of holograms.

31. A computer program product comprising computer readable instructions which, when loadedonto a computer, configures the computer to perform a method according to claim 16.

Images