Method and apparatus for lensless digital holographic microscopy with sub-pixel resolution using a single hologram

Abstract

Method for lensless digital holographic microscopy with sub-pixel resolution using a single hologram, comprising digital interpolation of a complex-valued hologram of an object in an object plane, recorded using an approximately spherical reference wave with an origin by means of an area detector, in particular a camera sensor, in its detector plane, onto a fine scanning grid with sub-pixel resolution, as well as numerical compensation of the reference wave, and exact reconstruction of the interpolated hologram, and apparatus for carrying out the same.

Description

[0001] The present invention relates to a method and a device for lensless digital holographic microscopy with sub-pixel resolution using a single hologram.

[0002] Digital holography, utilizing the principles of interference and diffraction, enables the recording and reconstruction of three-dimensional images by capturing and storing both the amplitude and phase information of light waves scattered by objects. Digital holograms are recorded using an optical holographic setup, while their reconstruction is performed entirely digitally on a computer. This approach eliminates the need for complex optical components and, through volumetric imaging, allows for lensless microscopy. Lensless microscopy finds applications in various fields, such as metrology and quality assurance, biomedical imaging, and environmental monitoring, and provides a cost-effective and portable alternative for achieving depth-resolution 3D microscopic imaging.

[0003] However, it is widely assumed that the spatial resolution of lensless holographic microscopes is limited by the pixel pitch of the camera sensor. This limitation is often laboriously circumvented using pixel super-resolution techniques to synthesize a larger and higher-resolution hologram. This is typically achieved by recording multiple holograms and scanning them by shifting the camera sensor, the object, or the illumination, or by placing a diffraction grating between the sample and the camera.

[0004] Another alternative for reconstructing lensless digital holograms is the use of neural networks. However, the reconstruction using this method is ambiguous and can be highly error-prone due to overfitting.

[0005] The present invention is based on the objective of improving the optical resolution of lensless holographic microscopes.

[0006] According to the invention, this problem is solved according to a first aspect by a method for lensless digital holographic microscopy with sub-pixel resolution using a single hologram, comprising - digital interpolation of a complex-valued hologram of an object in an object plane, recorded in the detector plane of an area detector, in particular a camera sensor, using an approximately spherical reference wave with an origin, onto a fine scanning grid with sub-pixel resolution, - numerical compensation of the reference wave, and - Exact reconstruction of the interpolated hologram. The detector plane is identical to the hologram plane. The scanning grid corresponds to the pixel grid of the area detector, such as a camera sensor. Typical pixel pitches range from approximately 1.5 µm to several µm. The minimum resolution of the sub-pixel grid is determined by the numerical aperture of the lensless microscopy setup; typically, minimum resolutions of up to approximately 0.5 µm can be achieved.

[0007] Sub-pixel resolution means that the interpolated hologram contains pixels with spacing smaller than the pixels of the area detector, such as the camera sensor used to measure the hologram.

[0008] According to another aspect, this task is solved by a device for lensless digital holographic microscopy with sub-pixel resolution using a single hologram, comprising - a lensless digital holographic microscope with Fourier holography setup, comprising an illumination device for generating object and reference illumination and an area detector with one detector plane, and - a computing device connected to the area detector and configured to perform the method described in claims 1 to 5.

[0009] According to a particular embodiment, the method further comprises: recording the complex-valued digital hologram of the object in the object plane by means of Fourier holography using the approximately spherical reference wave in the detector plane by means of the area detector.

[0010] Advantageously, the origin of the reference wave and the object are equidistant from the area detector.

[0011] According to a particular embodiment of the present invention, digital interpolation comprises: - Fresnel reconstruction of the object to near the object plane, possibly taking the reference wave into account, - Applying zero-padding by a factor of ρ around the reconstruction, and - Inverse Fresnel reconstruction into the detector plane. ρ is also the factor by which the pixel pitch of the interpolated hologram is smaller than the pixel pitch of the area detector, such as a camera sensor. One could therefore also describe ρ as the scaling factor of the pixel pitch.

[0012] Advantageously, the exact reconstruction of the interpolated hologram includes: - Exact reconstruction of the interpolated hologram by applying the Angular Spectrum method.

[0013] The reconstruction is an exact solution of the Rayleigh-Sommerfeld integral, which results directly from Maxwell's equations. However, there will always be apparent degradation in the reconstructed wave field, which is a consequence of scanning the hologram at discrete pixels. Furthermore, measurement errors that occurred during the initial recording of the hologram cannot, of course, be compensated for by the reconstruction.

[0014] The device may be designed so that the microscope is set up for a transmitted light mode.

[0015] In particular, the microscope may include a beam splitter.

[0016] According to a particular embodiment, the microscope includes a Mach-Zehnder interferometer.

[0017] Alternatively, the microscope can be set up for reflected light mode.

[0018] In particular, it may be provided that the microscope includes a beam splitter.

[0019] According to a particular embodiment, the microscope includes a Michelsen interferometer.

[0020] Ideally, the lighting system includes at least one point light source.

[0021] Finally, it may be provided in particular that the lighting device comprises at least one, preferably fiber-coupled, laser light source.

[0022] The present invention is based on the surprising finding that by using an approximately spherical reference wave in the recording of a digital hologram and a special digital reconstruction method, an exact reconstruction of the hologram is possible.

[0023] The concept of lensless microscopy with sub-pixel resolution from single image recordings (single hologram or single hologram) enables a precise determination of a propagated wave field over a fine scanning grid with sub-micrometer spacing, starting from a coarsely scanned wave field in the camera plane, which has a pixel spacing of several micrometers.

[0024] The invention essentially consists of a digital reconstruction method for reconstructing a high-resolution object wave field from a low-resolution complex-valued digital hologram. To apply the reconstruction method to the digital hologram, a specially adapted holographic setup is required for its recording. For accurate scanning of the hologram, the setup is based on Fourier holography principles, for example, as a Mach-Zehnder interferometer or Michelson interferometer. While the setup based on the principle of lensless Fourier holography represents the current state of the art, it must be adapted for the application of the novel reconstruction method.

[0025] In a particular embodiment, the digital reconstruction method can begin with the propagation of the wave field from the detector plane (area detector plane), especially the camera plane, to the vicinity of the object plane using Fresnel propagation. Generally, propagation describes the numerical spread of a wave field into another plane. One could also say that the wave field is reconstructed in this new plane by propagation. Fresnel propagation inherently provides reconstruction over a grid with spacings determined by the diffraction-limited resolution. However, the image quality in microscopy (NA>0.1) is generally unsatisfactory. Nevertheless, despite the blurring, the energy is localized in the region of the object.Therefore, in a particular embodiment, we can use this imprecise reconstruction and apply zero-padding to it so that subpixel interpolation to a fine scanning grid in the camera plane occurs during Fresnel backpropagation. In a further embodiment, we can then apply the Angular Spectrum method to the interpolated signal to compute an exact representation of the wave field over the fine (interpolated) grid in the object plane. The method works with a single hologram and can therefore also be used to record transient events.

[0026] Further features and advantages of the invention will become apparent from the attached claims and from the following description of advantageous embodiments with reference to the schematic drawings. These show: Fig. 1: a typical scanning scheme in lensless microscopy in the state of the art; Fig. 2a) and b): Flowcharts a) of a particular embodiment of a method for lensless digital holographic microscopy with sub-pixel resolution using a single hologram (diffraction-limited reconstruction process) and b) of a particular embodiment of the ‘interpolation step’ shown in a); Fig. 3a) to f): an illustrated flowchart of a method (diffraction-limited reconstruction process) for lensless digital holographic microscopy with sub-pixel resolution using a single hologram according to a particular embodiment of the present invention, using the example of an object wave field consisting of four Siemens stars; Fig. 4a) to c): schematic components of devices for lensless digital holographic microscopy with sub-pixel resolution with a single hologram according to particular embodiments of the present invention; and Fig. 5a) to d): schematic components of devices for lensless digital holographic microscopy with sub-pixel resolution with a single hologram according to particular embodiments of the present invention.

[0027] Let us consider a typical scanning scenario for lensless microscopy, as described in Fig. Figure 1 shows an object plane OE on the left and, parallel to it at a distance z, a sensor plane or detector plane DE of a sensor or area detector. The area detector has a pixel pitch of Δp, which is typically several times larger than the optical resolution (distance) d of a system (object structures), such that Δp = ρ1 · d with ρ1 > 1. ρ1 is a factor by which the pixel pitch of the area detector (e.g., camera sensor) is larger than the diffraction-limited resolution. For example, ρ1 could be chosen such that ρ = ρ1. However, other values ​​are also conceivable, for example, to reduce the amount of data in the interpolated hologram.

[0028] In the example shown, the factor between the grids is ρ = 4. The space-bandwidth product of the system, i.e., the number of pixels N, is invariant. Consequently, the edge length L is S the detector plane is larger by a factor of ρ1 than the edge length L Othe object level.

[0029] If we denote the number of pixels along an edge by N, we can define the edge length of the detector plane as L. S = N · Δp and the edge length of the object plane divided by L O = N · d. Therefore, the detector plane is much larger than the object plane, while the resolution of the object plane (object structures) is much higher than that of the detector plane. We can now relate all parameters if we consider the system as diffraction-limited. In this case, the optical resolution d is defined by the Abbe diffraction limit: d=λ2⋅NA≈λ⋅zLS, with the numerical aperture NA = n · sin (α) and the wavelength λ. In the last step of equation 1, we have the paraxial approximation sin(α) ≈ tan(α) = L S / (2z) assumed.

[0030] When light reflected or scattered by an object in the object plane is superimposed with a reference wave R(x, y), the sensor will record an interference pattern. I(x,y)=|UH|2+|R|2+UH⋅R*+UH*⋅R, where U H (x,y) denotes the complex-valued hologram, i.e., the object wave across the sensor or detector grating, and we have partially omitted the function arguments for better readability. We can use any phase-shift method to determine the term M(x, y) = U H To extract (x,y) · R* (x, y), from which we can obtain the complex-valued hologram if the structure of R is known, the Fresnel propagation formula is applicable. It is based on the Fresnel approximations and allows the propagation of an object wave field U. O (u,v), e.g. into the hologram plane, so that UH(x,y)=iλzS1(x,y)⋅F{U0(u,v)⋅S2(u,v)}(xλz,yλz), where F denotes the Fourier transform operator and S1 and S2 are two exponential functions with quadratic phase S1(x,y)=exp[ik(x2+y2)2z+ikz] and S2(u,v)=exp[ik(u2+v2)2z]

[0031] The propagation formula above has significant advantages over diffraction-limited, lensless imaging. Its inverse is the Fresnel propagation formula for reconstructing an object wave field from a complex-valued hologram: U0(u,v)=iλz⋅S2*(u,v)⋅F−1{UH(x,y)⋅S1*(x,y)}(uλz,vλz).

[0032] This propagation formula transforms a complex-valued hologram scanned over the sensor or detector grid into an object wave field over another scanning grid, whose pitch Δq = (λ Z ) -1This coincides with the diffraction limit d, as can be seen from the structure of the Fourier transform in Equation 5. Therefore, the Fresnel propagation formula (Eq. 5) inherently yields diffraction-limited imaging. Unfortunately, however, it only provides acceptable results for numerical apertures NA < 0.1, making it a poor choice for microscopy applications.

[0033] A precise reconstruction beyond this limit and without further prior knowledge is not a simple task. The main problem is the high-frequency chirp S1(x,y) in Equation 3 (and Equation 5 accordingly), which is significantly undersampled in the detector plane and therefore hinders the application of precise methods such as wave propagation by plane wave decomposition (or the angular spectrum method). To address this, we can write the gradient of the phase ϕ(x,y) = arg {S1}, e.g., in the x-direction, as ∇xϕ(x,y)=k⋅x / z.

[0034] If we x = L S Substituting / 2 to determine the highest phase gradient near the edges of the detector, we obtain ∇xϕ(x,y)|x=LS / 2=2πLS2λz=2π12d=vN, In the last step, we used the Abbe diffraction limiting equation (1). As a result, we obtain exactly the Nyquist frequency v. N of the densely sampled grating in the object plane. Therefore, S1 is undersampled in the detector plane by a factor of ρ. This has two essential consequences: First, we can

[0035] We cannot experimentally scan the wave field itself with a camera and, for example, a plane reference wave, because the interference fringes would be too dense and the finite pixels would average out the high-frequency parts of the interference pattern. Secondly, if we do not know the entire wave field above the finely scanned grating, we cannot reconstruct it precisely.

[0036] The solution to the first problem has been widely described in the literature under the term lensless Fourier holography. The basic idea is to use a spherical reference wave R(x, y) that appears to originate from the center of the object. This can be achieved, for example, with a beam splitter / combiner. Since S1 can be considered a parabolic approximation of a point source at the center of the object, we can assume R ≈ S1 for small numerical apertures, and the measured cross term becomes M=UH⋅R*≈UH⋅S1*.

[0037] In this situation, the reconstruction task reduces to a simple Fourier transformation of M(x, y), as can be seen from equation 5, hence the name of the technique.

[0038] The origin of the reference wave can also be shifted laterally to achieve off-axis holography by modulating the interference pattern with a spatial carrier frequency. In this case, no additional phase shift is required (single-frame operation) to separate the real and conjugate images. Furthermore, the source of the reference wave, e.g., a fiber optic cable, can be placed next to the object, thus eliminating the need for a beam splitter.

[0039] A spherical reference wave works well to prevent averaging during recording, but the assumption that the reconstruction process can be carried out by a simple Fourier transform is still dependent on the validity of Equation 5 and is therefore hardly more precise than this Fresnel propagation formula.

[0040] The core of the invention lies in solving the second problem mentioned above concerning lensless microscopy. According to the current state of the art, the digital reconstruction of a digital hologram is only possible with the resolution of the pixel spacing of the digital hologram. Therefore, for high-resolution object wave fields, the measured hologram must currently have an equally high resolution.

[0041] The solution to this reconstruction problem can be described as a two-stage process. First, the complex-valued digital hologram is interpolated onto a fine scanning grid (with subpixel resolution) and then reconstructed as an object wave field by precise propagation into the object plane.

[0042] Similar to the lensless Fourier holography scheme, we assume that an approximately spherical reference wave with a known structure is used during the acquisition. In a preferred embodiment of the present invention, Equation 5 and the measured cross term M(x, y) are used in a first step to interpolate the hologram and obtain an approximate reconstruction of the object via Fresnel propagation. If the origin of the reference spherical wave and the object are equidistant from a planar detector, e.g., a camera sensor, this propagation becomes a pure Fourier transform. At a high numerical aperture, the reconstruction quality will be poor in the sense that we do not achieve complete diffraction-limited resolution. However, by choosing a measurement setup similar to Fourier holography, the energy will still be concentrated close to the object support.We can then apply zero-padding by a factor of ρ around this "poor" reconstruction and transform it back into the sensor plane. The result will be an interpolated representation of M(x, y) over a fine scanning grid that has the size of the detector but the pitch of the object plane.

[0043] In a preferred embodiment, in a second step, the finely sampled M can be modulated with the high-frequency, but known, analytical structure of the reference wave R for numerical compensation of the spherical reference wave. We obtain M · R = U H · R* · R = U H (assuming that the amplitude of the reference wave does not vary too much, so that we can set |R| ≈ 1), which completely reconstructs the complex-valued wave field digitally over the finely sampled grid in the sensor plane.

[0044] In a preferred embodiment, we can now simply apply the Angular Spectrum method to obtain an accurate reconstruction of the complex-valued wave field in the object plane.

[0045] Fig. Figures 2a) and b) show flowcharts of the entire reconstruction process (including recording) according to a particular embodiment of the present invention ( Fig. 2a)) and according to a particularly preferred embodiment of the present invention, in which a particularly preferred implementation of the interpolation of the hologram onto a fine scanning grid is used ( Fig. 2b)). Additionally, show Fig. 3a) to f) a flowchart of the reconstruction process of Fig. 2b) using the example of an object wave field consisting of four Siemens stars 12.

[0046] Fig. Figure 3a) shows an example of a complex-valued digital hologram 10 recorded with a detector. This is digitally propagated in a computer in the first step by Fresnel propagation (in ideal Fourier holography by Fourier transform FT) into an object plane (see Figure 3a). Fig. 3b)).

[0047] In the next step, zero padding (ZP) is used to pad around the signal at the object level. Fig. 3c)). The inverse Fresnel propagation (or inverse Fourier transform FT) -1 In the next step, a high-resolution hologram is generated along a fine scanning grid (with subpixel resolution) in the detector plane, the phase of which (represented in Fig. 3 d)) is superimposed by an (approximately) spherical reference wave.

[0048] In the next step, the reference wave is numerically compensated ( Fig. 3 e)). The resulting hologram along the fine scanning grid can be propagated precisely into the object plane at high resolution using the Angular Spectrum method ASM, making the Siemens stars 12 shown as examples exactly resolvable.

[0049] A device for lensless digital holographic microscopy with sub-pixel resolution using a single hologram can be implemented with lensless microscopes in various configurations. All setups share the common feature that, following the principles of lensless Fourier holography, the sampling problem of lensless microscopy is solved through the specific selection of object and reference illumination. Thus, in every case, an (approximately) spherical wave is used as the reference wave, positioned at a similar distance to the area detector as the object. Furthermore, an implicit assumption of the reconstruction method (or Fresnel propagation) is that all recorded wave fields originate from the field of view 28 (see...). Fig. 4a) to c)). To validate this assumption, experimental measures must ensure that only the field of view is illuminated. Since microscopy typically has a field of view much smaller than the area detector, using plane-wave illumination restricted to the field of view would concentrate a large portion of the recorded energy on a small area of ​​the area detector. To avoid this and consequently improve the signal-to-noise ratio, we advantageously use divergent object illumination instead.

[0050] A distinction must first be made between transmitted light mode and reflected light mode. Fig. Figure 4a) schematically illustrates the realization of object and reference illumination in transmitted light mode according to special embodiments. Here, a point light source 14 and a reference point light source 16 are used for the object and reference waves, respectively, and superimposed on a surface detector 20 by means of a beam splitter 18. Fig. Figure 4b shows an example of an implementation of the illumination in reflected light mode, in which a reference point light source 16 is also used for the reference wave. The (approximately) spherical object illumination is achieved via an additional lens 22 in an illumination beam path 24. This creates a focal point FP in front of or behind an object 26.

[0051] Fig. Figure 4c) shows another implementation of object and reference illumination in incident light mode. The object illumination is achieved exactly as in Fig. 4b) is generated. A reference mirror 30 is used for the reference wave, which maps the focal point FP1 of the lens 22 at a similar distance from the area detector 20 as the object 26.

[0052] For complete scanning, the reference wave is in any case generated by a reference point light source 16, which has a similar distance from the area detector 20 as the object 26 (cf. Fourier holography).

[0053] The transmitted light mode can, as in Fig. 5a) and b) are shown by way of example, and in a preferred embodiment can be realized by a fiber-based Mach-Zehnder interferometer 32 with, for example, a fiber-coupled laser 50 and, for example, a fiber splitter 52. Divergent illumination can be achieved by using point light sources, for example, (single-mode) fibers 34 (see Fig. 5a)). However, conventional single-mode fibers only offer a numerical aperture of about 0.1 to 0.2, which does not allow the entire target to be illuminated. Instead, lenses 36 can also be used to achieve divergent illumination, as in the setup in Fig. 5b) shown as an example. It is worth noting that “lensless imaging” or “lensless microscopy” here refers to the absence of lenses between object 26 and area detector 20. The lenses 36 in Fig. 5b) are used only for beam guidance.

[0054] The incident light mode can, for example, be used as shown in Fig. 5c) can also be realized by a Mach-Zehnder interferometer 32 (this corresponds to the illumination situation from Fig. 4b)). The reference point light sources can be, for example, as in Fig. 5a) and b) can also be generated either via fiber ends 38 or by lenses 40. In this example, the object illumination is generated by an additional lens 42. Alternatively, the incident light mode can be generated, for example, by a Michelson interferometer 44, as shown in Fig. 5d) be implemented (accordingly Fig. 4c)), wherein a reference mirror 30 generates the reference wave.

[0055] To generate complex-valued digital holograms from the intensity distributions recorded by the area detector, any phase-shifting method can be used in each configuration. A preferred embodiment, particularly for single-image operation, is off-axis holography, which can be achieved by a lateral offset of the reference point light source or by tilting the reference mirror.

[0056] Fig. Figures 5a) and b) show the transmitted light mode as a Mach-Zehnder interferometer. Fig. 5a) The point light sources are realized by fiber ends; alternatively, the point light sources can be, for example, as in Fig. 5b) are produced by lenses to achieve a sufficiently high NA. Fig. Figures 5c) and 5d) show the incident light mode as a Mach-Zender interferometer ( Fig. 5c)), or as a Michelson interferometer ( Fig. 5d)).

[0057] In addition, depending on requirements, for example polarizers 46 and / or λ / 2 plates 48 may be provided.

[0058] The in the Fig. 4a)- d) and Fig. The embodiments shown in Figures 5a) to d) also include a computing device (not shown) which is connected to the area detector 20 and is configured to perform a method according to any one of claims 1 to 5. The components belong to a lensless digital holographic microscope with a Fourier holographic setup.

[0059] The features of the invention disclosed in the foregoing description, in the drawings and in the claims can be essential for the realization of the invention in its various embodiments, both individually and in any combination. Reference symbol list 10 Hologram 12 Siemens stars 14 Point light source 16 Reference point light source 18 beam splitters 20 area detector 22 lens d optical / spatial resolution DE Detector level FP Focus Point FP1 Focus Point OE object level AP pixel pitch L O Edge length L S Edge length N number of pixels Z distance 24 Lighting beam path 26 objects 28 field of vision 30 reference mirrors 32 Mach-Zehnder interferometers 34 fibers 36 lenses 38 fiber ends 40 lenses 42 lenses 44 Michelson interferometers 46 Polarizer 48 λ / 2 plate 50 lasers 52 fiber shards

Claims

[1] Method for lensless digital holographic microscopy with sub-pixel resolution using a single hologram, comprising - digital interpolation of a complex-valued hologram of an object in an object plane, recorded in the detector plane of an area detector, in particular a camera sensor, using an approximately spherical reference wave with an origin, onto a fine scanning grid with sub-pixel resolution, - numerical compensation of the reference wave, and - exact reconstruction of the interpolated hologram. [2] The method of claim 1, further comprising: Recording the complex-valued digital hologram of the object in the object plane using Fourier holography with the use of the approximately spherical reference wave in the detector plane by means of the area detector. [3] Method according to claim 1 or 2, wherein the origin of the reference wave and the object are equidistant from the area detector. [4] Method according to any one of claims 1 to 3, wherein the digital interpolation comprises: - Fresnel reconstruction of the object to near the object plane, possibly taking the reference wave into account, - Applying zero-padding by a factor of ρ around the reconstruction, and - Inverse Fresnel reconstruction into the detector plane. [5] Method according to any of the preceding claims, comprising the exact reconstruction of the interpolated hologram: - Exact reconstruction of the interpolated hologram by applying the Angular Spectrum method. [6] Device for lensless digital holographic microscopy with sub-pixel resolution with a single hologram (10), comprising - a lensless digital holographic microscope with Fourier holography setup, comprising an illumination device for generating object and reference illumination and an area detector (28) with a detector plane (DE), and - a computing device connected to the area detector (20) and configured to perform the method described in claims 1 to 5. [7] Device according to claim 6, wherein the microscope is set up for a transmitted light mode. [8] Device according to claim 7, wherein the microscope comprises a beam splitter (18). [9] Device according to claim 7, wherein the microscope comprises a Mach-Zehnder interferometer. [10] Device according to claim 6, wherein the microscope is set up for a reflected light mode. [11] Device according to claim 10, wherein the microscope comprises a beam splitter (18). [12] Device according to claim 10, wherein the microscope comprises a Michelson interferometer (44). [13] Device according to one of claims 6 to 12, wherein the lighting device comprises at least one point light source (14, 16). [14] Device according to one of claims 6 to 13, wherein the lighting device comprises at least one, preferably fiber-coupled, laser light source (50).

Images