Method and apparatus for determining the position of an optical interface along a first direction

The method improves the accuracy of determining the optical interface position by averaging and transforming image data to frequency space, addressing noise and lighting issues in existing microscopy techniques.

DE102016113068B4Active Publication Date: 2024-12-05CARL ZEISS MICROSCOPY GMBH
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
DE102016113068
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Filing Date
2016-07-15
Publication Date
2024-12-05
Estimated Expiration
2036-07-15

AI Technical Summary

Technical Problem

Existing methods for determining the position of an optical interface using structured illumination in microscopy are prone to noise and lighting variations, leading to unstable results and requiring empirical parameter adjustments.

Method used

A method involving imaging a periodic pattern, averaging images along a transverse direction, transforming into frequency space, and determining the position based on the largest frequency amplitude, eliminating the need for derivative-based contrast functions and reducing sensitivity to noise and lighting variations.

Benefits of technology

The method provides a robust and accurate determination of the optical interface position, reducing the need for parameter adjustments and enhancing precision by using data reduction techniques.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Method for determining the position of an optical interface along a first direction, in which the following steps are performed, a) Mapping a pattern onto a plane perpendicular to the first direction, wherein the pattern in the plane varies periodically in a second direction, and recording a two-dimensional image of the pattern mapped onto the plane, b) Repeating step a) for different positions in the first direction, wherein the different positions cover an area in the first direction in which the optical interface lies, c) By means of each image from step a) along one direction perpendicular to the second direction such that a one-dimensional data set is generated in each case, d) Transforming each data set from step c) into a frequency space, e) Determining the frequency of the largest amplitude of all data sets transformed in step d), f) Determining the position along the first direction by evaluating the data sets transformed in step d) at the frequency determined in step e).
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The present invention relates to a method and a device for determining the position of an optical interface along a first direction.

[0002] Methods using structured illumination are known in microscopy to determine the position of an optical interface. As described, for example, in US 2008 / 0292135 A1, the contrast of the grating pattern imaged on the interface and detected by the camera depends on the axial position, so that the axial position of the interface can be determined. Various forms of derivatives are used to determine the contrast function from the recorded image data. A general disadvantage of derivatives is their sensitivity to disturbances such as noise and differences in illumination, which are often reflected in the contrast function. Furthermore, a derivative of a contrast function obtained in this way is often used to find the position via the zero crossings.However, it has been shown that such approaches are unstable, and that this usually requires additional filters and segmentations with parameters for adaptation. The choice of these parameters is usually empirical and only valid in a limited application space, thus providing no robust method for determining the position of an optical interface.

[0003] Based on this, the object of the invention is to provide a method for determining the position of an optical interface along a first direction, which overcomes the difficulties mentioned above as completely as possible. Furthermore, a corresponding device for determining the position of the optical interface along a first direction is to be provided.

[0004] DE 10 2014 216 227 A1 describes a method and a device for determining a distance between two boundary surfaces spaced apart along a first direction. US 2009 / 0022401 A1 describes a method and a device for image processing, image feature extraction and image comparison.

[0005] The invention is defined in the independent claims. Advantageous further developments are specified in the dependent claims.

[0006] According to the invention, the object is achieved by a method for determining the position of an optical interface along a first direction, in which the following steps are carried out: a) imaging a pattern in a plane transverse to the first direction, the pattern in the plane varying periodically in a second direction, and recording a two-dimensional image of the pattern imaged in the plane, b) repeating step a) for different positions in the first direction, wherein the different positions cover a region in the first direction in which the optical interface lies, c) averaging each image from step a) along a direction transverse (preferably perpendicular) to the second direction such that a one-dimensional data set is generated in each case, d) Transforming each data set from step c) into a frequency space, e) Determining the frequency of the largest amplitude of all data sets transformed in step d), f) Determining the position along the first direction by evaluating the data sets transformed in step d) at the frequency determined in step e).

[0007] By averaging according to step c) and transforming according to step d), the contrast function and its maxima are determined not by derivatives, but by summation. Therefore, the method according to the invention is very robust against unavoidable disturbances, such as noise or illumination differences. Additional adjustment by parameters is therefore no longer necessary.

[0008] Step c) essentially performs a data reduction to one dimension using projection, which increases the robustness of the inventive method. Steps d) and e) then essentially perform a data reduction to one spectral component, which further increases the robustness of the inventive method.

[0009] The method according to the invention for determining the position of an optical interface along a first direction can also be characterized by the following steps: A) imaging a pattern in a plane transverse to the first direction, the pattern in the plane varying periodically in a second direction, and taking an image of the pattern imaged in the plane, B) Repeating step A) for different positions in the first direction, wherein the different positions cover a region in the first direction in which the optical interface lies, C) Reducing the data of each image from step A) to one dimension using projection, D) reducing the data generated in step C) to a spectral component having the highest amplitude, E) Determine the frequency of the spectral component of step D), F) Determining the position along the first direction by evaluating the data according to step C) at the frequency determined in step E).

[0010] It is possible to replace a step labeled with a capital letter with the step mentioned above with the same but lowercase letters. The same applies vice versa.

[0011] In the method according to the invention, the pattern in step a) or A) can be provided such that it varies periodically only in the second direction. In step c), the averaging is preferably performed such that it occurs only along the direction in which the pattern does not vary or oscillate. The same applies to the projection in step c).

[0012] The averaging according to step c) can be carried out by means of an appropriate projection.

[0013] In the method according to the invention, the pattern can be a striped grating. The pattern can, in principle, have any periodic grating function along the second direction. For example, the grating function can be a triangular function, a rectangular function, a sine function, etc.

[0014] Furthermore, a discrete Fourier transform can be performed in step d) or d). This can, in particular, be a one-dimensional discrete Fourier transform. Of course, other transformations can also be performed in step d) or d). For example, transformations based on cosine, sine, or other suitable functional transformations can be performed.

[0015] In particular, a maximum can be sought in step f) or f). Thus, in step f) or f), an approximation can be performed using a cubic polynomial function or cubic splines. These have very good properties at the connecting points, as they are doubly differentiable there. This allows for a very precise determination of zeros and extrema. Of course, the approximation in step f) or f) can also be performed using other interpolants.

[0016] In particular, in step f) or F), all spectral data generated in step d) or D) can be evaluated at the frequency determined in step e) or E).

[0017] Furthermore, in the method according to the invention, the optical interface can be an interface of a slide, and in step f) or f), a spherical aberration caused by the slide can be corrected. This leads to a further improvement in the measurement result.

[0018] In the method according to the invention, the position of two optical interfaces and, from this, the distance between the two optical interfaces can be determined using steps a) - f) or A) - F). In particular, the mechanical distance between the optical interfaces can also be determined from this.

[0019] The optical interface can thus be, in particular, an optical interface of a microscope slide. This slide can be designed, for example, as a cover glass or a Petri dish.

[0020] Furthermore, in the method according to the invention, the frequency can be determined in step e) or e) using a parabolic fit. This allows very precise results to be achieved. Furthermore, it is possible to determine the frequency in step e) or e) using other interpolants.

[0021] In the method according to the invention, the direction transverse to the second direction can be determined based on the respective image in step c) or C). For this purpose, for example, the averaging or projection can be defined as rotation-dependent, so that a lateral rotation of the pattern in the plane is included as a parameter or variable. The averaging or rotation can then be evaluated such that the direction transverse to the second direction is preferably perpendicular to the second direction. This ensures that no component along the second direction, in which the pattern varies periodically, is taken into account during the averaging or projection. This procedure can ensure that non-optimal alignments of the pattern can be compensated for in step a) or A). This leads to better results when determining the position of the optical interface.

[0022] In the method according to the invention, the various positions according to step b) or B) can be equidistant. However, it is also possible for the distances to be non-constant.

[0023] In particular, the different positions according to step b) or B) can have a smaller distance the closer they are to the position of the optical interface.

[0024] In the method according to the invention, the pattern is preferably imaged in an enlarged manner in step a) or A).

[0025] The method according to the invention can be used in particular in reflected light or transmitted light microscopes.

[0026] Furthermore, a device for determining the position of an optical interface along a first direction is provided, which device has an illumination module for imaging a pattern in a plane transverse to the first direction, an imaging module for recording the pattern imaged in the plane and a control module which is designed to carry out the steps of the method according to the invention (including its further developments).

[0027] The device can be designed, in particular, as a microscope. The microscope can be designed, for example, as a reflected-light microscope or a transmitted-light microscope.

[0028] It is understood that the features mentioned above and those to be explained below can be used not only in the combinations indicated, but also in other combinations or in isolation, without departing from the scope of the present invention.

[0029] The invention is explained in more detail below, for example, with reference to the accompanying drawings, which also disclose essential features of the invention. They show: Fig. 1 is a schematic representation of the structure of an embodiment of the device 1 according to the invention for determining the position of an optical interface along a first direction; Fig. 2 an enlarged view of the objective 9 and the sample 14 of the device 1 of Fig. 1; Fig. 3 a schematic representation of the strip grating 24 to be imaged into the focal plane, and Fig. 4 a flow chart for the method according to the invention.

[0030] In Fig. Figure 1 schematically shows the structure of an embodiment of the device according to the invention for determining the position of an optical interface along a first direction. In this embodiment, the device 1 is designed as an inverted reflected-light microscope comprising an illumination module 2 and an imaging module 3.

[0031] The illumination module 2 can, for example, have a first light source 4 with a first optic 5, a second light source 6 with a second optic 7, an illumination optic 8 and a lens 9.

[0032] The first light source 4 can, for example, emit light with a wavelength of 420 nm, and the second light source 6 can, for example, emit light with a wavelength of 625 nm. In principle, light with a wavelength from the visible wavelength range, the infrared range, or the UV range can be used.

[0033] Furthermore, deflecting mirrors 10, 11 and 12 and a beam combiner 13 are provided, so that the light L of the first or second light source 4, 6 can preferably be selectively directed for the illumination optics 8 and the objective 9 for illumination onto a sample 14.

[0034] The imaging module 3 comprises the objective lens 9, a beam splitter 15, an imaging optics 16 and a camera 17. Thus, a (preferably enlarged) image of the sample 14 can be recorded via the objective lens 9, the beam splitter 15 and the imaging optics 16 with the camera 17.

[0035] Furthermore, the device 1 also has a control module M, which controls the device 1 during operation and performs the determination of the position of the optical interface described below. The control module M can be part of the device 1, can be designed as a separate module, or can be a combination of both.

[0036] In Fig. Figure 2 shows an enlarged detailed view of the objective 9 and the sample 14. The sample 14 can be, for example, a cell sample floating in a liquid medium 18 contained in a Petri dish 19. An immersion medium T can be present between a bottom 20 of the Petri dish 19 and the objective. For high-quality imaging, it may be necessary, for example, to determine the precise position of an upper interface 22 of the bottom 20.

[0037] For this purpose, in a plane 23 conjugate to the focal plane of the imaging module 3 ( Fig. 1) inside the lighting module 2 a strip grid 24 (as shown schematically in Fig. 3) so that a corresponding striped grid-shaped structured illumination is present in the focal plane. The structured illumination varies periodically in the x-direction, which can also be referred to as the second direction.

[0038] With this illumination, several images are taken by means of the imaging module 3, with the images being taken at different z-positions (the z-direction can also be referred to as the first direction). Thus, a so-called z-stack is recorded. The z-positions are selected such that they cover a z-range 25 within which at least the upper boundary surface 22 lies. In the embodiment described here, the z-range 25 is selected such that the lower boundary surface 21 of the base 20 of the Petri dish 19 also lies within the z-range 25.

[0039] Since there is a refractive index jump at the interface 22, reflection occurs here, whereas no reflection occurs at other locations inside or outside the base 20. This can be used to determine the position of the upper interface 22 along the first direction (in the z-direction) with high accuracy. Of course, it is also possible to determine the position of the lower interface 21 with high accuracy, since a refractive index jump also occurs here. By determining the positions of the two interfaces 21 and 22, the thickness of the base 20 of the Petri dish 19 can also be determined with high accuracy.

[0040] In the device 1 according to the invention, a predetermined z-range 25 is thus traversed (the focal plane is shifted in the z-direction), with a sharp image of the striped grating 24 and thus the desired strip-shaped structured illumination always being present in the focal plane. Traversing the predetermined z-range 25 can, for example, be performed in equidistant steps. However, it is also possible to vary the step size. For example, the possible position of the refractive index jump can be located in large steps, in order to then achieve a finer scan around this range in smaller steps.

[0041] Structured illumination is used because it is known from epiluminescence bright-field microscopy that lateral planar objects, such as reflections from planar surfaces, cannot be resolved axially. The reason for this is that the optical transfer function (OTF) does not allow axial spatial frequency components to pass through. Therefore, it is impossible to find the focal position of such a reflection. A rather convenient solution to this difficulty is to introduce a lateral detail, such as a periodic grating composed of a conjugate illumination plane, such as plane 23. Using the Fourier shift theorem, it can be shown that this modulation shifts the optical transfer function laterally in the grating direction, allowing axial zero-frequency components to pass through.

[0042] As already mentioned, a reflection occurs at an interface with a refractive index jump or an abrupt change in the refractive index, which can be explained by the Huygens-Fresnel principle. However, the distance between two such reflections at the upper and lower interfaces 21 and 22 does not yet represent the possible thickness of the intermediate layer. Derived from Snell's law, the measured distance d a Calculate the actual thickness d of the soil according to equation 1: d=dangni

[0043] Where n i the refractive index of the immersion medium T and n gthe refractive index of the base 20. Strictly speaking, equation 1 is only valid in the absence of spherical aberration. Assuming that the spherical aberration directly beneath the base 20 is corrected, this cannot generally be true for the other side of the base 20. The additional focus shift at this first reflection (reflection at the upper interface 22) due to the spherical aberration can be taken into account in the interest of smaller measurement errors. A further measurement error can be generated if the reflection of the second interface 22 is caused by an embedding medium, which contradicts the optical design parameters. To eliminate such measurement errors, a model-based approach is proposed that determines the shifts in the focus positions caused by the spherical aberration.

[0044] The mode of operation is described using a mathematical model of structured illumination. The image formation (observation) on camera 17 is as follows: g=hd∗f(hi∗s)

[0045] Where g is the observation, f is the reflective surface, ( * ) a convolution operator, h l the PSF (= point spread function) for the illumination and h d the PSF for detection. All quantities and operators are three-dimensional in the ℝ domain. The quantity s is the incoherent, grating-modulated intensity, which for simplicity is represented here as a cosine function with the grating frequency ω: s(x,y,z)=12δ(z−zI)[1+cos(ωx)]

[0046] The special case of structured lighting, where f is an axis-orthogonal plane, dictates that f(x,y,z) = δ(z - z g ), where z g the focal position of the observation plane and z lis the focal position of the modulated illumination plane. If these two planes do not coincide exactly, we observe that only a weaker signal is observed, and the measurement does not suffer any additional error. However, in the interest of a good signal-to-noise ratio (SNR), a parfocal alignment should be performed. Since f now represents an axial Dirac distribution, it can be seen from Equation 2 that only the point spread function h d The detection influences the observed intensities in the axial direction. In the presence of spherical aberration, a significant shift in the focus position can occur.

[0047] For patterns defined in Equation 3, the detection signal-to-noise ratio can be significantly improved by averaging the repeating structures. To do this, we define a projection along the y-coordinate: p(x,z)=∫Y g(x,y,z)dy=Y[1+mD(z−zg)cosωx]

[0048] Here, m D (z) = c ∫ cosωx[∫h D (x,y,z)dy]dx and c is a constant. It can be seen that exactly at z = z g the amplitude reaches a maximum in p. Using the Fourier transform, it can be shown that due to the following proportionality, it is possible to determine the unknown location z g to determine: P(u,z)=Y[2πδ(u)+mD(z−zg)π(δ(u+ω)+δ(u−ω))]‖P(±ω,z)‖∼mD(z−zg)

[0049] The application of a discrete Fourier transform (DFT) has the advantage that only ± ω components arise from summation and thus repeatedly significantly improves the signal-to-noise ratio, which further increases the robustness of the algorithm.

[0050] Technical implementations often show that a precise alignment of the grating 24 cannot always be achieved mechanically in order to correctly perform the simple projection from equation 4 above. To take this into account, we define a rotation-dependent projection as pc(x,y;φ)=1Y∑yg(Rφ(x,y)T,z); Rφ=(cos(φ)-sin(φ)sin(φ)cos(φ)) and use the norm of the 1 D Fourier transform over x at ± ω as a cost function to determine the lateral rotation angle of the grating φ by maximizing: ξ(φ)=‖FT{pc(x,zg,φ)}(±ω)‖→max.

[0051] Working with discretely sampled data always requires interpolation adapted to the expected measurement accuracy. In the embodiment described here, a piecewise approximation using a cubic polynomial function, as shown in Equation 8, is used. This has the additional advantage of being able to find the exact position of the desired peak or maximum using simple analytical expressions: k(x)=Ui{a0i+a1ix+a2ix2+a3ix3}

[0052] This type of approximation is also known as cubic splines. These have the excellent property of being doubly differentiable at the connecting points. This allows for the precise determination of zeros and extrema. The generation of the coefficients (a i ) will not be discussed in detail here, as this is sufficiently known to the expert.

[0053] The workflow of the method according to the invention for determining the position of an optical interface along a first direction can be described as follows.

[0054] In step S1 ( Fig. 4) The z-stack is acquired in the manner described. This can include the two interfaces 21, 22. Alternatively, it is possible to acquire a first z-stack that includes the upper interface 22 and a second z-stack that includes the lower interface 21. Each image from a z-plane (which can also be referred to as a z-slice) contains the precise z-position, e.g., of a sample stage on which the Petri dish 19 is positioned. The width of each z-stack should be at least 2-3 times the full width at half maximum (FWHM) in the axial direction. The sampling should not be coarser than half the axial resolution (Nyquist). A finer sampling is preferred.

[0055] In step S2, with approximately known rotation of the grating 24 (if it is a line grating, the preferred direction is horizontal), positions of the dominant reflections in the z-stack are searched as follows: In substep a), all recorded z-slices are converted into a 2D array of zx-positions using equation 6.

[0056] Then, in substep b), each projection is transformed using a one-dimensional discrete Fourier transform.

[0057] In the following substep c), the spectral component of the largest norm (largest amplitude) that differs from zero frequency is found along z according to the relationship in Equation 5. The exact grating frequency is determined using a parabolic fit. This will later be used to find all maxima.

[0058] Now, in the optional substep d), the lateral rotation angle of grating 24 can be determined according to equation 7 if the given angle deviates from the assumed angle by approximately more than ± 10°. In this case, substeps a) and b) are repeated with the correct angle. Since this angle determination is time-consuming, it is recommended to reuse the angle value determined in this way when repeating the measurement in substep a) and then skip substep b).

[0059] In sub-step e), the spectral data obtained in sub-step b) are evaluated for all z-spectra at the grating frequency determined in sub-step c). From this point on, all data are continuously analyzed using cubic splines. Along the optical axis (z-direction), all maxima of the spectrum norm and their corresponding positions are continuously found using the derivatives of the polynomials. In the case of z-stacks fragmented in step S1, the application of a simple low-pass filter before the derivation can help reduce discontinuous discrete jumps at the transitions of these z-stacks. This can prevent error detections at these transitions. This measure is not necessary for a continuous z-stack.

[0060] In substep f), the positions of the desired N interfaces can be obtained from a sorting step. If the number N is usually known, higher entries can be considered as existing maxima after sorting, but these pseudo-maxima can be neglected. N is an integer greater than or equal to 1.

[0061] In step S3, the distances are now available in units of continuous table coordinates. Using Equation 1 and known refractive indices, the mechanical distance can be calculated.

[0062] In the optional step S4, a correction of the spherical aberration can be carried out, for example, using the Gibson-Lanni model (Gibson, Lanni, 1992, Experimental Test of an Analytical Model of Aberration in an Oil-Immersion Objective Lens used in three Dimensional Light Microscopy, JOSA, 9 (1), 154 ff).

[0063] This model geometrically determines the optical path lengths through three different optical media, resulting in a pupil function. Here, the three optical media are sample 14, ground 20, and immersion medium T. Using this model, the axial shift of the focus position can be obtained from the calculation of the point spread function. To determine the shift of the plane closest to objective 9, we modify the Gibson-Lanni model to a 2-layer model. To achieve this, and assuming that the refractive indices of immersion medium T and sample 14 are approximately the same, the modulated thickness of ground 20 can be set to zero. If we now calculate a point spread function along the lateral central line in the axial direction (often referred to as the line spread function), the shift of the global maximum is approximately equal to the expected measurement error.You then only need to add this to the previously determined result.

Claims

[1] Method for determining the position of an optical interface along a first direction, in which the following steps are carried out, a) imaging a pattern in a plane transverse to the first direction, the pattern in the plane varying periodically in a second direction, and recording a two-dimensional image of the pattern imaged in the plane, b) repeating step a) for different positions in the first direction, wherein the different positions cover a region in the first direction in which the optical interface lies, c) averaging each image from step a) along a direction transverse to the second direction such that a one-dimensional data set is generated in each case, d) Transforming each data set from step c) into a frequency space, e) Determining the frequency of the largest amplitude of all data sets transformed in step d), f) Determining the position along the first direction by evaluating the data sets transformed in step d) at the frequency determined in step e). [2] A method according to claim 1, wherein the pattern in step a) varies periodically only in the second direction. [3] A method according to claim 1 or 2, wherein the pattern in step a) is a striped grid. [4] Method according to one of the above claims, wherein in step d) a discrete Fourier transformation is carried out. [5] Method according to one of the above claims, wherein in step f) an approximation is carried out with a cubic polynomial function. [6] Method according to one of the above claims, in which the optical interface is an interface of a slide and in step f) a spherical aberration caused by the slide is corrected. [7] Method according to one of the above claims, in which the position of two optical interfaces and, therefrom, the distance between the two optical interfaces are determined by steps a) - f). [8] Method according to one of the above claims, in which in step e) the frequency is determined by means of a parabolic fit and / or in which in step c) the direction transverse to the second direction is determined on the basis of the respective image. [9] Device for determining the position of an optical interface along a first direction, with an illumination module (2) for imaging a pattern in a plane transverse to the first direction, an imaging module (3) for recording the image of the pattern imaged in the plane and a control module (M) designed to carry out steps a) to f) according to claim 1. [10] Apparatus according to claim 9, wherein the control module (M) is further configured to perform the steps of any one of claims 2 to 8.

Citation Information

Patent Citations

  • Method and Device For Reconstructing Images

    US20080292135A1

  • microscope with autofocus device

    DE102006027836A1

  • Autofocus methods for microscopes and microscopes with autofocus devices

    DE102011077236A1

  • Method and apparatus for determining the distance between two optical interfaces spaced apart along a first direction

    DE102014216227A1

  • Image Processing, Image Feature Extraction and Image Matching Apparatuses, Methods and Programs, and Image Matching System

    US20090022401A1