A method for providing transfer functions in light-field microscopy.

Abstract

This efficiently provides a wide range of transfer functions for light-field microscopy. [Solution] A sample volume to be imaged with multiple sub-images (1~12xy) from different capture angles (Φ) is captured, and for the sub-images (1~12xy) intended for 3D reconstruction, the respective point image distribution function (PSF) is identified. The sub-images (1~12xy) are examined for optical symmetry, the symmetry is captured, and stored along with a calculation rule. The calculation rule can start with the point image distribution function (PSF) and / or processed point image distribution function of the first sub-image (1~12) and generate the corresponding point image distribution function (PSF) and / or processed point image distribution function of further sub-images (2x~12xy) that are symmetric with respect to the first sub-image (1~12), and the point image distribution function (PSF) or processed point image distribution function of further sub-images (2x~12xy) is generated only when necessary.

Description

[Technical Field]

[0001] The present invention relates to a method according to the preceding part of an independent claim. [Background technology]

[0002] For example, in the field of microscopy of dynamic systems, it is desirable to capture a three-dimensional sample volume in a very short time while minimizing the load on the sample, which is often living. In recent years, so-called light-field microscopy has proven to be very suitable for this purpose. In this microscopy technique, the detection radiation coming from the sample volume is essentially divided into numerous sub-images. In the simplest case, the sample in question is captured only once using a two-dimensional detector, but a 3D reconstruction of the sample volume can be calculated based on the captured sub-images, each containing different angular information of the sample volume.

[0003] Each subimage inherently possesses specific optical properties that describe the optical transfer of the sample from a point source to a detector. Such mathematical relationships are sometimes referred to as point spread functions (PSFs). When such transfer functions are provided for subsequent 3D reconstruction, they typically need to be transmitted from the (main) data storage, data memory, or data storage device (Datenspeicher) to the processing unit where the actual 3D reconstruction is performed.

[0004] Because such transfer functions describe complex three-dimensional relationships, they require a large amount of memory. When passed to one or more transfer function processing units, the large amount of data and the associated access time cause a time delay in the process flow. Furthermore, the memory (cache) of processing units is typically designed for relatively small amounts of data (intermediate) storage. [Overview of the project]

[0005] The problem that this invention aims to solve is to propose a method that can efficiently provide a wide range of transfer functions.

[0006] This problem is solved by the method described in the main claim of the patent claims. A favorable development is the subject of the dependent claim.

[0007] This method is particularly useful for providing transfer functions in light-field microscopy. In this method, detection radiation from a sample volume to be imaged in multiple sub-images is captured at different capture angles. All or some of the selected sub-images are provided for subsequent virtual 3D reconstruction of the sample volume. For at least these selected sub-images, the respective point image transfer function (hereinafter referred to as PSF) is identified, assigned to the sub-image sub-image, and stored. The determination of the PSF can be performed, for example, by measurement, simulation, and / or calculation.

[0008] The method according to the present invention is characterized in that a partial image is examined with respect to the existing symmetry of the underlying optical technology (image) data. Here, symmetry is understood to mean primarily the intrinsic properties of each partial image, such as the PSF. Symmetry in the sense of this specification exists when the transfer function of the first partial image can be transformed by virtual mirroring into a transfer function that is effective for further partial images of the captured partial image. Mirroring can occur, for example, on an axis that extends orthogonally to and intersects with the optical axis of the captured radiant flux of the detection radiation. In the following description, the x and y axes of the Cartesian coordinate system are used as examples, and the z axis of the Cartesian coordinate system coincides in particular with the optical axis of the detection beam path.

[0009] Once the symmetry is determined, these are captured and stored along with the calculation rules. The calculation rules allow for the generation of corresponding PSFs and / or processed PSFs of further subimages symmetric to the first subimage, starting from the PSF and / or processed PSF (see below) of the first subimage, which can also be understood as the original. In this case, the PSFs and / or processed PSFs of other subimages can preferably be generated only when necessary. For example, this may be necessary if a current 3D reconstruction of the sample volume is actually performed. Since the PSF or processed PSF can be converted to the desired symmetric PSF at any time using the PSF of the first subimage and the associated calculation rules, the storage of PSFs and / or processed PSFs generated using the calculation rules is advantageously only needed temporarily.

[0010] In the sense of this specification, the processed PSF is obtained by performing at least one mathematical operation on each PSF. The results of the relevant mathematical operation are stored in relation to the relevant subimage. In one aspect of this method, the processed PSF is an optical transfer function (OTF), which is generated using the Fourier transform of the PSF. In one aspect of the method according to the present invention, the discrete Fourier transform (DFT) is used.

[0011] In the context of this invention, the term "transfer function" means a point image distribution function, or a processed point image distribution function such as an OTF.

[0012] Symmetrical partial images, more precisely, can arise in the field of light-field microscopy based on the fact that partial images are produced by the action of optical elements in the detection beam path. If the optical elements have symmetry in their design, or if symmetry exists during actual image recording, these symmetries can be used in the sense of the present invention. For example, the microlenses of a microlens array used to generate partial images can be arranged symmetrically with respect to a virtual axis. If, during image recording, the microlens array is arranged such that the optical axis and the virtual axis of the beam path coincide, then the microlenses and, in each case, the partial images they produce can be considered symmetric, and their positions can be transformed from one another by mirroring (Spiegelungen).

[0013] Additionally or alternatively, the calculation rules can be used, if necessary, to generate further symmetric subsets within the image data of at least one subimage based on a first subset of the transfer functions of the subimages. In this way, a subset or section, e.g., half, of the PSF can be generated based on another section, another half, as will be explained in more detail below. In this manner, by taking advantage of the symmetry of the subimage properties, in particular the symmetry of the resulting PSF itself, all the necessary PSFs and / or processed PSFs (hereinafter also referred to as OTFs) can be provided with significantly reduced memory space requirements. Subsets of OTFs can be captured as symmetric to each other if, for example, they can be represented as complex conjugates with opposite imaginary parts. Thus, for example, a symmetric subset of a PSF is a PSF invert (x,y) can be written as PSF(-x,-y). In 2D Fourier space, this corresponds to the following relationship: OTF invert (kx,ky) = OTF * (kx,ky)

[0014] Therefore, the core of the present invention is to take advantage of the fact that when recording images by this method of light-field microscopy, transfer functions are generated that are assigned to different spatial positions but are otherwise identical. Furthermore, symmetry may arise in the optical technical data of the partial image, which can be advantageously utilized in the process of the method according to the present invention.

[0015] The present invention makes it possible to advantageously utilize typical technical configurations of different data storage devices. Data storage devices such as hard disks, RAM (random access memory) of CPUs (central processing units), and VRAM (video random access memory) of GPUs (graphics processing units) have large storage capacities (viel Speicherplatz), but their data access rates are relatively low, resulting in long data access times. For example, loading a large amount of data, such as that given by a fully computed and stored transfer function, takes a considerably long time.

[0016] In contrast, data in the buffer memory (cache, on-chip memory, GPU shared memory) of a processing unit (processor) can be accessed up to 20 times faster (compared to internal transfer speeds L1 / L2 / L3) based on its high data transfer rate. However, the storage capacity of such buffer memory is 200 to 2000 times smaller than the storage capacity of the aforementioned data storage. CPUs and GPUs can also be used as processing units.

[0017] This invention enables the use of high data transfer speeds in buffer storage devices, while still efficiently providing and utilizing the complex transfer functions required for 3D reconstruction.

[0018] In particular, if necessary, for example during 3D reconstruction, only the PSF of the first partial image and / or the processed PSF can be stored in the buffer memory, and based on this, the necessary PSF and / or the processed PSF of further partial images can be generated by the processing unit itself ("on demand"). Therefore, there is no need to hold them or load them from a data storage device with a low data transfer rate.

[0019] The present invention can be advantageously used, for example, when processing a plurality of image recordings obtained with the same PSF. This situation occurs, for example, when the same microscope with the same settings and the same optical elements is used for all of these image recordings, especially within the detection beam path. Subsequently, the PSF of the first partial image and / or the processed PSF can be stored only once (einmalig) in the buffer memory of the processing unit and can be used, for example, as a basis for 3D reconstruction of most of the image recordings.

[0020] To reconstruct the sample volume, in each form of the method according to the invention, the experimentally collected data, i.e., the captured measurements and the image data generated therefrom, can be compared with predicted data (forward model). For a specific partial image, i.e., an image of the sample at a specific capture angle (viewing direction: Blickrichtung), the forward model can be understood as the 2D projection P of the 3D model M of the sample folded with the associated PSF (mit der zugehoerigen PSF gefaltet ist).

[0021] To calculate the forward model, the transformation of the PSF using the discrete Fourier transform DFT can be utilized.

[0022]

Number

[0024]

Number

[0025] Here, M z and the PSF z are the individual 2D layers of the object model M and the corresponding PSF, respectively. The operator "·" represents the element-by-element product.

[0026] Equation (2) reduces the computational complexity and access time (= number of RAM accesses) and supports the use of the existing symmetry of the transfer function.

[0027] In an advantageous embodiment of the method according to the invention, the processed PFS, in particular the OTF, of the captured light field is stored. Such an OTF can be defined as OTF LF = 2DDFT(PSF z (x,y) and can be in the form of Equation (3) below.

[0028]

Number

[0029] The method for providing the proposed transfer function according to the invention can be advantageously used particularly in a system configured for imaging using the concept of a light field. The method according to the invention can be used particularly within the framework of a reconstruction method for virtually reconstructing a sample volume captured in a plurality of partial images from different capture angles.

Brief Description of the Drawings

[0030] Hereinafter, the present invention will be described in detail with reference to the drawings and examples. [Figure 1] FIG. 1 is a diagram schematically showing a microlens array and the selected symmetry of the existing microlenses. [Figure 2] Figures 2a to 2c schematically illustrate the symmetry between subsets of point image distribution functions and the processing according to the present invention. [Figure 3] Figure 3 schematically illustrates the adaptation of the focal position based on the spatial positioning of the point image distribution function. [Figure 4] Figure 4 is a schematic diagram illustrating an example of a light-field microscope. [Modes for carrying out the invention]

[0031] Figure 1 illustrates a first form of possible symmetry in the optically-technical image data of a hypothetically assumed detection beam. In conventional imaging systems using a light field, such as a light-field microscope 12 (see Figure 4), the detection beam coming from the sample volume to be imaged is directed to an optically effective element, which splits the detection beam and images it as multiple sub-images on a subsequent detector 16 (see Figure 4). Each sub-image is directed to the sample volume at a different capture angle.

[0032] Figure 1 shows a microlens array 15 having a total of 37 microlenses 1 to 12xy. Since each of the microlenses 1 to 12xy generates a partial image, for simplicity of explanation, the microlenses 1 to 12xy are identified with their corresponding partial images 1 to 12xy.

[0033] Microlenses 1 to 12xy are arranged substantially concentrically around the central microlens 1. To carry out the method according to the present invention, microlens 1 is positioned on the optical axis of the detection beam path. When the microlens array 15 is uniformly illuminated, symmetry can be determined with respect to the relative arrangement of some of the microlenses. In this example, microlenses 1 to 12, located in the upper right quadrant, are selected as the reference point (first partial image). The action of microlenses 1 to 12 generates partial images 1 to 12.

[0034] Figure 1 shows, for example, the x and y axes, which are oriented orthogonal to each other with respect to the optical axis (z axis). Microlenses described as symmetric can be determined by assumed mirroring of some of the microlenses 2 through 12, either sequentially along the x, y, or both x and y axes. A group of symmetric microlenses can be assumed to have the same point image distribution function, and for evaluation, it is only necessary to associate this with the respective positions of specific microlenses within the microlens array 15.

[0035] In this example, the microlens or partial image produced by mirroring along the x-axis is indicated by the sign "x". The same applies to mirroring along the y-axis, or both the x and y axes. Since microlens 1 is located at the origin of the coordinate system (=optical axis), it cannot be mirrored along either the x or y axis.

[0036] This symmetry can be explained using microlens 3 as an example. Microlens 3, located in the upper right quadrant, can be virtually mirrored along the x-axis. Here we have a microlens denoted as 3x. Correspondingly, by mirroring microlens 3 along the y-axis, we obtain the corresponding microlens 3y. Mirroring the latter along the x-axis yields microlens 3xy. Correspondingly, microlens 3x can also be mirrored along the y-axis.

[0037] In this sense, the determined mutually symmetrical microlenses or partial images are captured and stored.

[0038] For microlens 3, the associated point image distribution function (PSF) is identified and stored. The processed PSF, particularly the optical transfer function (OTF), for microlens 3 can be calculated by the Fourier transform of the point image distribution function (PSF). In the sense of the present invention, in order to efficiently provide PSFs or OTFs for microlenses 3x, 3y, and 3xy, the PSF and / or OTF of microlens 3 are supplemented with the respective calculation rules for each of the microlenses 3x, 3y, and 3xy. For example, to generate the PSF for microlens 3xy, it is only necessary to retrieve the stored PSF for microlens 3xy, taking into account the relative position and capture angle of microlens 3xy according to the calculation rules (e.g., by calculating the complex conjugate OTF). This calculation can be performed, for example, by the processing unit 17, and the PSF thus generated is stored in the buffer memory 18. This also applies to the generation and provision of the respective processed PSFs, such as the OTF. Similarly, other microlenses or partial images with determined symmetric correspondences can also be handled.

[0039] In addition to the symmetry of the resulting transfer function, the fact that the transfer function, such as the point image distribution function (PSF), is formed mirror-symmetrically, for example, along its longitudinal extent, can also be utilized. Figure 2a shows an exemplary diagram of the PSF in the xz plane. The PSF is tilted at certain angles with respect to the x and z axes, respectively. The spatial position and orientation of the PSF represent the current line of sight (capture angle Φ; approximate).

[0040] For example, to save memory space, only half of the PSF (PSF / 2) can be stored and retained for subsequent use (Figure 2b). If necessary, the missing second half of PSF / 2 can be stored. invert If rules are stored in addition to the PSF / 2 data so that they can be supplemented, then when these rules are executed, the first part of the PSF / 2 data will start with the second part of the PSF / 2 data. invert It can be calculated quickly and a complete PSF can be generated.

[0041] To calculate the remaining half of the PSF using the above equations (1) and (2) for the 2D projection P, a mirrored PSF (see above) can be generated starting from the stored half PSF / 2 of the original PSF. When supplementing the OTF / 2, the OTF of the light field of Equation (3) OTF LF can be used. In this case, half of the OTF to be captured is the OTF in the x, y, and z directions LF which can be calculated by the complex conjugate.

[0042] In partial figure 2c, the lower half of the captured PSF / 2 is shown as PSF / 2 by different types of hatching invert as shown.

[0043] In an actual light field imaging system, the focal position of each microlens can be shifted along the line of sight direction and / or the capture angle Φ. Such a shift is different for each microlens and further depends on the wavelength of the detection beam.

[0044] Figure 3 illustrates such a case. As described above, the first subset of the PSF is transferred back to the buffer memory 18 of the processing unit 17. In this case, it is necessary to transfer data from a subset of the PSF that slightly exceeds half of the PSF in the z direction (hereinafter, referred to as PSF / 2 for simplicity). When calculating the subset of PSF / 2 to be captured, it is offset by an offset vector of the following form:

[0045]

Number

[0046] It shifts in the x, y, and z directions according to the offset of the focal position (focal offset). The focal offset is specified by arranging the actual focal position in the x, y, and z directions in relation to the theoretical focal position.

[0047] The focal offset relative to the center of the PSF is transmitted across the corresponding z level of the PSF / 2, thus shifting the PSF in the z direction (z-shift, z-offset). Furthermore, the entire PSF is shifted accordingly in the x and / or y directions. This can be done by individually determining a fixed xy offset (xy offset) for each microlens 1-12xy, or by using a determined xy offset for the PSF.

[0048] Furthermore, as a result of the shifted focal position, a correspondingly modified target region (region of interest, ROI, not shown) in the x, y, and z directions is determined to which the PSF should be applied.

[0049] As already explained with respect to Figure 2, preferably, a light field OTF according to formula (3) LF The optical transfer function of is used in the calculation. The PSF to be captured or a subset of the processed PSF is the light field OTF. LF The optical transfer function is recalculated using its complex conjugate with the opposite imaginary part. The shifts in the x and / or y directions are calculated using multiple 2D phase operators for the unshifted OTF. LF It is generated based on the following. The z-direction shift (z-shift) is calculated by adding the corresponding z-shift of the shift vector, and then OTF LF This is done computationally by calculating the region of interest (z-ROI) in the z-direction for both subsets. The calculation is performed by the processing unit 17 as backward projections, instead of calculating the full PSF or fully processed PSF in corresponding intermediate steps.

[0050] The method of the present invention can be carried out, for example, on the detection beam path of a light field microscope 20 (Figure 4). Detection radiation coming from the sample volume to be imaged is collected by the objective lens 13 and guided along the detection beam path by a transmission optics unit 14 (shown in a very simplified manner) and directed to an optical element, which in turn generates a number of partial images on the detector 16 that image the sample volume from different observation directions. In this embodiment, the optical element is configured as a microlens array 15 having a number of microlenses 1 to 12xy (see Figure 1).

[0051] The measurements captured by the detector 16 are transmitted to the processing unit 17. This unit is configured to retrieve from the data storage 19 data on the transfer functions of the selected microlenses 1-12xy, information on their existing symmetries regarding the generated partial images, and information on symmetries arising in relation to each transfer function itself. The information on the selected transfer functions and, if applicable, the calculation rules is used by the processing unit 17 to calculate the transfer functions as needed and store them in the processing unit 17's buffer storage 18, in particular so that they can be repeatedly called. [Explanation of Symbols]

[0052] 1-12xy microlens / partial image (Mikrolinse / Teilabbildung) 13. Objective lens (Objektiv) 14. Transmission Optics (Uebertragungsoptik) 15. Microlens array (Mikrolinsenarray) 16 Detector 17 Process Units (Prozessierungseinheit) 18. Buffer memory (Pufferspeicher) 19. Data Storage (Non-volatile, RAM) 20. Microscopes, Light-field microscopes (Mikroskop, Lichtfeldmikroskop) OTF-processed PSF, optical transfer function (prozessierte PSF, optische Transferfunktion) PSF (Point Scatter Distribution Function) Φ Acquisition angle (Erfassungswinkel)

Claims

1. A method for providing a transfer function in a light field microscope, Capture the sample volume to be imaged with multiple partial images (1–12xy) from different capture angles (Φ); For the partial images (1 to 12xy) intended for 3D reconstruction of the sample volume, each point image distribution function (PSF) is identified and assigned to the partial images (1 to 12xy) for use; The aforementioned partial image (1-12xy) is examined with respect to the symmetry of its underlying optical technical data; The determined symmetry is captured and stored along with the calculation rules, and the calculation rules are: - Starting from the point image distribution function (PSF) and / or processed point image distribution function of the first subimage (1 to 12), it is possible to generate the corresponding point image distribution function (PSF) and / or processed point image distribution function of further subimages (2x to 12xy) that are symmetric with respect to the first subimage (1 to 12), generate the point image distribution function (PSF) or processed point image distribution function of the further subimages (2x to 12xy) as a transfer function only when necessary, temporarily store them, and obtain the processed point image distribution function by performing at least one mathematical operation on each point image distribution function (PSF), and / or - If necessary, it is possible to generate a further symmetric subset within the transfer function of at least one subimage (1–12xy) based on a first subset of the transfer functions of the subimages (1–12xy). method.

2. Such partial images (2-12xy) are captured as symmetrical to one another and can be transformed to one another by mirroring on at least one mirror axis, the at least one mirror axis being orthogonal to and intersecting the optical axis of the captured beam beam of the detection beam. The method according to claim 1.

3. The processed point image distribution function is the optical transfer function (OTF). The method according to claim 1 or 2.

4. A subset of the point image distribution function (PSF) is PSF ｉｎｖｅｒｔｅｄ If (x, y) can be expressed in the form PSF(-x, -y), then they are determined to be symmetric to each other. The method according to any one of claims 1 to 3.

5. A subset of optical transfer functions (OTFs) is determined to be symmetric to one another if they can be expressed as complex conjugates with opposite imaginary parts. The method according to claim 3.

6. During 3D reconstruction, only the point image distribution function (PSF) or processed point image distribution function of the first partial image (1 to 12) is stored in the buffer memory (18) of the processing unit (17) that performs the 3D reconstruction, and based on this, the processing unit (17) generates the necessary processed point image distribution functions, in particular optical transfer functions (OTF), of further partial images (2x to 12xy). The method according to any one of claims 1 to 5.

7. If there are multiple image recordings captured with the same point image distribution function (PSF), the point image distribution function (PSF) of the first partial image (1 to 12) or the processed point image distribution function is stored only once in the buffer memory (18) of the processing unit (17). The method according to claim 6.

8. A reconstruction method for virtually reconstructing a sample volume captured in multiple partial images (1 to 12xy) from different capture angles (Φ) in light field microscopy, wherein the method according to any one of claims 1 to 7 is used to provide a transfer function.

Images