Method and reconstruction method for providing transfer function in light field microscopy
By utilizing the symmetry of optical elements to generate symmetric transfer functions, the storage space and time delay problems of storing and processing transfer functions in light field microscopy are solved, achieving efficient three-dimensional reconstruction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-30
- Publication Date
- 2026-03-31
AI Technical Summary
In light field microscopy, the storage and processing of transfer functions require a large amount of storage space and time, resulting in data access delays and making it difficult to effectively provide and reconstruct the volume of three-dimensional samples.
By detecting the symmetry of optical elements, a symmetrical transfer function is generated using computational rules. Only a portion of the original transfer function is stored, and other transfer functions are generated on demand when needed. The high data transfer rate of the buffer memory is used for processing.
It effectively reduces storage requirements, improves data access speed, ensures the efficiency and speed of 3D reconstruction, and is suitable for various data storage configurations.
Smart Images
Figure CN121767543A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for providing a transfer function in light field microscopy. Background Technology
[0002] For example, in the field of microscopy for dynamic systems, it is often desirable to detect the three-dimensional volume of a sample (typically a living sample) in a very short time and with minimal burden on the sample. In recent years, so-called light-field microscopy has proven well-suited for this purpose. In principle, using this microscopy technique, the probe rays from the sample volume are divided into multiple sub-images. Based on the detected sub-images, each containing different angular information about the sample volume, a 3D reconstruction of the sample volume can be performed mathematically, although in the simplest case, the sample in question is detected only once using a two-dimensional detector.
[0003] Each sub-image inherently possesses a specific function as an optical technique characteristic, which describes the optical transfer from the point source of the sample to the detector. This mathematical relationship is known, for example, as the point spread function (PSF). If such transfer functions are ready for subsequent 3D reconstruction, they typically need to be transferred from (main) data memory to a processing unit where the actual 3D reconstruction is performed.
[0004] Because these transfer functions describe complex three-dimensional relationships, they require a significant amount of storage space. When one or more transfer functions are passed to a processing unit, time delays occur in the processing sequence due to the large amount of data and associated access time. Furthermore, the processing unit's memory (cache) is typically designed only for (temporarily) storing relatively small amounts of data. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to propose the possibility of providing even a wide range of transfer functions in an efficient manner.
[0006] This technical problem is solved by a method for providing the transfer function and a reconstruction method in light field microscopy.
[0007] This method is used to provide a transfer function, particularly in light field microscopy. Multiple sub-images from different detection angles are used to detect probe rays from the volume of the sample to be imaged. All or more selected sub-images are intended for subsequent virtual 3D reconstruction of the sample volume. At least based on these selected sub-images, the corresponding point image transfer function (hereinafter referred to as PSF) is determined and stored in correspondence with the sub-images. The PSF can be determined, for example, by measurement, simulation, and / or calculation.
[0008] According to the present invention, the method is characterized by examining the symmetry of the basic optical technology (image) data of the sub-image. Here, symmetry primarily refers to the inherent properties of the sub-image, such as the corresponding PSF. Symmetry exists in the sense of this application when the transfer function of the first sub-image can be converted by a virtual mirror to a transfer function applicable to other detected sub-images. Reflection can occur, for example, on an axis orthogonal to and intersecting with the optical axis of the detected beam of the probe ray. In a further process, taking the x-axis or y-axis of a Cartesian coordinate system as an example, its z-axis coincides with the optical axis, particularly the optical axis of the detected beam path.
[0009] If symmetry is confirmed, it is detected and saved along with the computational rules. The computational rules allow the generation of corresponding PSFs and / or processed PSFs for another sub-image that is symmetrical to the first sub-image (which can also be understood as the original or source image) from the PSF and / or processed PSF (see below). The PSFs and / or processed PSFs of the other sub-image can advantageously be generated only when needed. For example, this might be necessary if a 3D reconstruction of the sample volume is currently underway. Since the PSF or processed PSF can be converted to the desired symmetrical PSF at any time using the PSF of the first sub-image and the associated computational rules, it is advantageous to only temporarily store the PSFs and / or processed PSFs generated using the computational rules.
[0010] The processed PSF in this application is obtained by performing at least one mathematical operation on the corresponding PSF. The result of the mathematical operation in question is saved as an assignment to the associated sub-image. In one embodiment of the method, the processed PSF may be an Optical Transfer Function (OTF). This is generated using a Fourier transform of the PSF. In one embodiment of the method according to the invention, a Discrete Fourier Transform (DFT) is used.
[0011] In this application, the term "transfer function" is understood to refer to a point image transfer function or a processed point image transfer function, such as OTF.
[0012] Symmetrical sub-images, or more precisely, symmetry in the properties of sub-images, may appear in the field of light field microscopy because sub-images are generated by the action of optical elements in the probe beam path. These symmetries can be used in the sense of this invention if the optical elements have symmetry in their structure or if symmetry exists during actual image capture. For example, the microlenses of a microlens array used to generate sub-images can be arranged symmetrically with respect to a virtual axis. If the microlens array is arranged during image recording such that the virtual axis of the beam path coincides with the optical axis, then those microlenses and their respective sub-images can be considered symmetrical, their positions interchangeable by reflection.
[0013] If needed, the computational rules can be additionally or alternatively used to generate symmetrical additional subsets within the image data of at least one sub-image based on a first subset of the transfer function of the sub-image. As will be explained in more detail below, subsets or parts (e.g., half) of the PSF can be generated in this way based on another part (e.g., based on the other half). In this way, the symmetry of the properties of the sub-images, especially the symmetry of the PSF itself, can also be used to still provide all the required PSF and / or processed PSF (which is also referred to below as: OTF), while significantly reducing storage space requirements. For example, if subsets of the OTF can be represented as complex conjugates with opposite imaginary parts, they can be detected as symmetrical to each other. For example, a symmetrical subset of the PSF can be described as a PSF inverted (x, y) = PSF(-x, -y). This corresponds to the relation in 2D Fourier space: OTF inverted (kx, ky) = OTF*(kx, ky).
[0014] Therefore, the core of this invention utilizes the fact that transfer functions emerge during image acquisition using light field microscopy. These transfer functions are assigned to different spatial locations but correspond to each other in other respects. Furthermore, symmetry can appear within the optical technical data of sub-images and can be advantageously used in the process according to the method of the invention.
[0015] This invention allows for the advantageous use of typical technical configurations of different data storage devices. Data storage devices such as hard disk drives, RAM (Random Access Memory) of CPUs (Central Processing Units), and VRAM (Video Random Access Memory) of GPUs (Graphics Processing Units) have large storage capacity but relatively low data access rates, resulting in long data access times. Loading large amounts of data (e.g., data given by a fully computed and stored transfer function) requires a correspondingly long time.
[0016] In contrast, processing units (processors) can access data in buffer memory (cache, on-chip memory, GPU shared memory) much faster, for example, up to 20 times faster (based on internal transfer rates L1 / L2 / L3) due to the higher data transfer rates. However, the storage capacity of such buffer memory is, for example, 200 to 2000 times lower than the aforementioned data memory. For example, a CPU or GPU can also be used as a processing unit.
[0017] This invention now allows for high data transfer rates using buffer memory while still efficiently providing and using the complex transfer functions required for 3D reconstruction.
[0018] In particular, when needed, such as during 3D reconstruction, only the PSF and / or processed PSF of the first sub-image can be stored in the buffer memory, and based on this, additional sub-images with required PSF and / or processed PSF can be generated by the processing unit itself (“on demand”). They do not need to be saved, nor do they need to be loaded from the data memory using a lower data transfer rate.
[0019] For example, the present invention can be advantageously used if multiple image records detected using the same PSF are to be processed. This is, for example, if the same microscope with the same setup and the same optical elements (especially in the detection beam path) is used for all these image records. Then the PSF of the first sub-image and / or the processed PSF can be stored once in the buffer memory of the processing unit and used as the basis for, for example, 3D reconstruction of most of the image records.
[0020] To reconstruct the sample volume, in each embodiment of the method according to the invention, experimentally collected data (i.e., detected measurements and the resulting image data) can be compared with predicted data (the forward model). For a specific sub-image, i.e., an image of the sample at a specific detection angle ("viewing direction"), the forward model can be understood as a 2D projection P of the sample's 3D model M, which is folded with the associated PSF.
[0021] To compute the forward model, the Discrete Fourier Transform (DFT) can be used to transform the PSF:
[0022] (1).
[0023] When the PSF changes only slowly in the z-direction, equation (1) can be approximated as follows:
[0024] (2)
[0025] Here, Mz and PSFz are the various 2D layers of the object model M and their associated PSFs. The operator "·" represents element-wise multiplication.
[0026] Equation (2) can reduce computational complexity and access time (= RAM access count) and support the existing symmetry of using transfer functions.
[0027] In an advantageous embodiment of the method according to the invention, the processed PFS, particularly the OTF, of the detected optical field is stored. Such an OTF can be defined as... This leads to the following equation (3).
[0028] (3).
[0029] The method for providing the transfer function proposed according to the invention can be advantageously used in systems specifically designed for imaging using the concept of light field. The method according to the invention can in particular be used as part of a reconstruction method in which the sample volume detected using multiple sub-images from different detection angles is virtually reconstructed. Attached Figure Description
[0030] The invention will now be explained in more detail with reference to the accompanying drawings and exemplary embodiments. In the drawings:
[0031] Figure 1 A schematic diagram showing the selected symmetry of a microlens array and existing microlenses;
[0032] Figures 2a to 2c A schematic diagram illustrating the symmetry between subsets of point image transfer functions and their processing according to the present invention is shown.
[0033] Figure 3 A schematic diagram illustrating spatial positioning adjustment of focus position based on point image transfer function; and
[0034] Figure 4 A schematic diagram illustrating an exemplary embodiment of light field microscopy. Detailed Implementation
[0035] exist Figure 1 The text explains the first form of possible symmetry in optical technique image data, using the detection of rays as an example. In conventional systems that use light fields for imaging, such as light field microscopes 12 (see...),... Figure 4 The probe rays from the sample volume to be imaged are guided to an optically effective element, which separates the probe rays and then detects them at a subsequent detector 16 (see [reference]). Figure 4 The sample volume is imaged as multiple sub-images. Each sub-image is aligned with the sample volume at a different detection angle.
[0036] Figure 1 A microlens array 15 with a total of 37 microlenses 1 to 12xy is shown. Each of the microlenses 1 to 12xy generates a sub-image; therefore, for simplicity, microlenses 1 to 12xy are equivalent to the associated sub-images 1 to 12xy.
[0037] Microlenses 1-12 are arranged concentrically around the central microlens 1. To perform the method according to the invention, microlens 1 is placed on the optical axis of the probe ray path. If the microlens array 15 is uniformly illuminated, the symmetry of the arrangement of some microlenses relative to each other can be confirmed. In this example, microlenses 1 to 12, arranged in the upper right quadrant, are selected as reference points (first sub-image). Sub-images 1 to 12 are generated by the action of microlenses 1 to 12.
[0038] exist Figure 1 The x-axis and y-axis are illustrated exemplarily, and they are both perpendicular to the optical axis (z-axis) and orthogonal to each other. Microlenses 2 to 12 can be identified by imagining some of their mirror images along the x-axis, y-axis, or one after another along the x-axis and y-axis. It can be assumed that symmetrical microlens groups have the same point image transfer function, which only needs to be evaluated in relation to the corresponding positions of the relevant microlenses in the microlens array 15.
[0039] In this example, the microlens or sub-image generated by reflection along the x-axis can be labeled with the additional marker "x". The same applies to reflections along the y-axis or both x and y axes. Microlens 1 cannot reflect along either the x or y axis due to its position at the origin of the coordinate system (= optical axis).
[0040] The current symmetry can be explained using the example of microlens 3. Imagine that microlens 3, existing in the upper right quadrant, can be mirrored on the x-axis. There exists a microlens here called 3x. Correspondingly, by mirroring microlens 3 on the y-axis, we can find the corresponding microlens 3y. If the latter is mirrored on the x-axis, we obtain microlens 3xy. Correspondingly, microlens 3x can also be mirrored on the y-axis.
[0041] In this sense, mutually symmetrical microlenses or sub-images are detected and saved.
[0042] For the microlens 3, the relevant point image transfer function (PSF) is determined and stored. The processed PSF of the microlens 3, particularly the optical transfer function (OTF), can be calculated through the Fourier transform of the PSF. To efficiently provide PSFs or OTFs for microlenses 3x, 3y, and 3xy in the sense of this invention, the PSFs and / or OTFs of the microlenses 3 are supplemented with corresponding calculation rules for each of the microlenses 3x, 3y, and 3xy. For example, to generate the PSF of microlens 3xy, the already stored PSF of microlens 3 is simply called, and the relative position and detection angle of microlens 3xy are considered according to calculation rules (e.g., by calculating the complex conjugate OTF). This calculation can be performed, for example, in processing unit 17, and the PSF generated in this way can be stored in buffer memory 18. The same applies to the generation and provision of corresponding processed PSFs (e.g., OTFs). Other microlenses or sub-images with confirmed symmetrical correspondences can be processed similarly.
[0043] In addition to the symmetry of the transfer function, the fact that the transfer function (e.g., the point image transfer function PSF) is designed to be mirror symmetric, for example, along its longitudinal range. Figure 2aAn example of a PSF is shown in an xz-plane view. The PSF is tilted at an angle relative to the x-axis and z-axis. The spatial position and orientation of the PSF describe the current viewing direction (detection angle Ф; as shown in the figure).
[0044] For example, to save storage space, you can save only half of the PSF (PSF / 2) and reserve it for later use. Figure 2b In order to be able to supplement the missing second half of PSF / 2 when necessary. invert In addition to the PSF / 2 data, a rule is also stored. When this rule is executed, the second half of the PSF / 2 can be quickly calculated based on the first half of the PSF / 2 data. invert And then generate a complete PSF again.
[0045] To calculate the other half of the PSF of the 2D projection P using equations (1) and (2) given above, a mirrored PSF can be generated from half of the stored original PSF, PSF / 2 (see above). If OTF / 2 is to be added, the light field OTF of equation (3) can be used. LF OTF. In this case, OTF can be used. LF The half of the OTF to be supplemented is calculated by using the complex conjugates in the x, y, and z directions.
[0046] In some Figure 2c In the diagram, the lower half of the PSF / 2 supplemented using this rule is shown as PSF / 2 by different types of shading. invert .
[0047] In a real light field imaging system, the focal position of each microlens can be moved along the observation direction and / or the detection angle Ф. This offset may vary from microlens to microlens and also depends on the wavelength of the probe rays.
[0048] Figure 3 This situation is illustrated by example. As described above, the first subset of the PSF is again passed to the buffer memory 18 of the processing unit 17. In this case, it is necessary to transfer data from a subset of the PSF slightly more than half of the PSF in the z direction (hereinafter referred to as: PSF / 2 for simplicity). When calculating the PSF / 2 subset to be supplemented, it shifts by an offset vector, which is of the form: This corresponds to the offset of the focal position in the x, y, and z directions (focal offset). The focal offset is determined by the offset of the actual focal position in the x, y, and z directions relative to the theoretical focal position.
[0049] The focal offset associated with the PSF center is transferred to the entire relevant z-plane of PSF / 2, thus moving the PSF in the z-direction (z-offset). Furthermore, the entire PSF moves accordingly in the x and / or y directions. This can be accomplished either by setting a fixed xy offset (xy offset) individually for each microlens 1 to 12xy, or by using the determined xy offset of the PSF.
[0050] Additionally, due to the shift in the focus position, correspondingly altered target regions (regions of interest, ROI, not shown) are defined in the x, y, and z directions, and PSF will be applied to these target regions.
[0051] As explained in Figure 2, the optical field OTF according to equation (3) LF The optical transfer function is preferably used for calculation. Again, it is used in conjunction with the optical field OTF. LF The complex conjugate of the opposite imaginary part of the optical transfer function is used to calculate the supplemented PSF or a subset of the processed PSF. Multiple 2D phase operators are used based on the unshifted OTF. LF Generate offsets in the x and / or y directions. The z-direction offset (z-offset) is calculated by adding the corresponding z-offset to the offset vector, and then calculating the OTF. LF Regions of interest (z-ROI) in the z-direction of two subsets. The computation is performed by means of processing unit 17 as backward projections, rather than calculating the full PSF or the full processed PSF in the corresponding intermediate steps.
[0052] The method according to the invention can be applied, for example, to a light field microscope 20 ( Figure 4 The detection beam is implemented in the detection beam path of the detector 16. The detection beam from the sample volume to be imaged is collected by objective lens 13, guided along the detection beam path by transmission optics 14 (shown in a very simplified manner), and directed onto an optical element. This optical element generates multiple sub-images that detect the sample volume from different viewing directions on detector 16. In an exemplary embodiment, the optical element is designed as a microlens array 15 having multiple microlenses 1 to 12xy (see...). Figure 1 ).
[0053] The measurements detected by detector 16 are transmitted to processing unit 17. This is configured to receive data from data memory 19 regarding the transfer functions of the selected microlenses 1 to 12xy, as well as information about their existing symmetries with respect to the generated sub-images and information about symmetries occurring with respect to the respective transfer functions themselves. Information about the selected transfer functions and any calculation rules are used to calculate the transfer functions using processing unit 17 when necessary and store them in buffer memory 18 of processing unit 17, particularly for repeated retrieval.
[0054] Figure Labels
[0055] 1-12xy microlenses / partial imaging
[0056] 13 Objective Lenses
[0057] 14 Transmission Optical Components
[0058] 15 microlens array
[0059] 16 detectors
[0060] 17 processing units
[0061] 18 Buffer Memory
[0062] 19. Data storage (permanent, RAM)
[0063] 20. Microscopes, Light Field Microscopes
[0064] OTF-treated PSF, optical transfer function
[0065] PSF Point Image Transfer Function
[0066] Ф detection angle
Claims
1. A method for providing a transfer function in light field microscopy. in, -Use multiple sub-images (1 to 12xy) from different detection angles (Ф) to determine the sample volume to be imaged; - Determine the appropriate point image transfer function (PSF) for the sub-images (1 to 12xy) for 3D reconstruction of the sample volume, and assign the point image transfer function to the sub-images (1 to 12xy). Its features are, - Check the symmetry of the basic optical technical data of the sub-images (1 to 12xy); - Confirmed symmetries are detected and saved along with computational rules, wherein the computational rules allow: • Using the point image transfer function (PSF) and / or processed point image transfer function of the first sub-image (1 to 12), generate point image transfer functions (PSF) and / or processed point image transfer functions for another sub-image (2x to 12xy) symmetrical to the first sub-image (1 to 12), wherein the point image transfer functions (PSF) and / or processed point image transfer functions of the other sub-image (2x to 12xy) are generated as transfer functions only when necessary and are temporarily stored; wherein the processed point image transfer function is obtained by performing at least one mathematical operation on the corresponding point image transfer function (PSF). and / or • If necessary, generate another symmetric subset within the transfer function of at least one sub-image (1 to 12xy) based on the first subset of the transfer function of the sub-image (1 to 12xy).
2. The method according to claim 1, characterized in that, Sub-images (2 to 12xy) that can be transformed into each other by reflection on at least one mirror axis are detected as symmetrical, wherein the at least one mirror axis extends orthogonally to and intersects the optical axis of the detected radiation beam of the probe ray.
3. The method according to any one of the preceding claims, characterized in that, The processed point image transfer function is the optical transfer function (OTF).
4. The method according to any one of claims 1 to 3, characterized in that, A subset of a point image transfer function (PSF) is considered to be symmetric to each other if it can be represented in the form PSFinverted(x, y) = PSF(-x, -y).
5. The method according to claim 3, characterized in that, When a subset of an optical transfer function (OTF) can be expressed in complex conjugates with opposite imaginary parts, these subsets are confirmed to be symmetric to each other.
6. The method according to any one of the preceding claims, characterized in that, During the 3D reconstruction process, only the point image transfer function (PSF) and / or the processed point image transfer function of the first sub-image (1 to 12) are stored in the buffer memory (18) of the processing unit (17) that performs the 3D reconstruction, and on this basis, the processing unit (17) generates the required processed point image transfer function, in particular the optical transfer function (OTF), for the other sub-image (2x to 12xy).
7. The method according to claim 6, characterized in that, If there are multiple image records detected using the same point image transfer function (PSF), the point image transfer function (PSF) of the first sub-image (1 to 12) and / or the processed point image transfer function are stored once in the buffer memory (18) of the processing unit (17).
8. A reconstruction method for virtually reconstructing, in light field microscopy, a sample volume detected using multiple sub-images (1 to 12xy) from different detection angles (Ф), wherein, The transfer function is provided using the method according to any one of the preceding claims.