METHOD FOR EVALUATING MEASUREMENT DATA FROM A LIGHT FIELD MICROSCOPE AND DEVICE FOR LIGHT FIELD MICROSCOPY

DE502022002121D1Active Publication Date: 2024-11-28CARL ZEISS MICROSCOPY GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE502022002121
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-09-07
Filing Date
2022-09-05
Publication Date
2024-11-28
Estimated Expiration
2042-09-05

AI Technical Summary

Technical Problem

Current light field microscopy methods struggle to achieve high temporal resolution and good signal-to-noise ratio, especially in dense samples, due to overshadowing of detailed information by background noise and non-linear intensity ratio issues in existing reconstruction algorithms.

Method used

A method and device for evaluating measurement data from a light field microscope that selectively extracts and integrates image signals from specific positions in a sequence of partial images, allowing for high temporal resolution without complete image reconstruction, using a control and evaluation unit to control the light source and detector for improved signal extraction and processing.

Benefits of technology

This approach provides better contrast and significant time savings compared to Shift&Sum methods, while maintaining linear sensitivity and allowing for reliable quantitative evaluation of raw data, achieving high temporal resolution and good signal-to-noise ratio without the need for complete image reconstruction.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

[0001] In a first aspect, the invention relates to a method for evaluating measurement data from a light field microscope. In a second aspect, the invention relates to a device for light field microscopy.

[0002] A generic device for light field microscopy includes at least the following components: a light field microscope with a light source for emitting excitation light, with an illumination beam path with at least one microscope objective for directing the excitation light onto or into a sample, with a two-dimensional spatially resolving detector for detecting light emitted by the sample, with a detection beam path with at least one microlens array for directing light emitted by the sample onto the detector and with a control and evaluation unit for controlling the light source and the detector and for evaluating the measurement data obtained from the detector.

[0003] In biomedical research, processes in organs, tissues, and organisms are increasingly being investigated. For example, the neuronal response to stimuli or the propagation of impulses in muscle tissue are of interest. What is needed here is not primarily structural studies with the highest resolution, but rather the rapid acquisition of three-dimensional image data. A temporal and / or spatial relative change in fluorescent signals, i.e. Δ F / F 0

[0004] Here, F 0 is the intensity measured at a first time or at a first point in space, and ΔF is the change in signal from the first to the second time or from the first to a second point in space. The actual image is only a data source and is generally not used for further analysis. For example, since neuronal structures remain essentially stationary during rapid volume acquisition, the fast temporal information can be assigned to specific regions of the neuronal structure if, for example, an acquisition of the structure is performed at the beginning or at another time of the measurement.

[0005] Light field microscopy (LFM) is increasingly being used for rapid image acquisition. Light field microscopy has the advantage that a volume of several tens to hundreds of micrometers can be reconstructed from a single camera image, which is usually taken within fractions of a second. Various configurations are possible with light field microscopy. In so-called spatial light field microscopy, a microlens array (MLA) is arranged in a plane conjugated to the object plane in the detection beam path. A camera sensor detects in its back focal plane, which is then optically conjugated to the back focal plane of the microscope objective (BFP).

[0006] A more intuitive method for microscopy is Fourier light field microscopy, in which the microlens array (MLA) is arranged in a plane conjugate to the pupil plane of the objective. Detection again occurs in the back focal plane (BFP) of the microlens array. In this case, real images of the sample volume are captured as partial images within the recorded camera image, each of which, however, corresponds to a different parallax viewing direction of the sample. Due to parallax, an imaginary axial displacement of an emitting fluorescent molecule in the object space causes a lateral shift of the signal on the camera sensor, depending on the position of the respective microlens.

[0007] Various algorithms are available for reconstructing the structural volume information from the raw data. A first algorithm first separates all partial image data from the camera image into an image stack, with each partial image of this stack corresponding to a viewing direction of the sample. To calculate the image content of a specific axial sample plane, each partial image of this stack is shifted by the parallactic shift expected for the selected axial plane. The resulting stack is then summed to form an image of a z-plane. To obtain a three-dimensional image, also known as a 3D stack, this process is then repeated for the additional planes, and a three-dimensional image is then composed from images for the individual z-planes.The disadvantage of this method, which is also called Shift&Sum (S&S) method, is that detailed information can be heavily overlaid by background, especially in densely populated samples.

[0008] Another method uses a joint deconvolution approach (jDCV). This requires knowledge of the point distribution function (PSF) for each microlens, and the computation times are significantly longer than with the Shift&Sum method. In return, a significantly higher-contrast image is obtained. The disadvantage of this method, however, is that due to its nonlinear nature, intensity ratios may not be preserved.

[0009] Both the Shift&Sum method and the jDCV method serve to provide the three-dimensional image information and to deliver the best possible structural image of the sample.

[0010] US 2017 / 0205615 A1 relates to a method and apparatus for light field microscopy with enhanced resolution of three-dimensional imaging. In a first approach, the image resolution is increased by modeling the light scattering occurring in an imaging target and using the model in deconvolution of the light field information. In a second approach, the resolution is improved by additionally using a second imaging mode, such as two-photon, multiphoton, or confocal microscopy.

[0011] As a Task The invention can be considered to provide a method and a device that enable the investigation of local dynamic processes with high temporal resolution and a good signal-to-noise ratio.

[0012] This object is achieved by the method having the features of claim 1 and by the device having the features of claim 15.

[0013] Advantageous variants of the method according to the invention and preferred embodiments of the device according to the invention are explained below, in particular in connection with the dependent claims and the figures.

[0014] In the method according to the invention for evaluating measurement data from a light field microscope, the following process steps are carried out: a) at least one sample position to be analyzed is selected in a sample; b) images of the sample are recorded using a light field microscope at a sequence of recording times, each image comprising a set of partial images; c) in the partial images of the images recorded in step b), the positions corresponding to the selected sample positions are determined; d) the image signals are extracted from at least some of the partial images at the positions determined in partial step c); e) an integrated signal is obtained for a specific recording time by integrating the image signals extracted in step d) for a specific position from partial images of this recording time to form the integrated signal; and f) a temporal profile of the integrated signal is obtained by carrying out step e) for a plurality of recording times.

[0015] The device for light field microscopy of the type specified above is further developed according to the invention in that the control and evaluation unit is designed to A) to control the light source and the detector to record images of the sample in a sequence of recording times, wherein the images each comprise a set of partial images; B) to determine positions in the partial images of the images recorded in step A) which correspond to sample positions selected in the sample by a user or by machine; C) to extract the image signals from at least some of the partial images at the positions determined in partial step B); D) to generate an integrated signal for a specific recording time by integrating the image signals extracted in step C) for a specific position from partial images of this recording time and; E) to carry out step D) for a plurality of recording times.

[0016] The method and device according to the invention are fundamentally suitable for any type of sample that is accessible for examination using light field microscopy and in which dynamic processes are to be investigated with high temporal resolution and a good signal-to-noise ratio. The invention is particularly suitable for the investigation of biological samples, in particular physiological processes in living samples. For example, sample positions along a trajectory, such as a nerve pathway, can be observed. It is then possible to investigate, for example, how an excitation propagates along the nerve pathway. Movements of cell particles can also be investigated. Such processes are also referred to as functional traces. The method and device according to the invention are also particularly suitable for the investigation of calcium transients, for example.

[0017] In the method according to the invention, the raw data from the two-dimensional spatially resolving detector are preferably evaluated. This enables high temporal resolution.

[0018] The essential findings of the present invention can be considered that in order to investigate dynamic processes with high temporal resolution and a good signal-to-noise ratio, it is not necessary to completely reconstruct the images of the light field microscope and that it is rather sufficient to evaluate the data of the light field microscope only at positions of interest.

[0019] Because the present invention allows the relevant data to be extracted directly from the raw data, a significant advantage is achieved compared to shift-and-sum methods, and a significant time saving is achieved compared to joint deconvolution methods. Furthermore, it is a clear advantage to start with unprocessed raw data. This can be quantitatively evaluated more reliably. If one remains within the linear sensitivity range of the camera, the changes are linear. This cannot be guaranteed in all cases with processed data.

[0020] The device according to the invention can be configured to carry out the methods according to the invention.

[0021] Light field microscopy can be performed as fluorescence microscopy, and lasers are preferably used as light sources, which are suitable for the fluorescence excitation of the dyes with which the samples to be examined were prepared.

[0022] In principle, well-known cameras such as CMOS, CCD or SPAD cameras can be used as two-dimensional spatially resolving detectors.

[0023] The images acquired with the light field microscope each comprise a set of partial images. The partial images are the images generated by a single microlens of the microlens array on the two-dimensional spatially resolved detector.

[0024] In a particularly advantageous embodiment of the device according to the invention, the microlens array is arranged in a plane optically conjugate to the rear pupil of the microscope objective. The partial images associated with the individual microlenses then correspond to images of the sample from different parallactic angles. In this variant, known as Fourier light-field microscopy, the partial images are clearly understandable. However, the present invention is not limited to Fourier light-field microscopy.

[0025] Rather, the variant referred to as spatial light field microscopy can be used for the method and the device according to the invention, in which the microlens array is arranged in a plane optically conjugated to a plane of the sample.

[0026] There is considerable freedom with regard to selecting the sample positions to be analyzed. For example, the sample position(s) to be analyzed can be selected by a user and / or automatically from a two-dimensional or three-dimensional overview image of the sample. Furthermore, it is possible for the sample position(s) to be selected by a user based on visual observation of the sample with an eyepiece. Alternatively or additionally, the sample position(s) to be analyzed can be selected by a user and / or automatically based on a two-dimensional camera image acquired using an EDOF method (EDOF = Enhanced Depth Of Focus: improved or increased depth of field) or a lens with a large depth of field.

[0027] It is also important that, in the event that the sample position(s) to be analyzed are selected by a user and / or machine-based on images of the sample, these images do not have to be captured or created immediately before carrying out the method according to the invention.

[0028] Rather, the method according to the invention can be carried out as downstream data processing of previously recorded, and therefore existing, image data.

[0029] The analysis according to the method of the invention can therefore be performed either immediately after the image data has been acquired or purely as post-processing. For the generation of the two-dimensional or three-dimensional images, which are used to select the positions to be analyzed, all known methods for capturing two- or three-dimensional structural information from a sample, even if they are comparatively slow, are generally suitable.

[0030] A two- or three-dimensional microscopic overview image of the sample is a microscopic image of the sample to be examined in two or all three spatial directions.

[0031] An overview image of the sample can be obtained, for example, using a laser scanning microscope. Three-dimensional images are typically obtained with a laser scanning microscope as a z-stack, i.e., a set of images of planes of the sample at different positions along the optical axis running in the z-direction.

[0032] In the device according to the invention, a module for laser scanning microscopy can be expediently present for recording the three-dimensional overview image of the sample.

[0033] Alternatively or additionally, a three-dimensional overview image of the sample can be obtained with the light-field microscope, particularly by reconstructing a set of partial images obtained with the light-field microscope. Additional equipment, such as a module for laser scanning microscopy, is then not necessarily required.

[0034] The reconstruction of the three-dimensional overview image can be obtained with a deconvolution algorithm using the point distribution functions of the microlenses of a multi-lens array of the light field microscope.

[0035] Such methods are also known as joint deconvolution methods or jDCV methods. Joint deconvolution methods are comparatively computationally complex, but produce high-contrast images.

[0036] Alternatively or additionally, the reconstruction of the three-dimensional overview image can be obtained by calculating the image content for the individual axial sample planes. The image content of a specific axial sample plane is calculated by shifting each sub-image in the set of sub-images by a parallactic shift expected for the axial sample plane, and summing the set of shifted sub-images. The three-dimensional overview image is composed of the image content for the individual axial sample planes.

[0037] Such methods are called shift-and-sum methods. These methods require less computational effort than joint deconvolution methods, but image quality is often inferior because image details are heavily overlaid by background, especially in densely populated samples.

[0038] The term sample position refers to those locations in the sample that are to be selected at which dynamic processes, such as a neuronal response to stimuli or the propagation of impulses in muscle tissue, are to be recorded. A sample position therefore always includes a specific location in the sample and a microscopic volume that is to be selected at that location either mechanically or by a user, possibly with mechanical assistance. The microscopic volume can have linear dimensions, typically in the single-digit to double-digit micrometer range. For the images to be evaluated, this means that for a specific sample position, a plurality of pixels, namely those pixels that lie in the volume belonging to the sample position, must always be evaluated.

[0039] The step of selecting one or more sample positions to be analyzed refers to the process step in which the three-dimensional overview image is examined with regard to structures of interest and those sample positions at which such structures of interest are located are selected for further investigation.

[0040] The term "sequence of acquisition times" initially means that a number of images are acquired sequentially. The frequency of acquisition times is limited by the equipment. Wherever possible, the frequency of acquisition times is preferably adapted to the temporal structure of the process under investigation, particularly the biological process, such as the excitation of an axon. The time intervals between the individual acquisition times can vary.

[0041] The sample positions in the three-dimensional overview image are uniquely assigned to positions in the partial images of the light field microscope. The assignment rule is determined by the optical parameters of the light field microscope's detection beam path, in particular by the geometric parameters of the multi-lens array used. This assignment rule can be derived, for example, from the parallactic shifts applied in the Shift&Sum method.

[0042] The method step c) of determining those positions in the partial images that correspond to the sample positions selected in the overview image therefore involves finding the respective positions using the assignment rule defined by the optical parameters of the detection beam path.

[0043] In advantageous variants of the method according to the invention, the positions in the partial images of the light field data are calculated which correspond to the sample positions selected in the three-dimensional overview image and to be analyzed.

[0044] In principle, the invention is implemented when the partial images are evaluated at only one position. In preferred variants of the method according to the invention, the images of the sequence of acquisition times are each evaluated at a plurality of positions according to steps a) to f). For example, the partial images can be evaluated at positions along a neural pathway in a living specimen.

[0045] The sample positions to be analyzed can generally be selected by a user in the three-dimensional overview image, for example, using a graphical representation of the three-dimensional overview image as a z-stack on a computer monitor. Alternatively or additionally, in other advantageous variants, the selection of sample positions to be analyzed can also be supported by computer algorithms or performed completely automatically by an evaluation unit.

[0046] For example, sample positions selected by a user can be passed along with a set of partial images to a convolutional neural network, which, in subsequent executions of the procedure, suggests a preselection of sample positions to be analyzed. It is to be expected that an experiment will be repeated many times and that the sample structure will be very similar across the different executions of the experiment. Consequently, predictions about the selection of positions to be analyzed are realistic after only a few tens of runs.

[0047] It can also be provided that the image data of the three-dimensional overview image and / or the images of the light field microscope are analyzed mechanically and that the image data are searched for structures of interest, i.e. sample positions that can potentially be analyzed.

[0048] In further advantageous variants of the method according to the invention, algorithms can be used for image evaluation, which automatically assign image signals to specific structural features.

[0049] The method according to the invention is implemented if, according to method step d), the image signals are extracted from at least some of the partial images at the positions determined in partial step c). If all partial images contain meaningful measurement data, for example, if the structures of interest in none of the partial images overlap or are obscured by other structures, the image signals at the positions determined in partial step c) can preferably also be extracted from all partial images. The measurement data are then optimally utilized, and the signal-to-noise ratio can be increased compared to evaluating only a selection of partial images. Against this background, one will always strive to evaluate as many partial images as possible that contain meaningful image data.

[0050] The partial images from which the image signals are extracted at the positions determined in sub-step c) can in principle be selected by a user.

[0051] For example, for the selection of the partial images from which the image signals are extracted at the positions determined in sub-step c), a set of partial images of an image of the sample taken with the light field microscope can be partially or completely displayed graphically, for example on a computer monitor.

[0052] In addition or alternatively, the selection of those partial images from which the image signals are to be extracted can also be carried out partially, for example assisted, by an evaluation unit or completely automatically by an evaluation unit.

[0053] It is conceivable, for example, that an evaluation unit suggests to a user which partial images should be selected and / or rejected.

[0054] In the simplest case, integrating the extracted image signals into the integrated signal involves summing the respective image signals. The image signals to be integrated can be further weighted with a weighting function that, for example, gives signals from pixels in the center of a volume under consideration, such as an ellipsoid, greater weight than signals from the edge of the volume under consideration.

[0055] A time course of the integrated signal is obtained by performing step e) for a plurality of recording times.

[0056] In preferred variants of the method according to the invention, integration regions for calculating the integrated signals can be graphically defined in a graphical representation, in particular of sections of the partial images. This can be done interactively by a user and, in particular, supported by a segmentation algorithm.

[0057] For example, a user can graphically define an integration region for calculating a signal in the displayed extracted signals by enclosing the region to be integrated, for example with an ellipse.

[0058] It is also possible that the definition of the integration areas for calculating the integrated signals is carried out fully automatically by a segmentation algorithm, in particular not graphically.

[0059] It is also expedient to subtract a minimum value or a starting value from the integrated signals determined at a specific position in step e), in particular the minimum value of the integrated signal obtained for the evaluated images of the sequence of acquisition times or the integrated signal obtained for the first image in time. Alternatively, the integrated signal can be divided by the minimum value or the starting value.

[0060] A temporal course of the integrated signal, also referred to as a signal trace, can be graphically displayed for at least one, preferably for several and in particular for each evaluated position.

[0061] It is also possible to correlate the measurement data of selected pixels in order to achieve a higher signal intensity compared to the noise.

[0062] In the case of living biological samples, artifacts in the image data can occur, for example, due to the heartbeat or breathing of a living animal being examined, such as a mouse.

[0063] With the further developments of the method according to the invention described below, such artifacts can be detected and corrected.

[0064] In a preferred variant of the method according to the invention, a time-varying shift of images of the sample taken at different recording times can be determined by comparing the images taken at different recording times.

[0065] It is particularly advantageous to compare images that are immediately consecutive in time.

[0066] For example, to compare images taken at different times, a spatial correlation can be calculated for the images to be compared.

[0067] In principle, images that deviate significantly, such as those that are significantly shifted, and therefore do not match, can simply be removed from the time series of images being analyzed, so that only matching images are evaluated. However, this may result in wasted measurement data, potentially to a considerable extent.

[0068] In principle, a spatial correlation can also be used to determine the direction and magnitude of a shift in the content of the correlated images. Thus, a shift vector can be determined with which the images can be shifted onto one another, allowing the shifted images to be evaluated together again.

[0069] In such methods, a displacement vector relative to a reference image, for example the last image recorded before the relevant recording time, is determined for at least one recording time, in particular for several and in particular for each recording time, and the set of partial images of the image is corrected by the determined displacement vector before further processing.

[0070] The compensation of shifts can be achieved more quickly if, for the analysis of time-varying shifts of images of the sample taken at different acquisition times, only a selection of favorable partial images, in particular of fewer than ten, preferably fewer than five and particularly preferably two favorable partial images, is evaluated.

[0071] For example, in the case of images taken at different acquisition times, the central partial image can be analyzed in order to determine displacements lateral to the optical axis, and a partial image of a lens lying on an outer ring, in particular at the edge, of the multi-lens array can be analyzed in order to detect axial displacements, in particular displacements parallel to the optical axis.

[0072] Further advantages and features of the present invention are described below with reference to the accompanying figures, in which: Figure 1: a schematic overview of a light field microscope according to the invention; Figure 2: an example of a set of partial images recorded with the light field microscope; and Figure 3: sections from the partial images of the Figure 2 with selected positions.

[0073] An example of a device according to the invention which is designed to carry out the method according to the invention is described with reference to Figure 1explained. The device shown therein includes a light field microscope 100, which has the following essential components: a light source 1, typically one or more lasers, for emitting excitation light 2, an illumination beam path with at least one microscope objective 4 for directing the excitation light 2 onto or into a sample 5, a two-dimensional spatially resolving detector 11 for detecting light 6 emitted by the sample 5, and a detection beam path with at least one microlens array 9 for directing the light 6 emitted by the sample 5 onto the detector 11. The detector 11 is arranged in or near a focal plane of the microlens array 9 and can typically be a CMOS, CCD, or SPAD camera. The microlens array 9 could also be part of the objective 4 and arranged in its rear focal plane.

[0074] Finally, a control and evaluation unit 12 is provided for controlling the light source 1 and the detector 11 and for evaluating the measurement data obtained from the detector 11, which may in particular be a computer of a generally known nature.

[0075] The light 2 emitted by the light source 1, in particular excitation light for fluorescent dyes with which the sample 5 has been prepared, passes through a dichroic beam splitter 3 to the microscope objective 4 and is focused by it into a sample plane on or in the sample 5. Emission light emitted by the sample 5, in particular fluorescent light emitted by fluorescent dyes, passes through the microscope objective 4 back to the dichroic beam splitter 3 and is reflected therein towards a relay optics 8. The relay optics 8 consists of two lenses arranged telescopically relative to one another. At position 7 is an intermediate image plane, i.e., a plane optically conjugate to the sample plane. After passing through the relay optics 8, the emission light reaches a microlens array 9, which is arranged in a plane optically conjugate to the rear focal plane of the microscope objective 4 (objective pupil BFP).The individual lenses of the microlens array 9 generate partial images 201,..,289 on the detector 11 arranged in a focal plane of the microlens array 9 (see . Figure 2 ), each showing individual images of sample 5 from different angles, more precisely: different parallax angles. An image 200 recorded with the light field microscope 100 therefore comprises a set of partial images 201,..,289 (see Figure 2 ).

[0076] At the Figure 1The arrangement shown, with the microlens array 9 arranged in a pupil plane, is a setup for Fourier light field microscopy. Alternatively, so-called spatial light field microscopy would also be possible for the implementation of the invention, in which a microlens array is arranged in a plane optically conjugate to the object plane (instead of the rear focal plane of the microscope objective 4) in the detection beam path. The raw image information obtained with spatial light field microscopy is related to that obtained with Fourier light sheet microscopy via a Fourier transformation. However, the end result of both methods is essentially the same.

[0077] To take a three-dimensional overview image of sample 5, a Figure 1 A module for laser scanning microscopy (not shown) may be present.

[0078] In actual embodiments, the device 100 according to the invention can comprise numerous further optical components, in particular mirrors, lenses, color filters and apertures, the function of which is known per se and which are therefore not specifically described in the present description.

[0079] The control and evaluation unit 12 is designed according to the invention to A) to control the light source 1 and the detector 11 to record images 200 of the sample 5 in a sequence of recording times, B) in the partial images 201,,..,289 (see Figure 2 ) to determine 200 positions a of the images recorded in step A), which correspond to sample positions selected by a user or by machine in the sample, in particular in a previously obtained microscopic overview image of the sample; C) at least some of the partial images p1, p2, p3, p8, p9 (see Figures 2 and 3) extract the image signals at the positions a determined in sub-step B); D) generate an integrated signal for a specific recording time by integrating the image signals extracted in step C) for a specific position a from partial images of this recording time and; E) carry out step D) for a plurality of recording times.

[0080] Variants of methods according to the invention for evaluating measurement data from a light field microscope are described with reference to Figures 1 to 3 , especially the Figures 2 and 3 , described.

[0081] First, in the variant of the method according to the invention described here, a three-dimensional overview image of sample 5 can be obtained. However, as explained above, this is not absolutely necessary. The three-dimensional overview image can be obtained, for example, with a Figure 1not shown laser scanning microscope and / or by reconstructing a set 200 of partial images 201,..,289 obtained with the light field microscope 100.

[0082] The methods described above can be used to reconstruct the three-dimensional overview image from a set of 200 partial images obtained with the light field microscope 100.

[0083] Then, in step a), at least one sample position to be analyzed is selected in the three-dimensional overview image. This selection of the sample positions to be analyzed can be performed in the three-dimensional overview image by a user and / or partially or completely automatically by the control and evaluation unit 12. For example, sample positions selected by a user can be transferred together with a set of partial images 201,...,289 to a convolutional neural network (CNN), which, in subsequent executions of the method, suggests a preselection of sample positions to be analyzed to the user.

[0084] In step b), images 200 of the sample 5 are then recorded with the light field microscope 100 in a sequence of recording times, each comprising a set of partial images 201,...,289.

[0085] Figure 2shows a set of partial images 201-289, which were recorded with a device of the type shown in Figure 1. Overall, the Figure 2 The image 200 shown as an example corresponds to the number of microlenses in the microlens array 9, corresponding to 89 partial images. These partial images are shown in Figure 2 Numbered from top left to bottom right from 201 to 289, with the number indicated for some of the images. The sample examined is the dead brain of a mouse. In principle, living organisms can and should also be examined.

[0086] In the pictures of the Figures 2 and 3 Bright areas mean low light input and dark areas mean high light input to the camera.

[0087] In step c), the positions a corresponding to the sample positions selected in the overview image are determined in the partial images 201,...,289 of the images 200 acquired in step b). Because the optical parameters of the light field microscope are known, this is always possible.

[0088] In step d), the image signals are extracted from at least some of the partial images at the positions a determined in step c), and in step e), an integrated signal is obtained for a specific recording time by integrating the image signals extracted in step d) for a specific position a from partial images of this recording time to form the integrated signal. This is calculated with reference to Figure 3 described.

[0089] For the selection of the partial images from which the image signals are extracted at the positions a determined in step c), a set of partial images 201,..,289 of an image 200 of the sample 5 recorded with the light field microscope 100 can be partially or completely displayed graphically.

[0090] Figure 3 shows sections p1 to p9 from different drawings. Section p9 is a section from drawing 227, which is shown in Figure 2 is also shown (not to scale). Reference symbol a denotes a position to be analyzed. Specifically, it is a nerve cell structure in the brain of a mouse. From the sections p1 to p9, it is clear that the sample appears at a different parallax angle when viewed from the different microlenses.

[0091] In the graphical representation of sections p1, .., p9 of the partial images, a user can graphically define integration regions for calculating the integrated signals. To limit the area in which the signals are to be integrated, a border in the form of a circle or an ellipse is drawn around position a in section p1, which can be selected by a user, for example. Corresponding borders are also drawn in the image sections p2, p3, p8, and p9. Such a border can also be the result of a segmentation algorithm applied to the three-dimensional overview image, allowing potentially interesting sample regions to be interactively selected.

[0092] In sections p1, p2, p3, p8 and p9 it is also evident that there is another structure b in the vicinity of the structure (a neuron) at position a, which, however, is located at a different location relative to position a in the different images depending on the viewing angle (parallax) into the sample. For correct evaluation of the data, it is necessary to count only those signals that actually originate from the structure at position a and not those that originate from the structure (another neuron) at position b. For example, in image section p8, structures a and b are immediately adjacent to one another. However, a separation, as indicated by a border in image section p8, still seems possible. In image sections p4, p5, p6 and p7, however, structures a and b have merged into one another, i.e., structures a and b are located one behind the other in the viewing direction belonging to these image sections.Because separation of the signals corresponding to structures a and b is no longer possible, image sections p4, p5, p6, and p7 are discarded (represented by X in these image sections) and these sub-images are not considered in further analysis. Discarding image sections p4, p5, p6, and p7 increases the contrast of the extracted traces and also avoids erroneous temporal signatures.

[0093] The necessary discarding of image information can be done interactively by the user. Alternatively, information belonging to two overlapping image sections derived from a segmentation algorithm can be discarded automatically.

[0094] However, discarding the data from image sections p4, p5, p6, and p7 is not absolutely necessary. As long as the camera operates in a linear range, it may be possible to separate the signals again using the known ratio of the radiation of both cell regions in the separate partial images. Such separation will only become impossible if shadowing effects from the respective cell structures in front of them become so strong that the ratio of the radiation of two cell regions in separate partial images is no longer maintained.

[0095] In a situation where the position a to be analyzed is free in all partial images and is not overlaid by signals from other structures, no partial images need to be discarded; instead, the signals from all partial images can be integrated. The aim is to make the most optimal use of the available measurement data and therefore only discard those partial images in which the signals cannot be meaningfully separated.

[0096] Finally, in step f), a temporal course of the integrated signal is obtained by carrying out step e) for a plurality of recording times, in particular for all recording times of the series of measurements under consideration.

[0097] In order to obtain comparable data, a minimum value or a starting value can be subtracted from the integrated signals determined for a specific position a in step e) or the integrated signal can be divided by the minimum value or the starting value.

[0098] Finally, the obtained time courses of the integrated signal, also called signal traces, can be graphically displayed for the evaluated positions a.

[0099] The heartbeat and / or respiration of the mouse under investigation can cause temporally varying shifts in images 200 of sample 5 taken at different acquisition times. These shifts must be separated from the actual measurement signal.

[0100] The images 200 captured at different recording times can be compared with each other. In particular, a spatial correlation can be calculated for the images 200 to be compared.

[0101] Raw data acquisition via Fourier light-field microscopy enables the detection of movements without reconstructing the full three-dimensional information. For example, if one assumes that the object's movement occurs lateral to the optical axis of the lens, the entire set of structures in all partial images shifts laterally on the sensor in a correlated and simultaneous manner by approximately the same amount. Such a temporally varying shift in all partial images can be detected relatively easily, for example, by spatially correlating consecutive images. The affected images can then either be removed from the time series or all partial images can be corrected using the determined shift vector before further processing.

[0102] For each recording time, a displacement vector relative to a reference image, for example the last image recorded before the respective recording time, can be determined for the image 200 recorded at this recording time, and the set of partial images of the respective image 200 can be corrected by the determined displacement vector before further processing.

[0103] A faster method for determining the displacement vectors is achieved if, for the analysis of time-varying displacements of images 200 of the sample 5 taken at different recording times, only a few, for example only two, favorable partial images are evaluated. For example, for images 200 taken at different recording times, only the central partial image 245 and a partial image of a lens located at the edge of the multi-lens array 9, in Figure 2 for example one of the partial images 221 or 240, can be analyzed.

[0104] By analyzing the central partial image 245, the displacements lateral to the optical axis are obtained, and the analysis of an outer partial image 221, 240 yields axial displacements, in particular displacements parallel to the optical axis. Axial displacements also lead to displacements of the entire structure lateral to the camera plane, which, however, are different for each subaperture. For a central microlens, i.e., a microlens located on the optical axis, the displacement is zero. The further the microlens in question is from the optical axis, the greater the lateral displacement. Consequently, it is also possible to correlate the corresponding subapertures of two consecutive raw data images. Lateral and axial motion artifacts can thus be automatically detected and corrected.

[0105] The present invention provides a novel method for evaluating measurement data from a light field microscope and a device for light field microscopy, with which significant improvements in the investigation of local dynamic processes with high temporal resolution and a good signal-to-noise ratio are achieved. List of reference symbols and abbreviations used

[0106] 1 Light source (laser) 2 Excitation light 3 Dichroic beam splitter 4 Microscope objective 5 Sample 6 Emission light 7 Intermediate image plane 8 Relay optics 9 Microlens array 10 Fourier plane (optically conjugated to BFP of microscope objective 4) 11 Detector (camera, in image plane) 12 Evaluation unit 100 Light field microscope 200 Image of light field microscope 201 Partial image 227 Partial image 245 Central partial image 289 Partial image a Selected position to be analyzed in partial image b Position p 1 -p adjacent to position a 9 Sections from different partial images X means: Image is not evaluated BFP Back focal plane j DCV Joint deconvolution method LFM Light field microscope LSM Laser scanning microscope MLA Microlens array PSF Point distribution function S&SShift&Sum method

Claims

1. Method for evaluating measurement data from a light field microscope, wherein the following steps are carried out: a) at least one sample position to be analyzed is selected in a sample (5); b) images (200) of the sample (5), which each comprise a set of partial images (201, ..., 289), are recorded at a sequence of recording times using a light field microscope (100); c) the positions (a) corresponding to the selected sample positions are determined in the partial images (201, ..., 289) of the images (200) recorded in step b); d) the image signals are extracted from at least some of the partial images (p1, p2, p3, p8, p9) at the positions (a) determined in partial step c); e) an integrated signal is obtained for a certain recording time by virtue of the image signals extracted for a certain position (a) from partial images of this recording time in step d) being integrated to form the integrated signal; and f) a time profile of the integrated signal is obtained by virtue of step e) being carried out for a plurality of recording times.

2. Method according to claim 1, wherein the at least one sample position to be analyzed is selected by a user from an overview image of the sample (5), in particular a two-dimensional or three-dimensional overview image of the sample, in particular an overview image of the sample that is already available; on the basis of an observation of the sample using an eyepiece, and / or on the basis of a 2D image obtained using an EDOF method or an objective having a large depth of field.

3. Method according to claim 2, wherein the overview image of the sample (5) is obtained using a laser scanning microscope, or wherein the overview image of the sample (5) is obtained using the light field microscope (100), in particular by reconstructing a set (200) of partial images (201, ..., 289) obtained by the light field microscope (100).

4. Method according to any one of the preceding claims, wherein the positions in the partial images of the light field data that correspond to the sample positions which are selected in the three-dimensional overview image and which are intended to be analyzed are calculated.

5. Method according to any one of claims 1 to 4, wherein the images (200) of the sequence of recording times are each evaluated at a multiplicity of positions (a) in accordance with steps a) to f).

6. Method according to any one of claims 1 to 5, wherein the partial images (p1, p2, p3, p8, p9) from which the image signals are extracted at the positions (a) determined in partial step c) are selected by a user and / or automatically by an evaluation unit (12).

7. Method according to any one of claims 1 to 6, wherein a set of partial images (201, ..., 289) of an image (200) of the sample (5) recorded by the light field microscope (100) is partly or completely represented graphically for the selection of partial images (p1, p2, p3, p8, p9) from which the image signals are extracted at the positions (a) determined in partial step c).

8. Method according to claim 7, wherein integration regions for calculating the integrated signals are defined graphically by a user, in particular with assistance by a segmentation algorithm, or fully automatically by a segmentation algorithm in the graphical representation, in particular of portions (p1, ..., p9) of the partial images.

9. Method according to any one of claims 1 to 8, wherein a minimum value or a start value, in particular the minimum value of the integrated signal obtained for the evaluated images (200) from the sequence of recording times or the integrated signal obtained for the first image in time, is subtracted from the integrated signals determined for a certain position (a) in step f), or wherein the integrated signal is divided by the minimum value or the start value.

10. Method according to any one of claims 1 to 9, wherein a time-varying shift of images (200) of the sample (5) recorded at different recording times is determined by virtue of the images (200) recorded at different recording times being compared to one another.

11. Method according to claim 10, wherein a spatial correlation is calculated for the images (200) to be compared for the purposes of the comparison of images (200) recorded at different recording times.

12. Method according to claim 10 or 11, wherein a displacement vector relative to a reference image, for example the last image recorded prior to the relevant recording time, is determined for at least one recording time, in particular for a plurality of recording times and in particular for each recording time, for an image (200) recorded at this recording time, and wherein the set of partial images of the image (200) is corrected using the determined displacement vector prior to the further processing.

13. Method according to any one of claims 1 to 12, wherein a selection of advantageous partial images, in particular less than ten partial images, preferably less than five partial images and particularly preferably two partial images, is evaluated in each case for the analysis of time-varying shifts of images (200) of the sample (5) recorded at different recording times.

14. Method according to any one of claims 1 to 13, wherein the central partial image (245) is in each case analyzed for images (200) recorded at different recording times in order to determine lateral shifts with respect to the optical axis, and wherein a partial image (221, 240) from a lens located on an outer ring of the micro-lens array (9), in particular on the edge, is analyzed in order to detect axial displacements, in particular displacements parallel to the optical axis.

15. Apparatus for light field microscopy, comprising a light field microscope (100) comprising a light source (1) for emitting excitation light (2), an illumination beam path with at least one microscope objective (4) for guiding the excitation light (2) onto or into a sample (5), comprising a two-dimensionally spatially resolving detector (11) for detecting light (6) emitted by the sample (5), comprising a detection beam path with at least one micro-lens array (9) for guiding light (6) emitted by the sample (5) onto the detector (11), and comprising a control and evaluation unit (12) for controlling the light source (1) and the detector (11) and for evaluating the measurement data (200) obtained by the detector (11), wherein the control and evaluation unit (12) is configured A) to control the light source (1) and the detector (11) to record images (200) of the sample (5) in a sequence of recording times, the images (200) each comprising a set of partial images (201, ..., 289); B) to determine positions (a) in the partial images (201, ..., 289) of the images (200) recorded in step A), the positions corresponding to sample positions selected in the sample (5) by a user or by a machine; C) to extract the image signals from at least some of the partial images (p1, p2, p3, p8, p9) at the positions (a) determined in partial step B); D) to generate an integrated signal for a certain recording time by integrating the image signals extracted for a certain position (a) from partial images of this recording time in step C); and E) to carry out step D) for a plurality of recording times.

16. Apparatus according to claim 15, wherein the micro-lens array (9) is arranged in a plane (10) optically conjugate to the back pupil of the microscope objective (4).