Data processing apparatus and method for processing digital images with photon arrival time data

Abstract

The present invention uses the input image position ( x i ) input photon arrival time data (P i ( x i The data processing device (100) is configured to calculate a digital output image (O) based on the digital input image (I) having output image positions ( x o ) output photon arrival time data (P o ( x o , t)) and output photon arrival time data (P o ( x o ,t) is the input photon arrival time data (P i ( x i ,t)) (m) ( x i ,t)). The deconvolution (102) is calculated by an iterative algorithm (104) using an update function (U). The update function (U) is a function of the point spread function (psf( x i [,t]) and the ground truth (I GT ( x i ,t)) (m-1) ( x i ,t) and the input photon arrival time data (P i ( x i ,t)) and depends on.

Description

[Technical Field]

[0001] The present invention relates to a data processing apparatus and method for processing a digital input image, the digital input image comprising input photon arrival time data at image locations. The invention further relates to a fluorescence lifetime imaging device comprising such a data processing apparatus. [Background technology]

[0002] Processing of photon arrival time data is known, for example, from fluorescence lifetime imaging microscopy (FLIM), in which at each image location, i.e. at each pixel or voxel, an arrival time histogram of detected photons is stored.

[0003] The main goal of FLIM is to estimate the lifetime, or decay time, of fluorescence from the arrival time of photons. Fluorescence is emitted from a fluorophore illuminated by light with a fluorescence trigger wavelength band. The lifetime is indicative of the chemical state of the fluorophore, among other things, and therefore conclusions can be drawn about the chemical environment of the fluorophore.

[0004] Photon arrival time data are usually very noisy, and the detection system inevitably introduces spatial blur into the recorded images, so images calculated from the photon arrival time data tend to be blurry.

[0005] Unlike typical image data, which has one or more time-independent intensity values ​​at each image location, standard deconvolution algorithms, such as Wiener deconvolution, cannot be applied to sharpen images containing a time series of intensity values ​​(when the intensity values ​​are considered to be represented by photon counts).

[0006] However, it is known to apply deconvolution to arrival time data, for example from O'Connor, DV, Ware, WR and Andre, JC, "Deconvolution of Fluorescence Decay Curves. A Critical Comparison of Techniques", Journal of Physical Chemistry, 83(10), 1333-1343, 1979. However, this deconvolution only removes blurring from the arrival time data; it does not improve spatial resolution.

[0007] It is also known to apply multi-image deconvolution to digital input images with arrival time data. This is described, for example, in Castello, M., Diaspro, A., and Vicidomini, G., "Multi-images Deconvolution Improves Signal-to-Noise Ratio on Gated Stimulation Emission Depletion Microscopy," Applied Physics Letters, 105(23): 234106, 2014. However, this approach is unsatisfactory in that it results in digital output images in which information about photon arrival times is lost. Therefore, analyses such as FLIM are no longer possible. Furthermore, multi-image deconvolution has other drawbacks. For example, it is highly affected by noise. Multi-image deconvolution is also computationally inefficient because it requires separate deconvolution for every time step. Finally, multi-image deconvolution does not provide a solution when there are more than one fluorophore in the input image data, and therefore more than one fluorescence emission spectrum. Summary of the Invention [Problem to be solved by the invention]

[0008] Therefore, there is a strong need to provide an apparatus and method that can deblur digital images containing photon arrival time data, that is computationally efficient, and that results in preservation of the arrival time information. [Means for solving the problem]

[0009] This need is met by a data processing apparatus for processing a digital input image, the data processing apparatus configured to calculate a digital output image based on the digital input image by deconvolution, the digital output image having output photon arrival time data at output image locations, the output photon arrival time data representing an unblurred ground truth estimate of the input photon arrival time data, the data processing apparatus further configured to calculate the deconvolution by an iterative algorithm using an update function, the update function depending on a point spread function, a prior estimate of the ground truth, and the input photon arrival time data.

[0010] The above needs are also met by a computer-implemented method of processing a digital input image, the digital input image having input photon arrival time data at input image locations, computing a digital output image from the digital input image by deconvolution, the digital output image having output photon arrival time data at output image locations, the output photon arrival time data representing an unblurred ground truth estimate of the input photon arrival time data, and computing the deconvolution by an iterative algorithm using an update function, the update function depending on a point spread function, a prior estimate of the ground truth, and the input photon arrival time data.

[0011] The above needs are also met by a fluorescence lifetime imaging apparatus having a data processing apparatus as described above, and by a computer program product and / or computer readable medium having instructions that, when executed by a computer, cause the computer to perform a data processing method.

[0012] Further, the above needs are met by a machine learning product or apparatus for processing a digital input image, the digital input image having input image data at an input image location, the input image data having input photon arrival time data, the machine learning product configured to calculate a digital output image, the digital output image having output image data at an output image location, the digital output image data having output photon arrival time data at the output image location, the machine learning product being trained with different digital input image and digital output image pairs, each digital output image of the pair calculated from the pair's digital input image using the digital output signal processing method described above.

[0013] Finally, the above needs are also addressed by a method for training a machine learning product using the above-described data processing method.

[0014] The above solution results in a digital output image containing photon arrival time data. Therefore, the above solution is fully compatible with FLIM-related analyses. The iterative algorithm and update function allow for computationally efficient algorithms. Finally, the ground truth, i.e., the "real" picture free of noise and distortion, estimates both the photon arrival time data and the spatial location, improving the spatial resolution and signal-to-noise ratio of the photon arrival time data.

[0015] The above-described solution can be further improved by the following features, which can be combined independently of one another: In this context, the features described below with respect to an apparatus or device can also be used to improve the method, and vice versa.

[0016] For example, the digital input image may be a line image or one-dimensional image, a two-dimensional image, a three-dimensional image or a higher dimensional image.

[0017] The digital output image preferably has the same number of dimensions as the digital input image, so that no information is lost. In particular, the digital input image and the digital output image may have the same number of image locations. Preferably, the image locations of the digital input image and the digital output image correspond to one another. Any of these features ensures that the information contained in the digital output image can be correlated with one another. Depending on whether the digital input image and / or the output image represent surface data or volume data, the input image locations and / or the output image locations may be one of pixels and voxels.

[0018] At least one, preferably at least two of the input and / or output image dimensions may be spatial dimensions.

[0019] In general, the input photon arrival time data may include photon arrival time data for one or more fluorophores.

[0020] In another embodiment, the input photon arrival time data can be dye-separable. For example, at each image location, there can be two or more sets of photon arrival time data, each set representing photons of a different fluorophore, i.e., a different fluorescence band. It should be noted that these sets can be combined into a single digital input image and / or digital output image, representing additional dimensions of the respective digital input image and / or digital output image. For example, a digital output image has three spatial dimensions, X, Y, and Z, and at each image location, two sets of photon arrival time data can be considered a five-dimensional digital output image.

[0021] Furthermore, although it is preferred that the photon arrival time data be included at each input image location, even if the data is an empty set or NULL, it is not essential.

[0022] In another embodiment, the digital input image is or has been recorded by an optical device, in particular by a (confocal) laser scanning microscope.

[0023] The digital input image may be or may have been recorded by a photon counting device, such as a time-correlated photon counting device and / or an assembly having a gated light intensifier and a CCD sensor or camera, etc. More generally, any device having input image data and for generating a digital input image, itself having input photon arrival time data at input image locations, may be used.

[0024] Optical devices and / or photon-counting devices can be characterized by a point spread function, which can be time-invariant or time-dependent. The same applies to the update function.

[0025] According to another embodiment, input photon arrival time data at an input image location may be different from output photon arrival time data at an output image location corresponding to the input image location. In particular, output photon arrival time data at a particular time point and output image location may result from superposition of input photon arrival time data at this particular time point from at least one different input image location. In particular, input photon arrival time data at a particular arrival time may be shifted to a different location in the output image data by an iterative algorithm. This can occur because an update function assigns these arrival time data to different image locations to minimize some error constraint or cost function. In other words, the iterative algorithm or update function may shift "stray" photons captured at one input image location to an output image location that is deemed more appropriate.

[0026] Hence, the update function can represent a redistribution of the photon arrival time data according to the point spread function.

[0027] In general, deconvolution, especially iterative algorithms, can assume one of the following mathematical forms: E (m) ( x i ,t)= E (m-1) ( x i ,t)·U(E (m-1) ( x i ,t),psf( x i [,t]),P i ( x i ,t)) or E (m) ( x i ,t)= E (m-1) ( x i ,t)+U(E (m-1) ( x i ,t),psf( x i [,t]),P i ( x i ,t)) where, x i represents a set of input image locations, typically coordinates and / or array indices, t represents time, and psf( x i [,t]) represents the point spread function, which may be time dependent as symbolically represented by the square brackets, and P i represents the input photon arrival time data, and E (m-1) represents the estimate of the ground truth at iteration step m-1, and U is the update function.

[0028] Thus, the iterative algorithm may comprise a combination of an update function and a priori estimates of the unblurred ground truth of the input photon arrival time data, the combination being at least one of additive and multiplicative.

[0029] However, it is preferred that at least one of the point spread function, the ground truth estimate, and the update function be time-independent. To this end, these functions need not themselves be time-invariant. For example, deconvolution can include an operation on at least one of the input image data, the point spread function, and a prior estimate of unblurred ground truth, the operation being configured to remove the time dependency of that operand.

[0030] Hence, a time-dependent point spread function as an operand of an operation T can be made time-independent by this operation as follows: psf( x i )=T(psf( x i ,t)) Of course, if a time-invariant point spread function is used, then the operation T does not need to be applied.

[0031] The operation T can provide a time-independent ground truth estimate by using it as an operand as follows: E (m-1) ( x i )=T(E (m-1) ( x i ,t)) is.

[0032] Input photon arrival time data as operands of operation T can be made time independent by applying operation T in a similar manner, i.e. P i ( x i )=T(P i ( x i ,t)) is.

[0033] For each of the above examples, different operations T can be used. For example, operation T can include statistical moments, sums over time, and / or values ​​of operands of operation T at a particular point in time t1. Examples of operations T and their applications are provided below. Any of these examples can be used in any combination.

number

[0034] If the operation T represents a statistical moment, then this is

number

[0035] If the operation T involves summing input photon arrival time data and / or ground truth estimates, the signal-to-noise ratio is significantly reduced.

[0036] By using the operation T and / or a time-invariant point spread function, we can obtain a time-independent update function as follows: E (m) ( x i ,t)= E (m-1) ( x i ,t)+U(T(E (m-1) ( x i ,t)),psf( x i ),T(P i (x i ,t))) or E (m) ( x i ,t)= E (m-1) ( x i ,t)·U(T(E (m-1) ( x i ,t)),psf( x i ),T(P i ( x i ,t))) or E (m) ( x i ,t)= E (m-1) ( x i ,t)+U(T(E (m-1) ( x i ,t)),T(psf( x i ,t)),T(P i ( x i ,t))) or E (m) ( x i ,t)= E (m-1) ( x i ,t)·U(T(E (m-1) ( x i ,t)),T(psf( x i ,t)),T(P i ( x i ,t))) is.

[0037] The above equation can be physically interpreted as saying that the photon count at a particular time is maintained at that particular time across all iterations. However, depending on the update function, the location of that particular photon count can change. Thus, the above update function "redistributes" the photon counts across space.

[0038] This form of iterative algorithm produces a ground truth estimate E (m) is time-dependent, i.e., the update function U is time-independent, the ground truth estimate E(m) still contains the arrival time data at each image location, which allows for FLIM-related analysis of the results.

[0039] Another advantage is that the use of time-invariant updates allows the use of known and efficient deconvolution algorithms, and therefore no special solutions are required. For example, the Richardson-Lucy deconvolution can be used to calculate the update function. In this case, the update function has the following form:

number

number

[0040] The iterative algorithm terminates, for example, after the Mth iteration if a convergence criterion is met, where m=1,...,M. Possible convergence criteria are:

number

[0041] If the convergence criterion is met, the ground truth estimate at the Mth iteration is the output photon arrival time data P o ( x i ,t), i.e., P o ( x o ,t)=E (m) ( x i ,t).

[0042] In principle, the deconvolution described above can be used regardless of the number of fluorophores represented in the input photon arrival time data at one input image location.

[0043] However, if the photon arrival times of more than one fluorophore are included in the input photon arrival time data, it may be advisable to separate the different fluorescence photons before performing the deconvolution.

[0044] Thus, in one embodiment, the input photon arrival time data P i ( x i , t) is the input photon arrival time data P i (f) ( x i ,t) (where f=1,…,F, and the number F is greater than 1), for example,

number

[0045] In such a case, the input photon arrival time data P i ( x i , t) are the different input photon arrival time data P i (f) ( x i, t). The resulting input photon arrival time data P i (f) ( x i , t) will be referred to as dye-separated input photon arrival time data hereafter.

[0046] Preferably, the combined input photon arrival time data is separated into dye-separated input photon arrival time data, the number of dye-separated input photon arrival time data in the set corresponding to the number of different fluorophores. Each of the different dye-separated input photon arrival time data represents the fluorescence evolution over time of a different fluorophore. In principle, it is also possible to calculate the dye-separated photon arrival time data after deconvolution, but better results are obtained if deconvolution is performed separately for each of the dye-separated input photon arrival time data, as this allows different point spread functions to be applied for the different fluorophores.

[0047] The result of the dye separation of the input photon arrival time data is the input image I( x i , t) are F color-separated input images I (f) ( x i , t) (where f=1,...,F), and the fth dye-separated input image is divided into the dye-separated input photon arrival time data P i (f) ( x i ,t).

[0048] The separation of the input photon arrival time data into a plurality of dye-separated input photon arrival time data can be performed, for example, according to at least one predetermined dye decay curve, which is different for each of the different fluorophores.

[0049] For example, the dye decay curve for one particular fluorophore f has the following form:

number

[0050] As a first step, the lifetimes τ of different fluorophores f can be estimated from the digital input image. In this step, the lifetime can be estimated by using input photon arrival time data at multiple input image locations, in particular at all input image locations of the digital input image.

[0051] For example, aggregate input photon arrival time data can be calculated from a combination of input photon arrival time data from multiple input image locations, for example, by combining these input photon arrival time data additively and / or multiplicatively.

[0052] Such aggregated input photon arrival time data P Σ An example of calculating (t) is as follows:

number

[0053] From the aggregated input photon arrival time data, dye decay curves can be calculated for each number of different fluorophores. Such calculations may include, for example, fitting a preferably linear combination of predetermined decay curves to the aggregated input photon arrival time data. The fitting may, for example, be performed using a minimization scheme for each fluorophore f, e.g., in the form:

number

[0054] As a result of this minimization, the weighting coefficient a for the fth fluorophore is fand lifetime τ f An estimate of the parameter

number

number

[0055] The above minimization can be solved using least-squares fitting, which may involve Gauss-Newton and / or Levenberg-Marquardt algorithms. An example of Levenberg-Marquardt is given in Marquardt, D., W., "An Algorithm for Least-Squares Estimation of Nonlinear Parameters," Journal of the Society for Industrial and Applied Mathematics, 11(2), 431-441, 1963.

[0056] In the next step, the aggregate dye lifetimes for the different fluorophores calculated from the aggregate input photon arrival time data are used to generate the dye-separated photon arrival time data P at each input image location. i (f) ( x i ,t) can be calculated.

[0057] This step involves generating, at each input image location, or at least one input image location, the combined input photon arrival time data P i ( x i , t), and the lifetime of each dye, τ f The fth dye decay curve D contains (f) (t;τ f) to determine the different dye lifetimes. In other words, the steps initially performed on the data aggregation, i.e., using the input photon arrival time data for multiple input image locations, to determine the different dye lifetimes are now repeated at each input image location and fitting these lifetimes to the input photon arrival time data present at that location.

[0058] The fitting of the dye decay curve at an input image location to the photon arrival time data at this input image location can be calculated by solving a minimization problem of the following form:

number

number

[0059] Again, this minimization can be solved by using a least-squares fitting algorithm, such as the Gauss-Newton or Levenberg-Marquardt algorithm.

[0060] An alternative solution to the above minimization problem can be analytically derived in the following form:

number

[0061] The combined input photon arrival time data P i ( x i,t) at each input image position x i Dye decay curve D for each fluorophore f in (f) (t;τ f ) at each input image location, resulting in f dye-separated input photon arrival time data P i (f) ( x i , t) is generated. A set of dye-separated input photon arrival time data is calculated at a plurality of input image locations, preferably at all input image locations.

number

[0062] Of course, it is not necessary to calculate dye lifetimes if predetermined dye decay curves are already available, e.g., from a library of dye decay curves and / or lifetimes for different fluorophores. In this case, it may not be necessary to calculate the lifetime of each different fluorophore, and instead, for example, the predetermined dye decay curves and / or lifetimes may be used to calculate the lifetime for each input image position. x i Using the above minimization problem in (f) ( x i ) can proceed directly to the calculation of the predetermined dye decay curves and / or lifetimes can be stored in the memory of the data processing device.

[0063] When deconvolution is performed as described above, each dye-separated input image I (f) ( x i , t) is the input photon arrival time data P i (f) ( x i , t) can be used for deconvolution as separate input images.

[0064] Therefore, after deconvolution, the input photon arrival time data P i (f) ( x i , t) (f) ( x i , t), the output photon arrival time data P o (f) ( x o , t) (f) ( x o ,t) is obtained.

[0065] From the dye-separated output photon arrival time data, a single output photon arrival time data can be calculated as follows:

number

[0066] Equivalently, the dye separation output image O (f) can be added up to arrive at a single output image, i.e.

number

[0067] In this case, the output photon arrival time data P o ( x i , t) is the output image data O( x o , t), which may be part of x i but x o otherwise, spatial interpolation may be involved when the input image positions do not correspond to the output image positions.

[0068] A fluorescence lifetime imaging device including the data processing apparatus of any of the above-described embodiments and / or configured to perform the data processing method of any of the above-described embodiments can have a light source, such as a pulsed laser. The fluorescence lifetime imaging device may further include illumination optics for guiding light from the light source to the sample. The light source is configured to emit light at the fluorescence excitation frequency of at least one fluorophore. When multiple fluorophores are present, i.e., when two or more fluorophores are present, the light source is preferably configured to emit light at the fluorescence excitation frequency of each of the multiple fluorophores.

[0069] The fluorescence lifetime imaging device may further comprise a photon counting unit configured to record input photon arrival time data, which can be recorded as combined input photon arrival time data, where photons from different fluorophores are superimposed at an image location, or alternatively, the input photon arrival time data can be recorded as dye-separated input photon arrival time data.

[0070] The fluorescence lifetime imaging device may include detection optics for directing the emitted fluorescence to a photon counting device.

[0071] The detection optics may include fluorescence separation filter elements capable of separating photons from different fluorophores and recording dye-separated input photon arrival time data.

[0072] The fluorescence lifetime imaging device can have a stage that accommodates the sample and is configured to generate relative motion between the at least one light source and the photon counting device on the one hand, and between the at least one light source and the sample on the other hand, such that input photon arrival time data can be collected at different input image locations.

[0073] Preferably, the fluorescence lifetime imaging device is a confocal laser scanning microscope, which may have at least one pulsed light source or a supercontinuum laser light source (emitting pulsed laser light) and suitable selection means for selecting at least one emission light wavelength band for exciting at least one fluorophore in the sample, and further has a spectral detector for separately spectrally detecting the fluorescence emitted by the at least two excited fluorophores by at least two photon counting devices.

[0074] It should be noted that the above features, whether described in the context of a method or in the context of an apparatus, may apply independently to both the method and the apparatus configured to perform the method.

[0075]

[0023] In the following, one embodiment will be described by way of example with reference to the drawings. According to the above description, the features described in this embodiment can be omitted if their technical effects are not required or are disadvantageous in a specific application. Conversely, features not included in the embodiment described below but included in the above-mentioned specification can be added if their specific effects are necessary or beneficial for a specific application.

[0076] In the drawings, elements that correspond to one another in terms of function and / or structure have the same reference numerals. [Brief explanation of the drawings]

[0077] [Figure 1] FIG. 1 is a schematic diagram of a fluorescence lifetime imaging device and a data processing unit. [Figure 2] FIG. 1 is a schematic diagram of separating combined photon arrival time data into dye-separated photon arrival time data. [Figure 3] FIG. 1 is a schematic diagram of an iterative algorithm for calculating deconvolution using photon arrival time data. [Figure 4] FIG. 2 is a schematic diagram of a digital output image. [Figure 5] FIG. 1 is a schematic diagram of a technique for deblurring a digital input image with photon arrival time data. [Figure 6] Figure 6a is a diagram showing an example of a digital input image at a particular time point, Figure 6b is a diagram showing an example of a digital output image at a particular time point of the digital input image of Figure 6a, Figure 6c is a schematic diagram of photon count values ​​for both the ground truth, the digital input image and the digital output image along the line of the position shown in Figure 6a, and Figure 6d is a schematic diagram of arrival time histograms of the digital input image, the digital output image and the ground truth at the position shown in Figure 6a. [Figure 7] FIG. 7a is a schematic diagram of a phasor plot of the photon arrival time data in the digital input image of FIG. 6a, and FIG. 7b is a schematic diagram of a phasor plot of the photon arrival time data in the output image of FIG. 6b. [Figure 8] FIG. 10 is another schematic diagram of the data processing unit and the fluorescence lifetime imaging device. DETAILED DESCRIPTION OF THE INVENTION

[0078] Referring first to FIG. 1, the structure and function of a fluorescence lifetime imaging device 128, a data processing apparatus 100 which may be part of a microscope, for example, will be described.

[0079] A sample 106 , such as a cell or a portion of a cell, is illuminated by a light source 114 , such as a laser 134 or a pulsed laser 136 .

[0080] The sample 106 may include one or more phosphors 108. The total number F of phosphors 108 is 1 or greater.

[0081] Preferably, light source 114 emits light at a wavelength that triggers fluorescence of one or more fluorophores 108 .

[0082] The fluorescence lifetime imaging device 128 may further include illumination optics 116 that focus a light beam 118 from the light source 114 onto the sample 106. The sample 106 and the light beam 118 or light source 114 are movable relative to one another, as indicated by arrow 138. The light beam 118 or light pulse 120 is focused onto a position on the sample 106. x Fluorescence lifetime imaging device 128 may further include detection optics 110.

[0083] The photon counting device 112 may be fixedly coupled to the light source 114 so that when there is relative motion between the light source 114 and the sample 106, the photon counting device 112 maintains its relative position with respect to the light source 114. This ensures that the photon counting device 112 always points to the position where the light source 114 illuminates the sample 106.

[0084] The photon counting device 112 is x In this case, the input photon arrival time data P i , for example, to generate a histogram of photons 132 received in response to a single light pulse 120. The input photon arrival time data P j For example, the input image I is generated from x Input photon arrival time data P i is the position in the input image x i , and as a result, the input photon arrival time data P i ( x i ,t) is obtained.

[0085] After a certain time has elapsed, for example when no more photons reach the photon counting device 112, or after a preset time, the sample 106 or the light beam 118 is moved to another position. x and moved to this other position x At position , the next light pulse 120 triggers the emission of another set of photons 132. The input photon arrival time data from this position is then transferred to another position in the input image, x i and this other position x i Preferably, at a plurality of subsequent positions in the sample x In this way, the positional relationship between x is moved along the image raster 122.

[0086] More reliable input photon arrival time data P i ( x i , t) at one position by, for example, triggering a sequence of light or laser pulses 120 to emit photons 132, and by advancing time t from the same relative starting point for each light pulse in the sequence, superimposing various photon counts from different pulses in the sequence. x More than one measurement can be performed in

[0087] In Figure 1, histograms showing photon counts N over time are shown for three different sample locations, corresponding to three different input image locations. These histograms correspond to the input photon arrival time data.

[0088] An input image I is generated by scanning the entire sample 106 along the image raster 122. The input image I includes input image data I( x i , t), where each position in the input image I is x i In this case, the input photon arrival time data P i( x i , t). Additional information can also be encoded in the input image at each location, so that the input photon arrival time data P i ( x i ,t) is simply the sum of all input image positions x i Input image data I( x i , t).

[0089] The input image I is the ground truth I GT , i.e., to a completely faithful representation of the sample 106 because noise and artifacts are introduced by, for example, the detection optics 110 and the photon counting device 112.

[0090] For example, stray photons 134 that are emitted from a position different from the current position due to the finite width of the point spread function or that are not formed by fluorescence may be recorded by the photon counting device 112 .

[0091] If the sample 106 has more than one fluorophore 108, i.e., F is greater than 1, there are two possibilities: First, the photons of different fluorophores can be combined into a single histogram, resulting in the input photon arrival time data P i ( x i , t) is obtained, and the input photon arrival time data P i ( x i , t) is essentially the individual input photon arrival time data P for each individual fluorophore f. i (f) ( x i , t), where f is the count of the different fluorophores, f = 1, ..., F. The notation P i (f) denotes the input photon arrival time data of the fth fluorophore. Second, the position of the input image Ix i Each histogram in σ contains dye-separated input photon arrival time data P for a single fluorophore, i.e., fluorophore f. i (f) , F, where f=1,...,F. To this end, the detection optics 110 may, for example, comprise one or more dye separation filters, each associated with a different photon counting device. This allows the different photon counting devices 112 to collect input photon arrival time data P for each fluorophore. i (f) In such a case, the fluorescence lifetime imaging device 128 may have two or more photon counting devices 112, each configured to record a different fluorophore. In this way, different dye-separated input images I (f) is recorded by each photon counting device 112.

[0092] In another alternative embodiment, the light source 114 may be adjustable. For example, the light source 114 may be tuned to a different position on the sample 106. x , can be configured to emit a predetermined number of subsequent light pulses 120, each of which is different and therefore triggers the fluorescence of a different fluorophore 108. In this case, the input photon arrival time data P i (f) are recorded separately and successively, and the different dye separation input images I (f) can be assigned to

[0093] The processor 144, which may be part of the fluorescence lifetime imaging device 128, relies on the recording method described above to generate the input image I( x i ,t) or I (f) ( x i , t), different input photon arrival time data P i ( x ,t) or P i(f) ( x i ,t).

[0094] The processor 144 may be, for example, a general-purpose computer having a CPU and preferably a GPU and / or vector processor.

[0095] The data processing device 100 may be the same as the processor 144 or a separate entity from the processor 144. This is particularly the case when the processor 144 is part of a general purpose computer or a dedicated image processing computer, which can then take over the functions of the data processing device 100.

[0096] The data processing device 100 is configured to calculate an output image O from an input image I by deconvolution 102. The input image is obtained by deconvolving the input photon arrival time data P i The input image may be a combined input image or a dye-separated input image.

[0097] The digital output image O is calculated as the input image I at the output image position x o The output image data O( x o , t). Preferably, the output image O contains the same number of positions and the same dimensions as the input image. Output Image Positions x o is the input image position x i It may be compatible with.

[0098] In particular, the output image data O( x o , t) is the output photon arrival time data P o ( x , t). The input image I (f) is the dye-separated input image, then the output image O is also dye-separated output photon arrival time data P o (f) ( x, t) (f) The data processing device 100 is configured to calculate a digital output image O from a digital input image I by deconvolution 102. The deconvolution 102 can be implemented by software, i.e. computer instructions executed by the data processing device 100, and / or by hardware that is part of the data processing device 100.

[0099] The digital output image O is the unblurred ground truth I GT More specifically, the digital output image O represents an estimate of the output image position x o Output photon arrival time data P o ( x o , t), and this output photon arrival time data P o ( x o , t) is the unblurred ground truth estimate E of the input photon arrival time data. (m) ( x i ,t) can be represented as

[0100] The photon counting device 112 generates an unblurred ground truth I GT may add noise and artifacts to the input image I, which are then reflected in the input image I. The properties of the photon counting device 112 and / or the detection optics 110 may affect the point spread function psf( x i [,t]).

[0101] To reduce blur in the input image I, a deconvolution 102 can be performed, resulting in a sharpened or deblurred output image O. The deconvolution 102 preferably comprises an iterative algorithm 104 that, at each iteration m, iterates over the ground truth I until a convergence criterion is met.GT The estimated value of E (m) ( x i , t) is updated. An update function U is used to update the estimate E.

[0102] In particular, the data processing device 100 calculates the estimated value E (m) ( x i ,t) as follows, i.e., E (m) ( x i ,t)= E (m-1) ( x i ,t)·U(E (m-1) ( x i ,t),psf( x i [,t]),P i ( x i ,t)) or E (m) ( x i ,t)= E (m-1) ( x i ,t)+U(E (m-1) ( x i ,t),psf( x i [,t]),P i ( x i ,t)) It can be configured to calculate: Here, the update function U is the current estimate E (m-1) and the point spread function psf( x i [,t]) and the dye-separable input photon arrival time data P i The update function can depend on both time and location, U = U( x i ,t), E (m) ( x i ,t)=E (m-1)( x i ,t)·U( x i ,t) or E (m) ( x i ,t)=E (m-1) ( x i ,t)+U( x i ,t) is.

[0103] However, since iterative algorithms that include time-dependent update functions can result in computationally inefficient and prohibitively complex deconvolution algorithms, the update algorithm 104 uses a time-independent update function U(x i ), i.e. E (m) ( x i ,t)=E (m-1) ( x i ,t)·U( x i ) or E (m) ( x i ,t)=E (m-1) ( x i ,t)+U( x i ) is.

[0104] This allows the use of standard deconvolution algorithms for deconvolution 102, for example Lucy-Richardson deconvolution.

[0105] The behavior of the time-invariant update function is shown in Figure 4, where the position x i1 The leading estimate E at time t1 (m-1) The input photon arrival time data of is maintained at the same time instant t1, but the updated estimate E (m) Now, another positionx i2 In Figure 4, the shifted photon counts are shown in black.

[0106] A time-independent update function can be obtained when an operation T is introduced that makes its arguments time-independent. For example, the operation T may include a sum over time, a statistical moment, or a value of a time-dependent function at a particular point in time, as described in the general part of this specification. The operation T may be implemented in the data processing apparatus 100 by hardware and / or software.

[0107] Using an operation T, the iterative algorithm can be assumed to be of the following form: E (m) ( x i ,t)= E (m-1) ( x i ,t)+U(T(E (m-1) ( x i ,t)),T(psf( x i ,t)),T(P i ( x i ,t))) or E (m) ( x i ,t)= E (m-1) ( x i ,t)·U(T(E (m-1) ( x i ,t)),T(psf( x i ,t)),T(P i ( x i ,t))) is.

[0108] Of course, if the point spread function is already time independent, then there is no need to apply the operation T to the point spread function, i.e., psf = psf( xi )

[0109] In the exemplary described embodiment, the operation T involves a summation over time, and the point spread function is assumed to be time-invariant. Thus, the iterative algorithm 104 may have the following form:

number

[0110] In such a case, the data processing device 100 may use, for example, the following update function:

number

number

[0111] Alternatively to the deconvolution algorithm 102, the data processing device 100 may have a machine learning product 130 that has been trained with an input image I and an output image O, where the output image O has been calculated from the input image I using the deconvolution 102. Such a machine learning product does not perform the deconvolution 102, but processes the input image I as if the deconvolution 102 did.

[0112] Photon arrival time data P i contains the fluorescence of two or more fluorophores, for example in superposition, i.e.

number

[0113] In the upper part of Figure 2, the input photon arrival time data P i ( x i , t) are illustratively three different input image positions x i1 , x i2 , x i3 In P i ( x i1 ,t),P i ( x i2 ,t),P i ( x i3 , t). At each of these three positions, and at any given time t, the photon count N is calculated by dividing the photon count N by two unknown P i (1) ( x i ,t) and P i (2) ( x i , t), where P i (1) ( x i , t) represents the input photon arrival time data for the first fluorophore, f=1, and P i (2) ( x i , t) represents the input photon arrival time data for the second fluorophore, f=2.

[0114] Multiple input image positions to calculate individual dye-separated input photon arrival time data for each fluorophorex i From the combined input photon arrival time data, the aggregate input photon arrival time data P Σ is calculated as follows, for example:

number

[0115] Aggregated input photon arrival time data P derived from input photon arrival time data Σ is shown diagrammatically in the center of Figure 2.

[0116] The aggregated input photon arrival time data is calculated based on the input image position. x i or from only a subset of the input image positions x i It can be calculated from the whole.

[0117] Furthermore, the input photon arrival time data P for different fluorophores i (f) is some known dye decay curve D (f) (t;τ f ), where τ f is the unknown fluorescence lifetime of the fth fluorophore.

[0118] Lifespan τ f can be calculated for F fluorophores using, for example, minimization, in particular least-squares minimization, as follows:

number

number

[0119] As a result of this minimization, the weighting coefficient a for the fth fluorophore isf and lifetime τ f An estimate of the parameter

number

number

[0120] This minimization of equation (2) can be calculated using the Levenberg-Marquardt algorithm. As a result, we obtain the aggregate cumulative lifetime τ for each of the F fluorophores (f = 1,…,F). f In other words, at multiple input image positions x i By combining the input photon arrival time data, the decay curve for each fluorophore can be calculated by curve fitting. Due to combining the aggregate photon arrival time data with multiple input photon arrival time data, the resulting aggregate photon arrival time data has reduced noise, resulting in a more accurate estimate of the decay curve.

[0121] Dye decay curves available for minimization D (f) (t;τ f ) is a simple example: The following, i.e.

number

[0122] Of course, the step of calculating the dye decay curve for each fluorophore can be omitted if a predetermined decay curve or a predetermined lifetime is used. Such predetermined decay curves can be retrieved, for example, from the storage device or memory 146 (FIG. 1) of the data processing device 100. The predetermined dye decay curves can be determined empirically and / or analytically.

[0123] Once the dye decay curves are known for all fluorophores, these can be calculated for each input image position x i In the dye separation, the input photon arrival time data P i (f) This is used to separate the input photon arrival time data into x i In the input photon arrival time data P i , with a known lifetime τ f Dye decay curve with D (f) (t;τ f ) by fitting or linearly combining them.

[0124] For example, the following minimization problem can be solved to calculate dye-separated input photon arrival time data for each fluorophore at each input image location:

number

number

[0125] The above minimization can again be solved at each input image location using the Levenberg-Marquardt algorithm.

[0126] An alternative solution to the above minimization problem can be analytically derived in the following form:

number

[0127] Dye separation input photon arrival time data P i (f) can be determined using the following formula:

number

[0128] Once this is done, the dye-separated input photon arrival time data P i (f) are the input image positions x i for each of the F fluorophores.

[0129] Then, as described above, the input photon arrival time data P i Instead of the respective dye-separated input photon arrival time data P i (f) or equivalently, instead of input image I, each dye-separated input image I (f) Deconvolution 102 is performed separately for each input image I (f) Regarding the output image O (f) , or equivalently, the respective dye-separated input photon arrival time data P i (f) , the dye separation output photon arrival time data P o (f) Calculate.

[0130] Then, dye separation output photon arrival time data P o(f) The images can be summed across the phosphors, which gives the output photon arrival time data P o That is,

number

[0131] Equivalently, the dye separation output image O (f) can be added together to arrive at the output image, i.e.

number

[0132] The schematic flow chart of Figure 5 summarizes the above-mentioned processes that can be performed by the data processing device 100. x i ), i.e., at each input image position x i Input photon arrival time data P i ( x i , t), step 510 first calculates the input photon arrival time data P i Determine whether more than one fluorophore is represented in F, i.e., whether F is greater than 1.

[0133] For F=1, the input photon arrival time data can be used to generate the output image O( x i ,t) is calculated.

[0134] F>1, i.e., the input photon arrival time data P i If contains two or more fluorophores, dye separation input photon arrival time data P i (f) Divide the input photon arrival time data into x i over the respective dye separations, the input photon arrival time data P i(f) The convolution is calculated separately for each of the

[0135] Further processing is performed using the already predefined dye decay curves D available for each fluorophore. (f) (t;τ f ) and / or fluorescence lifetime τ f This is checked in step 512.

[0136] Predefined dye decay curve D (f) (t;τ f ) or a predetermined lifetime τ f If P is already available for each fluorophore, then in step 500, the dye-separated input photon arrival time data P is calculated using predetermined dye decay curves, e.g., a linear combination of the individual predetermined dye decay curves, e.g., by fitting these to the combined input photon arrival time data. i (f) can be calculated. This can be done using the Gauss-Newton or Levenberg-Marquardt algorithm to solve equation (4).

[0137] If pre-defined dye decay curves are not available for the different fluorophores contained in the input image, then these curves need to be determined in step 502. Step 502 may include two steps, namely steps 504 and 506.

[0138] In step 504, the aggregated input photon arrival time data P Σ In step 506, the aggregated input photon arrival time data P Σ Using the dye decay function shown in, for example, Eq. (3), the dye decay curves D of different fluorophores can be calculated, for example, according to Eq. (2). (f) (t;τ f ) or lifetime τ f Calculate.

[0139] In this case, these decay curves can be used in the same way as the pre-defined dye decay curves used in step 500 to calculate dye-separated input photon arrival time data for each fluorophore.

[0140] Dye separation input photon arrival time data P i (f) By using (f) , where each dye-separated input image contains dye-separated input photon arrival time data for only one fluorophore. In step 514, deconvolution 102 is then performed separately for each dye-separated input image, resulting in output image O (f) Of course, if only one fluorophore is present, the input photon arrival time data P i only exists, and the output photon arrival time data P o There is only a single input image and a single output image with only .times. ...

[0141] If the original input image I contained more than one fluorophore, the output image O (f) can be combined into a single output image O in step 508, for example as described above using equation (5).

[0142] The output image can be displayed on any device, such as a monitor, a projector, goggles, etc. The monitor can be part of the data processing unit and / or the fluorescence lifetime imaging device.

[0143] To evaluate the deconvolution 102, random fluorescent spots with two lifetimes τ = 3.8 ns and τ = 1.3 ns were simulated. The simulated data had a spatial dimension of 256 × 256 pixels. The input photon arrival time data included 100 time steps, i.e., t = t, ..., t 100 The number of fluorophores was 2, i.e., F=2. The maximum signal-to-noise ratio was set to 5.

[0144] FIG. 6a shows the input photon arrival time data P i =Σ f P i (f) In Figure 6b, the result P o =Σ f P o (f) , i.e. P o ( x o ,t1=0), where x i = x o The effect of deblurring is clearly visible in the output photon arrival time data.

[0145] The deblurring effect can also be seen in the intensity cross section of Figure 6c, which is taken along the input image position on the white line in Figure 6a.

[0146] In Figure 6c, the input photon arrival time data along this line is shown as "raw," the ground truth intensity distribution, i.e., the noise-free fluorescent spot, is shown as "GT," and the output photon arrival time data is shown as "decon." The results of the deconvolution process are very close to the ground truth, representing a significant improvement over the "raw," i.e., noisy, input image data.

[0147] Figure 6d shows the input photon arrival times, "raw," the two raw data, the ground truth, and the deconvolution for one fluorescent spot indicated by a star on the line in Figure 6a. Again, the deconvolution results are very close to the ground truth and represent a significant improvement over the raw data. In Figures 6c and 6d, the input image position x i , i.e., output image positions along the same line and from the same fluorescent spot. x o In, data from the output image is obtained.

[0148] Clearly, deconvolution improves resolution and significantly reduces the signal-to-noise ratio: the peak signal-to-noise ratio is significantly improved from 36 in the raw input image data to 44 in the deconvolved output image data.

[0149] Using the same data as in Figures 6a-6d, the same quality improvement can be seen in the phasor plots shown in Figure 7a for the input image data and in Figure 7b for the output image data. The phasor plot in Figure 7a for the input image data shows large fluctuations due to noise. After dye-separated deconvolution, all points in phasor space can be assigned to one fluorophore.

[0150] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0151] While some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or apparatus corresponds to a step or feature of a step, and similarly, aspects described in the context of a step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0152] Some embodiments relate to a microscope that includes a system such as that described in connection with one or more of Figures 1 to 7. Alternatively, the microscope may be part of or connected to a system such as that described in connection with one or more of Figures 1 to 7.

[0153] FIG. 8 shows a schematic diagram of a system 800 configured to implement the methods described herein. The system 800 includes a microscope 810 and a computer system 820. The microscope 810 is configured to capture images and is connected to the computer system 820. The computer system 820 is configured to implement at least a portion of the methods described herein. The computer system 820 may be configured to execute machine learning algorithms. The computer system 820 and the microscope 810 may be separate entities or may be integrated into a common housing. The computer system 820 may be part of a central processing system of the microscope 810 and / or may be part of a subsidiary component of the microscope 810, such as a sensor, actor, camera, or lighting unit of the microscope 810.

[0154] Computer system 820 may be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) with one or more processors and one or more storage devices, or may be a distributed computing system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed at various locations, such as local clients and / or one or more remote server farms and / or data centers). Computer system 820 may include any circuit or combination of circuits. In one embodiment, computer system 820 may include one or more processors, which may be of any type. As used herein, processor may contemplate any type of computing circuit, such as, but not limited to, a microprocessor of a microscope or microscope component (e.g., a camera), a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit. Other types of circuits that may be included in computer system 820 may be custom circuits, application specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communications circuits) used in wireless devices such as cell phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. Computer system 820 may also include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives that handle removable media, such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.Computer system 820 may also include a display device, one or more speakers and a controller which may include a keyboard and / or mouse, trackball, touch screen, voice recognition device, or any other device that allows a user of the system to input information to and receive information from computer system 820.

[0155] Some or all of the steps may be performed by (or using) a hardware apparatus, such as, for example, a processor, microprocessor, programmable computer, or electronic circuitry. In some embodiments, any one or more of the critical steps may be performed by such an apparatus.

[0156] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation may be performed by a non-transitory storage medium, such as a digital storage medium, for example, a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to implement the respective methods. Therefore, the digital storage medium may be computer-readable.

[0157] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform any of the methods described herein.

[0158] Generally, embodiments of the present invention may be implemented as a computer program product comprising program code that is operative to perform any of the methods when the computer program product is run on a computer, and that may be stored, for example, on a machine-readable carrier.

[0159] Further embodiments comprise the computer program for performing any of the methods described herein, stored on a machine readable carrier.

[0160] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing any of the methods described herein when the computer program runs on a computer.

[0161] Therefore, another embodiment of the invention is a recording medium (or data carrier or computer readable medium) containing a computer program stored thereon for performing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a recording medium.

[0162] A further embodiment of the present invention is, therefore, a data stream or a sequence of signals representing the computer program for performing any of the methods described herein, the data stream or sequence of signals being for example adapted to be transmitted via a data communication connection, for example the Internet.

[0163] Another embodiment comprises a processing means, for example a computer, or a programmable logic device configured to or adapted to perform any of the methods described herein.

[0164] Another embodiment comprises a computer having installed thereon the computer program for performing any of the methods described herein.

[0165] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0166] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are advantageously performed by any hardware apparatus.

[0167] Embodiments may be based on the use of machine learning models or algorithms. Instead of relying on models and inference, machine learning may refer to algorithms and statistical models that a computer system may use to perform a particular task without using explicit instructions. For example, machine learning may use data transformations inferred from an analysis of past data and / or training data instead of rule-based data transformations. For example, image content may be analyzed using a machine learning model or algorithm. For a machine learning model to analyze image content, the machine learning model may be trained using training images as input and training content information as output. By training the machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize image content, such that image content not included in the training data can be recognized using the machine learning model. The same principle may be used for other types of sensor data as well: by training the machine learning model with training sensor data and a desired output, the machine learning model "learns" a transformation between sensor data and output, which can be used to provide an output based on the non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata and / or image data) may be pre-processed to obtain feature vectors that are used as input to machine learning models.

[0168] A machine learning model may be trained using training input data. The above example uses a training method called "supervised learning." In supervised learning, a machine learning model is trained using multiple training samples, where each sample may include multiple input data values ​​and multiple desired output values, i.e., each training sample is associated with a desired output value. By specifying both the training samples and the desired output value, the machine learning model "learns" during training which output value to provide based on input samples similar to the provided sample. In addition to supervised learning, semi-supervised learning may be used. In semi-supervised learning, some of the training samples lack a corresponding desired output value. Supervised learning may be based on a supervised learning algorithm (e.g., a classification algorithm, a regression algorithm, or a similarity learning algorithm). A classification algorithm may be used when the output is restricted to a limited set of values ​​(categorical variables), i.e., the input is classified into one of a limited set of values. A regression algorithm may be used when the output may have any numerical value (within a range). Similarity learning algorithms may be similar to both classification and regression algorithms, but are based on learning from examples using a similarity function that measures how similar or related two objects are. In addition to supervised or semi-supervised learning, unsupervised learning may also be used to train machine learning models. In unsupervised learning, input data may be provided (only), and unsupervised learning algorithms may be used to find structure in the input data (e.g., by grouping or clustering the input data, finding commonalities in the data). Clustering is the assignment of input data containing multiple input values ​​into multiple subsets (clusters) such that input values ​​within the same cluster are similar according to one or more (predefined) similarity criteria, but dissimilar to input values ​​contained in another cluster.

[0169] Reinforcement learning is a third group of machine learning algorithms. In other words, reinforcement learning may be used to train machine learning models. In reinforcement learning, one or more software actors (referred to as "software agents") are trained to take actions in their surroundings. Based on the actions taken, a reward is calculated. Reinforcement learning is based on training one or more software agents to select actions that increase cumulative rewards (as manifested by increasing rewards), resulting in the software agent becoming better at a given task.

[0170] Furthermore, some techniques may be applied to parts of the machine learning algorithm. For example, feature representation learning may be used. In other words, a machine learning model may be trained at least in part using feature representation learning, and / or a machine learning algorithm may include a feature representation learning component. A feature representation learning algorithm, which may be referred to as a representation learning algorithm, may not only preserve information in its input, but may also transform the information to make it useful, often as a preprocessing step before performing classification or prediction. Feature representation learning may be based on, for example, principal component analysis or cluster analysis.

[0171] In some examples, anomaly detection (i.e., outlier detection) may be used, which aims to provide identification of input values ​​that raise suspicion by differing significantly from the majority of the input or training data. In other words, a machine learning model may be trained at least in part with anomaly detection and / or a machine learning algorithm may include an anomaly detection component.

[0172] In some examples, a machine learning algorithm may use a decision tree as a predictive model. In other words, the machine learning model may be based on a decision tree. In a decision tree, an observation about an item (e.g., a set of input values) may be represented by a branch of the decision tree, and an output value corresponding to the item may be represented by a leaf of the decision tree. The decision tree may support both discrete and continuous values ​​as output values. If discrete values ​​are used, the decision tree may be represented as a classification tree, and if continuous values ​​are used, the decision tree may be represented as a regression tree.

[0173] Association rules are another technique that can be used in machine learning algorithms. In other words, a machine learning model may be based on one or more association rules. Association rules are created by identifying relationships between variables in large amounts of data. A machine learning algorithm may identify and / or utilize one or more association rules that represent knowledge derived from the data. These rules may be used, for example, to store, manipulate, or apply the knowledge.

[0174] Machine learning algorithms are typically based on machine learning models. In other words, the term "machine learning algorithm" may refer to a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" may refer to a data structure and / or a set of rules that represent learned knowledge (e.g., based on training performed by a machine learning algorithm). In embodiments, the use of a machine learning algorithm may refer to the use of an underlying machine learning model (or underlying machine learning models). The use of a machine learning model may refer to the machine learning model and / or the set of data structures / rules that are the machine learning model being trained by a machine learning algorithm.

[0175] For example, the machine learning model may be an artificial neural network (ANN). An ANN is a system influenced by biological neural networks, such as those found in the retina or brain. An ANN includes multiple interconnected nodes and multiple junctions, or edges, between the nodes. Typically, there are three types of nodes: input nodes that receive input values, hidden nodes that are (only) connected to other nodes, and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may transmit information from one node to another. The output of a node may be defined as a (nonlinear) function of its input (e.g., the sum of its inputs). The input of a node may be used in a function based on the "weights" of the edges or nodes that provide the input. The weights of the nodes and / or edges may be adjusted during the learning process. In other words, training an artificial neural network may involve adjusting the weights of the nodes and / or edges of the artificial neural network to obtain a desired output for a given input.

[0176] Alternatively, the machine learning model may be a support vector machine, a random forest model, or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with an associated learning algorithm that can be used to analyze data (e.g., in classification or regression analysis). A support vector machine may be trained by providing input with multiple training input values ​​that belong to one of two categories. A support vector machine may be trained to assign new input values ​​to one of two categories. Alternatively, the machine learning model may be a Bayesian network, which is a probabilistic directed acyclic graphical model. A Bayesian network may use a directed acyclic graph to represent a set of random variables and their conditional dependencies. Alternatively, the machine learning model may be based on a genetic algorithm, a search algorithm and a heuristic method that mimics the process of natural selection. [Explanation of symbols]

[0177] 100 Data processing device 102 Deconvolution 104 Iterative Algorithms 106 samples 108 Phosphor 110 Detection optical system 112 Photon Counting Device 114 Light source 116 Illumination optical system 118 Light Beam 120 Light Pulse 122 Image Raster 128 Fluorescence Lifetime Imaging Device 130 Machine Learning Products 132 Photon 134 Laser 136 Pulsed Laser 138 Relative motion between the sample and the photon counting device 140 Floating Photons 142 pulse period 144 processors 146 Storage / Memory 500 Dye-separated input photon arrival time data calculations 502 Calculation of dye decay curves for different fluorophores 504 Calculation of fluorescence lifetimes of different fluorophores 506 Calculation of dye decay curves at input image positions 508 Combine the dye separation output images for each fluorophore 510 Identify the number of fluorophores in the input photon arrival time data 512 Identify whether predefined dye decay curves and / or fluorescence lifetimes are available 514 Performing deconvolution a (f) ,b (f) Weighting coefficients for fitting the dye decay curve of fluorophore f

number

Claims

1. A data processing apparatus (100) for processing a digital input image (I), comprising: The digital input image (I) is input at an input image position (x ｉ ) the input photon arrival time data (P ｉ (x ｉ , t) The data processing device (100) is configured to calculate a digital output image (O) based on the digital input image (I) by deconvolution (102), The digital output image (O) is ｏ ) output photon arrival time data (P ｏ (x ｏ , t)), and the output photon arrival time data (P ｏ (x ｏ , t)) is the input photon arrival time data (P ｉ (x ｉ , t)) （ｍ） (x ｉ , t)), the data processing device (100) is configured to calculate the deconvolution (102) by an iterative algorithm (104) using an update function (U), The update function (U) is the point spread function (psf(x ｉ [, t]) and the ground truth (I ＧＴ (x ｉ , t)) （ｍ－１） (x ｉ , t)) and the input photon arrival time data (P ｉ (x ｉ ,t)) and depends on, A data processing device (100).

2. The point spread function (psf(x ｉ )) and at least one of the update functions (U) is time independent; The data processing apparatus (100) of claim 1.

3. The deconvolution (102) includes the input photon arrival time data (P ｉ (x ｉ ,t), P ｉ （ｆ） (x ｉ , t), the point spread function, and / or the prior estimate, and the operation (T) is ｉ (x ｉ ,t), P ｉ （ｆ） (x ｉ , t), psf(x ｉ [,t]), E （ｍ－１） (x ｉ , t)) time dependency, A data processing device (100) according to claim 1 or 2.

4. The operation (T) has its operands (P ｉ (x ｉ ,t), P ｉ （ｆ） (x ｉ , t), psf(x ｉ [,t]), E （ｍ－１） (x ｉ , t) over time and the input image position and a specific time point (t １ and the value of the operand in The data processing apparatus (100) of claim 3.

5. The iterative algorithm (104) calculates the update function (U) and the prior estimate (E （ｍ－１） (x ｉ , t)), wherein the combination is at least one of addition and multiplication; A data processing device (100) according to any one of claims 1 to 4.

6. The input photon arrival time data (P ｉ (x ｉ , t)) has input photon arrival time data for a number (F) of different fluorophores (108), said number (F) being greater than 1; The data processing device (100) processes different dye-separated input photon arrival time data (P ｉ （ｆ） (x ｉ , t)), wherein the number of dye-separated input photon arrival times (F) corresponds to the number of distinct fluorophores (108); The data processing device (100) processes the input photon arrival time data (P ｉ (x ｉ , t) as the dye-separated input photon arrival time data (P ｉ （ｆ） (x ｉ , t)) separately for calculating the deconvolution (102), A data processing device (100) according to any one of claims 1 to 5.

7. The data processing device (100) Multiple input image positions (x ｉ ) from the input photon arrival time data (P ｉ (x ｉ , t) from the combination of the aggregated input photon arrival time data (P Σ (x ｉ , t) and For each of the number (F) of phosphors (108), the aggregated input photon arrival time data (P Σ (x ｉ , t)) to the dye decay curve (D （ｆ） (x ｉ , t) It is configured as follows: The data processing apparatus (100) of claim 6.

8. The dye decay curve (D) for the number (F) of phosphors (108) （ｆ） (x ｉ , t) is calculated based on the lifetime (τ ｆ ) calculations are included. The data processing apparatus (100) of claim 7.

9. The data processing device (100) calculates the dye decay curve (D （ｆ） (x ｉ , t)) and a predetermined dye decay curve (D （ｆ） (x ｉ , t)) at the input image position (x ｉ ) the input photon arrival time data (P ｉ (x ｉ , t) to the input image position (x ｉ ) the dye-separated input photon arrival time data (P ｉ （ｆ） (x ｉ , t)), A data processing device (100) according to claim 7 or 8.

10. A fluorescence lifetime imaging device (128) comprising a data processing device (100) according to any one of claims 1 to 9.

11. A method for processing a digital input image (I), comprising the steps of: The digital input image (I) is input at an input image position (x ｉ ) the input photon arrival time data (P ｉ (x ｉ , t) a digital output image (O) is computationally calculated from said digital input image (I) by deconvolution (102); The digital output image (O) is ｏ ) output photon arrival time data (P ｏ (x ｏ , t)), and the output photon arrival time data (P ｏ (x ｏ , t)) is the input photon arrival time data (P ｉ (x ｉ , t)) （ｍ） (x ｉ , t)), The deconvolution (102) is computed by an iterative algorithm (104) using an update function (U), The update function (U) is the point spread function (psf(x ｉ [, t]) and the ground truth (I ＧＴ (x ｉ , t)) （ｍ－１） (x ｉ , t)) and the input photon arrival time data (P ｉ (x ｉ ,t)) and depends on, method.

12. 12. A method according to claim 11, adapted to operate a data processing device (100) according to any one of claims 1 to 9.

13. 13. A computer program comprising instructions which, when executed by a computer, cause said computer to carry out the method according to claim 11 or 12.

14. A computer readable storage medium having instructions which, when executed by a computer, cause the computer to perform the method of claim 11 or 12.

15. An apparatus (130) for processing a digital input image (I), comprising: The digital input image (I) is input at an input image position (x ｉ ) the input photon arrival time data (P ｉ (x ｉ , t) the device is configured to calculate a digital output image (O), The digital output image (O) is ｏ ) output photon arrival time data (P ｏ (x ｏ , t)), and the output photon arrival time data (P ｏ (x ｏ , t)) is the input photon arrival time data (P ｉ (x ｉ , t)) （ｍ） (x ｉ , t)), The device has been trained with pairs of different digital input images (I) and digital output images (O), each digital output image (O) of the pair being calculated from the digital input images (I) of the pair using at least one of the method of claim 11 or 12 and the data processing device (100) of any one of claims 1 to 9. Apparatus (130).

16. A method for training a device (130) with pairs of different digital input images (I) and digital output images (O), each digital output image (O) of the pair calculated from the digital input images (I) of the pair, using at least one of the method of claim 11 or 12 and a data processing device (100) of any one of claims 1 to 9.

Images