An exposure time learnable end-to-end fluorescence lifetime imaging method

Abstract

This invention discloses an end-to-end fluorescence lifetime imaging method with learnable exposure time. The specific steps are as follows: (1) Construct a fluorescence lifetime dataset containing fluorescence lifetime maps and corresponding fluorescence intensity maps; (2) Establish a differentiable physical imaging model, use a pulse function as the modulation function, set the exposure time and demodulation function as learnable parameters, and consider the physical constraints in the actual scene, input fluorescence lifetime and fluorescence intensity data to obtain noise-free multiple measurements; (3) Model a noise model, derive the weighted Fisher information considering the exposure time allocation, and input the noise-added measurements into the fluorescence lifetime reconstruction network; (4) Use the weighted Fisher information as the loss function, and combine it with the fluorescence lifetime fidelity loss function to jointly train the exposure time, demodulation function and fluorescence lifetime reconstruction network; (5) Input the test data into the trained exposure time, demodulation function and reconstruction network to reconstruct the fluorescence lifetime map.

Description

Technical Field

[0001] This invention belongs to the field of computational imaging and deep learning technology, and in particular relates to an end-to-end fluorescence lifetime imaging method with learnable exposure time. Background Technology

[0002] Fluorescence lifetime imaging, a powerful technique for resolving fluorophores and their unique molecular environments, is widely used in the biomedical field. Compared to intensity-based fluorescence imaging, fluorescence lifetime imaging offers unique advantages: it is less sensitive to fluorophore concentration, can distinguish fluorophores with similar spectra, detects changes in molecular environmental parameters (such as temperature, pH, and oxygen levels), and studies cellular metabolism. For live-cell imaging or real-time applications, dynamic measurements of rapidly changing environmental factors, surrounding cells, and cellular structures are necessary. Furthermore, considering that excessively long exposure times and high optical power during live-cell imaging can impair the photostability of fluorescent samples, leading to photodamage and photobleaching, rapidly, accurately, and safely measuring fluorescence lifetime has always been a significant challenge.

[0003] Currently, there are two main methods for measuring fluorescence lifetime. One is time-domain fluorescence lifetime imaging (TDFF), which uses a high-frequency pulsed light source to illuminate the fluorescent sample. By statistically analyzing the photon distribution in the acquired time-domain signal, a fluorescence intensity decay curve is fitted to calculate the corresponding fluorescence lifetime. TDFF can directly capture the ultrafast fluorescence decay process, but it suffers from complex data processing, slow imaging speed, and requires high-speed sensors, resulting in high costs. The other method is frequency-domain fluorescence lifetime imaging (FROM). This method uses modulation techniques to modulate the light source, emitting continuously periodic modulated light. The incident light is absorbed by the fluorescent sample, emitting fluorescence with altered amplitude and phase. This fluorescence is received by a sensor, and a demodulation function extracts the lifetime information. This method can measure fluorescence lifetime quickly and efficiently, but it can produce significant measurement errors under low signal-to-noise ratio conditions.

[0004] With the development of deep learning, an increasing number of studies in time-domain fluorescence lifetime imaging are utilizing neural networks to process complex photon data, thereby improving the imaging speed and measurement accuracy of time-domain methods. In frequency-domain fluorescence lifetime imaging, the exploration of optimal encoding schemes has never ceased. However, how to optimize the encoding function under different noise levels to achieve accurate fluorescence lifetime imaging remains a challenge. Furthermore, besides optimal encoding schemes, how to rationally allocate the exposure time for each measurement under exposure constraints is also worth exploring. Summary of the Invention

[0005] This invention proposes an end-to-end framework for jointly optimizing the exposure time, demodulation function, and fluorescence lifetime reconstruction network in a fluorescence lifetime imaging model. By considering the weighted Fisher loss and fidelity loss of the exposure time during training, the optimized exposure time, demodulation function, and reconstruction network can be used in a practical fluorescence lifetime imaging hardware prototype system to achieve high-precision fluorescence lifetime imaging under low exposure and high noise conditions.

[0006] The technical solution adopted in this invention is as follows:

[0007] An end-to-end fluorescence lifetime imaging method with learnable exposure time includes the following steps:

[0008] Step 1: Construct a fluorescence lifetime database, including a training set and a test set;

[0009] Step 2: Establish a differentiable physical imaging model to simulate the fluorescence lifetime imaging process. Use the pulse function as the modulation function, set the exposure time and demodulation function as learnable parameters, and consider the physical constraints in the actual scene. Input the data in the training set into the differentiable physical imaging model to obtain noise-free multiple measurement values.

[0010] Step 3: Model the noise model and obtain multiple measurements after adding noise;

[0011] Step 4: Input the noise-added multiple measurements into the fluorescence lifetime reconstruction network to obtain the reconstructed fluorescence lifetime map; the weighted Fisher information of exposure time is considered during reconstruction.

[0012] Step 5: Construct a loss function, including a weighted Fisher information-guided loss function and a fidelity loss function, and perform joint training and optimization of the exposure time, demodulation function and fluorescence lifetime reconstruction network in an end-to-end framework.

[0013] Step 6: On the test set, the imaging process is simulated using the exposure time and demodulation function optimized in Step 5 to obtain multiple measurement maps with noise. Then, the measurement maps are input into the trained fluorescence lifetime reconstruction network to obtain the final fluorescence lifetime reconstruction map.

[0014] Furthermore, step 6 is followed by step 7: building a hardware prototype system for fluorescence lifetime imaging, using the optimized exposure time and demodulation function, combined with the pulse modulation function, acquiring measurement images of real fluorescence samples, inputting them into the trained fluorescence lifetime reconstruction network, and finally outputting lifetime images of real fluorescence samples to complete fluorescence lifetime imaging.

[0015] This invention proposes an end-to-end fluorescence lifetime imaging method based on a learnable forward model of exposure time and a reconstruction network. The exposure time and demodulation function are set as learnable parameters and combined with the reconstruction network. Joint optimization is performed by considering the weighted Fisher loss and fidelity loss of the exposure time allocation. The optimized exposure time allocation scheme and demodulation function are applied to a prototype fluorescence lifetime imaging system to obtain actual measurements, which are then fed into the reconstruction network to obtain an accurate fluorescence lifetime map. Simulation experiments have demonstrated that, compared with existing fluorescence lifetime encoding schemes, this invention achieves superior performance under different exposure time constraints. Attached Figure Description

[0016] Figure 1 This is a schematic flowchart of the method of the present invention;

[0017] Figure 2 The overall framework diagram for implementing the method of this invention;

[0018] Figure 3 This is a comparison diagram of the simulation results of the present invention. Detailed Implementation

[0019] The method of the present invention will be described in detail below, with detailed implementation methods and specific operation procedures provided. However, the scope of protection of the present invention is not limited to the following implementations.

[0020] This embodiment provides an end-to-end fluorescence lifetime imaging method with learnable exposure time, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0021] Step 1: Construct a fluorescence lifetime database, which is built from real fluorescence lifetime datasets and synthetic fluorescence lifetime datasets. The synthetic fluorescence lifetime dataset is made from a fluorescence denoising dataset and is divided into training data and test data. Each sample contains an intensity map and a lifetime map of the fluorescence sample.

[0022] Step 2: Model the fluorescence lifetime imaging process as a differentiable physical imaging model. Use a pulse function as the modulation function, set the exposure time and demodulation function as learnable parameters, and consider the physical constraints in the actual scene. Input the data from the training set into the differentiable physical imaging model to obtain noise-free multiple measurements B. i :

[0023]

[0024] Where M(t) and D(t) are the pulse modulation function and the learnable demodulation function, respectively, h(t) is the pulse response function of the fluorescent sample, K is the number of measurements, and Δt iExposure time allocated to each measurement. D(t) is set as a superposition of multiple learnable amplitude and phase sine functions:

[0025]

[0026] Where a j φ j b j Let ω be the amplitude, phase, and offset of the j-th sine wave in the demodulation function, and let ω0 be the fundamental frequency of the demodulation function.

[0027] Step 3: Model the noise model to obtain multiple measurements I with noise. i :

[0028]

[0029] Where n i (Δt i ) represents the noise corresponding to the measured value. The actual measured value, after considering noise modeling, is Gaussian distributed, μ. i and σ i The mean and variance of the actual measured values ​​are Gaussian distributions:

[0030]

[0031]

[0032] Where g represents a Gaussian distribution, b i λ is the expected value of the theoretically noise-free measurement. d It is the expectation of dark current noise with a Poisson distribution, σ r The standard deviation of the noise is given by the Gaussian distribution.

[0033] The fluorescence lifetime imaging measurements at a single pixel (x, y) with respect to lifetime, weighted by exposure time and the amount of Fisher information, are as follows:

[0034]

[0035] Where σ i (x,y) represents the standard deviation of the actual measured value at pixel (x,y), b i (x,y) represents the theoretically uninfluenced measurement value at pixel (x,y), and τ represents the true fluorescence lifetime. From this, the weighted Fisher information content considering exposure time is derived.

[0036] Step 4, convert the noisy multiple measurements I i The input is fed into a fluorescence lifetime reconstruction network consisting of a feature extraction part and a reconstruction part to obtain the reconstructed fluorescence lifetime map. The exposure time-weighted Fisher information is considered during reconstruction.

[0037] Step 5: Construct loss functions for exposure time, demodulation function, and fluorescence lifetime reconstruction network, including a weighted Fisher information-guided loss function and a fidelity loss function that consider exposure time. Perform end-to-end joint training and optimization of the exposure time, demodulation function, and fluorescence lifetime reconstruction network framework. The weighted Fisher information-guided loss function is:

[0038]

[0039] The fidelity loss function is the mean squared error loss function.

[0040] Step 6: In the testing phase, the optimized exposure time and demodulation function are used to simulate the imaging process, resulting in multiple noisy measurement maps. These maps are then input into the trained reconstruction network to obtain the final fluorescence lifetime reconstruction map.

[0041] Step 7, Build a hardware prototype system for fluorescence lifetime imaging: such as Figure 2 As shown, a laser source capable of being modulated by any waveform is first used to emit laser light. The emitted laser light is modulated using an optimized exposure time combined with a pulse modulation function. After passing through a dichroic mirror and a microscope objective, it is irradiated onto the fluorescent sample. The fluorescence emitted by the sample is focused onto the sensor by a lens group after passing through the microscope objective and the dichroic mirror. The acquired signal is multiplied by the optimized demodulation function and integrated to obtain the measurement map of the real fluorescent sample. This map is then input into the optimized fluorescence lifetime reconstruction neural network, and finally, the reconstructed lifetime map of the real fluorescent sample is output, thus completing fluorescence lifetime imaging.

[0042] This example uses ADAM as the network optimizer, with an initial learning rate of 0.01. The learning rate decays linearly at a rate of 0.8 every 10 epochs. The parameters of the learnable encoding function are set using Kaiming initialization. Finally, the network stops training after 200 epochs.

[0043] Please see Figure 3 Our proposed method was compared with existing coding schemes on real datasets in simulation experiments. Our method achieved the best results in terms of MAE (Magnitude of Effect) across three exposure time levels.

Claims

1. An end-to-end fluorescence lifetime imaging method with learnable exposure time, characterized in that, Includes the following steps: Step 1: Construct a fluorescence lifetime database, including a training set and a test set; Step 2: Establish a differentiable physical imaging model to simulate the fluorescence lifetime imaging process. Use the pulse function as the modulation function, set the exposure time and demodulation function as learnable parameters, and consider the physical constraints in the actual scene. Input the data in the training set into the differentiable physical imaging model to obtain noise-free multiple measurement values. Step 3: Model the noise model and obtain multiple measurements after adding noise; Step 4: Input the noise-added multiple measurements into the fluorescence lifetime reconstruction network to obtain the reconstructed fluorescence lifetime map; the weighted Fisher information of exposure time is considered during reconstruction. Step 5: Construct a loss function, including a weighted Fisher information-guided loss function and a fidelity loss function, and perform joint training and optimization of the exposure time, demodulation function and fluorescence lifetime reconstruction network in an end-to-end framework. Step 6: On the test set, the imaging process is simulated using the exposure time and demodulation function optimized in Step 5 to obtain multiple measurement maps with noise. Then, the measurement maps are input into the trained fluorescence lifetime reconstruction network to obtain the final fluorescence lifetime reconstruction map.

2. The end-to-end fluorescence lifetime imaging method with learnable exposure time according to claim 1, characterized in that, In step 1, the database includes synthetic fluorescence images and real fluorescence images. Each sample in the training set and test set contains an intensity map and a lifetime map of the fluorescence sample.

3. The end-to-end fluorescence lifetime imaging method with learnable exposure time according to claim 1, characterized in that, The noise-free multiple measurements in step 2 are as follows: in , These are the pulse modulation function and the learnable demodulation function, respectively. Let be the impulse response function of the fluorescent sample. The number of measurements. Exposure time allocated to each measurement.

4. The end-to-end fluorescence lifetime imaging method with learnable exposure time according to claim 3, characterized in that, The learnable demodulation function is configured as a superposition of multiple learnable amplitude and phase sine functions: in , , For the demodulation function, the first The amplitude, phase, and offset of a sine wave. This is the fundamental frequency of the demodulation function.

5. The end-to-end fluorescence lifetime imaging method with learnable exposure time according to claim 3, characterized in that, The multiple measurements after adding noise in step 3 are: in This represents the noise corresponding to the measured value.

6. The end-to-end fluorescence lifetime imaging method with learnable exposure time according to claim 5, characterized in that, The measured values ​​follow a Gaussian distribution: in Represents a Gaussian distribution. and The mean and standard deviation of the actual measured values ​​are Gaussian distributions. The expected value of the measurement without added noise in theory. It is the expectation of dark current noise with a Poisson distribution. The standard deviation of the noise is given by the Gaussian distribution.

7. The end-to-end fluorescence lifetime imaging method with learnable exposure time according to claim 1, characterized in that, In step 4, fluorescence lifetime imaging is performed on a single pixel. The measured values, considering lifetime, and the exposure time-weighted Fisher information content are: in, For at pixel The standard deviation of the actual measured value, For at pixel Theoretically, the expected value of the measurement without noise is... This represents the true fluorescence lifetime value.

8. The end-to-end fluorescence lifetime imaging method with learnable exposure time according to claim 7, characterized in that, In step 5, the weighted Fischer information-guided loss function is: Where K is the number of measurements.

9. The end-to-end fluorescence lifetime imaging method with learnable exposure time according to claim 1, characterized in that, In step 5, the mean squared error loss function is used as the fidelity loss function.

10. The end-to-end fluorescence lifetime imaging method with learnable exposure time according to claim 1, characterized in that, Step 6 is followed by step 7: building a hardware prototype system for fluorescence lifetime imaging, using the optimized exposure time and demodulation function, combined with the pulse modulation function, acquiring measurement images of real fluorescence samples, inputting them into the trained fluorescence lifetime reconstruction network, and finally outputting lifetime images of real fluorescence samples to complete fluorescence lifetime imaging.

Images