Fluorescence lifetime super-resolution imaging method based on deep learning

Abstract

The present invention relates to the technical field of optical microscope imaging. Disclosed is a fluorescence lifetime super-resolution imaging method based on deep learning. The method comprises: step one, performing fluorescence microscopic imaging on a sample, so as to obtain a confocal intensity image and an STED intensity image which are at the same position; step two, performing a registration operation on the collected confocal intensity image and STED intensity image; step three, respectively using the registered confocal intensity image and STED intensity image as an input and a real value, and forming a dataset in pairs; step four, dividing the dataset into a training set and a validation set according to a certain proportion; and step five, establishing a network, and selecting hyper-parameters and an optimizer. In the fluorescence lifetime super-resolution imaging method based on deep learning, fluorescence lifetime super-resolution microscopic imaging is realized in a conventional confocal fluorescence lifetime imaging (FLIM) system, the limitation of the spatial resolution of FLIM technology is eliminated, the bottleneck of resolutions in conventional optical microscopic imaging is eliminated, and the normal fluorescence lifetime feature of a fluorescent probe is also displayed.

Description

A fluorescence lifetime super-resolution imaging method based on deep learning Technical Field

[0001] The present invention relates to the technical field of optical microscope imaging, and in particular to a fluorescence lifetime super-resolution imaging method based on deep learning. Background Art

[0002] Fluorescence lifetime imaging (FLIM), a cutting-edge optical imaging method, has broad applications in scientific research. By precisely measuring the average time it takes for fluorescent molecules to return from an excited state to a ground state—the fluorescence lifetime—FLIM provides researchers with a unique perspective on the structure and properties of fluorophores. In biology, FLIM is widely used for cell imaging and biolabeling. By measuring the fluorescence lifetime of fluorescent molecules within cells, researchers can understand cellular structure and function, as well as interactions between biomolecules, providing powerful support for disease diagnosis and treatment. In chemistry and materials science, fluorescence lifetime measurements can reveal the chemical structure, defect states, and energy transfer processes within materials, providing important insights for optimizing and designing material properties. Therefore, FLIM plays a vital role in a wide range of fields, including biology, chemistry, and materials science.

[0003] However, due to the diffraction of light, the spatial resolution of FLIM technology is limited, just like traditional optical microscopy. Therefore, it cannot meet the requirements for high-resolution research on the complex internal structures and dynamic processes of cells, limiting its application at the microscopic scale. Currently, mainstream super-resolution imaging technologies include stimulated emission depletion (STED) microscopy, structured illumination microscopy (SIM), single-molecule localization microscopy (SMLM), and minimum light flux (MINFLUX) microscopy. These technologies have significantly improved imaging resolution and promoted the advancement of microscopic imaging technology. However, these super-resolution imaging technologies are plagued by problems in their implementation, such as excessive laser power, slow imaging speed, and expensive imaging systems.

[0004] In recent years, deep learning technology has achieved significant breakthroughs and progress in numerous fields. As a computational model that mimics the structure and function of the human brain's neural networks, deep learning, by building deep neural networks, can automatically extract useful features from large amounts of data and efficiently perform tasks such as classification, recognition, and prediction. In the field of image recognition, deep learning has surpassed traditional image processing algorithms and become one of the most advanced methods available. Furthermore, deep learning can be applied to noise reduction and enhancement of fluorescence images, improving image clarity and contrast, and further enhancing imaging quality.

[0005] Given the current demand for super-resolution fluorescence lifetime imaging in optical microscopy, this patent is dedicated to the development of a fluorescence lifetime imaging technology with super-resolution capabilities. The core of this technology lies in achieving a significant improvement in spatial resolution while maintaining the accuracy of fluorescence lifetime measurements without increasing the complexity and cost of the imaging system. The development and application of this innovative technology is expected to inject new vitality into the field of optical microscopy, further promote in-depth exploration of related research, and contribute to the vigorous development of related fields. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention provides a fluorescence lifetime super-resolution imaging method based on deep learning, which realizes fluorescence lifetime super-resolution microscopy imaging in the traditional confocal fluorescence lifetime imaging system, breaks through the spatial resolution limitation of FLIM technology, and solves the problems raised by the background technology. Technical Solution

[0007] To achieve the above objectives, the present invention provides the following technical solution: a fluorescence lifetime super-resolution imaging method based on deep learning, the specific steps of which are as follows:

[0008] Step 1: Perform fluorescence microscopy imaging on the sample to obtain a confocal intensity image and stimulated emission depletion (STED) intensity image at the same location;

[0009] Step 2: perform registration on the acquired confocal intensity image and stimulated emission depletion (STED) intensity image;

[0010] Step 3: The registered confocal intensity image and stimulated emission depletion (STED) intensity image are used as input and ground truth, respectively, and form a dataset in pairs;

[0011] Step 4: The data set is divided into training set and validation set according to a certain ratio;

[0012] Step 5: Build the network, select hyperparameters and optimizer;

[0013] Step 6: Input the training set into the network for training;

[0014] Step 7: The network is forward and backward propagated to update the weights and determine whether the network has converged. If not, repeat step 6.

[0015] Step 8: Determine whether the network performs well on the validation set. If not, repeat step 5.

[0016] Step 9: Prepare samples for FLIM imaging;

[0017] Step 10: Acquire FLIM data from the confocal FLIM microscope system;

[0018] Step 11: Combine the super-resolution intensity information with the fluorescence lifetime information to obtain a FLIM super-resolution image.

[0019] Preferably, the sample in step 1 is a fluorescently stained sample, and confocal fluorescence lifetime imaging is performed on it to obtain FLIM data containing fluorescence spatiotemporal information.

[0020] Preferably, in step six, the trained network is loaded, the confocal FLIM data is read, the intensity part is taken, and after scaling, it is input into the network test to obtain super-resolution intensity information I(x, y).

[0021] Preferably, in the step eleven, the fluorescence decay curve of the FLIM data is fitted to obtain the fluorescence lifetime value of each pixel, i.e., τ(x, y), where τ(x, y) represents the normal fluorescence lifetime information, and I(x, y) represents the super-resolution fluorescence intensity information. Finally, a full-one matrix ones(x, y) with the same pixels as the acquired image is generated, and then the fluorescence intensity I(x, y) and the fluorescence lifetime τ(x, y) are normalized, and the normalized fluorescence intensity and fluorescence lifetime information are respectively used as brightness and color and merged with the full-one matrix into a three-channel HSV image. The HSV image is converted into an RGB image to obtain an intensity-weighted fluorescence lifetime image, i.e., a fluorescence lifetime super-resolution image.

[0022] Preferably, the step 10 of obtaining FLIM data from the confocal FLIM microscope system is divided into two parts, one part is: taking the intensity information of the FLIM data, scaling it according to a certain mean and variance, and inputting it into the trained network to obtain super-resolution intensity information; the other part is: taking the FLIM data and fitting the fluorescence decay curve to obtain fluorescence lifetime information.

[0023] Preferably, a stage is required during use of the sample, and the stage is used to place and fix the sample to be tested and to perform three-dimensional movement control on the sample.

[0024] Compared with the existing technology, the present invention provides a fluorescence lifetime super-resolution imaging method based on deep learning, which has the following beneficial effects:

[0025] This deep learning-based fluorescence lifetime super-resolution imaging method involves the acquisition of paired fluorescence images, specifically non-super-resolution and super-resolution images of the same sample location. A deep learning model is then constructed and trained using a large number of paired fluorescence intensity image data. When fed a diffraction-limited non-super-resolution image, the model outputs a high-quality super-resolution fluorescence intensity image. Furthermore, this super-resolution intensity image is combined with fluorescence lifetime information after fluorescence lifetime fitting to produce a super-resolution fluorescence lifetime image. The proposed method significantly improves the spatial resolution of fluorescence lifetime images, providing powerful technical support for research in fields such as biology, chemistry, and materials science. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1 is a flow chart of a fluorescence lifetime super-resolution imaging method based on deep learning proposed by the present invention;

[0027] FIG2 is a schematic diagram of a fluorescence lifetime super-resolution imaging system based on deep learning proposed in the present invention;

[0028] FIG3 is a schematic diagram of the laser pulse and fluorescence decay curves proposed in the present invention;

[0029] FIG4 is a comparison diagram of the low-resolution fluorescence intensity image and the super-resolution fluorescence lifetime image proposed in the present invention. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0031] Please refer to Figure 1 for a deep learning-based fluorescence lifetime super-resolution imaging method. The specific steps are as follows:

[0032] Step 1: Perform fluorescence microscopy on the sample to obtain a confocal intensity image and stimulated emission depletion (STED) intensity image at the same position. The sample is a fluorescently stained sample, and confocal fluorescence lifetime imaging is performed on it to obtain FLIM data containing fluorescence spatiotemporal information. The sample requires a stage during use, which is used to place and fix the sample to be tested and perform three-dimensional movement control of the sample;

[0033] Step 2: perform registration on the acquired confocal intensity image and stimulated emission depletion (STED) intensity image;

[0034] Step 3: The registered confocal intensity image and stimulated emission depletion (STED) intensity image are used as input and ground truth, respectively, and form a dataset in pairs;

[0035] Step 4: The data set is divided into training set and validation set according to a certain ratio;

[0036] Step 5: Build the network, select hyperparameters and optimizer;

[0037] Step 6: Input the training set into the network for training, load the trained network, read the confocal FLIM data, take its intensity part, input it into the network test after scaling, and obtain the super-resolution intensity information I(x,y);

[0038] Step 7: The network is forward and backward propagated to update the weights and determine whether the network has converged. If not, repeat step 6.

[0039] Step 8: Determine whether the network performs well on the validation set. If not, repeat step 5.

[0040] Step 9: Prepare samples for FLIM imaging;

[0041] Step 10: Obtain FLIM data from the confocal FLIM microscope. Obtaining FLIM data from the confocal FLIM microscope is divided into two parts: one part is to obtain the intensity information of the FLIM data, scale it according to a certain mean and variance, and then input it into the trained network to obtain super-resolution intensity information I(x,y); the other part is to obtain the FLIM data and fit the fluorescence decay curve to obtain the fluorescence lifetime information τ(x,y);

[0042] Step 11: Combine the super-resolution intensity information with the fluorescence lifetime information to obtain a FLIM super-resolution image. Fit the fluorescence decay curve of the FLIM data to obtain the fluorescence lifetime value of each pixel, i.e., τ(x,y). τ(x,y) represents the normal fluorescence lifetime information, while I(x,y) represents the super-resolution fluorescence intensity information. Finally, generate an all-one matrix ones(x,y) with the same pixels as the acquired image. Then normalize the fluorescence intensity I(x,y) and fluorescence lifetime τ(x,y). The normalized fluorescence intensity and fluorescence lifetime information are respectively used as brightness and color and merged with the all-one matrix into a three-channel HSV image. Convert the HSV image into an RGB image to obtain an intensity-weighted fluorescence lifetime image, i.e., a fluorescence lifetime super-resolution image.

[0043] For network training, data acquisition involves capturing confocal intensity images and stimulated emission depletion (STED) super-resolution intensity images from the same sample location each time. These images are acquired using a commercial STED super-resolution microscope system. In this imaging system, when the STED laser power is zero, confocal intensity images are obtained; when the STED laser power is constant, STED super-resolution intensity images are obtained. Typically, due to system undercalibration and drift between imaging modes, confocal and STED images from the same field of view exhibit some rigid or non-rigid distortion. Therefore, image registration is required between these data pairs. This is achieved using feature point detection and pairing based on ORB (Oriented FAST and Rotated BRIEF). At this point, both low-resolution (confocal) and super-resolution (STED) intensity images are obtained and serve as the input and ground truth (GT), respectively, forming the dataset. Afterwards, m groups of data are generated by random combination as training sets and n groups as validation sets (m,n∈N,m:n=5:1). The data constituting the training set and the data constituting the validation set must be strictly separated.

[0044] This algorithm is further configured as follows: For network training: mini batch training (batch_size=2 k ,k∈N) to accelerate training. The loss function is loss=α×MSELoss(X,Y)+β×SSIMLoss(X,Y), where X is the network output and Y is the ground truth. MSELoss is the root mean square error to ensure that the individual pixel values ​​of the output conform to the ground truth, and SSIMloss is the structural similarity error to ensure that the global structural features of the output conform to the ground truth. α and β represent the coefficients of the two errors, respectively. The optimizer uses the RMSprop optimizer, with a learning rate of lr=3e-4. To accelerate network convergence, each batch is first scaled to the range [-1,1] before being fed into the network during training. The network epochs are set to 500 or an early stop module is used. Network performance is verified on the validation set every five epochs.

[0045] Data collection and testing: After the network has been trained to convergence and performs well on the validation set, testing can begin. Fluorescently stained samples are prepared and subjected to confocal fluorescence lifetime (confocal FLIM) imaging to obtain FLIM data containing spatiotemporal fluorescence information. The trained network is then loaded, and the confocal FLIM data is read. The intensity portion is extracted and scaled before being fed into the network for testing. This yields super-resolution intensity information, I(x,y). The fluorescence decay curve of the FLIM data is then fitted to obtain the fluorescence lifetime value for each pixel, τ(x,y). τ(x,y) represents the normalized fluorescence lifetime information, while I(x,y) represents the super-resolution fluorescence intensity information. Finally, an all-ones matrix, ones(x,y), is generated, containing the same pixels as the acquired image. The fluorescence intensity I(x,y) and fluorescence lifetime τ(x,y) are then normalized. The normalized fluorescence intensity and lifetime information are then merged with the all-ones matrix to form a three-channel HSV image, representing brightness and color, respectively. Converting the HSV image to an RGB image yields an intensity-weighted fluorescence lifetime image, a super-resolution fluorescence lifetime image that incorporates both normal fluorescence lifetime information and super-resolution structural information. Based on this principle, the present invention achieves super-resolution fluorescence lifetime microscopy within a conventional confocal fluorescence lifetime imaging system, overcoming the spatial resolution limitations of FLIM technology.

[0046] As shown in Figure 2, in the process of steps 1 to 11 above, you need to use:

[0047] Laser, outputting picosecond pulse laser;

[0048] Half-wave plate, used to adjust the polarization direction of the laser;

[0049] Polarization beam splitter, used for laser beam splitting, can be used in conjunction with a half-wave plate to control the energy ratio of reflected and transmitted laser light;

[0050] Reflector, used to change the transmission direction of laser;

[0051] A dichroic mirror, used to transmit excitation light and reflect fluorescence signals;

[0052] Galvanometer x, used for performing horizontal synchronous scanning of the two laser beams;

[0053] Galvanometer y is used to perform longitudinal synchronous scanning of the two laser beams and work with galvanometer x to achieve area array imaging of the sample;

[0054] The scanning lens is placed after the galvanometer and is used to collect the laser beam for area array scanning;

[0055] Tube lens, which works with the objective lens to form a microscope system;

[0056] The objective lens is used to focus the laser onto the sample and collect the fluorescence signal reflected by the sample;

[0057] The stage is used to place and fix the sample to be tested and to control the three-dimensional movement of the sample;

[0058] A lens, used to focus the light beam;

[0059] Filters are used to transmit fluorescence, remove stray light other than fluorescence, and improve the image signal-to-noise ratio;

[0060] Detector 1, using a photomultiplier tube or avalanche photodiode, is used to collect signals and amplify the fluorescence signal;

[0061] Detector 2 is used to detect the laser reflected by the polarization beam splitter in the Gaussian excitation light path as a reference signal in fluorescence lifetime imaging;

[0062] Time-correlated single-photon counter (TCSPC), used to record the spatiotemporal information of fluorescence signals;

[0063] Computer, used to control the software to acquire images, store data and process image data, etc.

[0064] As shown in Figure 2, this system uses only one picosecond pulsed laser source. After emission, the laser's polarization direction is adjusted by a half-wave plate and then split into two by a polarizing beam splitter. The reflected light is collected by detector 2 and used as a reference signal for fluorescence lifetime imaging. The transmitted light is reflected by the reflector, passes through a dichroic mirror, and is scanned by galvanometer mirrors in the x and y directions. It then passes through a scanning lens, a tube lens, and an objective lens before being focused on the sample. After being irradiated by the laser, the sample emits fluorescence. The fluorescence signal is collected by the objective lens and then returns along the same path. After passing through the tube lens, scanning lens, and galvanometer mirror, it is reflected by the dichroic mirror, focused by a lens, filtered by a filter, and finally reaches detector 1. The TCSPC simultaneously collects the fluorescence signal from detector 1 and the reference signal from detector 2, and transmits the data to a computer for storage and processing.

[0065] [Corrected 24 April 2025 in accordance with Regulation 26] As shown in Figure 3, the excitation light pulse frequency is used as the detection period (T) of the fluorescence signal. Therefore, a detection period includes one laser pulse and one complete spontaneous emission process. The solid black line in the figure represents the laser pulse, and the dashed black line represents the fluorescence decay curve resulting from spontaneous emission after laser irradiation. By fitting the fluorescence decay curve, the fluorescence lifetime information for each pixel in the image, i.e., τ(x,y), can be obtained.

[0066] As shown in Figure 4, the cell microtubule structure is stained with a fluorescent dye, and then FLIM data of the sample is collected using a confocal fluorescence lifetime imaging system. Figure 4a is a low-resolution confocal intensity image directly obtained based on the photon number distribution in the FLIM data, and Figure 4b is a super-resolution fluorescence lifetime image obtained using the method proposed in this patent. From the image comparison results, it can be seen that the image obtained by the present invention simultaneously displays more detailed structural information and richer fluorescence lifetime information of the sample.

[0067] The method of the present invention can be applied to any confocal fluorescence lifetime imaging system and is very suitable for dynamic imaging of living cells and studying the interaction processes between different subcellular structures, thus providing strong technical support for further exploration in the biomedical field.

[0068] It should be noted that the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or apparatus that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0069] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A fluorescence lifetime super-resolution imaging method based on deep learning, characterized in that: The specific steps are as follows: Step 1: Perform fluorescence microscopy imaging on the sample to obtain a confocal intensity image and stimulated emission depletion (STED) intensity image at the same position; Step 2: Perform registration operation on the collected confocal intensity image and STED intensity image; Step 3: The registered confocal intensity image and STED intensity image are used as input and true value respectively, and form a dataset in pairs; Step 4: The data set is divided into training set and validation set according to a certain ratio; Step 5: Build the network, select hyperparameters and optimizer; Step 6: Input the training set into the network for training, load the trained network, read the confocal FLIM data, take its intensity part, input it into the network test after scaling, and obtain the super-resolution intensity information I(x,y); Step 7: The network is forward and backward propagated to update the weights and determine whether the network has converged. If not, repeat step 6. Step 8: Determine whether the network performs well on the validation set. If not, repeat step 5. Step 9: Prepare samples for FLIM imaging; Step 10: Obtain FLIM data from the confocal FLIM microscope. Obtaining FLIM data from the confocal FLIM microscope is divided into two parts: one part is to obtain the intensity information of the FLIM data, scale it according to a certain mean and variance, and then input it into the trained network to obtain super-resolution intensity information; the other part is to obtain the FLIM data and fit the fluorescence decay curve to obtain fluorescence lifetime information; Step 11: Combine the super-resolution intensity information with the fluorescence lifetime information to obtain a FLIM super-resolution image. Fit the fluorescence decay curve of the FLIM data to obtain the fluorescence lifetime value of each pixel, i.e., τ(x,y). τ(x,y) represents the normal fluorescence lifetime information, while I(x,y) represents the super-resolution fluorescence intensity information. Finally, generate an all-one matrix ones(x,y) with the same pixels as the acquired image. Then normalize the fluorescence intensity I(x,y) and fluorescence lifetime τ(x,y). The normalized fluorescence intensity and fluorescence lifetime information are respectively used as brightness and color and merged with the all-one matrix into a three-channel HSV image. Convert the HSV image into an RGB image to obtain an intensity-weighted fluorescence lifetime image, i.e., a fluorescence lifetime super-resolution image.

2. The fluorescence lifetime super-resolution imaging method based on deep learning according to claim 1, characterized in that: The sample in step 1 is a fluorescently stained sample, and confocal fluorescence lifetime imaging is performed on it to obtain FLIM data containing fluorescence spatiotemporal information.

3. The fluorescence lifetime super-resolution imaging method based on deep learning according to claim 1, characterized in that: During use of the sample, a stage is required, which is used to place and fix the sample to be tested and to perform three-dimensional movement control on the sample.

Images