Fluorescence lifetime assisted deep learning driven fluorescence multicolor super-resolution imaging method

Abstract

The invention discloses a fluorescence lifetime-assisted deep learning-driven fluorescence multicolor super-resolution imaging method, which comprises the following steps of: 1, carrying out fluorescence microscopic super-resolution imaging on various fluorescence samples to obtain super-resolution images of different components; 2, through a degradation model, converting the super-resolution images of different components into a multi-channel super-resolution image, a single-channel low-resolution image and a classification graph for training a deep learning model; and 3, converting the single-channel low-resolution image into a multi-channel high-resolution image by using a deep learning model and combining the intensity image and the fluorescence lifetime information. According to the method, fluorescence lifetime imaging and a deep learning model are combined, improvement from single-channel low-resolution images to multi-channel high-resolution images is achieved, on the basis that only a single-channel fluorescence lifetime microscopic imaging system is used and imaging system constitution is not increased, the number of available channels and spatial resolution are remarkably improved, and the method is suitable for large-scale popularization and application. And meanwhile, the accuracy of the multi-channel super-resolution image can be maintained.

Description

Technical Field

[0001] The present invention relates to the field of optical microscope imaging, and specifically to a fluorescence lifetime-assisted deep learning-driven fluorescence multi-color super-resolution imaging method. Background Art

[0002] For the development of life sciences, ultra-high-precision observation of dynamic processes is a very important part, which requires multi-component, high-resolution biological observation methods. However, due to the existence of optical diffraction limits, traditional optical microscopy methods cannot achieve the spatial resolution required for such observations, which poses a challenge to multi-component, high-resolution, multi-color high-resolution observations. In recent years, a variety of super-resolution technologies have been proposed, such as stimulated emission depletion (STED) microscopy, structured illumination microscopy (SIM), single-molecule localization microscopy (SMLM), and minimum light flux (MINFLUX) microscopy. For multi-component imaging, that is, multi-color imaging, spectral separation methods are commonly used, using multiple light sources and detectors, and using wavelength-sensitive devices to separate signals based on the wavelength differences between different fluorescent dyes. This multi-color imaging method is widely used in multi-color super-resolution imaging. However, multi-color super-resolution optical methods are often accompanied by many limitations, such as high cost, low stability, requirements for fluorescent dyes, and destructiveness to organisms. This hinders the widespread application of high-precision biological observations.

[0003] In recent years, deep learning technology has achieved significant breakthroughs and progress in fluorescence imaging. Current applications of deep learning in fluorescence microscopy often utilize convolutional neural networks, which can automatically learn image features from large amounts of data, achieving results that are difficult to achieve with traditional algorithms. While deep learning has made significant progress in pushing the boundaries of fluorescence microscopy resolution, its application to multicolor super-resolution imaging still requires a multicolor system, and is not a one-step process. There is a lack of a deep learning method to advance from single-channel, low-resolution system images to multi-channel, super-resolution images.

[0004] Therefore, we propose a fluorescence lifetime-assisted deep learning-driven fluorescence multi-color super-resolution imaging method. Summary of the Invention

[0005] The purpose of the present invention is to provide a fluorescence lifetime-assisted deep learning-driven fluorescence multi-color super-resolution imaging method, which solves the problems raised in the background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a fluorescence lifetime-assisted deep learning-driven fluorescence multi-color super-resolution imaging method, comprising the following method steps:

[0007] Step 1: Perform fluorescence microscopy super-resolution imaging on a variety of fluorescent samples to obtain super-resolution images of different components;

[0008] Step 2: Using the degradation model, the super-resolution images of different components are converted into multi-channel super-resolution images, single-channel low-resolution images, and classification images for training the deep learning model.

[0009] Step 3: Use a deep learning model to combine intensity images and fluorescence lifetime information to convert single-channel low-resolution images into multi-channel high-resolution images.

[0010] As a preferred embodiment of the present invention, the training process of the deep learning model includes:

[0011] The single-channel low-resolution noise image and fluorescence lifetime classification map are used as the input and auxiliary information of the network respectively;

[0012] Mini batch training is used for training, and the loss function includes root mean square error and adversarial loss error.

[0013] As a preferred embodiment of the present invention, the loss function is:

[0014] los s=α×MSELoss(X,Y)+β×GANLoss(X,Y)

[0015] Among them, X is the model output, Y is the true value, and α and β are weight coefficients.

[0016] As a preferred embodiment of the present invention, the root mean square error in the loss function is used to ensure that the pixel values output by the network conform to the true values, and the adversarial loss error is used to achieve better generated image quality through adversarial training between the generator and the discriminator.

[0017] As a preferred embodiment of the present invention, the deep learning model adopts a generative network structure based on U-Net, which includes:

[0018] The downsampling part consists of convolution blocks and pooling layers;

[0019] The upsampling part consists of a convolution block and an upsampling block;

[0020] The cascade acts as the skip connection part of the network.

[0021] As a preferred embodiment of the present invention, the network structure further includes:

[0022] The FLIM attention gate is used to combine features of different scales in the network and the fluorescence lifetime classification image, providing multi-scale fluorescence lifetime information assistance.

[0023] As a preferred embodiment of the present invention, it also includes:

[0024] During the model testing phase, the fluorescence lifetime data were preprocessed, including thresholding, K-means classification, and Gaussian blurring, to obtain lifetime classification images;

[0025] The preprocessed intensity image and lifetime classification image are input into the trained deep learning model to obtain a multi-color super-resolution image.

[0026] As a preferred embodiment of the present invention, the preprocessing of the fluorescence lifetime data further includes:

[0027] Thresholding is performed on the intensity image to obtain a binary image;

[0028] Multiply the initial classification image and the binary image to obtain the threshold classification image.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention combines fluorescence lifetime imaging and deep learning models to achieve an upgrade from single-channel low-resolution images to multi-channel high-resolution images. While using only a single-channel fluorescence lifetime microscopy system and without increasing the composition of the imaging system, it achieves a significant increase in the number of available channels and spatial resolution, while ensuring that the accuracy of multi-channel super-resolution images is maintained. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0032] Figure 1 This is a flow chart of the fluorescence lifetime-assisted deep learning-driven multi-color super-resolution imaging implemented in the present invention;

[0033] Figure 2 2 is a diagram of a method for obtaining deep learning model training data implemented by the present invention;

[0034] Figure 3 It is a diagram of the deep learning model training method implemented by the present invention;

[0035] Figure 4 It is a method for processing fluorescence lifetime distribution diagrams in real data implemented by the present invention;

[0036] Figure 5 Schematic diagram of the deep learning network structure proposed in the present invention;

[0037] Figure 6 This is a schematic diagram of the convolutional block structure in the deep learning network structure proposed in the present invention;

[0038] Figure 7This is a schematic diagram of the FLIM attention gate in the deep learning network structure proposed in this invention. DETAILED DESCRIPTION

[0039] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0040] Example

[0041] like Figure 1-7 As shown, the present invention proposes an embodiment of a fluorescence lifetime-assisted deep learning-driven fluorescence multi-color super-resolution imaging method, and the specific implementation steps are as follows:

[0042] Data acquisition and preprocessing

[0043] Prepare a variety of fluorescently labeled biological samples and image them using a high-resolution fluorescence microscope to obtain super-resolution images of different components as ground truth;

[0044] By decoupling the super-resolution degradation model, namely combining, blurring and denoising, a single-channel low-resolution image is obtained, and then a certain amount of Poisson noise is added to simulate the detector noise during imaging, and the classified image is obtained through thresholding and K-means classification.

[0045] Deep learning model construction and training

[0046] Construct a generative network structure based on U-Net, which includes a downsampling part (consisting of convolution blocks and pooling layers), an upsampling part (consisting of convolution blocks and upsampling blocks), and a jump connection part. The specific structure of the network is referenced Figure 5 、 Figure 6 and Figure 7 ;

[0047] The convolution block consists of two Conv 3*3-Batch normalization-Leaky ReLU combinations. To avoid the checkerboard effect caused by the upsampling transpose matrix, the upsampling block uses the neighbor difference upsampling method and adds Conv1*1 convolution.

[0048] A FLIM attention gate is added to the network. This gate structure can combine features of different scales in the network and the fluorescence lifetime classification image to provide multi-scale fluorescence lifetime information assistance.

[0049] The simulated single-channel low-resolution noise image and fluorescence lifetime classification map are used as the network input and auxiliary information to train the deep learning model. During the training process, mini batch training is adopted, and the following loss function is used to optimize the model:

[0050] los s=α×MSELoss(X,Y)+β×GANLoss(X,Y);

[0051] Among them, X is the model output, Y is the true value, α and β are weight coefficients;

[0052] MSE Loss s is the root mean square error to ensure that each pixel value conforms to the true value. GAN loss s is the adversarial loss error. Through adversarial training between the generator and the discriminator to achieve better generated image quality, the optimizer uses the Adam optimizer and the learning rate is set to lr = 1e-5;

[0053] To speed up network convergence, each batch is first scaled to the range of [-1, 1] before being input into the network during training. The network epochs is set to 1000 or the Early Stop module is used. The network performance is verified on the validation set every 10 epochs.

[0054] Model testing and validation

[0055] The samples were imaged using a confocal microscope system to obtain intensity maps and distribution maps of single channels;

[0056] Preprocess the fluorescence lifetime data. First, use thresholding and K-means classification to classify the distribution map to obtain an initial classification image. Then, threshold the intensity image I to obtain a binary image. Next, multiply the initial classification image and the binary image to obtain a threshold classification image. Finally, Gaussian blur the threshold classification image to obtain the final lifetime classification image.

[0057] The preprocessed intensity image and lifetime classification image are input into the trained deep learning model to obtain a multi-color super-resolution image;

[0058] Compare the multi-color super-resolution image output by the model with the true value Y, and evaluate the accuracy and performance of the model by calculating indicators such as MSE Loss s.

[0059] In summary, the fluorescence lifetime-assisted deep learning-driven fluorescence multi-color super-resolution imaging method proposed in the present invention realizes the transformation from single-channel low-resolution images to multi-channel high-resolution images by combining deep learning technology and the physical information of fluorescence lifetime imaging.

[0060] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as illustrative and non-restrictive in all respects. The scope of the present invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be included therein.

[0061] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A fluorescence lifetime-assisted deep learning-driven fluorescence multi-color super-resolution imaging method, characterized by: The method comprises the following steps: Step 1: Perform fluorescence microscopy super-resolution imaging on a variety of fluorescent samples to obtain super-resolution images of different components; Step 2: Using the degradation model, the super-resolution images of different components are converted into multi-channel super-resolution images, single-channel low-resolution images, and classification images for training the deep learning model. Step 3: Use a deep learning model to combine intensity images and fluorescence lifetime information to convert single-channel low-resolution images into multi-channel high-resolution images.

2. The fluorescence lifetime-assisted deep learning-driven fluorescence multi-color super-resolution imaging method according to claim 1, characterized in that: The training process of the deep learning model includes: The single-channel low-resolution noise image and fluorescence lifetime classification map are used as the input and auxiliary information of the network respectively; Mini batch training is used for training, and the loss function includes root mean square error and adversarial loss error.

3. The fluorescence lifetime-assisted deep learning-driven fluorescence multi-color super-resolution imaging method according to claim 1, characterized in that: The loss function is: los s=α×MSELoss(X,Y)+β×GANLoss(X,Y) Among them, X is the model output, Y is the true value, and α and β are weight coefficients.

4. The fluorescence lifetime-assisted deep learning-driven fluorescence multi-color super-resolution imaging method according to claim 2, characterized in that: The root mean square error in the loss function is used to ensure that the pixel values output by the network are consistent with the true values, and the adversarial loss error is used to achieve better generated image quality through adversarial training between the generator and the discriminator.

5. The fluorescence lifetime-assisted deep learning-driven fluorescence multi-color super-resolution imaging method according to claim 1, characterized in that: The deep learning model adopts a generative network structure based on U-Net, which includes: The downsampling part consists of convolution blocks and pooling layers; The upsampling part consists of a convolution block and an upsampling block; The cascade acts as the skip connection part of the network.

6. The fluorescence lifetime-assisted deep learning-driven fluorescence multi-color super-resolution imaging method according to claim 5, characterized in that: The network structure also includes: The FLIM attention gate is used to combine features of different scales in the network and the fluorescence lifetime classification image, providing multi-scale fluorescence lifetime information assistance.

7. The fluorescence lifetime-assisted deep learning-driven fluorescence multi-color super-resolution imaging method according to claim 1, characterized in that: Also includes: During the model testing phase, the fluorescence lifetime data were preprocessed, including thresholding, K-means classification, and Gaussian blurring, to obtain lifetime classification images; The preprocessed intensity image and lifetime classification image are input into the trained deep learning model to obtain a multi-color super-resolution image.

8. The fluorescence lifetime-assisted deep learning-driven fluorescence multi-color super-resolution imaging method according to claim 7, characterized in that: The preprocessing of the fluorescence lifetime data further includes: Thresholding is performed on the intensity image to obtain a binary image; Multiply the initial classification image and the binary image to obtain the threshold classification image.

Images