Training Method for Self-Supervised Microscopic Image Processing Neural Network

Abstract

The present application discloses a training method for a self-supervised microscopic image processing neural network, including: obtaining a fluorescence microscopic image dataset by using an optical microscopic imaging system with time-correlated single photon counting function, wherein the fluorescence microscopic image dataset includes a photon counting distribution set {G mn (t)} MN , where M and N are integers greater than 1, m ranges from 1 to M and n ranges from 1 to N, and t represents time; constructing a training dataset for a neural network including a fluorescence lifetime prediction convolutional module and / or a denoising convolutional module from the photon counting distribution set of the M×N pixel points by means of photon flow redistribution.

Description

Technical Field

[0001] The present application generally relates to a self-supervised denoising neural network for photon flow recombination based on fluorescence images, a training method for a fluorescence lifetime prediction neural network, and denoising of fluorescence images and prediction of fluorescence lifetimes using the trained neural network. Background Art

[0002] Fluorescence microscopy imaging is currently an important branch of microscopy imaging technology and can observe biological cells, tissues, and organs under in-vivo conditions. Generally, in fluorescence microscopy imaging, specific structures of a sample are specifically labeled with fluorescent components to have fluorescent molecules, and when the fluorescent molecules are excited by laser irradiation, fluorescence is generated, and a fluorescence image of the sample can be obtained by collecting the fluorescence with a detector. For in-vivo observation, it is difficult to simultaneously obtain high temporal resolution and high image quality. Often, the image quality needs to be sacrificed for extremely high temporal resolution, resulting in the useful information in the obtained fluorescence image usually being submerged in background noise, seriously affecting the performance of the fluorescence microscopy imaging system.

[0003] Regarding the problem of background denoising for microscopic images, there are already existing denoising algorithms such as mean filtering, Gaussian filtering, median filtering, non-local means denoising, wavelet transform denoising, and three-dimensional block matching denoising (BM3D). However, these algorithms are difficult to achieve good denoising effects for images with low signal-to-noise ratios such as dynamic fluorescence microscopic image sets. Although deep learning network algorithms have made great progress in the field of fluorescence microscopic image denoising, convolutional neural networks such as denoising convolutional neural network (DnCNN), fast and flexible denoising network (FFDNet), and convolutional blind denoising network (CBDNet) must be supervised and trained based on noisy-clear microscopic images. However, in application scenarios such as in vivo observation where it is difficult to obtain clear fluorescence images in the time domain, this supervised training for convolutional neural networks is difficult to complete. The training of this supervised neural network requires obtaining high-definition images for biological samples in advance. However, for the scenario of in vivo observation of samples, it is difficult to obtain high-definition fluorescence images that meet the requirements, which limits the application of supervised neural networks. In addition, although there are also neural network models based on noisy-noisy images or self-supervised training such as Noise2Noise, Noise2Void, Noise2self, and Neighbor2Neighbor in the prior art, the training of these neural network models usually requires at least two noisy images with the same signal distribution or adding a constraint term at the loss function, which also limits the application of these neural network models in fluorescence microscopic imaging denoising.

[0004] In addition, fluorescence lifetime imaging in the field of fluorescence microscopy has gradually received more and more attention in recent years. Molecules in biological samples specifically labeled with fluorescent components (such as bioluminescent dyes or antibodies) can become fluorescent molecules. Fluorescence lifetime is a special property of fluorescent molecules themselves, which can be used to distinguish different fluorescent molecules and even dynamically reflect changes in the microenvironment around fluorescent molecules as a sensor, thereby enabling highly sensitive and highly quantitative description of intracellular microenvironment changes in biological samples. A fluorescence lifetime imaging system characterizes the specific dynamic changes of cells by collecting and recording the time for fluorescent molecules to transition from the excited state energy level to the ground state (this time scale is usually in the nanosecond range). By using the principle of Fluorescence resonance energy transfer (FRET), the fluorescence lifetime changes of fluorescent molecules are collected / recorded, and the distance between two adjacent fluorescent molecules in a biological sample is inferred based on the changes in fluorescence lifetime, thereby determining / describing the intracellular microstructural changes in the biological sample. However, for fluorescence lifetime imaging, good fitting results for the fluorescence lifetime of each pixel in the final image often require collecting a sufficient number of photons. However, in order to achieve a sufficient number of photons, the imaging time of the photon-scanning system for collecting photons often needs to be extended, which makes it difficult to apply fluorescence lifetime imaging to rapid fluorescence imaging of living cells. The neural networks developed in recent years have also proposed a unique solution for this field. Neural network models such as fluorescence lifetime imaging network (FLI-Net), Few-photon fluorescence lifetime imaging (FPFLI), and fluorescence lifetime imaging based on generative adversarial network estimation (flimGANE) have provided some solutions for fluorescence lifetime prediction in the case of a small number of photons. However, these methods complete the training of the network through simulated datasets, and there is a problem of domain shift for real samples. Summary of the Invention

[0005] To address the above problems, the present application aims to propose a method for self-supervised training of a denoising network and a fluorescence lifetime prediction network based on photon flow redistribution from fluorescence images, so that a dataset for training a neural network (including a denoising or fluorescence lifetime prediction neural network) can be obtained by collecting the minimum number of fluorescence images of biological samples, especially living biological samples, and the trained neural network is used to denoise the fluorescence images of biological samples and predict the fluorescence lifetime.

[0006] According to one aspect of the present application, a method for training a self-supervised microscopic image processing neural network is provided, including:

[0007] Using an optical microscopic imaging system with time-correlated single-photon counting function to obtain a fluorescence microscopic image dataset for a biological sample, wherein the fluorescence microscopic image dataset includes a photon counting distribution set {G mn (t)} MN for M×N pixel points, where M and N are integers greater than 1, m ranges from 1 to M, n ranges from 1 to N, and t represents time;

[0008] Using the method of photon flow redistribution to construct a training dataset for a neural network including a fluorescence lifetime prediction convolution module and / or a denoising convolution module from the photon counting distribution set of M×N pixel points. Each pixel point's photon counting distribution in the photon counting distribution set has p time intervals △t, where p is an integer greater than 1. In the photon flow redistribution, for each pixel point's photon counting distribution among the M×N pixel points, a pair of photon counting distributions after photon flow redistribution is generated. Each of the pair of photon counting distributions after photon flow redistribution also has p time intervals △t, and the photon counts of each pixel point's photon counting distribution within the p time intervals △t are randomly assigned to the p time intervals △t of the pair of photon counting distributions after photon flow redistribution. The above photon flow redistribution is repeated Q times to generate Q pairs of photon counting distribution sets after photon flow redistribution, where Q is an integer greater than 1.

[0009] Among them, select the first group of Q photon counting distribution sets after photon flow redistribution from the Q pairs of photon counting distribution sets after photon flow redistribution, and calculate using the maximum likelihood algorithm respectively to generate Q training fluorescence lifetime images and perform maximum value normalization processing to form an input set for fluorescence lifetime prediction training. And select the second group of Q photon counting distribution sets after photon flow redistribution from the Q pairs of photon counting distribution sets after photon flow redistribution, and calculate using the maximum likelihood algorithm respectively to obtain Q training fluorescence lifetime images to form a target set or a true value set for fluorescence lifetime prediction training; or

[0010] Select the first set of Q photon-counting distribution sets with photon flow redistribution from the Q pairs of photon-counting distribution sets with photon flow redistribution. For each photon-counting distribution set in the selected Q photon-counting distribution sets with photon flow redistribution, the photon-counting distributions of each pixel are summed over time and then normalized by the maximum value to generate Q fluorescence intensity images for training, so as to form the input set for image denoising training. Select the second set of Q photon-counting distribution sets with photon flow redistribution from the Q pairs of photon-counting distribution sets with photon flow redistribution. For each photon-counting distribution set in the selected second set of Q photon-counting distribution sets with photon flow redistribution, the photon-counting distributions of each pixel are summed over time and then normalized by the maximum value to generate Q fluorescence intensity images for training, so as to form the target set or ground truth set for image denoising training.

[0011] Wherein, the training data set includes the input set for fluorescence lifetime prediction training and the target set or ground truth set for fluorescence lifetime prediction training and / or the input set for image denoising training and the target set or ground truth set for image denoising training;

[0012] The method further includes training the neural network using the training data set.

[0013] Optionally, in the photon flow redistribution, the first set of Q and the second set of Q photon-counting distribution sets with photon flow redistribution selected from the Q pairs of photon-counting distribution sets with photon flow redistribution are randomly selected.

[0014] Optionally, in the photon flow redistribution, the first set of Q photon-counting distribution sets with photon flow redistribution selected from the Q pairs of photon-counting distribution sets with photon flow redistribution includes one photon-counting distribution in each pair of photon-counting distribution sets with photon flow redistribution, and the second set of Q photon-counting distribution sets with photon flow redistribution selected from the Q pairs of photon-counting distribution sets with photon flow redistribution includes the other photon-counting distribution in one photon-counting distribution in each pair of photon-counting distribution sets with photon flow redistribution.

[0015] Optionally, before calculating using the maximum likelihood algorithm, the data of each pixel to be calculated is superimposed / merged with the pixel points with a correlation degree (Bin) of 0, 1 or 2 adjacent to it.

[0016] Optionally, the model used to build the neural network includes but is not limited to a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.

[0017] Optionally, the loss function of the neural network is:

[0018]

[0019] Among them, represents the overall loss function for building the entire neural network, represents the loss function of the fluorescence lifetime prediction convolutional module, represents the loss function of the denoising convolutional module, and μ is a number between 0 and 1. Among them, the model for constructing the loss function includes, but is not limited to, the mean squared error loss function (MSE), the cross-entropy loss function (CE), or the L2 loss function.

[0020] Optionally, the output features of the model for building the neural network respectively pass through the fluorescence lifetime prediction convolutional module and the denoising convolutional module independently.

[0021] Optionally, the fluorescence microscopy image dataset obtained by using an optical microscopy imaging system with time-correlated single-photon counting function is a fluorescence microscopy image dataset or a selected fluorescence microscopy image dataset from multiple obtained fluorescence microscopy image datasets.

[0022] According to another aspect of the present application, there is also provided a method for processing a fluorescence microscopy image dataset obtained by using an optical microscopy imaging system with time-correlated single-photon counting function, including:

[0023] Constructing a neural network including a fluorescence lifetime prediction convolutional module and / or a denoising convolutional module;

[0024] Training the neural network by using the foregoing training method; and

[0025] Processing the fluorescence microscopy image dataset obtained by using an optical microscopy imaging system with time-correlated single-photon counting function by using the trained neural network, and the processing includes fluorescence lifetime prediction and / or denoising.

[0026] According to another aspect of the present application, there is also provided a computer program product, including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the steps of the foregoing method are implemented.

[0027] By adopting the above technical means of the present application, it is possible to obtain a dataset sufficient in quantity for training a denoising neural network and a fluorescence lifetime prediction neural network only by photon flow redistribution of a single fluorescence image containing photon counting for a biological sample, avoiding the deficiency in traditional neural network training that at least two fluorescence images with the same signal distribution must be collected, enabling truly fully self-supervised neural network training, simplifying the acquisition process of neural network training and the difficulty of obtaining the training dataset, and more importantly, providing a basis for the easy implementation of denoising of fluorescence microscopic images of living biological samples and fluorescence lifetime prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The principles and various aspects of the present application can be more comprehensively understood from the following detailed description in conjunction with the following drawings. It should be noted that the scales of the various drawings may be different for the purpose of clear illustration, but this will not affect the understanding of the present application. In the drawings:

[0029] Figure 1A Schematically shows the distribution diagram of photons (or photon flow) recorded by a certain pixel point in the image taken by an optical microscopy imaging system with time-correlated single-photon counting function;

[0030] Figure 1B and 1C Schematically shows the result of photon flow redistribution for Figure 1A the photon distribution;

[0031] Figure 2 Schematically shows the result of photon flow redistribution for all pixel points;

[0032] Figure 3 Schematically shows the result of multiple photon flow redistributions for all pixel points;

[0033] Figure 4 Schematically shows the block diagram of an optical microscopy imaging system with time-correlated single-photon counting function according to an embodiment of the present application;

[0034] Figure 5 Schematically shows the flowchart of a method for denoising a fluorescence image of a biological sample and / or predicting a fluorescence lifetime according to an embodiment of the present application;

[0035] Figure 6A and 6B Schematically show the cases when the correlation degree Bin between adjacent pixel points for superposition / merging is 1 and 2 respectively;

[0036] Figure 7A and 7BSchematically shown are the image results of fluorescence lifetime prediction and denoising using a trained neural network. Detailed implementation manners

[0037] In the various drawings of the present application, features with the same structure or similar functions are denoted by the same reference numerals.

[0038] Figure 1A Schematically shown is a distribution diagram of photons (or photon streams) recorded at a certain pixel point in an image captured by an optical microscopy imaging system with time-correlated single-photon counting function. It should be noted that within the scope of the present application, an optical microscopy imaging system with time-correlated single-photon counting function means that using this optical microscopy imaging system, fluorescence microscopy imaging can be performed on a biological sample (abbreviated as "fluorescent sample"), especially a living biological sample such as a biological cell, which is labeled with a biological fluorescent dye or antibody, under the irradiation of an excitation light, detecting and recording the fluorescent photons (or photon amounts) emitted by the fluorescent sample due to stimulated emission and resulting in energy level transitions, and at the same time being able to record the microscopic time required for each fluorescent photon from being excited to being emitted (the scale of this microscopic time is often at the nanosecond level). An optical microscopy imaging system with time-correlated single-photon counting function can, for example, adopt the Luminosa time-correlated single-photon counting confocal scanning system produced by PicoQuant, which is commercially available. For example, a fluorescence microscopy image (or fluorescence microscopy image dataset) with microscopic time of photon counting results after an optical microscopy imaging system captures a biological sample labeled with a biological fluorescent dye can be expressed as:

[0039] {I(m,n)} MN |{G mn (t)} MN

[0040] Wherein, m is any integer between 1 and M (M is an integer greater than 1), n is any integer between 1 and N (N is an integer greater than 1), I represents the fluorescence image data captured by the optical microscopy imaging system for the fluorescence generated by the biological sample after being irradiated by the excitation light, that is, the fluorescence intensity data of the (m,n)-th pixel point, and G mn (t) represents the photon counting distribution over time for the (m,n)-th pixel point in the fluorescence image data. The above symbol {} can, for example, represent the fluorescence intensity data for all pixel points in the entire pixel point M×N or the photon counting distribution over time for all pixel points. The total time of the photon counting distribution depends on the recording time of the above optical microscopy imaging system for photon counting. Therefore, {I(m,n)} MN can be referred to as the fluorescence intensity dataset for the entire pixel point M×N, {G mn (t)}MN It can be called the photon count distribution set for the entire pixel point M×N. Those skilled in the art know that there is a correlation between the fluorescence intensity data set and the photon count distribution set. This is because the fluorescence intensity of each pixel point is ultimately determined by the total number of photons generated by the pixel point. It can also be considered that the fluorescence intensity of each pixel point is the result of the sum of the total number of photons generated by the pixel point. The fluorescence intensity of a fluorescence image {I(m,n)} MN It is associated with the photon count distribution set of all pixels of the fluorescence image over time. For example, the fluorescence intensity of each pixel can be reflected as the sum of the photon count distribution of the pixel over time.

[0041] For example, Figure 1A Schematically shows the photon count distribution G of the (m,n)th pixel in the fluorescence microscopy image taken by an optical microscopy imaging system with time-correlated single photon counting function over time. mn (t). Figure 1A In the figure, the horizontal axis is time t. For example, in this figure, it is shown that there is a photon recording time T L , the photon recording time is divided into p equal time intervals Δt, where p is an integer greater than or equal to 1, in particular in the embodiment shown, p = 8. It should be clear that in an alternative embodiment not shown, p = 2×P, where P is an integer greater than or equal to 1. Photon recording time T L The size of depends on the relevant parameter settings of the optical microscopy imaging system and is determined by the specific operation of the optical microscopy imaging system. From a time microscopic perspective, within each time interval △t, the accumulated number of photons represents the probability of the fluorescent molecule transitioning from the excited state back to the ground state. The more accumulated photons, the greater the probability of its transition back to the ground state. As the vertical coordinate increases, the time it takes to transition from the excited state back to the ground state is longer, indicating that the probability of such an event is small. Photons within the same time interval △t can be approximately considered to have the same time to transition from the excited state to the ground state, which facilitates statistical fitting. This probability distribution also intuitively reflects the properties of the fluorescent molecule itself. Therefore, the above G mn (t) can also be called the photon distribution / photon flow distribution for the (m,n)th pixel point.

[0042] Refer to the following Figure 1A , 1B, 1C introduces a method for photon flow redistribution according to an embodiment of the present application. It should be noted that the methods or method steps mentioned in the context of the present application can be encoded and stored in the form of computer programs / instructions, and can be called and executed by a processor such as a computer chip when needed. For example, the computer programs / instructions can be stored in a data memory such as a computer-readable storage medium, a cloud server, etc., and can be called and executed by the cloud server via a corresponding computer interface or network interface.

[0043] In the technical solution of the present application, it is assumed that when training a neural network, the theoretical true values corresponding to the fluorescence imaging results and / or fluorescence lifetimes obtained for the same biological sample are the same. Starting from this assumption, the basic idea of the technical solution of the present application is to use the fluorescence microscopy image (or fluorescence microscopy image data) with microscopic time photon counting results obtained once to obtain a set of fluorescence microscopy images (or fluorescence microscopy image datasets) for training the neural network by means of photon flow redistribution.

[0044] Specifically, the photon flow redistribution for Figure 1A the photon counting distribution G mn (t) of a pixel point (the (m, n) - th pixel point) involved is as follows: First, for all photons with microscopic time information recorded related to Figure 1A a pixel point (the (m, n) - th pixel point) involved, each photon is randomly assigned to the same pixel position of two (images of the same specification) by generating random numbers, so as to generate photon counting distributions G Figure 1B and 1C as shown in G mn ′(t) and G mn ″(t) after photon flow redistribution. An example way of random assignment is, for example, to generate a random number between 0 and 1. When the random number is greater than a specific value between 0 and 1, a photon to be assigned is assigned to the photon counting distribution G mn ′(t), and when the random number is less than or equal to the specific value between 0 and 1, a photon to be assigned is assigned to the photon counting distribution G mn ″(t). Then, the photon counting distributions G mn ′(t) and G mn ″(t) are divided into p parts within the microscopic time acquisition period (such as the photon recording time T L ) with a time interval of △t, so as to obtain the photon counting distributions G_mn'(t) and G_mn”(t) corresponding to this pixel point after photon flow redistribution. Again, for example or alternatively, taking the first time interval △t in Figure 1A as an example, four photons are successively recorded within this time interval △t. Then, as inFigure 1B The first time interval Δt as shown, and as Figure 1C shown, four photons are distributed randomly in the first time interval Δt. In the context of the present application, the random distribution of numbers can be implemented by any suitable known algorithm in a computer. Then, Figure 1A the other time intervals Δt in Figure 1B can also be randomly distributed in a similar manner to the other corresponding time intervals Δt as shown in Figure 1C and as shown in Figure 1C shown. It should be clear that when redistributing the photon stream, the photons to be distributed can be distributed in the order in which the photons are recorded (rather than the order in micro-time); alternatively, it can also be distributed in the order of micro-time or in segments of the order of micro-time (such as the order of each time interval as shown in Figure 1A ).

[0045] The process of redistributing the photon stream for a single pixel can be extended to all pixels. Thus, as shown in Figure 2 , for the photon count distribution over time (shown on the left in the figure) {G mn (t)} MN for all pixels in the entire pixel array M×N, two (or a pair) of photon count distribution sets {G mn ′(t)} MN and {G mn ″(t)} MN (shown on the right in the figure) can be obtained by using the process of redistributing the photon stream. The process of redistributing the photon stream for the above-mentioned dataset of the entire pixel array M×N is repeated Q times to finally obtain 2×Q photon count distribution sets, which can be represented as {G mnq ′(t)} MN and {G mnq ″(t)} MN respectively, where Q is an integer greater than 1, and the subscript q ranges from 1 to Q, as shown in Figure 3 .

[0046] In an embodiment of the present application, considering the hardware limitations of the optical imaging system, the maximum number of photons in the photon count distribution over time recorded by the optical imaging system for all pixels in the entire pixel array M×N is not high. Therefore, in the above process of redistributing the photon stream, for each pixel, only one photon count distribution G mn (t) is randomly used to generate two photon count distributions G mn ′(t) and G mnIt is implemented in the manner of ″(t). Of course, those skilled in the art should understand that on the premise that the maximum number of photons in the photon count distribution over time for all pixel points in the entire pixel point recorded by the optical imaging system is sufficient to support, during the above-mentioned photon stream redistribution process, each pixel point can also have a photon count distribution G each time mn It is implemented by randomly generating more than two photon count distributions of (t).

[0047] Therefore, through the above-mentioned photon stream redistribution process, for the photon count distribution {G mn (t)} MN in the dataset of M×N for the entire pixel point, 2×Q photon count distribution sets {G mnq ′(t)} MN and {G mnq ″(t)} MN can be obtained. Or it can also be called Q pairs of photon count distribution sets {G mnq ′(t)} MN and {G mnq ″(t)} MN . Among them, because each photon count distribution set is independently generated in the manner of random numbers as described above, it can be considered that the fluorescence imaging results and / or the theoretical true values corresponding to the fluorescence lifetimes obtained for the same biological sample reflected by them are the same. From this point of view, a training set for training a neural network (such as a neural network for denoising fluorescence images and / or a neural network for predicting photon lifetimes) can be generated from these 2×Q photon count distribution sets.

[0048] According to an embodiment of the present application, for the q-th pair of photon count distribution sets in the Q pairs of photon count distribution sets, for example, the two photon count distribution sets {G mnq ′(G)} MN and {G mnq ″(t)} MN are used to calculate and obtain two fluorescence lifetime images through the maximum likelihood algorithm familiar to those skilled in the art; then, these two fluorescence lifetime images are subjected to maximum value normalization processing familiar to those skilled in the art to generate two corresponding training fluorescence lifetime images, for example, G′ q and G″ q . One of the training fluorescence lifetime images, for example, G′ q is stored in the input set for fluorescence lifetime prediction training, and the other training fluorescence lifetime image, for example, G″ q is stored in the target set or true value set for fluorescence lifetime prediction training. The above process is repeated for all photon count distribution sets (for example, Q pairs), and finally, the input set {G' q} Qand the target set or true value set {G” for fluorescence lifetime prediction training q} Q .

[0049] According to another embodiment of the present application, Q is added to two photon count distributions in the q-th pair of photon count distributions in the photon count distribution set, such as G mn ′(t) and G mn ″(t) respectively in time to obtain two fluorescence intensity images; then, these two fluorescence lifetime images for training are subjected to maximum normalization well-known to those skilled in the art to generate two corresponding fluorescence intensity images for training, such as I′ q and I″ q . One of the fluorescence intensity images for training, such as I′ q , is stored in the input set for fluorescence denoising prediction training, and the other fluorescence intensity image for training, such as I″ q , is stored in the target set or true value set for fluorescence denoising prediction training. The above process is repeated for all photon count distribution sets (such as Q pairs), and finally the input set {I' q} Q for denoising training and the target set or true value set {I” q} Q are obtained.

[0050] Then, using the input set {G' q} Q for fluorescence lifetime prediction training and the target set or true value set {G” q} Q obtained in the above two embodiments, as well as the input set {I' q} Q for denoising training and the target set or true value set {I” q} Q the constructed denoising neural network and fluorescence lifetime prediction neural network can be trained respectively.

[0051] Figure 4A block diagram of an optical microscopy imaging system with time-correlated single-photon counting function according to an embodiment of the present application is schematically shown. For example, it includes an imaging module 100 and a data processing module 200. The imaging module 100 includes a fluorescence imaging device and a single-photon counting device 110. For example, the single-photon counting device 110 may include a picosecond pulsed laser, a single-photon detector, and a time-correlated single-photon counting acquisition device. When a biological sample 300 treated with a fluorescent reagent is placed on a workbench 400, the picosecond pulsed laser can emit a laser pulse towards the biological sample 300 as needed to correspondingly excite a fluorescence signal in the biological sample 300. At the same time, the excited fluorescence signal can be detected by the single-photon detector, and the information of the recorded photons (for example, the time it takes for the photons to transition from the excited state to the ground state) is sent to the time-correlated single-photon counting acquisition device for storage as specific encoded information (for example, as photon counting (distribution) data). Such cyclic recording is repeated until the specified photon recording time is reached. The data processing module 200 can include processing for the fluorescence image data captured by the fluorescence imaging device and the photon counting (distribution) data received from the imaging module 100. The photon counting (distribution) data recorded in the above single-photon counting manner has the characteristic of being independent and uncorrelated with each other for the data of each recorded photon.

[0052] Figure 5 A flowchart of a method for denoising a fluorescence image and / or predicting a fluorescence lifetime of a biological sample (which can be a living biological sample) according to an embodiment of the present application is schematically shown. First, in step S10, at least one fluorescence microscopy image dataset is acquired using an optical microscopy imaging system with time-correlated single-photon counting function. The fluorescence microscopy image dataset may include a fluorescence intensity dataset and a photon counting distribution set, and the fluorescence microscopy image dataset can be represented as

[0053] {I(m,n)} MN |{G mn (t)} MN

[0054] where m is any integer between 1 and M (M is an integer greater than 1), n is any integer between 1 and N (N is an integer greater than 1), I represents the fluorescence image data obtained after the fluorescence generated by the biological sample irradiated with the excitation light is received by the optical microscopy imaging system, that is, the fluorescence intensity data of the (m,n)th pixel point, and G mn (t) represents the photon counting distribution over time for the (m,n)th pixel point in the fluorescence image data, and {} represents the fluorescence intensity data for all pixel points in the entire pixel point M×N or the photon counting distribution over time for all pixel points.

[0055] In step S20, a data set for training a neural network (such as a denoising neural network or a fluorescence lifetime prediction neural network) is constructed. For example, the training data set for the fluorescence lifetime prediction neural network and the training data set for the denoising neural network can be constructed independently of each other or simultaneously.

[0056] Specifically for the implementation of the above step S20, as a non-limiting example, using a photon count distribution set {G mn (t)} MB in the fluorescence microscopy image data set obtained in step S10, 2×Q photon count distribution sets {G mnq ′(t)} MN and {G mnq ″(t)} Mj are reconstructed by the method of photon flow redistribution described above as the training data set for the fluorescence lifetime prediction neural network. According to a non-limiting example, Q photon count distribution sets can be randomly selected from the 2×Q photon count distribution sets {G mnq ′(t)} MN and {G mnq ″(t)} MN , and the fluorescence lifetime image is calculated for each selected photon count distribution set using the maximum likelihood algorithm familiar to those skilled in the art, so as to finally obtain Q training fluorescence lifetime images to form the training input set for fluorescence lifetime prediction; then, each of the remaining Q photon count distribution sets in the 2×Q photon count distribution sets {G mnq ′(t)} MN and {G mnq ″(t)} MN is respectively used to calculate the fluorescence lifetime image using the maximum likelihood algorithm familiar to those skilled in the art, so as to finally obtain Q training fluorescence lifetime images to form the training target set or true value set for fluorescence lifetime prediction.

[0057] In another non-limiting example, the Q photon count distribution sets {G mnq ′(t)} MN can be used to calculate the fluorescence lifetime image using the maximum likelihood algorithm familiar to those skilled in the art, so as to finally obtain Q training fluorescence lifetime images to form the training input set for fluorescence lifetime prediction; then, the Q photon count distribution sets {G mnq ″(t)} MN are used to calculate the fluorescence lifetime image using the maximum likelihood algorithm familiar to those skilled in the art, so as to finally obtain Q training fluorescence lifetime images to form the training target set or true value set for fluorescence lifetime prediction.

[0058] Those skilled in the art should clearly understand that the fluorescence lifetime images in the above step S20 can be subjected to maximum normalization before being used as images in the input set, target set, or ground truth set for fluorescence lifetime prediction training.

[0059] The above input set for fluorescence lifetime prediction training and the above target set or ground truth set for fluorescence lifetime prediction training constitute a training dataset for a neural network for fluorescence lifetime prediction.

[0060] In the technical solution of this application, considering from a spatial perspective, to improve accuracy, the data of the pixel points to be calculated can be superimposed / merged with the data of their adjacent pixel points, and then the maximum likelihood algorithm is used for calculation. The degree of association between adjacent pixel points for superimposition / merging can be represented by Bin. For example Figure 6A and Figure 6B respectively show the cases of Bin = 1 or 2. As Figure 6A shown, in the case of Bin = 1, the black squares in the figure represent the pixel points to be calculated using the maximum likelihood algorithm, and their adjacent pixel points with a step size of 1 pixel are represented by hollow squares. The photon count distributions G mn (t) represented by these hollow squares are respectively superimposed / merged into the photon count distribution G mn (t) represented by the black squares, and then the maximum likelihood algorithm is used for calculation. As Figure 6B shown, in the case of Bin = 2, the black squares in the figure represent the pixel points to be calculated using the maximum likelihood algorithm, and their adjacent pixel points with a step size of 2 pixels are represented by hollow squares. Those skilled in the art should clearly understand that when the pixel points represented by the black squares are at the boundary of the image, the photon count distributions to be superimposed / merged outside the corresponding boundary can be regarded as zero. The process of superimposition / merging can be simply understood as summing the photon counts recorded for each time interval △t for these photon count distributions G mn (t). Those skilled in the art should clearly understand that for the case of Bin = 0, the above superimposition / merging is not required, and the maximum likelihood algorithm is directly used to calculate the corresponding photon count distribution set.

[0061] In addition, according to another non - restrictive example, using a photon count distribution set {G mn (t)} MN in the fluorescence microscopy image dataset obtained in step S10, 2×Q photon count distribution sets {G mnq ′(t)} MN and {G mnq ″(t)} MNAfter that, they can be summed over time to create a training dataset for the denoising neural network. According to a non - limiting example, Q photon - counting distribution sets can be randomly selected from the 2×Q photon - counting distribution sets {G mnq ′(t)} mN and {G mnq ″(t)} MN For each selected photon - counting distribution set, the photon - counting distributions for each pixel are summed over time and then subjected to maximum normalization, which is well - known to those skilled in the art, to generate Q training fluorescence intensity images, which are used to form the input set for image denoising training. Then, for each of the remaining Q photon - counting distribution sets in the 2×Q photon - counting distribution sets {G mnq ′(t)} MN and {G mnq ″(t)} MN each photon - counting distribution set is summed over time and then subjected to maximum normalization, which is well - known to those skilled in the art, to generate Q training fluorescence intensity images, which are used to form the target set or ground - truth set for image denoising training. According to another non - limiting example, the Q photon - counting distribution sets {G mnq ′(t)} MN can be summed over time respectively and then subjected to maximum normalization, which is well - known to those skilled in the art, to generate Q training fluorescence intensity images, which are used to form the input set for image denoising training; and the Q photon - counting distribution sets {G mnq ″(t)} MN can be summed over time respectively and then subjected to maximum normalization, which is well - known to those skilled in the art, to generate Q training fluorescence intensity images, which are used to form the target set or ground - truth set for image denoising training.

[0062] The above - mentioned input set for image denoising training and the above - mentioned target set or ground - truth set for image denoising training constitute the training dataset for the denoising neural network.

[0063] Those skilled in the art should be clear that the fluorescence microscopy image dataset used to construct the dataset for training the neural network in step S20 can be one fluorescence microscopy image dataset obtained by using an optical microscopy imaging system with time - correlated single - photon counting function in step S10 or one selected from multiple fluorescence microscopy image datasets obtained by using an optical microscopy imaging system with time - correlated single - photon counting function in step S10.

[0064] In step S30, a neural network for predicting the fluorescence lifetime of the dataset obtained by the optical microscopy imaging system and / or a neural network for denoising the fluorescence image are constructed. The construction of the neural network can select neural networks familiar to those skilled in the art, including but not limited to U-shaped neural network models, residual neural network models, residual channel attention convolutional neural network models, or Fourier channel attention convolutional neural network models. The selected neural network, such as the UNet neural network model or other neural network models, is configured to extract features from the input dataset with time information, and the extracted features can be respectively sent to two convolutional modules (the fluorescence lifetime prediction convolutional module and the denoising convolutional module), and the two modules can respectively output the denoised image and the fluorescence lifetime image. In this case, the loss function of the neural network is:

[0065]

[0066] where represents the overall loss function of the entire constructed neural network, represents the loss function of the fluorescence lifetime prediction convolutional module, represents the loss function of the denoising convolutional module, and μ is a number between 0 and 1. For example, when μ = 0, it can be considered that the constructed neural network is specifically used for denoising, and when μ = 1, it can be considered that the constructed neural network is specifically used for fluorescence lifetime prediction. By adjusting the value of μ (for example, adjusting between 0 and 1), the processing weights between denoising and fluorescence lifetime in the constructed neural network can be correspondingly adjusted.

[0067] Those skilled in the art should be aware that the constructed neural network can also only include the fluorescence lifetime prediction convolutional module or the denoising convolutional module.

[0068] In step S40, the neural network constructed in step S30 is trained using the training dataset for the fluorescence lifetime prediction neural network and / or the training dataset for the denoising neural network constructed in step S20. For example, during the training of the neural network, the initial learning rate can be set to 1×10 -4 , the training batch size is 1, and the Adam optimizer is used for backpropagation iterative optimization.

[0069] In step S50, the trained neural network is used to perform denoising and / or fluorescence lifetime prediction processing on the fluorescence microscopy image dataset obtained by the optical microscopy imaging system with time-correlated single photon counting function.

[0070] According to a non-limiting example, when the neural network is simultaneously used for denoising and fluorescence lifetime prediction, the denoised image and the fluorescence lifetime image can be combined for pseudo-color display.

[0071] Figure 7A It shows that the photon distribution set (represented by G) obtained by using an optical microscopy imaging system with time-correlated single-photon counting function is processed by a trained neural network (represented by S) to obtain a fluorescence lifetime prediction image (represented by G'). Figure 7B It shows that the fluorescence image (represented by I) obtained by using an optical microscopy imaging system with time-correlated single-photon counting function is processed by a trained neural network (represented by S) to obtain a denoised fluorescence image (represented by I'). According to an embodiment of the present application, the neural network S here may simultaneously include a fluorescence lifetime prediction convolution module and a denoising convolution module, but according to the setting of the neural network (such as the setting of μ above), the fluorescence lifetime prediction convolution module and / or the denoising convolution module can be selected respectively, and the processing weights between the two are changed accordingly when processing simultaneously.

[0072] Although specific embodiments of the present application are described in detail herein, they are given for explanatory purposes only and should not be considered as limiting the scope of the present application. In addition, those skilled in the art should be clear that the embodiments described in this specification can be used in combination with each other. Various substitutions, changes and modifications can be conceived without departing from the spirit and scope of the present application.

Claims

1. A method for training a neural network for self-supervising microscopic image processing, comprising: A fluorescence microscopic image data set for a biological sample is obtained by using an optical microscopic imaging system with a time-correlated single photon counting function, wherein the fluorescence microscopic image data set includes a photon count distribution set {G mn (t)} MN , where M and N are integers greater than 1, respectively, and m ranges from 1 to M and n ranges from 1 to N, and t represents time; A training data set for a neural network including a fluorescence lifetime prediction convolution module and / or a denoising convolution module is constructed from the photon count distribution set of the M×N pixels by means of photon flow redistribution, wherein the photon count distribution of each pixel in the photon count distribution set has p time intervals Δt, wherein p is an integer greater than 1, and in the photon flow redistribution, a pair of photon count distributions after photon flow redistribution is generated for the photon count distribution of each pixel of the M×N pixels, wherein the pair of photon count distributions after photon flow redistribution also each has p time intervals Δt and the photon counts of the photon count distribution of each pixel within the p time intervals Δt are respectively randomly allocated to the p time intervals Δt of the pair of photon count distributions after photon flow redistribution, and the photon flow redistribution is repeated Q times to generate Q pairs of photon count distribution sets after photon flow redistribution, wherein Q is an integer greater than 1, wherein a first group of Q photon count distribution sets after photon flow redistribution are selected from the Q pairs of photon count distribution sets after photon flow redistribution, and Q training fluorescence lifetime images are generated by maximum likelihood algorithm calculation and subjected to maximum value normalization processing to form a fluorescence lifetime prediction training input set, and a second group of Q photon count distribution sets after photon flow redistribution are selected from the Q pairs of photon count distribution sets after photon flow redistribution, and Q training fluorescence lifetime images are obtained by maximum likelihood algorithm calculation to form a fluorescence lifetime prediction training target set or true value set; or A first group of Q photon count distribution sets after photon flow redistribution is selected from the Q pairs of photon count distribution sets after photon flow redistribution, and for each of the selected Q photon count distribution sets after photon flow redistribution, the photon count distribution of each pixel is summed up in time and normalized by the maximum value to generate Q training fluorescence intensity images to form an input set for image denoising training, and a second group of Q photon count distribution sets after photon flow redistribution is selected from the Q pairs of photon count distribution sets after photon flow redistribution, and for each of the selected second group of Q photon count distribution sets after photon flow redistribution, the photon count distribution of each pixel is summed up in time and normalized by the maximum value to generate Q training fluorescence intensity images to form a target set or true value set for image denoising training, Wherein, the training data set includes the fluorescence lifetime prediction training input set and the fluorescence lifetime prediction training target set or true value set and / or the image denoising training input set and the image denoising training target set or true value set; The method also includes training the neural network using the training data set.

2. The training method according to claim 1, characterized in that: In the photon flux redistribution, the first group of Q and the second group of Q photon flux redistributed photon count distribution sets selected from the Q pairs of photon flux redistributed photon count distribution sets are randomly selected.

3. The training method according to claim 1, characterized in that: In the photon flux redistribution, a first group of Q photon flux redistributed photon count distribution sets selected from the Q pairs of photon flux redistributed photon count distribution sets include one photon count distribution in each pair of photon flux redistributed photon count distributions, and a second group of Q photon flux redistributed photon count distribution sets selected from the Q pairs of photon flux redistributed photon count distribution sets include another photon count distribution in one of the photon count distributions in each pair of photon flux redistributed photon count distributions.

4. The training method according to any one of claims 1 to 3, characterized in that: Before calculating using the maximum likelihood algorithm, the data of each pixel point to be calculated is superimposed / merged with the adjacent pixel points whose correlation degree (Bin) is 0, 1 or 2.

5. The training method according to claim 3 or 4, characterized in that: Models used to build the neural network include but are not limited to a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.

6. The training method according to claim 5, characterized in that: The loss function of the neural network is: in, Represents the overall loss function of the entire neural network. represents the loss function of the fluorescence lifetime prediction convolution module, Represents the loss function of the denoising convolution module, μ is a number between 0 and 1, wherein the model for constructing the loss function includes but is not limited to the mean square error loss function (MSE), the cross entropy loss function (CE), or the L2 loss function.

7. The training method according to claim 6, characterized in that: The output features of the model used to build the neural network are independently passed through a fluorescence lifetime prediction convolution module and a denoising convolution module.

8. The training method according to claim 1 or 2, characterized in that: The fluorescence microscopic image data set acquired by using the optical microscopic imaging system with time-correlated single photon counting function is a fluorescence microscopic image data set or a fluorescence microscopic image data set selected from a plurality of acquired fluorescence microscopic image data sets.

9. A method for processing a fluorescence microscopy image data set acquired using an optical microscopy imaging system with a time-correlated single photon counting function, comprising: Constructing a neural network including a fluorescence lifetime prediction convolution module and / or a denoising convolution module; Training the neural network using the training method described in any one of claims 1 to 8; as well as The trained neural network is used to process a fluorescence microscopy image data set acquired by an optical microscopy imaging system with a time-correlated single photon counting function, wherein the processing includes fluorescence lifetime prediction and / or denoising.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images