Training method of self-supervised microscopic image processing neural network

Abstract

The invention discloses a training method for a self-supervised microscopic image processing neural network, and the method comprises the steps: obtaining a fluorescent microscopic image data set through employing an optical microscopic imaging system with a time-dependent single photon counting function, and enabling the fluorescent microscopic image data set to comprise a photon counting distribution set {Gmn (t)} MN for M * N pixel points, m and N are respectively integers greater than 1, m is from 1 to M, n is from 1 to N, and t represents time; and constructing a training data set for a neural network comprising a fluorescence lifetime prediction convolution module and / or a denoising convolution module from the photon counting distribution set of the M * N pixel points in a photon flow redistribution mode.

Description

Technical Field

[0001] The present application generally relates to a self-supervised denoising neural network based on photon flow recombination of fluorescence images and a training method for a fluorescence lifetime prediction neural network, and using the trained neural network to denoise fluorescence images and predict fluorescence lifetime. Background Art

[0002] Fluorescence microscopy is currently an important branch of microscopic imaging technology, which can observe biological cells, tissues, and organs under living conditions. Usually in fluorescence microscopy, the specific structure of the sample is specifically labeled with fluorescent components and has fluorescent molecules, and when the fluorescent molecules are excited by laser irradiation, fluorescence is generated, and the fluorescence image of the sample can be obtained by collecting the fluorescence with the detector. For in vivo observation, it is difficult to obtain high temporal resolution and high image quality at the same time. It is often necessary to sacrifice image quality for extremely high temporal resolution, resulting in the useful information in the obtained fluorescence image being usually submerged in background noise, which seriously affects the performance of the fluorescence microscopy system.

[0003] To address the problem of background denoising in microscopic images, there are currently denoising algorithms such as mean filtering, Gaussian filtering, median filtering, non-local mean denoising, wavelet transform denoising, and three-dimensional block matching denoising (BM3D). However, these algorithms are difficult to achieve good denoising effects for images with low signal-to-noise ratios, such as dynamic fluorescence microscopic image sets. Although deep learning network algorithms have made great progress in the field of fluorescence microscopic image denoising, convolutional neural networks such as denoising convolutional neural network (DnCNN), fast and flexible denoising network (FFDNet), and convolutional blind denoising network (CBDNet) must be supervised based on noise-clear microscopic images. However, in application scenarios such as in vivo observations where it is difficult to obtain clear fluorescence images in the time domain, this supervised training of convolutional neural networks is difficult to complete. The training of this supervised neural network requires obtaining high-definition images of biological samples in advance, but for the scene of in vivo observation of samples, it is difficult to obtain high-definition fluorescence images that meet the requirements, which limits the application of supervised neural networks. In addition, although there are neural network models based on noise image-noise image or self-supervised training such as Noise2Noise, Noise2Void, Noise2self, Neighbor2Neighbor, etc. in the prior art, the training of these neural network models usually requires at least two noise images with the same signal distribution or adding constraints to the loss function, which also limits the application of these neural network models in fluorescence microscopy denoising.

[0004] In addition, fluorescence lifetime imaging in the field of fluorescence microscopy has also gradually received more and more attention in recent years. Molecules in biological samples that are specifically labeled with fluorescent components (such as bioluminescent dyes or antibodies) can become fluorescent molecules. Fluorescence lifetime is a special property of fluorescent molecules themselves, which can be used to distinguish different fluorescent molecules, and even as a receptor to dynamically reflect the changes in the microenvironment around fluorescent molecules, so that the intracellular microenvironment changes of biological samples can be described with high sensitivity and high quantification. The fluorescence lifetime imaging system characterizes the specific dynamic changes of cells by collecting and recording the time (the time scale is usually nanoseconds) for fluorescent molecules to transition from excited state energy levels to ground states. The fluorescence lifetime changes of fluorescent molecules are collected / recorded using the principle of fluorescence resonance energy transfer (FRET), and the distance between two adjacent fluorescent molecules of biological samples is inferred by the changes in fluorescence lifetime, thereby determining / describing the intracellular microstructural changes of biological samples. However, for fluorescence lifetime imaging, the fluorescence lifetime fitting of each pixel point of the final image often requires the collection of a sufficient number of photons to obtain a good fitting effect. However, in order to achieve a sufficient number of photons, the imaging time of the scanning system for collecting photons often needs to be extended, which makes it difficult to apply fluorescence lifetime imaging to fast fluorescence imaging of living cells. The neural network developed in recent years has also proposed a unique solution for this field. Neural network models such as fluorescence lifetime imaging network (FLI-Net), few-photon fluorescence lifetime imaging (FPFLI), and fluorescence lifetime imaging based on generative adversarial network estimation (flimGANE) have provided some solutions for solving the problem of fluorescence lifetime prediction in the case of a small number of photons. However, these methods all use simulated data sets to complete the network training, which will have the problem of domain offset for real samples. Summary of the invention

[0005] In response to the above problems, the present application aims to propose a method for self-supervised training of denoising networks and fluorescence lifetime prediction networks based on fluorescence images by photon flow redistribution, so that a data set for training neural networks (including denoising or fluorescence lifetime prediction neural networks) can be obtained by collecting a minimum number of fluorescence images of biological samples, especially living biological samples, and the trained neural network can be used to denoise the fluorescence images of biological samples and predict the fluorescence lifetime.

[0006] According to one aspect of the present application, a method for training a self-supervised microscopic image processing neural network is provided, comprising:

[0007] A fluorescence microscopic image data set for a biological sample is obtained by using an optical microscopic imaging system with a time-correlated single photon counting function, wherein the fluorescence microscopic image data set includes a photon count distribution set {G mn (t)} MN , where M and N are integers greater than 1, respectively, and m ranges from 1 to M and n ranges from 1 to N, and t represents time;

[0008] A training data set for a neural network including a fluorescence lifetime prediction convolution module and / or a denoising convolution module is constructed from the photon count distribution set of the M×N pixels by means of photon flow redistribution, wherein the photon count distribution of each pixel in the photon count distribution set has p time intervals Δt, wherein p is an integer greater than 1, and in the photon flow redistribution, a pair of photon count distributions after photon flow redistribution is generated for the photon count distribution of each pixel of the M×N pixels, wherein the pair of photon count distributions after photon flow redistribution also each has p time intervals Δt and the photon counts of the photon count distribution of each pixel within the p time intervals Δt are respectively randomly allocated to the p time intervals Δt of the pair of photon count distributions after photon flow redistribution, and the photon flow redistribution is repeated Q times to generate Q pairs of photon count distribution sets after photon flow redistribution, wherein Q is an integer greater than 1,

[0009] wherein a first group of Q photon count distribution sets after photon flow redistribution are selected from the Q pairs of photon count distribution sets after photon flow redistribution, and Q training fluorescence lifetime images are generated by maximum likelihood algorithm calculation and subjected to maximum value normalization processing to form a fluorescence lifetime prediction training input set, and a second group of Q photon count distribution sets after photon flow redistribution are selected from the Q pairs of photon count distribution sets after photon flow redistribution, and Q training fluorescence lifetime images are obtained by maximum likelihood algorithm calculation to form a fluorescence lifetime prediction training target set or true value set; or

[0010] A first group of Q photon count distribution sets after photon flow redistribution is selected from the Q pairs of photon count distribution sets after photon flow redistribution, and for each of the selected Q photon count distribution sets after photon flow redistribution, the photon count distribution of each pixel is summed up in time and normalized by the maximum value to generate Q training fluorescence intensity images to form an input set for image denoising training, and a second group of Q photon count distribution sets after photon flow redistribution is selected from the Q pairs of photon count distribution sets after photon flow redistribution, and for each of the selected second group of Q photon count distribution sets after photon flow redistribution, the photon count distribution of each pixel is summed up in time and normalized by the maximum value to generate Q training fluorescence intensity images to form a target set or true value set for image denoising training,

[0011] Wherein, the training data set includes the fluorescence lifetime prediction training input set and the fluorescence lifetime prediction training target set or true value set and / or the image denoising training input set and the image denoising training target set or true value set;

[0012] The method also includes training the neural network using the training data set.

[0013] Optionally, in the photon flux redistribution, the first group of Q and the second group of Q photon flux redistributed photon count distribution sets selected from the Q pairs of photon flux redistributed photon count distribution sets are randomly selected.

[0014] Optionally, in the photon flux redistribution, a first group of Q photon flux redistributed photon count distribution sets selected from the Q pairs of photon flux redistributed photon count distribution sets include one photon count distribution in each pair of photon flux redistributed photon count distributions, and a second group of Q photon flux redistributed photon count distribution sets selected from the Q pairs of photon flux redistributed photon count distribution sets include another photon count distribution in one of the photon count distributions in each pair of photon flux redistributed photon count distributions.

[0015] Optionally, before calculating using the maximum likelihood algorithm, the data of each pixel point to be calculated is superimposed / merged with the data of its adjacent pixel points whose correlation degree (Bin) is 0, 1 or 2.

[0016] Optionally, the model used to build the neural network includes but is not limited to a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.

[0017] Optionally, the loss function of the neural network is:

[0018]

[0019] in, Represents the overall loss function of the entire neural network. represents the loss function of the fluorescence lifetime prediction convolution module, Represents the loss function of the denoising convolution module, μ is a number between 0 and 1, wherein the model for constructing the loss function includes but is not limited to the mean square error loss function (MSE), the cross entropy loss function (CE), or the L2 loss function.

[0020] Optionally, the output features of the model used to build the neural network are independently passed through a fluorescence lifetime prediction convolution module and a denoising convolution module.

[0021] Optionally, the fluorescence microscopic image data set acquired using the optical microscopic imaging system with a time-correlated single photon counting function is a fluorescence microscopic image data set or a fluorescence microscopic image data set selected from a plurality of acquired fluorescence microscopic image data sets.

[0022] According to another aspect of the present application, a method for processing a fluorescence microscopy image data set acquired by an optical microscopy imaging system with a time-correlated single photon counting function is also provided, comprising:

[0023] Constructing a neural network including a fluorescence lifetime prediction convolution module and / or a denoising convolution module;

[0024] Using the aforementioned training method to train the neural network; and

[0025] The trained neural network is used to process a fluorescence microscopy image data set acquired by an optical microscopy imaging system with a time-correlated single photon counting function, wherein the processing includes fluorescence lifetime prediction and / or denoising.

[0026] According to another aspect of the present application, a computer program product is provided, including a computer program / instruction, wherein the computer program / instruction implements the steps of the aforementioned method when executed by a processor.

[0027] By adopting the above-mentioned technical means of the present application, it is possible to obtain a data set sufficient for training a denoising neural network and a fluorescence lifetime prediction neural network by redistributing the photon flow using only a single fluorescence image containing photon counts for a biological sample, thereby avoiding the deficiency of having to collect at least two fluorescence images with the same signal distribution in traditional neural network training, and realizing truly fully self-supervised neural network training. At the same time, it simplifies the acquisition process of neural network training and the difficulty of obtaining training data sets, and provides a basis for the denoising of fluorescence microscopy images of living biological samples and the easy implementation of fluorescence lifetime prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The principles and various aspects of the present application can be more fully understood from the following detailed description in conjunction with the following drawings. It should be noted that the scales of the drawings may be different for the purpose of clear description, but this will not affect the understanding of the present application. In the drawings:

[0029] Figure 1A The schematic diagram shows the distribution of photons (or photon flow) recorded at a certain pixel in an image captured by an optical microscopic imaging system with a time-correlated single photon counting function;

[0030] Figure 1B and 1C Schematically shows the Figure 1A The result of photon flow redistribution based on the photon distribution;

[0031] Figure 2 The result of photon flow redistribution for all pixels is schematically shown;

[0032] Figure 3 The result of multiple photon flow redistribution for all pixels is schematically shown;

[0033] Figure 4 A block diagram of an optical microscopic imaging system with a time-correlated single photon counting function according to an embodiment of the present application is schematically shown;

[0034] Figure 5 A flowchart of a method for denoising a fluorescence image of a biological sample and / or predicting a fluorescence lifetime according to an embodiment of the present application is schematically shown;

[0035] Fig. 6A and 6B The cases where the correlation degree Bin between adjacent pixels for superposition / merging is 1 and 2 are schematically shown respectively;

[0036] Fig. 7A and 7BThe fluorescence lifetime prediction and image denoising results using the trained neural network are schematically shown respectively. DETAILED DESCRIPTION

[0037] In the various figures of the present application, features with the same structure or similar functions are represented by the same reference numerals.

[0038] Figure 1A The schematic diagram shows the distribution of photons (or photon flow) recorded at a certain pixel in an image captured by an optical microscopic imaging system with a time-correlated single photon counting function. It should be noted that within the scope of the present application, an optical microscopic imaging system with a time-correlated single photon counting function means that the optical microscopic imaging system can be used to perform fluorescence microscopic imaging of biological samples (referred to as "fluorescent samples") labeled with biological fluorescent dyes or antibodies, especially living biological samples such as biological cells, under the irradiation of excitation light in a manner familiar to those skilled in the art, detect and record the fluorescent photons (or photon quantity) emitted by the fluorescent samples due to energy level transitions caused by stimulated radiation, and can also record the microscopic time required for each fluorescent photon from being excited to being emitted (the scale of the microscopic time is often in the nanosecond level). The optical microscopic imaging system with a time-correlated single photon counting function can, for example, use the Luminosa time-correlated single photon counting scanning confocal system produced by PicoQuant, which is commercially available. For example, a fluorescence microscopic image (or fluorescence microscopic image data set) with microscopic time photon counting results after the optical microscopic imaging system captures a biological sample labeled with a biological fluorescent dye can be expressed as:

[0039] {I(m,n)} MN |{G mn (t)} MN

[0040] Wherein, m is any integer between 1 and M (M is an integer greater than 1), n ​​is any integer between 1 and N (N is an integer greater than 1), I represents the fluorescence image data captured by the optical microscope imaging system on the fluorescence generated by the biological sample after being irradiated with the excitation light, that is, the fluorescence intensity data of the (m, n)th pixel point, G mn (t) represents the photon count distribution over time for the (m,n)th pixel in the fluorescence image data. The above symbol {} can, for example, represent the fluorescence intensity data for all pixels in the entire pixel M×N or the photon count distribution over time for all pixels. The total time of the photon count distribution depends on the recording time of the above optical microscopy imaging system for the photon count. Therefore, {I(m,n)} MN It can be called the fluorescence intensity data set for the entire pixel point M×N, {G mn (t)}MN It can be called the photon count distribution set for the entire pixel point M×N. Those skilled in the art know that there is a correlation between the fluorescence intensity data set and the photon count distribution set. This is because the fluorescence intensity of each pixel point is ultimately determined by the total number of photons generated by the pixel point. It can also be considered that the fluorescence intensity of each pixel point is the result of the sum of the total number of photons generated by the pixel point. The fluorescence intensity of a fluorescence image {I(m,n)} MN It is associated with the photon count distribution set of all pixels of the fluorescence image over time. For example, the fluorescence intensity of each pixel can be reflected as the sum of the photon count distribution of the pixel over time.

[0041] For example, Figure 1A Schematically shows the photon count distribution G of the (m,n)th pixel in the fluorescence microscopy image taken by an optical microscopy imaging system with time-correlated single photon counting function over time. mn (t). Figure 1A In the figure, the horizontal axis is time t. For example, in this figure, it is shown that there is a photon recording time T L , the photon recording time is divided into p equal time intervals Δt, where p is an integer greater than or equal to 1, in particular in the embodiment shown, p = 8. It should be clear that in an alternative embodiment not shown, p = 2×P, where P is an integer greater than or equal to 1. Photon recording time T L The size of depends on the relevant parameter settings of the optical microscopy imaging system and is determined by the specific operation of the optical microscopy imaging system. From a time microscopic perspective, within each time interval △t, the accumulated number of photons represents the probability of the fluorescent molecule transitioning from the excited state back to the ground state. The more accumulated photons, the greater the probability of its transition back to the ground state. As the vertical coordinate increases, the time it takes to transition from the excited state back to the ground state is longer, indicating that the probability of such an event is small. Photons within the same time interval △t can be approximately considered to have the same time to transition from the excited state to the ground state, which facilitates statistical fitting. This probability distribution also intuitively reflects the properties of the fluorescent molecule itself. Therefore, the above G mn (t) can also be called the photon distribution / photon flow distribution for the (m,n)th pixel point.

[0042] Refer to the following Figure 1A , 1B1C introduces a method for photon flow redistribution according to an embodiment of the present application. It should be noted that the methods or method steps mentioned in the context of the present application can be encoded and stored in the form of computer programs / instructions, and called and executed by a processor such as a computer chip when necessary. For example, the computer program / instructions can be stored in a data storage such as a computer-readable storage medium, a cloud server, etc., and called and executed by a cloud server via a corresponding computer interface or network interface.

[0043] In the technical solution of the present application, it is assumed that when training a neural network, the theoretical true values ​​corresponding to the fluorescence imaging results and / or fluorescence lifetimes obtained for the same biological sample are the same. Based on this assumption, the basic idea of ​​the technical solution of the present application is to obtain a fluorescence microscopic image set (or fluorescence microscopic image data set) for training a neural network by means of a single fluorescence microscopic image (or fluorescence microscopic image data) with a microscopic time photon counting result obtained by photon flow redistribution.

[0044] Specifically, photon flux redistribution targets Figure 1A The photon count distribution G of a pixel involved (the (m,n)th pixel) mn The implementation process of (t) is as follows: First, for Figure 1A All the photons with microscopic time information recorded about a pixel point (the (m,n)th pixel point) are randomly assigned to the same pixel position of two images (of the same size) by generating random numbers to produce the following: Figure 1B and 1C The photon count distribution G after photon flow redistribution is shown mn ′(t) and G mn ″(t). An example of random allocation is to generate a random number between 0 and 1. When the random number is greater than a specific value between 0 and 1, a photon to be allocated is allocated to the photon counting distribution G mn ′(t), and when the random number is less than or equal to a specific value between 0 and 1, a photon to be allocated is allocated to the photon counting distribution G mn ″(t). Then, the photon count distribution G mn ′(t) and G mn ″(t) is the time microscopic acquisition period (e.g. photon recording time T L ) is divided into p parts with a time interval of △t to obtain the photon count distribution G_mn'(t) and G_mn"(t) corresponding to the pixel after the photon flow is redistributed. For example or alternatively, Figure 1A Take the first time interval △t in the example as an example. Four photons are recorded in sequence in this time interval △t. Then, Figure 1B The first time interval △t shown and Figure 1C In the first time interval Δt shown, four photons are distributed in a random number manner. In the context of this application, the distribution of random numbers can be implemented in a computer using any suitable known algorithm. Next, Figure 1A The other time intervals △t in can also be randomly assigned to the Figure 1B As shown and Figure 1C It should be clear that when redistributing the photon flow, the photons to be distributed can be distributed in the order in which the photons are recorded (rather than in the order of microscopic time); alternatively, it can also be distributed in the order of microscopic time or in segments of the order of microscopic time (for example, Figure 1A The allocation is performed in the order of the time intervals shown.

[0045] The above-mentioned photon flow redistribution process for one pixel can be extended to all pixels. Figure 2 As shown, the photon count distribution over time for all pixels in the entire pixel M×N (as shown on the left) {G mn (t)} MN The photon flow redistribution process can be used to obtain two (or a pair of) photon count distribution sets {G mn ′(t)} MN and {G mn ″(t)} MN (As shown in the right figure). The photon flow redistribution process for the entire pixel point M×N data set is repeated Q times to finally obtain 2×Q photon count distribution sets, which can be represented by {G mnq ′(t)} MN and {G mnq ″(t)} MN In other words, Q is an integer greater than 1, and the subscript q ranges from 1 to Q, such as Figure 3 shown.

[0046] In the embodiment of the present application, considering the hardware limitation of the optical imaging system, the maximum number of photons recorded by the optical imaging system for the photon count distribution over time for all the pixel points in the entire pixel point M×N is not high, so in the above-mentioned photon flow redistribution process, only one photon count distribution G is used for each pixel point at a time. mn (t) Randomly generate two photon count distributions G mn ′(t) and G mn″(t) is implemented. Of course, those skilled in the art should be aware that, under the premise that the maximum number of photons recorded by the optical imaging system for the photon count distribution of all the pixels in the entire pixel point over time is sufficient to support, in the above-mentioned photon flow redistribution process, a photon count distribution G can be obtained for each pixel point each time. mn (t) This is achieved by randomly generating two or more photon counting distributions.

[0047] Therefore, through the above-mentioned photon flow redistribution process, the photon count distribution {G mn (t)} MN We can obtain 2×Q photon counting distribution sets {G mnq ′(t)} MN and {G mnq ″(t)} MN , or it can also be called the Q-to-photon count distribution set {G mnq ′(t)} MN and {G mnq ″(t)} MN , where, because each photon count distribution set is independently generated as described above in a random number manner, it can be considered that the fluorescence imaging results and / or theoretical true values ​​corresponding to the fluorescence lifetime obtained for the same biological sample embodied by them are the same. From this point of view, a training set for training a neural network (e.g., a neural network for denoising fluorescence images and / or a neural network for predicting photon lifetime) can be generated from these 2×Q photon count distribution sets.

[0048] According to one embodiment of the present application, two photon counting distribution sets in the qth pair of photon counting distribution sets in the Q pairs of photon counting distribution sets, for example {G mnq ′(G)} MN and {G mnq ″(t)} MN Two fluorescence lifetime images are obtained by calculating the maximum likelihood algorithm familiar to those skilled in the art; then, the two fluorescence lifetime images are subjected to maximum value normalization processing well known to those skilled in the art to generate two corresponding training fluorescence lifetime images such as G′ q and G″ q , one of the training fluorescence lifetime images, such as G′ q Stored in the fluorescence lifetime prediction training input set, and another training fluorescence lifetime image such as G″ q The above process is repeated for all photon count distribution sets (such as Q pairs), and finally the input set {G' q} Qand the target set or true value set {G” for fluorescence lifetime prediction training q} Q .

[0049] According to another embodiment of the present application, two photon count distributions in the qth pair of photon count distributions in the Q pair of photon count distributions, for example, G mn ′(t) and G mn ″(t) are added in time to obtain two fluorescence intensity images; then, the two training fluorescence lifetime images are subjected to maximum value normalization processing well known to those skilled in the art to generate two corresponding training fluorescence intensity images such as I′ q and I″ q , one of the training fluorescence intensity images, such as I′ q Stored in the fluorescence denoising prediction training input set, and another training fluorescence intensity image such as I″ q The above process is repeated for all photon count distribution sets (such as Q pairs), and finally the denoising training input set {I' q} Q And the target set or true value set for denoising training {I” q} Q .

[0050] Then, the fluorescence lifetime prediction training input set {G′ q} Q and the target set or true value set {G” for fluorescence lifetime prediction training q} Q And the denoising training input set {I' q} Q And the target set or true value set for denoising training {I” q} Q The constructed denoising neural network and fluorescence lifetime prediction neural network can be trained separately.

[0051] Figure 4The block diagram of an optical microscopic imaging system with a time-correlated single photon counting function according to an embodiment of the present application is schematically shown, for example, it includes an imaging module 100 and a data processing module 200. The imaging module 100 includes a fluorescence imaging device and a single photon counting device 110. For example, the single photon counting device 110 may include a picosecond pulse laser, a single photon detector, and a time-correlated single photon counting acquisition device. When the biological sample 300 treated with a fluorescent agent is placed on the workbench 400, the picosecond pulse laser can emit a laser pulse toward the biological sample 300 as needed to excite a fluorescence signal in the biological sample 300. At the same time, the excited fluorescence signal can be detected by the single photon detector, and the recorded photon information (such as the time taken for the photon to transition from the excited state to the ground state) is sent to the time-correlated single photon counting acquisition device as a specific coded information storage (for example, as photon counting (distribution) data), and the recording is repeated in this way until the specified photon recording time is reached. The data processing module 200 can include a module for processing the fluorescence image data and the photon counting (distribution) data taken by the fluorescence imaging device and received from the imaging module 100. The photon counting (distribution) data recorded in the above-mentioned single photon counting manner has the characteristic that the data of each recorded photon is independent and unrelated to each other.

[0052] Figure 5 The flowchart of the method for fluorescence image denoising and / or fluorescence lifetime prediction of a biological sample (which may be a living biological sample) according to an embodiment of the present application is schematically shown. First, in step S10, at least one fluorescence microscopic image data set is acquired using an optical microscopic imaging system with a time-correlated single photon counting function. The fluorescence microscopic image data set may include a fluorescence intensity data set and a photon count distribution set. The fluorescence microscopic image data set may be represented as

[0053] {I(m,n)} MN |{G mn (t)} MN

[0054] Wherein, m is any integer between 1 and M (M is an integer greater than 1), n ​​is any integer between 1 and N (N is an integer greater than 1), I represents the fluorescence image data obtained by the optical microscopic imaging system after the fluorescence generated by the biological sample after being irradiated by the excitation light is received by the system, that is, the fluorescence intensity data of the (m, n)th pixel point, G mn (t) represents the photon count distribution over time for the (m,n)th pixel in the fluorescence image data, and {} represents the fluorescence intensity data for all pixels in the entire pixel M×N or the photon count distribution of all pixels over time.

[0055] In step S20, a data set for training a neural network (e.g., a denoising neural network or a fluorescence lifetime prediction neural network) is constructed. For example, a training data set for a fluorescence lifetime prediction neural network and a training data set for a denoising neural network can be constructed independently or simultaneously.

[0056] Specifically, as a non-limiting example, a photon count distribution set {G mn (t)} MB , reconstructed by redistributing the photon flow as described above to generate 2×Q photon count distribution sets {G mnq ′(t)} MN and {G mnq ″(t)} Mj As a training data set for the fluorescence lifetime prediction neural network. According to a non-limiting example, a 2×Q photon count distribution set {G mnq ′(t)} MN and {G mnq ″(t)} MN Q photon count distribution sets are randomly selected from the set, and each selected photon count distribution set is used to calculate the fluorescence lifetime image using the maximum likelihood algorithm familiar to those skilled in the art, so as to finally obtain Q training fluorescence lifetime images to form an input set for fluorescence lifetime prediction training; then, 2×Q photon count distribution sets {G mnq ′(t)} MN and {G mnq ″(t)} MN Each of the remaining Q photon count distribution sets in the is respectively calculated using the maximum likelihood algorithm familiar to those skilled in the art to obtain a fluorescence lifetime image, so as to finally obtain Q training fluorescence lifetime images to form a target set or true value set for fluorescence lifetime prediction training.

[0057] In another non-limiting example, the Q photon count distribution sets {G mnq ′(t)} MN The fluorescence lifetime image is calculated using the maximum likelihood algorithm familiar to those skilled in the art, so as to finally obtain Q training fluorescence lifetime images to form an input set for fluorescence lifetime prediction training; then, the Q photon count distribution sets {G mnq ″(t)} MN The fluorescence lifetime image is calculated using a maximum likelihood algorithm familiar to those skilled in the art, so as to finally obtain Q training fluorescence lifetime images to form a target set or a true value set for fluorescence lifetime prediction training.

[0058] Those skilled in the art should be aware that the fluorescence lifetime image in the above step S20 may be subjected to maximum normalization processing before being used as an input set for fluorescence lifetime prediction training or an image in a target set or a true value set for fluorescence lifetime prediction training.

[0059] The above-mentioned input set for fluorescence lifetime prediction training and the above-mentioned target set or true value set for fluorescence lifetime prediction training constitute a training data set for a fluorescence lifetime prediction neural network.

[0060] In the technical solution of the present application, in order to improve the accuracy from the spatial perspective, the data of the pixel point to be calculated can be superimposed / merged with its adjacent pixel points, and then the maximum likelihood algorithm is used for calculation. The correlation between the adjacent pixel points to be superimposed / merged can be represented by Bin, for example Fig. 6A and Figure 6B The cases of Bin = 1 or 2 are shown respectively. Fig. 6A As shown in the figure, when Bin=1, the black squares in the figure represent the pixels to be calculated using the maximum likelihood algorithm, and the adjacent pixels with a step length of 1 pixel are represented by hollow squares. The photon count distribution G represented by these hollow squares mn (t) Superimposed / merged to the photon count distribution G represented by the black square mn (t), and then use the maximum likelihood algorithm to calculate. Figure 6B As shown in the figure, in the case of Bin=2, the black squares in the figure represent the pixels to be calculated using the maximum likelihood algorithm, and the adjacent pixels with a step size of 2 are represented by hollow squares. It should be clear to those skilled in the art that when the pixels represented by the black squares are at the boundary of the image, the photon count distribution to be superimposed / merged outside the corresponding boundary can be regarded as zero. The superposition / merging process can be simply understood as the calculation of these photon count distributions G mn (t) Sum the photon counts recorded in each time interval Δt. Those skilled in the art should know that, for the case of Bin=0, there is no need to perform the above superposition / merging, and the corresponding photon count distribution set can be directly calculated using the maximum likelihood algorithm.

[0061] In addition, according to another non-limiting example, in utilizing a photon count distribution set {G mn (t)} MN , reconstructed by redistributing the photon flow as described above to generate 2×Q photon count distribution sets {G mnq ′(t)} MN and {G mnq ″(t)} MNAfter that, it can be summed in time to create a training data set for the denoising neural network. According to a non-limiting example, it can be obtained from a 2×Q photon count distribution set {G mnq ′(t)} mN and {G mnq ″(t)} MN Q photon count distribution sets are randomly selected from the image processing system, and the photon count distribution of each pixel point of each selected photon count distribution set is summed up in time, and subjected to maximum value normalization processing well known to those skilled in the art to generate Q training fluorescence intensity images to form an input set for image denoising training; then, 2×Q photon count distribution sets {G mnq ′(t)} MN and {G mnq ″(t)} MN Each of the remaining Q photon count distribution sets in the sum is added in time, and subjected to maximum value normalization processing well known to those skilled in the art to generate Q training fluorescence intensity images to form a target set or a true value set for image denoising training. According to another non-limiting example, the Q photon count distribution sets {G mnq ′(t)} MN The images are summed up in time and subjected to maximum value normalization processing known to those skilled in the art to generate Q training fluorescence intensity images to form an input set for image denoising training; and the Q photon count distribution sets {G mnq ″(t)} MN The images are summed up in time and subjected to maximum value normalization processing known to those skilled in the art to generate Q training fluorescence intensity images to form a target set or true value set for image denoising training.

[0062] The above-mentioned input set for image denoising training and the above-mentioned target set or true value set for image denoising training constitute a training data set for a denoising neural network.

[0063] Those skilled in the art should be aware that the fluorescence microscopy image data set used to construct a data set for training the neural network in step S20 can be a fluorescence microscopy image data set acquired in step S10 using an optical microscopy imaging system with a time-correlated single photon counting function, or one selected from multiple fluorescence microscopy image data sets acquired in step S10 using an optical microscopy imaging system with a time-correlated single photon counting function.

[0064] In step S30, a neural network for predicting the fluorescence lifetime of a data set acquired by an optical microscopy imaging system and / or a neural network for denoising a fluorescence image is constructed. The construction of the neural network can select a neural network familiar to those skilled in the art, including but not limited to a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model. The selected neural network, such as a UNet neural network model or other neural network model, is configured to extract features from an input data set with time information, and the extracted features can be respectively transmitted to two convolution modules (a fluorescence lifetime prediction convolution module and a denoising convolution module), and the two modules can respectively output a denoised image and a fluorescence lifetime image. In this case, the loss function of the neural network is:

[0065]

[0066] in, Represents the overall loss function of the entire neural network. represents the loss function of the fluorescence lifetime prediction convolution module, represents the loss function of the denoising convolution module, μ is a number between 0 and 1. For example, when μ=0, it can be considered that the constructed neural network is specifically used for denoising, and when μ=1, it can be considered that the constructed neural network is specifically used for fluorescence lifetime prediction. By adjusting the size of μ (for example, between 0 and 1), the processing weight between denoising and fluorescence lifetime in the constructed neural network can be adjusted accordingly.

[0067] Those skilled in the art should be aware that the constructed neural network may also only include a fluorescence lifetime prediction convolution module or a denoising convolution module.

[0068] In step S40, the neural network constructed in step S30 is trained using the training data set for the fluorescence lifetime prediction neural network and / or the training data set for the denoising neural network constructed in step S20. For example, in the training of the neural network, the initial learning rate can be set to 1×10 -4 , the training batch size is 1, and the Adam optimizer is used for back-propagation iterative optimization.

[0069] In step S50, the trained neural network is used to perform denoising and / or fluorescence lifetime prediction processing on a fluorescence microscopy image data set acquired by an optical microscopy imaging system with a time-correlated single photon counting function.

[0070] According to a non-limiting example, when a neural network is used for both denoising and fluorescence lifetime prediction, the denoised image can be combined with the fluorescence lifetime image for pseudo-color display.

[0071] Fig. 7A It demonstrates the use of a trained neural network (represented by S) to process a photon distribution set (represented by G) acquired using an optical microscopy imaging system with time-correlated single photon counting function to obtain a fluorescence lifetime prediction image (represented by G′). Figure 7B It is demonstrated that a fluorescence image (represented by I) acquired by an optical microscopic imaging system with a time-correlated single photon counting function is processed using a trained neural network (represented by S) to obtain a denoised fluorescence image (represented by I′). According to an embodiment of the present application, the neural network S here can include a fluorescence lifetime prediction convolution module and a denoising convolution module at the same time, but according to the setting of the neural network (such as the setting of μ above), the fluorescence lifetime prediction convolution module and / or the denoising convolution module can be selected separately, and the processing weight between the two can be changed accordingly when processing at the same time.

[0072] Although the specific embodiments of the present application are described in detail herein, they are provided only for the purpose of explanation and should not be considered to limit the scope of the present application. In addition, it should be clear to those skilled in the art that the various embodiments described in this specification can be used in combination with each other. Various replacements, changes and modifications can be conceived without departing from the spirit and scope of the present application.

Claims

1. A method for training a neural network for self-supervising microscopic image processing, comprising: A fluorescence microscopic image data set for a biological sample is obtained by using an optical microscopic imaging system with a time-correlated single photon counting function, wherein the fluorescence microscopic image data set includes a photon count distribution set {G mn (t)} MN , where M and N are integers greater than 1, respectively, and m ranges from 1 to M and n ranges from 1 to N, and t represents time; A training data set for a neural network including a fluorescence lifetime prediction convolution module and / or a denoising convolution module is constructed from the photon count distribution set of the M×N pixels by means of photon flow redistribution, wherein the photon count distribution of each pixel in the photon count distribution set has p time intervals Δt, wherein p is an integer greater than 1, and in the photon flow redistribution, a pair of photon count distributions after photon flow redistribution is generated for the photon count distribution of each pixel of the M×N pixels, wherein the pair of photon count distributions after photon flow redistribution also each has p time intervals Δt and the photon counts of the photon count distribution of each pixel within the p time intervals Δt are respectively randomly allocated to the p time intervals Δt of the pair of photon count distributions after photon flow redistribution, and the photon flow redistribution is repeated Q times to generate Q pairs of photon count distribution sets after photon flow redistribution, wherein Q is an integer greater than 1, wherein a first group of Q photon count distribution sets after photon flow redistribution are selected from the Q pairs of photon count distribution sets after photon flow redistribution, and Q training fluorescence lifetime images are generated by maximum likelihood algorithm calculation and subjected to maximum value normalization processing to form a fluorescence lifetime prediction training input set, and a second group of Q photon count distribution sets after photon flow redistribution are selected from the Q pairs of photon count distribution sets after photon flow redistribution, and Q training fluorescence lifetime images are obtained by maximum likelihood algorithm calculation to form a fluorescence lifetime prediction training target set or true value set; or A first group of Q photon count distribution sets after photon flow redistribution is selected from the Q pairs of photon count distribution sets after photon flow redistribution, and for each of the selected Q photon count distribution sets after photon flow redistribution, the photon count distribution of each pixel is summed up in time and normalized by the maximum value to generate Q training fluorescence intensity images to form an input set for image denoising training, and a second group of Q photon count distribution sets after photon flow redistribution is selected from the Q pairs of photon count distribution sets after photon flow redistribution, and for each of the selected second group of Q photon count distribution sets after photon flow redistribution, the photon count distribution of each pixel is summed up in time and normalized by the maximum value to generate Q training fluorescence intensity images to form a target set or true value set for image denoising training, Wherein, the training data set includes the fluorescence lifetime prediction training input set and the fluorescence lifetime prediction training target set or true value set and / or the image denoising training input set and the image denoising training target set or true value set; The method also includes training the neural network using the training data set.

2. The training method according to claim 1, characterized in that: In the photon flux redistribution, the first group of Q and the second group of Q photon flux redistributed photon count distribution sets selected from the Q pairs of photon flux redistributed photon count distribution sets are randomly selected.

3. The training method according to claim 1, characterized in that: In the photon flux redistribution, a first group of Q photon flux redistributed photon count distribution sets selected from the Q pairs of photon flux redistributed photon count distribution sets include one photon count distribution in each pair of photon flux redistributed photon count distributions, and a second group of Q photon flux redistributed photon count distribution sets selected from the Q pairs of photon flux redistributed photon count distribution sets include another photon count distribution in one of the photon count distributions in each pair of photon flux redistributed photon count distributions.

4. The training method according to any one of claims 1 to 3, characterized in that: Before calculating using the maximum likelihood algorithm, the data of each pixel point to be calculated is superimposed / merged with the adjacent pixel points whose correlation degree (Bin) is 0, 1 or 2.

5. The training method according to claim 3 or 4, characterized in that: Models used to build the neural network include but are not limited to a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.

6. The training method according to claim 5, characterized in that: The loss function of the neural network is: in, Represents the overall loss function of the entire neural network. represents the loss function of the fluorescence lifetime prediction convolution module, Represents the loss function of the denoising convolution module, μ is a number between 0 and 1, wherein the model for constructing the loss function includes but is not limited to the mean square error loss function (MSE), the cross entropy loss function (CE), or the L2 loss function.

7. The training method according to claim 6, characterized in that: The output features of the model used to build the neural network are independently passed through a fluorescence lifetime prediction convolution module and a denoising convolution module.

8. The training method according to claim 1 or 2, characterized in that: The fluorescence microscopic image data set acquired by using the optical microscopic imaging system with time-correlated single photon counting function is a fluorescence microscopic image data set or a fluorescence microscopic image data set selected from a plurality of acquired fluorescence microscopic image data sets.

9. A method for processing a fluorescence microscopy image data set acquired using an optical microscopy imaging system with a time-correlated single photon counting function, comprising: Constructing a neural network including a fluorescence lifetime prediction convolution module and / or a denoising convolution module; Training the neural network using the training method described in any one of claims 1 to 8; as well as The trained neural network is used to process a fluorescence microscopy image data set acquired by an optical microscopy imaging system with a time-correlated single photon counting function, wherein the processing includes fluorescence lifetime prediction and / or denoising.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images