A fluorescence image conversion method based on tissue sections

Abstract

The application provides a kind of fluorescence image conversion method based on tissue slice, for converting the fluorescence image to be converted of certain channel of tissue slice into the fluorescence image of other channel in same field of view, it is characterized in that, including the following steps: constructing deep learning network Pix2Pix;Loss function is constructed;The data set required for training is obtained;The data set is input to deep learning network Pix2Pix, the deep learning network Pix2Pix is optimized training using loss function and Adam optimization algorithm, and the trained deep learning network Pix2Pix is used as fluorescence image conversion model;The fluorescence image to be converted is input to fluorescence image conversion model, to obtain the fluorescence image of other channel in same field of view corresponding.The fluorescence image conversion method of tissue slice utilizes deep learning technology to find the prediction relationship between different channels of fluorescence image, so as to realize the image conversion of different channels, reduces the time-consuming and laborious work in (multi-label) fluorescence imaging process, saves cost and time.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of fluorescence imaging technology, and particularly relates to a fluorescence image conversion method based on tissue sections. BACKGROUND

[0002] Fluorescence microscopy is mainly used for detecting specific fluorescent materials and plays a crucial role in understanding the structure of macromolecules, metabolites and other subcellular components. However, at present, the use of fluorescence microscopy still has some limitations. For example, fluorescence microscopy involves time-consuming preparation steps, expensive reagents and signal bias caused by photobleaching.

[0003] Deep learning is a new technology that has been widely applied in various fields. With the in-depth application of deep learning in the field of microscopy, deep neural networks provide the possibility of converting the fluorescence image of a certain channel into the fluorescence image of other channels in the same field of view. In other words, deep learning technology can find a predictive relationship between different channels of fluorescence images. Based on this, the feasibility of different fluorescence conversion imaging based on tissue sections is provided. SUMMARY

[0004] To solve the above problems, a fluorescence image conversion method based on deep learning strategy for tissue sections is provided, which can convert the fluorescence image of a certain channel of the tissue section into the fluorescence image of other channels in the same field of view, thereby reducing the repeated and time-consuming work in the (multi-label) fluorescence imaging process, saving cost and time. The present application adopts the following technical scheme:

[0005] The present application provides a fluorescence image conversion method based on tissue sections, which is used for converting the fluorescence image of a certain channel of the tissue section into the fluorescence image of other channels in the same field of view, and is characterized by comprising the following steps: step S1, constructing a deep learning network Pix2Pix; step S2, constructing a loss function; step S3, obtaining a data set required for training; step S4, inputting the data set into the deep learning network Pix2Pix, and using the loss function and Adam optimization algorithm to optimize and train the deep learning network Pix2Pix, and taking the trained deep learning network Pix2Pix as a fluorescence image conversion model; and step S5, inputting the fluorescence image to be converted into the fluorescence image conversion model to obtain the fluorescence image of other channels in the same field of view.

[0006] The fluorescence image conversion method based on tissue sections provided by the present application can also have the following technical features, wherein the deep learning network Pix2Pix comprises a generator G and a discriminator D, the generator G adopts a U-Net network structure, and the discriminator D adopts a conditional discriminator PatchGAN.

[0007] The tissue slice-based fluorescent image conversion method provided by the application can also have the following technical features: the loss function is obtained by adding an L1 loss function and an MS-SSIM loss function on the basis of an adversarial loss function, and the loss function is:

[0008]

[0009] In the formula, represents the adversarial loss function, represents the L1 loss function, represents the MS-SSIM loss function, λ1 represents a regularization parameter of the L1 loss function, and λ2 represents a regularization parameter of the MS-SSIM loss function, represents that, in the case that G is a constant, a suitable D is selected so that can be maximized, and then a suitable G is selected so that can be minimized.

[0010] The tissue slice-based fluorescent image conversion method provided by the application can also have the following technical features: the adversarial loss function is:

[0011]

[0012] In the formula, p data (x) is a real image distribution in a training set, p data (x, y) is a joint probability distribution of the fluorescent image x and the fluorescent image y, is a log-likelihood expectation of (x, y), G represents a target that is attempted to be minimized, and D represents a target that is attempted to be maximized.

[0013] The L1 loss function is:

[0014]

[0015] The MS-SSIM loss function is:

[0016]

[0017] The tissue slice-based fluorescent image conversion method provided by the application can also have the following technical features: step S3 comprises the following sub-steps: step S3-1, acquiring and imaging the tissue slice based on the imaging device to obtain fluorescent images containing three channels; step S3-2, performing image enhancement on the fluorescent images obtained in step S3-1 by using a limited contrast adaptive histogram equalization method to obtain enhanced fluorescent images; step S3-3, randomly dividing all the enhanced fluorescent images into a plurality of small image blocks of 512x512 pixels; and step S3-4, regarding the plurality of small image blocks as a data set and dividing the data set into training data and test data.

[0018] The tissue slice-based fluorescent image conversion method provided by the application can also have the following technical features: the acquisition and imaging process comprises selection of a region of interest, automatic focusing, and image stitching.

[0019] The tissue slice-based fluorescent image conversion method provided by the application can also have the following technical features: in the optimization training of step S4, 150 epochs are trained, the number N of small batches of images that are traversed by the network each time for optimization is 4, and the initial learning rate is 0.0002.

[0020] Inventive action and effect

[0021] According to the tissue slice-based fluorescent image conversion method provided by the application, the deep learning network Pix2Pix with a generator and a discriminator is constructed to address the deficiencies in existing tissue slice fluorescent imaging, the deep learning network Pix2Pix is optimized and trained by using a loss function and an Adam optimization algorithm to obtain a fluorescent image conversion model, and as long as the tissue fluorescent image of a certain channel is input into the fluorescent image conversion model, the conversion of the fluorescent image of other channels of the corresponding tissue slice can be realized. The tissue slice fluorescent image conversion method uses deep learning technology to find a prediction relationship between different channels of the fluorescent image, thereby realizing the conversion of images of different channels, reducing the time-consuming and laborious work in the (multi-label) fluorescent imaging process, and saving costs and time. Meanwhile, based on the deep learning technology, multi-fluorescent label tissue slice imaging can also be realized. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of a tissue slice-based fluorescent image conversion method in an embodiment of the application;

[0023] Figure 2 is a structural schematic diagram of a deep learning network Pix2Pix in an embodiment of the application;

[0024] Figure 3is a schematic diagram of three-channel fluorescence images of mouse kidney tissue sections in the embodiments of the present application.

[0025] Figure 4 is a schematic diagram of converting the fluorescence image of the DAPI channel tissue section into the fluorescence image of the Phalloidin channel in the embodiments of the present application. DETAILED DESCRIPTION

[0026] The present application constructs a deep learning model that can realize the conversion of tissue section fluorescence images, including the deep learning network Pix2Pix architecture and the loss function; the constructed deep learning model is trained based on the data set obtained by experiment; then, based on the trained deep learning model, the fluorescence image of a certain channel of tissue section is converted into the fluorescence image of another channel; finally, the Mean Absolute Error (MAE), Structural Similarity (SSIM) and Peak signal-to-noise ratio (PSNR) are used to evaluate the quality of fluorescence image conversion.

[0027] In order to make the technical means, creative features, purposes and effects realized by the present application easy to understand, the following will make a specific description of a fluorescence image conversion method based on tissue sections according to the embodiments and the accompanying drawings.

[0028] <EMBODIMENT>

[0029] This embodiment is realized based on a desktop computer with an Intel Xeon Gold 6248R CPU@3.00GHz processor and 256G RAM, running a Windows 10 operating system. Among them, the prediction network is realized by using Python 3.8.3 version.cGAN is realized by using Pytorch version 1.6.0. Other python libraries used include torchvision, dominate, visdom, os, time, Python Imaging Library (PIL), and numpy.

[0030] Figure 1 is a flowchart of a fluorescence image conversion method based on tissue sections in the embodiments of the present application.

[0031] As shown in Figure 1 , the fluorescence image conversion method based on tissue sections includes the following steps:

[0032] The purpose of image transformation is to convert a source domain image into a target domain image using an end-to-end network model. In this embodiment, the deep learning network Pix2Pix employs a fully supervised approach, that is, it trains the model using fully paired input and output images, and then uses the trained model to generate the target image for the specified task from the input image.

[0033] Step S1: Construct the deep learning network Pix2Pix.

[0034] Figure 2 This is a schematic diagram of the structure of the deep learning network Pix2Pix in an embodiment of the present invention.

[0035] like Figure 2 As shown, the deep learning network Pix2Pix includes a generator G and a discriminator D. The generator G employs a U-Net-based architecture, and its training objective is to maximize the probability value output by the discriminator when the generated G(x) and x are used as inputs. The discriminator D uses the PatchGAN conditional discriminator, and its training objective is to output a small probability value when the input is not a pair of real images (x and G(x)) and a large probability value when the input is a pair of real images (x and y). In other words, the discriminator attempts to distinguish between the target image and the generator's output image.

[0036] Step S2: Construct a loss function by adding the L1 loss function and the MS-SSIM loss function to the adversarial loss function. The loss function is expressed as follows:

[0037]

[0038] In the formula, Represents the adversarial loss function. Describes the L1 loss function. Let λ1 represent the MS-SSIM loss function, λ2 represent the regularization parameter of the L1 loss function, and λ3 represent the regularization parameter of the MS-SSIM loss function. This means that, with G being a constant, we should choose a suitable D such that... To maximize, and then choose an appropriate G to make It can be minimized.

[0039] The loss function has a great influence on network training. The L1 loss function can better maintain the similarity between the input image and the output image, but the L1 loss function does not consider human visual perception. The SSIM loss function considers human visual perception indicators such as luminance, contrast and structure. Further, the MS-SSIM (Multi-Scale SSIM) loss function is based on a multi-layer SSIM loss function, and also considers resolution. Therefore, the embodiment adds the L1 loss function and the MS-SSIM loss function on the basis of the adversarial loss function.

[0040] wherein the adversarial loss function is:

[0041]

[0042] wherein p data (x) is a real image distribution in the training set, p data (x, y) is a joint probability distribution of the fluorescence image x and the fluorescence image y, is a log-likelihood expectation of (x, y), wherein G represents a target to be minimized, and D represents a target to be maximized.

[0043] The L1 loss function is:

[0044]

[0045] The MS-SSIM loss function is:

[0046]

[0047] Step S3, obtain the data set required for training.

[0048] This step S3 includes the following sub-steps:

[0049] Step S3-1, based on the imaging device, collect and image the tissue section to obtain fluorescence images containing three channels.

[0050] In this embodiment, the fluorescence imaging mode of the scanning microscope (Olympus, VS-BX) equipped with a 40x / 0.95NA objective lens is used to obtain fluorescence images of three channels of DAPI, Phalloidin and WGA of the mouse kidney tissue section.

[0051] wherein DAPI (4', 6-Diamidino-2-Phenylindole) is used to label the cell nucleus (this channel is simply referred to as DAPI); Alexa 488WGA conjugate for labeling cell membrane (this channel is abbreviated as WGA), Alexa Fluor Phalloidin conjugate for labeling cytoskeleton (this channel is abbreviated as Phalloidin). For DAPI channel, the emission wavelength is 455 nm, and the exposure time is 100 ms; for Phalloidin channel, the emission wavelength is 480 nm, and the exposure time is 50 ms; for WGA channel, the emission wavelength is 656 nm, and the exposure time is 50 ms. The acquisition and imaging process includes the selection of the region of interest (ROI), automatic focusing, and image stitching, etc.

[0052] Step S3-2, the fluorescence images obtained in step S3-1 are subjected to image enhancement by using a limited contrast adaptive histogram equalization method to obtain enhanced fluorescence images.

[0053] Since the acquired partial fluorescence images have low contrast, the embodiment uses a limited contrast adaptive histogram equalization (CLAHE) method for image enhancement. This method can improve the local contrast of the image, will not cause damage to the image details, and can avoid amplification of image noise with the increase of image contrast. The final fluorescence images of three different channels are as shown in Figure 3

[0054] Step S3-3, in order to reduce the memory requirement and speed up the training process, all the enhanced fluorescence images are randomly divided into a plurality of small image blocks of 512x512 pixels.

[0055] Step S3-4, the plurality of small image blocks are used as a data set, and the training data is divided into 1280, and the test data is 320.

[0056] Step S4, the data set is input into the deep learning network Pix2Pix, the loss function and the optimization algorithm are used to optimize and train the deep learning network Pix2Pix, and the trained deep learning network Pix2Pix is used as a fluorescence image conversion model.

[0057] In this embodiment, the Adam optimization algorithm is used to optimize and train the data set and the deep learning network Pix2Pix, and the total training is 150 epochs, wherein the number N of small batches of images that the network optimizes each time is 4, and the initial learning rate is 0.0002. In order to make the error converge to a smaller value, a learning rate decay strategy is used.

[0058] ​Step S5, inputting any channel fluorescence image of the tissue slice into the fluorescence image conversion model to obtain the fluorescence image of other channels in the same field of view.

[0059] In this embodiment, in order to evaluate the performance of the fluorescence image conversion model, the mean absolute error (MAE), the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM) of the real image and the model generated image are calculated. Specifically:

[0060] MAE represents the average of the absolute error between the predicted value and the observed value, which can avoid the problem of mutual offset of errors, and thus can accurately reflect the actual prediction error. The smaller the value is, the better the model effect is. The calculation method is as follows:

[0061]

[0062] In the formula, N represents the number of pixels in the image.

[0063] SSIM is an index for measuring the similarity of two images. Its value range is [0, 1], and the larger the SSIM value is, the smaller the image distortion is. The calculation method is as follows:

[0064]

[0065] In the formula, μ x is the average value of x, μ y is the average value of y, is the variance of x, is the variance of y, σ xy is the covariance of x and y, c1=(k1L) 2 , c2=(k2L) 2 is a constant used to maintain stability, L is the dynamic range of pixel value, k1=0.01, k2=0.03. PSNR is the most common and widely used image objective evaluation index. The larger the value is, the smaller the distortion is. The calculation method is as follows:

[0066]

[0067]

[0068] Wherein, MSE (Mean Square Error) represents the mean square error of the current image x and the reference image y, h and w are the height and width of the image respectively, and n is the number of bits per pixel.

[0069] In this embodiment, the performance evaluation index results of the tissue slice fluorescence image conversion realized by the trained deep learning network Pix2Pix are shown in Table 1. The experimental results of converting the DAPI channel to the Phalloidin channel are shown in FIG. 2.Figure 4 as shown.

[0070] Fluorescence channel conversion MAE SSIM PSNR (dB) DAPI -> Phalloidin 14.62±2.576 0.621±0.014 16.891±0.446 DAPI -> WGA 10.38±2.928 0.828±0.012 19.364±0.506

[0071] Table 1

[0072] Compared with the existing (multi-label) tissue slice fluorescence imaging method, the present application constructs a general framework to conveniently solve the fluorescence image conversion problem, thereby realizing multi-color tissue slice fluorescence imaging; at the same time, the time-consuming and laborious preparation work in the (multi-label) fluorescence imaging process is reduced, and the cost and time are saved.

[0073] Effects of the embodiments

[0074] According to the tissue slice-based fluorescence image conversion method provided in the embodiment, the deep learning network Pix2Pix is constructed in view of the deficiencies existing in the existing tissue slice fluorescence imaging, and the deep learning network Pix2Pix is optimized and trained by using a loss function and an Adam optimization algorithm to obtain a fluorescence image conversion model. As long as the tissue fluorescence image of a certain channel is input into the fluorescence image conversion model, the fluorescence image conversion of other channels of the corresponding tissue slice can be realized. The tissue slice fluorescence image conversion method uses deep learning technology to find the prediction relationship between different channels of the fluorescence image, thereby realizing the image conversion of different channels, reducing the time-consuming and laborious work in the (multi-label) fluorescence imaging process, and saving the cost and time; at the same time, based on the deep learning technology, multi-fluorescent label tissue slice imaging can also be realized.

[0075] In the embodiment, since the constructed deep learning network Pix2Pix has a generator and a discriminator, a good output is generated through the mutual game of the generator and the discriminator. At the same time, the deep learning technology also better realizes the prediction conversion of different channel images.

[0076] In the embodiment, since the loss function used for training is constructed by adding an L1 loss function and an MS-SSIM loss function on the basis of an adversarial loss function, not only the similarity between the input image and the output image can be better maintained, but also the brightness, contrast, structure and other human visual perception indicators are considered. At the same time, since the MS-SSIM loss function is based on a multi-layer SSIM loss function, the high resolution of the image is also ensured.

[0077] The above embodiments are only used to illustrate the specific embodiments of the present application, and the present application is not limited to the description range of the above embodiments.

Claims

1. A method for converting fluorescence images of a tissue section, for converting a to-be-converted fluorescence image of a certain channel of a tissue section into a fluorescence image of another channel in the same field of view, comprising the steps of: data ​ Step S1, constructing a deep learning network Pix2Pix, wherein the deep learning network Pix2Pix comprises a generator G and a discriminator D, The generator G adopts a U-Net network structure, and the discriminator D adopts a conditional discriminator PatchGAN; Step S2, constructing a loss function, wherein the loss function is obtained by adding an L1 loss function and an MS-SSIM loss function on the basis of an adversarial loss function, and the loss function is: wherein, represents the adversarial loss function, represents the L1 loss function, represents the MS-SSIM loss function, λ1 represents a regularization parameter of the L1 loss function, and λ2 represents a regularization parameter of the MS-SSIM loss function, represents that, in the case where G is a constant, a suitable D is selected such that can be maximized, a suitable G is selected such that can be minimized; Step S3, obtaining a data set required for training; Step S4, inputting the data set into the deep learning network Pix2Pix, and optimizing and training the deep learning network Pix2Pix by using the loss function and an Adam optimization algorithm, and taking the trained deep learning network Pix2Pix as a fluorescence image conversion model; Step S5, inputting the to-be-converted fluorescence image into the fluorescence image conversion model to obtain a fluorescence image of another channel in the same field of view.

2. The fluorescence image conversion method based on tissue slices according to claim 1, wherein: wherein The adversarial loss function is: where p data (x) is the true image distribution in the training set, p data (x,y) is the joint probability distribution of fluorescence image x and fluorescence image y, is the log-likelihood expectation of (x,y), G represents the goal to be minimized, and D represents the goal to be maximized; The L1 loss function is: The MS-SSIM loss function is:

3. The fluorescence image conversion method based on tissue slices according to claim 1, wherein: wherein, The step S3 comprises the following sub-steps: Step S3-1, acquiring and imaging tissue slices based on an imaging device to obtain fluorescence images containing three channels; Step S3-2, performing image enhancement on the fluorescence images obtained in the step S3-1 by using a limited contrast self-adaptive histogram equalization method to obtain enhanced fluorescence images; Step S3-3, randomly dividing all the enhanced fluorescence images into a plurality of small image blocks with a size of 512x512 pixels; Step S3-4, taking the plurality of small image blocks as the data set, and dividing the data set into training data and test data.

4. The fluorescence image conversion method based on tissue slices according to claim 3, wherein: wherein The acquisition and imaging process comprises selection of a region of interest, automatic focusing, and image stitching.

5. The fluorescence image conversion method based on tissue slices according to claim 1, wherein: wherein In the optimization training of the step S4, a total of 150 epochs are trained, the number N of small batches of images that are optimized by the network each time is 4, and the initial learning rate is 0.0002.

Images