Plasmon tissue section staining-free imaging method based on generative adversarial network

Abstract

A dyeing-free imaging method for a plasmon tissue section based on a generative adversarial network relates to the technical field of biomedical detection, and comprises the following steps: 1) recording the tissue section placed on a plasmon glass slide by using a high-resolution bright field microscope and passing through a Hamp; e, dividing the dyed tissue slices into a training set and a verification set in proportion; 2) amplifying the number of paired images in the training set to a plurality of times through data enhancement; 3) constructing a GAN model, and training the GAN model by using the paired image after data enhancement as a data set; and 4) based on the trained GAN model, performing virtual staining of the staining-free imaging image of the plasmon tissue section. According to the invention, the generated virtual dyeing image is visually and statistically compared with the real Hamp; the E-dyed image has high similarity, and provides pathological interpretable images and more effective discrimination information for pathologists. The time is remarkably shortened, the cancer diagnosis efficiency is improved, and a rapid and effective tissue slice analysis means is provided for clinic.

Description

Technical Field

[0001] The present invention relates to the field of biomedical detection technology, and in particular to a plasmon tissue slice staining-free imaging method based on a generative adversarial network. Background Art

[0002] At present, tissue sections are the most commonly used criteria for pathology. General histopathological staining, such as hematoxylin and eosin (H&E), is the gold standard for cancer diagnosis. It has high resolution and good contrast and can evaluate tissue morphology, but its process is usually cumbersome and time-consuming. However, traditional H&E staining technology has the problems of long time consumption and easy damage to sample structure, which affects subsequent analysis. Summary of the invention

[0003] The purpose of the present invention is to solve the above-mentioned problem that traditional H&E staining in the prior art is time-consuming and not suitable for intraoperative diagnosis, and to provide a plasmon tissue section stain-free imaging method based on a generative adversarial network to achieve fast and high-quality stain-free tissue section imaging. The present invention enhances the plasmon near-field effect through a metal nanoarray, significantly improving the contrast and details of the stain-free image. Based on a generative adversarial network, the plasmon category (SPR) image is virtually stained into an H&E category image, thereby providing more image details, making the image staining pathologically interpretable, and then providing more discriminant information for pathologists, thereby improving the accuracy of cancer diagnosis.

[0004] In order to achieve the above object, the present invention adopts the following technical scheme:

[0005] A plasmon tissue section stain-free imaging method based on a generative adversarial network comprises the following steps:

[0006] 1) Using a high-resolution bright-field microscope, observe and record tissue sections placed on plasmon slides and tissue sections stained with H&E to obtain paired tissue section SPR images and H&E images;

[0007] 2) Collect enough paired tissue section SPR images and H&E images and divide them into training and validation sets in proportion;

[0008] 3) Expand the number of paired images in the training set several times through data augmentation;

[0009] 4) Build a GAN model and use the paired images after data enhancement as the data set to train the GAN model;

[0010] 5) Based on the trained GAN model, virtual staining of plasmon tissue section stain-free imaging images is performed.

[0011] The GAN model includes a generator and a discriminator, wherein the generator is used to generate a traditional H&E image from an input SPR image, and the discriminator is used to determine whether the image is a real H&E image or is generated by conversion of the generator.

[0012] The generator in the GAN model includes an encoder, a converter and a decoder. The encoder is composed of three convolutional layers, the converter is composed of 9 residual module layers, and the decoder is composed of two transposed convolutional layers and one convolutional layer.

[0013] The discriminator in the GAN model consists of four convolutional layers, and the discrimination result is output through a 1-channel convolutional layer.

[0014] In step 1), the data is divided into training set and validation set in a ratio of 4:1.

[0015] In step 2), the number of paired images in the training set is increased by 5 times through data augmentation.

[0016] The tissue sections include non-diseased tissue, invasive cancer tissue, and hyperplastic tissue. The tissue section types may be paraffin sections or frozen sections. The tissue sections may be selected to be of different tissue types, including breast tissue, liver tissue, ovarian tissue, etc., to increase the applicability and robustness of the model.

[0017] In one embodiment, 160 pairs of the 200 paired images collected are used as a data set for data augmentation, the data set is expanded 5 times, and the data set is input into the GAN model for training. After about 500 epochs, the loss function converges, the GAN model training is completed, and the remaining 40 pairs of images are used to verify the performance of the model.

[0018] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0019] The method of the present invention obtains the SPR image of the tissue section and the corresponding H&E staining image, uses 200 pairs of images for data enhancement, trains the constructed GAN model, and converts the plasmon SPR image into the corresponding H&E staining image, thereby realizing the virtual staining of the stain-free imaging image. Compared with the traditional H&E staining method, the present invention enhances the image contrast after imaging and obtains the spectral information of the tissue section by utilizing the near-field enhancement characteristics of the plasmon, and combines the generative adversarial network to realize the virtual staining of the stain-free imaging of the tissue section. This method not only significantly shortens the time required for traditional staining, but also can quickly provide key tissue information during surgery, avoiding delays caused by the staining process. Based on the change in the refractive index of the cell body, the present invention can accurately distinguish cancerous tissue from healthy tissue, and the generated virtual staining image is highly consistent with the real H&E staining image in terms of visual and statistical aspects, providing pathologists with pathologically interpretable images, and providing a more convenient and efficient means for clinical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the plasmon tissue section stain-free imaging method based on generative adversarial network of the present invention;

[0021] Figure 2 Schematic diagram of the structure of the plasmon slide in the embodiment;

[0022] Figure 3 This is a diagram of the structure of the generative adversarial network of the present invention;

[0023] Figure 4 The results of SPR image, SPR image virtually stained by the adversarial network generated by the present invention and H&E image are compared. DETAILED DESCRIPTION

[0024] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0025] Example 1

[0026] like Figure 1 As shown, the plasmon tissue section stain-free imaging method based on a generative adversarial network of the present invention comprises the following steps:

[0027] S1. Collect enough paired tissue section SPR images and H&E images and divide them into training set and validation set in a ratio of 4:1.

[0028] S2. The number of paired images in the training set is increased by 5 times through data enhancement; the data enhancement includes operations such as rotation, random brightening, and flipping to increase the richness of the data set and improve the generalization ability of the model;

[0029] S3, build a GAN model, and use the paired images after data enhancement as the data set to train the GAN model;

[0030] S4. Based on the trained GAN model, virtual staining of plasmon tissue section stain-free imaging images is performed.

[0031] The paired tissue slice SPR images and H&E images were observed and recorded under a high-resolution bright-field microscope. Specifically, the recorded images were two consecutive tissue slices of the same tissue placed on the periodic plasmon nanopores and stained with H&E.

[0032] Figure 2 Schematic diagram of the two-layer structure of the plasmon slide in this embodiment. The bottom layer is a glass substrate with periodic plasmon nanopores, and the top layer is metallic silver, which covers the glass substrate. The periodic plasmon nanopores are used to excite plasmons.

[0033] Periodic plasmon nanoholes can be prepared by electron beam lithography (EBL), focused electron beam (FIB), etc. The period is 300~600nm, and the diameter of the periodic plasmon nanoholes is 100~200nm. A layer of 10~100nm thick silver film is fixed on the surface of the periodic plasmon nanohole array by evaporation deposition (ED).

[0034] In this embodiment, breast cancer tissue is used as an example to analyze tissue. Breast cancer tissue sections placed on plasmon slides and tissue sections stained with H&E are observed and recorded using a high-resolution bright-field microscope. The recorded SPR images of breast cancer tissue are input into a GAN deep learning model trained with a large number of paired SPR images and H&E images of breast cancer tissue for virtual staining, aiming to convert SPR-style images into pathologically interpretable H&E-style images.

[0035] The GAN deep learning network model in this embodiment is a combination of two networks, one is the generator network responsible for generating simulated data, and the other is the discriminator network responsible for judging whether the input data is real or generated by the generator. The data here refers to the virtual stained image. The generator is used to generate the virtual stained image, and the discriminator is used to evaluate the similarity between the generated image and the real image. The generator must continuously optimize the H&E images it generates so that the discriminator network cannot judge it, and the discriminator must also be continuously optimized to make its judgment more accurate. The relationship between the two forms an adversarial relationship, so it is called a generative adversarial network.

[0036] See also Figure 3The generator network structure in the GAN model consists of three parts: encoder, converter and decoder. The encoder part consists of three convolutional layers, the converter part consists of 9 residual module layers, and the decoder consists of two transposed convolutional layers and one convolutional layer. The activation functions used by the encoder and decoder are both ReLU functions. The generator is used to generate traditional H&E images from the input SPR images. The discriminator consists of four convolutional layers. The activation function used in the convolutional layer is the LeakyReLU function, which can avoid the gradient disappearance problem. The discrimination result is output through a 1-channel convolutional layer to determine whether the image is a real H&E image or generated by an SPR image.

[0037] In this embodiment, the input SPR image is 256×256 pixels and 3 channels. The image first passes through a convolution layer of the encoder, using a 7×7 convolution kernel with a step size of 1, and is converted into a feature map of 64 channels; then, in the second convolution layer, the convolution kernel is 3×3, the step size is 2, and the number of channels is 128; the third convolution layer uses the same size convolution kernel 3×3, the step size is 2, and the number of channels is 256. After these three layers of convolution processing, the size of the output feature map is 64×64×256. Then, a 1×1 convolution layer is applied, and the number of channels remains at 256. The next converter part consists of 9 identical residual blocks (ResNet Block), each of which includes two convolution layers, the convolution kernel size is 3×3, the step size is 1, and the number of channels is 256. After the feature migration of the encoder and converter, the image features are passed to the decoder for upsampling and decoding. The first convolution layer of the decoder uses a 3×3 convolution kernel, a stride of 1, 256 channels, and a padding value of 1; the second convolution layer also uses a 3×3 convolution kernel, a stride of 1, 256 channels, and a padding value of 1; finally, the third convolution layer uses a 3×3 convolution kernel, a stride of 1, 3 channels, and a padding value of 1. The size of the output virtual stained image is the same as the size of the input SPR image, which is 256×256×3.

[0038] 200 pairs of paired images of non-diseased breast tissue, invasive cancer tissue, and hyperplastic tissue were collected, and 160 pairs of paired breast tissue images were used as training sets. After data augmentation, the training set was expanded 5 times and the dataset was input into the GAN model for training. After about 500 epochs, the loss function converged and the GAN model training was completed. The remaining 40 pairs of images were virtually stained with plasmon tissue section stain-free imaging images to verify the performance of the model. The reliability of the virtual staining results was verified by comparing the pathological features of the virtually stained pathological images generated by SPR images with those of the H&E stained pathological images.

[0039] Figure 4 is a virtual dyeing result diagram in this embodiment, Figure 4(a), (b), and (c) are SPR images observed under a microscope. Figure 4 (d), (e), and (f) are SPR images virtually dyed using a trained generative adversarial network. Figure 4 (g), (h), and (i) are H&E stained images of consecutive tissue sections in the same tissue block. Comparing two consecutive tissue sections of the same tissue, one is a traditional H&E stained image and the other is an SPR image. The verification results show that the H&E style image generated by using the SPR image after virtual staining with the generative adversarial network has a high degree of style consistency with the corresponding tissue section H&E stained image, and provides a structure for pathology interpretation. The feasibility of the plasmonic tissue section stain-free imaging method based on the generative adversarial network is proved.

[0040] The present invention utilizes a high-resolution bright-field microscope to observe and record tissue sections placed on plasmon slides and tissue sections stained with H&E, and inputs the recorded tissue SPR images into a GAN deep learning model trained with a large number of paired tissue SPR images and H&E images for virtual staining, thereby converting the SPR-style images into pathologically interpretable H&E-style images.

[0041] The present invention uses a metal nanoarray structure to achieve near-field enhancement of plasmons, enhance the image contrast after imaging, and obtain the spectral information of tissue sections. By analyzing the changes in the refractive index of the cell body, it is possible to accurately distinguish cancerous tissue from healthy tissue, and use the GAN deep learning model to generate virtual staining images, which are highly consistent with the real H&E staining images visually and statistically, providing pathologists with pathologically interpretable images, and providing a fast and effective tissue section analysis method for cancer clinical testing.

[0042] Although the contents of the present invention are specifically shown and described in conjunction with the preferred embodiments, those skilled in the art should understand that various changes made to the present invention in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims and without making any creative effort are within the scope of protection of the present invention.

Claims

1. A plasmon tissue section stain-free imaging method based on a generative adversarial network, characterized in that: The following steps are involved: 1) Using a high-resolution bright-field microscope, observe and record tissue sections placed on plasmon slides and tissue sections stained with H&E to obtain paired tissue section SPR images and H&E images; 2) Collect enough paired tissue section SPR images and H&E images and divide them into training and validation sets in proportion; 3) Expand the number of paired images in the training set several times through data augmentation; 4) Build a GAN model and use the paired images after data enhancement as the data set to train the GAN model; 5) Based on the trained GAN model, virtual staining of plasmon tissue section stain-free imaging images is performed.

2. The method for plasmon tissue section stain-free imaging based on generative adversarial networks according to claim 1, characterized in that: The GAN model includes a generator and a discriminator, wherein the generator is used to generate a traditional H&E image from an input SPR image, and the discriminator is used to determine whether the image is a real H&E image or is generated by conversion of the generator.

3. The method for plasmon tissue section stain-free imaging based on generative adversarial networks according to claim 2, characterized in that: The generator in the GAN model includes an encoder, a converter and a decoder. The encoder is composed of three convolutional layers, the converter is composed of 9 residual module layers, and the decoder is composed of two transposed convolutional layers and one convolutional layer.

4. The method for plasmon tissue section stain-free imaging based on generative adversarial networks according to claim 2, characterized in that: The discriminator in the GAN model consists of four convolutional layers, and the discrimination result is output through a 1-channel convolutional layer.

5. The method for plasmon tissue section stain-free imaging based on generative adversarial networks according to claim 1, characterized in that: In step 1), the data is divided into training set and validation set in a ratio of 4:

1.

6. The method for plasmon tissue section stain-free imaging based on generative adversarial networks according to claim 1, characterized in that: In step 2), the number of paired images in the training set is increased by 5 times through data augmentation.

7. The method for plasmon tissue section stain-free imaging based on generative adversarial networks according to claim 1, characterized in that: The tissue sections include non-diseased tissue, invasive cancer tissue, and hyperplastic tissue.

Images