Prostate cancer tissue HE virtual staining algorithm based on deep learning

Abstract

The invention relates to a prostate cancer tissue Hamp based on deep learning; the invention discloses an E virtual staining algorithm, and belongs to the field of malignant tumors. The method comprises the following steps: 1, obtaining a prostate cancer tissue slice white piece and a corresponding Hamp; e, an image data set; 2, performing data preprocessing on the data set; 3, inputting the preprocessed standard data set into the model for training; step 4, inputting the data set into a designed Attention-CycleGAN network, after inputting a source domain image, outputting a conversion image of a target domain by the model, and training a prostatic cancer tissue Hamp; and E, virtualizing the dyeing model. The prostate cancer tissue Hamp based on deep learning; according to the E virtual dyeing algorithm, the model is more excellent in performance when processing a complex image conversion task, the generated image is more vivid and meticulous, the discriminator extracts the multi-scale features of the input image through the multilayer convolutional network, and the authenticity of the input image can be more accurately judged in combination with the enhancement characteristic of the space attention mechanism, so that the image conversion efficiency is improved. Therefore, the precision of the whole model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of malignant tumors, and in particular to a H&E virtual staining algorithm for prostate cancer tissue based on deep learning. Background Art

[0002] Prostate cancer (PC) is one of the most common malignant tumors in men worldwide and the sixth leading cause of cancer-related death. In clinical practice, pathological biopsy has always been recognized as an effective and accurate method for diagnosing prostate cancer. In almost all large-scale prostate cancer outcome studies, prostate puncture biopsy is the gold standard for diagnosing prostate cancer. Under the real-time guidance of ultrasound, a fine needle is used to puncture 9 to 16 needles in the prostate tissue to obtain prostate tissue. After hematoxylin-eosin staining (H&E staining) is performed according to professional personnel and equipment, it is sent to the pathology department for final pathological diagnosis. Hematoxylin-eosin staining (H&E staining) is one of the commonly used staining methods in paraffin section technology. It is the most basic and widely used technical method in teaching and scientific research in histology, embryology, and pathology.

[0003] Traditional H&E staining requires well-trained professionals to perform cell smear, water washing, staining and other steps, which consumes a lot of time and space costs during the operation. In addition, doctors with different medical experience mainly make subjective judgments based on their own judgments, and the diagnostic analysis results of the stained biopsy sections may be different, resulting in missed diagnosis or misdiagnosis. At the same time, in the traditional metabolomics analysis of cancer tissue cells, there are two main platforms: mass spectrometry (MS) and nuclear magnetic resonance (NMR) spectroscopy, but the price of both instruments is high. The detection requires professional equipment and professionals to destroy the cells, separate the cell substances, and then analyze the metabolism of each substance. The steps are cumbersome. Therefore, a rapid, non-invasive, and automatic detection method is needed for the research and analysis of the pathology and metabolomics of PCa. Summary of the invention

[0004] 1. Technical issues to be resolved In view of the shortcomings of the existing technology, the present invention provides a deep learning-based H&E virtual staining algorithm for prostate cancer tissue, which has the advantages of being fast and non-invasive, and solves the problem that different diagnostic analysis results may be obtained for stained biopsy sections, thereby causing missed diagnosis or misdiagnosis.

[0005] (II) Technical solution To achieve the above object, the present invention provides the following technical solution: a H&E virtual staining algorithm for prostate cancer tissue based on deep learning, comprising the following steps: Step 1: Obtain prostate cancer tissue slices and corresponding H&E image datasets; Step 2: Preprocess the above data set; Step 3: Input the above preprocessed standard data set into the model for training; Step 4: Input the dataset into the designed Attention-CycleGAN network. After inputting the source domain image, the model will output the converted image of the target domain and train the H&E virtual staining model of prostate cancer tissue, which consists of a generator, a discriminator, and a spatial attention mechanism. Step 5: After the training is completed, the H&E pseudo-staining model of prostate cancer tissue is obtained.

[0006] A total of 169 white film images and 169 corresponding H&E stained images used in step 1 were provided by the Urology Experimental Center of Renji Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, including tissue section images of patients scored as benign hyperplasia, Gleason-3, Gleason-4, and Gleason-5.

[0007] In step 2, each image exported from the K-Viewer software is enhanced and various image transformations are performed to expand the data set, including rotation, translation, scaling, cropping, flipping, adjusting brightness and contrast, and adding noise. Through these transformations, multiple versions of the original image can be generated to increase the diversity of training data and reduce overfitting of the model.

[0008] Generators G_A and G_B map prostate cancer tissue slices from the source domain to the target domain, respectively, and introduce a spatial attention mechanism to enable the generator to pay more attention to the key areas in the image; Specifically, the input image first passes through the Attention module to calculate the importance of each pixel and generate an attention map. The attention map is then concatenated with the input image and passed through a series of convolutional and deconvolutional layers to finally generate an image of the target domain.

[0009] In step 3: The discriminators D_A and D_B are used to distinguish the generated virtual H&E stained images from the real H&E stained images. The discriminator is composed of several convolutional layers. It extracts features layer by layer and makes judgments, and finally outputs a probability value to judge the authenticity of the input image. Specifically, in the Attention-CycleGAN model, both the generator and the discriminator contain spatial attention mechanism modules. The spatial attention mechanism in the generator is used to enhance the features of specific areas in the input image, so that the generator can pay more attention to key areas and improve the quality and detail performance of the generated image; the spatial attention mechanism in the discriminator is used to improve the ability to judge the authenticity of the input image; each spatial attention mechanism module is composed of multiple convolutional layers, which generates an attention map by calculating the importance of each pixel and weights the feature map.

[0010] Step 3: For generator training: The preprocessed prostate cancer tissue slices are input into the generator G_A, which converts the input prostate cancer tissue slices into virtual H&E stained images; during the generation process, the feature map is weighted by the spatial attention mechanism; in the process of applying the spatial attention mechanism, a convolution operation is first performed to extract the features of the input image, and the ReLU activation function is used to increase the nonlinearity of the model. A single-channel attention map is generated through the convolution operation, and the Softmax function is applied to the attention map. The sum of the attention weights along the spatial dimension is 1. Finally, the weighted feature map is input to the subsequent part of the generator to generate a high-quality virtual H&E stained image. The generated virtual H&E stained image and the real H&E stained image are used for loss calculation, and the parameters of the generator are updated.

[0011] Step 3: For generator training: The generated virtual H&E stained image and the real H&E stained image are input into the discriminator D_A respectively. The discriminator D_A extracts the multi-scale features of the input image through a multi-layer convolutional network and judges the authenticity of the input image. The adversarial loss is used to calculate the difference between the virtual image and the real image, which is formula (1). Finally, according to the loss result, the parameters of the discriminator D_A are updated to improve its ability to distinguish between real images and virtual images.

[0012] The generated virtual H&E stained image is converted back to the original prostate cancer tissue slice using the generator G_B. The L1 loss between the original image and the converted image is calculated as formula (2). According to the cycle consistency loss, the parameters of the generators G_A and G_B are updated to ensure that the conversion between the source domain and the target domain maintains the consistency of the image content. The above steps are repeated until the loss functions of the generator and the discriminator reach a stable state and the model converges. .

[0013] (III) Beneficial effects Compared with the prior art, the present invention provides a prostate cancer tissue H&E virtual staining algorithm based on deep learning, which has the following beneficial effects: This deep learning-based H&E virtual staining algorithm for prostate cancer tissue introduces a spatial attention mechanism, and Attention-CycleGAN can better focus on key areas in the image. The spatial attention mechanism generates an attention map by calculating the importance of each pixel and weights the feature map output by the generator, thereby improving the quality and detail retention of the generated image. The generator weights the feature map through the spatial attention mechanism, which can more efficiently generate high-quality target domain images. The model performs better when processing complex image conversion tasks, and the generated images are more realistic and detailed. The discriminator extracts multi-scale features of the input image through a multi-layer convolutional network. Combined with the enhanced characteristics of the spatial attention mechanism, it can more accurately judge the authenticity of the input image, thereby improving the accuracy of the overall model. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is an overall step diagram of a deep learning-based H&E virtual staining algorithm for prostate cancer tissue proposed by the present invention; Figure 2 This is a schematic diagram of the structure of the Attention-CycleGAN algorithm for H&E virtual staining of prostate cancer tissue based on deep learning proposed in the present invention; Figure 3 Schematic diagram of the SAM structure of the deep learning-based H&E virtual staining algorithm for prostate cancer tissue proposed in the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] Embodiment 1: A deep learning-based H&E virtual staining algorithm for prostate cancer tissue, comprising the following steps: Step 1: Obtain a dataset of prostate cancer tissue slices and H&E images. A total of 169 white slice images and 169 corresponding H&E stained images were provided by the Urology Experimental Center of Renji Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, including tissue slice images of patients scored as benign hyperplasia, Gleason-3, Gleason-4, and Gleason-5.

[0017] Step 2: Preprocess the above dataset. Perform image enhancement on each image exported from the K-Viewer software, and perform various transformations on the image to expand the dataset. This includes rotation, translation, scaling, shearing, flipping, adjusting brightness, contrast, adding noise, etc. Through these transformations, multiple different versions of the original image can be generated, thereby increasing the diversity of training data, reducing model overfitting, and improving the generalization ability of the model, which can significantly improve the performance and robustness of the deep learning model.

[0018] Step 3: Input the above preprocessed standard data set into the model for training: Step 4: The preprocessed prostate cancer tissue slice images were input into the Attention-CycleGAN model. The Attention-CycleGAN model mainly consists of two generators, two discriminators, and a spatial attention mechanism (SAM) module. The network structure is shown in the figure below. Figure 2 As shown in the figure, the generators G_A and G_B respectively map prostate cancer tissue slices from the source domain to the target domain, and introduce the spatial attention mechanism to enable the generator to pay more attention to the key areas in the image. The input prostate cancer tissue slices are first subjected to several convolutional layers for feature extraction, and then the attention map is generated by the spatial attention module, and the feature map is weighted to improve the quality of the generated image. Specifically, the input image first passes through the Attention module to calculate the importance of each pixel and generate an attention map. The attention map and the input image are then concatenated, and the image of the target domain is finally generated through a series of convolutional and deconvolutional layers. The output of the Attention module is used as the initial input of the generator model, and is used by the discriminators D_A and D_B to distinguish the generated virtual H&E staining images from the real H&E staining images. The discriminator consists of several convolutional layers, which extracts features layer by layer and makes judgments, and finally outputs a probability value to judge the authenticity of the input image.

[0019] like Figure 3As shown in the figure, in the Attention-CycleGAN model, both the generator and the discriminator contain spatial attention mechanism modules. The spatial attention mechanism in the generator is used to enhance the features of specific areas in the input image, so that the generator can pay more attention to key areas, thereby improving the quality and detail performance of the generated image. The spatial attention mechanism in the discriminator is used to improve the ability to judge the authenticity of the input image. Each spatial attention mechanism module consists of multiple convolutional layers, which generates an attention map by calculating the importance of each pixel and weights the feature map.

[0020] Training process: For generator training: the preprocessed prostate cancer tissue slices are input into the generator G_A, which converts the input prostate cancer tissue slices into virtual H&E stained images. During the generation process, the feature map is weighted by the spatial attention mechanism. In the process of applying the spatial attention mechanism, a convolution operation is first performed to extract the features of the input image, and the ReLU activation function is used to increase the nonlinearity of the model. A single-channel attention map is generated through the convolution operation. The Softmax function is applied to the attention map, and the sum of the attention weights along the spatial dimension is 1. Finally, the weighted feature map is input to the subsequent part of the generator to generate a high-quality virtual H&E stained image. The generated virtual H&E stained image and the real H&E stained image are used for loss calculation, and the parameters of the generator are updated.

[0021] For discriminator training, the generated virtual H&E staining image and the real H&E staining image are input into the discriminator D_A respectively. The discriminator D_A extracts the multi-scale features of the input image through a multi-layer convolutional network and judges the authenticity of the input image. The difference between the virtual image and the real image is calculated using adversarial loss, which is formula (1). Finally, according to the loss result, the parameters of the discriminator D_A are updated to improve its ability to distinguish between real images and virtual images.

[0022] The generated virtual H&E stained image is converted back to the original prostate cancer tissue slice using the generator G_B. The L1 loss between the original image and the converted image is calculated as formula (2). According to the cycle consistency loss, the parameters of the generators G_A and G_B are updated to ensure that the conversion between the source domain and the target domain maintains the consistency of the image content. The above steps are repeated until the loss functions of the generator and the discriminator reach a stable state and the model converges.

[0023]

[0024]

[0025] The beneficial effects of the present invention are: the deep learning-based prostate cancer tissue H&E virtual staining algorithm, CycleGAN, realizes unsupervised image translation from the source domain to the target domain by using two generators and two discriminators. It does not require paired training data, and only needs to provide unpaired image data of the source domain and the target domain. At the same time, the spatial attention mechanism (SAM) performs well in processing image data, and improves the model's ability to extract key features through weight distribution. The spatial attention module is introduced to capture significant features in the image, thereby enhancing the effect of image conversion. After inputting the source domain image, the model will output a converted image of the target domain. The generator converts the source domain image into the target domain image, and focuses on the key areas in the image by introducing the spatial attention mechanism. The discriminator is used to distinguish the generated target domain image from the real target domain image. The spatial attention mechanism generates an attention map by calculating the importance of each pixel, and weights the feature map output by the generator, thereby improving the quality and detail retention of the generated image.

[0026] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A deep learning-based H&E virtual staining algorithm for prostate cancer tissue, characterized in that: The following steps are involved: Step 1: Obtain prostate cancer tissue slices and corresponding H&E image datasets; Step 2: Preprocess the above data set; Step 3: Input the above preprocessed standard data set into the model for training; Step 4: Input the dataset into the designed Attention-CycleGAN network. After inputting the source domain image, the model will output the converted image of the target domain and train the H&E virtual staining model of prostate cancer tissue, which consists of a generator, a discriminator, and a spatial attention mechanism. Step 5: After the training is completed, the H&E virtual staining model of prostate cancer tissue is obtained.

2. The H&E virtual staining algorithm for prostate cancer tissue based on deep learning according to claim 1, characterized in that: A total of 169 white film images and 169 corresponding H&E stained images used in step 1 were provided by the Urology Experimental Center of Renji Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, including tissue section images of patients scored as benign hyperplasia, Gleason-3, Gleason-4, and Gleason-5.

3. The H&E virtual staining algorithm for prostate cancer tissue based on deep learning according to claim 1, characterized in that: In step 2, each image exported from the K-Viewer software is enhanced and various image transformations are performed to expand the data set, including rotation, translation, scaling, cropping, flipping, adjusting brightness and contrast, and adding noise. Through these transformations, multiple versions of the original image can be generated to increase the diversity of training data and reduce overfitting of the model.

4. The H&E virtual staining algorithm for prostate cancer tissue based on deep learning according to claim 1, characterized in that: In step 4: Generators G_A and G_B map prostate cancer tissue slices from the source domain to the target domain, respectively, and introduce a spatial attention mechanism to enable the generator to pay more attention to the key areas in the image, specifically, The input image first passes through the Attention module to calculate the importance of each pixel and generate an attention map. The attention map is then concatenated with the input image and passed through a series of convolutional and deconvolutional layers to finally generate an image of the target domain.

5. The H&E virtual staining algorithm for prostate cancer tissue based on deep learning according to claim 4, characterized in that: In step 4: The discriminators D_A and D_B are used to distinguish the generated virtual H&E stained images from the real H&E stained images. The discriminator consists of several convolutional layers. It extracts features layer by layer and makes judgments, and finally outputs a probability value to judge the authenticity of the input image. Specifically, In the Attention-CycleGAN model, both the generator and the discriminator contain spatial attention mechanism modules. The spatial attention mechanism in the generator is used to enhance the features of specific areas in the input image, allowing the generator to pay more attention to key areas and improve the quality and detail of the generated image. The spatial attention mechanism in the discriminator is used to improve the ability to judge the authenticity of the input image; each spatial attention mechanism module consists of multiple convolutional layers, which generates an attention map by calculating the importance of each pixel and weighting the feature map.

6. The H&E virtual staining algorithm for prostate cancer tissue based on deep learning according to claim 1, characterized in that: Step 4: For generator training: The preprocessed prostate cancer tissue slices are input into the generator G_A, which converts the input prostate cancer tissue slices into virtual H&E stained images; during the generation process, the feature map is weighted by the spatial attention mechanism; in the process of applying the spatial attention mechanism, a convolution operation is first performed to extract the features of the input image, and the ReLU activation function is used to increase the nonlinearity of the model. A single-channel attention map is generated through the convolution operation, and the Softmax function is applied to the attention map. The sum of the attention weights along the spatial dimension is 1. Finally, the weighted feature map is input to the subsequent part of the generator to generate a high-quality virtual H&E stained image. The generated virtual H&E stained image and the real H&E stained image are used for loss calculation, and the parameters of the generator are updated.

7. The H&E virtual staining algorithm for prostate cancer tissue based on deep learning according to claim 1, characterized in that: Step 4: For generator training: The generated virtual H&E stained image and the real H&E stained image are input into the discriminator D_A respectively. The discriminator D_A extracts the multi-scale features of the input image through a multi-layer convolutional network and judges the authenticity of the input image. The difference between the virtual image and the real image is calculated using the adversarial loss, which is Formula 1. Finally, according to the loss result, the parameters of the discriminator D_A are updated to improve its ability to distinguish between real images and virtual images. The generated virtual H&E stained image is converted back to the original prostate cancer tissue slice using the generator G_B. The L1 loss between the original image and the converted image is calculated as Formula 2. According to the cycle consistency loss, the parameters of the generators G_A and G_B are updated to ensure that the conversion between the source domain and the target domain maintains the consistency of the image content. The above steps are repeated until the loss functions of the generator and the discriminator reach a stable state and the model converges: 。

Images