Pathological section digital virtual staining method and system of CyclGAN-based unsupervised model

Abstract

The invention discloses a pathological section digital virtual staining method and system of a CyclGAN-based unsupervised model, and belongs to the field of digital pathology and artificial intelligence data processing. By combining multi-modal information, an autofluorescence image and a DAPI image of an unstained tissue section are used as input and are converted into a traditional hematoxylin-eosin stained output image for pathological diagnosis. According to the method, an unsupervised model is utilized, dependence on a pixel-level matching training set is avoided, and limitation of a traditional supervised model is broken through. Experimental results show that the generated virtual dyeing image is excellent in dyeing consistency and contrast, and the accuracy and efficiency of clinical diagnosis can be improved. Besides, a conventional fluorescence microscope is adopted in the method, so that the use of chemical reagents and complex experimental equipment is avoided, and a relatively high practical value is provided for clinical transformation of a virtual dyeing technology.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital pathology and artificial intelligence data processing, and specifically relates to a method and system for digital virtual staining of pathological sections based on a CycleGAN-based unsupervised model. Background Art

[0002] Pathological sections and their diagnosis play a central role in disease diagnosis and are important means for determining the nature of diseases, evaluating the degree of lesions, and guiding treatment decisions. They are widely used in fields such as tumors, inflammatory diseases, and infectious diseases. In pathological diagnosis, staining is an important step to highlight tissue characteristics and facilitate doctors' analysis. The traditional hematoxylin-eosin (H&E) staining method has become the "gold standard" in the field of pathology. The comparison between a blank section image and a real H&E-stained image is shown in Figure 1. Through H&E staining, the cell nucleus appears blue-purple, and the cytoplasm and stroma appear pink, thus clearly showing the tissue structure and lesion characteristics. However, this method takes a lot of time, relies on chemical reagents and professional equipment, and may cause loss of precious pathological samples. In clinical applications, histopathology is widely used for the diagnosis of various diseases, but the preparation process of pathological sections is usually time-consuming, laborious, and costly. To improve the diagnosis efficiency and reduce the impact of the staining process on samples, the field of pathology is gradually moving towards digitalization. Digital pathology has achieved high-resolution digitization of pathological sections through whole slide imaging (WSI) scanning and advanced image processing technologies. This progress enables the staining process to be replaced, providing doctors with intuitive diagnostic evidence through intelligent image analysis technologies. In this context, virtual staining technology has emerged and become a research hotspot in the field of digital pathology.

[0003] Virtual staining technology converts the original images of unstained tissue samples into images similar to the effects of traditional staining through deep learning models such as Generative Adversarial Network (GAN), providing visual performance comparable to physical staining. This process does not rely on chemical reagents and complex experimental procedures, significantly simplifies the preparation of pathological sections, and avoids sample loss at the same time. As shown in Figure 2 below, when comparing the flowcharts of virtual staining and traditional staining, both traditional staining and virtual staining need to obtain tissue paraffin sections first. The difference is that traditional staining requires the use of specific chemical reagents and goes through complex staining steps before obtaining WSI images using a digital scanner; virtual staining will avoid the use of chemical reagents, collect the original images using a physical microscope, and then use a virtual staining model to simulate the effects of traditional staining in the original images. Virtual staining technology has the following advantages: (1) Improve efficiency and reduce costs; the traditional staining process includes multiple complex steps, while virtual staining completes staining simulation in a short time through digital image processing, thus accelerating the speed of pathological diagnosis. (2) Reduce dependence on chemical reagents and equipment; virtual staining uses digital images, which are not restricted by geographical conditions and experimental equipment, and helps to promote the popularization and green development of pathological diagnosis. (3) Sample protection and information sharing; the digital images generated by virtual staining can be stored for a long time and are easy to share, providing convenience for remote diagnosis and cross-regional cooperation. At the same time, the samples are not damaged, which is convenient for the storage of rare samples. (4) Consistency and flexibility; virtual staining ensures consistent staining effects through algorithm standardization, eliminating differences in manual operations. In addition, "one-key switching" of multiple specific stainings can be achieved for the same section, providing the possibility of multi-angle analysis for complex pathological research.

[0004] At present, advanced imaging devices such as multiphoton microscopes and ultraviolet photoacoustic microscopes have shown important value in imaging unstained tissues. However, due to the lack of clear contrast of cell nuclei or chemical specificity, these methods are still difficult to meet the clinical diagnosis needs. By utilizing the autofluorescence characteristics of unstained tissues and collecting multimodal image information, especially the autofluorescence image at a wavelength of 488 nm and the DAPI image at a wavelength of 358 nm, histological features can be effectively presented and the information required for diagnosis can be provided. Based on this, virtual staining learns the mapping relationship between image domains through paired training of a large number of pre- and post-staining images, and realizes the conversion of unstained images to stained images, providing an innovative solution for pathological diagnosis. Although significant progress has been made in the field of virtual staining in existing research, the following challenges still remain: (1) The technical difficulty of obtaining unstained images; many advanced physical imaging devices (such as Raman scattering microscopes and nonlinear optical microscopes) can generate high-resolution images, but the devices are complex and have high operation requirements, which is not conducive to clinical promotion. (2) The limitation of the paired database; supervised deep learning models require precisely paired pre- and post-staining images. However, affected by problems such as section drift and tissue deformation, data acquisition is difficult and the matching algorithm is complex. Summary of the Invention

[0005] In order to overcome the above-mentioned disadvantages of the prior art, the purpose of the present invention is to provide a method and system for digital virtual staining of pathological sections based on a CycleGAN-based unsupervised model, so as to solve the technical problem of how to provide a multimodal virtual staining method based on unsupervised learning to replace traditional staining techniques and realize rapid and non-destructive analysis of pathological samples.

[0006] In order to achieve the above purpose, the present invention adopts the following technical solutions: The present invention discloses a method for digital virtual staining of pathological sections based on a CycleGAN-based unsupervised model, which is characterized by including the following steps: Collect the fluorescence image and DAPI image of an unstained paraffin section to obtain a bimodal fluorescence image; Perform H&E staining on the same paraffin section sample to obtain an H&E image; Based on the CycleGAN model, and using the bimodal fluorescence image and the H&E image to train the CycleGAN network to learn the mapping relationship of the conversion between image domains, and obtain the trained CycleGAN network; Input the original image of the unstained paraffin section into the trained CycleGAN network to obtain a virtual stained image.

[0007] Preferably, use a validation set to evaluate the virtual stained image, and adjust the hyperparameters of the CycleGAN network according to the staining consistency and visual effect.

[0008] Preferably, the CycleGAN model consists of four deep neural networks; The four deep neural networks include generator G, generator F, discriminator D x and discriminator D Y ; Generator G is used to convert fluorescence images into H&E images; Generator F is used to convert H&E images into fluorescence images; Discriminator D Y is used to distinguish between the fluorescence images generated by generator G and real fluorescence images; Generator G and discriminator D Y play a game of confrontation. When discriminator D Y cannot distinguish the H&E images generated by generator G, it indicates that generator G has learned the conversion from fluorescence images to H&E images; Discriminator D x is used to distinguish between the H&E images generated by generator F and real H&E images; Generator F and discriminator D x play a game of confrontation. When discriminator D x cannot distinguish the fluorescence images generated by generator F, it indicates that generator F has learned the conversion from H&E images to fluorescence images.

[0009] Further preferably, the objective function of the mapping relationship between generator G and generator F is:

[0010] In the formula, x ~ P ( x ) and y ~ P ( y ) are the original data distributions of fluorescence images and H&E images respectively; L GAN is the loss function of the discriminator; E is the mathematical expectation; x is the distribution of the fluorescence image domain; y is the distribution of the H&E image domain; G is the generator from the fluorescence image domain X to the H&E image domain Y; D Y is the discriminator of the H&E image domain; D Y ( y ) is the discriminant output of discriminator D Y for the real H&E image; D Y [G( x ) ] is the discriminant output of discriminator D Y for the H&E image generated by generator G; F is the generator from the H&E image domain Y to the fluorescence image domain X; D X is the discriminator of the fluorescence image domain; D X ( y ) is the discriminator DX Discriminant output for the real fluorescence image; D X [G( x ) ] is the discriminator D X Discriminant output for the fluorescence image generated by the generator F

[0011] Further preferably, the total loss function of the CycleGAN model is:

[0012]

[0013] In the formula: L is the total loss function of the CycleGAN model; Lcyc is the cycle consistency loss of the network; λ is the DX weighting factor, and the greater its value, the more meaningful it is to reduce the cycle consistency loss; G* is the optimization result of the generator G, representing the optimal mapping network from the fluorescence image domain X to the H&E image domain Y; F* is the optimization result of the generator F, representing the optimal mapping network from the H&E image domain Y to the fluorescence image domain X; is to find the best parameters of the generator G; is to find the best parameters of the generator F; is to maximize the loss functions of the discriminators DX and DY

[0014] Further preferably, structural similarity SSIM and learning perceptual image patch similarity LPIPS are added to the total loss function of the CycleGAN model to make the result closer to the real staining effect The structural similarity SSIM is:

[0015] Among them, x is the local patch of the x image patch; y is the local patch of the y image patch; µ x is the average brightness of the x image patch; µ y is the average brightness of the y image patch; σ 2 is the variance of the image patch, representing the contrast of the image patch; σ xy is the covariance of the image patch, used to measure the structural similarity between image patches; µ x 2 is the square of the average brightness of the fluorescence image patch x; µ y 2 is the square of the average brightness of the H&E image patch y; σ x 2 is the variance of the fluorescence image patch x; σ y 2 is the variance of the H&E image patch y; C 1 and C 2 are small constants used to prevent the denominator from being zero

[0016] Preferably, the learning perceptual image patch similarity LPIPS is:

[0017] Among them, x is the x image; y is the y image; Ф l represents the feature map of the neural network at the l-th layer; H l and W l respectively represent the height and width of the feature map at the l-th layer; the ⊙ symbol represents element-wise multiplication, which is used to adjust the weight of the feature difference; h is the index of a pixel in the height direction in the feature map at the l-th layer; w is the index of a pixel in the width direction in the feature map at the l-th layer; w l is the weight vector of the l-th layer, which is used to adjust the influence of each layer of features on the similarity, and w l can be trained or fixed; is the square of the L 2 norm regularization.

[0018] Preferably, label smoothing regularization technology is used to prevent overfitting of the CycleGAN network.

[0019] Preferably, the virtual stained image is subjected to staining normalization processing to standardize the hue and contrast of the output image.

[0020] The present invention also discloses a digital virtual staining system for pathological sections based on a CycleGAN-based unsupervised model, including: A dual-modal fluorescence image acquisition module for acquiring the autofluorescence image and DAPI image of an unstained paraffin section; An H&E staining image acquisition module for performing H&E staining on the same paraffin section sample to obtain an H&E staining image; A CycleGAN network training module for training the CycleGAN network based on the CycleGAN model and using the dual-modal fluorescence image and the H&E staining image to learn the mapping relationship of the conversion between image domains to obtain a trained CycleGAN network; A virtual staining image generation module for inputting the original image of the unstained paraffin section into the trained CycleGAN network to obtain a virtual staining image.

[0021] Compared with the prior art, the present invention has the following beneficial effects: A digital virtual staining method for pathological sections based on an unsupervised model of CycleGAN. This unsupervised virtual staining method based on the Cycle Generative Adversarial Network (CycleGAN) aims to combine multimodal information. It takes the autofluorescence (AF) image and DAPI image of unstained tissue sections as inputs and converts them into traditional hematoxylin and eosin (H&E) stained output images for pathological diagnosis. This method uses an unsupervised model, avoiding the dependence on pixel-level matching training sets and breaking through the limitations of traditional supervised models. Experimental results show that the generated virtual stained images perform excellently in terms of staining consistency and contrast, which can improve the accuracy and efficiency of clinical diagnosis. In addition, this method uses a conventional fluorescence microscope, avoiding the use of chemical reagents and complex experimental equipment, providing high practical value for the clinical transformation of virtual staining technology. It combines biomedical and digital image processing technologies to develop an intelligent virtual staining system that can replace traditional pathological staining techniques. The traditional staining process usually relies on chemical reagents to chemically stain pathological samples, which is time-consuming and laborious, may cause sample loss, and is also affected by the variability brought by operation dependence.

[0022] Through an advanced algorithm model, the present invention simulates the traditional staining effect, realizes the rapid and non-destructive analysis of pathological samples, and significantly reduces the variation dependent on operation. This system can automatically identify and process pathological images, and use intelligent diagnostic algorithms to extract key features in the samples, so as to achieve accurate analysis and classification. The application of this technology not only significantly improves the efficiency of pathological work, but also effectively avoids the potential damage of chemical staining to samples, showing broad application prospects. A deep learning model is used to convert bimodal fluorescence images into virtual H&E images. An unsupervised model is built to extract effective information from autofluorescence (AF) images and DAPI images to achieve virtual staining conversion, effectively solving the problem of difficult acquisition of original images and promoting the clinical practice of virtual staining; an unsupervised model is built by using and improving the CycleGAN model to achieve virtual staining conversion, effectively solving the problem that it is difficult to achieve pixel-level matching between the original image and the real staining image; staining normalization is introduced into the research of virtual staining to further standardize the staining results and make the virtual staining images more in line with the personal diagnosis habits of pathologists. The present invention uses the CycleGAN network model to convert unstained paraffin section images into virtual staining images similar to the traditional H&E staining effect. The acquisition of bimodal fluorescence images and H&E images of the same section is only used for the construction of the dataset. After the network training is completed, only unstained bimodal fluorescence images need to be acquired to generate virtual staining results, without the need for physical staining operations. The present invention completes the staining process under completely non-destructive conditions, avoiding the loss of precious samples in the traditional staining process. At the same time, the staining efficiency is greatly improved through digital image processing, without relying on chemical reagents, significantly reducing the operation complexity and experimental costs. In addition, the present invention not only generates high-quality images matching the H&E staining, but also provides a technical basis for the future realization of automatic conversion of various specific staining types (such as antibody-specific immunohistochemical staining). With the expansion ability of the deep learning model, this method has the potential for further development, providing more flexible and diverse solutions for complex pathological diagnosis and tissue research. Virtual staining technology performs excellently in the consistency and repeatability of staining results, providing strong support for the digital storage, remote diagnosis and cross-regional cooperation of pathological images. By simplifying the laboratory operation process and improving the diagnostic efficiency, the present invention provides a new technical path and application prospect for the development of the pathology field towards intelligence and digitization. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Comparison between a blank section image and a real H&E stained image; Figure 2 Comparison between the virtual staining flow chart and the traditional staining flow chart; Figure 3Flowchart of the digital virtual staining method for pathological sections based on the CycleGAN-based unsupervised model disclosed in the present invention; Figure 4 It is a CycleGAN image conversion model diagram; Figure 5 It is a flowchart of image homogenization based on the structure preservation method; Figure 6 It is the t-SNE distribution images before and after staining normalization; Figure 7 It is the cosine similarity distribution diagram between the Virtual-H&E image and the real H&E image patches; Figure 8 It is the perceptual hashing difference distribution diagram between the Virtual-H&E image and the real H&E stained image; Figure 9 It is the virtual staining and bimodal fluorescence local contrast diagram; Figure 10 It is the comparison diagram between the virtual staining result and the real H&E staining result. Detailed implementation manners

[0024] The present invention will be described in detail below with reference to the drawings and in combination with embodiments. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0025] The following detailed descriptions are all exemplary descriptions, aiming to provide further detailed descriptions of the present invention. Unless otherwise specified, all technical terms adopted by the present invention have the same meaning as commonly understood by those of ordinary skill in the art to which the present application belongs. The terms used in the present invention are only for describing specific implementation manners, and are not intended to limit the exemplary embodiments according to the present invention.

[0026] The present invention will be further described in detail below with reference to the drawings: Figure 1 It is the comparison between the blank section image and the real H&E stained image; It can be seen from the figure that through H&E staining, the cell nuclei show blue-violet, and the cytoplasm and stroma show pink, thus clearly showing the tissue structure and lesion characteristics. However, it takes a lot of time, depends on chemical reagents and professional equipment, and may cause loss of precious pathological samples.

[0027] Figure 2For the comparison between the virtual staining flow chart and the traditional staining flow chart, it can be seen from the figure that both traditional staining and virtual staining require obtaining tissue paraffin sections first. The difference is that traditional staining needs to use specific chemical reagents and go through complex staining steps, and then use a digital scanner to obtain WSI images; virtual staining will avoid the use of chemical reagents, collect the original images using a physical microscope, and then use a virtual staining model to simulate the effect of traditional staining in the original images.

[0028] Figure 3 This is the flow chart of the digital virtual staining method for pathological sections based on the CycleGAN-based unsupervised model disclosed by the present invention. It can be seen from the figure that the digital virtual staining method for pathological sections based on the CycleGAN-based unsupervised model includes the following steps: 1) Sample preparation and data collection 11) Obtain section samples from human or animal tissues, and perform standard paraffin embedding and formalin fixation treatments to obtain paraffin sections.

[0029] 12) Collect the autofluorescence image AF and DAPI image of the unstained tissue section under a physical microscope to obtain a dual-modal fluorescence image for network training and model inference.

[0030] 2) Training stage 21) Data collection: On the basis of collecting unstained images, perform H&E staining on the same samples to obtain the corresponding image data, i.e., H&E images, after staining.

[0031] 22) Network training: Construct a CycleGAN model and improve the model to better handle image conversion tasks. Use the pair of dual-modal fluorescence images and H&E images to train the CycleGAN network to learn the mapping relationship of the conversion between image domains, and obtain the trained CycleGAN network.

[0032] 23) Model optimization: Use the validation set to evaluate the output results of the model, and adjust the network hyperparameters according to the staining consistency and visual effects to ensure that the generated results meet the expected standards.

[0033] 3) Testing stage 31) Input the unstained original image into the trained CycleGAN network to generate a preliminary virtual staining result.

[0034] 32) Perform staining normalization on the preliminary virtual staining result to standardize the hue and contrast of the output image and ensure consistent staining effects.

[0035] 33) Finally, achieve high-quality conversion from the unstained image to the H&E image and provide virtual staining results for pathological diagnosis.

[0036] The present invention utilizes the CycleGAN network model to convert unstained paraffin section images into virtual stained images with similar effects to traditional H&E staining. By using a fluorescence microscope to collect the original AF images and DAPI images of the unstained sections and combining them with the CycleGAN model, it effectively avoids the dependence on a high-precision paired database in traditional supervised methods. During the network training stage, the AF image and DAPI image bimodal images are used as inputs, and the real H&E stained image is used as the output to construct a training dataset for network training. This method completes the staining process under completely non-destructive conditions, avoiding the loss of precious samples in the traditional staining process. At the same time, it greatly improves the staining efficiency through digital image processing, does not rely on chemical reagents, and significantly reduces the operation complexity and experimental costs. The present invention realizes the generation of high-quality virtual stained images, combining high efficiency and practicality, providing a new technical path for clinical pathological diagnosis. In addition, the present invention not only generates high-quality images matching H&E staining, but also provides a technical basis for the future realization of automated conversion of various specific staining types (such as antibody-specific immunohistochemical staining). With the expansion ability of the deep learning model, this method has the potential for further development, providing more flexible and diverse solutions for complex pathological diagnosis and tissue research. The virtual staining technology performs excellently in the consistency and repeatability of staining results, providing strong support for the digital storage, remote diagnosis, and cross-regional collaboration of pathological images. By simplifying the laboratory operation process and improving the diagnosis efficiency, the present invention provides a new technical path and application prospect for the development of the pathology field towards intelligence and digitization.

[0037] The digital virtual staining method for pathological sections based on the CycleGAN-based unsupervised model disclosed in the present invention specifically includes the following steps: 1) Sample preparation To meet the imaging requirements of a fluorescence microscope, the present invention uses a tissue paraffin section with a thickness of 2 μm as a sample. First, after dewaxing the section with xylene, it is adsorbed onto a standard glass slide and covered with a coverslip to protect the sample. Subsequently, DAPI reagent is dropped onto the unlabeled tissue sample, and the DAPI image under an excitation wavelength of 358 nm and the AF image under an excitation wavelength of 488 nm are respectively collected through a fluorescence microscope. After the collection is completed, the slide is immersed in xylene for about 48 hours or until the coverslip can be removed without damage; after removing the coverslip, the slide is immersed in absolute ethanol and 95% ethanol 30 times each in turn, and then washed with deionized water for about 1 minute to remove residual substances and prepare for the staining step. Finally, the section is stained with H&E dye, and the WSI is obtained through a digital pathology slide scanner. It should be particularly noted that the acquisition of the dual-modal fluorescence image and the H&E image of the same section is only used for the construction of the dataset. After the network training is completed, only the unstained dual-modal fluorescence image needs to be collected to generate the virtual staining result, and no physical staining operation is required anymore.

[0038] 2) Fluorescence microscope The dual-modal fluorescence image of the original section is collected using a fluorescence image microscope. The fluorescence light source range is 380 nm - 770 nm. An objective lens with a magnification of 40X and a numerical aperture of 0.95 is selected to collect the autofluorescence image. The collected fluorescence channels are Channel 1 (emission fluorescence wavelength range 380 nm - 399 nm) and Channel 2 (emission fluorescence wavelength range 490 nm - 507 nm), and the exposure times are 1 ms and 50 ms respectively. The AF images and DAPI images of tissue sections from 4 different parts of the human liver are collected, and the average size of each image is 2.5 GB.

[0039] 3) Model construction The CycleGAN model is adopted. The CycleGAN model consists of four deep neural networks, two generators (G, F), and two discriminators (D X 、D Y ). The structure of the CycleGAN model is shown in Figure 4.

[0040] Figure 4This is the diagram of the CycleGAN image conversion model. As can be seen from the figure, the image synthesized from the AF image and the DAPI image is used as the input, which is called the original fluorescence image (Real Autofluorescence image, denoted as Real-AF-image). It is converted into a Virtual-H&E image by the generator G, while the generator F converts the Virtual-H&E image back into a fluorescence image (Reconstructed Autofluorescence image, denoted as Rec-AF-image). The real H&E image (Real Hematoxylin-Eosin Staining image, denoted as Real-H&E) is converted into a fake fluorescence image (Fake Autofluorescence image, denoted as Fake-AF-image) by the generator F, and the Fake-AF-image is then converted into a reconstructed H&E (Reconstructed H&E, denoted as Rec-H&E) image by the generator G. The discriminator X (Discriminator X, abbreviated as D X ) aims to distinguish between the Real-AF-image and the generated Fake-AF-image, while the discriminator Y (D Y ) aims to distinguish between the Real-H&E image and the generated Virtual-H&E image. Once the generator G can produce a Virtual-H&E image that D Y cannot distinguish, it means that the generator G has learned the conversion from the fluorescence image to the H&E image. The principles of the generator F and the discriminator D X are similar.

[0041] To prevent the situation where the content in the Real-AF-image is lost when generating the Virtual-H&E image, the loss function is calculated by converting F(Virtual-H&E) into Rec-AF-image and then comparing it with the Real-AF-image, making F(G(Real-AF-image)) ≈ Real-AF-image. The lower part of the structure first maps the Real-H&E image through F to generate F(Real-H&E) = Fake-AF-image, and then maps it through G to generate G(Fake-AF-image) = Rec-H&E, finally achieving G(F(Real-H&E)) ≈ Rec-H&E.

[0042] The objective functions of the generation networks G and F are defined as follows:

[0043] In the formula, x ~P ( x ) and y ~ P ( y ) are the original data distributions of the fluorescence image and the H&E image respectively; L GAN is the loss function of the discriminator; E is the mathematical expectation; x is the distribution of the fluorescence image domain; y is the distribution of the H&E image domain; G is the generator G from the fluorescence image domain X to the H&E image domain Y; D Y is the discriminator of the H&E image domain; D Y ( y ) is the discriminant output of the discriminator D Y for the real H&E image; D Y [G ( x ) ] is the discriminant output of the discriminator D Y for the H&E image generated by the generator G; F is the generator F from the H&E image domain Y to the fluorescence image domain X; D X is the discriminator of the fluorescence image domain; D X ( y ) is the discriminant output of the discriminator D X for the real fluorescence image; D X [G ( x ) ] is the discriminant output of the discriminator D X for the fluorescence image generated by the generator F.

[0044] The total loss function of the CycleGAN model is:

[0045]

[0046] In the formula: L is the total loss function of the CycleGAN model; Lcyc is the cycle consistency loss of the network; λ is the DX weighting factor, and the larger its value, the more meaningful it is to reduce the cycle consistency loss; G* is the optimization result of the generator G, representing the optimal mapping network from the fluorescence image domain X to the H&E image domain Y; F* is the optimization result of the generator F, representing the optimal mapping network from the H&E image domain Y to the fluorescence image domain X; is to find the best parameters of the generator G to minimize the loss function; is to find the best parameters of the generator F to minimize the loss function; is to maximize the loss functions of the discriminators DX and DY.

[0047] To enable the model to better handle this task, the present invention proposes an improved loss function by adding Structural Similarity Index (SSIM for short) and Learned Perceptual Image Patch Similarity (LPIPS for short) to the original loss function. Its advantage lies in that it can better simulate the human eye's perception of images, making the Virtual-H&E results closer to the real staining effect within the visual range. The SSIM function is mainly used to measure the similarity of the structural information, brightness, and contrast of images. Compared with simple mean squared error (MSE) or peak signal-to-noise ratio (PSNR), it is more in line with the evaluation of image quality by the human visual system. Its expression is as follows:

[0048] where x and y are different local patches of two images respectively; x is the local patch of the x image patch; y is the local patch of the y image patch; µ x is the average brightness of the x image patch; µ y is the average brightness of the y image patch; σ 2 is the variance of the image patch, representing the contrast of the image patch; σ xy is the covariance of the image patch, used to measure the structural similarity between image patches; µ x 2 is the square of the average brightness of the fluorescence image patch x; µ y 2 is the square of the average brightness of the H&E image patch y; σ x 2 is the variance of the fluorescence image patch x; σ y 2 is the variance of the H&E image patch y; C 1 and C 2 are small constants used to prevent the denominator from being zero.

[0049] LPIPS is a deep learning-based perceptual similarity metric method, mainly used to measure the similarity between two images in human visual perception. The convolutional neural network AlexNet pre-trained on large datasets such as ImageNet is used to extract image features, and then the distance between the feature vectors is calculated. Compared with traditional methods such as SSIM, LPIPS pays more attention to the high-level semantic information of images (such as texture, shape, color, etc.). Its expression is as follows:

[0050] where x and y are two images to be compared; x is the x image; y is the y image; Ф lDenote the feature map of the neural network at the $l$-th layer; $H$ l and $W$ l respectively represent the height and width of the feature map at the $l$-th layer; the $\odot$ symbol represents element-wise multiplication, which is used to adjust the weights of feature differences; $h$ is the index of a pixel in the height dimension of the feature map at the $l$-th layer; $w$ is the index of a pixel in the width dimension of the feature map at the $l$-th layer; $w$ l is the weight vector (trainable or fixed) of the $l$-th layer, which is used to adjust the influence of each layer of features on similarity.

[0051] And to prevent overfitting of the network, label smoothing regularization technology is used to improve the generalization performance of the model and make it perform better on test data. In traditional classification tasks, labels are usually one-hot encoded. Such hard labels may cause the probabilities output by the model to tend to extreme values (such as approaching 1 or 0), which is prone to overfitting, especially when there is noise in the training data. The basic idea of label smoothing is to "smooth" the one-hot encoded labels into values that are close to one-hot but not exactly 0 or 1. For example, in the case of 3 classes, the label can be transformed from the original $y$ original to $y$ smoothrd in the following form:

[0052] In this way, the probability of the correct class of the label is reduced from 1 to a smaller value (such as 0.8), and the remaining classes are assigned lower but non-zero probabilities.

[0053] 4) Image post-processing In diagnosis, pathologists usually rely on the color and detail features of tissue sections under the microscope for analysis. However, due to differences in operation, staining conditions, and the specifications of scanning devices during the staining process, there may be significant variations in the color intensity of tissue sections. This inconsistency will interfere with the doctor's visual judgment, affect the accuracy of the diagnosis result, and at the same time reduce the reliability of the computer-aided diagnosis system.

[0054] To solve this problem, the present invention introduces a staining normalization technology in the post-processing stage of virtual staining. The core of this technology is to standardize the staining color of Virtual-H&E to make it consistent with the staining distribution of the reference image, thereby improving the visual consistency of the image. Through unified color correction, staining normalization ensures that the hue of the Virtual-H&E image is close to the true H&E staining effect, avoiding the situation where color deviation affects the doctor's diagnosis.

[0055] This post - processing technique can not only significantly improve the reliability of Virtual - H&E images in clinical applications, but also enhance the adaptability of pathologists, helping them interpret images more efficiently. By improving the consistency and readability of images, this technique effectively improves the speed and accuracy of pathological diagnosis, providing strong technical support for intelligent pathological diagnosis.

[0056] Figure 5 It is a flowchart of image normalization based on a structure - preserving method; as can be seen from the figure, in order to achieve the normalization of stained images, first, the image needs to be color - separated. The RGB image obtained after H&E staining is converted into a staining basis matrix W and a staining density map H. Based on the Beer - Lambert theorem, stained tissues attenuate light in a specific spectrum according to the type and amount of stain they absorb. Therefore, assuming that the RGB value of each pixel in a pathological section is represented by the symbol I, when acquiring the scanned section, the illumination intensity of the instrument is I 0 , and the relative optical density matrix V at this time is expressed as follows:

[0057] Then, use a sparse non - negative matrix to perform color separation on V to obtain the staining basis matrix W and the staining density map H, that is:

[0058] Then the stained image is:

[0059] Explanation of post - processing results. Using a structure - preserving method to normalize the network output image only multiplies the staining target H by a scalar, which can keep the relative staining density map of the source image unchanged, enabling the Virtual - H&E image output by the network to be further normalized in terms of color, brightness, etc. while retaining the original texture structure, reducing staining differences, and being more in line with the reading habits of pathologists.

[0060] To visually understand the impact of the normalization process on the image data distribution through data visualization, t - SNE (t - Distributed Stochastic Neighbor Embedding) images of the images before and after normalization are plotted. t - SNE is a dimensionality reduction method that maps high - dimensional data into two - dimensional or three - dimensional space while preserving the local structure of the data, capable of showing the relative positions of high - dimensional data points in the low - dimensional space and reflecting the data distribution pattern.

[0061] Figure 6t-SNE distribution images before and after staining normalization; it can be seen from the figure that the t-SNE dimensionality reduction results of the original Virtual-H&E results and the normalized staining results are almost the same in the three-dimensional space. The image data distribution patterns before and after staining normalization are almost the same, and the organizational structure after normalization is preserved intact, indicating that this post-processing operation will not affect the basic structure of the virtual staining data, but can achieve the matching effect of color and brightness, and improve the accuracy of virtual staining.

[0062] Example A digital virtual staining method for pathological sections based on the CycleGAN unsupervised model disclosed by the present invention includes the following steps: 1) Dataset First, select a liver tissue sample with a thickness of 2 μm, and use a fluorescence microscope to collect AF images at an excitation light wavelength of 488 nm and DAPI images at an excitation light wavelength of 358 nm. Subsequently, perform H&E staining on the sample, and use a digital pathological section scanner to collect the stained H&E images. During the construction of the dataset, ensure that the AF images and DAPI images are from the same field of view as the H&E images to ensure data consistency. The images are cropped into image patches with a size of 256×256 pixels, and the initial database is screened to remove blurred or stained images. In the finally constructed dataset, there are 2952 dual-modal autofluorescence images and 2952 H&E images each, providing high-quality data for subsequent model training and verification.

[0063] 2) Experimental process The image size is adjusted to 256*256, the batch size is 1, that is, 1 image is used for each training, the number of iterations in one training epoch is 2952 times, the model is trained for 200 epochs, the learning rate for the first 100 epochs is 0.0002, the learning rate for the last 100 epochs is 0.0001, and the Adam optimizer with a hyperparameter of 0.5 is used. The generator G network of the CycleGAN model selects the RENet_9blocks model, and the experiment uses the Python language.

[0064] 3) Experimental results 216 liver section image patches with a size of 256*256 are obtained in the experiment to verify the correctness of the virtual staining model. Since the SSIM and LPIPS perception functions are added as loss functions in the loss function, the cosine similarity and perceptual hashing difference are selected to evaluate the similarity between the Virtual-H&E results and the real H&E results during evaluation.

[0065] In the field of images, cosine similarity evaluation relies on the similarity of the feature vectors of two different images. It is a method of measuring the similarity between two vectors by the cosine value of the angle between the vectors. Its value range is [-1, 1], where 1 indicates complete similarity, 0 indicates complete dissimilarity, and -1 indicates complete opposition. Extracting the feature vectors of an image can convert the information of the image into high-dimensional feature vectors, while preserving the content of the image and avoiding the influence of pixel-level deviations between two different images on the evaluation of the image structure. When calculating the feature vectors, cosine similarity focuses on the overall semantics and global feature expression of the image, and can effectively measure the visual similarity of the image. The expression formula of cosine similarity is as follows:

[0066] where, A·B represents the inner product of two vectors; ||A|| represents the norm of vector A; ||B|| represents the norm of vector B; A i represents the i-th element in vector A; B i represents the i-th element in vector B; A i 2 represents A i squared; B i 2 represents B i squared.

[0067] Figure 7 is the cosine similarity distribution graph between Virtual-H&E images and real H&E image patches; it can be seen from the figure that for the 216 Virtual-H&E image patches obtained in the experiment, the cosine similarity between Virtual-H&E and real H&E image patches is obtained, and the statistical results of the cosine similarity are shown in Table 1 below, and the statistical image is drawn as Figure 7 shown.

[0068] Table 1 Statistical results of cosine similarity

[0069] It can be seen from the statistical results graph 7 and Table 1 of the experimental data that Virtual-H&E images show relatively high quality in terms of visual similarity. The analysis based on cosine similarity shows that the average value of the cosine similarity between Virtual-H&E and real H&E staining results is 0.9688, and the median is 1. Among them, the number of image patches with a cosine similarity of 1 (completely similar) is 205, accounting for 94.60% of the total experimental samples. This result indicates that in terms of overall visual characteristics, Virtual-H&E images are highly consistent with the results of real H&E images and can better restore the visual features of traditional H&E staining.

[0070] Figure 8 It is the distribution diagram of the perceptual hashing difference between the Virtual-H&E image and the real H&E stained image; it can be seen from the figure that the cosine similarity mainly compares the similarity based on the high-dimensional information of the image, while the perceptual hashing difference focuses on the analysis of the differences in the low-level visual features of the image. The perceptual hashing method quickly extracts the global information of the image by reducing the image to a binary hash value, and the statistical distribution of the perceptual hashing difference is shown in Figure 8.

[0071] The experimental results show that the mean value of the perceptual hashing difference is 31.40 and the median is 32, indicating that there are certain visual differences between the Virtual-H&E image and the real H&E stained image, but this difference is generally small. The median of 32 indicates that the perceptual differences of most images are concentrated between 30 and 34. The first quartile (Q1) is 28 and the third quartile (Q3) is 34, further indicating that most of the range of image differences is between 28 and 34, showing the proximity of Virtual-H&E to the real staining results in terms of low-level visual features.

[0072] For images with low perceptual differences (the difference value is close to 20), the Virtual-H&E results are almost visually identical to the real staining results, and the model shows a high reduction ability on these samples. However, for images with high perceptual difference values (close to 42), the performance of the model is relatively weak, especially in areas with edge details or complex structures. This difference is mainly reflected in the restoration of complex features in the image, indicating that there is still room for improvement in the model's detail processing.

[0073] A comparative display analysis of the autofluorescence image (the image combined with the AF image and the DAPI image), the virtual staining result, and the real H&E staining result is presented.

[0074] Figure 9For the virtual staining and bimodal fluorescence local contrast images, it can be seen from the figure that Figure 9 shows the original bimodal fluorescence image and the Virtual-H&E image. Among them, four images labeled ABCD show the Virtual-H&E results, and four images labeled abcd show the bimodal fluorescence image combined with the AF image and DAPI image of the original tissue section. The information marked by the red arrows in the figure is the nuclear information. It can be seen from the four groups of comparison images that the Virtual-H&E results can not only mark the position of the nucleus at the exact position but also ensure the integrity of the contour. Even in the case of multiple nuclear adhesions, as shown by the black arrows in the figure, the cases of binuclear adhesion and multinuclear adhesion, each nucleus can be accurately marked as blue-violet in the Virtual-H&E results. Comparing the tissue texture details of the four groups of images, the Virtual-H&E stained image can not only effectively process the nucleus, cytoplasm, proteins in the extracellular matrix, etc. into different colors, but also the tissue texture distribution of the original fluorescence image can be seen in the Virtual-H&E image, indicating that the virtual staining results can highlight the tissue texture details within the naked eye range.

[0075] Figure 10 For the comparison chart of the virtual staining results and the real H&E staining results; it can be seen from the figure that the Virtual-H&E results and the real H&E staining results are shown as in Figure 10. Image A represents the Virtual-H&E results, and image a represents the real H&E results. The details of the red and green boxes in image A and image a are respectively shown on the right side of Figure 10. The red image block represents the detailed display of the Virtual-H&E results, and the green image block represents the detailed display of the real H&E results. From the overall experimental results, the virtual staining technology can better simulate the effect of traditional H&E staining. The Virtual-H&E image can clearly reproduce the staining of the nucleus and cytoplasm in the tissue section. The nucleus is blue-violet, and the cytoplasm is pink or light red, which is highly similar to the real H&E staining effect, and it performs excellently in the global staining mode and the presentation of tissue structure.

[0076] Specific analysis shows that the colors of Virtual-H&E are more uniform, without color transition regions. The cell nuclei appear blue-violet, and the cytoplasm, proteins in the extracellular matrix, etc. appear pink. In real H&E staining, as shown in the blue circle area in Image a, the staining results of the cytoplasm, proteins in the extracellular matrix, etc. vary in color intensity. From a detailed perspective, Virtual-H&E can not only accurately label the cell nuclei, as indicated by the small blue arrows in the right image block of Figure 10, but also effectively handle the problem of cell nucleus holes in real staining results. In the green image frame of Figure 10 (i.e., the real H&E staining image), the black arrow indicates an obvious hole in the cell nucleus, but there is no hole in the cell nucleus in the red image block (i.e., the Virtual-H&E image).

[0077] In summary, by comparing the dual-modal fluorescence images, Virtual-H&E images, and real H&E staining images, it can be found that the present invention can effectively combine multi-modal information to convert dual-modal fluorescence images into high-quality virtual H&E images. The converted Virtual-H&E images clearly highlight the detailed features of tissue sections in the H&E staining space, conform to the reading habits of pathologists, and provide reliable support for clinical diagnosis. In addition, the virtual staining technology has extremely high efficiency and can complete the image staining conversion of 216 images (256×256 pixels) within seconds, significantly shortening the diagnosis time. This technology not only effectively reduces the waiting burden of patients, but also significantly improves the diagnostic efficiency of doctors, providing a new technical means for rapid and accurate pathological diagnosis.

[0078] Based on the CycleGAN model, the present invention successfully solves the problems of complex operation, time-consuming, and sample loss in traditional staining techniques, and proposes an efficient and non-destructive virtual staining method suitable for rapid pathological diagnosis. Experimental results show that the generated Virtual-H&E images are highly similar to real H&E staining images in visual effects and can meet the accuracy requirements of clinical diagnosis. At the same time, this method significantly reduces the fluctuations in diagnostic results caused by operation differences, improves the consistency of staining results, and shows broad application value. In addition, by adopting the post-processing technology of staining homogenization, the visual consistency of the images is further improved, optimizing the diagnostic experience of pathologists, thereby effectively improving the efficiency of clinical diagnosis. The present invention provides an innovative solution for the intelligent development of the pathology field and has important potential for clinical promotion.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: it is still possible to modify the specific implementation manners of the present invention or make equivalent substitutions, and any modification or equivalent substitution that does not depart from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. A digital virtual staining method for pathological sections based on a CycleGAN-based unsupervised model, characterized in that: The following steps are involved: Fluorescence images and DAPI images of unstained paraffin sections were collected to obtain dual-modality fluorescence images; The same paraffin section samples were subjected to H&E staining to obtain H&E images; Based on the CycleGAN model, the dual-modality fluorescence images and H&E images are used to train the CycleGAN network to learn the mapping relationship between image domains, and the trained CycleGAN network is obtained; The original image of the unstained paraffin section was input into the trained CycleGAN network to obtain a virtual stained image.

2. The pathological section digital virtual staining method of the CycleGAN-based unsupervised model according to claim 1, characterized in that: The virtual dyed images were evaluated using the validation set, and the CycleGAN network hyperparameters were adjusted based on dyeing consistency and visual effects.

3. The pathological section digital virtual staining method of the CycleGAN-based unsupervised model according to claim 1, characterized in that: The CycleGAN model consists of four deep neural networks; The four deep neural networks include generator G, generator F, discriminator D x and the discriminator D Y ; The generator G is used to convert the fluorescence image into an H&E image; The generator F is used to convert the H&E image into a fluorescent image; The discriminator D Y Used to distinguish the fluorescence image generated by the generator G and the real fluorescence image; generator G and discriminator D Y Game confrontation, when the discriminator D Y When the H&E image generated by generator G cannot be identified, it indicates that generator G has learned the conversion from fluorescent image to H&E image; The discriminator D x Used to distinguish the H&E images generated by the generator F and the real H&E images; generator F and discriminator D x Game confrontation, when the discriminator D x When the fluorescence image generated by generator F cannot be identified, it indicates that generator F has learned the conversion from H&E images to fluorescence images.

4. The pathological section digital virtual staining method of the CycleGAN-based unsupervised model according to claim 3, characterized in that: The mapping relationship objective function between the generator G and the generator F is: In the formula, x ~ P ( x )and y ~ P ( y ) are the raw data distributions of fluorescence image and H&E image respectively; L GAN is the loss function of the discriminator; E is the mathematical expectation; x is the distribution of the fluorescence image domain; y is the distribution of H&E image domain; G is the generator from fluorescence image domain X to H&E image domain Y; D Y is the discriminator for the H&E image domain; D Y ( y ) is the discriminator D Y The discriminant output of the real H&E image; D Y [G( x ) ] is the discriminator D Y The discriminant output of the H&E image generated by the generator G; F is the generator from the H&E image domain Y to the fluorescence image domain X; D X is the discriminator in the fluorescence image domain; D X ( y ) is the discriminator D X The discriminant output of the real fluorescence image; D X [G( x ) ] is the discriminator D X The discriminant output of the fluorescence image generated by generator F.

5. The pathological section digital virtual staining method of the CycleGAN-based unsupervised model according to claim 4, characterized in that: The total loss function of the CycleGAN model is: Where: L is the total loss function of the CycleGAN model; L cyc is the cycle-consistent loss of the network; λ is D X The weighting factor, the larger its value, the more meaningful it is to reduce the cycle consistency loss; G* is the optimization result of the generator G, which represents the optimal mapping network from the fluorescence image domain X to the H&E image domain Y; F* is the optimization result of the generator F, which represents the optimal mapping network from the H&E image domain Y to the fluorescence image domain X; To find the best parameters of the generator G; To find the best parameters of generator F; To maximize the discriminator D X and D Y The loss function of .

6. The pathological section digital virtual staining method of the CycleGAN-based unsupervised model according to claim 5, characterized in that: Adding structural similarity SSIM and learning-perceptual image patch similarity LPIPS to the total loss function of the CycleGAN model makes the result closer to the real coloring effect; The structural similarity SSIM is: Where x is a local block of the x image block; y is a local block of the y image block; µ x is the average brightness of the x image block; µ y is the average brightness of the y image block; σ 2 is the variance of the image block, indicating the contrast of the image block; σ xy is the covariance of the image blocks, which is used to measure the structural similarity between image blocks; µ x 2 is the square of the average brightness of the fluorescent image block x; µ y 2 is the square of the average brightness of H&E image block y; σ x 2 is the variance of the fluorescence image patch x; σ y 2 is the variance of the H&E image patch y; C1 and C2 are small constants used to prevent the denominator from being zero.

7. The pathological section digital virtual staining method based on the CycleGAN unsupervised model according to claim 1, characterized in that: The learning-perceptual image block similarity LPIPS is: Among them, x is the x image; y is the y image; Ф l H represents the feature map of the neural network at layer l; l and W l denote the height and width of the feature map of the first layer, respectively; the ⊙ symbol denotes element-wise multiplication, which is used to adjust the weight of feature differences; h is the index of a pixel in the feature map of the first layer in terms of height; w is the index of a pixel in the feature map of the first layer in terms of width; w l is the weight vector of layer l, which is used to adjust the influence of each layer feature on the similarity. l Can be trained or fixed; is the square of L2 norm regularization.

8. The pathological section digital virtual staining method based on the CycleGAN unsupervised model according to claim 1, characterized in that: Label smoothing regularization technique is used to prevent overfitting of the CycleGAN network.

9. The pathological section digital virtual staining method of the CycleGAN-based unsupervised model according to claim 1, characterized in that: The virtual dyed image is subjected to dyeing homogenization processing to standardize the tone and contrast of the output image.

10. A digital virtual staining system for pathological sections based on a CycleGAN-based unsupervised model, characterized in that: include: Dual-modality fluorescence image acquisition module, used to acquire autofluorescence images and DAPI images of unstained paraffin sections; H&E staining image acquisition module, used to perform H&E staining on the same paraffin section sample to obtain H&E staining images; The CycleGAN network training module is used to train the CycleGAN network based on the CycleGAN model and use the dual-modal fluorescence image and H&E staining image to learn the mapping relationship between the image domains and obtain the trained CycleGAN network; The virtual stained image generation module is used to input the original image of the unstained paraffin section into the trained CycleGAN network to obtain a virtual stained image.

Images