An IHC image generation method and system based on a generative adversarial network

Abstract

The application relates to an IHC image generation method and system based on a generative adversarial network, and relates to the technical field of computer image processing.The method comprises the following steps: acquiring a frozen H&E image of a target part slice of a detector; performing segmentation on the frozen H&E image to obtain a plurality of frozen H&E subimages; inputting the plurality of frozen H&E subimages into a trained paraffin H&E image generator respectively to obtain a paraffin H&E subimage corresponding to each frozen H&E subimage; inputting each paraffin H&E subimage into a trained IHC image generator respectively to obtain an IHC subimage corresponding to each paraffin H&E subimage; and splicing all the IHC subimages to obtain an IHC image.The paraffin H&E image generator is used to generate a paraffin H&E image from a frozen H&E image, and then the IHC image generator is used to generate an IHC image from the paraffin H&E image, so that the time for generating the IHC image is reduced.

Description

Technical Field

[0001] This invention relates to the field of computer image processing technology, and in particular to an IHC image generation method and system based on generative adversarial networks. Background Technology

[0002] Cancer is the second leading cause of death today, and pathological analysis plays a crucial role in intraoperative cancer diagnosis. As a core component of pathological analysis, histochemical staining distinguishes different tissue components by enhancing color differences, allowing different parts of the tissue to exhibit different colors. Some common staining agents, such as hematoxylin and eosin (H&E) staining, can stain the cell nucleus and cytoplasm purple and pink respectively, facilitating pathologists' observation of cellular structure. However, based solely on H&E images, pathologists cannot determine the positive or negative expression of certain genes in a patient, thus hindering surgical guidance. Immunohistochemical staining (IHC staining), through antigen-antibody binding, can differentiate between negative and positive cells, enabling pathologists to make more accurate diagnoses. For example, in the analysis of thyroid cancer tissue, pathologists typically use IHC staining to analyze the expression of genes such as Ki67, TTF-1, and TG, thereby determining the type of gene mutation and enabling targeted treatment.

[0003] However, in clinical practice, IHC staining takes much longer than H&E staining. Typically, it takes about 15 minutes to prepare an H&E slide during surgery, while it takes three to five days to prepare an IHC slide. This seriously hinders the application of IHC slide images during surgery. Summary of the Invention

[0004] The purpose of this invention is to provide an IHC image generation method and system based on generative adversarial networks, which reduces the time required to generate IHC images.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] An IHC image generation method based on generative adversarial networks includes:

[0007] Acquire frozen H&E images of tissue sections from the target area of ​​the subject;

[0008] The frozen H&E image is segmented to obtain several frozen H&E sub-images, and the location information of the frozen H&E sub-images is labeled.

[0009] Several frozen H&E sub-images are input into a trained paraffin H&E image generator to obtain a paraffin H&E sub-image corresponding to each frozen H&E sub-image; the trained paraffin H&E image generator is a generator trained with sample frozen H&E sub-images as input and sample paraffin H&E sub-images as labels.

[0010] Each of the paraffin H&E sub-images is input into the trained IHC image generator to obtain the IHC sub-image corresponding to each of the paraffin H&E sub-images; the trained IHC image generator is a generator trained with sample paraffin H&E sub-images as input and sample IHC sub-images as labels.

[0011] All the IHC sub-images are stitched together based on the labeled location information to obtain the IHC image.

[0012] Optionally, before inputting the plurality of frozen H&E sub-images into the trained paraffin H&E image generator, the method further includes:

[0013] Construct a first generative adversarial network; the first generative adversarial network includes a paraffin H&E image generator, a high-resolution decision maker, and a low-resolution decision maker;

[0014] The first generative adversarial network is trained to obtain a trained first generative adversarial network; the paraffin H&E image generator of the trained first generative adversarial network is the trained paraffin H&E image generator.

[0015] When training the first generative adversarial network, the loss functions used include kernel constraint loss function, content consistency constraint loss function and texture consistency constraint loss function;

[0016] The expression for the kernel constraint loss function is:

[0017] L H =||f(x)-f(x')||1

[0018] Among them, L H denoted by kernel constraint loss function value, x represents the frozen H&E sub-image of the sample, x' represents the virtual paraffin H&E sub-image of the sample, f() is the cell kernel segmentation algorithm, f(x) is the cell kernel of the frozen H&E sub-image of the sample, and f(x′) is the cell kernel of the virtual paraffin H&E sub-image of the sample.

[0019] The expression for the content consistency constraint loss function is:

[0020]

[0021] Among them, LNCE (E,T) represents the content consistency constraint loss function value between the sample augmented frozen H&E sub-image E and the texture feature map T, and g(T,m) represents a vector in the m-th layer feature block of the texture feature map T. + Then g(E,m) represents the vector at the position corresponding to g(T,m) in the m-th layer feature block of the sample augmented frozen H&E sub-image. n - represents the vector in the m-th layer feature block of the augmented frozen H&E sub-image that does not correspond to g(T,m); τ represents the temperature coefficient used for balancing the distribution; N represents the number of times the vector at the non-corresponding position is sampled; S L M represents the number of vectors randomly selected from the feature blocks of the texture feature map; M represents the total number of feature layers.

[0022] The expression for the texture consistency constraint loss function is:

[0023] L Texture =||x' sf -T cj ||1

[0024] Among them, L Texture x' represents the value of the texture consistency constraint loss function. sf T represents the image after the sample virtual paraffin H&E sub-image has been scaled up by a preset ratio. cj This represents an image after the texture feature map has been cropped to a preset cropping range.

[0025] Optionally, training the first generative adversarial network specifically includes:

[0026] Obtain a first sample set; the first sample set includes several sample frozen H&E sub-images and sample paraffin H&E sub-images, sample expanded frozen H&E sub-images, and sample expanded paraffin H&E sub-images corresponding to each sample frozen H&E sub-image.

[0027] The frozen H&E sub-image of the sample is input into the paraffin H&E image generator to generate a virtual paraffin H&E sub-image of the sample; based on the frozen H&E sub-image of the sample and the virtual paraffin H&E sub-image of the sample, the kernel constraint loss function value is calculated using the kernel constraint loss function;

[0028] The sample-enlarged frozen H&E sub-image is input into the paraffin H&E image generator to generate a texture feature map; based on the sample-enlarged frozen H&E sub-image and the texture feature map, the content consistency constraint loss function value is calculated using the content consistency constraint loss function.

[0029] Based on the sample virtual paraffin H&E sub-image and the texture feature map, the texture consistency constraint loss function value is calculated using the texture consistency constraint loss function.

[0030] The high-resolution decision-maker is used to distinguish between real and fake samples of virtual paraffin H&E sub-images and samples of paraffin H&E sub-images, and to obtain the high-resolution score loss value of the sample virtual paraffin H&E sub-image and the loss function value of the first paraffin H&E decision-maker.

[0031] The low-resolution decision maker is used to distinguish between real and fake texture feature maps and sample augmented paraffin H&E sub-images, and the low-resolution score loss value of the texture feature map and the loss function value of the second paraffin H&E decision maker are obtained.

[0032] The total loss function value of the paraffin H&E generator is obtained by weighted summing of the kernel constraint loss function value, the content consistency constraint loss function value, the texture consistency constraint loss function value, the high resolution score loss value, and the low resolution score loss value.

[0033] The paraffin H&E image generator is updated based on the total loss function value of the paraffin H&E generator to obtain the updated paraffin H&E image generator.

[0034] The total loss function value of the decision device is calculated based on the loss function value of the first paraffin H&E decision device and the loss function value of the second paraffin H&E decision device.

[0035] The high-resolution decision and the low-resolution decision are updated based on the total loss function value of the decision, resulting in the updated high-resolution decision and the updated low-resolution decision.

[0036] Determine whether the preset iteration condition has been met. If yes, stop the iteration and use the updated paraffin H&E image generator, the updated high-resolution decision maker, and the updated low-resolution decision maker as the paraffin H&E image generator, high-resolution decision maker, and low-resolution decision maker in the trained first generative adversarial network, respectively. If no, return to the step "input the sample frozen H&E sub-image into the paraffin H&E image generator" until the preset iteration condition is met.

[0037] Optionally, before calculating the kernel constraint loss function value using the kernel constraint loss function based on the frozen H&E sub-image of the sample and the virtual paraffin H&E sub-image of the sample, the method further includes:

[0038] The H-staining images of the frozen H&E sub-image and the virtual paraffin H&E sub-image of the sample are extracted using the color deconvolution method, resulting in the H-staining images of the frozen H&E sub-image and the virtual paraffin H&E sub-image of the sample.

[0039] The H-stained image of the frozen H&E sub-image of the sample is segmented and extracted according to the set nuclear threshold to obtain the cell nucleus of the frozen H&E sub-image of the sample;

[0040] The cell nuclei in the H-stained image of the virtual paraffin H&E sub-image of the sample are segmented and extracted according to the set nuclear threshold to obtain the cell nuclei of the virtual paraffin H&E sub-image of the sample.

[0041] Optionally, before obtaining the first sample set, the method further includes:

[0042] Obtain frozen H&E sub-images of the samples;

[0043] The frozen H&E sub-images of the samples were annotated;

[0044] Based on the marked location information, the frozen H&E sub-image of the sample is stitched together with some frozen H&E sub-images of other samples within a preset range outside the frozen H&E sub-image of the sample to obtain the sample expanded frozen H&E sub-image.

[0045] Optionally, before inputting each of the paraffin H&E sub-images into the trained IHC image generator, the method further includes:

[0046] A second generative adversarial network (GAN) is constructed. This GAN includes an IHC image generator, an IHC decision maker, an H&E image generator, and an H&E decision maker. The GAN is an improved CycleGAN model, with a first yin-yang classification module added to the IHC decision maker and a second yin-yang classification module added to the H&E decision maker. The first yin-yang classification module is a model trained using sample IHC sub-images as input and sample IHC sub-images with yin-yang labels as labels. The yin-yang labels are the yin-yang information obtained after IHC staining of sample patient tissue sections. The second yin-yang classification module is a model trained using sample paraffin H&E sub-images as input and sample paraffin H&E sub-images with yin-yang labels as labels.

[0047] The second generative adversarial network is trained to obtain a trained second generative adversarial network; the IHC image generator in the trained second generative adversarial network is the trained IHC image generator.

[0048] Optionally, training the second generative adversarial network specifically includes:

[0049] Obtain a second sample set; the second sample set includes several sample paraffin H&E sub-images and a sample IHC sub-image corresponding to each sample paraffin H&E sub-image.

[0050] When training the second generative adversarial network, the loss functions used include the first Yin-Yang loss function, the second Yin-Yang loss function, the third Yin-Yang loss function, and the fourth Yin-Yang loss function;

[0051] The loss value of the first yin-yang loss function is calculated based on the sample IHC sub-image and the first predicted classification output; the first predicted classification output is the predicted classification output obtained by inputting the sample virtual IHC sub-image into the first yin-yang classification module; the sample virtual IHC sub-image is obtained by inputting the sample paraffin H&E sub-image into the IHC image generator;

[0052] The loss value of the second yin-yang loss function is calculated based on the sample paraffin H&E sub-image and the first predicted classification output;

[0053] The loss value of the third yin-yang loss function is calculated based on the sample IHC sub-image and the second predicted classification output; the second predicted classification output is the predicted classification output obtained by inputting the sample virtual H&E sub-image into the second yin-yang classification module; the sample virtual H&E sub-image is obtained by inputting the sample IHC sub-image into the H&E image generator;

[0054] The loss value of the fourth yin-yang loss function is calculated based on the sample paraffin H&E sub-image and the second predicted classification output.

[0055] Optionally, obtaining the second sample set specifically includes:

[0056] Obtain tissue samples from the affected area of ​​the patient; each tissue sample comprises i layers;

[0057] IHC staining is performed on the slice of the i-th sample layer to obtain the sample IHC sub-image;

[0058] H&E staining was performed on the i-th layer sample slice to obtain the H&E sub-image of the adjacent paraffin layer;

[0059] A registration network is used to register the negative and positive labels of the sample IHC sub-image and the adjacent paraffin H&E sub-image to obtain the sample paraffin H&E sub-image.

[0060] Optionally, the first yin-yang classification module and the second yin-yang classification module are ResNet-18 networks.

[0061] This invention also provides an IHC image generation system based on generative adversarial networks, comprising:

[0062] The frozen H&E image acquisition module is used to acquire frozen H&E images of slices from the target area of ​​the inspector;

[0063] The segmentation and annotation module is used to segment the frozen H&E image to obtain several frozen H&E sub-images, and to annotate the location information of the frozen H&E sub-images;

[0064] The paraffin H&E sub-image generation module is used to input several frozen H&E sub-images into a trained paraffin H&E image generator to obtain a paraffin H&E sub-image corresponding to each frozen H&E sub-image; the trained paraffin H&E image generator is a generator trained with sample frozen H&E sub-images as input and sample paraffin H&E sub-images as labels.

[0065] The IHC sub-image generation module is used to input each of the paraffin H&E sub-images into the trained IHC image generator to obtain the IHC sub-image corresponding to each of the paraffin H&E sub-images; the trained IHC image generator is a generator trained with sample paraffin H&E sub-images as input and sample IHC sub-images as labels.

[0066] The IHC image generation module is used to stitch together all the IHC sub-images according to the labeled location information to obtain an IHC image.

[0067] According to specific embodiments provided by the present invention, the following technical effects are disclosed: The present invention provides an IHC image generation method and system based on generative adversarial networks, including acquiring frozen H&E images of slices of the target body part of a detector; segmenting the frozen H&E images to obtain several frozen H&E sub-images, and labeling the position information of the frozen H&E sub-images; inputting the several frozen H&E sub-images into a trained paraffin H&E image generator to obtain a paraffin H&E sub-image corresponding to each frozen H&E sub-image; the trained paraffin H&E image generator is a generator trained with sample frozen H&E sub-images as input and sample paraffin H&E sub-images as labels; inputting each paraffin H&E sub-image into a trained IHC image generator to obtain an IHC sub-image corresponding to each paraffin H&E sub-image; the trained IHC image generator is a generator trained with sample paraffin H&E sub-images as input and sample IHC sub-images as labels; and stitching all IHC sub-images together according to the labeled position information to obtain an IHC image. This invention trains a generative adversarial network to obtain a paraffin H&E image generator and an IHC image generator. The paraffin H&E image generator is used to generate paraffin H&E images from frozen H&E images, and then the IHC image generator is used to generate IHC images from paraffin H&E images, thus reducing the time required to generate IHC images. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a schematic diagram of the IHC image generation method based on generative adversarial networks provided in Embodiment 1 of the present invention;

[0070] Figure 2 This is a flowchart illustrating the specific implementation of the IHC image generation method provided in Embodiment 1 of the present invention.

[0071] Figure 3 A schematic diagram of the training framework for the first generative adversarial network provided in Embodiment 1 of the present invention;

[0072] Figure 4 This is a schematic diagram of the nuclear constraint framework provided in Embodiment 1 of the present invention;

[0073] Figure 5 The registration and patch tags provided in Embodiment 1 of the present invention are given a frame;

[0074] Figure 6 This is a schematic diagram of the training framework for the second generative adversarial network provided in Embodiment 1 of the present invention;

[0075] Figure 7 This is a block diagram of the IHC image generation system based on generative adversarial networks provided in Embodiment 2 of the present invention. Detailed Implementation

[0076] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0077] The purpose of this invention is to provide an IHC image generation method and system based on generative adversarial networks, which can reduce the time required to generate IHC images.

[0078] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0079] Example 1

[0080] This embodiment provides an IHC image generation method based on generative adversarial networks. See [link to relevant documentation]. Figure 1 The method includes:

[0081] Step 1: Obtain frozen H&E images of slices from the target area of ​​the subject.

[0082] Step 2: Segment the frozen H&E image to obtain several frozen H&E sub-images, and label the location information of the frozen H&E sub-images.

[0083] Step 3: Input several frozen H&E sub-images into the trained paraffin H&E image generator to obtain the paraffin H&E sub-image corresponding to each frozen H&E sub-image; the trained paraffin H&E image generator is a generator trained with sample frozen H&E sub-images as input and sample paraffin H&E sub-images as labels.

[0084] Step 4: Input each of the paraffin H&E sub-images into the trained IHC image generator to obtain the IHC sub-image corresponding to each of the paraffin H&E sub-images; the trained IHC image generator is a generator trained with sample paraffin H&E sub-images as input and sample IHC sub-images as labels.

[0085] Step 5: Stitch together all the IHC sub-images according to the marked position information to obtain the IHC image.

[0086] In this embodiment, as Figure 2 As shown, firstly, rapid frozen H&E sections were prepared from the patient's lesion tissue and scanned using a microscope to obtain frozen H&E images. Next, the scanned frozen H&E images were segmented into frozen H&E epatch (i.e., frozen H&E sub-images), and these frozen H&E epatch were input into G... FS2FFPE (G FS2FFPE Generate paraffin H&E sub-images for the paraffin H&E image generator, and input the resulting paraffin H&E sub-images into G. H&E2IHC (G H&E2IHC The IHC image generator sequentially converts the images into IHC sub-images. Finally, the resulting IHC patches (IHC sub-images) are stitched together into a slide according to the coordinate information to obtain the IHC image. The entire process takes approximately 15 minutes, which significantly reduces the generation time compared to directly creating the IHC image.

[0087] This embodiment consists of two parts. The first part involves converting frozen H&E sections into paraffin-embedded H&E sections to remove interfering factors. The second part aims to predict IHC sections that can characterize gene expression based on the tissue information provided by the H&E images. Using the method in this embodiment, pathologists can virtually generate IHC sections intraoperatively based on frozen H&E sections to understand the patient's gene expression and thus guide the surgical direction. The specific processes of the two parts are as follows:

[0088] Part 1: Unsupervised Virtual Paraffin H&E Subimage Prediction Based on Sample Frozen H&E Subimages

[0089] The main function of this section is to remove factors that interfere with the prediction of tissue positivity and negativity, such as ice crystals and wrinkles, from the obtained frozen H&E images. Frozen H&E images are usually fixed using a freezing method, which is faster (15 minutes), but often introduces many interfering factors, such as ice crystals, over-staining, and wrinkles. Conventional H&E sections are usually fixed using paraffin embedding, which is slower (36 hours), but generally does not introduce interfering factors.

[0090] Therefore, in this embodiment, H&E sub-images of frozen samples and paraffin H&E sub-images of conventionally paraffin-embedded samples are collected respectively. The first generative adversarial network is used to obtain the mapping relationship between the sample H&E sub-images and the sample paraffin H&E sub-images, and then the sample H&E sub-images are transformed into sample paraffin H&E sub-images, thereby achieving the purpose of removing interference factors.

[0091] In this embodiment, before using the first generative adversarial network to obtain the mapping relationship between the sample H&E sub-image and the sample paraffin H&E sub-image, it is necessary to train the first generative adversarial network. The training process can be as follows:

[0092] First, the frozen H&E and paraffin-embedded H&E pathological sections are scanned under a microscope at 20x magnification. Next, the scanned images are segmented into numerous patches, resulting in several sample frozen H&E sub-images and a corresponding sample paraffin-embedded H&E sub-image for each sample frozen H&E sub-image. In this embodiment, the size of the sample frozen H&E sub-image can be 256×256.

[0093] After obtaining the above images, construct as follows: Figure 3 The first generative adversarial network shown includes a paraffin H&E image generator, a high-resolution decision maker, and a low-resolution decision maker.

[0094] Then, the first generative adversarial network is trained to obtain a trained first generative adversarial network; the paraffin H&E image generator of the trained first generative adversarial network is the trained paraffin H&E image generator.

[0095] In this embodiment, the loss functions used when training the first generative adversarial network include kernel constraint loss function, content consistency constraint loss function, and texture consistency constraint loss function. The process of establishing the three loss functions is as follows:

[0096] For a 256×256 frozen H&E sub-image, the H&E image generator generates a virtual paraffin H&E sub-image based on this frozen H&E sub-image. In tissue images, the nuclear structure is crucial; however, due to the similarity between nuclear structures and other tissue structures, existing methods often confuse nuclear structures with other tissue structures, resulting in missing or redundant nuclei in the generated image. Taking thyroid tissue as an example, since thyroid tissue images contain many thyroid follicle structures, which are similar to nuclear structures, the H&E image generator may mismatch thyroid follicle structures with nuclear structures, leading to missing or redundant nuclei in the virtual paraffin H&E sub-image. Therefore, this embodiment designs a kernel constraint to make the generation of nuclei in the virtual paraffin H&E sub-image more accurate, such as... Figure 4 As shown, the kernel constraint process is as follows:

[0097] In the kernel constraint framework, for both the frozen H&E sub-image and the virtually generated virtual paraffin H&E sub-image, firstly, H staining (hematoxylin staining, used to stain cell nuclei) and E staining (eosin staining, used to stain cytoplasm) are separated and the H staining result is extracted based on color deconvolution, and then the cell nuclei are segmented according to a set kernel threshold. Next, by constraining the kernel segmentation result of the virtual paraffin H&E sub-image to be consistent with that of the frozen H&E sub-image, the generation of cell nuclei in the virtual paraffin H&E sub-image is made more accurate. The expression for the kernel constraint loss function is:

[0098] L H =||f(x)-f(x')||1

[0099] Among them, L H Let f(x) represent the kernel constraint loss function value, x represent the frozen H&E sub-image of the sample, x' represent the virtual paraffin H&E sub-image of the sample, f() is the cell kernel segmentation algorithm, f(x) is the cell kernel of the frozen H&E sub-image of the sample, and f(x′) is the cell kernel of the virtual paraffin H&E sub-image of the sample.

[0100] In this embodiment, before calculating the kernel constraint loss function value using the kernel constraint loss function, the following steps are also included:

[0101] The H-staining images of the frozen H&E sub-image and the virtual paraffin H&E sub-image of the sample are extracted using the color deconvolution method, resulting in the H-staining images of the frozen H&E sub-image and the virtual paraffin H&E sub-image of the sample.

[0102] The H-stained image of the frozen H&E sub-image of the sample is segmented and extracted according to the set nuclear threshold to obtain the cell nucleus of the frozen H&E sub-image of the sample;

[0103] The cell nuclei in the H-stained image of the virtual paraffin H&E sub-image of the sample are segmented and extracted according to the set kernel threshold to obtain the cell nuclei of the virtual paraffin H&E sub-image of the sample. The expression of the cell nucleus segmentation algorithm is as follows:

[0104]

[0105] Among them, H x The image is the H-stained image obtained after separating the frozen H&E sub-image x through the H channel, and t is the set kernel threshold.

[0106] In this embodiment, the process of acquiring sample augmentation frozen H&E sub-images also includes:

[0107] The frozen H&E sub-images of the samples obtained above are labeled;

[0108] Based on the marked location information, the frozen H&E sub-image of the sample is stitched together with some frozen H&E sub-images of other samples within a preset range outside the frozen H&E sub-image of the sample to obtain the sample expanded frozen H&E sub-image.

[0109] The process of obtaining sample augmentation frozen H&E sub-images is described in detail below:

[0110] Because frozen H&E sub-images of samples contain interfering factors such as ice crystals and wrinkles, generating virtual paraffin H&E sub-images requires inferring the texture at the location of these interfering factors based on the morphological structure of the tissue surrounding them. However, for the edges of a patch, there is limited tissue structure information available. Therefore, the edges of patches are often not accurate enough during the generation of virtual paraffin H&E sub-images. To address this issue, this embodiment designs a multi-resolution generator paradigm. For a patch, based on its coordinate position information, the surrounding tissue regions are located in the slide to better generate the texture of the patch's edges. Figure 3 The query retrieves the surrounding tissue region of the frozen H&E sub-image x256 and then retrieves the frozen H&E sub-image x 256 The frozen H&E sub-images of some other samples within a preset external range are added together to obtain an image of size N×N. Meanwhile, since some unimportant features (such as blurring due to defocusing during sampling) can be suppressed at lower resolutions, this embodiment scales the N×N image to 256×256 to highlight some important texture features. This 256×256 image serves as the sample augmented frozen H&E image. Subsequently, the augmented frozen H&E image is input into the generator to obtain the corresponding texture feature map, and a content consistency constraint is established between the augmented frozen H&E image and the texture feature map to ensure the accuracy of the generated image. Its expression is as follows:

[0111]

[0112] Among them, L NCE (E,T) represents the content consistency constraint loss function value between the sample augmented frozen H&E sub-image E and the texture feature map T, and g(T,m) represents a vector in the m-th layer feature block of the texture feature map T. + Then g(E,m) represents the vector at the position corresponding to g(T,m) in the m-th layer feature block of the sample augmented frozen H&E sub-image. n -represents the vector in the m-th layer feature block of the augmented frozen H&E sub-image that does not correspond to g(T,m); τ represents the temperature coefficient used for balancing the distribution; N represents the number of times the vector at the non-corresponding position is sampled; S L M represents the number of vectors randomly selected from the feature blocks of the texture feature map; M represents the total number of feature layers.

[0113] In this embodiment, after obtaining the texture of the sample virtual paraffin H&E sub-image by augmenting the frozen H&E sub-image, a texture consistency constraint needs to be applied to the sample virtual paraffin H&E sub-image. First, the region corresponding to the real frozen H&E image (size n×n) is cropped from the central region of the obtained texture feature map, and then the sample virtual paraffin H&E sub-image is scaled to n×n proportionally. Finally, a texture consistency constraint is constructed between the two to make the texture information of the sample virtual paraffin H&E sub-image more accurate. Its expression is as follows:

[0114] L Texture =||x' sf -T cj ||1

[0115] Among them, L Texture x' represents the value of the texture consistency constraint loss function. sf T represents the image after the sample virtual paraffin H&E sub-image has been scaled up by a preset ratio. cj This represents an image after the texture feature map has been cropped to a preset cropping range.

[0116] In this embodiment, the training process of the first generative adversarial network includes the following steps:

[0117] Obtain a first sample set; the first sample set includes several sample frozen H&E sub-images and sample paraffin H&E sub-images, sample expanded frozen H&E sub-images, and sample expanded paraffin H&E sub-images corresponding to each sample frozen H&E sub-image.

[0118] The frozen H&E sub-image of the sample is input into the paraffin H&E image generator to generate a virtual paraffin H&E sub-image of the sample; based on the frozen H&E sub-image of the sample and the virtual paraffin H&E sub-image of the sample, the kernel constraint loss function value is calculated using the kernel constraint loss function.

[0119] The sample-expanded frozen H&E sub-image is input into the paraffin H&E image generator to generate a texture feature map; based on the sample-expanded frozen H&E sub-image and the texture feature map, the content consistency constraint loss function value is calculated using the content consistency constraint loss function.

[0120] Based on the sample virtual paraffin H&E sub-image and the texture feature map, the texture consistency constraint loss function value is calculated using the texture consistency constraint loss function.

[0121] The high-resolution decision-maker is used to distinguish between real and fake samples of virtual paraffin H&E sub-images and samples of paraffin H&E sub-images, and to obtain the high-resolution score loss value of the sample virtual paraffin H&E sub-image and the loss function value of the first paraffin H&E decision-maker.

[0122] The low-resolution decision maker is used to distinguish between real and fake texture feature maps and sample-enhanced paraffin H&E sub-images, resulting in the low-resolution score loss value of the texture feature map and the loss function value of the second paraffin H&E decision maker.

[0123] The total loss function value of the paraffin H&E generator is obtained by weighted summing of the kernel constraint loss function value, the content consistency constraint loss function value, the texture consistency constraint loss function value, the high resolution score loss value, and the low resolution score loss value.

[0124] The paraffin H&E image generator is updated based on the total loss function value of the paraffin H&E generator to obtain the updated paraffin H&E image generator.

[0125] The total loss function value of the decision maker is calculated based on the loss function value of the first paraffin H&E decision maker and the loss function value of the second paraffin H&E decision maker.

[0126] The high-resolution decision and the low-resolution decision are updated based on the total loss function value of the decision, resulting in updated high-resolution decision and updated low-resolution decision.

[0127] Determine whether the preset iteration condition has been met. If yes, stop the iteration and use the updated paraffin H&E image generator, the updated high-resolution decision maker, and the updated low-resolution decision maker as the paraffin H&E image generator, high-resolution decision maker, and low-resolution decision maker in the trained first generative adversarial network, respectively. If no, return to the step "input the sample frozen H&E sub-image into the paraffin H&E image generator" until the preset iteration condition is met.

[0128] The relevant loss function expression for the above training process is as follows:

[0129]

[0130]

[0131]

[0132]

[0133]

[0134]

[0135] In the formula, x represents the sample paraffin H&E sub-image, E represents the sample expanded frozen H&E sub-image, x' is the sample virtual paraffin H&E sub-image, and T is the texture feature map. Where x' = G FS2FFPE (x), T=G FS2FFPE (E), This represents the high-resolution score loss value. This represents the low-resolution score loss value. This is the total score loss value. The first paraffin H&E decision loss function value. The loss function value of the second paraffin H&E decision maker. The value is the loss function value of the paraffin H&E decision maker.

[0136] For the H&E image generator, the total loss function is obtained by weighted summation of the above loss functions, and its expression is as follows:

[0137]

[0138] For the discriminator, the total loss function is as follows:

[0139]

[0140] In the formula, L G L represents the total loss function value of the paraffin H&E generator. D λ1, λ2, λ3, and λ4 are the total loss function values ​​of the decision maker, and λ1, λ2, λ3, and λ4 are the weighting coefficients of each loss function. In this embodiment, λ1 = 10, λ2 = λ3 = λ4 = 1.

[0141] Part 2: Semi-supervised Virtual IHC Image Prediction Based on Paraffin H&E Images

[0142] The main function of this section is to predict IHC sections based on the tissue structure information provided by paraffin-embedded H&E sections. Post-operatively, pathologists often use adjacent tissue sections for H&E staining and different types of IHC staining (e.g., thyroid tissue is usually stained with TG, TTF-1, and Ki-67). These adjacent tissue sections are roughly similar in morphology and structure. After registration using a registration network, the positive and negative results for the same tissue location are completely consistent. The registration process specifically includes the following steps:

[0143] Obtain tissue samples from the affected area of ​​the patient; each tissue sample comprises i layers;

[0144] IHC staining is performed on the slice of the i-th sample layer to obtain the sample IHC sub-image;

[0145] H&E staining was performed on the i-th layer sample slice to obtain the H&E sub-image of the adjacent paraffin layer;

[0146] A registration network is used to register the negative and positive labels of the sample IHC sub-image and the adjacent paraffin H&E sub-image to obtain the sample paraffin H&E sub-image.

[0147] like Figure 5 As shown, in IHC images, due to the significant color difference between positive (brown) and negative (blue) tissues, the positive / negative information of IHC slices can be easily obtained, and thus the positive / negative information of adjacent H&E slices can be obtained. In this embodiment, the second generative adversarial network (GAN) can accurately generate IHC images by constraining the positive / negative nature of the H&E image and the generated IHC image to be consistent. First, a registration network is used to register adjacent H&E slides (layer i-1) with IHC slides (layer i). Based on the proportion of positive areas (brown areas), a threshold method is used to assign positive / negative labels to IHC patches. Furthermore, since the registered IHC patch and H&E patch at the same location have approximately consistent pathological information, the positive / negative label of the IHC patch is assigned to the H&E patch, thereby obtaining several paraffin H&E sub-images of the samples and the corresponding sample IHC sub-image for each paraffin H&E sub-image.

[0148] In this embodiment, before inputting each of the paraffin H&E sub-images into the trained IHC image generator in step 4, the method further includes:

[0149] A second generative adversarial network (GAN) is constructed. This GAN includes an IHC image generator, an IHC decision maker, an H&E image generator, and an H&E decision maker. The second GAN is an improved CycleGAN model, with a first yin-yang classification module added to the IHC decision maker and a second yin-yang classification module added to the H&E decision maker. The first yin-yang classification module is a model trained using sample IHC sub-images as input and sample IHC sub-images with yin-yang labels as labels. The yin-yang labels are the yin-yang information obtained after IHC staining of sample patient tissue sections. The second yin-yang classification module is a model trained using sample paraffin H&E sub-images as input and sample paraffin H&E sub-images with yin-yang labels as labels. In this embodiment, the yin-yang classification module is composed of ResNet-18, initialized with weights pre-trained on ImageNet.

[0150] The second generative adversarial network is trained to obtain a trained second generative adversarial network; the IHC image generator in the trained second generative adversarial network is the trained IHC image generator.

[0151] In this embodiment, the above-described IHC image generation method based on generative adversarial networks may further include a training process for a second generative adversarial network, the training process specifically including the following steps:

[0152] Obtain a second sample set; the second sample set includes several sample paraffin H&E sub-images and a sample IHC sub-image corresponding to each sample paraffin H&E sub-image.

[0153] When training the second generative adversarial network, the loss functions used include the first Yin-Yang loss function, the second Yin-Yang loss function, the third Yin-Yang loss function, and the fourth Yin-Yang loss function;

[0154] The loss value of the first yin-yang loss function is calculated based on the sample IHC sub-image and the first predicted classification output; the first predicted classification output is the predicted classification output obtained by inputting the sample virtual IHC sub-image into the first yin-yang classification module; the sample virtual IHC sub-image is obtained by inputting the sample paraffin H&E sub-image into the IHC image generator;

[0155] The loss value of the second yin-yang loss function is calculated based on the sample paraffin H&E sub-image and the first predicted classification output;

[0156] The loss value of the third yin-yang loss function is calculated based on the sample IHC sub-image and the second predicted classification output; the second predicted classification output is the predicted classification output obtained by inputting the sample virtual H&E sub-image into the second yin-yang classification module; the sample virtual H&E sub-image is obtained by inputting the sample IHC sub-image into the H&E image generator;

[0157] The loss value of the fourth yin-yang loss function is calculated based on the sample paraffin H&E sub-image and the second predicted classification output.

[0158] like Figure 6 As shown in Figure (a), the process of inputting the sample paraffin H&E sub-image into the IHC image generator to obtain the sample virtual IHC sub-image, and then inputting the sample virtual IHC sub-image into the H&E image generator to obtain the reconstructed paraffin H&E sub-image. Figure (b) shows the process of inputting the sample IHC sub-image into the H&E image generator to obtain the sample virtual paraffin H&E sub-image, and then inputting the sample virtual paraffin H&E sub-image into the IHC image generator to generate the reconstructed IHC image.

[0159] The training process is illustrated below using the H&E-IHC-H&E process shown in Figure (a) as an example:

[0160] Based on CycleGAN, the IHC decision maker D IHC Add a first positive / negative classification module to distinguish the positive / negative nature of IHC images, thus realizing the function of identifying the positive / negative nature of virtual IHC sub-images of samples. Train D IHC At that time, for the IHC decision device D IHC The first positive / negative classification module in the model is input with sample IHC sub-images and its classification results are constrained to match the positive / negative labels of the sample IHC sub-images. IHC To maintain consistency, as shown in the following formula:

[0161]

[0162] In the formula, It is D IHC The first predicted classification output of the first Yin-Yang classification module in the middle, l IHC These are the positive and negative labels of the sample IHC sub-images.

[0163] When training the generator, for the IHC image generator G H&E2IHC Input a sample paraffin H&E sub-image (the sample paraffin H&E sub-image has positive and negative labels) to it. H&E This yields a sample virtual IHC sub-image. The generator G is expected to... H&E2IHC The positive / negative sign of the generated sample virtual IHC sub-image is consistent with that of the sample paraffin H&E sub-image. Therefore, in this embodiment, the sample virtual IHC sub-image is input into D. IHC The first yin-yang classification module in the model aims to output a classification result, i.e., the first predicted classification output, that is, the label of the sample paraffin H&E sub-image, as shown in the following formula:

[0164]

[0165] In the formula, It is the first predicted classification output, l H&E These are the positive and negative labels for the sample paraffin H&E sub-image.

[0166] exist Figure 6 (a) During the H&E-IHC-H&E process, the generator's total loss function is as follows:

[0167]

[0168] For the discriminator, the total loss function is as follows:

[0169]

[0170] Among them, L G1 Let L be the total loss function of the H&E-IHC-H&E process generator. D1 Let x be the total loss function of the H&E-IHC-H&E process discriminator. H&E Represents the sample paraffin H&E sub-image, x IHC G represents the sample IHC sub-image. H&E2IHC For IHC image generator, G IHC2H&E For paraffin H&E image generator, The score loss value is input into the paraffin H&E sub-image of the sample and generated by the paraffin H&E image generator. For IHC decision device D IHC Loss function value, L cyc1 =||G IHC2H&E (G H&E2IHC (x H&E ))-x H&E ||1 represents the loss function value calculated based on the sample paraffin H&E sub-image and the reconstructed paraffin H&E sub-image, L idt1 =||G IHC2H&E (x H&E )-x H&E 1. In this embodiment, λ5=λ9=10, λ6=λ7=λ8=λ 10 =1.

[0171] exist Figure 6 (b) During the IHC-H&E-IHC process, the generator's total loss function is as follows:

[0172]

[0173] For the discriminator, the total loss function is as follows:

[0174]

[0175] Among them, L G2 Let L be the total loss function of the generator during the IHC-H&E-IHC process. D2 The total loss function of the discriminator during the IHC-H&E-IHC process. The score loss value generated by inputting the IHC sub-image of the IHC-H&E-IHC process sample into the paraffin H&E image generator. L represents the DH&E loss function value of the IHC decision unit during the IHC-H&E-IHC process. cyc2 =||G H&E2IHC (G IHC2H&E (x IHC ))-x IHC||1 represents the loss function value calculated during the IHC-H&E-IHC process based on the sample IHC sub-images and the reconstructed IHC sub-images, L idt2 =||G H&E2IHC (x IHC )||1, It is the second predicted classification output, l H&E These are the positive and negative labels for the sample paraffin H&E sub-images. IHC These are the positive and negative labels of the sample IHC sub-images.

[0176] This invention can predict IHC images based on frozen H&E images. In clinical practice, immunohistochemical staining typically takes three to five days, which severely hinders the application of immunohistochemical techniques in intraoperative guidance. This invention trains a generative adversarial network to obtain a paraffin H&E image generator and an IHC image generator. The paraffin H&E image generator generates paraffin H&E images from frozen H&E images, and then the IHC image generator generates IHC images from the paraffin H&E images. This method can quickly predict IHC images (approximately 15 minutes), reducing the time required to generate IHC images.

[0177] Example 2

[0178] This embodiment provides an IHC image generation method system based on generative adversarial networks. See [link to documentation]. Figure 7 The system includes:

[0179] The frozen H&E image acquisition module T1 is used to acquire frozen H&E images of slices from the target area of ​​the inspector.

[0180] The segmentation and annotation module T2 is used to segment the frozen H&E image to obtain several frozen H&E sub-images, and to annotate the position information of the frozen H&E sub-images.

[0181] The paraffin H&E sub-image generation module T3 is used to input several frozen H&E sub-images into a trained paraffin H&E image generator to obtain a paraffin H&E sub-image corresponding to each frozen H&E sub-image; the trained paraffin H&E image generator is a generator trained with sample frozen H&E sub-images as input and sample paraffin H&E sub-images as labels.

[0182] The IHC sub-image generation module T4 is used to input each of the paraffin H&E sub-images into the trained IHC image generator to obtain the IHC sub-image corresponding to each of the paraffin H&E sub-images; the trained IHC image generator is a generator trained with sample paraffin H&E sub-images as input and sample IHC sub-images as labels.

[0183] The IHC image generation module T5 is used to stitch together all the IHC sub-images according to the marked position information to obtain an IHC image.

[0184] The system disclosed in the embodiments is described in a relatively simple manner because it corresponds to the method disclosed in the embodiments. For relevant details, please refer to the method section.

[0185] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for generating IHC images based on generative adversarial networks, characterized in that, include: Acquire frozen H&E images of tissue sections from the target area of ​​the subject; The frozen H&E image is segmented to obtain several frozen H&E sub-images, and the location information of the frozen H&E sub-images is labeled. Several frozen H&E sub-images are input into a trained paraffin H&E image generator to obtain a paraffin H&E sub-image corresponding to each frozen H&E sub-image; the trained paraffin H&E image generator is a generator trained using sample frozen H&E sub-images as input and sample paraffin H&E sub-images as labels; before inputting several frozen H&E sub-images into the trained paraffin H&E image generator, the process further includes: Construct a first generative adversarial network; the first generative adversarial network includes a paraffin H&E image generator, a high-resolution decision maker, and a low-resolution decision maker; The first generative adversarial network is trained to obtain a trained first generative adversarial network; the paraffin H&E image generator of the trained first generative adversarial network is the trained paraffin H&E image generator. When training the first generative adversarial network, the loss functions used include kernel constraint loss function, content consistency constraint loss function and texture consistency constraint loss function; The expression for the kernel constraint loss function is: in, L H This represents the value of the kernel constraint loss function. x This represents the frozen H&E sub-image of the sample. This represents the sample virtual paraffin H&E sub-image. This is a cell nucleus segmentation algorithm. The cell nuclei are from frozen H&E sub-images of the sample. Cell nuclei for a sample virtual paraffin H&E sub-image; The expression for the content consistency constraint loss function is: in, This represents the content consistency constraint loss function value for augmenting the frozen H&E sub-image E and the texture feature map T. This represents a vector in the m-th layer feature block of the texture feature map T. This indicates that the m-th layer feature block of the augmented frozen H&E sub-image is related to... The vector at the corresponding position, This indicates that the m-th layer feature block of the augmented frozen H&E sub-image is related to... Vectors at non-corresponding positions; This represents the temperature coefficient, used for equilibrium distribution. This indicates the number of times the vector at non-corresponding positions is sampled; This represents the number of vectors randomly selected from the feature blocks of the texture feature map; Indicates the total number of feature layers; The expression for the texture consistency constraint loss function is: in, This represents the value of the texture consistency constraint loss function. This represents the image after the sample virtual paraffin H&E sub-image has been scaled up by a preset ratio. This represents an image after the texture feature map has been cropped to a preset cropping range. Each of the paraffin H&E sub-images is input into the trained IHC image generator to obtain the IHC sub-image corresponding to each of the paraffin H&E sub-images; the trained IHC image generator is a generator trained with sample paraffin H&E sub-images as input and sample IHC sub-images as labels. All the IHC sub-images are stitched together based on the labeled location information to obtain the IHC image.

2. The IHC image generation method based on generative adversarial networks according to claim 1, characterized in that, The training of the first generative adversarial network specifically includes: Obtain a first sample set; the first sample set includes several sample frozen H&E sub-images and sample paraffin H&E sub-images, sample expanded frozen H&E sub-images, and sample expanded paraffin H&E sub-images corresponding to each sample frozen H&E sub-image. The frozen H&E sub-image of the sample is input into the paraffin H&E image generator to generate a virtual paraffin H&E sub-image of the sample; based on the frozen H&E sub-image of the sample and the virtual paraffin H&E sub-image of the sample, the kernel constraint loss function value is calculated using the kernel constraint loss function; The sample-enlarged frozen H&E sub-image is input into the paraffin H&E image generator to generate a texture feature map; based on the sample-enlarged frozen H&E sub-image and the texture feature map, the content consistency constraint loss function value is calculated using the content consistency constraint loss function. Based on the sample virtual paraffin H&E sub-image and the texture feature map, the texture consistency constraint loss function value is calculated using the texture consistency constraint loss function. The high-resolution decision-maker is used to distinguish between real and fake samples of virtual paraffin H&E sub-images and samples of paraffin H&E sub-images, and to obtain the high-resolution score loss value of the sample virtual paraffin H&E sub-image and the loss function value of the first paraffin H&E decision-maker. The low-resolution decision maker is used to distinguish between real and fake texture feature maps and sample augmented paraffin H&E sub-images, and the low-resolution score loss value of the texture feature map and the loss function value of the second paraffin H&E decision maker are obtained. The total loss function value of the paraffin H&E generator is obtained by weighted summing of the kernel constraint loss function value, the content consistency constraint loss function value, the texture consistency constraint loss function value, the high resolution score loss value, and the low resolution score loss value. The paraffin H&E image generator is updated based on the total loss function value of the paraffin H&E generator to obtain the updated paraffin H&E image generator. The total loss function value of the decision device is calculated based on the loss function value of the first paraffin H&E decision device and the loss function value of the second paraffin H&E decision device. The high-resolution decision and the low-resolution decision are updated based on the total loss function value of the decision, resulting in the updated high-resolution decision and the updated low-resolution decision. Determine whether the preset iteration condition has been met. If yes, stop the iteration and use the updated paraffin H&E image generator, the updated high-resolution decision maker, and the updated low-resolution decision maker as the paraffin H&E image generator, high-resolution decision maker, and low-resolution decision maker in the trained first generative adversarial network, respectively. If no, return to the step "input the sample frozen H&E sub-image into the paraffin H&E image generator" until the preset iteration condition is met.

3. The IHC image generation method based on generative adversarial networks according to claim 2, characterized in that, Before calculating the kernel constraint loss function value using the kernel constraint loss function based on the sample frozen H&E sub-image and the sample virtual paraffin H&E sub-image, the method further includes: The H-staining images of the frozen H&E sub-image and the virtual paraffin H&E sub-image of the sample are extracted using the color deconvolution method, resulting in the H-staining images of the frozen H&E sub-image and the virtual paraffin H&E sub-image of the sample. The H-stained image of the frozen H&E sub-image of the sample is segmented and extracted according to the set nuclear threshold to obtain the cell nucleus of the frozen H&E sub-image of the sample; The cell nuclei in the H-stained image of the virtual paraffin H&E sub-image of the sample are segmented and extracted according to the set nuclear threshold to obtain the cell nuclei of the virtual paraffin H&E sub-image of the sample.

4. The IHC image generation method based on generative adversarial networks according to claim 2, characterized in that, Before obtaining the first sample set, the method further includes: Obtain frozen H&E sub-images of the samples; The frozen H&E sub-images of the samples were annotated; Based on the marked location information, the frozen H&E sub-image of the sample is stitched together with some frozen H&E sub-images of other samples within a preset range outside the frozen H&E sub-image of the sample to obtain the sample expanded frozen H&E sub-image.

5. The IHC image generation method based on generative adversarial networks according to claim 1, characterized in that, Before inputting each of the paraffin H&E sub-images into the trained IHC image generator, the method further includes: A second generative adversarial network (GAN) is constructed. This GAN includes an IHC image generator, an IHC decision maker, an H&E image generator, and an H&E decision maker. The GAN is an improved CycleGAN model, with a first yin-yang classification module added to the IHC decision maker and a second yin-yang classification module added to the H&E decision maker. The first yin-yang classification module is a model trained using sample IHC sub-images as input and sample IHC sub-images with yin-yang labels as labels. The yin-yang labels are the yin-yang information obtained after IHC staining of sample patient tissue sections. The second yin-yang classification module is a model trained using sample paraffin H&E sub-images as input and sample paraffin H&E sub-images with yin-yang labels as labels. The second generative adversarial network is trained to obtain a trained second generative adversarial network; the IHC image generator in the trained second generative adversarial network is the trained IHC image generator.

6. The IHC image generation method based on generative adversarial networks according to claim 5, characterized in that, The training of the second generative adversarial network specifically includes: Obtain a second sample set; the second sample set includes several sample paraffin H&E sub-images and a sample IHC sub-image corresponding to each sample paraffin H&E sub-image. When training the second generative adversarial network, the loss functions used include the first Yin-Yang loss function, the second Yin-Yang loss function, the third Yin-Yang loss function, and the fourth Yin-Yang loss function; The loss value of the first yin-yang loss function is calculated based on the sample IHC sub-image and the first predicted classification output; the first predicted classification output is the predicted classification output obtained by inputting the sample virtual IHC sub-image into the first yin-yang classification module; the sample virtual IHC sub-image is obtained by inputting the sample paraffin H&E sub-image into the IHC image generator; The loss value of the second yin-yang loss function is calculated based on the sample paraffin H&E sub-image and the first predicted classification output; The loss value of the third yin-yang loss function is calculated based on the sample IHC sub-image and the second predicted classification output; the second predicted classification output is the predicted classification output obtained by inputting the sample virtual H&E sub-image into the second yin-yang classification module; the sample virtual H&E sub-image is obtained by inputting the sample IHC sub-image into the H&E image generator; The loss value of the fourth yin-yang loss function is calculated based on the sample paraffin H&E sub-image and the second predicted classification output.

7. The IHC image generation method based on generative adversarial networks according to claim 5, characterized in that, The acquisition of the second sample set specifically includes: Obtain tissue samples from the affected area of ​​the patient; the tissue samples include i layer; For the first i IHC staining was performed on sample slices to obtain IHC sub-images of the samples; For the first i H&E staining was performed on slices of the layer sample to obtain H&E sub-images of adjacent paraffin layers; A registration network is used to register the negative and positive labels of the sample IHC sub-image and the adjacent paraffin H&E sub-image to obtain the sample paraffin H&E sub-image.

8. The IHC image generation method based on generative adversarial networks according to claim 5, characterized in that, The first yin-yang classification module and the second yin-yang classification module are ResNet-18 networks.

9. A system for generating IHC images based on generative adversarial networks, based on the IHC image generation method of claim 1, characterized in that, include: The frozen H&E image acquisition module is used to acquire frozen H&E images of slices from the target area of ​​the inspector; The segmentation and annotation module is used to segment the frozen H&E image to obtain several frozen H&E sub-images, and to annotate the location information of the frozen H&E sub-images; The paraffin H&E sub-image generation module is used to input several frozen H&E sub-images into a trained paraffin H&E image generator to obtain a paraffin H&E sub-image corresponding to each frozen H&E sub-image; the trained paraffin H&E image generator is a generator trained with sample frozen H&E sub-images as input and sample paraffin H&E sub-images as labels. The IHC sub-image generation module is used to input each of the paraffin H&E sub-images into the trained IHC image generator to obtain the IHC sub-image corresponding to each of the paraffin H&E sub-images; the trained IHC image generator is a generator trained with sample paraffin H&E sub-images as input and sample IHC sub-images as labels. The IHC image generation module is used to stitch together all the IHC sub-images according to the labeled location information to obtain an IHC image.

Images