A staining generation method from HE image to IHC image based on image patch similarity

Abstract

The present invention relates to a method for generating staining from a HE image to an IHC image based on image block similarity, comprising: obtaining a corresponding HE image and a real IHC image; inputting the HE image into an initial image generator to obtain a first generated IHC image; obtaining a total loss function; training the initial image generator; inputting the HE image into the trained initial image generator to obtain a second generated IHC image. The beneficial effects of the present invention are as follows: the present invention proposes a computational semantic supervision PatchNCE loss function, which integrates other image blocks as positive examples to reduce the negative impact of incompletely calibrated HE-IHC images on the model, and the present invention proposes a semantic relationship consistency loss function, which regularizes the uncalibrated regions between image pairs, so that the generated image has consistent overall semantic information with the real target image, rather than just consistent with the input image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically, to a method for generating staining from a HE image to an IHC image based on image block similarity. Background Art

[0002] Immunohistochemistry (IHC) staining is a technique that uses the specific binding between antibodies and antigens to detect the expression and location of target proteins in tissues. It is widely used in basic research and clinical pathology testing. Hematoxylin-Eosin staining is the staining of cell nuclei and cytoplasm with hematoxylin and eosin. It is mainly used to show the structural morphology of cells. In order to assist doctors in making accurate diagnoses, many diseases require the use of these two staining methods to stain cell sections at the same time. However, due to the material limitations and operational complexity of IHC staining, this technology is time-consuming and expensive. Experienced doctors can make a rough assessment of the IHC staining expression of the same section by observing the digitized whole-slice image (WSI) based on the results of HE staining. Under this premise, researchers tried to study whether IHC staining images can be generated directly from HE staining images at a lower cost. The most challenging part for the HE-IHC image translation task is the lack of well-calibrated HE-IHC image pairs. Because the same section cannot be stained with two staining reagents, each HE-IHC staining image pair comes from two consecutive sections of the same tissue. The two slices are similar in overall structure, but there are still inconsistencies in the details. As shown in Figure 1, the different slices and different staining operations make pixel-level registration of the image pair almost impossible.

[0003] Some researchers have tried some methods to accomplish this task, most of which are based on the GAN model. This model can convert data from the source domain to the target domain. In addition, in order to ensure that the generated image is highly consistent with the original image in content, cycle consistency loss and L1 regularization loss terms are added to the GAN loss function. These loss terms help to preserve the structural integrity of the image and the consistency of relevant features during the conversion process. However, some existing studies have shown that if the image pairs used for model training are not well aligned and calibrated, these overly restrictive constraints will cause the generated images to be severely distorted.

[0004] To address this challenge, researchers have tried to explore other model methods to mitigate the adverse effects of such strong constraints on the model. For example, they have tried to introduce a style transfer module into their model and add some auxiliary heads to maintain the consistency of content style between the generated image and the input image. Specifically, a classification head can be added to control the model to generate a variety of different IHC staining images. However, from the perspective of individual cells in the image, it is obvious that most IHC images generated by the style transfer model cannot accurately reflect the staining details of specific cells in the HE staining image. In addition, some methods based on the cycle-GAN model inject expert annotation information during training to improve the performance of the model. However, the collection and processing of this knowledge brings a lot of additional pre-work to the entire algorithm process. The prior art also proposes an adaptive weighting scheme based on the similarity between image blocks. However, this method still does not pay attention to the image blocks that are offset in position but consistent in meaning due to insufficient calibration. In addition, the prior art discloses the use of a diffusion model framework to generate corresponding images. However, researchers have found that the speed of training and reasoning using a diffusion model is often relatively slow. Moreover, the output of diffusion models often exhibits higher randomness, which is inconsistent with the goal of generating images with the same structure as the input image. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the prior art and to propose a method for generating staining from HE images to IHC images based on image block similarity.

[0006] In a first aspect, a method for generating staining from a HE image to an IHC image based on image block similarity is provided, comprising:

[0007] Step 1: acquiring an image pair, wherein the image pair includes a HE image and a real IHC image that correspond in structure;

[0008] Step 2: obtaining an initial image generator; inputting the HE image into the initial image generator to obtain a first generated IHC image;

[0009] Step 3: Calculate the semantic supervision PatchNCE loss function, the semantic relationship consistency loss function, the supervised PatchNCE loss function and the GAN loss function according to the image pair and the first generated IHC image;

[0010] Step 4: Obtain the total loss function based on the given weight parameters and the loss functions in step 3;

[0011] Step 5: training the initial image generator through the total loss function;

[0012] Step 6: Input the HE image into the trained initial image generator to obtain a second generated IHC image.

[0013] Preferably, in step 3, calculating the semantic supervision PatchNCE loss function includes:

[0014] Performing image block sampling on the image pair and the first generated IHC image;

[0015] Perform feature extraction based on the sampled image blocks;

[0016] Based on the extracted features, contrastive learning is performed and the semantic supervision PatchNCE loss function is calculated.

[0017] Preferably, in step 3, the performing image block sampling includes:

[0018] Each image block in the first generated IHC image is used as an anchor point, and the positive examples with the highest similarity to the anchor point are selected around the same position as the anchor point in the real IHC image. image blocks as auxiliary positive examples.

[0019] Preferably, in step 3, calculating the semantic relationship consistency loss function includes:

[0020] According to the HE image, obtaining a semantic relationship distribution of an input image;

[0021] According to the first step of generating an IHC image, obtaining a semantic relationship distribution of the generated image;

[0022] According to the real IHC image, obtaining a real image semantic relationship distribution;

[0023] Calculating a first distance between the input image semantic relationship distribution and the generated image semantic relationship distribution, and calculating a second distance between the generated image semantic relationship distribution and the real image semantic relationship distribution;

[0024] A semantic relationship consistency loss function is calculated according to the first distance and the second distance.

[0025] Preferably, in step 3, the semantic relationship distribution of the real image is equal to the semantic relationship distribution of the input image.

[0026] In a second aspect, a HE image to IHC image staining generation system based on image block similarity is provided, which is used to execute any method described in the first aspect, including:

[0027] A first acquisition module is used to acquire an image pair, wherein the image pair includes a HE image and a real IHC image that correspond in structure;

[0028] A second acquisition module is used to acquire an initial image generator; input the HE image into the initial image generator to acquire a first generated IHC image;

[0029] A calculation module, used for calculating a semantic supervision PatchNCE loss function, a semantic relationship consistency loss function, a supervised PatchNCE loss function and a GAN loss function according to the image pair and the first generated IHC image;

[0030] The third acquisition module is used to obtain the total loss function according to the given weight parameters and the loss functions in the calculation module;

[0031] A training module, used for training the initial image generator through the total loss function;

[0032] The fourth acquisition module is used to input the HE image into the trained initial image generator to obtain a second generated IHC image.

[0033] According to a third aspect, a computer storage medium is provided, wherein a computer program is stored in the computer storage medium; when the computer program is executed on a computer, the computer executes any method described in the first aspect.

[0034] In a fourth aspect, an electronic device is provided, including:

[0035] Memory, used to store computer programs;

[0036] A processor is used to execute the computer program to implement any method as described in the first aspect.

[0037] The beneficial effects of the present invention are:

[0038] 1. This paper proposes a computational semantic supervision PatchNCE loss function, which integrates other image patches as positive examples to alleviate the negative impact of incompletely calibrated HE-IHC images on the model.

[0039] 2. The present invention proposes a semantic relationship consistency loss function to regularize the uncalibrated regions between image pairs, so that the generated image has consistent overall semantic information with the true target image, rather than just consistent with the input image. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of a method for generating staining from a HE image to an IHC image based on image block similarity provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a HE-IHC image pair provided by an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of the overall framework of the network structure provided by an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of the overall framework of PatchNCE loss and SSP loss provided in an embodiment of the present invention;

[0044] Figure 5 A diagram showing the SRC regularization term provided in an embodiment of the present invention;

[0045] Figure 6 A schematic diagram of an IHC image generated on a dataset provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0047] Embodiment 1:

[0048] HE and IHC are two different staining techniques frequently used in cancer diagnosis. HE staining is relatively more economical and convenient. In contrast, the cost of IHC staining is relatively high, and the operating procedures involved are more complicated, which has prompted researchers to want to directly use algorithm models to directly generate IHC staining images from HE staining images. However, due to the specificity of the staining procedure, the number of well-calibrated images is limited. Limited high-quality training data greatly affects the performance of current image generation models.

[0049] To solve this problem, this application considers the semantic similarity between image patches as a key factor in optimizing the image generator, and proposes a new image patch contrast learning strategy, Semantic Supervised PatchNCELoss (SSP). SSP loss prompts the generator to focus on image patches with higher semantic similarity around the anchor point, and add these image patches as positive examples to contrast learning, thereby alleviating the adverse effects of image miscalibration. To further improve performance, this application directly incorporates semantic similarity between image patches as a regularization term into the loss function. This prompts the model of this application to focus on the overall semantic consistency between the generated image and the original image.

[0050] As an example, Figure 1 As shown, the present application provides a method for generating staining from a HE image to an IHC image based on image block similarity, comprising:

[0051] Step 1: Acquire an image pair, wherein the image pair includes a HE image and a real IHC image that correspond in structure.

[0052] For natural images, the real output images in the image dataset are usually obtained by hand-painting, PS technology or image segmentation of the input image. Figure 2 As shown in the figure, although it cannot be said that HE images and real IHC images are completely calibrated at the pixel level, it is generally acceptable to use pixel-level constraints when training models with these data. Similarly, after the medical image is registered, the HE-IHC image has image blocks with the same semantic structure, although there is a slight shift in position, it can still be observed and identified as a pair of image blocks with the same structure.

[0053] Furthermore, although the calibration between the real IHC image and the HE image may not be sufficient, the semantic structure of the two images has a certain degree of similarity when observed from a large area. This shows that although there may be a large offset between small-sized image patches at the same position, the entire image can still be considered to have a consistent structure and can be used as supervised model training data for this staining conversion task.

[0054] Step 2: obtaining an initial image generator; inputting the HE image into the initial image generator to obtain a first generated IHC image.

[0055] For example, the present application provides an image generator G, which can Convert to . This application first describes InfoNCE loss, which is the most widely used contrastive learning loss and is widely used in unsupervised visual representation learning. The main idea of ​​this method is to transform the image representation task into an image classification task. Specifically, in a multi-classification framework, it regards the original image enhanced as a positive example and other images as negative examples. This framework allows the model to learn robust features under unsupervised data learning conditions:

[0056]

[0057] in , ,and Represent anchor boxes, positive examples and negative examples respectively, is the temperature coefficient in the contrastive learning loss function.

[0058] In addition, inspired by the contrastive learning loss, the unpaired image patches in the image-to-image translation task are regarded as semi-supervised labels, and the PatchNCE loss can be calculated. have Layer semantic extractor, where Indicates the number of image blocks in each layer of feature maps. Represented as the vector embedding for each image patch, represents the set of image block vectors at non-corresponding positions, Represented as the generated image, then there is a loss function The supervised PatchNCE loss function introduces a bidirectional image block contrast loss in the PatchNCE loss function, which provides a more sufficient supervision signal for the generator, so that the reconstruction result looks closer to the real target. Then there is a loss function .

[0059] Step 3: Calculate a semantic supervision PatchNCE loss function, a semantic relation consistency (Semantic Relation Consistency, SRC) loss function, a supervised PatchNCE loss function and a GAN loss function according to the image pair and the first generated IHC image.

[0060] Step 3 includes:

[0061] Step 3.1: Calculate the semantically supervised PatchNCE loss function. The semantically supervised PatchNCE loss function encourages the generator to focus on image patches with higher semantic similarity around the anchor points and add these image patches as positive examples to contrastive learning, thereby alleviating the adverse effects of image miscalibration.

[0062] Specifically, step 3.1 includes:

[0063] Step 3.1.1: Perform image block sampling on the image pair and the first generated IHC image.

[0064] Specifically, each image block in the first generated IHC image is used as an anchor point, and the positive examples with the same position as the anchor point are selected in the real IHC image, and the image blocks with the highest similarity to the anchor point are selected. image blocks as auxiliary positive examples.

[0065] Step 3.1.2: Extract features based on the sampled image blocks.

[0066] Step 3.1.3: Based on the extracted features, perform contrastive learning and calculate the semantic supervision PatchNCE loss function.

[0067] It should be noted that the dimensional size of the image blocks used in the contrast learning is Figure 4The crop size shown is much smaller; therefore, this application uses such small boxes to accurately describe the steps of the method. The vectors of the sampled image patches are extracted by the shared MLP network and then used for contrastive learning loss calculation. The SSP loss of this application adds additional image patch vectors (such as Figure 4 The red box shows the positive samples as supplementary samples for contrastive learning to alleviate the negative impact of incompletely calibrated real IHC images.

[0068] For example, Figure 4 The framework of PatchNCE loss and semantically supervised PatchNCE loss is presented, where It is a HE image; Generate IHC images for the first time; This is a real IHC image.

[0069] This application is based on the supervised PatchNCE loss function and adds auxiliary positive examples to the loss function. The semantic supervised PatchNCE loss function can be written as:

[0070]

[0071] in, represents the number of positive examples used when calculating the loss, is the number of sampled anchor points. Each embedding vector in the formula is passed through a two-layer MLP This application uses each image block in the generated image as an anchor point, and selects the positive examples with the same position as the anchor point in the real image, and the ones with the highest similarity to the anchor point. image blocks are used as positive examples. The similarity is calculated using the cosine similarity of the image block vectors. Anchor points , the selected positive example embedding vectors form a set ,and Represents all positive example vectors corresponding to all anchor points in the input image.

[0072] For feature vectors of small-sized image blocks, we may find positive example vectors with high feature vector similarity but far-distance in spatial position from the entire feature map. This pairing is obviously not what we want. However, since the image pairs have been preliminarily registered, although the pairing of each small-area image block may not be perfect, the overall image registration is still acceptable. Therefore, we can select supplementary positive examples only from the 8 image blocks around the corresponding positive example. Subsequent experimental results show that this method of selecting positive examples is both convenient and efficient.

[0073] Step 3.2: Calculate the semantic relationship consistency loss function, the supervised PatchNCE loss function, and the GAN loss function.

[0074] Step 4: Obtain the total loss function based on the given weight parameters and the loss functions in step 3.

[0075] Step 5: Train the initial image generator using the total loss function.

[0076] Step 6: Input the HE image into the trained initial image generator to obtain a second generated IHC image.

[0077] Embodiment 2:

[0078] Based on Example 1, Example 2 of the present application provides a more specific method for generating staining from HE image to IHC image based on image block similarity, including:

[0079] Step 1: Acquire an image pair, wherein the image pair includes a HE image and a real IHC image that correspond in structure.

[0080] Step 2: obtaining an initial image generator; inputting the HE image into the initial image generator to obtain a first generated IHC image.

[0081] Step 3: Calculate the semantic supervision PatchNCE loss function, the semantic relationship consistency loss function, the supervised PatchNCE loss function and the GAN loss function according to the image pair and the first generated IHC image.

[0082] Step 3 includes:

[0083] Step 3.1, calculate the semantic relationship consistency loss function. This application incorporates the semantic similarity between image blocks as a regularization term into the loss function. This prompts the model of this application to focus on the overall semantic consistency between the generated image and the original image.

[0084] Specifically, step 3.1 includes:

[0085] Step 3.1.1, obtaining the input image semantic relationship distribution according to the HE image;

[0086] Step 3.1.2, according to the first generated IHC image, obtaining the generated image semantic relationship distribution;

[0087] Step 3.1.3, obtaining a real image semantic relationship distribution according to the real IHC image;

[0088] Step 3.1.4, calculating a first distance between the input image semantic relationship distribution and the generated image semantic relationship distribution, and calculating a second distance between the generated image semantic relationship distribution and the real image semantic relationship distribution;

[0089] Step 3.1.5: Calculate the semantic relationship consistency loss function based on the first distance and the second distance.

[0090] For example, Figure 5 shows the regularization term, Indicates the generated image block The distribution of semantic relationships with other patches in the same image. For each patch vector of the generated image , with another patch vector of the same image The semantic relationship is defined as:

[0091]

[0092] Similarly, the image patch in the real image The semantic relationship distribution of is defined as:

[0093]

[0094] in and Respectively represent and The embedding vectors of the same position but belonging to different images. Semantic Relation Consistency (SRC) brings the semantic distributions of the two closer, ensuring that the generated image retains the overall semantic relationship consistent with the input image. The loss function term uses the Jensen-Shannon (JS) divergence to calculate the distance between the two distributions:

[0095]

[0096] As mentioned above, in HE-IHC image pairs, small-sized image patches may be significantly misaligned, but the overall image pair can still maintain the consistency of semantic relations. Inspired by this observation and the supervised PatchNCE loss, this application believes that the semantic relationship between patches in the real output image should not only be consistent with the input image, but also with the generated image. Here, the additional SRC regularization term can be written as:

[0097]

[0098] The SSP loss focuses on the semantic consistency of image patches, while the SRC regularization works at the level of the entire image.

[0099] Step 3.2: Calculate the semantic supervised PatchNCE loss function, the supervised PatchNCE loss function, and the GAN loss function.

[0100] Step 4: Obtain the total loss function based on the given weight parameters and the loss functions in step 3.

[0101] Specifically, the total loss function can be described as

[0102]

[0103] in, , ,and is the weight parameter.

[0104] Step 5: Train the initial image generator using the total loss function.

[0105] Step 6: Input the HE image into the trained initial image generator to obtain a second generated IHC image.

[0106] In addition, this application uses two data sets, BCI and MIS, to evaluate the technical solution of this application.

[0107] Among them, the BCI dataset manually selects feature points on the whole slice image (WSI) and applies projection transformation for initial image correction, and then uses the Elastix tool for fine-tuning. BCI has only one type of IHC staining image, namely HER2, and the dataset contains 3896 pairs of images for training and 977 pairs of images for testing. MIST uses the Valis method to automatically align HE-IHC whole slice images. MIST has four pairs of HE-IHC staining images, namely HER2, ER, Ki67, and PR. Each part of MIST has approximately 4500 pairs of images for training and 1000 pairs of images for testing.

[0108] In the experiment, this application used the entire dataset of BCI and MIST. It should be noted that the size of all images is 1024 1024, and this application does not make any modifications to the image data before applying image augmentation in training.

[0109] In all experiments, this application uses the same parameter settings as ASP loss to train the model of this application. The restriction property of the loss term is too strong, so this application removes it from the loss function. This application uses a two-layer MLP, similar to the CUT process, with 256 image blocks sampled per layer. In order to highlight the effectiveness of this application's model, this application only uses ResNet-9blocks as generators in the CycleGAN and Pix2Pix (P2P) models, while all other models use ResNet-6blocks as generators. The hyperparameters are set to =10, =10, = 1.

[0110] It should be noted that this application uses FID (Fréchet Inception Distance) and KID (Kernel Inception Distance) as evaluation indicators to evaluate the distance between the data distribution of the generated image and the real image. This non-pixel level comparison provides a more objective evaluation, especially for datasets with imperfect image alignment. In addition, this application also uses PHV (perceptual hash value) to evaluate the similarity between the generated image and the real image at the feature map level. The points in the feature map extracted by the pre-trained network can reflect the characteristics of the surrounding large area.

[0111] In addition, in Table 1, the present application shows quantitative indicators comparing the present application's method with various baseline methods (such as CycleGAN, Pix2Pix, CUT, PyramidP2P, ASP). From the results in Table 1, overall, the present application's method performs very well on all five datasets. The performance on the BCI dataset is slightly inferior, which may be due to the lower brightness of the images in the BCI dataset. This may result in the inability to obtain accurate feature maps of the input image using only the image block level loss function (PatchNCE).

[0112] Table 1

[0113]

[0114] Table 1 shows the experimental results of five datasets. It means the lower the index, the better. It can be noted that the best value of each evaluation index is marked in bold.

[0115] also, Figure 6The results of qualitative comparison of generated images are shown. By comparing the images generated by the method of this application with those generated by several other benchmark models, the images generated by the model using L1 loss are obviously blurred or lack realism, while the method proposed in this application can significantly capture the specific semantic features of the image and generate output in a targeted manner to obtain a more realistic output image.

[0116] Table 2 is the ablation experiment of the method proposed in this application. This application adopts the CUT model as the basic framework and adds additional loss terms to its loss function. SP refers to the supervised PatchNCE loss function. SRC and SSP are two methods proposed in this application. Total refers to the comprehensive effect of adding SRC and SSP to the model.

[0117] Table 2

[0118]

[0119] Table 2 verifies that the auxiliary positive examples and SRC regularization method of this application can improve the performance of CUT. This application gradually adds various components of the method of this application, and observes that the experimental indicators are gradually improved in Table 2. The above results show that both components of the method of this application can effectively utilize the semantic information between the real image and the output result, thereby improving the quality of the generated image.

[0120] Table 3 shows the PHV, FID and KID scores of the method of this application under different settings, mainly including the number of auxiliary image patches used in this application and the selection of supervised contrastive learning function.

[0121] Table 3

[0122]

[0123] In order to determine the number of auxiliary image patches with the best experimental results, this application conducted experiments on the HER2 dataset with different settings of 0, 1, 2, and 3 auxiliary image patches. The results in Table 3 show that the experimental results are best when a single auxiliary image patch is used. This is because too many auxiliary image patches may distract the model from paying attention to accurately aligning image patch pairs. In addition, for specific image generation tasks involving multiple positive examples, this application also compares another supervised contrastive learning method in Table 3, which is expressed as:

[0124]

[0125] The results in the table show Compared to the initial application The settings in , which will lead to worse model results.

[0126] In summary, this application explores the value of semantic similarity in the model and uses it as an important part of the loss function. This application introduces this concept into the framework proposed in this application from two different perspectives: regularization and contrastive learning. Bringing the semantic similarity distribution of image patches close solves the problem of overall image consistency, while semantic supervision PatchNCELoss focuses on the semantic consistency between the results and the supervision signal image patch level. Through extensive experiments on different datasets, this application proves that the method of this application has superior results compared to previous models.

[0127] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0128] Embodiment 3:

[0129] Based on embodiments 1 and 2, embodiment 3 of the present application provides a HE image to IHC image staining generation system based on image block similarity, including:

[0130] A first acquisition module is used to acquire an image pair, wherein the image pair includes a HE image and a real IHC image that correspond in structure;

[0131] A second acquisition module is used to acquire an initial image generator; input the HE image into the initial image generator to acquire a first generated IHC image;

[0132] A calculation module, used for calculating a semantic supervision PatchNCE loss function, a semantic relationship consistency loss function, a supervised PatchNCE loss function and a GAN loss function according to the image pair and the first generated IHC image;

[0133] The third acquisition module is used to obtain the total loss function according to the given weight parameters and the loss functions in the calculation module;

[0134] A training module, used for training the initial image generator through the total loss function;

[0135] The fourth acquisition module is used to input the HE image into the trained initial image generator to obtain a second generated IHC image.

[0136] Specifically, the system provided in this embodiment is a system corresponding to the method provided in Embodiments 1 and 2. Therefore, the parts in this embodiment that are the same or similar to Embodiments 1 and 2 can be referenced to each other and will not be repeated in this application.

Claims

1. A method for generating staining from HE image to IHC image based on image block similarity, characterized in that: include: Step 1: acquiring an image pair, wherein the image pair includes a HE image and a real IHC image that correspond in structure; Step 2: obtaining an initial image generator; inputting the HE image into the initial image generator to obtain a first generated IHC image; Step 3: Calculate the semantic supervision PatchNCE loss function, the semantic relationship consistency loss function, the supervised PatchNCE loss function and the GAN loss function according to the image pair and the first generated IHC image; In step 3, the semantic supervision PatchNCE loss function is calculated, including: Performing image block sampling on the image pair and the first generated IHC image; Perform feature extraction based on the sampled image blocks; Based on the extracted features, contrastive learning is performed and the semantic supervision PatchNCE loss function is calculated; In step 3, calculating the semantic relationship consistency loss function includes: According to the HE image, obtaining a semantic relationship distribution of an input image; According to the first step of generating an IHC image, obtaining a semantic relationship distribution of the generated image; According to the real IHC image, obtaining a real image semantic relationship distribution; Calculating a first distance between the input image semantic relationship distribution and the generated image semantic relationship distribution, and calculating a second distance between the generated image semantic relationship distribution and the real image semantic relationship distribution; Calculate a semantic relationship consistency loss function according to the first distance and the second distance; Step 4: Obtain the total loss function based on the given weight parameters and the loss functions in step 3; The total loss function is described as: in, is the weight parameter, is the GAN loss function, is the supervised PatchNCE loss function, is the semantic supervision PatchNCE loss function, represents the semantic relationship consistency loss function; Step 5: training the initial image generator through the total loss function; Step 6: Input the HE image into the trained initial image generator to obtain a second generated IHC image.

2. The method for generating staining from HE image to IHC image based on image block similarity according to claim 1, characterized in that: In step 3, the image block sampling includes: Each image block in the first generated IHC image is used as an anchor point, and the positive examples with the highest similarity to the anchor point are selected around the same position as the anchor point in the real IHC image. image blocks as auxiliary positive examples, Indicates the number of positive examples used when computing the loss.

3. The method for generating staining from HE image to IHC image based on image block similarity according to claim 2, characterized in that: In step 3, the semantic relationship distribution of the real image is equal to the semantic relationship distribution of the input image.

4. A HE image to IHC image staining generation system based on image block similarity, characterized in that: The method for executing any one of claims 1 to 3 comprises: A first acquisition module is used to acquire an image pair, wherein the image pair includes a HE image and a real IHC image that correspond in structure; A second acquisition module is used to acquire an initial image generator; input the HE image into the initial image generator to acquire a first generated IHC image; A calculation module, used for calculating a semantic supervision PatchNCE loss function, a semantic relationship consistency loss function, a supervised PatchNCE loss function and a GAN loss function according to the image pair and the first generated IHC image; The third acquisition module is used to obtain the total loss function according to the given weight parameters and the loss functions in the calculation module; A training module, used for training the initial image generator through the total loss function; The fourth acquisition module is used to input the HE image into the trained initial image generator to obtain a second generated IHC image.

5. A computer storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed on a computer, the computer executes any one of the methods described in claims 1 to 3.

6. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 3.

Images