A pathological image virtual staining method based on multi-task learning

Abstract

The application discloses a kind of pathological image virtual dyeing methods based on multi-task learning, steps include: a pair of virtual dyeing model is constructed, respectively including the generator and discriminator of corresponding source domain and target domain;For generator, it is reconstructed, conversion across dyeing domain in self-supervision;For discriminator, add pathological section category supervision loss in self-supervision adversarial loss;After such alternation training, update, finally realize the image conversion between source domain and target domain and corresponding pathological classification task.The application uses unpaired dyeing dataset, introduces classification loss by pathological category label, optimizes objective function design, adds classification module, back gradient guides model to learn inter-class difference dyeing feature, and improves the generation effect of pathological image virtual dyeing.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field related to computer-aided diagnosis, image processing and deep learning, and particularly relates to a pathological image virtual staining method based on multi-task learning. BACKGROUND

[0002] As the gold standard for tumor diagnosis, pathologists perform pathological examination under an optical microscope, and complete the diagnosis and grading of tumors by observing the morphology and cell characteristics of tumor tissues after staining. Hematoxylin-eosin (HE) is the most commonly used staining agent in pathological diagnosis, which focuses on showing the organizational structure of cells. However, HE staining does not have all the information required for disease grading, and for different diseases, pathologists still need different immunohistochemistry (IHC) staining agents to analyze specific structures or abnormal substances. Common immunohistochemical staining, such as P57, can better color the trophoblast layer; Ki67 shows specific representation of cell nuclei and is commonly used for cervical cancer screening, etc. However, the preparation of such special staining is time-consuming and laborious, and different IHC reagents are required for specific diseases, which is not feasible for downstream hospitals to promote and develop.

[0003] With the advent of digital scanners and deep learning, the diagnosis of pathological sections has gradually changed from microscopic examination under a microscope to digital section reading. As a computer-aided diagnosis method, a model based on deep learning methods can generate different virtual staining effects on a single staining section, to some extent, solving the limitations of traditional pathological section staining methods and improving the effect of pathological diagnosis. A trained deep learning model can also provide consistent staining style and reduce the impact of changes in staining reagents in different centers on reading.

[0004] Conditional generative adversarial network (Pix2Pix) and cycle generative adversarial network (CycleGAN) are two commonly used algorithm frameworks for image style transfer. The performance of the method based on Pix2Pix largely depends on a large number of pixel-level paired training datasets. However, in the process of pathological staining, tissue loss and movement often occur, and the annotation of training paired images requires a lot of manpower, which is difficult to achieve accurate correspondence at the pixel level, resulting in the subsequent model being prone to artifacts and affecting the quality of virtual staining.

[0005] CycleGAN does not require a paired dataset to complete the conversion between two image styles, so the method is sensitive to the domain distribution of the converted image. For some immunohistochemical staining methods, images between different pathological categories will show obvious positive and negative differences, that is, there is class-specific information. Using the conventional CycleGAN method, the IHC staining is simply classified into a staining domain for staining transfer, which does not work well. If further classification is performed between classes, on the one hand, multiple CycleGAN models need to be trained, increasing the data collection cost, and more importantly, the staining task itself is to better classify the pathology. If the training and inference of the virtual staining model are performed according to the pathological categories, there is no practical significance, which deviates from the original intention of the virtual staining task. SUMMARY

[0006] In view of the above, the purpose of the present application is to provide a pathological image virtual staining method and device based on multi-task learning, aiming to combine virtual staining and classification tasks to jointly supervise model training, add pathological diagnosis level categories instead of image histology categories in the original unpaired training dataset, without increasing the difficulty of dataset annotation, optimize the model structure design, add a class classification module, introduce a pathological diagnosis level category loss, adjust the loss function design, guide the model to learn the inter-class difference features of pathological images, and improve the generation effect of pathological image virtual staining.

[0007] To achieve the above-mentioned purpose of the application, the embodiment provides a pathological image virtual staining method based on multi-task learning, comprising the following steps:

[0008] A pair of virtual staining models are constructed, each of which includes a generator and a discriminator, and one virtual staining model corresponds to a virtual staining task from a source domain to a target domain and a pathological image classification task, and the other virtual staining model corresponds to a virtual staining task from the target domain to the source domain and a pathological image classification task;

[0009] The source domain image and the target domain image of the same pathological category are input, and the generator and the discriminator are alternately trained, and the model objective function includes the generator loss function and the discriminator loss function during training, wherein the generator loss function includes ontology loss, adversarial loss, cycle loss and classification loss, and the discriminator loss function includes another adversarial loss and another classification loss;

[0010] After training is completed, the source domain image is input into the generator included in the virtual staining model corresponding to the source domain to the target domain, and a virtual staining image of the target domain can be generated without pathological categories.

[0011] Preferably, the generator adopts a Unet structure composed of an encoder and a decoder, and a Resnet residual structure is added in the middle to optimize the feature extraction of the encoder.

[0012] Preferably, the discriminator employs a stacked convolutional neural network structure, with a fully connected layer added as the pathological category classification part.

[0013] Preferably, for the virtual staining model M from the source domain to the target domain s2t Includes target domain generator G s2t and target domain discriminator D s2t Virtual coloring model M from target domain to source domain t2s Includes source domain generator G t2s Source Domain Discriminator D t2s ;

[0014] Source domain images with pathology category c x s Input to source domain generator G t2s The generated image is obtained from the process. x s ', based on source domain image x s and generating images x s 'Construct the first ontology loss, and classify the source domain image as pathology category c.' x s Input to target domain generator G s2t Generating pseudo-images and for pseudo-images Assigning the first true label, the target domain discriminator D s2t For the input pseudo image Perform the first true / false image discrimination and generate the first predicted pathological category. Construct the first adversarial loss based on the first true label and the first true / false image discrimination result. Construct the first classification loss based on the pathological category label c and the first predicted pathological category. Generate the pseudo image. Input to source domain generator G t2s Generate reconstructed images ', based on source domain image x s and reconstructed images 'Construct the first cycle loss;'

[0015] The target domain image of pathology category c x t Input to target domain generator G s2t The generated image is obtained from the process. x t ', based on target domain image x t and generating images x t 'Construct a second ontology loss to transform the target domain image of pathology category c.' x tInput to source domain generator G t2s Generating pseudo-images and for pseudo-images Assigning a second true label, source domain discriminator D t2s For the input pseudo image A second true / false image is generated, and a second predicted pathological category is produced. A second adversarial loss is constructed based on the second true label and the second true / false image discrimination results. A second classification loss is constructed based on the pathological category label c and the second predicted pathological category. The generated pseudo-image... Input to target domain generator G s2t Generate reconstructed images ', based on target domain image x t and reconstructed images 'Construct a second-cycle loss;'

[0016] Based on all the losses constructed above, the target domain generator G... s2t Source domain generator G t2s Conduct training.

[0017] Preferably, the first and second ontology losses are the expected values ​​of the LI norm of the two images.

[0018] Preferably, the first cyclic loss and the second cyclic loss are the expected values ​​of the LI norm of the two images.

[0019] Preferably, the first adversarial loss is the mean squared error loss of the difference between the first true / false judgment result and the first true label;

[0020] The second adversarial loss is the mean squared error loss of the difference between the second true / false judgment result and the second true label.

[0021] Preferably, both the first classification loss and the second classification loss are the expected values ​​of the cross-entropy loss between the predicted pathology category and the pathology label.

[0022] Source domain image with pathology category label c x s Assign a third true label and input it into the source domain discriminator D. t2s The third true / false judgment result and the third predicted pathological category were obtained, indicating a false image. Assign a fourth pseudo-label and input it into the source domain discriminator D. t2s The fourth true / false discrimination result and the fourth predicted pathological category are obtained. Based on the third true label and the third true / false discrimination result, the fourth false label and the fourth true / false discrimination result, the third adversarial loss is constructed. Based on the pathological category label c and the third predicted pathological category and the fourth predicted pathological category, the third classification loss is constructed.

[0023] For the target domain image with pathology category label cx t The fifth true label is given, and input to the target domain discriminator D s2t The fifth true-false discrimination result and the fifth predicted pathological category are obtained, and the image is a false image The sixth false label is given, and input to the source domain discriminator D t2s The sixth true-false discrimination result and the sixth predicted pathological category are obtained, and based on the fifth true label and the fifth true-false discrimination result, the sixth false label and the sixth true-false discrimination result, a fourth adversarial loss is constructed, and based on the pathological category label c and the fifth predicted pathological category and the sixth predicted pathological category, a fourth classification loss is constructed.

[0024] The target domain discriminator D s2t and the source domain discriminator D t2s are trained based on the third adversarial loss, the third classification loss, the fourth adversarial loss, and the fourth classification loss.

[0025] Preferably, the third adversarial loss adopts the mean square error loss of the gap between the third true-false discrimination result and the third true label and the mean square error loss of the gap between the fourth true-false discrimination result and the fourth false label.

[0026] The fourth adversarial loss adopts the mean square error loss of the gap between the fifth true-false discrimination result and the fifth true label and the mean square error loss of the gap between the sixth true-false discrimination result and the sixth false label.

[0027] Preferably, the third classification loss and the fourth classification loss both adopt the expected value of the cross-entropy loss of the corresponding predicted pathological category and the pathological label.

[0028] Compared with the prior art, the present application has at least the following beneficial effects:

[0029] The pathological image virtual staining method based on multi-task learning provided by the present application adds class supervision during model training, guiding the generator to learn the staining features of different categories. Due to the hidden affinity between classification and virtual staining tasks, the virtual staining method not only imitates the statistical density of the color space, but also learns the distinguishing features between different pathological categories and generates corresponding staining patterns, thereby improving the virtual staining quality. It is worth noting that the tile pathological category label used for training in the present application is derived from the diagnosis result of the whole pathological section, not the histological classification label of a specific image, and no additional annotation is required, so it can replace the current virtual staining model and has practical value. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative effort based on these drawings.

[0031] Figure 1 is a flowchart of a pathological image virtual staining method based on multi-task learning provided by the embodiment;

[0032] Figure 2 is a schematic diagram of a pathological image virtual staining framework based on multi-task learning provided by the embodiment;

[0033] Figure 3 is a flowchart of generator training provided by the embodiment;

[0034] Figure 4 is a flowchart of discriminator training provided by the embodiment;

[0035] Figure 5 is a comparison diagram of P57 virtual staining generation results provided by the embodiment. DETAILED DESCRIPTION

[0036] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative effort based on these drawings.

[0037] As shown in Figure 1 , the embodiment provides a pathological image virtual staining method based on multi-task learning, comprising the following steps:

[0038] S1, a pair of virtual staining models are constructed, each virtual staining model comprising a generator G and a discriminator D.

[0039] In the embodiment, a virtual staining model M s2t corresponding to the virtual staining task of the source domain to the target domain and the pathological image classification task, comprising a target domain generator G s2t and a target domain discriminator D s2t , and another virtual staining model M t2s corresponding to the virtual staining task of the target domain to the source domain and the pathological image classification task, comprising a source domain generator G t2s and a source domain discriminator D t2s .

[0040] Each generator employs a UNet structure consisting of an encoder and a decoder, with a ResNet residual structure added in between to optimize the encoder's feature extraction. Each discriminator uses a stacked convolutional neural network structure, with fully connected layers added as the pathology category classification part to control its convergence speed and better guide gradient propagation.

[0041] S2, input source and target domain images of the same pathological category, and train the generator and discriminator alternately. During training, the pathological category... To correspond to the image diagnostic labels, the model's objective function during training includes a generator loss function and a discriminator loss function. The generator loss function includes ontology loss, adversarial loss, recurrence loss, and classification loss, while the discriminator loss function includes additional adversarial loss and additional classification loss.

[0042] During training, such as Figure 2 As shown, the generator takes real images as input and generates pseudo images without specifying the pathological category; the discriminator learns to distinguish between real and pseudo images and attempts to classify the images into their corresponding pathological categories; the generator reconstructs images from the pseudo images, hoping to obtain the same results as the real images.

[0043] Specifically, source domain images with pathology category c x s Input to source domain generator G t2s The generated image is obtained from the process. x s ', based on source domain image x s and generating images x s Construct the first ontology loss for the source domain image with pathology category c. x s Input to target domain generator G s2t Generating pseudo-images and for pseudo-images Assigning the first true label, the target domain discriminator D s2t For the input pseudo image Perform the first true / false image discrimination and generate the first predicted pathological category. Construct the first adversarial loss based on the first true label and the first true / false image discrimination result. Construct the first classification loss based on the pathological category label c and the first predicted pathological category. Generate the pseudo image. Input to source domain generator G t2s Generate reconstructed images ', based on source domain image x s and reconstructed images 'Construct the first cycle loss.'

[0044] target domain image with pathology category c x t input into the target domain generator G s2t to generate a generated image x t based on the target domain image x t and the generated image x t construct a second ontology loss, input the target domain image with pathology category c x t into the source domain generator G t2s to generate a pseudo image , and assign a second true label to the pseudo image , the source domain discriminator D t2s performs second true-false discrimination on the input pseudo image and generates a second predicted pathology category, constructs a second adversarial loss based on the second true label and the second true-false discrimination result, and constructs a second classification loss based on the pathology category c and the second predicted pathology category, inputs the generated pseudo image into the target domain generator G s2t to generate a reconstructed image based on the target domain image x t and the reconstructed image ;

[0045] All the losses constructed above are weighted to train the target domain generator G s2t and the source domain generator G t2s . Among them, the first ontology loss and the second ontology loss adopt the expected value of the LI norm of two images. The first cycle loss and the second cycle loss adopt the expected value of the LI norm of two images, the first adversarial loss adopts the mean square error loss of the gap between the first true-false discrimination result and the first true label; the second adversarial loss adopts the mean square error loss of the gap between the second true-false discrimination result and the second true label. The first classification loss and the second classification loss both adopt the expected value of the cross-entropy loss.

[0046] When training the discriminator, the preset training label for the generated image is true, which is different from the preset label (pseudo) when training the generator. Specifically, the source domain image with pathology category c x s is assigned a third true label and input into the source domain discriminator D t2s to obtain a third true-false discrimination result and a third predicted pathology category, the pseudo image is assigned a fourth pseudo label and input into the source domain discriminator D t2sobtain a fourth authenticity discrimination result and a fourth predicted pathological category, construct a third adversarial loss based on the third true label and the fourth authenticity discrimination result, and the fourth false label and the fourth authenticity discrimination result, and construct a third classification loss based on the pathological category label c and the fourth predicted pathological category;

[0047] a target domain image of which the pathological category is c x t assign a fifth true label, and input to the target domain discriminator D s2t obtain a fifth authenticity discrimination result and a fifth predicted pathological category, and the fifth predicted pathological category is a false image assign a sixth false label, and input to the source domain discriminator D t2s obtain a sixth authenticity discrimination result and a sixth predicted pathological category, construct a fourth adversarial loss based on the fifth true label and the fifth authenticity discrimination result, and the sixth false label and the sixth authenticity discrimination result, and construct a fourth classification loss based on the pathological category label c and the fifth predicted pathological category and the sixth predicted pathological category;

[0048] train the target domain discriminator D s2t and the source domain discriminator D t2s by weighting the third adversarial loss, the third classification loss, the fourth adversarial loss, and the fourth classification loss. Wherein, the third adversarial loss adopts the mean square error loss of the gap between the third authenticity discrimination result and the third true label and the mean square error loss of the gap between the fourth authenticity discrimination result and the fourth false label; the fourth adversarial loss adopts the mean square error loss of the gap between the fifth authenticity discrimination result and the fifth true label and the mean square error loss of the gap between the sixth authenticity discrimination result and the sixth false label; the third classification loss and the fourth classification loss both adopt the expected value of the cross entropy loss of the corresponding predicted pathological category and the pathological label.

[0049] Wherein, the ontology loss aims to guide the generator to pay attention to the staining difference between different conversion domains, and distinguish it from the ontology structure of the picture. When the image is input to the generator, it is expected to perform generator ontology mapping, that is, the output image is the same as the input image.

[0050] The adversarial loss is applied to the training of the generator and the discriminator. The optimization direction of the parameters of the generator is to minimize the loss, which aims to make the distribution of the generated false image close to that of the real image to deceive the discriminator, while the discriminator tries to maximize the adversarial loss to judge the authenticity of the input image as much as possible. The two are alternately optimized to achieve the purpose of adversarial promotion.

[0051] The classification loss aims to guide the generator to pay attention to the staining difference between different categories. Taking the source domain generator as an example, its optimization goal is to convert the image into a false image , and correctly classify it into the pathological category of the target domain . The method adds an auxiliary classifier on top of the discriminator, in embodiments, trying to minimize the classification loss to generate images that can be classified as c the target domain image of the class , then tries to learn to classify the input image correctly.

[0052] The cycle loss aims to suppress the mode vanishing problem, expecting the image after dye transfer to only change the color domain related part, i.e. the generated image maintains similar morphological structure with the original image. For the source domain image , the optimization goal of the source domain generator is that the reconstructed image ’ after the second conversion is consistent with the original image .

[0053] The above generator and discriminator are alternately trained to carry out the training process of the two models, and when a given training round is completed or the early stopping strategy is triggered, i.e. before the generated image appears obvious artifacts, the training of the two models is completed.

[0054] S3, after the training is completed, the source domain image is input into the generator included in the virtual dyeing model corresponding to the target domain from the source domain, and the virtual dyeing image of the target domain can be generated without the pathology class.

[0055] In actual use, the generator part can be deployed separately for inference, and the source domain image can be input without the pathology class label to complete the virtual dyeing task.

[0056] Next, taking HE dyeing as the source domain and P57 dyeing as the target domain as an example, the following is described, at this time, the target domain generator G s2t and the target domain discriminator D s2t are named as and , respectively, and the source domain generator G t2s and the source domain discriminator D t2s are named as and , respectively. For convenience of description, the part of judging authenticity is recorded as , and the part of judging class is recorded as , , and so on. In each training process, the source domain image c and the target domain image of the pathology class are extracted from the data set, preprocessed, and data augmentation (including flipping, rotation, appropriate scaling, etc.) is completed.

[0057] During the model training process, the generator loss function is calculated first, and the corresponding parameters are updated,Figure 3 For generator training in virtual staining task, loss function calculation flow chart, when generator parameter is updated, discriminator is in frozen state, only inference, no parameter update, generator loss function calculation flow as follows:

[0058] (1) input P57 staining image into generator , output picture , calculate L1 loss as ontology loss with input image , guide generator pay attention to the staining state of input image, pay attention to the color distribution difference between different staining domains, distinguish it from the tissue structure characteristics of picture, its formula is:

[0059]

[0060] For and generator , similar operation is carried out to obtain , the total ontology loss function calculation formula of generator is , the following corresponding generator total loss function calculation method is the same, not described again.

[0061] (2) input HE staining picture into generator , generate pseudo P57 staining picture , send it into discriminator to calculate the adversarial loss and classification loss. In the training of generator , the generated should be real enough to deceive discriminator , so the adversarial loss is calculated as:

[0062]

[0063] For and generator , similar operation is carried out to obtain , the total ontology loss function calculation formula of generator is ;

[0064] (3) for the calculation of classification loss, it is known that the input picture label is , the optimization goal of generator is to generate pseudo P57 image which can be classified to the correct pathological category by discriminator . The definition of classification loss function is:

[0065] ​​

[0066] For generator G , the similar operations are performed to obtain , and the generator total ontology loss function is calculated as ;

[0067] (4) The pseudo P57 staining picture is input into the generator , to generate the reconstructed HE image , and the L1 loss is calculated as the cycle loss between the reconstructed HE image and the original HE image . The cycle loss aims to suppress the mode collapse problem, and hopes that the generated image maintains similar morphological structures as the original image, and only modifies the relevant part of the staining domain. The cycle loss is defined as:

[0068]

[0069] For generator G , the similar operations are performed to obtain , and the generator total ontology loss function is calculated as ;

[0070] The overall loss function of the two generators consists of four parts, namely

[0071]

[0072] wherein , , are hyperparameters, which respectively control the relative importance of the classification loss, the reconstruction loss and the ontology loss compared with the adversarial loss.

[0073] Figure 4 The loss function flowchart for the discriminator training in the virtual staining task. When the discriminator parameters are updated, the generator parameters are in a frozen state, and the optimization goal of the discriminator is to correctly judge the authenticity of the input image and its corresponding pathological category, that is, compared with the adversarial loss and the classification loss part of the generator, the loss calculation of the real image and the true and false label, and the pathological label. For the input HE image , the adversarial loss is calculated as follows:

[0074]

[0075] For and , the similar operations are performed to obtain , and the total adversarial loss function of the discriminator is ​The total loss function of the discriminator is calculated in the same way as follows.

[0076] The classification loss function is calculated as follows:

[0077]

[0078] For and , the total classification loss function of the discriminator is ,

[0079] The overall discriminator loss function is composed of two parts, i.e.

[0080]

[0081] wherein is an adjustment parameter, and the overall loss function is composed of the discriminator loss and the generator loss

[0082]

[0083] After the virtual staining model completes the preset training rounds or reaches the stop training standard, the generator part is deployed for inference application according to actual use requirements, and a target domain virtual staining image is output.

[0084] In the embodiments, Figure 5 The embodiments are compared with the conventional staining transfer method CycleGAN. Since there are no paired images in the data set, the P57 staining images given are the staining results of the same subject slice. It can be seen from the figure that both the CycleGAN method and the method of the present application can generate images similar in P57 staining style, but the embodiments can further maintain the P57 positive and negative features of the corresponding images. For example, the P57 negative images in the first two rows, the trophoblast layer of the generated images in the embodiments is not colored black-brown, which shows the effectiveness of the method of the present application.

[0085] The specific embodiments described above have explained the technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modifications, supplements and equivalent replacements made within the principle range of the present application shall be included in the protection scope of the present application.​​

Claims

1. A pathological image virtual staining method based on multi-task learning, characterized in that, The method comprises the following steps: A pair of virtual staining models are constructed, each of which comprises a generator and a discriminator, one virtual staining model corresponding to a virtual staining task from a source domain to a target domain and a pathological image classification task, and the other virtual staining model corresponding to a virtual staining task from the target domain to the source domain and a pathological image classification task; The source domain images and the target domain images of the same pathological category are input, and the generator and the discriminator are alternately trained, and the model objective function comprises a generator loss function and a discriminator loss function during training, wherein the generator loss function comprises an ontology loss, an adversarial loss, a cycle loss and a classification loss, and the discriminator loss function comprises an additional adversarial loss and an additional classification loss; After training, the source domain images are input into the generator in the virtual staining model corresponding to the source domain to the target domain, and the virtual stained images of the target domain can be generated without the pathological category. Virtual staining model M from source domain to target domain s2t Comprising a target domain generator G s2t And a target domain discriminator D s2t Virtual staining model M from target domain to source domain t2s Comprising a source domain generator G t2s And a source domain discriminator D t2s ; source domain images of pathology class c x s input into the source domain generator G t2s to generate images x s based on the source domain images x s and the generated images x s construct a first ontology loss based on the source domain images of pathology class c x s input into the target domain generator G s2t to generate pseudo images , and assign the pseudo images a first true label, the target domain discriminator D s2t conduct a first true and false discrimination on the input pseudo images and generate a first predicted pathology class, construct a first adversarial loss based on the first true label and the first true and false discrimination result, construct a first classification loss based on the pathology class label c and the first predicted pathology class, input the pseudo images into the source domain generator G t2s to generate reconstructed images based on the source domain images x s and the reconstructed images construct a first cycle loss; target domain image of which the pathology category is c x t input into the target domain generator G s2t to generate an image x t based on the target domain image x t and the generated image x t construct a second ontology loss, input the target domain image of which the pathology category is c x t into the source domain generator G t2s to generate a pseudo image , and give the pseudo image a second true label, the source domain discriminator D t2s conduct a second true and false discrimination on the input pseudo image and generate a second predicted pathology category, construct a second adversarial loss based on the second true label and the second true and false discrimination result, construct a second classification loss based on the pathology category label c and the second predicted pathology category, input the pseudo image into the target domain generator G s2t to generate a reconstructed image based on the target domain image x t and the reconstructed image construct a second cycle loss; Based on all the losses constructed above, the target domain generator G s2t Source domain generator G t2s Training is performed.

2. The method of claim 1, wherein the method is based on multi-task learning. The generator adopts a Unet structure composed of an encoder and a decoder, and a Resnet residual structure is added in the middle to optimize the feature extraction of the encoder.

3. The method of claim 1, wherein the method is based on multi-task learning. The discriminator adopts a stacked convolutional neural network structure, and a fully connected layer is added as a pathological category classification part.

4. The multi-task learning based pathological image virtual staining method of claim 1, wherein, The first ontology loss and the second ontology loss adopt the expected value of the LI norm of two images.

5. The multi-task learning based pathological image virtual staining method of claim 1, wherein, The first cycle loss and the second cycle loss adopt the expected value of the LI norm of two images.

6. The multi-task learning based pathological image virtual staining method of claim 1, wherein, The first adversarial loss adopts the mean square error loss of the gap between the first true and false discrimination result and the first true label. The second adversarial loss adopts the mean square error loss of the gap between the second true and false discrimination result and the second true label.

7. The multi-task learning based pathological image virtual staining method of claim 1, wherein, The first classification loss and the second classification loss both adopt the expected value of the cross-entropy loss of the corresponding predicted pathological category and the pathological label.

8. The multi-task learning based pathological image virtual staining method of claim 1, wherein, Further comprising: source domain image labeled as c for pathology category x s third true label is given and input to source domain discriminator D t2s third true-false discrimination result and third predicted pathology category are obtained, and the third true-false discrimination result is for a false image fourth false label is given and input to source domain discriminator D t2s fourth true-false discrimination result and fourth predicted pathology category are obtained, a third adversarial loss is constructed based on the third true label and the third true-false discrimination result, the fourth false label and the fourth true-false discrimination result, and a third classification loss is constructed based on the pathology category label c and the third predicted pathology category and the fourth predicted pathology category; target domain image labeled with the pathological category label c x t a fifth true label is assigned and input to the target domain discriminator D s2t a fifth true-false discrimination result and a fifth predicted pathological category are obtained, and the image is a fake image a sixth fake label is assigned and input to the source domain discriminator D t2s a sixth true-false discrimination result and a sixth predicted pathological category are obtained, and based on the fifth true label and the fifth true-false discrimination result, the sixth fake label and the sixth true-false discrimination result, a fourth adversarial loss is constructed, and based on the pathological category label c and the fifth predicted pathological category, the sixth predicted pathological category, a fourth classification loss is constructed; The target domain discriminator D is trained based on the third adversarial loss, the third classification loss, the fourth adversarial loss, and the fourth classification loss. s2t and the source domain discriminator D t2s are trained.

9. The method of claim 8, wherein the method is based on multi-task learning. The third adversarial loss adopts the mean square error loss of the gap between the third true and false discrimination result and the third true label and the mean square error loss of the gap between the fourth true and false discrimination result and the fourth false label. The fourth adversarial loss adopts the mean square error loss of the gap between the fifth true and false discrimination result and the fifth true label and the mean square error loss of the gap between the sixth true and false discrimination result and the sixth false label. The third classification loss and the fourth classification loss both adopt the expected value of the cross-entropy loss of the corresponding predicted pathological category and the pathological label.

Images