Pathological image virtual dyeing method based on multi-task learning

Abstract

The invention discloses a pathological image virtual dyeing method based on multi-task learning, and the method comprises the steps: constructing a pair of virtual dyeing models which respectively comprise a generator and a discriminator corresponding to a source domain and a target domain; carrying out self-supervised cross-dyeing domain reconstruction and conversion on the generator; for the discriminator, adding pathological section category supervision loss in self-supervision adversarial loss; and after alternate training and updating, image conversion between the source domain and the target domain and a corresponding pathological classification task are finally realized. According to the method, a non-paired staining data set is used, classification loss is introduced through pathological category labels, objective function design is optimized, a classification module is added, a gradient guide model is returned to learn inter-class difference staining features, and the generation effect of virtual staining of the pathological image is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field involving the combination of computer-aided diagnosis, image processing, and deep learning, and specifically relates to a virtual staining method for pathological images based on multi-task learning. Background Art

[0002] As the gold standard for tumor diagnosis, pathologists perform pathological examinations under an optical microscope and complete the diagnosis and grading of tumors by observing the morphology and cell characteristics of tumor tissues after staining and preparation. Hematoxylin-eosin (HE) is the most commonly used stain in pathological diagnosis, focusing on showing the tissue structure of cells. However, HE staining does not have all the information needed to evaluate the grading of the disease. For different diseases, pathologists still need different immunohistochemistry (IHC) stains to analyze specific structures or abnormal substances. Common immunohistochemical stains, such as P57, can better stain the trophoblast layer; Ki67 shows specific expression of the cell nucleus and is often used in cervical cancer screening. However, the preparation of such special stains is time-consuming and laborious, and specific diseases require different IHC reagents, which is not feasible for promotion in downstream hospitals.

[0003] With the emergence of digital scanners and deep learning, the diagnosis of pathological sections has gradually changed from microscopic examination to digital section reading. As a new era of computer-aided diagnosis, models based on deep learning methods can generate different virtual staining effects on a single stained section, which to a certain extent solves the limitations of traditional pathological section staining methods and improves pathological diagnosis results. The trained deep learning model can also provide a consistent staining style, reducing the impact of changes in staining reagents in different centers on film reading.

[0004] Conditional Generative Adversarial Network (Pix2Pix) and Cycle Generative Adversarial Network (CycleGAN) are two commonly used algorithm frameworks for image style transfer. The performance of the Pix2Pix-based method depends largely on a large number of pixel-level paired training data sets. However, tissue loss and movement often occur during the pathological staining process, and it takes a lot of manpower to annotate the paired images required for training. It is difficult to achieve accurate pixel-level correspondence, which makes it easy for subsequent models to produce artifacts and affects the quality of virtual staining.

[0005] CycleGAN does not require paired datasets to complete the conversion between two image styles, so the method is sensitive to the domain distribution of the converted image. For some immunohistochemical staining methods, images between different pathological categories will show obvious yin-yang differences, that is, there is category-specific information. Using the conventional CycleGAN method, simply classifying IHC staining into a staining domain for staining transfer is not effective. If the categories are further subdivided for inter-class conversion, on the one hand, it is necessary to train multiple groups of CycleGAN models, which increases the cost of data collection. More importantly, the staining task itself is to better classify pathology. If the virtual staining model is trained and inferred according to the pathological category, it has no practical significance and deviates from the original intention of the virtual staining task. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide a method and device for virtual staining of pathological images based on multi-task learning, which aims to jointly supervise model training by combining virtual staining and classification tasks, add pathological diagnosis level categories instead of image histology categories to the original unpaired training data set, and do not increase the difficulty of data set annotation. By optimizing the model structure design, adding a category classification module, introducing pathological diagnosis level category loss, and adjusting the loss function design, the model is guided to learn the difference characteristics between pathological image classes, thereby improving the generation effect of virtual staining of pathological images.

[0007] To achieve the above-mentioned purpose of the invention, an embodiment provides a pathological image virtual staining method based on multi-task learning, comprising the following steps:

[0008] A pair of virtual staining models is constructed, each of which includes a generator and a discriminator. One virtual staining model corresponds to the virtual staining task from the source domain to the target domain and the pathological image classification task, and the other virtual staining model corresponds to the virtual staining task from the target domain to the source domain and the pathological image classification task.

[0009] Input source domain images and target domain images of the same pathological category, alternately train the generator and the discriminator, and the model objective function during training includes a generator loss function and a discriminator loss function, wherein the generator loss function includes ontology loss, adversarial loss, cycle loss and classification loss, and the discriminator loss function includes another adversarial loss and another classification loss;

[0010] After the training is completed, the source domain image is input into the generator contained in the virtual staining model corresponding to the source domain to the target domain, and the virtual staining image of the target domain can be generated without the need for pathological classification.

[0011] Preferably, the generator adopts a Unet structure consisting of an encoder and a decoder, and adds a Resnet residual structure in the middle to optimize the feature extraction of the encoder.

[0012] Preferably, the discriminator adopts a stacked convolutional neural network structure, adding a fully connected layer as a pathology category classification part.

[0013] Preferably, the virtual coloring model M from the source domain to the target domain s2t Contains the target domain generator G s2t and the target domain discriminator D s2t , the virtual coloring model M from the target domain to the source domain t2s Contains the source domain generator G t2s and source domain discriminator D t2s ;

[0014] The source domain image x with pathological category c s Input to the source domain generator G t2s The generated image x is obtained s ', based on the source domain image x s and generate image x s 'Construct the first ontology loss, and convert the source domain image x with pathological category c s Input to the target domain generator G s2t Generate fake images in And for the pseudo image Assign the first true label, the target domain discriminator D s2t For the input fake image Perform the first true-false discrimination and generate the first predicted pathological category, construct the first adversarial loss based on the first true label and the first true-false discrimination result, and construct the first classification loss based on the pathological category label c and the first predicted pathological category to generate a pseudo image Input to the source domain generator G t2s Generate the reconstructed image Based on the source domain image x s and reconstruct the image Construct the first cycle loss;

[0015] The target domain image x with pathological category c t Input to the target domain generator G s2t The generated image x is obtained t ', based on the target domain image x t and generate image x t 'Construct the second ontology loss and transform the target domain image x with pathological category c t Input to the source domain generator G t2s Generate fake images in And for the pseudo image Assign the second true label, source domain discriminator D t2s For the input fake image Perform a second true-false discrimination and generate a second predicted pathology category, construct a second adversarial loss based on the second true label and the second true-false discrimination result, and construct a second classification loss based on the pathology category label c and the second predicted pathology category to generate a pseudo image Input to the target domain generator G s2t Generate the reconstructed image Based on the target domain image x t and reconstruct the image Construct the second loop loss;

[0016] Based on all the losses constructed above, the target domain generator G s2t Source domain generator G t2s Conduct training.

[0017] Preferably, the first ontology loss and the second ontology loss adopt the expected value of the LI norm of the two images.

[0018] Preferably, the first loop loss and the second loop loss adopt the expected value of the LI norm of the two images.

[0019] Preferably, the first adversarial loss adopts the mean square error loss of the difference between the first true and false discrimination result and the first true label;

[0020] The second adversarial loss uses the mean square error loss of the difference between the second true and false discrimination result and the second true label.

[0021] Preferably, the first classification loss and the second classification loss both adopt expected values ​​of the cross entropy losses corresponding to the predicted pathology category and the pathology label.

[0022] is the source domain image x with pathological category label c s Assign the third true label and input it to the source domain discriminator D t2s The third true-false discrimination result and the third predicted pathology category are obtained, which is a pseudo image. Assign the fourth pseudo label and input it into the source domain discriminator D t2s Obtain a fourth true-false discrimination result and a fourth predicted pathological category, construct a third adversarial loss based on the third true label and the third true-false discrimination result, the fourth pseudo label and the fourth true-false discrimination result, and construct a third classification loss based on the pathological category label c and the third predicted pathological category and the fourth predicted pathological category;

[0023] is the target domain image x with pathological category label c t Assign the fifth true label and input it to the target domain discriminator D s2t The fifth true-false discrimination result and the fifth predicted pathology category are obtained, which is a pseudo image. Assign the sixth pseudo label and input it into the source domain discriminator D t2sObtain the sixth true-false discrimination result and the sixth predicted pathological category, construct a fourth adversarial loss based on the fifth true label and the fifth true-false discrimination result, the sixth false label and the sixth true-false discrimination result, and construct a fourth classification loss based on the pathological category label c and the fifth predicted pathological category and the sixth predicted pathological category;

[0024] Based on the third adversarial loss, the third classification loss, the fourth adversarial loss, and the fourth classification loss, the target domain discriminator D s2t and source domain discriminator D t2s Conduct training.

[0025] Preferably, the third adversarial loss adopts the mean square error loss of the gap between the third true and false discrimination result and the third true label and the mean square error loss of the gap between the fourth true and false discrimination result and the fourth pseudo label;

[0026] The fourth adversarial loss adopts the mean square error loss of the gap between the fifth true and false discrimination result and the fifth true label and the mean square error loss of the gap between the sixth true and false discrimination result and the sixth pseudo label.

[0027] Preferably, the third classification loss and the fourth classification loss both adopt the expected value of the cross entropy loss corresponding to the predicted pathology category and the pathology label.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The virtual staining method for pathological images based on multi-task learning provided by the present invention adds category supervision during model training to guide the generator to learn staining features of different categories. Due to the hidden affinity between classification and virtual staining tasks, the virtual staining method does not simply imitate the statistical density of the color space, but learns the distinguishing features between different pathological categories and generates corresponding staining patterns, thereby improving the quality of virtual staining. It is worth noting that the pathological category labels of the blocks used for training in the present invention are derived from the diagnostic results of the entire pathological section, rather than the histological classification labels of specific images. No additional annotation is required, so it can replace the current virtual staining model and has practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 is a flowchart of a pathological image virtual staining method based on multi-task learning provided in an embodiment;

[0032] Figure 2 is a schematic diagram of a virtual staining framework of pathological images based on multi-task learning provided by an embodiment;

[0033] Figure 3 is a flowchart of generator training provided by an embodiment;

[0034] Figure 4 is a flowchart of discriminator training provided by an embodiment;

[0035] Figure 5 It is a comparison chart of the P57 virtual staining generation results provided in the embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0037] like Figure 1 As shown, the embodiment provides a pathological image virtual staining method based on multi-task learning, comprising the following steps:

[0038] S1, build a pair of virtual coloring models, each of which includes a generator G and a discriminator D.

[0039] In the embodiment, a virtual dyeing model M is constructed s2t Corresponding to the virtual staining task from the source domain to the target domain and the pathological image classification task, it includes the target domain generator G s2t and the target domain discriminator D s2t , and construct another virtual dyeing model M t2s The corresponding target domain to source domain virtual staining task and pathological image classification task, which includes the source domain generator G t2s and source domain discriminator D t2s .

[0040] Each generator uses a Unet structure consisting of an encoder and a decoder, and adds a Resnet residual structure in the middle to optimize the encoder's feature extraction. Each discriminator uses a stacked convolutional neural network structure and adds a fully connected layer as the pathology category classification part to control its convergence speed and better guide gradient conduction.

[0041] S2, input source domain images and target domain images of the same pathology category, alternately train the generator and discriminator. During the training process, the pathology category c is the corresponding image diagnosis label. The model objective function during training includes the generator loss function and the discriminator loss function, where the generator loss function includes ontology loss, adversarial loss, cycle loss and classification loss, and the discriminator loss function includes additional adversarial loss and additional classification loss.

[0042] During training, Figure 2 As shown, the generator takes a real image as input and generates a pseudo image without specifying the pathology category; the discriminator learns to distinguish between real and pseudo images and tries to classify the images into their corresponding pathology categories; the generator reconstructs the image from the pseudo image, expecting to obtain the same result as the real image.

[0043] Specifically, the source domain image x with pathology category c is s Input to the source domain generator G t2s The generated image x is obtained s ', based on the source domain image x s and generate image x s 'Construct the first ontology loss, and convert the source domain image x with pathological category c s Input to the target domain generator G s2t Generate fake images in And for the pseudo image Assign the first true label, the target domain discriminator D s2t For the input fake image Perform the first true-false discrimination and generate the first predicted pathological category, construct the first adversarial loss based on the first true label and the first true-false discrimination result, and construct the first classification loss based on the pathological category label c and the first predicted pathological category to generate a pseudo image Input to the source domain generator G t2s Generate the reconstructed image Based on the source domain image x s and reconstruct the image Construct the first cycle loss.

[0044] The target domain image x with pathological category c t Input to the target domain generator G s2t The generated image x is obtained t ', based on the target domain image x t and generate image x t 'Construct the second ontology loss and transform the target domain image x with pathological category c t Input to the source domain generator G t2s Generate fake images in And for the pseudo image Assign the second true label, source domain discriminator Dt2s For the input fake image Perform a second true-false discrimination and generate a second predicted pathology category, construct a second adversarial loss based on the second true label and the second true-false discrimination result, and construct a second classification loss based on the pathology category c and the second predicted pathology category to generate a pseudo image Input to the target domain generator G s2t Generate the reconstructed image Based on the target domain image x t and reconstruct the image Construct the second loop loss;

[0045] By weighting all the losses constructed above, the target domain generator G s2t Source domain generator G t2s Training is performed. The first ontology loss and the second ontology loss use the expected value of the LI norm of the two images. The first cycle loss and the second cycle loss use the expected value of the LI norm of the two images. The first adversarial loss uses the mean square error loss of the difference between the first true-false discrimination result and the first true label; the second adversarial loss uses the mean square error loss of the difference between the second true-false discrimination result and the second true label. The first classification loss and the second classification loss both use the expected value of the cross entropy loss.

[0046] When training the discriminator, the preset training label for the generated image is true, which is different from the preset label (false) during generator training. Specifically, the source domain image x with pathology category c s Assign the third true label and input it to the source domain discriminator D t2s Get the third true and false discrimination result and the third predicted pathology class

[0047] No, it is a fake image Assign the fourth pseudo label and input it into the source domain discriminator D t2s Obtain a fourth true-false discrimination result and a fourth predicted pathological category, construct a third adversarial loss based on the third true label and the third true-false discrimination result, the fourth pseudo label and the fourth true-false discrimination result, and construct a third classification loss based on the pathological category label c and the third predicted pathological category and the fourth predicted pathological category;

[0048] is the target domain image x with pathological category c t Assign the fifth true label and input it to the target domain discriminator D s2t The fifth true-false discrimination result and the fifth predicted pathology category are obtained, which is a pseudo image. Assign the sixth pseudo label and input it into the source domain discriminator D t2sObtain the sixth true-false discrimination result and the sixth predicted pathological category, construct a fourth adversarial loss based on the fifth true label and the fifth true-false discrimination result, the sixth false label and the sixth true-false discrimination result, and construct a fourth classification loss based on the pathological category label c and the fifth predicted pathological category and the sixth predicted pathological category;

[0049] The target domain discriminator D is trained by weighting the third adversarial loss, the third classification loss, the fourth adversarial loss, and the fourth classification loss. s2t and source domain discriminator D t2s The third adversarial loss uses the mean square error loss of the gap between the third true and false discrimination result and the third true label, and the mean square error loss of the gap between the fourth true and false discrimination result and the fourth pseudo label; the fourth adversarial loss uses the mean square error loss of the gap between the fifth true and false discrimination result and the fifth true label, and the mean square error loss of the gap between the sixth true and false discrimination result and the sixth pseudo label; the third classification loss and the fourth classification loss both use the expected value of the cross entropy loss of the corresponding predicted pathological category and pathological label.

[0050] Among them, the ontology loss aims to guide the generator to focus on the coloring differences between different conversion domains, distinguish them from the ontology structure of the image, and when the image is input into the generator, it is expected to perform generator ontology mapping, that is, the output image is the same as the input image.

[0051] Adversarial loss is applied to the training of the generator and the discriminator, where the parameter optimization direction of the generator is to minimize the loss, aiming to make the distribution of the generated pseudo images close to that of the real images to deceive the discriminator, while the discriminator attempts to maximize the adversarial loss and judge the authenticity of the input image as much as possible. The two are optimized alternately to achieve the purpose of adversarial improvement.

[0052] The classification loss aims to guide the generator to focus on the coloring differences between different categories. Taking the source domain generator as an example, its optimization goal is to convert x s Convert to pseudo image and The method adds an auxiliary classifier on top of the discriminator. In the embodiment, G s2t Try to minimize the classification loss to generate target domain images that can be classified into class c D t It tries to learn to correctly classify the input image.

[0053] The cycle loss aims to suppress the pattern disappearance problem, and expects that the image after color transfer only changes the part related to the color domain, that is, the generated image maintains a similar morphological structure to the original image. s , the optimization goal of the source domain generator is: the reconstructed image after the secondary transformation With the original image xs Be consistent.

[0054] The generator and discriminator are trained alternately to carry out the training process for the two models. The training of the two models is completed after a given training round is completed or the early stopping strategy is triggered, that is, before obvious artifacts appear in the generated image.

[0055] S3, after the training is completed, the source domain image is input into the generator contained in the virtual staining model corresponding to the source domain to the target domain, and the virtual staining image of the target domain can be generated without the need for pathological classification.

[0056] In actual use, the generator part can be deployed separately for reasoning, and the source domain image can be input to complete the virtual staining task without the need for pathological category labels.

[0057] The following example takes HE dyeing as the source domain and P57 dyeing as the target domain for explanation. s2t and the target domain discriminator D s2t Named G HE_P57 and D P57 , source domain generator G t2s and source domain discriminator D t2s Named G P57_HE and D HE For the convenience of expression, D HE The part that determines the authenticity is recorded as D HE_adv , the part for judging the category is recorded as D HE_cls , D P57 In each round of training, a source domain image x with pathological category c in the dataset is extracted. h , target domain image x p , and perform preprocessing to complete data enhancement (including flipping, rotation, appropriate scaling, etc.).

[0058] During the model training process, the generator loss function is calculated and the corresponding parameters are updated first. Figure 3 This is a flowchart of generator training and loss function calculation in the virtual coloring task. When the generator parameters are updated, the discriminator is in a frozen state and only performs reasoning without updating parameters. The generator loss function calculation process is as follows:

[0059] (1) To the generator G HE_P57 Input P57 staining image x p , output image With the input image x p Calculate the L1 loss as the body loss L idt_HE , guide the generator G HE_P57Focus on the dyeing state of the input image, pay attention to the color distribution differences between different dyeing domains, and distinguish them from the organizational structure characteristics of the image. The formula is:

[0060]

[0061] For x h and the generator G P57_HE Perform similar operations to obtain The calculation formula of the total ontology loss function of the generator is: The calculation method of the total loss function of the corresponding generators below is the same and will not be repeated here.

[0062] (2) To the generator G HE_P57 Input HE staining picture x h , generate pseudo P57 staining images Send to the discriminator D P57 Calculate the adversarial loss and classification loss. In the generator G HE_P57 During training, we hope to generate Real enough to fool the discriminator D P57 , so the adversarial loss is calculated as:

[0063] L adv_HE =MSE(1-D P57_adv (G HE_P57 (x h ))))

[0064] For x P57 and the generator G P57_HE Perform similar operations to obtain L adv_P57 , the total ontology loss function of the generator is calculated as L adv =(L adv_HE +L adv_P57 ) / 2;

[0065] (3) For the calculation of classification loss, the input image label is known to be c, and the generator G HE_P57 The optimization goal is to generate a pseudo P57 image Can be used as a discriminator D P57 Classification to the correct pathological category c. Classification loss function L cls_P57 It is defined as:

[0066]

[0067] For x P57 and the generator G P57_HE Perform similar operations to obtain L cls_HE , the total ontology loss function of the generator is calculated as L cls =(L cls_P57 +L cls_HE) / 2;

[0068] (4) Pseudo-P57 staining images Input generator G P57_HE In the above example, the reconstructed HE image is generated. With the original HE image x h Calculate the L1 loss as the loop loss L rec_HE The cycle loss aims to suppress the pattern disappearance problem, hoping that the generated image maintains a similar morphological structure to the original image and only modifies the relevant parts of the color domain. The cycle loss is defined as:

[0069]

[0070] For the generator G HE_P57 Perform similar operations to obtain L rec_P57 , the total ontology loss function of the generator is calculated as L rec =(L rec_HE +L rec_P57 ) / 2;

[0071] The overall loss function of the two generators consists of four parts, namely

[0072]

[0073] where λ cls ,λ rec ,λ idt are hyperparameters that control the relative importance of classification loss, reconstruction loss, and ontology loss, respectively, compared to the adversarial loss.

[0074] Figure 4 The figure is a flowchart of the discriminator training and loss function in the virtual staining task. When the discriminator parameters are updated, the generator parameters are frozen. The optimization goal of the discriminator is to correctly judge the authenticity of the input image and its corresponding pathological category. That is, compared with the adversarial loss and classification loss of the generator, adding the real image x h The loss calculation of the true and false labels and pathological labels. For the input HE image x h , the adversarial loss is calculated as follows:

[0075]

[0076] x p and G P57_HE Perform similar operations to obtain L adv_P57 ', the total adversarial loss function of the discriminator is L adv '=(L adv_HE '+L adv_P57 ') / 2, the following corresponding discriminator total loss function is calculated in the same way.

[0077] The classification loss function calculation formula is as follows:

[0078]

[0079] For x p and G P57_HE Perform similar operations to obtain L cls_P57 ', the total classification loss function of the discriminator is L cls '=(L cls_HE '+L cls_P57 ') / 2,

[0080] The overall discriminator loss function consists of two parts, namely

[0081] L D =-L adv ′+λ cls L cls '

[0082] Among them, λ cls To improve the adjustment parameters, the overall loss function By and discriminator loss and the generator loss It consists of two parts:

[0083]

[0084] After the virtual dyeing model completes the preset training rounds using the above method or reaches the training stop criteria, it will deploy the generator part for reasoning application according to actual usage requirements and output the target domain virtual dyeing image.

[0085] In an embodiment, Figure 5 The embodiment is compared with the conventional dye transfer method CycleGAN. Since there is no paired image in the data set, the P57 dyeing image given is the dyeing result of the slice of the same subject. As can be seen from the figure, the CycleGAN method and the invention method can both generate images with similar P57 dyeing style, but this embodiment can further maintain the P57 positive and negative characteristics of the corresponding image. For example, in the first two rows of P57 negative images, the trophoblast layer of the image generated by this embodiment is not colored dark brown, which shows the effectiveness of the method of the present invention.

[0086] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A virtual staining method for pathological images based on multi-task learning, characterized in that: The following steps are involved: A pair of virtual staining models is constructed, each of which includes a generator and a discriminator. One virtual staining model corresponds to the virtual staining task from the source domain to the target domain and the pathological image classification task, and the other virtual staining model corresponds to the virtual staining task from the target domain to the source domain and the pathological image classification task. Input source domain images and target domain images of the same pathological category, alternately train the generator and the discriminator, and the model objective function during training includes a generator loss function and a discriminator loss function, wherein the generator loss function includes ontology loss, adversarial loss, cycle loss and classification loss, and the discriminator loss function includes another adversarial loss and another classification loss; After the training is completed, the source domain image is input into the generator contained in the virtual staining model corresponding to the source domain to the target domain, and the virtual staining image of the target domain can be generated without the need for pathological classification.

2. The pathological image virtual staining method based on multi-task learning according to claim 1 is characterized in that: The generator adopts a Unet structure consisting of an encoder and a decoder, and adds a Resnet residual structure in the middle to optimize the feature extraction of the encoder.

3. The pathological image virtual staining method based on multi-task learning according to claim 1 is characterized in that: The discriminator adopts a stacked convolutional neural network structure and adds a fully connected layer as the pathology category classification part.

4. The pathological image virtual staining method based on multi-task learning according to claim 1, characterized in that: Virtual coloring model M for source domain to target domain s2t Contains the target domain generator G s2t and the target domain discriminator D s2t , the virtual coloring model M from the target domain to the source domain t2s Contains the source domain generator G t2s and source domain discriminator D t2s ; The source domain image x with pathological category c s Input to the source domain generator G t2s The generated image x is obtained s ', based on the source domain image x s and generate image x s 'Construct the first ontology loss, and convert the source domain image x with pathological category c s Input to the target domain generator G s2t Generate fake images in And for the pseudo image Assign the first true label, the target domain discriminator D s2t For the input fake image Perform the first true-false discrimination and generate the first predicted pathological category, construct the first adversarial loss based on the first true label and the first true-false discrimination result, and construct the first classification loss based on the pathological category label c and the first predicted pathological category. Input to the source domain generator G t2s Generate the reconstructed image Based on the source domain image x s and reconstruct the image Construct the first cycle loss; The target domain image x with pathological category c t Input to the target domain generator G s2t The generated image x is obtained t ', based on the target domain image x t and generate image x t 'Construct the second ontology loss and transform the target domain image x with pathological category c t Input to the source domain generator G t2s Generate fake images in And for the pseudo image Assign the second true label, source domain discriminator D t2s For the input fake image Perform a second true-false discrimination and generate a second predicted pathology category, construct a second adversarial loss based on the second true label and the second true-false discrimination result, and construct a second classification loss based on the pathology category label c and the second predicted pathology category. Input to the target domain generator G s2t Generate the reconstructed image Based on the target domain image x t and reconstruct the image Construct the second loop loss; Based on all the losses constructed above, the target domain generator G s2t Source domain generator G t2s Conduct training.

5. The method for virtual staining of pathological images based on multi-task learning according to claim 4, characterized in that: The first ontology loss and the second ontology loss use the expected value of the LI norm of the two images.

6. The pathological image virtual staining method based on multi-task learning according to claim 4 is characterized in that: The first loop loss and the second loop loss use the expected value of the LI norm of the two images.

7. The pathological image virtual staining method based on multi-task learning according to claim 4 is characterized in that: The first adversarial loss uses the mean square error loss of the difference between the first true and false discrimination result and the first true label; The second adversarial loss uses the mean square error loss of the difference between the second true and false discrimination result and the second true label.

8. The pathological image virtual staining method based on multi-task learning according to claim 4 is characterized in that: The first classification loss and the second classification loss both use the expected value of the cross entropy loss between the predicted pathological category and the pathological label.

9. The method for virtual staining of pathological images based on multi-task learning according to claim 4, characterized in that: Also includes: is the source domain image x with pathological category label c s Assign the third true label and input it to the source domain discriminator D t2s The third true-false discrimination result and the third predicted pathology category are obtained, which is a pseudo image. Assign the fourth pseudo label and input it into the source domain discriminator D t2s Obtain a fourth true-false discrimination result and a fourth predicted pathological category, construct a third adversarial loss based on the third true label and the third true-false discrimination result, the fourth pseudo label and the fourth true-false discrimination result, and construct a third classification loss based on the pathological category label c and the third predicted pathological category and the fourth predicted pathological category; is the target domain image x with pathological category label c t Assign the fifth true label and input it to the target domain discriminator D s2t The fifth true-false discrimination result and the fifth predicted pathology category are obtained, which is a pseudo image. Assign the sixth pseudo label and input it into the source domain discriminator D t2s Obtain the sixth true-false discrimination result and the sixth predicted pathological category, construct a fourth adversarial loss based on the fifth true label and the fifth true-false discrimination result, the sixth false label and the sixth true-false discrimination result, and construct a fourth classification loss based on the pathological category label c and the fifth predicted pathological category and the sixth predicted pathological category; Based on the third adversarial loss, the third classification loss, the fourth adversarial loss, and the fourth classification loss, the target domain discriminator D s2t and source domain discriminator D t2s Conduct training.

10. The pathological image virtual staining method based on multi-task learning according to claim 9, characterized in that: The third adversarial loss uses the mean square error loss of the gap between the third true and false discrimination result and the third true label and the mean square error loss of the gap between the fourth true and false discrimination result and the fourth pseudo label; The fourth adversarial loss uses the mean square error loss of the gap between the fifth true and false discrimination result and the fifth true label and the mean square error loss of the gap between the sixth true and false discrimination result and the sixth pseudo label; The third classification loss and the fourth classification loss both use the expected value of the cross entropy loss between the predicted pathological category and the pathological label.

Images