A prototype multi-task learning method for generating from DAPI to mIHC markers

Abstract

The present invention discloses a prototype multi-task learning method for generating from DAPI to mIHC markers, comprising the following steps: collecting original DAPI and marker data and performing preprocessing; constructing a prototype multi-task network model; designing a loss function; training the prototype multi-task network model; preprocessing test data and inputting the trained network to obtain the generation results of the markers. By adopting the above-mentioned prototype multi-task learning method for generating from DAPI to mIHC markers, the present invention captures the relationships between different virtual staining tasks by learning shared prototypes and task-specific prototypes; in the original attention layer, re-weighting and combining the task-specific prototypes and the shared prototypes to indicate the generation of different mIHC markers; and using the generations of multiple different markers to locate each other to achieve the purpose of multi-task generation, and can efficiently and interpretably generate multiple markers from DAPI staining.

Description

Technical Field

[0001] The present invention relates to the technical field of histopathology, and in particular to a prototype multi-task learning method for generating from DAPI to mIHC markers. Background Art

[0002] Multiplex immunohistochemistry (mIHC) technology has become a powerful tool for simultaneously detecting multiple markers on a single tissue section while preserving valuable information about their location and interaction. Its unique capabilities have the potential to address key questions regarding the development of various complex diseases, making it particularly valuable in the field of cancer immunotherapy. This technology will bring great convenience and impact to the field of pathology and is therefore also known as "next-generation pathology".

[0003] The distribution of multiple markers enriches the understanding of disease pathophysiology; however, few studies have explored the possible interactions between markers. The combination of protein markers such as DAPI and CD8 provides a comprehensive view, providing information about the spatial distribution of cell nuclei and protein expression within tissues. This close connection between DAPI and other markers is crucial for ensuring accurate and informative analysis of tissue sections. This connection has motivated the generation of virtual staining for other biomarkers in DAPI, thereby revealing the interactions and relationships between DAPI and other markers.

[0004] This concept can be achieved through image translation in the field of computer vision, which aims to transfer an image from a source domain to a target domain, with the goal of initiating this process from DAPI and generating virtual distributions for multiple other biomarkers. The task of generating images for multiple target domains from a single source domain can be accomplished in two ways: one involves multiple iterative applications of image-to-image translation, while another potential solution is to directly generate images for multiple target domains simultaneously. Due to the interaction between DAPI and the markers, a multi-task framework is introduced into the latter. Summary of the Invention

[0005] The objective of the present invention is to provide a prototype multi-task learning method for generating from DAPI to mIHC markers. Aiming at the problems that the existing virtual staining methods in histopathology lack interpretability and a single generation method cannot consider the positional relationship between the generations of different markers, etc., an interpretable and multi-task image translation method is proposed. By learning shared prototypes and task-specific prototypes, the relationships between different virtual staining tasks are captured; then in the original attention layer, both the task-specific prototypes and the shared prototypes will be re-weighted and combined to indicate the generation of different mIHC markers; the generations of multiple different markers are used to locate each other to achieve the purpose of multi-task generation, and it is possible to efficiently and interpretably generate multiple markers from DAPI staining.

[0006] To achieve the above object, the present invention provides a prototype multi-task learning method for generating from DAPI to mIHC markers, including the following steps:

[0007] Step S1, collect original DAPI and marker data, and preprocess the collected data;

[0008] Step S2, construct a prototype multi-task network model, including a multi-task prototype layer, a prototype attention layer, and an image decoder layer;

[0009] Step S3, design a loss function, including a single-task loss , an activation loss , a diversity loss and a generation loss ;

[0010] Step S4, train the prototype multi-task network model;

[0011] Step S5, based on the above steps, preprocess the test data in the same way as in training, and then input it into the trained network to obtain the generation result of the marker.

[0012] Preferably, in step S1, collect original DAPI and marker data, and preprocess the collected data. The specific process is as follows:

[0013] Step S11, collect original DAPI and marker data;

[0014] Step S12, preprocess the collected original DAPI and marker data. The specific steps include:

[0015] Step S121, for an image , its mean and variance are calculated as follows:

[0016] ;

[0017] ;

[0018] Among them, represents the pixel value of image I; represents the number of pixel points of the image;

[0019] Step S122, let and be the mean and variance of the DAPI image respectively, and They are the mean and variance of the marker image respectively, and the normalization ratio is calculated as follows:

[0020] ;

[0021] ;

[0022] Among them, represents the normalization ratio; represents the overall translation correction amount after the linear normalization of the marker image;

[0023] Step S123: For each marker image M, calculate its normalized mean and variance, and unify the marker image in terms of mean and variance to the reference space of the DAPI image as follows:

[0024] ;

[0025] Among them, means that the pixel value distribution of the marker image is linearly transformed to have the same mean and standard deviation as the DAPI image.

[0026] Preferably, in step S11, the original DAPI and marker data are collected, and the specific experimental process includes:

[0027] (1) mIHC experiment: Use IHC to analyze the positive conditions of CD8, CD45RO, CD68, and vimentin in patient samples;

[0028] (2) Deparaffinization: Put the sections into xylene for 10 minutes, repeat 2 times, then sequentially put them into 95% alcohol, 85% alcohol, and 75% alcohol for 5 minutes, and then wash them with phosphate buffer PBS;

[0029] (3) Antigen repair: Boil the tissue sections in antigen repair buffer for 10 minutes and keep warm for 30 minutes; After cooling at room temperature for 30 minutes, take out the slides and wash them 3 times with PBS, 3 minutes each time;

[0030] (4) Endogenous peroxidase blocking: Put the slides into 3% hydrogen peroxide solution and incubate them at room temperature in the dark for 10 minutes, then wash them 3 times with PBS;

[0031] (5) Blocking: Drop Super Blocker on the sections and incubate them at room temperature for 30 minutes;

[0032] Among them, primary antibody incubation: Drop the primary antibody diluted in proportion with the primary antibody diluent on the tissue; Incubate the sections overnight at 4°C in a humidified chamber;

[0033] Secondary antibody incubation: After washing three times with PBS, add a host-specific secondary antibody solution of the primary antibody dropwise to cover the tissue and incubate for 30 min at room temperature;

[0034] Finally, develop the color with DAB chromogenic solution, obtain the corresponding mIHC image using a tool, and extract the images of its different channels.

[0035] Preferably, in step S2, construct a prototype multi-task network model, including a multi-task prototype layer, a prototype attention layer, and an image decoder layer;

[0036] In the prototype multi-task network model, use the pre-trained model ResNet-50 as the image encoder , input the enhanced image into the encoder to obtain the feature representation of the image, as shown below:

[0037] ;

[0038] Among them, represents the image feature representation; represents the enhanced image.

[0039] Preferably, the specific process of constructing the multi-task prototype layer is as follows:

[0040] Let the input image be , and extract its feature map through the encoder layer as , where and represent the height and width of the feature map;

[0041] Divide the prototypes of different tasks into task-specific prototypes and shared prototypes; the task-specific prototypes are independently learned in each task to capture task-specific features; the shared prototypes are used to learn the global representation across different tasks;

[0042] Utilize the correlation between different tasks to construct a multi-task prototype layer, which simultaneously learns task-specific activation maps and shared activation maps;

[0043] First, construct learnable task-specific prototypes for each task; then, calculate the similarity between these prototypes and each patch in the feature map obtained through the image encoder to obtain the corresponding activation map; specifically, each position (i, j) of the k-th activation map represents the similarity score between the patch at the (i, j) position in the feature map and the k-th prototype;

[0044] Among them, for task k, define m task-specific prototypes and n shared prototypes, and their shapes are , as shown below:

[0045] ;

[0046] ;

[0047] Among them, represents the specific prototype of task k; represents the shared prototype of task k; .

[0048] Preferably, the specific process of constructing the prototype attention layer is as follows:

[0049] For task k, the original attention layer sums all activation maps as follows:

[0050] ;

[0051] Among them, represents element-wise multiplication; represents the i-th specific activation map of task k;

[0052] Assign different weights to different prototype activation maps, where each weight represents the importance of its corresponding prototype activation map for the combination, as follows:

[0053] ;

[0054] ;

[0055] Among them, represents the new activation map obtained by weighting and summing all prototype activation maps; represents feature extraction; represents the weight of the i-th specific prototype activation map of task k; represents the weight of the j-th shared activation map of task k; represents the j-th shared activation map of task k.

[0056] Preferably, in order to calculate the weights of different activation maps , a prototype channel attention module PCAM is designed; the PCAM module first compresses the global spatial information of the prototype activation map into a prototype channel vector , as follows:

[0057] ;

[0058] Then, an MLP layer with a gated sigmoid function is applied to capture the dependencies between different activation maps, and the weights of task k are obtained, as follows:

[0059] ;

[0060] Among them, represents the Sigmoid gating function; represents the weight vector of the prototype activation map for task k.

[0061] Preferably, an image decoder layer is constructed. For each virtual staining task, a decoder layer is implemented to generate each mIHC label; according to the obtained prototype attention indication feature map, the target image is further obtained through the image decoder.

[0062] Among them, the architectures of the image encoder and the image decoder follow the U-Net construction, and the encoder information is transmitted to the decoder using skip.

[0063] Preferably, in step S3, a loss function is designed, including the following steps:

[0064] (1) When calculating the similarity score between a patch and the corresponding prototype, encourage each training image to have at least one patch close to the current prototype as at least one highly activated point in the activation map; to avoid all points being highly activated, at least one patch is kept away from the prototype component; thus, an activation loss is proposed , as follows:

[0065] ;

[0066] Among them, represents the features of all patches in the feature map of the input image j, corresponding to the maximum activation score on the prototype activation map; represents a small positive smoothing term used to prevent the denominator from being zero;

[0067] (2) In the virtual staining task, in order to focus the activation map on different parts of the image, a diversity loss between different prototypes is proposed , that is, calculate the pairwise cosine similarity between different activation maps, as follows:

[0068] ;

[0069] Among them, and respectively represent the i-th and j-th prototype activation maps;

[0070] (3) Use the L1 loss to perform the image generation task. For task k, the generation loss , as follows:

[0071] ;

[0072] Among them, represents the output image generated by task k;

[0073] (4) The total loss function of multi-task virtual staining consists of a single-task loss , an activation loss , and a diversity loss , and is composed as follows:

[0074] ;

[0075] Among them, represents the total loss; represents the weight coefficient of the single-task loss ; represents the weight coefficient of the activation loss ; represents the weight coefficient of the diversity loss .

[0076] Preferably, in step S4, the prototype multi-task network model is trained to minimize the loss function. A network with a U-Net structure is used, and the backbone part of its encoder adopts ResNet50. The specific process is as follows:

[0077] Step S41, optimizer selection: The stochastic gradient descent (SGD) optimizer is selected, and the default hyperparameter settings are adopted;

[0078] Step S42, learning rate and weight decay settings: The learning rate is set to 0.01, and the weight decay is set to 0.0001 to control the size of the model parameters;

[0079] Step S43, backpropagation and gradient update: The gradient of the loss function with respect to the model parameters is calculated using the backpropagation algorithm; then, according to the rules of the SGD optimizer, the model parameters are updated to minimize the loss function;

[0080] Step S44, iterative training: Through multiple iterations of training, the model optimizes the parameters and improves the performance on the training data;

[0081] Step S45, hyperparameter tuning: During the training process, hyperparameters such as the learning rate are adjusted according to the performance of the validation set to ensure the generalization performance of the model on the training and validation sets;

[0082] Step S46, model saving: The weights of the model are saved regularly so that the model can be reloaded at any time during the training process;

[0083] Based on the above training process, the model learns task-specific feature representations, realizes effective encoding and normalization of DAPI images and marker images, and the ability to perform multi-task learning.

[0084] Therefore, the present invention adopts the above-mentioned prototype multi-task learning method for generating from DAPI to mIHC markers, captures the relationships between different virtual staining tasks by learning shared prototypes and specific-task prototypes; then in the original attention layer, both the specific-task prototypes and the shared prototypes will be re-weighted and combined to indicate the generation of different mIHC markers; the generation of multiple different markers is used to locate each other to achieve the purpose of multi-task generation, and it is possible to efficiently and interpretably generate multiple markers from DAPI staining.

[0085] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 is a flowchart of a prototype multi-task learning method for generating from DAPI to mIHC markers according to the present invention;

[0087] Figure 2 is a schematic diagram of the network structure of a prototype multi-task learning method for generating from DAPI to mIHC markers according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0088] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0089] As Figure 1 shown, a prototype multi-task learning method for generating from DAPI to mIHC markers includes the following steps:

[0090] Step S1, collect the original DAPI and marker data, and preprocess the collected data;

[0091] Step S2, construct a prototype multi-task network model, including a multi-task prototype layer, a prototype attention layer, and an image decoder layer;

[0092] Step S3, design a loss function, including a single-task loss , an activation loss , a diversity loss , and a generation loss ;

[0093] Step S4, train the prototype multi-task network model;

[0094] Step S5, based on the above steps, preprocess the test data in the same way as in the training, and then input it into the trained network to obtain the generation result of the marker.

[0095] Embodiment

[0096] Step S1. Collect the original DAPI and marker data and preprocess the collected data.

[0097] Step S11. Collection of the original DAPI and marker data.

[0098] (1) mIHC experiment: Use IHC to analyze the positive conditions of CD8, CD45RO, CD68, and vimentin in patient samples.

[0099] (2) Deparaffinization: Place the sections in xylene for 10 minutes, repeat 2 times, then sequentially place them in 95% alcohol, 85% alcohol, and 75% alcohol for 5 minutes, and then wash with phosphate buffer (PBS).

[0100] (3) Antigen retrieval: Boil the tissue sections in antigen retrieval buffer for 10 minutes and keep warm for 30 minutes. After cooling at room temperature for 30 minutes, take out the slides and wash them 3 times with PBS, 3 minutes each time.

[0101] (4) Endogenous peroxidase blocking: Place the slides in 3% hydrogen peroxide solution and incubate at room temperature in the dark for 10 minutes, then wash 3 times with PBS.

[0102] (5) Blocking: Drop Super Blocker on the sections and incubate at room temperature for 30 minutes.

[0103] Among them, primary antibody incubation: Drop the primary antibody diluted in proportion with the primary antibody diluent on the tissue. Incubate the sections overnight at 4°C in a humid chamber; secondary antibody incubation: After washing 3 times with PBS, drop the primary antibody host-specific secondary antibody solution to cover the tissue and incubate at room temperature for 30 min; finally, develop the color with DAB chromogenic solution, obtain the corresponding mIHC image using relevant tools, and extract the images of its different channels.

[0104] In this embodiment, a multiplex immunohistochemistry (mIHC) dataset containing colon cancer and liver cancer tissues is utilized. The original images contain DAPI, CD8, CD45RO, CD68, and Vimentin channels. Use DAPI as the source image to generate other channels, thus forming a multi-task method involving four subtasks, including CD8, CD45RO, CD68, and Vimentin. In the experiment, 4 regions of interest (ROIs) were selected for training and 1 ROI was reserved for testing. The resolution of the images at 40x magnification is 0.24 microns per pixel, and ROI images with a size of 256 pixels and a step size of 128 pixels are extracted, obtaining 3325 training data samples and 765 test data samples for the colon tissue dataset. In the liver tissue dataset, 1654 training images and 559 test images were obtained respectively.

[0105] Step S12: Preprocess the collected original DAPI and marker data;

[0106] In the data preprocessing stage, consider normalizing the mean and variance of the marker image to the mean and variance of the DAPI image, and adjust the statistical characteristics of the marker image to be consistent with the DAPI image to achieve better data consistency and stability.

[0107] Step S121: For an image , its mean and variance are calculated as follows:

[0108] ;

[0109] ;

[0110] where represents the pixel value of image I; represents the number of pixel points of the image.

[0111] Step S122: Let and be the mean and variance of the DAPI image respectively, and and be the mean and variance of the marker image respectively. Calculate the normalization ratio as follows:

[0112] ;

[0113] ;

[0114] where represents the normalization ratio; represents the overall translation correction amount after linear normalization of the marker image.

[0115] Step S123: For each marker image M, calculate its normalized mean and variance, and unify the marker image in terms of mean and variance to the reference space of the DAPI image as follows:

[0116] ;

[0117] where indicates that the pixel value distribution of the marker image is linearly transformed to have the same mean and standard deviation as the DAPI image.

[0118] Step S2: Construct a prototype multi-task network model as Figure 2 shown.

[0119] Step S21: Use the pre-trained model ResNet-50 as the image encoder , and the residual structure of ResNet-50 helps to learn more complex feature representations. Therefore, the enhanced image is input into the encoder to obtain the feature representation of the image, as follows:

[0120] ;

[0121] where, represents the feature representation of the image; represents the enhanced image.

[0122] Step S22: Construct a multi-task prototype layer.

[0123] Let the input image , and its feature map is extracted through the encoder layer as , where and represent the height and width of the feature map. In the multi-task learning scenario, the correlation between different tasks is utilized to construct a multi-task prototype layer, which can simultaneously learn task-specific activation maps and shared activation maps. To this end, the prototypes of different tasks are divided into task-specific prototypes and shared prototypes. Task-specific prototypes are independently learned for each task and can capture task-specific features; shared prototypes are dedicated to learning global representations across different tasks.

[0124] First, construct learnable task-specific prototypes for each task. Then, calculate the similarity between these prototypes and each patch in the feature map obtained through the image encoder to obtain the corresponding activation maps. Specifically, each position (i, j) of the k-th activation map represents the similarity score between the patch at the (i, j) position in the feature map and the k-th prototype.

[0125] where, for task k, m task-specific prototypes and n shared prototypes are defined, and their shapes are , as follows:

[0126] ;

[0127] ;

[0128] where, represents the task-specific prototype of task k; represents the shared prototype of task k; . In this embodiment, is set.

[0129] Step S23: Construct a prototype attention layer.

[0130] For task k, the original attention layer sums up all the activation maps as follows:

[0131] ;

[0132] where, represents the element-wise product; represents the i-th specific activation map of task k.

[0133] However, each activation map may contribute different weights to the final attention map. Therefore, a weighted sum strategy is proposed to assign different weights to different prototype activation maps, where each weight represents the importance of its corresponding prototype activation map for the combination, as follows:

[0134] ;

[0135] ;

[0136] where, represents the new activation map obtained by weighting and summing all prototype (specific + shared) activation maps; represents the extracted features; represents the weight of the i-th specific prototype activation map of task k; represents the weight of the j-th shared activation map of task k; represents the j-th shared activation map of task k.

[0137] To calculate the weights of different activation maps , a prototype channel attention module (PCAM) is designed. The PCAM module first compresses the global spatial information of the prototype activation map into a prototype channel vector , as follows:

[0138] ;

[0139] Then, an MLP layer with a gated sigmoid function is applied to capture the dependencies between different activation maps, and the weights of task k are obtained, as follows:

[0140] ;

[0141] where, represents the Sigmoid gating function; represents the weight vector of each prototype activation map of task k.

[0142] Step S24, construct an image decoder layer.

[0143] Finally, for each virtual staining task, a decoder layer is implemented to generate each mIHC marker. Based on the prototype attention indication feature map obtained in the previous step, the target image is then obtained through the image decoder. Among them, the architectures of the image encoder and the image decoder follow the U-Net construction, and skip is used to transfer the encoder information to the decoder.

[0144] Step S3: Design a loss function.

[0145] Since the similarity between patches and prototypes may lead to insignificant activation maps and the problem of failing to capture the activated regions; and there is also the problem that if the prototypes lack diversity and do not show obvious differences, they may not be able to fully capture or learn the prototype features of the image. In order to obtain a better prototype representation for image generation, the present invention designs a loss function to overcome the above problems and defects.

[0146] (1) When calculating the similarity score between a patch and the corresponding prototype, it is encouraged that each training image has at least one patch close to the current prototype as at least one highly activated point in the activation map. To avoid all points being highly activated, at least one patch is kept away from the prototype component. Therefore, the activation loss is proposed , as follows:

[0147] ;

[0148] Among them, represents the features of all patches in the feature map of the input image, corresponding to the maximum activation score on the prototype activation map; represents a small positive smoothing term used to prevent the denominator from being zero.

[0149] (2) In the virtual staining task, it is desired that the activation map can focus on different parts of the image. Therefore, the diversity loss between different prototypes is proposed , that is, the pairwise cosine similarity between different activation maps is calculated, as follows:

[0150] ;

[0151] Among them, and respectively represent the i-th and j-th prototype activation maps. Intuitively, the smaller the value of the diversity loss, the greater the diversity between different activation maps, and vice versa.

[0152] (3) The L1 loss is used to perform the image generation task. For task k, the generation loss , as follows:

[0153] ;

[0154] Among them, represents the output image generated by task k.

[0155] (4)The total loss function of multi-task virtual staining consists of the single-task loss , the activation loss and the diversity loss and is composed as follows:

[0156] ;

[0157] Among them, represents the total loss; represents the weight coefficient of the single-task loss ; represents the weight coefficient of the activation loss ; represents the weight coefficient of the diversity loss .

[0158] Step S4: Train the prototype multi-task network model.

[0159] The network training process is a process in which the deep learning model optimizes parameters to minimize the loss function. Here, a network with a structure similar to U-Net is used, and the backbone of its encoder adopts ResNet50.

[0160] Step S41: Optimizer selection: The stochastic gradient descent (SGD) optimizer is selected, and the default hyperparameter settings are adopted.

[0161] Step S42: Learning rate and weight decay settings: The learning rate is set to 0.01, and the weight decay is set to 0.0001 to control the size of the model parameters.

[0162] Step S43: Backpropagation and gradient update: Use the backpropagation algorithm to calculate the gradient of the loss function with respect to the model parameters. Then, according to the rules of the SGD optimizer, update the model parameters to minimize the loss function.

[0163] Step S44: Iterative training: Through multiple iterative trainings, the model gradually optimizes the parameters and improves the performance on the training data.

[0164] Step S45: Hyperparameter tuning: During the training process, hyperparameters such as the learning rate can be adjusted according to the performance of the validation set to ensure that the model has good generalization performance on both the training and validation sets.

[0165] Step S46: Model saving: Regularly save the weights of the model so that the model can be reloaded at any time during the training process.

[0166] Based on the above training process, the model can gradually learn task-specific feature representations, achieve effective encoding and normalization of DAPI images and marker images, and the ability to perform multi-task learning.

[0167] Step S5: Based on the above steps, preprocess the test data in the same way as in training, and then input it into the trained network to obtain the generation result of the biomarker.

[0168] Therefore, the present invention adopts the above-mentioned prototype multi-task learning method for generating mIHC biomarkers from DAPI, captures the relationship between different virtual staining tasks by learning shared prototypes and task-specific prototypes; then in the original attention layer, both the task-specific prototype and the shared prototype will be reweighted and combined to indicate the generation of different mIHC markers; the generation of multiple different markers is used to locate each other to achieve the purpose of multi-task generation, and it can efficiently and interpretably generate multiple markers from DAPI staining.

[0169] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that: they can still modify or equivalently replace the technical solutions of the present invention, and these modifications or equivalent replacements do not make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A prototype multi-task learning method for generating from DAPI to mIHC markers, characterized in that, It includes the following steps: Step S1: Collect the original DAPI and marker data, and preprocess the collected data; Step S2: Construct a prototype multi-task network model, including a multi-task prototype layer, a prototype attention layer, and an image decoder layer; Among them, the specific process of constructing the prototype attention layer is as follows: For task k, the original attention layer sums all activation maps as follows: ； Among them, represents the element-wise product; represents the i-th specific activation map of task k; Assign different weights to different prototype activation maps, where each weight represents the importance of its corresponding prototype activation map for the combination, as follows: ； ； Among them, represents the new activation map obtained by weighting and summing all prototype activation maps; represents feature extraction; represents the weight of the i-th specific prototype activation map of task k; represents the weight of the j-th shared activation map of task k; represents the j-th shared activation map of task k; To calculate the weights of different activation maps , a prototype channel attention module PCAM is designed; the PCAM module first compresses the global spatial information of the prototype activation map into a prototype channel vector , as follows: ； Then, apply an MLP layer with a gated sigmoid function to capture the dependencies between different activation maps, and the weights for task k are obtained as follows: ； Among them, represents the Sigmoid gating function; represents the weight vector of the activation maps of each prototype for task k; Step S3: Design a loss function, including a single-task loss , an activation loss , a diversity loss and a generation loss ; Step S4: Train the prototype multi-task network model; Step S5: Based on the above steps, preprocess the test data in the same way as in training, and then input it into the trained network to obtain the generation results of the markers.

2. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 1, characterized in that, In step S1, collect the original DAPI and marker data, and preprocess the collected data. The specific process is as follows: Step S11: Collect the original DAPI and marker data; Step S12: Preprocess the collected original DAPI and marker data. The specific steps include: Step S121: For an image , its mean value and variance are calculated as follows: ； ； Among them, represents the pixel value of image I; represents the number of pixel points of the image; Step S122, set and be the mean and variance of the DAPI image respectively, and be the mean and variance of the marker image respectively, and calculate the normalization ratio as follows: ； ； Among them, represents the standardization ratio; represents the overall translation correction amount after the linear standardization of the marker image. Step S123: For each marker image M, calculate its normalized mean and variance, and unify the marker image in terms of mean and variance to the reference space of the DAPI image, as follows: ； Among them, indicates that the pixel value distribution of the marker image is linearly transformed to have the same mean and standard deviation as the DAPI image.

3. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 2, characterized in that: In step S11, collect the original DAPI and marker data. The specific experimental process includes: (1) mIHC experiment: Use IHC to analyze the positive conditions of CD8, CD45RO, CD68, and vimentin in patient samples; (2) Deparaffinization: Place the sections in xylene for 10 minutes, repeat 2 times, then sequentially place them in 95% alcohol, 85% alcohol, and 75% alcohol for 5 minutes, and then wash with phosphate buffer PBS; (3) Antigen repair: Boil the tissue sections in antigen repair buffer for 10 minutes and keep warm for 30 minutes; After cooling at room temperature for 30 minutes, take out the slides and wash them 3 times with PBS, 3 minutes each time; (4) Endogenous peroxidase blocking: Place the slides in 3% hydrogen peroxide solution and incubate at room temperature in the dark for 10 minutes, then wash 3 times with PBS; (5) Blocking: Drop Super Blocker on the sections and incubate at room temperature for 30 minutes; Among them, primary antibody incubation: Drop the primary antibody diluted in proportion with the primary antibody diluent on the tissue; Incubate the sections overnight at 4°C in a humid chamber; Secondary antibody incubation: After washing 3 times with PBS, drop the primary antibody host-specific secondary antibody solution to cover the tissue and incubate at room temperature for 30 min; Finally, develop the color with DAB chromogenic solution, use tools to obtain the corresponding mIHC images, and extract the images of different channels.

4. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 1, characterized in that, In step S2, construct a prototype multi-task network model, including a multi-task prototype layer, a prototype attention layer, and an image decoder layer; In the prototype multi-task network model, the pre-trained model ResNet-50 is used as the image encoder , and the enhanced image is input into the encoder to obtain the feature representation of the image as follows: ； Among them, represents the image feature representation; represents the enhanced image.

5. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 4, characterized in that, The specific process of constructing the multi-task prototype layer is as follows: Assume the input image , and extract its feature map through the encoder layer as , where and represent the height and width of the feature map; The prototypes for different tasks are divided into task-specific prototypes and shared prototypes; task-specific prototypes are independently learned in each task to capture task-specific features; shared prototypes are used to learn the global representation across different tasks; Utilize the correlation between different tasks to construct a multi-task prototype layer, which simultaneously learns task-specific activation maps and shared activation maps; First, construct learnable task-specific prototypes for each task; then, calculate the similarity between these prototypes and each patch in the feature map obtained through the image encoder to obtain the corresponding activation maps; specifically, each position (i, j) in the k-th activation map represents the similarity score between the patch at the (i, j) position in the feature map and the k-th prototype; Among them, for task k, m specific prototypes and n shared prototypes are defined, and their shapes are , as follows: ； ； Among them, represents the specific prototype of task k; represents the shared prototype of task k; .

6. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 4, characterized in that, Construct an image decoder layer. For each virtual staining task, implement a decoder layer to generate each mIHC marker; based on the obtained prototype attention-indicating feature map, further obtain the target image through the image decoder; Among them, the architectures of the image encoder and the image decoder follow the U-Net construction, and skip is used to transfer encoder information to the decoder.

7. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 1, characterized in that, In step S3, design the loss function, including the following steps: When calculating the similarity score between a patch and the corresponding prototype, each training image is encouraged to have at least one patch close to the current prototype as at least one highly activated point in the activation map; to avoid all points being highly activated, at least one patch is kept away from the prototype component; thus, the activation loss is proposed , as follows: ； Among them, represents the features of all patches in the feature map of the input image j, corresponding to the maximum activation score on the prototype activation map; represents a small positive smoothing term used to prevent the denominator from being zero; (2)In the virtual staining task, to focus the activation maps on different parts of the image, a diversity loss between different prototypes is proposed , that is, the pairwise cosine similarity between different activation maps is calculated as follows: ； Among them, and represent the i-th and j-th prototype activation maps, respectively; (3) Use the L1 loss to perform the image generation task. For task k, the generation loss is as follows: ； Among them, represents the output image generated by task k; (4)The total loss function of multi-task virtual staining consists of a single-task loss , an activation loss , and a diversity loss , and is as follows: ； Among them, represents the total loss; represents the weight coefficient of the single task loss ; represents the weight coefficient of the activation loss ; represents the weight coefficient of the diversity loss .

8. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 1, characterized in that In step S4, train the prototype multi-task network model to minimize the loss function. Use a network with a U-Net structure, and the backbone part of its encoder adopts ResNet50. The specific process is as follows: Step S41, Optimizer selection: Select the Stochastic Gradient Descent (SGD) optimizer and adopt the default hyperparameter settings; Step S42, Learning rate and weight decay settings: The learning rate is set to 0.01, and the weight decay is set to 0.0001 to control the size of the model parameters; Step S43, Backpropagation and gradient update: Use the backpropagation algorithm to calculate the gradient of the loss function with respect to the model parameters; then, according to the rules of the SGD optimizer, update the model parameters to minimize the loss function; Step S44, Iterative training: Through multiple iterations of training, the model optimizes the parameters and improves the performance on the training data; Step S45, Hyperparameter tuning: During the training process, adjust the learning rate hyperparameter according to the performance of the validation set to ensure the generalization performance of the model on the training and validation sets; Step S46, Model saving: Regularly save the weights of the model so that the model can be reloaded during the training process; Based on the above training process, the model learns task-specific feature representations, realizes the effective encoding and normalization of DAPI images and marker images, and the ability to perform multi-task learning.

Images