Prototype multi-task learning method for generating markers from DAPI to mIHC

Abstract

The invention discloses a prototype multi-task learning method for generating markers from DAPI to mIHC, and the method comprises the following steps: collecting original DAPI and marker data, and carrying out the preprocessing; constructing a prototype multi-task network model; designing a loss function; training the prototype multi-task network model; and preprocessing test data, and inputting the test data into the trained network to obtain a generation result of the marker. According to the prototype multi-task learning method for generating the markers from the DAPI to the mIHC, the relationship between different virtual dyeing tasks is captured by learning a shared prototype and a specific task prototype; in the original attention layer, re-weighting and combining the specific task prototype and the shared prototype to indicate the generation of different mIHC markers; the purpose of multi-task generation is achieved through generation positioning of a plurality of different markers, and various markers can be generated through DAPI dyeing in an efficient and explainable mode.

Description

Technical Field

[0001] The present invention relates to the technical field of histopathology, and in particular, to a prototype multi-task learning method for generating from DAPI to mIHC markers. Background Art

[0002] Multiplex immunohistochemistry (mIHC) technology has become a powerful tool to simultaneously detect multiple markers on a single tissue section while preserving valuable information about their location and interaction. Its unique capabilities have the potential to solve key questions regarding the development of various complex diseases, making it particularly valuable in the field of cancer immunotherapy. This technology will bring great convenience and impact to the field of pathology, and thus is also known as "next-generation pathology".

[0003] The distribution of multiple markers enriches the understanding of disease pathophysiology; however, few works have explored the possible interactions between markers. The combination of protein markers such as DAPI and CD8 provides a comprehensive view, providing information about the spatial distribution of cell nuclei and protein expression within tissues. This close connection between DAPI and other markers is crucial for ensuring accurate and informative analysis of tissue sections. This connection has motivated the generation of virtual staining for other biomarkers in DAPI, thereby revealing the interactions and relationships between DAPI and other markers.

[0004] This concept can be achieved through image translation in the field of computer vision, the aim of which is to transfer an image from a source domain to a target domain, with the goal of initiating this process from DAPI and generating virtual distributions for multiple other biomarkers. The task of generating images for multiple target domains from a single source domain can be accomplished in two ways: one involves multiple iterative applications of image-to-image translation, while another potential solution is to directly generate images for multiple target domains simultaneously. Due to the interaction between DAPI and the markers, a multi-task framework is introduced into the latter. Summary of the Invention

[0005] The object of the present invention is to provide a prototype multi-task learning method for generating from DAPI to mIHC markers. Aiming at the problems that the existing virtual staining methods in histopathology lack interpretability and a single generation method cannot consider the positional relationship between the generations of different markers, etc., an interpretable and multi-task image translation method is proposed. By learning shared prototypes and task-specific prototypes to capture the relationships between different virtual staining tasks; then in the original attention layer, both the task-specific prototypes and the shared prototypes will be re-weighted and combined to indicate the generation of different mIHC markers; the generations of multiple different markers are used to locate each other to achieve the purpose of multi-task generation, and it is possible to efficiently and interpretably generate multiple markers from DAPI staining.

[0006] To achieve the above object, the present invention provides a prototype multi-task learning method for generating from DAPI to mIHC markers, including the following steps: Step S1, collect the original DAPI and marker data, and preprocess the collected data; Step S2, construct a prototype multi-task network model, including a multi-task prototype layer, a prototype attention layer, and an image decoder layer; Step S3, design a loss function, including a single-task loss , an activation loss , a diversity loss and a generation loss ; Step S4, train the prototype multi-task network model; Step S5, based on the above steps, preprocess the test data in the same way as in training, and then input it into the trained network to obtain the generation result of the marker.

[0007] Preferably, in step S1, collect the original DAPI and marker data, and preprocess the collected data. The specific process is as follows: Step S11, collect the original DAPI and marker data; Step S12, preprocess the collected original DAPI and marker data. The specific steps include: Step S121, for an image , its mean and variance are calculated as follows: ; ; wherein, represents the pixel value of image I; represents the number of pixel points of the image; Step S122, let and be the mean and variance of the DAPI image respectively, and be the mean and variance of the marker image respectively, and calculate the normalization ratio as follows: ; ; wherein, represents the normalization ratio; represents the overall translation correction amount after linear normalization of the marker image; Step S123: For each marker image M, calculate its normalized mean and variance, and unify the marker image in terms of mean and variance to the reference space of the DAPI image as follows: ; where, indicates that the pixel value distribution of the marker image is linearly transformed to have the same mean and standard deviation as the DAPI image.

[0008] Preferably, in step S11, collect the original DAPI and marker data. The specific experimental process includes: (1) mIHC experiment: Use IHC to analyze the positive conditions of CD8, CD45RO, CD68, and vimentin in patient samples; (2) Deparaffinization: Place the sections in xylene for 10 minutes, repeat 2 times, then sequentially place them in 95% alcohol, 85% alcohol, and 75% alcohol for 5 minutes, and then wash with phosphate buffer PBS; (3) Antigen retrieval: Boil the tissue sections in the antigen retrieval buffer for 10 minutes and keep warm for 30 minutes; After cooling at room temperature for 30 minutes, take out the slides and wash them 3 times with PBS, 3 minutes each time; (4) Endogenous peroxidase blocking: Place the slides in 3% hydrogen peroxide solution and incubate at room temperature in the dark for 10 minutes, then wash 3 times with PBS; (5) Blocking: Drop Super Blocker on the sections and incubate at room temperature for 30 minutes; Among them, primary antibody incubation: Drop the primary antibody diluted in proportion with the primary antibody diluent on the tissue; Incubate the sections overnight at 4°C in a humidified chamber; Secondary antibody incubation: After washing 3 times with PBS, drop the primary antibody host-specific secondary antibody solution to cover the tissue and incubate at room temperature for 30 min; Finally, develop the color with DAB chromogenic solution, use the tool to obtain the corresponding mIHC image, and extract the images of its different channels.

[0009] Preferably, in step S2, construct a prototype multi-task network model, including a multi-task prototype layer, a prototype attention layer, and an image decoder layer; In the prototype multi-task network model, use the pre-trained model ResNet-50 as the image encoder , input the enhanced image into the encoder to obtain the feature representation of the image as follows: ; where, indicates the feature representation of the image; indicates the enhanced image.

[0010] Preferably, the specific process of constructing the multi-task prototype layer is as follows: Let the input image , and its feature map is extracted through the encoder layer as , where and represent the height and width of the feature map; The prototypes of different tasks are divided into task-specific prototypes and shared prototypes; the task-specific prototypes are independently learned in each task to capture task-specific features; the shared prototypes are used to learn the global representation across different tasks; Utilize the correlation between different tasks to construct a multi-task prototype layer, which simultaneously learns task-specific activation maps and shared activation maps; First, construct learnable task-specific prototypes for each task; then, calculate the similarity between these prototypes and each patch in the feature map obtained through the image encoder to obtain the corresponding activation maps; specifically, each position (i, j) of the k-th activation map represents the similarity score between the patch at the (i, j) position in the feature map and the k-th prototype; Among them, for task k, m task-specific prototypes and n shared prototypes are defined, and their shapes are , as follows: ; ; Among them, represents the task-specific prototype of task k; represents the shared prototype of task k; .

[0011] Preferably, the specific process of constructing the prototype attention layer is as follows: For task k, the original attention layer sums all activation maps, as follows: ; Among them, represents the element-wise product; represents the i-th task-specific activation map of task k; Assign different weights to different prototype activation maps, where each weight represents the importance of its corresponding prototype activation map for the combination, as follows: ; ; Among them, represents the new activation map obtained by weighting and summing all prototype activation maps; represents feature extraction; represents the weight of the i-th task-specific prototype activation map of task k; Denotes the weight of the j-th shared activation map for task k; Denotes the j-th shared activation map for task k.

[0012] Preferably, in order to calculate the weights of different activation maps , a prototype channel attention module PCAM is designed; the PCAM module first compresses the global spatial information of the prototype activation map into a prototype channel vector as follows: ; Then, an MLP layer with a gated sigmoid function is applied to capture the dependencies between different activation maps, and the weights for task k are obtained as follows: ; where, Denotes the Sigmoid gating function; Denotes the weight vector of each prototype activation map for task k.

[0013] Preferably, an image decoder layer is constructed. For each virtual staining task, a decoder layer is implemented to generate each mIHC label; based on the obtained prototype attention indication feature map, the target image is further obtained through the image decoder; wherein, the architectures of the image encoder and the image decoder follow the U-Net structure, and the encoder information is transmitted to the decoder using skip.

[0014] Preferably, in step S3, a loss function is designed, including the following steps: (1) When calculating the similarity score between a patch and the corresponding prototype, encourage each training image to have at least one patch close to the current prototype as at least one highly activated point in the activation map; to avoid all points being highly activated, at least one patch is kept away from the prototype component; thus, the activation loss is proposed, as follows: ; where, Denotes the features of all patches in the feature map of the input image j, corresponding to the maximum activation score on the prototype activation map; Denotes a small positive smoothing term used to prevent the denominator from being zero; (2) In the virtual staining task, in order to focus the activation map on different parts of the image, the diversity loss between different prototypes is proposed, that is, the pairwise cosine similarity between different activation maps is calculated, as follows: ; where, and represent the i-th and j-th prototype activation maps, respectively; (3) Use the L1 loss to perform the image generation task. For task k, the generation loss is as follows: ; where, represents the output image generated by task k; (4) The total loss function of multi-task virtual staining consists of the single-task loss , the activation loss and the diversity loss and is as follows: ; where, represents the total loss; represents the weight coefficient of the single-task loss ; represents the weight coefficient of the activation loss ; represents the weight coefficient of the diversity loss .

[0015] Preferably, in step S4, the prototype multi-task network model is trained to minimize the loss function. A network with a U-Net structure is used, and the backbone part of its encoder adopts ResNet50. The specific process is as follows: Step S41, optimizer selection: The stochastic gradient descent (SGD) optimizer is selected, and the default hyperparameter settings are adopted; Step S42, learning rate and weight decay setting: The learning rate is set to 0.01, and the weight decay is set to 0.0001 to control the size of the model parameters; Step S43, backpropagation and gradient update: Use the backpropagation algorithm to calculate the gradient of the loss function with respect to the model parameters; then, according to the rules of the SGD optimizer, update the model parameters to minimize the loss function; Step S44, iterative training: Through multiple iterative trainings, the model optimizes the parameters and improves the performance on the training data; Step S45, hyperparameter tuning: During the training process, adjust hyperparameters such as the learning rate according to the performance of the validation set to ensure the generalization performance of the model on the training and validation sets; Step S46, model saving: Regularly save the weights of the model so that the model can be reloaded at any time during the training process; Based on the above training process, the model learns task-specific feature representations, enabling effective encoding and normalization of DAPI images and marker images, as well as the ability to perform multi-task learning.

[0016] Therefore, the present invention adopts the above-mentioned prototype multi-task learning method for generating from DAPI to mIHC markers, captures the relationships between different virtual staining tasks by learning shared prototypes and task-specific prototypes; then in the original attention layer, both the task-specific prototypes and the shared prototypes will be re-weighted and combined to indicate the generation of different mIHC markers; by using the generation of multiple different markers to locate each other to achieve the purpose of multi-task generation, it can efficiently and interpretably generate multiple markers from DAPI staining.

[0017] The technical solution of the present invention will be further described in detail below with reference to the drawings and embodiments. Description of the Drawings

[0018] Figure 1 is a flowchart of a prototype multi-task learning method for generating from DAPI to mIHC markers according to the present invention; Figure 2 is a schematic diagram of the network structure of a prototype multi-task learning method for generating from DAPI to mIHC markers according to the present invention. Detailed Embodiments

[0019] The technical solution of the present invention will be further described below with reference to the drawings and embodiments.

[0020] As Figure 1 shown, a prototype multi-task learning method for generating from DAPI to mIHC markers includes the following steps: Step S1: Collect original DAPI and marker data, and preprocess the collected data; Step S2: Construct a prototype multi-task network model, including a multi-task prototype layer, a prototype attention layer, and an image decoder layer; Step S3: Design a loss function, including a single-task loss , an activation loss , a diversity loss and a generation loss ; Step S4: Train the prototype multi-task network model; Step S5: Based on the above steps, preprocess the test data in the same way as in training, and then input it into the trained network to obtain the generation result of the markers.

[0021] Embodiment Step S1: Collect the original DAPI and marker data, and preprocess the collected data.

[0022] Step S11: Collection of the original DAPI and marker data.

[0023] (1) mIHC experiment: Use IHC to analyze the positive conditions of CD8, CD45RO, CD68, and vimentin in patient samples.

[0024] (2) Deparaffinization: Place the sections in xylene for 10 minutes, repeat 2 times, then sequentially place them in 95% alcohol, 85% alcohol, and 75% alcohol for 5 minutes, and then wash with phosphate buffer (PBS).

[0025] (3) Antigen retrieval: Boil the tissue sections in antigen retrieval buffer for 10 minutes and keep warm for 30 minutes. After cooling at room temperature for 30 minutes, take out the slides and wash them 3 times with PBS, 3 minutes each time.

[0026] (4) Endogenous peroxidase blocking: Place the slides in 3% hydrogen peroxide solution and incubate at room temperature in the dark for 10 minutes, then wash 3 times with PBS.

[0027] (5) Blocking: Drop Super Blocker on the sections and incubate at room temperature for 30 minutes.

[0028] Among them, primary antibody incubation: Drop the primary antibody diluted in proportion with the primary antibody diluent on the tissue. Incubate the sections overnight at 4°C in a humidified chamber; secondary antibody incubation: After washing 3 times with PBS, drop the primary antibody host-specific secondary antibody solution to cover the tissue and incubate at room temperature for 30 min; finally, develop the color with DAB chromogenic solution, obtain the corresponding mIHC image using relevant tools, and extract the images of its different channels.

[0029] In this embodiment, a multiplex immunohistochemistry (mIHC) dataset containing colon cancer and liver cancer tissues is utilized. The original images contain DAPI, CD8, CD45RO, CD68, and Vimentin channels. Use DAPI as the source image to generate other channels, thus forming a multi-task method involving four sub-tasks, including CD8, CD45RO, CD68, and Vimentin. In the experiment, 4 regions of interest (ROIs) were selected for training, and 1 ROI was reserved for testing. The resolution of the images at 40x magnification is 0.24 microns per pixel. Extract ROI images with a size of 256 pixels and a step size of 128 pixels, obtaining 3325 training data samples and 765 test data samples for the colon tissue dataset. In the liver tissue dataset, 1654 training images and 559 test images were obtained respectively.

[0030] Step S12. Preprocess the collected original DAPI and marker data; In the data preprocessing stage, consider normalizing the mean and variance of the marker image to the mean and variance of the DAPI image, and adjust the statistical characteristics of the marker image to be consistent with the DAPI image to achieve better data consistency and stability.

[0031] Step S121. For an image , its mean and variance are calculated as follows: ; ; where represents the pixel value of image I; represents the number of pixel points of the image.

[0032] Step S122. Let and be the mean and variance of the DAPI image respectively, and and be the mean and variance of the marker image respectively. Calculate the normalization ratio as follows: ; ; where represents the normalization ratio; represents the overall translation correction amount after the marker image is linearly normalized.

[0033] Step S123. For each marker image M, calculate its normalized mean and variance, and unify the marker image in terms of mean and variance to the reference space of the DAPI image as follows: ; where means that the pixel value distribution of the marker image is linearly transformed to have the same mean and standard deviation as the DAPI image.

[0034] Step S2. Construct a prototype multi-task network model as Figure 2 shown.

[0035] Step S21. Use the pre-trained model ResNet-50 as the image encoder , and the residual structure of ResNet-50 helps to learn more complex feature representations. Therefore, input the enhanced image into the encoder to obtain the feature representation of the image as follows: ; Among them, represents the feature representation of the image; represents the enhanced image.

[0036] Step S22, construct a multi-task prototype layer.

[0037] Let the input image , and its feature map is extracted through the encoder layer as , where and represent the height and width of the feature map. In the multi-task learning scenario, the correlation between different tasks is utilized to construct a multi-task prototype layer, which can simultaneously learn task-specific activation maps and shared activation maps. To this end, the prototypes of different tasks are divided into task-specific prototypes and shared prototypes. Task-specific prototypes are independently learned for each task and can capture task-specific features; shared prototypes are dedicated to learning the global representation across different tasks.

[0038] First, construct learnable task-specific prototypes for each task. Then, calculate the similarity between these prototypes and each patch in the feature map obtained through the image encoder to obtain the corresponding activation maps. Specifically, each position (i, j) of the k-th activation map represents the similarity score between the patch at the (i, j) position in the feature map and the k-th prototype.

[0039] Among them, for task k, m task-specific prototypes and n shared prototypes are defined, and their shapes are , as follows: ; ; Among them, represents the task-specific prototype of task k; represents the shared prototype of task k; . In this embodiment, is set.

[0040] Step S23, construct a prototype attention layer.

[0041] For task k, the original attention layer sums all activation maps, as follows: ; Among them, represents element-wise multiplication; represents the i-th task-specific activation map of task k.

[0042] However, each activation map may contribute different weights to the final attention map. Therefore, a weighted sum strategy is proposed to assign different weights to different prototype activation maps, where each weight represents the importance of its corresponding prototype activation map for the combination, as follows: ; ; where, represents the new activation map obtained by weighting and summing all prototype (specific + shared) activation maps; represents feature extraction; represents the weight of the i-th specific prototype activation map for task k; represents the weight of the j-th shared activation map for task k; represents the j-th shared activation map for task k.

[0043] To calculate the weights of different activation maps , a prototype channel attention module (PCAM) is designed. The PCAM module first compresses the global spatial information of the prototype activation map into a prototype channel vector , as follows: ; Then, an MLP layer with a gated sigmoid function is applied to capture the dependencies between different activation maps, and the weights for task k are obtained, as follows: ; where, represents the Sigmoid gating function; represents the weight vector of each prototype activation map for task k.

[0044] Step S24, construct an image decoder layer.

[0045] Finally, for each virtual staining task, a decoder layer is implemented to generate each mIHC label. Through the prototype attention indication feature map obtained in the previous step, the target image is further obtained through the image decoder. Among them, the architectures of the image encoder and the image decoder follow the U-Net construction, and the encoder information is passed to the decoder using skip.

[0046] Step S3, design a loss function.

[0047] The problem that the similarity between patches and prototypes may lead to insignificant activation maps and fail to capture the activated regions; and the problem that if the prototypes lack diversity and do not show obvious differences, the prototype features of the image may not be fully captured or learned. To obtain a better prototype representation for image generation, the present invention designs a loss function to overcome the above problems and defects.

[0048] (1) When calculating the similarity score between a patch and the corresponding prototype, each training image is encouraged to have at least one patch close to the current prototype as at least one highly activated point in the activation map. To avoid all points being highly activated, at least one patch is kept away from the prototype component. Therefore, the activation loss is proposed , as follows: ; where represents the features of all patches in the feature map of the input image, corresponding to the maximum activation score on the prototype activation map; represents a small positive smoothing term used to prevent the denominator from being zero.

[0049] (2) In the virtual staining task, it is desired that the activation map can focus on different parts of the image. Therefore, the diversity loss between different prototypes is proposed , that is, the pairwise cosine similarity between different activation maps is calculated, as follows: ; where and represent the i-th and j-th prototype activation maps respectively. Intuitively, the smaller the value of the diversity loss, the greater the diversity between different activation maps, and vice versa.

[0050] (3) The L1 loss is used to perform the image generation task. For task k, the generation loss , as follows: ; where represents the output image generated by task k.

[0051] (4) The total loss function for multi-task virtual staining consists of the single-task loss , the activation loss and the diversity loss , as follows: ; where represents the total loss; represents the weight coefficient of the single-task loss ; represents the activation loss Weight coefficient; Indicates the diversity loss Weight coefficient.

[0052] Step S4: Train the prototype multi-task network model.

[0053] The network training process is a process of optimizing the parameters of the deep learning model to minimize the loss function. Here, a network with a structure similar to U-Net is used, and the backbone of its encoder adopts ResNet50.

[0054] Step S41: Optimizer selection: The Stochastic Gradient Descent (SGD) optimizer is selected, and the default hyperparameter settings are adopted.

[0055] Step S42: Learning rate and weight decay settings: The learning rate is set to 0.01, and the weight decay is set to 0.0001 to control the size of the model parameters.

[0056] Step S43: Backpropagation and gradient update: Use the backpropagation algorithm to calculate the gradient of the loss function with respect to the model parameters. Then, according to the rules of the SGD optimizer, update the model parameters to minimize the loss function.

[0057] Step S44: Iterative training: Through multiple iterative trainings, the model gradually optimizes the parameters and improves the performance on the training data.

[0058] Step S45: Hyperparameter tuning: During the training process, hyperparameters such as the learning rate can be adjusted according to the performance of the validation set to ensure that the model has good generalization performance on both the training and validation sets.

[0059] Step S46: Model saving: Regularly save the weights of the model so that the model can be reloaded at any time during the training process.

[0060] Based on the above training process, the model can gradually learn task-specific feature representations, achieve effective encoding and normalization of DAPI images and marker images, and the ability to perform multi-task learning.

[0061] Step S5: Based on the above steps, preprocess the test data in the same way as in the training, and then input it into the trained network to obtain the generation results of the markers.

[0062] Therefore, the present invention adopts the above-mentioned prototype multi-task learning method for generating from DAPI to mIHC markers, captures the relationships between different virtual staining tasks by learning shared prototypes and specific-task prototypes; then in the original attention layer, both the specific-task prototypes and the shared prototypes will be re-weighted and combined to indicate the generation of different mIHC markers; the generation of multiple different markers is used to locate each other to achieve the purpose of multi-task generation, and it can efficiently and interpretably generate multiple markers from DAPI staining.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that they can still modify or equivalently replace the technical solutions of the present invention, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A prototype multi-task learning method for generating from DAPI to mIHC markers, characterized in that, It includes the following steps: Step S1, collect the original DAPI and marker data, and preprocess the collected data; Step S2, construct a prototype multi-task network model, including a multi-task prototype layer, a prototype attention layer, and an image decoder layer; Step S3: Design a loss function, including a single-task loss , an activation loss , a diversity loss and a generation loss ; Step S4, train the prototype multi-task network model; Step S5, based on the above steps, preprocess the test data in the same way as in training, and then input it into the trained network to obtain the generation result of the marker.

2. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 1, characterized in that In step S1, collect the original DAPI and marker data, and preprocess the collected data. The specific process is as follows: Step S11, collect the original DAPI and marker data; Step S12, preprocess the collected original DAPI and marker data. The specific steps include: Step S121. For an image , its mean value and variance are calculated as follows: ； ； Among them, represents the pixel value of image I; represents the number of pixel points of the image; Step S122, set and be the mean and variance of the DAPI image respectively, and be the mean and variance of the marker image respectively, and calculate the normalization ratio as follows: ； ； Among them, represents the standardized ratio; represents the overall translation correction amount after the linear standardization of the marker image; Step S123, for each marker image M, calculate its standardized mean and variance, and unify the marker image in terms of mean and variance to the reference space of the DAPI image as follows: ； Among them, indicates that the pixel value distribution of the marker image is linearly transformed to have the same mean and standard deviation as the DAPI image.

3. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 2, characterized in that: In step S11, collect the original DAPI and marker data. The specific experimental process includes: (1) mIHC experiment: Use IHC to analyze the positive conditions of CD8, CD45RO, CD68, and vimentin in patient samples; (2) Deparaffinization: Immerse the sections in xylene for 10 minutes, repeat 2 times, then sequentially immerse them in 95% alcohol, 85% alcohol, and 75% alcohol for 5 minutes, and then wash them with phosphate buffer PBS; (3) Antigen repair: Boil the tissue sections in antigen repair buffer for 10 minutes and keep warm for 30 minutes; After cooling at room temperature for 30 minutes, take out the slides and wash them 3 times with PBS, 3 minutes each time; (4) Endogenous peroxidase blocking: Immerse the slides in 3% hydrogen peroxide solution and incubate them at room temperature in the dark for 10 minutes, then wash them 3 times with PBS; (5) Blocking: Drop Super Blocker on the sections and incubate them at room temperature for 30 minutes; Among them, primary antibody incubation: Drop the primary antibody diluted in proportion with the primary antibody diluent on the tissue; Incubate the sections in a humid chamber at 4°C overnight; Secondary antibody incubation: After washing 3 times with PBS, drop the primary antibody host-specific secondary antibody solution to cover the tissue and incubate it at room temperature for 30 min; Finally, develop the color with DAB chromogenic solution, use tools to obtain the corresponding mIHC images, and extract the images of different channels.

4. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 1, characterized in that In step S2, construct a prototype multi-task network model, including a multi-task prototype layer, a prototype attention layer, and an image decoder layer; In the prototype multi-task network model, the pre-trained model ResNet-50 is used as the image encoder , and the enhanced image is input into the encoder to obtain the feature representation of the image, as follows: ； Among them, represents the image feature representation; represents the enhanced image.

5. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 4, characterized in that, The specific process of constructing the multi-task prototype layer is as follows: Assume the input image , and extract its feature map through the encoder layer as , where and represent the height and width of the feature map; Divide the prototypes of different tasks into task-specific prototypes and shared prototypes; Task-specific prototypes are independently learned in each task to capture task-specific features; Shared prototypes are used to learn the global representation across different tasks; Utilize the correlation between different tasks to construct the multi-task prototype layer, which simultaneously learns task-specific activation maps and shared activation maps; First, construct learnable task-specific prototypes for each task; then, calculate the similarity between these prototypes and each patch in the feature map obtained by the image encoder to obtain the corresponding activation map. Specifically, each position (i, j) in the k-th activation map represents the similarity score between the patch at the (i, j) position in the feature map and the k-th prototype. Among them, for task k, m specific prototypes and n shared prototypes are defined, and their shapes are , as follows: ； ； Among them, represents the specific prototype of task k; represents the shared prototype of task k; .

6. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 4, characterized in that, The specific process of constructing the prototype attention layer is as follows: For task k, the original attention layer sums all the activation maps as follows: ； Among them, represents the element-wise product; represents the i-th specific activation map of task k; Assign different weights to different prototype activation maps, where each weight represents the importance of its corresponding prototype activation map for the combination, as follows: ； ； Among them, represents the new activation map obtained by weighting and summing all prototype activation maps; represents feature extraction; represents the weight of the i-th specific prototype activation map of task k; represents the weight of the j-th shared activation map of task k; represents the j-th shared activation map of task k.

7. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 6, characterized in that To calculate the weights of different activation maps , a prototype channel attention module PCAM was designed; The PCAM module first compresses the global spatial information of the prototype activation map into a prototype channel vector as follows: ； Then, apply an MLP layer with a gated sigmoid function to capture the dependencies between different activation maps, and the weights for task k are obtained as follows: ； Among them, represents the Sigmoid gating function; represents the weight vector of the activation maps of each prototype for task k.

8. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 4, characterized in that Construct an image decoder layer. For each virtual staining task, implement a decoder layer to generate each mIHC marker; based on the obtained prototype attention-indicating feature map, further obtain the target image through the image decoder. Among them, the architectures of the image encoder and the image decoder follow the U-Net construction, and the encoder information is passed to the decoder using skip.

9. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 1, characterized in that, In step S3, design the loss function, including the following steps: (1) When calculating the similarity score between a patch and the corresponding prototype, each training image is encouraged to have at least one patch close to the current prototype as at least one highly activated point in the activation map; to avoid all points being highly activated, at least one patch is kept away from the prototype part; thus, the activation loss is proposed , as follows: ； Among them, represents the features of all patches in the feature map of the input image j, corresponding to the maximum activation score on the prototype activation map; represents a small positive smoothing term used to prevent the denominator from being zero; (2) In the virtual staining task, to focus the activation maps on different parts of the image, a diversity loss between different prototypes is proposed , that is, the pairwise cosine similarity between different activation maps is calculated as follows: ； Among them, and represent the i-th and j-th prototype activation maps respectively; (3) Use the L1 loss to perform the image generation task. For task k, the generation loss is as follows: ； Among them, represents the output image generated by task k; (4)The total loss function of multi-task virtual staining consists of a single-task loss , an activation loss , and a diversity loss and is as follows: ； Among them, represents the total loss; represents the weight coefficient of the single task loss ; represents the weight coefficient of the activation loss ; represents the weight coefficient of the diversity loss .

10. A prototype multi-task learning method for generating from DAPI to mIHC markers according to claim 1, characterized in that, In step S4, train the prototype multi-task network model to minimize the loss function. Use a network with a U-Net structure, and the backbone part of its encoder adopts ResNet50. The specific process is as follows: Step S41, Optimizer selection: Select the stochastic gradient descent SGD optimizer and adopt the default hyperparameter settings; Step S42, Learning rate and weight decay settings: The learning rate is set to 0.01, and the weight decay is set to 0.0001 to control the size of the model parameters; Step S43, Backpropagation and gradient update: Use the backpropagation algorithm to calculate the gradient of the loss function with respect to the model parameters; then, according to the rules of the SGD optimizer, update the model parameters to minimize the loss function; Step S44, Iterative training: Through multiple iterations of training, the model optimizes the parameters and improves the performance on the training data; Step S45, Hyperparameter tuning: During the training process, adjust the learning rate hyperparameter according to the performance of the validation set to ensure the generalization performance of the model on the training and validation sets; Step S46, Model saving: Regularly save the weights of the model for reloading the model during the training process; Based on the above training process, the model learns task-specific feature representations, realizes the effective encoding and normalization of DAPI images and marker images, and the ability to perform multi-task learning.

Images