A Predictive Model Construction Method Based on Weakly Supervised Multimodal Contrast Learning and a HER2 Score Prediction Method for Breast Cancer

Abstract

This invention discloses a prediction model construction method based on weakly supervised multimodal contrastive learning and a method for predicting HER2 scores in breast cancer. The model construction method includes the following steps: acquiring a multimodal whole-slice pathology dataset with images and labels and performing block preprocessing; extracting features from the block images using a feature encoder; establishing a network model based on weakly supervised multimodal contrastive learning; and training the network model using a multimodal whole-slice pathology dataset containing weak labels. The HER2 score prediction method for breast cancer uses the trained model to predict the scores of whole-slices pathology. This invention can learn and integrate pathological knowledge related to different modalities of whole-slices pathology through weakly supervised multimodal contrastive learning, effectively achieving semantic complementarity between multiple modalities and improving the learning ability of each modality, completing the HER2 scoring task of whole-slices pathology and performing visual interpretive analysis.

Description

Technical Field

[0001] This invention relates to the field of whole-section image analysis technology in histopathology, specifically to a method for constructing a prediction model based on weakly supervised multimodal contrastive learning and a method for predicting HER2 scores in breast cancer. Background Technology

[0002] Traditional HER2 (Human Epidermal Growth Factor Receptor 2) scoring for breast cancer requires experienced pathologists to examine the tissue under a microscope, observing staining patterns and quantifying the HER2 score through immunohistochemical staining to arrive at an interpretation. With the increasing demand for pathological diagnosis, the workload of pathologists has gradually increased, negatively impacting the efficiency of the entire diagnostic process. Furthermore, the subjective nature of human observation poses a significant challenge to the accuracy and objectivity of the final judgment and quantification. With the rapid development of artificial intelligence technology, using deep learning techniques to train neural networks for HER2 scoring of breast cancer has become possible.

[0003] In recent years, the technology for digitizing pathology slides has matured, allowing pathologists to observe and interpret whole pathology slides on a screen using software. This naturally leads to the consideration of leveraging artificial intelligence in image analysis to process and analyze digitized whole pathology slides. However, compared to natural images, whole pathology slides have high resolution, making them unsuitable for direct input into networks for training. Therefore, it is often necessary to segment the entire digital pathology slide into several fixed-size images before analysis. However, this often only provides slide-level annotations, lacking fine-grained annotations, making it impossible to train using strongly supervised learning methods. Based on these challenges, a common solution is to employ weakly supervised learning methods. Consequently, numerous weakly supervised slide classification methods have emerged in recent years, demonstrating the enormous development potential of AI-enabled auxiliary tools and holding significant importance for the pathology industry.

[0004] However, current methods often only use hematoxylin-eosin HE-stained sections or immunohistochemical IHC-stained sections to train the model, without considering the use of both types of stained sections for joint training to improve the model's prediction accuracy. Summary of the Invention

[0005] The purpose of this invention is to provide a predictive model construction method and a breast cancer HER2 score prediction method based on weakly supervised multimodal contrastive learning. This method solves the problems of lack of detailed annotation in the current pathological whole slide analysis, and the problem of training the model using only HE-stained or IHC-stained slides without considering the use of joint training of the two types of stained slides to improve the prediction accuracy of the model.

[0006] The present invention achieves the above objectives through the following technical solutions:

[0007] A method for constructing a prediction model based on weakly supervised multimodal contrastive learning, comprising the following steps:

[0008] S1. Obtain a multimodal pathological whole slide dataset with image category labels. The dataset contains several pairs of slides, and each pair of slides contains one HE-stained slide and one IHC-stained slide. Then, the slides in the dataset are preprocessed into blocks to obtain a set of block images. Then, the block image features of each block image in the set are extracted.

[0009] S2. The block image features are used as input to a weakly supervised multimodal contrastive learning network model. After processing by the model, the global features of HE staining slices and IHC staining slices, as well as the attention score of each block image, are output.

[0010] S3. Based on the attention scores, the block images of HE-stained sections and IHC-stained sections are ranked by key importance. Key block images are selected from them and their block image features are subjected to multimodal attention contrast learning to obtain multimodal contrast loss.

[0011] S4. Classify and predict the global features of HE-stained sections and IHC-stained sections respectively to obtain the HE single-modal prediction probability and IHC single-modal prediction probability. Then, fuse the global features of HE-stained sections and IHC-stained sections to obtain multimodal fused features. Finally, classify and predict the multimodal fused features to obtain the multimodal prediction probability.

[0012] S5. Calculate the cross-entropy loss between the IHC single-modal prediction probability and the multi-modal prediction probability and the image category label, and then calculate the relative entropy loss between the HE single-modal prediction probability and the IHC single-modal prediction probability.

[0013] S6. Optimize the network weights of the weakly supervised multimodal contrastive learning network model based on cross-entropy loss, relative entropy loss, and multimodal contrastive loss to obtain the optimized HER2 score prediction model for breast cancer, thus completing the construction.

[0014] A further improvement is made in that the specific process of step S1 is as follows:

[0015] S1-1. Obtain a multimodal pathological whole-slice dataset D with image category labels. The dataset D contains several pairs of slices, and each pair contains one HE-stained slice and one IHC-stained slice. Represent the dataset D as follows: in and Y represents the HE-stained section and the IHC-stained section in the i-th pair of slices in dataset D, respectively. i The label represents the i-th slice, and N represents the number of slice pairs in dataset D;

[0016] S1-2. A tissue mask image of the entire pathological slide is generated using the Otsu thresholding algorithm. Then, the entire pathological slide is segmented into fixed-size blocks using a sliding window method. Background blocks are filtered out using the tissue mask image, retaining only the blocks representing the tissue regions. and After block preprocessing, block image sets are obtained respectively. and in This represents the j-th block image of the HE-stained section in the i-th whole slice. Let represent the j-th block image of the IHC-stained section in the i-th pair of whole sections, n represent the number of block images in the HE-stained section in the i-th pair of whole sections, and m represent the number of block images in the IHC-stained section in the i-th pair of whole sections.

[0017] S1-3. Use a pre-trained feature encoder to extract the block image features of each block image in the set, and each block image can generate a feature vector e←f(I), e∈R. d Where d is the feature dimension; thus, a pair of whole pathological sections is represented as All the segmented image features are represented as and

[0018] A further improvement is that, in step S2, the weakly supervised multimodal contrastive learning network model includes two backbone networks corresponding to single modalities, a multimodal attention contrastive learning module, and a multimodal feature fusion module. The multimodal feature fusion module includes a multimodal feature mapping linear layer and a multimodal classifier. The backbone network corresponding to each single modality includes a linear layer, a multi-head self-attention module, a layer normalization layer, and a classifier.

[0019] A further improvement lies in the fact that the specific process of model processing is as follows:

[0020] The segmented image features and The data are fed into the backbone network of the corresponding model, where information learning and communication between image blocks are achieved through linear layers and multi-head self-attention modules. Pooling operations are then used to obtain global features that can describe the entire slice. and Simultaneously, attention scores for each image block are generated through the multi-head attention module in each unimodal backbone network. and

[0021] A further improvement is made in that the specific process of step S3 is as follows:

[0022] S3-1, Based on the attention score of the segmented image and Features of segmented images and Perform a priority ranking, then from... and The k features with the highest attention scores and the k features with the lowest attention scores are selected respectively to obtain the key block image features. and

[0023] S3-2. Calculate the cosine similarity matrix after standardizing the key segment image features. and

[0024]

[0025]

[0026] In equation (1), e is the Euler number and λ represents the learnable parameter;

[0027] S3-3, The cosine similarity matrix is... and The comparative loss is calculated iteratively. For each iteration, the calculation process is as follows:

[0028]

[0029] In equations (3)-(6), CE(·) represents the cross-entropy loss function, ∥ represents the array concatenation operation, and M[j] represents the j-th column element of matrix M;

[0030] S3-4. Accumulate and sum the contrast losses from all iterations to obtain the multimodal contrast loss. The calculation process is as follows:

[0031]

[0032] A further improvement is made in that the specific process of step S4 is as follows:

[0033] S4-1. Global features of HE-stained and IHC-stained sections and The data are input into the classifier corresponding to each backbone network for prediction, so as to obtain the HE single-modal prediction for each modality. And IHC single-mode prediction P i ihc The process is as follows:

[0034]

[0035] In equations (9) and (10), W he ∈R d×C and W ihc ∈R d×C These are learnable parameters, and C is the number of categories;

[0036] S4-2. Global features of HE-stained and IHC-stained sections and Both are input into the multimodal feature fusion module to obtain multimodal fused features. The process is as follows:

[0037]

[0038] In equations (11)-(13), and These represent the element-wise addition and element-wise multiplication operations of vectors, respectively. GELU(·) is the Gaussian error linear unit activation function. and These are learnable parameters;

[0039] S4-3, Fusing features across multiple modalities The data is fed into a multimodal classifier for prediction to obtain the multimodal prediction probability.

[0040]

[0041] In equation (15), These are learnable parameters.

[0042] A further improvement is made in that the specific process of step S5 is as follows:

[0043] S5-1 Calculate the IHC single-mode prediction probability and multimodal prediction probability With image category label Y i Cross-entropy loss:

[0044]

[0045] In equations (16) and (17), It is the predicted probability. The probability of the c-th category. It is the predicted probability. The probability of Y for the c-th category i It is the tag in the one-hot encoded form of the i-th pair of full slices;

[0046] S5-2, Calculate the HE single-modal prediction probability rate and IHC single-mode prediction probability Relative entropy loss:

[0047]

[0048] In equation (18), KL(·) is the relative entropy function.

[0049] This invention also provides a method for predicting HER2 scores in breast cancer. The prediction method utilizes a prediction model constructed using the aforementioned method. The specific process is as follows:

[0050] S1. Obtain a breast cancer pathology whole slide dataset with image category labels. The dataset contains several pairs of slides, and each pair of slides contains one HE-stained slide and one IHC-stained slide. Then, the slides in the dataset are preprocessed into blocks to obtain a set of block images. Then, the block image features of each block image in the set are extracted.

[0051] S2. The block image features are used as input to the prediction model. After model processing, the global features of HE staining slices and IHC staining slices and the attention score of each block image are output.

[0052] S3. Classify and predict the global features of HE-stained sections and IHC-stained sections respectively to obtain the HE single-modal prediction probability and IHC single-modal prediction probability. Then, fuse the global features of HE-stained sections and IHC-stained sections to obtain multimodal fused features. Finally, classify and predict the multimodal fused features to obtain the multimodal prediction probability.

[0053] S4. The HER2 score for breast cancer is obtained based on the HE single-modal prediction probability, IHC single-modal prediction probability, and multimodal prediction probability.

[0054] The beneficial effects of this invention are as follows:

[0055] (1) The present invention can complete the training of the network model using only slice-level labels, without the need for detailed cell region annotation;

[0056] (2) This invention introduces multimodal attention contrastive learning, which can realize the alignment and information exchange between HE and IHC modes, thereby improving the network's ability to predict HER2 scores for breast cancer.

[0057] (3) The multimodal contrastive learning framework designed in this invention can support HE and IHC slices to be paired as long as they have the same category label, without needing to come from the same case, in the absence of paired data;

[0058] (4) The method designed in this invention supports both single-modal and multi-modal reasoning and has a wide range of application scenarios. Attached Figure Description

[0059] Figure 1 This is a flowchart of the HER2 score prediction method for breast cancer according to the present invention;

[0060] Figure 2 This is a network model diagram of the HER2 score prediction method for breast cancer according to the present invention. Detailed Implementation

[0061] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0062] like Figure 1-2 As shown, this embodiment proposes a prediction model construction method and a breast cancer HER2 score prediction method based on weakly supervised multimodal contrastive learning, addressing the problem of lacking detailed annotations. The designed multimodal attention contrastive learning module is used to jointly train HE and IHC slices to complete the model construction, and the constructed prediction model is used to predict the breast cancer HER2 score. The construction method includes the following steps:

[0063] Step 1: Obtain a multimodal pathological whole-slice dataset D with image category labels. Dataset D contains several pairs of slices, each pair consisting of one hematoxylin-eosin (HE) stained slice and one immunohistochemical (IHC) stained slice. Perform block preprocessing on the slices in dataset D to obtain a set of block images.

[0064] Step 1.1: Obtain a multimodal pathological whole-slice dataset D with image category labels. Dataset D contains several pairs of slices, each pair consisting of one HE-stained slice and one IHC-stained slice. Therefore, the multimodal dataset can represent... in and Y represents the HE slice and IHC slice in the i-th pair of slices in dataset D, respectively.i The label represents the i-th slice, and N represents the number of slice pairs in dataset D;

[0065] Step 1.2: Generate a tissue mask image of the entire pathological slide using the Otsu thresholding algorithm. Then, using a sliding window method, the entire pathological slide is segmented into fixed-size blocks, and the background blocks are filtered out using the tissue mask image, retaining only the blocks representing the tissue regions. and After block preprocessing, block image sets are obtained respectively. and in This represents the j-th block image of the HE slice in the i-th full slice. Let represent the j-th block image of the IHC slice in the i-th pair of whole slices, n represent the number of block images in the HE slice in the i-th pair of whole slices, and m represent the number of block images in the IHC slice in the i-th pair of whole slices.

[0066] In this embodiment, the data used included 358 pairs of breast cancer slides, containing four HER2 scores (0, 1+, 2+, and 3+). Information for each case was verified and confirmed by a professional pathologist. All data were divided into training and testing sets according to a 7:3 ratio based on the cases. For each pathology slide, it was divided into 256×256 block images without repetition at 20x magnification.

[0067] Step 2: Use a pre-trained feature encoder to extract features from the segmented image. Each segmented image can generate a feature vector e←f(I), e∈R. d , where d is the feature dimension. Therefore, a pair of whole pathological sections can be represented as The features of all the segmented images can be represented as and In this embodiment, the pre-trained feature encoder uses the PLIP large-scale model trained on a large number of pathological images, and the extracted features have a dimension of 512.

[0068] Step 3: The network model based on weakly supervised multimodal contrastive learning consists of two backbone network branches, a multimodal attention contrastive learning module, and a multimodal feature fusion module. The multimodal fusion module includes a multimodal feature mapping linear layer and a multimodal classifier. Each single-modality backbone network includes a linear layer, a multi-head self-attention module, a layer normalization layer, and a classifier. The features of the segmented image described in Step 2 are then processed... and The data are input into the backbone network of the corresponding model, and information learning and communication between the image blocks are achieved through linear layers and multi-head self-attention modules. Pooling operations are then used to obtain features that can describe the entire slice. and Meanwhile, the multi-head attention module in the backbone network of each modality can generate an attention score for each image block. and

[0069] Step 4: Sort the IHC slice block images of HE slice according to the score of each block image, and select the features of key block images from them to send to the multimodal attention contrast learning module for contrast learning to obtain the multimodal contrast loss.

[0070] Attention scores for the segmented images in steps 4.1 and 3 and Features of segmented images and Sort, then from and The k features with the highest attention scores and the k features with the lowest attention scores are selected respectively to obtain the key block image features. and

[0071] Step 4.2: Calculate the cosine similarity matrix after standardizing the key segment image features. and

[0072]

[0073] In equation (1), e is the Euler number and λ represents the learnable parameter;

[0074] Step 4.3: Calculate the cosine similarity matrix. and The comparative loss is calculated iteratively. For each iteration, the calculation process is as follows:

[0075]

[0076] In equations (3)-(6), CE(·) represents the cross-entropy loss function, ∥ represents the array concatenation operation, and M[j] represents the j-th column element of matrix M. Finally, the contrastive losses from all iterations are summed to obtain the total contrastive loss. The calculation process is as follows:

[0077]

[0078] Contrastive loss is used for parameter updates in the network model.

[0079] In this embodiment, k is set to 50.

[0080] Step 5: Features of the entire slice and The data is input into the classifier corresponding to each backbone network in step 3 for prediction, so as to obtain the single-modal prediction for each modality. and The process is as follows:

[0081]

[0082] In equations (9) and (10), W he ∈R d×C and W ihc ∈R d×C C is a learnable parameter, and C is the number of categories. In this embodiment, the HER2 score contains 4 levels. Therefore, C is set to 4.

[0083] Step 6: Input the features describing the HE and IHC slices together into the multimodal feature fusion module to obtain multimodal fused features. Then, input the multimodal fused features into the multimodal classifier to obtain the multimodal prediction probabilities;

[0084] Step 6.1: Describe the characteristics of the HE and IHC slices. and The data are input into the multimodal feature fusion module to obtain multimodal fused features. The process is as follows:

[0085]

[0086] In equations (11)-(13), and These represent the element-wise addition and element-wise multiplication operations of vectors, respectively. GELU(·) is the Gaussian error linear unit activation function. and These are learnable parameters.

[0087] Step 6.2: Fuse multimodal features The data is fed into a multimodal classifier for prediction to obtain the multimodal prediction probability.

[0088]

[0089] In equation (15), And are learnable parameters.

[0090] Step 7: Calculate the loss by comparing the IHC single-modal prediction probability and multi-modal prediction probability with the true label Y. iCalculate the cross-entropy loss. Additionally, calculate the relative entropy loss between the HE single-modal prediction probabilities and the IHC single-modal prediction probabilities. Finally, optimize the network weights together with the aforementioned multimodal contrastive loss.

[0091] Step 7.1: Calculate the prediction probability of a single-modal IHC slice. and multimodal prediction probability Compared with the real label Y i Cross-entropy loss:

[0092]

[0093] In equations (16) and (17), It is the predicted probability. The probability of the c-th category. It is the predicted probability. The probability of Y for the c-th category i It is the tag in the one-hot encoded form of the i-th pair of full slices;

[0094] Step 7.2: Calculate the prediction probability of a single-modal HE slice. Predicted probabilities of single-mode IHC slices Relative entropy:

[0095]

[0096] In equation (18), KL(·) is the relative entropy function. The cross-entropy loss is then described. and Relative entropy loss and contrastive learning loss Collaborate on optimizing and training the neural network model, for example, by incorporating cross-entropy loss. and Relative entropy loss and contrastive learning loss The sums are used to obtain a total loss, which is then used for iterative optimization.

[0097] This invention also provides a method for predicting HER2 scores in breast cancer. A trained model is used to predict digital pathology whole slides to obtain the HER2 score of each slide. In this embodiment, HE and IHC slides from the test set are input into the model separately for single-modal inference. Then, the HE and IHC slides are input together into the network for multimodal inference. The specific process includes data processing of the pathology whole slides (steps 1 and 2 above) and the neural network model inference process (steps 3, 5, and 6 above).

[0098] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for constructing a prediction model based on weakly supervised multimodal contrastive learning, characterized by the following steps: include: S1. Obtain a multimodal pathological whole-slice dataset with image category labels. The dataset contains several pairs of slices, and each pair of slices contains one HE-stained slice and one IHC-stained slice. The slices in the dataset are then preprocessed into blocks to obtain a set of block images. The block image features of each block image in the set are then extracted. The block image features are represented as follows: and ; S2. The segmented image features are used as input to a weakly supervised multimodal contrastive learning network model. After model processing, the global features of HE-stained and IHC-stained sections and the attention score of each segmented image are output. The attention score is expressed as... and ; S3. Based on the attention scores, the block images of HE-stained sections and IHC-stained sections are ranked by key importance. Key block images are selected from them and their block image features are subjected to multimodal attention contrast learning to obtain multimodal contrast loss. The specific process of step S3 is as follows: S3-1, Based on the attention score of the segmented image and Features of segmented images and Perform a priority ranking, then from... and Select the ones with the highest attention scores respectively k The feature with the lowest attention score k Features are used to obtain key segmented image features. , , and ; S3-2. Calculate the cosine similarity matrix after standardizing the key segment image features. and : (1) (2) In formula (1) It is the Euler number. Indicates learnable parameters; S3-3, The cosine similarity matrix is... and The comparative loss is calculated iteratively. For each iteration, the calculation process is as follows: (3) (4) (5) (6) (7) In equations (3)-(6), Represents the cross-entropy loss function. , Representation matrix The Column elements; S3-4. Accumulate and sum the contrast losses from all iterations to obtain the multimodal contrast loss. The calculation process is as follows: (8)； S4. Classify and predict the global features of HE-stained sections and IHC-stained sections respectively to obtain the HE single-modal prediction probability and IHC single-modal prediction probability. Then, fuse the global features of HE-stained sections and IHC-stained sections to obtain multimodal fused features. Finally, classify and predict the multimodal fused features to obtain the multimodal prediction probability. S5. Calculate the cross-entropy loss between the IHC single-modal prediction probability and the multi-modal prediction probability and the image category label, and then calculate the relative entropy loss between the HE single-modal prediction probability and the IHC single-modal prediction probability. S6. Optimize the network weights of the weakly supervised multimodal contrastive learning network model based on cross-entropy loss, relative entropy loss, and multimodal contrastive loss to obtain the optimized HER2 score prediction model for breast cancer, thus completing the construction.

2. The method for constructing a prediction model based on weakly supervised multimodal contrastive learning according to claim 1, characterized in that, The specific process of step S1 is as follows: S1-1. Obtain a multimodal pathological whole-slice dataset with image category labels. D The dataset D The dataset contains several pairs of slices, with each pair consisting of one HE-stained slice and one IHC-stained slice. D Represented as ,in Representing datasets respectively D The Middle HE-stained sections and IHC-stained sections, Indicates the first Labels on the slices, N Represents the dataset D The number of slices; S1-2. A tissue mask image of the entire pathological slide is generated using the Otsu thresholding algorithm. Then, the entire pathological slide is segmented into fixed-size blocks using a sliding window method. Background blocks are filtered out using the tissue mask image, retaining only the blocks representing the tissue regions. After block preprocessing, block image sets are obtained respectively. , in Indicates the first Zhang's first HE-stained section of whole slides A block image, Indicates the first Zhang's first IHC-stained section of whole slide. A block image, Indicates the first The number of block images in the HE-stained section of the whole slice. Indicates the first The number of block images in the IHC-stained section of the whole slice; S1-3. Use a pre-trained feature encoder to extract the block image features of each block image in the set, and each block image can generate a feature vector. ,in It is a feature dimension; thus, a pair of whole pathological sections is represented as All the features of the segmented image are represented as .

3. The method for constructing a prediction model based on weakly supervised multimodal contrastive learning according to claim 2, characterized in that, In step S2, the weakly supervised multimodal contrastive learning network model includes two backbone networks corresponding to single modalities, a multimodal attention contrastive learning module, and a multimodal feature fusion module. The multimodal feature fusion module includes a multimodal feature mapping linear layer and a multimodal classifier. The backbone network corresponding to each single modality includes a linear layer, a multi-head self-attention module, a layer normalization layer, and a classifier.

4. The method for constructing a prediction model based on weakly supervised multimodal contrastive learning according to claim 3, characterized in that, The specific process of model processing is as follows: The segmented image features and The data are fed into the backbone network of the corresponding model, where information learning and communication between image blocks are achieved through linear layers and multi-head self-attention modules. Pooling operations are then used to obtain global features that can describe the entire slice. and Simultaneously, attention scores for each image block are generated through multi-head attention modules in each unimodal backbone network. .

5. The method for constructing a prediction model based on weakly supervised multimodal contrastive learning according to claim 4, characterized in that, The specific process of step S4 is as follows: S4-1. Global features of HE-stained and IHC-stained sections and The data are input into the classifier corresponding to each backbone network for prediction, so as to obtain the HE single-modal prediction for each modality. and IHC single-mode prediction The process is as follows: (9) (10) In equations (9) and (10), and These are learnable parameters. It is the number of categories; S4-2. Global features of HE-stained and IHC-stained sections and Both are input into the multimodal feature fusion module to obtain multimodal fused features. The process is as follows: (11) (12) (13) (14) In equations (11)-(13), and These represent the element-wise addition and element-wise multiplication operations of vectors, respectively. It is a Gaussian error linear unit activation function. These are learnable parameters; S4-3, Fusing features across multiple modalities The data is fed into a multimodal classifier for prediction to obtain the multimodal prediction probability. : (15) In equation (15), These are learnable parameters.

6. The method for constructing a prediction model based on weakly supervised multimodal contrastive learning according to claim 5, characterized in that, The specific process of step S5 is as follows: S5-1 Calculate the IHC single-mode prediction probability and multimodal prediction probability Image category tags Cross-entropy loss: (16) (17) In equations (16) and (17), It is the predicted probability. For the first The probability of each category, It is the predicted probability. For the first The probability of each category, It is the first Labels in one-hot encoded form for the entire slice; S5-2, Calculate the HE single-mode prediction probability and IHC single-mode prediction probability Relative entropy loss: (18) In equation (18), It is a relative entropy function.

7. A method for predicting HER2 scores in breast cancer, characterized in that, The prediction method is to perform prediction using the prediction model constructed by the construction method described in any one of claims 1-6. The specific process is as follows: S1. Obtain a breast cancer pathology whole slide dataset with image category labels. The dataset contains several pairs of slides, and each pair of slides contains one HE-stained slide and one IHC-stained slide. Then, the slides in the dataset are preprocessed into blocks to obtain a set of block images. Then, the block image features of each block image in the set are extracted. S2. The block image features are used as input to the prediction model. After model processing, the global features of HE staining slices and IHC staining slices and the attention score of each block image are output. S3. Classify and predict the global features of HE-stained sections and IHC-stained sections respectively to obtain the HE single-modal prediction probability and IHC single-modal prediction probability. Then, fuse the global features of HE-stained sections and IHC-stained sections to obtain multimodal fused features. Finally, classify and predict the multimodal fused features to obtain the multimodal prediction probability. S4. Obtain the HER2 score for breast cancer based on the HE single-modal prediction probability, IHC single-modal prediction probability, and multimodal prediction probability.

Images