Pathological whole-slide image classification method based on multi-branch independent mask and dirichlet evidence fusion
By employing a multi-branch independent masking method combined with Dirichlet evidence fusion, the problems of excessive attention and insufficient feature diversity in the classification of whole pathological slide images are addressed, achieving higher classification accuracy and robustness, and making it suitable for classification tasks of whole pathological slide images.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- KUNMING UNIV OF SCI & TECH
- Filing Date
- 2025-07-25
- Publication Date
- 2026-06-26
AI Technical Summary
In weakly supervised classification tasks of whole pathological slide images, there are problems of over-attention and insufficient feature diversity, which affect classification accuracy and robustness.
We employ a multi-branch independent masking and Dirichlet evidence fusion method. By constructing multiple independent attention branches, we guide each branch to focus on different regional features. Based on Dirichlet distribution and Dempster-Shafer evidence theory, we dynamically fuse the attention branches and adjust their weights to improve feature diversity and classification accuracy.
It improves the accuracy and robustness of whole-section pathology image classification, enhances the focus on different pathological regions, and improves the generalization ability of classification results.
Smart Images
Figure CN120912969B_ABST
Abstract
Description
Technical Field
[0001] This invention provides a classification method for whole-slide images based on multi-branch independent masking and Dirichlet evidence fusion, which relates to the interdisciplinary fields of bioinformatics and artificial intelligence, specifically a weakly supervised classification method for whole-slide images (WSI). Background Technology
[0002] Whole-slice images (WSI) are large-size images (typically hundreds of thousands of pixels × hundreds of thousands of pixels) obtained by high-resolution scanning of tissue sections using a digital pathology scanner. They contain rich histological features (such as cell morphology and structural distribution) and are the gold standard for cancer diagnosis and classification. However, weakly supervised classification tasks using WSI face two major challenges: high data annotation costs and excessive attention concentration. Therefore, there is an urgent need for a weakly supervised WSI classification method that can enhance attention diversity and dynamically fuse multi-branch predictions. Summary of the Invention
[0003] The technical problem to be solved by the present invention is: the present invention provides a pathological whole slide image classification method based on multi-branch independent mask and Dirichlet evidence fusion, which is used to solve the problems of excessive attention and static fusion defects in traditional multi-instance learning methods in weakly supervised classification tasks of pathological whole slide images, as well as the problem of insufficient feature diversity caused by attention concentration. The present invention improves the accuracy and robustness of WSI classification.
[0004] The technical solution of this invention is: a pathological whole-slice image classification method based on multi-branch independent masking and Dirichlet evidence fusion. By constructing multiple independent attention branches and introducing differentiated dynamic masking strategies to them, each branch is guided to focus on different regional features in the pathological image, thereby improving the diversity of feature expression. Furthermore, based on the Dirichlet distribution, the confidence and uncertainty of the prediction results of each branch are modeled, and a conflict factor mechanism is designed in combination with Dempster-Shafer evidence theory to realize dynamic weighted fusion of multi-branch results and obtain the final classification result.
[0005] The method specifically includes:
[0006] First, the input whole-slice pathology image (WSI) is divided into multiple image patches (instances). Then, the pathological features are extracted using a pre-trained backbone model and passed to multiple classifiers through different masking strategies to output the predicted probabilities of each branch. Finally, the results of different branches are fused through dynamic evidence fusion to output the final slice classification result.
[0007] Furthermore, the method includes the following steps:
[0008] Step 1: Divide the input pathological whole slice image into multiple image blocks, and extract the local features of each instance through a shared feature extractor;
[0009] Step 2: Generate a basic attention distribution based on the gating attention mechanism. Dynamically mask the attention weights by setting the Top-K value and masking probability independently for each attention branch, and guide different attention branches to focus on differentiated pathological areas.
[0010] Step 3: Based on the attention weights after masking, the instance features are weighted and aggregated to obtain the slice-level feature representation of each branch, and the predicted probability of each attention branch is output through the classifier.
[0011] Step 4: Convert the predicted probabilities of each attention branch into Dirichlet distribution parameters, quantify the confidence quality and uncertainty, calculate the conflict factor based on Dempster-Shafer evidence theory, dynamically adjust the fusion weight of attention branches, and output the final slice-level classification results.
[0012] Further, Step 1 includes:
[0013] The pathological whole-slice image is divided into fixed-size image blocks with a fixed stride. Each image block is considered as an instance. After removing the background block, a set of instances is obtained. A pre-trained convolutional neural network (such as ResNet-50) is used as a shared feature extractor to extract feature vectors for each image instance.
[0014] Furthermore, Step 2 includes:
[0015] Step 2.1: First, by combining a gating attention mechanism with bilinear transformation and nonlinear activation operations (such as hyperbolic tangent function and sigmoid function), the features of each instance are embedded to generate a basic attention weight distribution for each image instance.
[0016] Step 2.2: For each attention branch, firstly select a certain number of instances (i.e., Top-K instances) with the highest attention ranking of that attention branch according to the initialized attention weight; then, perform random masking operation on these high-attention image instances according to the masking probability set for that attention branch.
[0017] Furthermore, Step 3 includes:
[0018] For each attention branch, the feature vectors of all image instances are weighted and summed using the corresponding masked attention weights. Specifically, the information of all image blocks in the entire pathological slice is weighted and aggregated to form a global slice-level feature representation with semantic expressive power.
[0019] Furthermore, Step 4 includes:
[0020] Step 4.1: Use the predicted probabilities output by each branch as "evidence" and convert them into the parametric form of a Dirichlet distribution to quantify the confidence quality of each branch's predictions for each category and the uncertainty of the overall prediction.
[0021] Step 4.2: Introduce the concept of conflict factor from Dempster-Shafer evidence theory to quantify the predictive consistency among multiple branches;
[0022] Step 4.3: Based on the magnitude of the conflict factor, the fusion module dynamically adjusts the weight of each branch in the final classification decision. If the predictions between branches are consistent (small conflict), the output of the branch with high confidence is emphasized. If the predictions between branches are inconsistent (large conflict), the weight of the uncertain part is increased to reduce the dominant role of an unreliable branch in the final result.
[0023] Step 4.4 Finally, the fused Dirichlet distribution parameters are used to calculate the expected probability of the predicted class, thus obtaining the final slice-level classification result.
[0024] The present invention also provides a pathological whole-slice image classification system based on multi-branch independent masking and Dirichlet evidence fusion, the system comprising: a module for executing the pathological whole-slice image classification method based on multi-branch independent masking and Dirichlet evidence fusion.
[0025] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the pathological whole-slice image classification method based on multi-branch independent mask and Dirichlet evidence fusion.
[0026] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the pathological whole-slice image classification method based on multi-branch independent mask and Dirichlet evidence fusion.
[0027] The beneficial effects of this invention are:
[0028] 1. This invention uses a multi-branch independent masking strategy (MIM) to force different branches to focus on differentiated pathological regions (e.g., branch 1 focuses on the cell nucleus, branch 2 focuses on the matrix, and branch 3 focuses on blood vessels), which solves the problem of excessive attention in the traditional MIL method and improves the diversity and generalization ability of feature representation.
[0029] 2. The present invention uses a DMF module based on Dirichlet distribution and Dempster-Shafer evidence theory to quantify the confidence and uncertainty of multi-branch prediction, dynamically adjust the fusion weights, avoid the excessive dependence of static average fusion on unreliable branches, and enhance the robustness of classification results.
[0030] 3. Experimental results on publicly available datasets such as CAMELYON-16 (breast cancer metastasis detection) and TCGA-BRCA (breast cancer subtype classification) show that, compared with 10 mainstream pathological image classification methods, the method of this invention achieves the best performance in terms of F1 macro-average score, AUC macro-average score, etc., and has good generalization ability and practical application value. Attached Figure Description
[0031] Figure 1 This is a structural diagram of the overall method model in this invention. Detailed Implementation
[0032] Example 1, such as Figure 1 As shown, a pathological whole-section image classification method based on multi-branch independent masking and Dirichlet evidence fusion includes the following steps:
[0033] Step 1: Divide the input pathological whole slice image into multiple image blocks, and extract the local features of each instance through a shared feature extractor;
[0034] Step 2: Generate a basic attention distribution based on the gating attention mechanism. Dynamically mask the attention weights by setting the Top-K value and masking probability independently for each attention branch, and guide different attention branches to focus on differentiated pathological areas.
[0035] Step 3: Based on the attention weights after masking, the instance features are weighted and aggregated to obtain the slice-level feature representation of each branch, and the predicted probability of each attention branch is output through the classifier.
[0036] Step 4: Convert the predicted probabilities of each attention branch into Dirichlet distribution parameters, quantify the confidence quality and uncertainty, calculate the conflict factor based on Dempster-Shafer evidence theory, dynamically adjust the fusion weight of attention branches, and output the final slice-level classification results.
[0037] Further, Step 1 includes:
[0038] The pathological whole-slice image is divided into image blocks of fixed size with a fixed stride. Each image block is treated as an instance. After removing the background block, a set of instances is obtained. A pre-trained convolutional neural network is used as a shared feature extractor to extract feature vectors for each image instance.
[0039] Furthermore, Step 2 includes:
[0040] Step 2.1: First, by combining the gating attention mechanism with bilinear transformation and nonlinear activation operation, the features of each instance are embedded to generate a basic attention weight distribution for each image instance.
[0041] Step 2.2: For each attention branch, firstly select a certain number of instances with the highest attention ranking based on the initialized attention weight; then, perform random masking operations on these high-attention image instances according to the masking probability set for the attention branch.
[0042] Furthermore, Step 3 includes:
[0043] For each attention branch, the feature vectors of all image instances are weighted and summed using the corresponding masked attention weights. Specifically, the information of all image blocks in the entire pathological slice is weighted and aggregated to form a global slice-level feature representation with semantic expressive power.
[0044] Furthermore, Step 4 includes:
[0045] Step 4.1: Use the predicted probabilities of each branch as "evidence" and convert them into the parametric form of a Dirichlet distribution;
[0046] Step 4.2: Introduce the concept of conflict factor from Dempster-Shafer evidence theory to quantify the predictive consistency among multiple branches;
[0047] Step 4.3: Based on the magnitude of the conflict factor, the fusion module dynamically adjusts the weight of each branch in the final classification decision; if the predictions between branches are consistent, the output of the branch with higher confidence is emphasized; if the predictions between branches are inconsistent, the weighting of the uncertain part is increased.
[0048] Step 4.4 Finally, the fused Dirichlet distribution parameters are used to calculate the expected probability of the predicted class, thus obtaining the final slice-level classification result.
[0049] The present invention also provides a pathological whole-slice image classification system based on multi-branch independent masking and Dirichlet evidence fusion, the system comprising: a module for executing the pathological whole-slice image classification method based on multi-branch independent masking and Dirichlet evidence fusion.
[0050] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the pathological whole-slice image classification method based on multi-branch independent mask and Dirichlet evidence fusion.
[0051] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the pathological whole-slice image classification method based on multi-branch independent mask and Dirichlet evidence fusion.
[0052] Example 2, as follows Figure 1 As shown, a pathological whole-slice image classification method based on multi-branch independent masking and Dirichlet evidence fusion is described. The specific steps of this method are as follows:
[0053] Step 1: First, the input whole-slice pathology image (WSI) is segmented into small patches, and the semantic features of each patch are extracted. A single whole-slice image is then sliced using a sliding window at a fixed size (e.g., 224×224) and step size (e.g., 112 pixels) to obtain several small image patches, each patch being considered an "instance". Then, background detection is performed on each patch using methods such as Otsu thresholding to remove blank areas, retaining only valid patches containing tissue structures, forming an instance set {x1, x2, ..., x...}. N} where N is the number of valid image patches; in the feature extraction stage, each image patch is input into a pre-trained CNN network (such as ResNet-50), and its convolutional layer output is taken as the feature vector, and each image patch x i Encoded as z i =ResNet(x i )∈R d Where d is the feature dimension.
[0054] Step 2: First, a gated attention mechanism combined with bilinear transformation and nonlinear activation operations is used to embed the features of each instance, generating a basic attention weight distribution for each image instance; specifically including:
[0055] The importance of each instance is calculated using a gated attention mechanism to guide the model in focusing on key regions. The formula is as follows:
[0056]
[0057] The specific operation involves processing each patch feature z... i Two linear layers V1, V2∈R are used respectively. d×hThe vector is mapped to a latent space of dimension h, then one side is passed through the tanh function and the other through σ, followed by element-wise multiplication to form a gating mechanism. Finally, the vector w∈R is used. h The importance score is calculated, normalized to obtain the attention probability, and finally the attention matrix is obtained.
[0058] For each attention branch, firstly, a certain number of instances with high attention ranking for that attention branch are selected based on the initialized attention weights; then, according to the mask probability set for that attention branch, a random masking operation is performed on these high-attention image instances; specifically, this includes:
[0059] To prevent all branches from over-concentrating on the same highly significant region and to enhance attention diversity, we set a Top-K for each branch j = 1, 2, ..., M. j Value and mask rate p j Then in each branch, starting from the initial attention Select Top-K j The set of instances corresponding to the maximum value. With probability p j Set the attention weights of these Top-K instances to 0 to form the attention weights after masking. The formula is:
[0060]
[0061] Step 3: Multi-branch prediction stage. For each branch j, use the attention weight of that branch. For all instance features z i Weighted aggregation yields slice-level feature representations:
[0062]
[0063] Subsequently, the aggregation feature h (j) Input to the classifier f corresponding to the branch j (·)(composed of a multilayer perceptron), output class probability distribution:
[0064] y (j) =f j (h (j) )∈R C
[0065] Where C is the number of categories in the classification task.
[0066] Step 4: Calculate the prediction results for each branch j obtained in Step 3. The probability of class k is used as evidence to construct the Dirichlet parameters. and calculate concentration parameters
[0067]
[0068] The confidence mass was then derived using Dirichlet parameters and calculated concentration parameters. and uncertainty u (j) :
[0069]
[0070] To measure the consistency or degree of conflict between prediction results from different branches, a conflict factor calculation function was added. Taking two branches as an example, the formula for calculating the conflict factor C is as follows:
[0071]
[0072] The greater the conflict, the greater the difference in predictions between branches. Therefore, the model adjusts the influence of each branch on the final result based on the conflict factor. First, a conflict adjustment coefficient λ∈[0,1] is set, and then the previously obtained confidence quality is considered. With uncertainty u (j) To merge:
[0073]
[0074] The final classification result has the following probabilities for each class:
[0075]
[0076] The probability p obtained k This represents the model's final prediction of whether the entire WSI image belongs to category k.
[0077] This invention evaluates the performance of the proposed model in a pathological image classification task. Experiments were conducted on several publicly available whole-slice pathological image datasets to verify the adaptability and generalization ability of the proposed method across different pathological tasks. The CAMELYON16 dataset contains 400 breast lymph node slice images (270 for training and 130 for testing), primarily used for binary classification detection of breast cancer lymph node metastasis; the BRACS dataset, annotated by medical experts, contains 500 breast cancer pathological slice images and provides multi-level labels to evaluate the model's discriminative ability in real-world clinical scenarios; the TCGA-LUNG dataset contains over 1000 lung cancer pathological slice images from TCGA, covering subtypes such as lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC), used to test the model's generalization performance in cross-cancer classification tasks. All datasets employ a weakly supervised learning setting, providing only slice-level labels and omitting pixel-level or instance-level annotation information.
[0078] Table 1 shows detailed statistics for the dataset.
[0079]
[0080] The experiment used two mainstream metrics, Macro-average AUC and Macro-average F1-score, as evaluation standards. Macro-average AUC (area under the curve) calculates the AUC value for each class and then averages them, effectively mitigating the impact of class imbalance and measuring the model's overall discriminative ability across classes. This metric is particularly suitable for multi-class classification tasks where there are differences in the number of samples in whole-slice images. Macro-average F1-score, similarly calculated based on the F1 score of each class, reflects the balance between classification precision and recall across different classes. It is especially suitable for evaluating the model's ability to identify minority classes (such as rare cancer subtypes), ensuring that the classifier does not favor the dominant class and ignore clinically important marginal types.
[0081] All the above indicators are reported using the mean and standard deviation of five independent experiments to ensure that the evaluation results have good stability and statistical significance.
[0082] To verify the effectiveness of the method of the present invention, it was compared with 10 current mainstream pathological image classification methods, including Max-pooling, Mean-pooling, Clam-SB, TransMIL, DSMIL, DTFD-MIL, IBMIL, MHIM-MIL, ABMIL, and ACMIL.
[0083] Table 2 shows the comparative experimental results on the BRCA and HNSC datasets.
[0084]
[0085]
[0086] As shown in Table 2, the present invention achieves optimal performance on all three datasets. Taking CAMELYON-16 as an example, the method of the present invention comprehensively outperforms other methods in both the macro-average F1 score (95.9%) and the macro-average area under the curve (98.3%). Compared with the second-best ACMIL model (AUC of 97.40%), it improves the macro-average area under the curve by 1.1 percentage points. On other datasets, the method of the present invention also achieves macro-average areas under the curve of 89% and 96.5% on BRACS and TCGA-LUNG, respectively, consistently maintaining its leading position. The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments, and various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.
Claims
1. A pathological whole-section image classification method based on multi-branch independent masking and Dirichlet evidence fusion, characterized in that: First, the input pathological whole slice image is divided into multiple image patches. Then, the pathological image features are extracted through a pre-trained backbone model and passed to multiple classifiers through different masking strategies to output the predicted probability of each branch. Finally, the results of different branches are fused through dynamic evidence fusion to output the final slice classification result. The method includes the following steps: Step 1: Divide the input pathological whole slice image into multiple image blocks, and extract the local features of each instance through a shared feature extractor; Step 2: Generate a basic attention distribution based on the gating attention mechanism. Dynamically mask the attention weights by setting the Top-K value and masking probability independently for each attention branch, and guide different attention branches to focus on differentiated pathological areas. Step 3: Based on the attention weights after masking, the instance features are weighted and aggregated to obtain the slice-level feature representation of each branch, and the predicted probability of each attention branch is output through the classifier. Step 4: Convert the predicted probabilities of each attention branch into Dirichlet distribution parameters, quantify the confidence quality and uncertainty, calculate the conflict factor based on Dempster-Shafer evidence theory, dynamically adjust the fusion weight of attention branches, and output the final slice-level classification results. Step 2 includes: Step 2.1: First, by combining the gating attention mechanism with bilinear transformation and nonlinear activation operation, the features of each instance are embedded to generate a basic attention weight distribution for each image instance. Step 2.2: For each attention branch, firstly select a certain number of instances with the highest attention ranking based on the initialized attention weight; then, perform random masking operations on these high-attention image instances according to the masking probability set for the attention branch.
2. The pathological whole-section image classification method based on multi-branch independent masking and Dirichlet evidence fusion as described in claim 1, characterized in that, Step 1 includes: The pathological whole-slice image is divided into image blocks of fixed size with a fixed stride. Each image block is treated as an instance. After removing the background block, a set of instances is obtained. A pre-trained convolutional neural network is used as a shared feature extractor to extract feature vectors for each image instance.
3. The pathological whole-section image classification method based on multi-branch independent masking and Dirichlet evidence fusion according to claim 1, characterized in that, Step 3 includes: For each attention branch, the feature vectors of all image instances are weighted and summed using the corresponding masked attention weights. Specifically, the information of all image blocks in the entire pathological slice is weighted and aggregated to form a global slice-level feature representation with semantic expressive power.
4. The pathological whole-slice image classification method based on multi-branch independent masking and Dirichlet evidence fusion according to claim 1, characterized in that, Step 4 includes: Step 4.1: Use the predicted probabilities output by each branch as "evidence" and convert them into the parametric form of a Dirichlet distribution; Step 4.2: Introduce the concept of conflict factor from Dempster-Shafer evidence theory to quantify the predictive consistency among multiple branches; Step 4.3: Based on the magnitude of the conflict factor, the fusion module dynamically adjusts the weight of each branch in the final classification decision; if the predictions between branches are consistent, the output of the branch with higher confidence is emphasized; if the predictions between branches are inconsistent, the weighting of the uncertain part is increased. Step 4.4 Finally, the fused Dirichlet distribution parameters are used to calculate the expected probability of the predicted class, thus obtaining the final slice-level classification result.
5. A pathological whole-slice image classification system based on multi-branch independent masking and Dirichlet evidence fusion, characterized in that, The system includes a module for performing the pathological whole-slice image classification method based on multi-branch independent masking and Dirichlet evidence fusion as described in any one of claims 1 to 4.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the pathological whole-slice image classification method based on multi-branch independent mask and Dirichlet evidence fusion as described in any one of claims 1 to 4.
7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the pathological whole-slice image classification method based on multi-branch independent mask and Dirichlet evidence fusion as described in any one of claims 1 to 4.