Pathological full-slice image classification method based on multi-branch independent mask and Dirichlet evidence fusion

Abstract

The invention relates to a pathological full-slice image classification method based on multi-branch independent mask and Dirichlet evidence fusion, and belongs to the crossing field of biological information and artificial intelligence. Aiming at the problems of excessive attention concentration and static fusion defects of a traditional multi-instance learning method in a weak supervision classification task of a pathological full-slice image, dynamic mask parameters are independently set through multiple branches, different branches are forced to pay attention to differentiated pathological areas, and the problem of insufficient feature diversity caused by attention concentration is solved; dirichlet distribution is combined to quantify the confidence and uncertainty of branch prediction, the branch fusion weight is dynamically adjusted based on the evidence theory, and the fusion robustness of a multi-branch prediction result is improved. Experiments prove that compared with an MIL method, the method disclosed by the invention has the advantages that the AUC index is improved by 1.1-2.4%, and the accuracy and generalization ability of pathological WSI classification are remarkably enhanced.

Description

TECHNICAL FIELD

[0001] The application provides a pathological whole slide image classification method based on multi-branch independent mask and Dirichlet evidence fusion, relates to the cross technical field of biological information and artificial intelligence, and particularly relates to a weakly supervised classification method for pathological whole slide images (WSI). BACKGROUND

[0002] A pathological whole slide image (WSI) is a large-size image (usually tens of thousands of pixels x tens of thousands of pixels) obtained by high-resolution scanning of a tissue slice through a digital pathology scanner, which contains rich histological features (such as cell morphology, structure distribution, etc.), and is the gold standard for cancer diagnosis and typing. However, the weakly supervised classification task of WSI faces two major challenges of high data labeling cost and excessive attention concentration. Therefore, there is an urgent need for a weakly supervised WSI classification method that can enhance attention diversity and dynamically fuse multi-branch predictions. SUMMARY

[0003] The technical problem to be solved by the application is that the application provides a pathological whole slide image classification method based on multi-branch independent mask and Dirichlet evidence fusion, which is used to solve the problems of attention over-concentration and static fusion defects in the traditional multi-instance learning method for the weakly supervised classification task of pathological whole slide images, and the problem of insufficient feature diversity caused by attention concentration, and improves the accuracy and robustness of WSI classification.

[0004] The technical solution of the application is a pathological whole slide image classification method based on multi-branch independent mask and Dirichlet evidence fusion, which constructs multiple independent attention branches and introduces a differentiated dynamic mask strategy to guide each branch to focus on different regional features in the pathological image, thereby improving the diversity of feature expression; further based on Dirichlet distribution, the confidence and uncertainty of the prediction results of each branch are modeled, and a conflict factor mechanism is designed combining Dempster-Shafer evidence theory to realize dynamic weighted fusion of the results of multiple branches and obtain the final classification result.

[0005] The method specifically includes:

[0006] First, the input pathological whole slide image (WSI) is divided into multiple image blocks (instances), then the pathological image features are extracted through a pre-trained backbone model and are respectively transmitted to multiple classifiers through different mask strategies to output the prediction probabilities of each branch; finally, the results of different branches are fused through dynamic evidence to output the final slide classification result.

[0007] Further, the method includes the following steps:

[0008] Step 1: Divide the input pathological whole slice image into multiple image blocks, and extract the local features of each instance through a shared feature extractor;

[0009] Step 2: Generate a basic attention distribution based on the gating attention mechanism. Dynamically mask the attention weights by setting the Top-K value and masking probability independently for each attention branch, and guide different attention branches to focus on differentiated pathological areas.

[0010] Step 3: Based on the attention weights after masking, the instance features are weighted and aggregated to obtain the slice-level feature representation of each branch, and the predicted probability of each attention branch is output through the classifier.

[0011] Step 4: Convert the predicted probabilities of each attention branch into Dirichlet distribution parameters, quantify the confidence quality and uncertainty, calculate the conflict factor based on Dempster-Shafer evidence theory, dynamically adjust the fusion weight of attention branches, and output the final slice-level classification results.

[0012] Further, Step 1 includes:

[0013] The pathological whole-slice image is divided into fixed-size image blocks with a fixed stride. Each image block is considered as an instance. After removing the background block, a set of instances is obtained. A pre-trained convolutional neural network (such as ResNet-50) is used as a shared feature extractor to extract feature vectors for each image instance.

[0014] Furthermore, Step 2 includes:

[0015] Step 2.1: First, by combining a gating attention mechanism with bilinear transformation and nonlinear activation operations (such as hyperbolic tangent function and sigmoid function), the features of each instance are embedded to generate a basic attention weight distribution for each image instance.

[0016] Step 2.2: For each attention branch, firstly select a certain number of instances (i.e., Top-K instances) with the highest attention ranking of that attention branch according to the initialized attention weight; then, perform random masking operation on these high-attention image instances according to the masking probability set for that attention branch.

[0017] Furthermore, Step 3 includes:

[0018] For each attention branch, the feature vectors of all image instances are weighted and summed using the corresponding masked attention weights. Specifically, the information of all image blocks in the entire pathological slice is weighted and aggregated to form a global slice-level feature representation with semantic expressive power.

[0019] Further, the Step4 comprises:

[0020] Step4.1, the prediction probability output by each branch is converted into the parameter form of Dirichlet distribution as "evidence" to quantify the confidence quality of each branch to predict each category and the uncertainty of the overall prediction;

[0021] Step4.2, the conflict factor concept in Dempster-Shafer evidence theory is introduced to quantify the prediction consistency between multiple branches;

[0022] Step4.3, according to the size of the conflict factor, the fusion module dynamically adjusts the weight of each branch in the final classification decision; if the branches are consistent in prediction (small conflict), the branches with high confidence are focused on; if the branches are inconsistent in prediction (large conflict), the weighting of the uncertainty part is increased to reduce the dominant role of an unreliable branch on the final result;

[0023] Step4.4, finally, the Dirichlet distribution parameters after fusion are used to calculate the expected probability of the predicted category, that is, the final slice-level classification result is obtained.

[0024] The application also provides a pathological whole slice image classification system based on multi-branch independent mask and Dirichlet evidence fusion, which comprises a module for executing the pathological whole slice image classification method based on multi-branch independent mask and Dirichlet evidence fusion.

[0025] The application also provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to realize the pathological whole slice image classification method based on multi-branch independent mask and Dirichlet evidence fusion.

[0026] The application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to realize the pathological whole slice image classification method based on multi-branch independent mask and Dirichlet evidence fusion.

[0027] The application has the following beneficial effects:

[0028] 1. The application uses a multi-branch independent mask strategy (MIM) to force different branches to focus on different pathological areas (such as branch 1 focusing on cell nuclei, branch 2 focusing on stroma, and branch 3 focusing on blood vessels), which solves the problem of excessive attention of traditional MIL methods and improves the diversity and generalization ability of feature representation.

[0029] 2. The present invention uses a DMF module based on Dirichlet distribution and Dempster-Shafer evidence theory to quantify the confidence and uncertainty of multi-branch prediction, dynamically adjust the fusion weights, avoid the excessive dependence of static average fusion on unreliable branches, and enhance the robustness of classification results.

[0030] 3. Experimental results on publicly available datasets such as CAMELYON-16 (breast cancer metastasis detection) and TCGA-BRCA (breast cancer subtype classification) show that, compared with 10 mainstream pathological image classification methods, the method of this invention achieves the best performance in terms of F1 macro-average score, AUC macro-average score, etc., and has good generalization ability and practical application value. Attached Figure Description

[0031] Figure 1 This is a structural diagram of the overall method model in this invention. Detailed Implementation

[0032] Example 1, such as Figure 1 As shown, a pathological whole-section image classification method based on multi-branch independent masking and Dirichlet evidence fusion includes the following steps:

[0033] Step 1: Divide the input pathological whole slice image into multiple image blocks, and extract the local features of each instance through a shared feature extractor;

[0034] Step 2: Generate a basic attention distribution based on the gating attention mechanism. Dynamically mask the attention weights by setting the Top-K value and masking probability independently for each attention branch, and guide different attention branches to focus on differentiated pathological areas.

[0035] Step 3: Based on the attention weights after masking, the instance features are weighted and aggregated to obtain the slice-level feature representation of each branch, and the predicted probability of each attention branch is output through the classifier.

[0036] Step 4: Convert the predicted probabilities of each attention branch into Dirichlet distribution parameters, quantify the confidence quality and uncertainty, calculate the conflict factor based on Dempster-Shafer evidence theory, dynamically adjust the fusion weight of attention branches, and output the final slice-level classification results.

[0037] Further, Step 1 includes:

[0038] The pathological whole-slice image is divided into image blocks of fixed size with a fixed stride. Each image block is treated as an instance. After removing the background block, a set of instances is obtained. A pre-trained convolutional neural network is used as a shared feature extractor to extract feature vectors for each image instance.

[0039] Further, the Step2 comprises:

[0040] Step2.1, first, the feature of each instance is embedded by a gating attention mechanism combined with a bilinear transformation and a nonlinear activation operation to generate a basic attention weight distribution for each image instance;

[0041] Step2.2, for each attention branch, first, a certain number of instances with high attention degrees are selected according to the initialized attention weights; then, a random mask operation is performed on the high-attention-degree image instances according to the mask probability set by the attention branch.

[0042] Further, the Step3 comprises:

[0043] For each attention branch, the corresponding masked attention weight is used to perform weighted summation operation on the feature vectors of all image instances; specifically, the information of all image blocks in the whole pathological section is weighted and aggregated to form a global section-level feature representation with semantic expression ability.

[0044] Further, the Step4 comprises:

[0045] Step4.1, the prediction probability output by each branch is converted into the parameter form of Dirichlet distribution as "evidence";

[0046] Step4.2, the conflict factor concept in Dempster-Shafer evidence theory is introduced to quantify the prediction consistency between multiple branches;

[0047] Step4.3, according to the size of the conflict factor, the fusion module dynamically adjusts the weight of each branch in the final classification decision; if the branches are consistent in prediction, the branch with high confidence is emphasized; if the branches are inconsistent in prediction, the weighting of the uncertain part is increased;

[0048] Step4.4, finally, the fused Dirichlet distribution parameters are used to calculate the expected probability of the prediction category, that is, the final section-level classification result is obtained.

[0049] The application also provides a pathological whole section image classification system based on multi-branch independent mask and Dirichlet evidence fusion, the system comprising: a module for executing the pathological whole section image classification method based on multi-branch independent mask and Dirichlet evidence fusion.

[0050] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the pathological whole-slice image classification method based on multi-branch independent mask and Dirichlet evidence fusion.

[0051] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the pathological whole-slice image classification method based on multi-branch independent mask and Dirichlet evidence fusion.

[0052] Example 2, as follows Figure 1 As shown, a pathological whole-slice image classification method based on multi-branch independent masking and Dirichlet evidence fusion is described. The specific steps of this method are as follows:

[0053] Step 1: First, the input whole-slice pathology image (WSI) is segmented into small patches, and the semantic features of each patch are extracted. A single whole-slice image is then sliced ​​using a sliding window at a fixed size (e.g., 224×224) and step size (e.g., 112 pixels) to obtain several small image patches, each patch being considered an "instance". Then, background detection is performed on each patch using methods such as Otsu thresholding to remove blank areas, retaining only valid patches containing tissue structures, forming an instance set {x1, x2, ..., x...}. N} where N is the number of valid image patches; in the feature extraction stage, each image patch is input into a pre-trained CNN network (such as ResNet-50), and its convolutional layer output is taken as the feature vector, and each image patch x i Encoded as z i =ResNet(x i )∈R d Where d is the feature dimension.

[0054] Step 2: First, a gated attention mechanism combined with bilinear transformation and nonlinear activation operations is used to embed the features of each instance, generating a basic attention weight distribution for each image instance; specifically including:

[0055] The importance of each instance is calculated using a gated attention mechanism to guide the model in focusing on key regions. The formula is as follows:

[0056]

[0057] The specific operation involves processing each patch feature z... i Two linear layers V1, V2∈R are used respectively. d×hThe vector is mapped to a latent space of dimension h, then one side is passed through the tanh function and the other through σ, followed by element-wise multiplication to form a gating mechanism. Finally, the vector w∈R is used. h The importance score is calculated, normalized to obtain the attention probability, and finally the attention matrix is ​​obtained.

[0058] For each attention branch, firstly, a certain number of instances with high attention ranking for that attention branch are selected based on the initialized attention weights; then, according to the mask probability set for that attention branch, a random masking operation is performed on these high-attention image instances; specifically, this includes:

[0059] To prevent all branches from over-concentrating on the same highly significant region and to enhance attention diversity, we set a Top-K for each branch j = 1, 2, ..., M. j Value and mask rate p j Then in each branch, starting from the initial attention Select Top-K j The set of instances corresponding to the maximum value. With probability p j Set the attention weights of these Top-K instances to 0 to form the attention weights after masking. The formula is:

[0060]

[0061] Step 3: Multi-branch prediction stage. For each branch j, use the attention weight of that branch. For all instance features z i Weighted aggregation yields slice-level feature representations:

[0062]

[0063] Subsequently, the aggregation feature h (j) Input to the classifier f corresponding to the branch j (·)(composed of a multilayer perceptron), output class probability distribution:

[0064] y (j) =f j (h (j) )∈R C

[0065] Where C is the number of categories in the classification task.

[0066] Step 4: Calculate the prediction results for each branch j obtained in Step 3. The probability of class k is used as evidence to construct the Dirichlet parameters. and calculate concentration parameters

[0067]

[0068] Subsequently, the Dirichlet parameter and the concentration parameter are used to derive the confidence quality and the uncertainty u (j) :

[0069]

[0070] In order to measure the consistency or conflict degree between different branch prediction results. A conflict factor calculation function is added, taking two branches as an example, the calculation formula of the conflict factor C is:

[0071]

[0072] The greater the conflict, the greater the difference between the branches. Therefore, the model adjusts the influence of each branch on the final result according to the conflict factor. First, set a conflict adjustment coefficient λ∈[0,1], and then combine the confidence quality and the uncertainty u (j) :

[0073]

[0074] Finally, the probability of each class of the final classification result is:

[0075]

[0076] The probability p k obtained is the final prediction result of the model that the WSI image belongs to class k.

[0077] The performance of the proposed model is evaluated in the pathological image classification task. The experiment is based on multiple public pathological whole section image datasets to verify the adaptability and generalization ability of the proposed method in different pathological tasks. Among them, the CAMELYON16 dataset contains 400 breast lymph node section images (270 for training and 130 for testing), mainly used for breast cancer lymph node metastasis binary classification detection task; The BRACS dataset is annotated by medical experts and contains 500 breast cancer pathological section images, and provides multi-level labels, which is used to evaluate the discrimination ability of the model in actual clinical scenarios; The TCGA-LUNG dataset contains more than 1000 lung cancer pathological section images from TCGA, covering subtypes such as lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC), which is used to test the generalization performance of the model in cross-cancer classification tasks. All datasets use weakly supervised learning settings, only provide section-level labels, and do not use pixel-level or instance-level annotation information.

[0078] Table 1 is a detailed statistics of the data set

[0079]

[0080] In the experiment, two mainstream indicators, Macro-average AUC and Macro-average F1-score, are selected as evaluation criteria. Macro-average AUC (macro-average area under the curve) can effectively alleviate the influence of class imbalance by calculating the AUC value of each class and taking the average, and can measure the overall discrimination ability of the model on each class. This indicator is particularly suitable for multi-classification tasks with sample quantity differences in whole slice images. Macro-average F1-score (macro-average F1 value) also calculates the average based on the F1 score of each class, reflecting the balance between classification accuracy and recall rate of the model on different classes, especially suitable for evaluating the recognition ability of the model for minority classes (such as rare cancer subtypes), and ensuring that the classifier will not be biased towards the dominant class and ignore the marginal types that are critical to clinical.

[0081] The above indicators use the average and standard deviation of five independent experiments to report, ensuring that the evaluation results have good stability and statistical significance.

[0082] To verify the effectiveness of the method, it is compared with 10 current mainstream pathological image classification methods, including support maximum pooling (Max-pooling), average pooling (Mean-pooling), Clam-SB, TransMIL, DSMIL, DTFD-MIL, IBMIL, MHIM-MIL, ABMIL and ACMIL.

[0083] Table 2 is the comparison experiment results on BRCA and HNSC data sets

[0084]

[0085]

[0086] From Table 2, it can be seen that the present application achieves the optimal performance on three types of data sets. Taking CAMELYON-16 as an example, the present application method comprehensively leads other methods in two indicators of macro-average F1 value (95.9%) and macro-average area under the curve (98.3%). Compared with the second best ACMIL model (AUC is 97.40%), it is improved by 1.1 percentage points in the macro-average area under the curve. On other data sets, the present application method also reaches 89% and 96.5% of the macro-average area under the curve on BRACS and TCGA-LUNG, respectively, and always maintains the leading position. The above describes the specific embodiments of the present application in detail in combination with the drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the purpose of the present application.

Claims

1. A pathological whole-slide image classification method based on multi-branch independent mask and Dirichlet evidence fusion, characterized in that: Firstly, the input pathological whole slice image is divided into multiple image blocks, and then the pathological image features are extracted through the pre-trained backbone model, and then the features are input into multiple classifiers through different mask strategies to output the prediction probability of each branch.

2. The pathological whole-slide image classification method based on multi-branch independent mask and Dirichlet evidence fusion according to claim 1, characterized in that, The method comprises the following steps: Step 1, dividing the input pathological whole slice image into multiple image blocks, and extracting local features of each instance through a shared feature extractor; Step 2, generating a basic attention distribution based on a gating attention mechanism, and dynamically masking the attention weight through the Top-K value and the mask probability independently set by each attention branch to guide different attention branches to focus on different pathological regions; Step 3, weighting and aggregating the instance features based on the masked attention weight to obtain the slice-level feature representation of each branch, and outputting the prediction probability of each attention branch through a classifier; Step 4, converting the prediction probability of each attention branch into Dirichlet distribution parameters to quantify the confidence quality and uncertainty, calculating the conflict factor based on Dempster-Shafer evidence theory, dynamically adjusting the fusion weight of the attention branch, and outputting the final slice-level classification result.

3. The pathological whole-slide image classification method based on multi-branch independent mask and Dirichlet evidence fusion according to claim 2, characterized in that, The Step 1 comprises: Divide the pathological whole slice image into image blocks of a fixed size according to a fixed step length, and each image block is an instance. After removing the background blocks, an instance set is obtained. A pre-trained convolutional neural network is used as a shared feature extractor to extract a feature vector for each image instance.

4. The pathological whole-slide image classification method based on multi-branch independent mask and Dirichlet evidence fusion according to claim 2, characterized in that, The Step 2 comprises: Step 2.1, first, the feature of each instance is embedded by the gating attention mechanism combined with bilinear transformation and nonlinear activation operation to generate a basic attention weight distribution for each image instance; Step 2.2, for each attention branch, first, a certain number of instances with high attention degree are selected according to the initialized attention weight; then, random masking operation is performed on these high-attention-degree image instances according to the mask probability set by the attention branch.

5. The pathological whole-slide image classification method based on multi-branch independent mask and Dirichlet evidence fusion according to claim 2, characterized in that, The Step 3 comprises: For each attention branch, the feature vectors of all image instances are weighted and summed using the corresponding masked attention weight; specifically, the information of all image blocks in the whole pathological slice is weighted and aggregated to form a global slice-level feature representation with semantic expression ability.

6. The pathological whole-slide image classification method based on multi-branch independent mask and Dirichlet evidence fusion according to claim 2, characterized in that, The Step 4 comprises: Step 4.1, the prediction probability output by each branch is converted into the parameter form of Dirichlet distribution as "evidence"; Step 4.2, introducing the concept of conflict factor in Dempster-Shafer evidence theory to quantify the prediction consistency between multiple branches; Step 4.3, according to the size of the conflict factor, the fusion module dynamically adjusts the weight of each branch in the final classification decision; if the branches are consistent in prediction, the branch with high confidence is emphasized; if the branches are inconsistent in prediction, the weighting of the uncertain part is increased. Step 4.4, Finally, the fused Dirichlet distribution parameters are used to calculate the expected probability of the predicted class, i.e. to obtain the final slice-level classification results.

7. A pathological whole-slide image classification system based on multi-branch independent mask and Dirichlet evidence fusion, characterized in that, The system comprises a module for performing the pathological whole-slide image classification method based on multi-branch independent mask and Dirichlet evidence fusion according to any one of claims 1 to 6.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the pathological whole-slide image classification method based on multi-branch independent mask and Dirichlet evidence fusion according to any one of claims 1 to 6 when executing the program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the pathological whole-slide image classification method based on multi-branch independent mask and Dirichlet evidence fusion according to any one of claims 1 to 6 when executed by the processor.

Images