A mask-based digital pathology image classification method

Abstract

The present invention proposes a mask-based digital pathology image classification method. The MHIM-MIL framework adopts a Siamese structure, i.e., a twin structure, in the training stage. The main component of the framework is a MIL model based on general attention, called the student model S(·). The student model is used to aggregate instance features. In order to increase the discriminative ability of the student model and force it to focus on difficult-to-classify instances, this framework introduces a momentum teacher model T(·). The teacher model is mainly used to mine difficult instances, using a difficult instance mining strategy based on mask operations to cover simple instances, while retaining difficult instances as input to the student model. The Siamese structure can make model training more stable and efficient, and has fewer training parameters. After mining the difficult instances, all mined instance features are input into the student model to infer the package labels.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a multi-instance learning method for mining difficult instances for digital pathology image classification. Background Art

[0002] Due to the large size of digital pathology images and the lack of pixel-level annotation, existing research often uses multiple instance learning (MIL) frameworks to solve the problem of digital pathology image classification. Multiple instance learning (MIL) is a weakly supervised learning framework that predicts the category information of new packets by learning from unlabeled instances. Classification methods based on MIL can be divided into instance-level methods and embedding-level methods.

[0003] Instance-level methods primarily train instance classifiers to obtain instance labels, then aggregate instance label information to predict package labels. Typical instance-level aggregation methods include max pooling and average pooling. Max pooling focuses on the most prominent information in an instance, while average pooling considers information from all instances. In pathological image analysis tasks, these two pooling methods enable the model to learn information about a small amount of discriminative samples, but also make the model susceptible to interference from negative instances, resulting in a higher risk of overfitting. Extensive existing research indicates that instance-level methods perform inferior to embedding-level methods.

[0004] Embedding-level methods aggregate instance features into package-level features, then use these features to learn a package-level classifier to obtain package-level prediction labels. The model training process can be divided into three steps: feature learning, feature aggregation, and classifier learning. Embedding-level methods primarily target the feature aggregation step, designing different instance feature aggregation operations.

[0005] Currently, embedding-level methods tend to design attention-based MIL models, applying learnable attention weights to aggregate the feature information of prominent instances. Considering that tumor regions in pathology images are relatively small, the ratio of positive instances is low, and the large amount of redundant information in the image bag can easily interfere with the model's ability to discriminate positive instances. Some MIL frameworks add additional constraints to further discover more prominent instances.

[0006] The above research demonstrates that attention-based instance feature aggregation is a powerful tool for improving the performance of multi-instance learning models. Furthermore, focusing on the most prominent instances can lead to performance gains. Another work, however, takes the opposite approach, enhancing the model's discriminative capabilities by focusing on difficult instances.

[0007] Difficult sample mining is a popular technique used to accelerate model convergence and improve model discrimination. It has been applied in many deep learning fields. The key idea behind difficult sample mining is to design a difficult sample evaluation mechanism, select samples that are difficult to correctly classify, and perform targeted processing on these samples to alleviate the imbalance between positive and negative samples and facilitate model training. Currently, methods for assessing sample difficulty can be divided into three categories: loss-based, similarity-based, and learnable weight-based.

[0008] One of the difficulties in digital pathology image classification is the small proportion of tumor regions. This makes multi-instance learning methods susceptible to interference from negative instances, which affects the representation of positive instances. To overcome this problem, instance-based mining methods can effectively screen valuable instances for classification and are of great research significance. However, due to the lack of instance-level label information, existing sample mining methods cannot be directly applied to this task. Summary of the Invention

[0009] The purpose of the present invention is to provide a mask-based digital pathology image classification method to solve the above-mentioned problems existing in the prior art.

[0010] To achieve the above object, the present invention adopts the following technical solution: a mask-based digital pathology image classification method, comprising the following steps:

[0011] S1: Obtain existing pathological images. Each pathological image is considered as a package, and the blocks obtained after image preprocessing are considered as instances. If there is an instance containing a tumor area in the package, the package is a positive package, otherwise it is a negative package. Let the dataset Represents a set of samples of known categories, where B i Indicates a package, Y i ∈C represents the label corresponding to the i-th packet, C is the class label set, and N is the number of packets in the dataset. Let B={x1,…x i ,…,x n} represents a collection of instances, is a set of instance features, where Y = {y1,…y i ,…,y n},y i ∈C is the label of the corresponding instance, and n is the number of instances in the bag.

[0012] S2: Construct a twin network structure HIM-MIL, which includes a teacher model and a student model with exactly the same structure. Both the teacher model and the student model adopt the MIL model based on general attention;

[0013] The HIM-MIL framework can be defined as:

[0014]

[0015] S3: Training of student model:

[0016] The existing pathological images are sequentially fed into the teacher model to mine difficult instances. The student model is trained using the difficult instances mined by the teacher model and its parameters are updated. When the loss of the student model no longer changes, a trained student model is obtained.

[0017] During the process of updating the student model parameters, the teacher model also updates its parameters;

[0018] S4: Detection: a pathological image to be predicted is sent to the trained student model after image preprocessing, and the output is the predicted label of the pathological image to be predicted.

[0019] As a preference, the steps for the teacher model T(·) in S3 to mine difficult instances are as follows:

[0020] S31: The complete sequence of a given instance As the input of T(·), the attention value of the instance is obtained through the attention network of T(·), as shown below:

[0021] A=[a1,…,a2,…,a n ]=T(Z) (6)

[0022] Among them, A represents the instance attention sequence, a i represents the attention value of the i-th instance, and T(Z) represents the instance attention sequence obtained by the instance feature Z through the teacher model;

[0023] S32: Then sort the instance attention sequence A and arrange it in descending order to obtain the index of the attention sequence:

[0024] I=[i1,i2,…,i n ]=Sort(A) (7)

[0025] Among them, i1 is the index of the instance with the highest attention score, i n is the index of the instance with the lowest attention score;

[0026] S33: Using the index set I, the mask-based hard case mining strategy is used to mine hard instances:

[0027] Define an n-dimensional binary vector M = [m1,…,m i ,…,m n ], a mask tag used to encode the instance, where m i ∈{0,1}. If m i =1, it means that the i-th instance is not masked; if m i=0, it means that the i-th instance is masked;

[0028] Obtain the new mask mark after masking through the mask strategy In getting After that, we can obtain the difficult instances mined by the teacher model T(·)

[0029]

[0030] in, is the number of unmasked instances.

[0031] As a preference, the new mask mark in S33 The steps to obtain are as follows:

[0032] Use high attention value mask strategy HAM to obtain

[0033] The instance mask token under HAM is first initialized to an all-zero vector, i.e., M h (:)=0. Then, collect the attention scores ranked in the top β h % of the instance index, i.e. Finally, the mask flags of these indices are set to 1, i.e., M h (I h )=1. Adjust the mask ratio β according to the training progress. h , the classic cosine decay function is used to adjust the decay speed;

[0034] Let the mask mark M obtained by the HAM mask strategy h Mark the new mask

[0035] As a preference, the new mask mark in S33 The steps to obtain are as follows:

[0036] L-HAM mask strategy:

[0037] Generate the mask mark M using the same process as HAM l , taking the first β with the lowest mask attention score l % instances,

[0038]

[0039] M l =[0,…,0,1,…,1](9)

[0040] in, and

[0041] Then the mask obtained by these two L-HAM masking strategies is marked as M l and the mask mark M obtained by the HAM mask strategy h Merge into a new mask mark

[0042] As a preference, the new mask mark in S33 The steps to obtain are as follows:

[0043] R-HAM mask strategy:

[0044] Given a random ratio β r %, generate a random mask labeled M r :

[0045]

[0046] M l =[0,1,…,0,1] (11)

[0047] in, M r With M h Combined to get a new mask mark

[0048] As a preference, the new mask mark in S33 The steps to obtain are as follows: LR-HAM mask strategy: M l and M h and M r The three mask marks are merged to obtain a new mask mark

[0049] As a preferred method, the steps for updating parameters of the S3 student model using the gradient descent method are as follows:

[0050] The S3 student model training process includes two loss functions, one of which is the cross entropy loss, which is used to measure the loss of package label prediction.

[0051]

[0052] in, is the category label prediction of the package, and Y represents the label corresponding to the package.

[0053] The other is the consistency loss between the classification results of the student model and the momentum teacher model

[0054]

[0055] Among them, p trepresents the category prediction of the bag output by the teacher model, A t represents the instance attention score, and τ>0 is the temperature coefficient.

[0056] Solve the optimal parameters of the student model through the back propagation algorithm:

[0057]

[0058] in is the objective function of the student model, θ s are the parameters of the student model S(·), represents the updated parameters, and α is the consistency loss scaling factor.

[0059] Preferably, the teacher model is updated by using the network parameters of the student model through an exponential moving average strategy, and the teacher model parameter update rules are as follows:

[0060] θ t ←λθ t +(1-λ)θ s (16)

[0061] Among them, λ is a hyperparameter, θ t are the network parameters of the teacher model.

[0062] Compared with the prior art, the present invention has at least the following advantages:

[0063] 1. Compared with existing methods, the classification performance of the present invention is significantly improved.

[0064] 2. This invention is a general multi-instance learning framework applicable to different attention networks.

[0065] 3. An effective mask-based difficult instance mining strategy is proposed, which avoids the dependence on instance supervision information. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 Schematic diagram of the flow framework of the method of the present invention.

[0067] Figure 2 Schematic diagram of hybrid masking strategy for hard instance mining.

[0068] Figure 3 Schematic diagram of the MHIM-MIL model inference process.

[0069] Figure 4 Comparison of stability during training for momentum teachers and non-batch gradient update students.

[0070] Figure 5 is the high attention value mask ratio, Figure 5 (a) represents the mask ratio of high attention value on the dataset CAMELYON-16, Figure 5 (b) shows the ratio of high attention value masks on the TCGA Lung Cancer dataset.

[0071] Figure 6 is the performance of MHIM-MIL under different loss scaling factors α, Figure 6 (a) shows the performance of MHIM-MIL under different loss scaling factors α on the dataset CAMELYON-16, Figure 6 (b) Shows the performance of MHIM-MIL under different loss scaling factors α on the dataset TCGA Lung Cancer. DETAILED DESCRIPTION

[0072] The present invention is described in further detail below.

[0073] This paper proposes a mask-based digital pathology image classification method, called Masked Hard Instance Mining-based MIL (MHIM-MIL for short), which aims to mine difficult samples to improve the WSI classification effect. Figure 1 As shown in Figure 2, the MHIM-MIL framework adopts a Siamese structure, i.e., a twin structure, during the training phase.

[0074] A mask-based digital pathology image classification method comprises the following steps:

[0075] S1: Obtain existing pathological images. Each pathological image is considered as a package, and the blocks obtained after image preprocessing are considered as instances. For a binary classification problem, if there is an instance containing a tumor area in the package, then the package is a positive package, otherwise it is a negative package. Let the dataset Represents a set of samples of known categories, where B i Indicates a package, Y i ∈C represents the label corresponding to the i-th packet, C is the class label set, and N is the number of packets in the dataset. Let B={x1,…x i ,…,x n} represents a collection of instances, is a set of instance features, where Y = {y1,…y i ,…,y n},y i ∈C is the label of the corresponding instance, and n is the number of instances in the bag.

[0076] The number of instances n in different bags is variable. The goal of the MIL model M(·) is to use all instances to predict the bag label, i.e. A common solution is to select from a set of instance features We learn the bag embedding F in the process, which is also called the instance aggregation step. Then, we train a classifier on F. To predict the bag label,

[0077] There are two aggregation methods for implementing packet embedding: attention-based aggregation and multi-head self-attention-based aggregation. The first is attention-based aggregation, which is expressed as follows:

[0078]

[0079] Among them, a i Represents instance feature z i The learnable scalar weights of D represent the package embedding vector F and the instance feature vector z i Many research works follow the above aggregation formula, but only in generating the attention score a i The methods are different.

[0080] The second is the aggregation based on Multi-Head Self-Attention (MSA). This method uses a category token z0 to aggregate instance features, and combines the category token z0 with the instance features to obtain the initial input sequence The aggregation process is expressed as:

[0081]

[0082] Z l =Concat(head1,head2,…,head h )W O ,l=1,2,…,L (3)

[0083] in, and is the learnable projection matrix of MSA. is the attention matrix of the lth layer, L is the number of MSAs, and h is the number of heads. The bag embedding F is the category token in the output of the last layer,

[0084]

[0085] It can be seen that self-attention-based packet embedding is essentially a special case of attention-based packet embedding in multi-instance learning. This paper defines two commonly used attention paradigms in multi-instance learning as a multi-instance learning paradigm based on general attention.

[0086] S2: Constructing a twin network structure HIM-MIL, which includes a teacher model and a student model with identical structures. Both the teacher model and the student model use the MIL model based on general attention. The teacher model is used to mine difficult instances from the acquired existing pathological images, and the student model is trained based on the difficult instances mined by the teacher model.

[0087] The HIM-MIL framework can be defined as:

[0088]

[0089] Among them, M T (·) represents the difficult instance mining strategy through the teacher model, Represents the mined instance features. The teacher model and student model can use any attention-based MIL as the backbone network.

[0090] S3: Training of student model:

[0091] The existing digital pathology images are fed into the teacher model in sequence to mine difficult instances. In specific implementation, the batch size of the model training is set to 1, that is, one digital pathology image is fed into the teacher model for training each time, but each image actually contains thousands of instances.

[0092] The student model is trained using the difficult examples mined by the teacher model and its parameters are updated. When the loss of the student model no longer changes, the trained student model is obtained.

[0093] During the process of updating the student model parameters, the teacher model also updates its parameters;

[0094] S4: Detection: After image preprocessing, a pathological image to be predicted is fed into the trained student model, and the output is the predicted label of the pathological image to be predicted. Given an image I, after image preprocessing, it is fed into the convolutional neural network and instance embedding network to obtain the instance feature vector, and then fed into the MIL network to aggregate the instance information, and the final prediction result is obtained through the classification network. For details of the inference process, see Figure 3 shown.

[0095] Specifically, the steps for the teacher model T(·) in S3 to mine difficult instances are as follows:

[0096] S31: The complete sequence of a given instance As the input of T(·), the attention value of the instance is obtained through the attention network of T(·), as shown below:

[0097] A=[a1,…,a2,…,a n ]=T(Z) (6)

[0098] Among them, A represents the instance attention sequence, a i represents the attention value of the i-th instance, and T(Z) represents the instance attention sequence obtained by the teacher model through the instance feature Z. The instance attention value output by the teacher model implicitly reflects the teacher model's attention to the pathological image. Instances with high scores tend to locate prominent features in the pathological image. Therefore, the attention value output by the teacher model is used as the instance score to complete the difficult instance mining task.

[0099] S32: Then sort the instance attention sequence A and arrange it in descending order to obtain the index of the attention sequence:

[0100] I=[i1,i2,…,i n ]=Sort(A) (7)

[0101] Among them, i1 is the index of the instance with the highest attention score, i n is the index of the instance with the lowest attention score;

[0102] S33: Using the index set I, the mask-based hard case mining strategy is used to mine hard instances:

[0103] Define an n-dimensional binary vector M = [m1,…,m i ,…,m n ], a mask tag used to encode the instance, where m i ∈{0,1}. If m i =1, it means that the i-th instance is not masked; if m i =0, it means that the i-th instance is masked;

[0104] Obtain the new mask mark after masking through the mask strategy In getting After that, we can obtain the difficult instances mined by the teacher model T(·)

[0105]

[0106] in, is the number of unmasked instances. The student model is sent for training to obtain the prediction results of the package, and then the model is reversely optimized through consistency loss.

[0107] Specifically, the new mask mark in S33 The steps to obtain are as follows:

[0108] Use high attention value mask strategy HAM to obtain High Attention Masking (HAM) is the most intuitive hard instance mining strategy, which simply masks the top β h % instances, driving the model to focus on instances that are difficult to classify.

[0109] The instance mask token under HAM is first initialized to an all-zero vector, i.e., M h (:)=0. Then, collect the attention scores ranked in the top β h % of the instance index, i.e. Finally, the mask flags of these indices are set to 1, i.e., M h (I h )=1. To ensure that positive instances are retained in the unmasked sequence, mask ratio decay is proposed. The mask ratio β is adjusted according to the training progress. h , the classic cosine decay function is used to adjust the decay speed to prevent the high initial ratio from affecting the subsequent training. In particular, the model of the present invention only adjusts the β of the HAM strategy during training. h Apply the attenuation strategy, and do not apply the attenuation strategy to the mask ratio of other mask strategies introduced below;

[0110] Introducing other masking strategies into HAM, three hybrid masking strategies are proposed to more comprehensively consider the attributes of instances, thereby effectively improving the mining effect of difficult instances. Figure 2 The obtained masked labels are regarded as a set and the union operation is used to perform masked label fusion.

[0111] Let the mask mark M obtained by the HAM mask strategy h Mark the new mask

[0112] Specifically, the new mask mark in S33 The steps to obtain are as follows:

[0113] L-HAM mask strategy:

[0114] Generate the mask mark M using the same process as HAM l , taking the first β with the lowest mask attention score l % instances,

[0115]

[0116] M l =[0,…,0,1,…,1](9)

[0117] in, and

[0118] Then the mask obtained by these two L-HAM masking strategies is marked as M l and the mask mark M obtained by the HAM mask strategy h Merge into a new mask mark

[0119] Specifically, the new mask mark in S33 The steps to obtain are as follows:

[0120] R-HAM mask strategy:

[0121] Randomness helps reduce the risk of model overfitting and improve the generalization of the model. In order to introduce randomness into difficult instance mining, the same process is used to obtain random mask labels. Given a random ratio β r %, generate a random mask labeled M r :

[0122]

[0123] M l =[0,1,…,0,1] (11)

[0124] in, M r With M h Combined to get a new mask mark

[0125] The present invention adopts AB-MIL and TransMIL which are popular in multiple instance learning as the backbone network.

[0126] Specifically, the new mask mark in S33 The steps to obtain are as follows:

[0127] LR-HAM masking strategy: [This strategy simultaneously filters the highest and lowest attention scores, which is expected to achieve the ideal state of the above two properties]

[0128] M l and M h and M r The three mask marks are merged to obtain a new mask mark

[0129] Specifically, the steps for updating parameters of the S3 student model using the gradient descent method are as follows:

[0130] The S3 student model training process includes two loss functions, one of which is the cross entropy loss, which is used to measure the loss of package label prediction.

[0131]

[0132] in, is the category label prediction of the package, and Y represents the label corresponding to the package.

[0133] The other is the consistency loss between the classification results of the student model and the momentum teacher model

[0134]

[0135] Among them, p t represents the category prediction of the bag output by the teacher model, A t represents the attention score of the instance, and τ>0 is the temperature coefficient.

[0136] Solve the optimal parameters of the student model through the back propagation algorithm:

[0137]

[0138] in is the objective function of the student model, θ s are the parameters of the student model S(·), represents the updated parameters, and α is the consistency loss scaling factor.

[0139] Specifically, the teacher model is updated by the network parameters of the student model through the exponential moving average strategy.

[0140] The update rules of the teacher model parameters are as follows:

[0141] θ t ←λθ t +(1-λ)θ s (16)

[0142] Among them, λ is a hyperparameter, θ t are the network parameters of the teacher model. The updated teacher model will be used in the next iteration of difficult instance mining. The teacher model and the student model share the same network structure, but do not require gradient-based updates. The teacher model updates its parameters using an exponential moving average (EMA) strategy, which enhances the stability of the teacher model, resulting in more stable instance attention values output by the model, enabling reliable examples to be provided to the student model.

[0143] 1. Experimental Setup

[0144] This paper uses the first three convolutional modules and global average pooling layer of the ResNet-50 model pre-trained on the ImageNet dataset as the backbone network of this model. An initial 1024-dimensional feature vector is extracted from each tile, and then the dimension is reduced from 1024 to 512 dimensions through a fully connected layer. Under the Siamese structure, EMA is used to update the parameters of the teacher model. The momentum of EMA is 0.9999, and the temperature τ in the consistency loss function is 0.1. The model is trained using the Adam model optimizer, and the initial learning rate is set to 2×10 -4 , weight decay is 10 -5 , and a cosine learning rate strategy was used. All models were trained for 200 epochs using an early stopping strategy with a patience of 30 for CAMELYON-16 and 20 for TCGA. The batch size during model training was set to 1. In particular, no techniques to improve model performance, such as gradient clipping or gradient accumulation, were used.

[0145] 2. Dataset

[0146] In the field of multiple instance learning (MIL) research, there are five classic MIL benchmark datasets: Elephant, Fox, Tiger, Musk1, and Musk2. In the field of pathology image classification research, commonly used public datasets include CAMELYON-16 and TCGA LungCancer. A detailed introduction to each dataset is as follows:

[0147] ①MIL benchmark dataset

[0148] The MIL benchmark dataset is one of the most commonly used datasets in multiple instance learning research. It includes five small-scale datasets, each of which consists of pre-extracted sample features. Musk1 and Musk2 are datasets for drug activity prediction. Elephant, Fox, and Tiger are animal image datasets. These three small datasets do not contain image data, and also only have extracted instance features. For each training sample, if it contains a region of interest of a certain animal, the label of the training sample is the category of the animal. The positive bag contains at least one region of interest of the corresponding animal, and the negative bag does not contain any information of the corresponding animal. Table 1 gives the detailed statistical information of the five datasets in the MIL benchmark dataset:

[0149] Table 1 Statistics of MIL benchmark dataset

[0150]

[0151] ②CAMELYON-16

[0152] CAMELYON-16 is a public dataset for detecting breast cancer lymph node metastasis, including 400 hematoxylin-eosin-stained lymph node pathology sections (WSI). WSI images at 20x magnification are usually used as raw data, and the images are cropped into a series of 256×256 tiles using pathology image preprocessing operations, with non-overlapping image areas. This dataset crops approximately 3.5 million tiles, with an average of 8,800 tiles per image. The entire image is considered a package, and the cropped tiles are considered instances of the package. This dataset provides image category label information and pixel-level annotations of the tumor area, and the cropped tiles can obtain corresponding category labels based on pixel-level annotations. The present invention ignores pixel-level annotations and only retains the category label information of the image. CAMELYON-16 provides an official data split, with 270 images in the training set and 130 images in the test set, and the test sample ratio is 13 / 40≈1 / 3. In order to reduce the impact of data partitioning on model evaluation, the present invention uses three-fold cross validation to ensure that each sample participates in training and testing, and each fold has approximately 133 images.

[0153] ③TCGA Lung Cancer

[0154] TCGA Lung Cancer (TCGA for short) originates from a cancer data sharing system established by the National Cancer Institute of the United States. TCGA includes two subtypes of data: TCGA-LUSC for lung squamous cell carcinoma and TCGA-LUAD for lung adenocarcinoma. TCGA-LUSC contains 478 WSIs, while TCGA-LUAD contains 478 WSIs, for a total of 956 WSIs. The TCGA dataset uses the same data preprocessing as CAMELYON-16. After processing, each image yields approximately 10,300 tiles at 20x magnification. The present invention uses four-fold cross-validation for experiments on this dataset.

[0155] 3. Experimental Results and Analysis

[0156] ① Experimental results of MIL benchmark dataset

[0157] Table 2 shows the experimental results of the method of the present invention on the MIL benchmark dataset. Through observation, it can be found that the MHIM-MIL proposed in the present invention has superior performance on two different MIL models. In particular, on TransMIL, the performance on each dataset has achieved a significant breakthrough. Compared with the traditional method mi-SVM, the performance of MHIM-MIL on the Musk1, Musk2, Fox, Tiger, and Elephant datasets has improved by 10.9%, 15%, 20.8%, 14%, and 13.8%, respectively. Compared with DSMIL, the performance of MHIM-MIL proposed in the present invention on the Musk1, Musk2, Fox, Tiger, and Elephant datasets has improved by 1.7%, 1.5%, 0.5%, 7.9%, and 5%, respectively. This shows that the MHIM-MIL model has demonstrated excellent performance on the MIL benchmark dataset.

[0158] Table 2 Experimental results of MHIM-MIL on benchmarks datasets

[0159]

[0160] ② Experimental results of pathological image dataset

[0161] Tables 3 and 4 show the experimental results of MHIM-MIL and other methods on the CAMELYON-16 and TCGA datasets. The best results are indicated in bold. As shown in Tables 3 and 4, Max-pooling and Mean-pooling perform poorly on both datasets compared to other methods. Simple pooling methods under-model key instance information and are easily affected by other noise in the image. Attention-based methods achieve better performance on both datasets by focusing on salient instances.

[0162] As shown in Table 3, the representative MIL framework DTFD-MIL benefits from further exploration of important instances, achieving superior performance on the CAMELYON-16 dataset. Specifically, DTFD-MIL achieves 88.7%, 95.1%, and 86.5% accuracy, AUC, and F1-score, respectively. MHIM-MIL achieves significant performance improvements over both MIL models by mining difficult instances during training, breaking through performance bottlenecks. Compared to DSMIRL, MHIM-MIL, using two backbone networks, improves AUC by 0.8% and 1.2%, respectively.

[0163] Table 2 Experimental results of MHIM-MIL on the CAMELYON-16 dataset

[0164]

[0165] Similar phenomena can also be observed on the TCGA dataset. As shown in Table 4, MHIM-MIL with AB-MIL as the backbone network achieved 89.6%, 99% and 89.3% performance in accuracy, AUC and F1-score respectively. MHIM-MIL improved the AUC indicator by 1.3%. MHIM-MIL with TransMIL as the backbone network achieved 90.0%, 98% and 89.6% performance in accuracy, AUC and F1-score respectively. These experimental results all show that the mask-based difficult instance mining strategy proposed in the present invention is conducive to improving the performance of the model, and MHIM-MIL is more superior than other MIL methods.

[0166] Table 3 Experimental results of MHIM-MIL on TCGA dataset

[0167]

[0168]

[0169] 4. Ablation Experiment

[0170] ①The importance of different modules in MHIM-MIL

[0171] Tables 6 and 7 report the ablation experiment results of different modules in MHIM-MIL on the CAMELYON-16 and TCGA datasets. The baseline models are two representative attention-based MIL methods, namely AB-MIL and TransMIL. First, a masked difficult instance mining strategy (MHIM) is introduced on the baseline models, using the model itself to mine difficult instances during training. This strategy improves the AUC of the two MIL models by 2% and 1.5% on the CAMELYON-16 dataset, and by 0.9% and 1.3% on the TCGA dataset, respectively, as shown in Tables 6 and 7. Second, a Siamese structure (Siam.) is introduced, which includes a student model and a momentum teacher model, and uses the momentum teacher to mine difficult instances. Compared with the naive MHIM strategy, the third row of Table 6 shows that the Siamese structure based on the momentum teacher is conducive to more stable and effective mining of difficult instances, improving model performance. Compared with the above structure, the consistency loss further improves the two MIL models of MHIM-MIL by 0.3% and 0.3% on the CAMLYON-16 dataset, and by 0.4% and 0.8% on the TCGA dataset.

[0172] Table 4 Module ablation experiments of MHIM-MIL on the CAMELYON-16 dataset

[0173] Baseline MHIM Siam. Con. AB-MIL TransMIL √ .937 .945 √ √ .959 .960 √ √ √ .958 .962 √ √ √ √ .961 .965

[0174] Table 5 Module ablation experiments on TCGA dataset

[0175] Baseline MHIM Siam. Con. AB-MIL TransMIL √ .932 .925 √ √ .941 .938 √ √ √ .946 .941 √ √ √ √ .950 .949

[0176] ②The impact of different difficult instance mining strategies

[0177] The mask-based difficult instance mining strategy is the core design of MHIM-MIL. The present invention designs three hybrid strategies, namely R-HAM, L-HAM and LR-HAM. Tables 8 and 9 show the experimental results of MHIM-MIL using different mask strategies on the CAMELYON-16 and TCGA datasets. Compared with the baseline model, the high attention value mask strategy (HAM) significantly improves the performance of the model. The AUC of the two MIL models on the CAMELYON-16 dataset increased by 2% and 1.4%, respectively, and on the TCGA dataset increased by 0.6% and 2%, respectively. By introducing the other two strategies on the baseline model, different MIL models achieved different degrees of performance improvement on both datasets. Specifically, AB-MIL showed a more significant performance improvement after introducing randomness. Under the R-HAM strategy, MHIM-MIL achieved a 4% performance gain over the AB-MIL model, as shown in Table 8. Unlike AB-MIL, TransMIL performed better under the L-HAM strategy, with MHIM-MIL achieving a 2% performance gain. The more complex three-hybrid strategy (LR-HAM) achieved the best performance on the TCGA dataset with larger tumor areas and more instances. Compared with the baseline model, MHIM-MIL improved the performance of AB-MIL and TrasnMIL models by 1.8% and 4%, respectively.

[0178] Table 6 Comparison of results of different difficult instance mining strategies on the CAMELYON-16 dataset

[0179] Strategy AB-MIL TransMIL Baseline .937 .945 HAM .957 .959 R-HAM .961 .958 L-HAM .958 .965 LR-HAM .959 .963

[0180] Table 7 Comparison of results of different difficult instance mining strategies on TCGA dataset

[0181] Strategy AB-MIL TransMIL Baseline .932 .925 HAM .938 .945 R-HAM .948 .946 L-HAM .943 .947 LR-HAM .950 .949

[0182] In addition, the impact of different MHIM strategies on the computational cost of model training. Tables 10 and 11 report the resource consumption of different difficult instance mining strategies on the CAMELYON-16 dataset, including model size (Para.), training time per epoch (Time), and peak memory usage (Mem.). Table 10 shows the efficiency gains brought by large-scale masked instances with low attention scores and random masked instances. It can be observed from Table 11 that these two masking strategies are particularly important for TransMIL, whose space and time complexities are twice the number of instances. Large-scale masked instances greatly reduce the input of the student model, thereby reducing memory and time consumption. Although the input of the teacher model is all instances, due to the application of momentum teachers, there is almost no additional training cost.

[0183] Table 8 Comparison of resource consumption of different difficult instance mining strategies on the AB-MIL benchmark model

[0184]

[0185]

[0186] Table 9 Comparison of resource consumption of different difficult instance mining strategies on the TransMIL benchmark model

[0187] Strategy Para. Time Mem. Baseline 67M 13.1s 10.6G HAM 67M 15.9s 10.3G R-HAM 67M 10.3s 5.5G L-HAM 67M 10.1s 5.5G LR-HAM 67M 10.1s 5.4G

[0188] ③The impact of teacher model selection on model performance

[0189] The MHIM-MIL proposed in the present invention uses the teacher model to mine difficult instances to promote the training of the student model. As shown in Tables 12 and 13, first, a single model structure is used, and the student model is regarded as the teacher model (Studentcopy). The student model first mines difficult instances before training. Due to non-batch gradient updates, the performance of the student model is unstable and easily affected by noise, and the model cannot achieve optimal performance. Secondly, a Siamese structure is adopted to use the momentum teacher model (Momentum) to mine difficult instances. The momentum teacher model has the same network structure as the student model, and its network parameters are updated using the EMA strategy. This update strategy enhances the stability of the momentum teacher, as follows Figure 4 As shown in Table 12, the performance of the teacher model fluctuates greatly, while the performance of the momentum teacher model improves steadily. Compared with the first structure, the use of the momentum teacher model improves the performance of MHIM-MIL by 0.2% and 0.6% under the two MIL models, respectively.

[0190] On the TCGA dataset, the momentum teacher model significantly improved the performance of the MHIM-MIL model. As shown in Table 13, performance gains of 1% and 1% were achieved under the two MIL models, respectively. By properly initializing the momentum teacher model (Init.), the model's performance is relatively stable in the early stages, and reliable difficult examples are mined, enabling the model to achieve optimal performance. On the TCGA dataset, after adding initialization, the model improved performance by 0.3% and 0.4% on the two backbone networks, respectively. However, a fixed initialized teacher model cannot learn new knowledge, which further emphasizes the importance of iterative optimization.

[0191] Table 10 Comparison of different types of teacher networks on the CAMELYON-16 dataset

[0192]

[0193]

[0194] Table 11 Comparison of different types of teacher networks on the TCGA dataset

[0195] Teacher AB-MIL TransMIL Baseline .932 .925 Studentcopy .937 .935 Init. .947 .941 Momentum .947 .945 Init.+Momentum .950 .949

[0196] ④Mask ratio

[0197] First, fix β l and β r The ratio of high attention value mask ratio β is discussed. h Impact on model performance. Figure 5 As shown, with β h The performance of the model on different datasets will be improved with the increase of . Especially on the TCGA dataset, when β h When the value increases from 0.5% to 2%, the model performance improves significantly. This can be explained by the low β h It reduces the difficulty of instance mining, thus reducing the overall performance of the model. From the two datasets, it can be observed that TransMIL has a good performance in β h The best performance is achieved when β is higher. Compared with AB-MIL, TransMIL has lower discrimination ability for salient instances, which explains why TransMIL needs a larger β h .

[0198] In addition, in order to prevent the high initial ratio from affecting subsequent training, the present invention proposes mask ratio decay. As training progresses, the classic cosine decay function is used to adjust the decay speed. It can be observed from Tables 14 and 15 that the use of mask ratio decay techniques can significantly improve performance. On the CAMELYON-16 dataset, MHIM-MIL improved the performance of the two backbone networks by 0.1% and 0.4% respectively after using the decay strategy. On the TCGA dataset, the same settings improved the performance by 0.7% and 0.3% respectively.

[0199] Table 12 Comparison results of high attention value mask ratio attenuation on CAMELYON-16 dataset

[0200] Strategy AB-MIL TransMIL <![CDATA[β h %]]> .960 .961 <![CDATA[β h %→0%]]> .961 .965

[0201] Table 13 Comparison results of high attention value mask ratio attenuation on TCGA dataset

[0202] Strategy AB-MIL TransMIL <![CDATA[β h %]]> .943 .946 <![CDATA[β h %→0%]]> .950 .949

[0203] Then fix β h , discuss different proportions of β l and β r The impact on model performance. Tables 16 and 17 show the discussion results of different datasets on different MIL models. From Table 16, we can observe that under different β l and β r Under these conditions, the performance difference of the models is small. l and β r The sensitivity is lower than β h However, choosing an appropriate mask ratio is still crucial for optimizing model performance. From Table 17, we can observe that combining the three strategies on the CAMELYON-16 dataset degrades classification performance. This can be attributed to the fact that the tumor region accounts for too small a proportion on the CAMELYON-16 dataset, and the excessive instance masks lose important information.

[0204] Table 14 Mask ratio comparison on TCGA dataset

[0205]

[0206] Table 15 Mask ratio comparison on CAMELYON-16 dataset

[0207]

[0208] ⑤Consistency loss scaling factor

[0209] There is an important hyperparameter α in the MHIM-MIL framework, which balances the influence of self-supervisory information and fully supervised information on model training. Figure 6 As shown in Figure 2, ablation experiments on both datasets consistently show that α affects model training. As α changes, the performance of MHIM-MIL on both baseline models changes significantly. In particular, large values of α undermine the positive effect of WSI label information on model learning.

[0210] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A mask-based digital pathology image classification method, characterized in that: The steps include: S1: Obtain existing pathological images. Each pathological image is considered as a package. The blocks obtained after image preprocessing are considered as instances. If there is an instance containing a tumor area in the package, the package is a positive package, otherwise it is a negative package. Let the dataset Represents a set of samples of known categories, where B i Indicates a package, Y i ∈C represents the label corresponding to the i-th package, C is the category label set, N is the number of packages in the dataset; let B={x1,…x i ,…,x n } represents a collection of instances, is a set of instance features, where Y = {y1,…y i ,…,y n },y i ∈C is the label of the corresponding instance, n is the number of instances in the bag; S2: Construct a twin network structure HIM-MIL, which includes a teacher model and a student model with exactly the same structure. Both the teacher model and the student model adopt the MIL model based on general attention; The HIM-MIL framework can be defined as: S3: Training of student model: The existing pathological images are sequentially fed into the teacher model to mine difficult instances. The student model is trained using the difficult instances mined by the teacher model and its parameters are updated. When the loss of the student model no longer changes, the trained student model is obtained. During the process of updating the student model parameters, the teacher model also updates its parameters; The steps of the teacher model T(·) to mine difficult examples are as follows: S31: The complete sequence of a given instance As the input of T(·), the attention value of the instance is obtained through the attention network of T(·), as shown below: A=[a1,…,a2,…,a n ]=T(Z) (6) Among them, A represents the instance attention sequence, a i represents the attention value of the i-th instance, and T(Z) represents the instance attention sequence obtained by the instance feature Z through the teacher model; S32: Then sort the instance attention sequence A and arrange it in descending order to obtain the index of the attention sequence: I=[i1,i2,…,i n ]=Sort(A) (7) Among them, i1 is the index of the instance with the highest attention score, i n is the index of the instance with the lowest attention score; S33: Using the index set I, the mask-based hard case mining strategy is used to mine hard instances: Define an n-dimensional binary vector M = [m1,…,m i ,…,m n ], a mask tag used to encode the instance, where m i ∈{0,1}; if m i =1, it means that the i-th instance is not masked; if m i =0, it means that the i-th instance is masked; Obtain the new mask mark after masking through the mask strategy In getting After that, we can obtain the difficult instances mined by the teacher model T(·) in, is the number of unmasked instances; S4: Detection: a pathological image to be predicted is sent to the trained student model after image preprocessing, and the output is the predicted label of the pathological image to be predicted.

2. The mask-based digital pathology image classification method according to claim 1, wherein: New mask markers in S33 The steps to obtain are as follows: Use high attention value mask strategy HAM to obtain The instance mask token under HAM is first initialized to an all-zero vector, i.e., M h (:)=0; Then, collect the attention scores ranked in the top β h % of the instance index, i.e. Finally, the mask flags of these indices are set to 1, i.e., M h (I h )=1; adjust the mask ratio β according to the training progress h , the classic cosine decay function is used to adjust the decay speed; Let the mask mark M obtained by the HAM mask strategy h Mark the new mask 3. The mask-based digital pathology image classification method according to claim 2, wherein: New mask markers in S33 The steps to obtain are as follows: L-HAM mask strategy: Generate the mask mark M using the same process as HAM l , taking the first β with the lowest mask attention score l % instances, M l ＝[0,…,0,1,…,1] (9) in, and Then the mask obtained by these two L-HAM masking strategies is marked as M l and the mask mark M obtained by the HAM mask strategy h Merge into a new mask mark 4. The mask-based digital pathology image classification method according to claim 2, wherein: New mask markers in S33 The steps to obtain are as follows: R-HAM mask strategy: Given a random ratio β r %, generate a random mask labeled M r : M l ＝[0,1,…,0,1] (11) in, M r With M h Combined to get a new mask mark 5. The mask-based digital pathology image classification method according to claim 4, wherein: New mask markers in S33 The steps to obtain are as follows: LR-HAM mask strategy: M l and M h and M r The three mask marks are merged to obtain a new mask mark 6. The mask-based digital pathology image classification method according to any one of claims 1 to 5, characterized in that: The steps for updating parameters using the gradient descent method in the S3 student model are as follows: The S3 student model training process includes two loss functions, one of which is the cross entropy loss, which is used to measure the loss of package label prediction. in, is the category label prediction of the package, and Y represents the label corresponding to the package; The other is the consistency loss between the classification results of the student model and the momentum teacher model Among them, p t represents the category prediction of the bag output by the teacher model, A t represents the attention score of each instance, τ>0 is the temperature coefficient; Solve the optimal parameters of the student model through the back propagation algorithm: in is the objective function of the student model, θ s are the parameters of the student model S(·), represents the updated parameters, and α is the consistency loss scaling factor.

7. The mask-based digital pathology image classification method according to claim 6, wherein: The teacher model is updated by the network parameters of the student model through the exponential moving average strategy. The teacher model parameter update rules are as follows: i t ←lth t +(1-λ)θ s (16) Among them, λ is a hyperparameter, θ t are the network parameters of the teacher model.

Images