Pathological image classification method and device based on multi-instance learning

Abstract

According to the pathological image classification method and device based on multi-instance learning, better feature representation can be generated, and the problems of domain migration, data hunger and thirst and model overfitting can be effectively solved. The method comprises the following steps: (1) preprocessing a pathological image; (2) feature extraction; (3) feature aggregation; the step (2) comprises the following steps: (2.1) training a feature extractor of a full-section pathological image WSI by adopting self-supervised contrast learning; (2.2) performing data enhancement based on clustering representation; and (2.3) gradually discarding a network layer by adopting a regularization technology.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a pathological image classification method based on multi-instance learning and a pathological image classification device based on multi-instance learning. Background Art

[0002] A whole-slide pathological image (WSI) is a digital scan of a pathological tissue section, which is convenient for evaluation, sharing and analysis, and plays a crucial role in the evaluation and treatment of cancer. Recently, driven by deep learning, many computer-aided WSI diagnosis systems based on deep learning (DL) methods have been developed. However, the high-resolution characteristics of WSI pose challenges to computer-aided automatic analysis. It is not feasible to directly apply existing classification networks to billion-pixel WSI with limited GPU memory. First, WSI is huge and difficult to be processed by a deep learning model at one time. The common practice now is to preprocess WSI first, divide WSI into many small "patches", and then hand them over to a deep learning network for computational analysis. Second, for tens of thousands of patches, obtaining fine-grained labels is very time-consuming and laborious, which makes it difficult to implement supervised learning at the "patch" level. Relatively speaking, it is feasible to obtain the label of the whole pathological section. WSI classification is usually formulated as a multi-instance learning (MIL) problem, where each WSI is regarded as a bag, and thousands of patches cut from each WSI are regarded as examples of the bag. In the multi-instance learning formulation, all examples in a negative bag are negative, and the bag is positive as long as one example is positive. A set of WSIs X = {X1, X2,..., X N} and the corresponding labels Y = {Y1, Y2,..., Y N} form a WSI dataset D = {(X i , Y i ), i = 1,..., N}, where N represents the number of bags. The goal is to train a new classification model based on this WSI dataset so that it can predict the binary classification label Y i of the WSI according to an input WSI image X i ∈ {0, 1}. In traditional binary classification MIL, taking any WSI X i in the dataset as an example, it consists of many examples and can be expressed as and its corresponding example-level label The labels of these examples are unknown, where k i represents the number of examples in bag X i , and the number of examples in each bag can be different. When all examples in the bag are negative, it means the bag is negative, that is, Yi = 0; if there is at least one positive example in the bag, then the bag is positive, i.e., Y i = 1. So WSIX i 's label Y i can be expressed as:

[0003]

[0004] This method first trains an example classifier, calculates the scores of each example, and then aggregates the scores of all examples using methods such as max or average pooling to generate the score of the bag. Since only bag-level labels are available for the training set WSI, the example labels are usually initialized to the bag labels. However, this approach generates a large amount of noise, which affects the final classification performance.

[0005] Another type of method is MIL classification based on bag representation. Specifically, a WSIX i is sliced into multiple examples (patches) and then the example features are obtained through the example feature extractor G θ as shown in Equation (2): As shown in Equation (2):

[0006] h i,j = G θ (x i,j ), j = (1, 2,..., k i )(2)

[0007] where d represents the length of the example feature vector, and k i represents the number of examples in the bag X i . Finally, the example features are aggregated into a bag representation. Most current methods use the attention mechanism to generate the bag representation, as shown in Equation (3), where a j is the attention weight of the feature vector h i,j , representing the importance of the j-th example, and ψ σ is the feature aggregator. The WSI label prediction is output based on the bag representation where g θ is the bag classifier, as shown in Equation (4):

[0008]

[0009] The method based on bag representation uses the label of the bag to directly supervise the generation of the bag prediction score. Compared with the example-based method, it usually has higher accuracy.

[0010] This study mainly uses the multi-instance learning method based on bag representation. The general process of WSI analysis by this method usually consists of three stages: pathological image preprocessing, feature extraction, and feature aggregation, asFigure 1 As shown. Some studies focus on aggregator architecture design and improving the feature extraction of patches. However, despite the progress made by these methods, the following problems remain inadequately addressed, and there is still room for improvement in the performance of pathological image classification.

[0011] (1) In image classification, current methods either use CNN to extract fixed patch features or use the selected high-score patches to update the feature extractor. However, for large bags like WSI, the end-to-end training cost of the feature extractor and aggregator is too high. To reduce GPU memory consumption, a frozen block embedder pre-trained on ImageNet is usually used to extract features from patches and embed them into feature vectors. Although using a frozen block embedder can significantly reduce GPU memory consumption, it also introduces the problem of domain transfer for WSI classification. WSI images are semantically very different from natural images, and the main objects in natural images can occupy most of the area, but this is often not the case for WSI images.

[0012] (2) Due to its weak supervision signal, the MIL model is prone to overfitting problems and cannot learn rich feature representations.

[0013] (3) Deep learning models have problems of data memorization and data hunger. The literature on learning theory and neural networks points out that the problem of data memorization is fundamentally caused by the nature of neural networks, and the result is overfitting of the model. Neural networks learn the specific sample features of the training data during the training process and rely too much on the learned sample features during subsequent prediction or generalization, unable to effectively learn or generalize. In addition, insufficient samples may also hinder the model from being fully trained, further reducing the generalization performance. Data hunger refers to the lack of sufficient training data during the training process of the model, resulting in the inability to fully learn the patterns and rules of the data. Data augmentation is a method of generating additional training samples by transforming existing data (such as rotation, scaling, flipping, color adjustment, etc.) to increase the diversity of the dataset. This technique effectively reduces the overfitting tendency of the model and alleviates the "data hunger" problem caused by insufficient data volume. However, simple processing such as cropping and flipping of images has little effect. Therefore, it is crucial to find an efficient and effective WSI data augmentation method. Summary of the Invention

[0014] To overcome the defects of the prior art, the technical problem to be solved by the present invention is to provide a pathological image classification method based on multi-instance learning, which can not only generate better feature representations but also effectively solve the problems of domain transfer, data hunger, and model overfitting.

[0015] The technical solution of the present invention is as follows: This pathological image classification method based on multi-instance learning includes the following steps:

[0016] (1) Preprocessing of pathological images;

[0017] (2) Feature extraction;

[0018] (3) Feature aggregation;

[0019] The step (2) includes:

[0020] (2.1) Using self-supervised contrast learning to train the feature extractor for whole-slide pathological images WSI;

[0021] (2.2) Using data augmentation based on clustering representations;

[0022] (2.3) Using regularization techniques to gradually discard network layers.

[0023] The present invention uses self-supervised contrast learning to train the feature extractor of WSI, which can not only generate better feature representations but also well solve the domain transfer problem; for the overfitting problem, regularization techniques are used to gradually discard network layers; for the data hunger problem, a strategy of data augmentation based on clustering representations is used, effectively solving the problems of domain transfer, data hunger and model overfitting.

[0024] There is also provided a pathological image classification device based on multi-instance learning, and the device includes:

[0025] A data preprocessing module, which performs preprocessing of pathological images;

[0026] A feature extraction module, which performs feature extraction;

[0027] A feature aggregation module, which performs feature aggregation;

[0028] The feature extraction module performs the following steps:

[0029] (2.1) Using self-supervised contrast learning to train the feature extractor for whole-slide pathological images WSI;

[0030] (2.2) Using data augmentation based on clustering representations;

[0031] (2.3) Using regularization techniques to gradually discard network layers. Description of the Drawings

[0032] Figure 1 Shows a schematic diagram of the WSI classification framework based on bag representations for multi-instance learning.

[0033] Figure 2 Shows the schematic diagram of the present invention.

[0034] Figure 3 Shows the working mechanism diagram of the progressive dropout layer.

[0035] Figure 4 Shows the performance of various methods on the Camelyon16 dataset.

[0036] Figure 5 Shows the performance of various methods on the TCGA-lung dataset.

[0037] Figure 6 Shows the clustering effect diagrams, Camelyon16 (left), TCGA-lung (right).

[0038] Figure 7 Shows the effect diagrams of introducing the progressive dropout layer, Camelyon16 (left), TCGA-lung (right), where the odd columns represent only clustering, and the even columns represent clustering plus the imprisonment dropout layer.

[0039] Figure 8 Shows the clustering number experiments, Camelyon16 (left), TCGA-lung (right).

[0040] Figure 9 Shows the ablation experiments of the hyperparameter enhancement probability, Camelyon16 (left), TCGA-lung (right). Specific implementation manner

[0041] As Figure 2 shown, this pathological image classification method based on multi-instance learning includes the following steps:

[0042] (1) Preprocessing of pathological images;

[0043] (2) Feature extraction;

[0044] (3) Feature aggregation;

[0045] The step (2) includes:

[0046] (2.1) Using self-supervised contrast learning to train the feature extractor of the whole-slide pathological image WSI;

[0047] (2.2) Using data augmentation based on clustering representations;

[0048] (2.3) Using regularization techniques to progressively discard network layers.

[0049] The present invention uses self-supervised contrastive learning to train the feature extractor of WSI, which can not only generate better feature representations but also effectively solve the domain transfer problem. For the overfitting problem, regularization techniques are used to gradually discard network layers. For the data hunger problem, a data augmentation strategy based on clustering representations is used, effectively solving the problems of domain transfer, data hunger, and model overfitting.

[0050] Preferably, step (2.1) includes:

[0051] (2.1.1) Extract features from the examples in the WSI package to obtain a set of feature vectors

[0052] (2.1.2) Reduce the number of examples in each package through clustering. For each package, use the clustering method on the representations of the patches in it to obtain K clusters, and use their centroids to represent the package. X′ i represents the package after clustering, while represents the k i th prototype in the package. The clustered package is the result of removing noise from the entire package.

[0053] (2.1.3) Construct a package dictionary for each package where corresponds to the within-class covariance matrix of the k i th cluster. A package dictionary captures how its examples are distributed by modeling a multivariate Gaussian distribution .

[0054] (2.1.4) Reflect the inherent semantic direction of each cluster through the covariance.

[0055] Preferably, step (2.2) includes:

[0056] (2.2.1) Mix the example prototypes of two packages of the same class while retaining the original labels. Define one of the packages as the query package and the other package as the key package Their examples c q and c k are respectively called the query prototype and the key prototype. For each query prototype find its nearest key prototype

[0057] (2.2.2) Use the data augmentation method of formula (5) to perform data augmentation on the query package. Formula (5) generates a new representation using the covariance matrix of the key package and append it to the package X q ′ to generate a new package representation

[0058]

[0059] where λ is a strength hyperparameter, uniformly sampled from (0, 1), and δ follows a multivariate Gaussian distribution. Is the same as the query prototype The closest key prototype The corresponding covariance matrix has two parts Add together to get a new packet representation m i The number of examples after data enhancement. When performing data enhancement, set the enhancement probability hyperparameter Control the amount of data augmentation in the dataset, where

[0060] Preferably, in step (2.3), multiple progressive dropout layers are used in the MIL structure, each layer includes MIL attention-based Dropout and progressive learning scheduler, and after the fully connected layer, the feature vector After dimensionality reduction, the feature vector is Where f is the length of the feature vector after dimensionality reduction, and then the vector undergoes MIL’s attention-based Dropout.

[0061] Preferably, in the step (2.3), the MIL attention-based Dropout includes average pooling-based attention APBA, which is expressed as follows using formulas (6) and (7):

[0062]

[0063] The embedding features after dimensionality reduction are m i ×f positions, obtain the example-level attention map with the current embedding dimension f to establish the intrinsic connection between examples; each example adopts average pooling to obtain the corresponding weight, and then normalizes it through the softmax function; after ReLU, the key position of each example is activated, and the strength of each position is directly related to its contribution to determining the bag label; APBA is used as an aggregation method to summarize the activated positions, and examples with more activated positions are considered to be positive examples with higher attention weights. APBA identifies the required positive examples.

[0064] Preferably, in step (2.3), the discard rate of each example is dynamically adjusted according to the attention value, and a nonlinear interpolation method is used to dynamically generate a discard rate for each example. The nonlinear interpolation method is expressed as follows using formula (8):

[0065] P / E*log G(linspace(0,G E -1,m i )+1) (8)

[0066] Generate m i dropout rates from 0 to P by formula (8), where m i represents the number of examples, P represents the maximum dropout rate, linspace(min, max, num) is a linear interpolation function that returns num evenly spaced samples from the interval [min, max], and E, G are hyperparameters that control the spacing of the generated set.

[0067] Preferably, in step (2.3), for any function t → P(t), if P(0) = 0 and lim t→T P(t) = P max , then the function is an asymptotic function, and P max is a hyperparameter, the global maximum dropout rate. Adjust the maximum dropout rate P in formula (8) for each epoch through the asymptotic function, so that it gradually increases from 0 to the maximum value P max .

[0068] Preferably, in step (2.3), the initial condition is P(0) = 0, which means that at the initial stage of training, the dropout rate is 0, suppressing the role of the dropout layer. After that, for any t, introduce dropout gradually in the way of P(t) ≤ P max , and finally converge to P(t) → P max . The asymptotic convergence process is expressed by formula (9) as follows:

[0069] P(t) = P max -l(t)P max (9)

[0070] where l(t) is a monotonically decreasing function with 0 ≤ l(t) ≤ 1, which ensures that P(t) increases gradually from 0 to P max . Using the logarithmic interpolation function of formula (8), for each epoch, generate a set of maximum dropout rates {P1, P2,..., P t} from P(t), and these values increase from 0 to P max . Each maximum dropout rate P t will be provided to the dropout layer to generate a set of example dropout rates

[0071] A pathological image classification device based on multi-instance learning is also provided. The device includes:

[0072] A data preprocessing module that performs preprocessing on pathological images;

[0073] Feature extraction module, which performs feature extraction;

[0074] Feature aggregation module, which performs feature aggregation;

[0075] The feature extraction module performs the following steps:

[0076] (2.1) Use self-supervised contrastive learning to train the feature extractor for whole-slide pathology images (WSIs);

[0077] (2.2) Use clustering-based representations for data augmentation;

[0078] (2.3) Use regularization techniques to gradually discard network layers.

[0079] Preferably, in step (2.3), multiple progressive dropout layers are adopted in the MIL structure. Each layer includes MIL attention-based Dropout and a progressive learning scheduler. After passing through the fully connected layer, the feature vector is dimensionally reduced, and the dimensionally reduced feature vector is where f is the length of the dimensionally reduced feature vector. Then this vector passes through MIL attention-based Dropout.

[0080] The present invention is described in more detail below.

[0081] This method is based on the MIL method of bag representations. The main work focuses on the feature extraction part in the MIL framework, and works around three aspects: reducing memory consumption, data augmentation, and combating overfitting. The specific process is as Figure 2 shown.

[0082] 1. Example prototype - reducing memory consumption. Usually, the number of examples in a bag is very large, which will lead to high memory consumption during training. The present invention first extracts features from the examples in the WSI bag to obtain a set of feature vectors (Full-Bag) After that, the number of examples in each bag is reduced by clustering. Considering that most tissues in WSIs are usually similar and there is redundancy, a set of example prototypes is used to replace the examples in the WSI bag. Specifically, for each bag, the representations of the patches in it are used to obtain K clusters by clustering methods, and their prototypes (centroids) are used to represent the bag, that is X′ i represents the bag after clustering (Reduced-Bag), and represents the k i th prototype in the bag. The bag after clustering can be regarded as the result of removing noise from the whole bag. In addition, in order to make full use of WSI information, a bag dictionary is constructed for each bag where corresponds to the k iThe within-cluster covariance matrix of a cluster. A bag dictionary captures how its examples are distributed by modeling a multivariate Gaussian distribution An example of how it captures is as follows. In addition, covariance can reflect the inherent semantic direction of each cluster. For example, how the features of the cluster change. The processing procedure is as shown in Figure 2 (b).

[0083] 2. Cluster-based data augmentation. Traditional image data augmentation methods, such as flipping, rotation, blurring, etc., may be relatively inefficient. Inspired by previous research

[40] , this paper adopts the method of mixing the example prototypes of the same class of bags to achieve data augmentation. We use the bag mixing method to expand the latent space and increase data diversity. Specifically, we mix the example prototypes of two bags of the same class while retaining the original labels. Define one bag as the query bag Define the other bag as the key bag Their examples c q and c k are called the query prototype and the key prototype respectively. For each query prototype Find its nearest key prototype Then use the data augmentation method of formula (5) to perform data augmentation on the query bag. Specifically, formula (5) uses the covariance matrix of the key bag to generate a new representation And append it to the bag X q ′ to generate a new bag representation

[0084]

[0085] where λ is the intensity hyperparameter, sampled uniformly from (0, 1). δ follows a multivariate Gaussian distribution is the covariance matrix corresponding to the key prototype closest to the query prototype . The two parts ( and ) are added together to obtain the new bag representation m i is the number of examples after data augmentation. When performing data augmentation, set the augmentation probability hyperparameter to control the amount of data in the dataset that undergoes data augmentation, where

[0086] 3. Against overfitting. When performing deep learning training on a limited WSI dataset, overfitting is likely to occur. Data augmentation is an effective method to combat overfitting, as described above. In addition, the present invention effectively introduces another form of data augmentation by adopting a progressive dropout layer in the MIL structure, randomly discarding elements to help the network see a different set of data combinations. Multiple progressive dropout layers are adopted in the MIL structure. Taking one of the layers as an example for description, the specific processing process of the progressive dropout layer is as Figure 3 shown. It mainly includes MIL-specific attention-based Dropout and a progressive learning scheduler. As Figure 3 shown, after passing through the fully connected layer, the feature vector is dimensionally reduced, and the dimensionally reduced feature vector is where f is the length of the dimensionally reduced feature vector. Then this vector passes through the MIL-specific attention-based Dropout part.

[0087] The MIL-specific attention-based Dropout includes the following components: an approximate attention mechanism, namely Average Pooling-Based Attention (APBA) based on average pooling, which does not introduce additional parameters like general average pooling methods. It is expressed by formulas (6) and (7) as follows:

[0088]

[0089] The dimensionally reduced embedded features have m i ×f positions. This method does not focus spatial attention on each position of the feature map but obtains an example-level attention map with the current embedding dimension of f to establish the internal connection between examples. Each example adopts average pooling to obtain the corresponding weight, and then it is normalized through the softmax function. After ReLU, the key positions (key features) of each example are activated, and the intensity of each position is directly related to its contribution to determining the bag label. APBA is used as an aggregation method to summarize the activation positions, and the example with more activation positions is considered a positive example with a higher attention weight. APBA can identify the required positive examples.

[0090] To address the overfitting problem, the network model forcibly discards examples with high attention weights while moderately retaining examples with low attention weights. This strategy can help the model avoid over-relying on obvious features and thus discover more potential important features. Based on the above considerations, the dropout rate of each example is dynamically adjusted according to the attention value. Here, a non-linear interpolation method is used to dynamically generate the dropout rate for each example The non-linear interpolation method is expressed by formula (8) as follows:

[0091] P / E*logG (linspace(0, G E - 1, m i ) + 1)(8)

[0092] Generate m from formula (8) i dropout rates from 0 to P, where m i represents the number of example instances, and P represents the maximum dropout rate. linspace(min, max, num) is a linear interpolation function that returns num evenly spaced samples from the interval [min, max]. E and G are hyperparameters that control the spacing of the generated set. As Figure 3 shown, a set of dropout rates generated using the non - linear interpolation function of formula (8) has more dense dropout rate values when approaching the maximum dropout rate P, thus ensuring that high - attention - weight examples obtain corresponding high dropout rates. Examples with high attention weights correspond to high dropout rates, and the dropout rates of other examples are proportional to their weight magnitudes to maintain randomness. Specifically, the dropout rate is assigned based on the attention weights of the examples. A set of dropout rates generated within the range from 0 to P is sorted in descending order, and according to Figure 3 the attention map, a dropout rate is specified for each example In Figure 3 , the gray part represents the dropped - out examples, and the colored part represents the retained examples.

[0093] For the value of the maximum dropout rate P, if a fixed threshold P is adopted in equation (8), that is, the maximum dropout rate P remains unchanged for each training epoch, it may lead to false positives in the model during training (i.e., misidentifying negative examples as positive examples). Because in the initial stage of training, the model's ability to recognize the features of positive examples is weak. A fixed high dropout rate will cause positive examples with high attention weights to be discarded too much. Since positive examples with high attention weights are discarded, negative examples become the main training objects. Without enough positive examples, the model will mistake the features of negative examples for the features of positive examples.

[0094] To solve the above problems, this paper adopts a progressive learning scheduler to guide attention - based Dropout. Its purpose is to suppress the dropout layer in the initial training stage, allowing the model to pay more attention to and learn the features of positive examples. After the model gradually learns the features of positive examples, the dropout rate is gradually increased, enabling the model to better handle noise and prevent overfitting in the later stage of training.

[0095] For any function t → P(t), if it satisfies P(0) = 0 and lim t→T P(t) = P max , then this function is called a progressive function, P maxis a hyperparameter, which is the global maximum dropout rate. The maximum dropout rate \(P\) in formula (8) is adjusted for each epoch \(t\) through a progressive function, gradually increasing from 0 to the maximum value \(P\). max The specific calculation process is as follows: The initial condition is \(P(0) = 0\), which means that at the initial stage of training, the dropout rate is 0, suppressing the role of the dropout layer. After that, for any \(t\), dropout is gradually introduced in the way of \(P(t)\leq P\). max and finally converges to \(P(t)\to P\). max The progressive convergence process can be expressed by formula (9) as follows:

[0096] \(P(t)=P - l(t)P\) max (9) max where \(l(t)\) is a monotonically decreasing function with \(0\leq l(t)\leq1\), which ensures that \(P(t)\) increases gradually from 0 to \(P\). In this paper, the logarithmic interpolation function of formula (8) is adopted. For each epoch \(t\), a set of maximum dropout rates \(\{P_1, P_2,\cdots, P\}\) is generated from \(P(t)\), and these values increase from 0 to \(P\).

[0097] Each maximum dropout rate \(P\) (as \(P\) in formula (8)) is provided to the dropout layer to generate a set of example dropout rates. max To verify the effectiveness of the proposed scheme, our model is trained on the Camelyon16 and TCGA - lung datasets, and our model is evaluated on the test set to verify the effectiveness of this method. t First, feature extraction is performed on the Camelyon16 and TCGA - lung datasets. The ResNet - 18 pre - trained using the contrastive learning framework SimCLR is used as the feature extractor to extract patch features as 512 - dimensional vectors. For Camelyon16, multiple experiments are carried out according to the officially divided training set and test set and the average is taken. For TCGA - lung, k - fold cross - validation is performed, and finally the average of the results is taken as the final result. max is a framework for WSI classification of multi - instance learning based on bag representation. It is divided into three parts in total. First, each WSI is sliced into patches, then the patch features are extracted and processed, and finally the feature aggregator aggregates these patch feature vectors into a bag representation and performs bag prediction. t

[0098]

[0099]

[0100] Figure 1 Figure 2 ​​​​It is the working principle diagram of the present invention, which is mainly divided into four parts: (a) preprocessing the WSI, (b) performing clustering after feature extraction to reduce memory consumption, and then performing data augmentation, (c) performing regularization, introducing a progressive dropout layer to combat overfitting, and (d) performing bag aggregation to predict bag labels. Figure 3 It is the working mechanism diagram of a progressive dropout layer. The progressive learning scheduler controls the generation of the maximum dropout rate P for each epoch, and a set of dropout rates is generated for the current epoch based on the maximum dropout rate of each epoch. The APBA module dynamically assigns a dropout rate to each example according to the attention weight of the example. Figure 4 It is the performance of our model compared with other methods on the Camelyon16 dataset, where ACC, AUC, and F1 represent various model evaluation metrics. The larger the value, the better the model performance. Figure 5 It is the performance of our model compared with other methods on the TCGA-Lung dataset. Figure 6 It is an ablation experiment on the model clustering module on the camelyon16 and tcga-lung datasets to verify its effectiveness. Figure 7 It is an ablation experiment on the model progressive dropout layer module on the camelyon16 and tcga-lung datasets to verify its effectiveness. Figure 8 It is an ablation experiment on selecting the number of clustering clusters on the camelyon16 and tcga-lung datasets. The number of clustering clusters is set to 1, 2, 4, and 8 respectively. Figure 9 It is an ablation experiment on selecting the data augmentation probability on the camelyon16 and tcga-lung datasets. The augmentation probabilities are set to 0.3, 0.5, 0.7, and 0.9 on the two datasets respectively, and multiple experiments are conducted and averaged to select the optimal hyperparameters.

[0101] As described above, it is only a preferred embodiment of the present invention, and there is no restriction on the present invention in any form. Any simple modification, equivalent change, and modification made to the above embodiments based on the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A pathological image classification method based on multi-instance learning, characterized in that: The method includes the following steps: (1) Preprocessing of pathological images; (2) Feature extraction; (3) Feature aggregation; The step (2) includes: (2.1) Using self-supervised contrastive learning to train a feature extractor for whole-slide pathological images (WSIs); (2.2) Using clustering-based representations for data augmentation; (2.3) Adopting regularization techniques to gradually discard network layers.

2. The method for classifying pathological images based on multi-instance learning according to claim 1, wherein: The step (2.1) includes: (2.1.1) Extract features from the examples in the WSI package to obtain a set of feature vectors (2.1.2) Reduce the number of examples in each bag by clustering. For each bag, use the clustering method to obtain K clusters for the representations of the patches in it, and use their centroids to represent the bag. X′ i represents the bag after clustering, while represents the k i th prototype in the bag. The bag after clustering is the result of removing noise from the entire bag. (2.1.3) Construct a bag dictionary for each bag where corresponding to the within-cluster covariance matrix of the k i th cluster, a bag dictionary captures how the examples of it are distributed by modeling a multivariate Gaussian distribution ; (2.1.4) Reflecting the inherent semantic direction of each cluster through covariance.

3. The method for classifying pathological images based on multi-instance learning according to claim 2, wherein: The step (2.2) includes: (2.2.1) Example prototype of mixing two packages of the same type while retaining the original tags, defining one of the packages as the query package Define the other package as the key package Their example c q And c k Are called the query prototype and the key prototype respectively. For each query prototype Find its nearest key prototype (2.2.2) Use the data augmentation method of formula (5) to perform data augmentation on the query packet. Formula (5) generates a new representation using the covariance matrix of the key packet and append it to packet X q ′ to generate a new packet representation where λ is the intensity hyperparameter, sampled uniformly from (0, 1), and δ follows a multivariate Gaussian distribution is the key prototype closest to the query prototype and the corresponding covariance matrix. The two parts ( and ) are added together to obtain the new bag representation m m i where n is the number of examples after data augmentation, and the augmentation probability hyperparameter is set to control the number of data for data augmentation in the dataset, where 4. The method for classifying pathological images based on multi-instance learning according to claim 3, characterized in that: In the step (2.3), multiple progressive dropout layers are adopted in the MIL structure. Each layer includes MIL attention-based Dropout and a progressive learning scheduler. After passing through the fully connected layer, the feature vector is dimensionally reduced, and the dimensionally reduced feature vector is where f is the length of the dimensionally reduced feature vector. Then this vector passes through the MIL attention-based Dropout.

5. The method for classifying pathological images based on multi-instance learning according to claim 4, characterized in that: In the step (2.3), the attention-based Dropout in MIL includes the attention-based average pooling (APBA), which is expressed by formulas (6) and (7) as follows: The embedded features after dimensionality reduction have m i ×f positions to obtain an instance-level attention map with the current embedding dimension of f, so as to establish the internal connection between instances; each instance uses average pooling to obtain the corresponding weight, and then normalizes it through the softmax function; after ReLU, the key positions of each instance are activated, and the intensity of each position is directly related to its contribution to determining the packet label; APBA, as an aggregation method to summarize activation locations, an example with more activation locations is considered a positive example with a higher attention weight, and APBA identifies the required positive examples.

6. The method for classifying pathological images based on multi-instance learning according to claim 5, characterized in that: In the step (2.3), the dropout rate of each example is dynamically adjusted according to the attention value, and the dropout rate for each example is dynamically generated using the non-linear interpolation method. The non-linear interpolation method is expressed by the formula (8) as follows: P / E*log G (linspace(0,G E -1,m i )+1) (8) Generate m from formula (8). i discard rates from 0 to P, where m i represents the number of example, P represents the maximum discard rate, linspace(min, max, num) is a linear interpolation function that returns num evenly spaced samples from the interval [min, max], and E, G are hyperparameters that control the spacing of the generated set.

7. The method for classifying pathological images based on multi-instance learning according to claim 6, wherein: In the said step (2.3), for any function t → P(t), if it satisfies P(0) = 0 and lim t→T P(t) = P max , then this function is an asymptotic function, and P max is a hyperparameter, which is the global maximum dropout rate. The maximum dropout rate P in formula (8) is adjusted for each epoch t through the asymptotic function, so that it gradually increases from 0 to the maximum value P max .

8. The method for classifying pathological images based on multi-instance learning according to claim 7, wherein: In the step (2.3), the initial condition is P(0) = 0, which means that at the initial stage of training, the dropout rate is 0, suppressing the function of the dropout layer. After that, for any t, dropout is gradually introduced in the way of P(t) ≤ P max and finally converges to P(t) → P max . The asymptotic convergence process is expressed by formula (9) as follows: P(t) = P max -l(t)P max (9) where \(l(t)\) is a monotonically decreasing function with \(0\leq l(t)\leq1\), which ensures that \(P(t)\) increases gradually from 0 to \(P\). max Using the logarithmic interpolation function in formula (8), for each epoch \(t\), a set of maximum dropout rates \(\{P_1, P_2, \cdots, P\}\) is generated from \(P(t)\). t These values increase from 0 to \(P\). max Each maximum dropout rate \(P\) t will be provided to the dropout layer to generate a set of example dropout rates.

9. A pathological image classification device based on multi-instance learning, characterized in that: The device includes: A data preprocessing module that performs preprocessing of pathological images; A feature extraction module that performs feature extraction; A feature aggregation module that performs feature aggregation; The feature extraction module performs the following steps: (2.1) Using self-supervised contrastive learning to train a feature extractor for whole-slide pathological images (WSIs); (2.2) Using clustering-based representations for data augmentation; (2.3) Adopting regularization techniques to gradually discard network layers.

10. The pathological image classification device based on multi-instance learning according to claim 9, characterized in that: In the step (2.3), a plurality of progressive dropout layers are adopted in the MIL structure. Each layer includes MIL attention-based Dropout and a progressive learning scheduler. After passing through the fully connected layer, the feature vector is dimensionally reduced, and the dimensionally reduced feature vector is where f is the length of the dimensionally reduced feature vector. Then this vector passes through the MIL attention-based Dropout.

Images