Pathological picture classification method based on multi-instance learning

Abstract

The invention discloses a pathological picture classification method based on multi-instance learning, which is used for weakly supervised full-section pathological image classification. Firstly, a positive and negative sample pair is constructed by integrating a self-supervised contrast learning strategy, so that different enhanced versions of the same pathological picture are closer in a feature space, the feature distinguishing ability is effectively enhanced, and the performance of the model in a weak supervision environment is improved. Secondly, introducing a solution based on hematoxylin-eosin (Hamp; and E) a dyeing separation data enhancement technology is adopted to generate more diversified training data, so that the generalization ability and robustness of the model are improved. And finally, causal intervention is carried out through backdoor adjustment to reduce the data set deviation, so that the influence of the data set deviation on the model training and reasoning process is effectively reduced, and the reliability of the model is improved. The model can accurately position a pathological area, and powerful technical support is provided for clinical application.

Description

Technical Field

[0001] The present invention relates to the improvement of algorithms in the field of slice-level pathological image classification, and specifically to a pathological image classification method based on multi-instance learning. Background Art

[0002] Cancer is a global health challenge, and early diagnosis and treatment are crucial to improving patient survival. Histopathological examination of tissue sections is widely considered to be the most reliable and accurate standard for clinical diagnosis of various diseases. However, this process requires pathologists to perform a lot of work under a microscope, which is time-consuming and prone to errors. Using deep learning methods to automatically and accurately classify whole slide images (WSIs) can provide doctors with more objective diagnostic references and improve diagnostic efficiency and accuracy.

[0003] Convolutional neural networks (CNN) are widely used in the field of pathological images due to their powerful feature extraction capabilities. However, the CNN-based WSI diagnostic method requires professional pathologists to manually annotate billion-pixel pathological images, which is extremely complex and time-consuming in actual operation, making it very difficult and costly to obtain a large amount of professionally annotated data. Summary of the invention

[0004] In view of the above-mentioned problems existing in the prior art, the present invention proposes a weakly supervised method (CL-MIL) for WSI classification based on contrastive learning. As a self-supervised learning method, contrastive learning can effectively distinguish pathological images in feature space. The present invention also introduces a data enhancement strategy of random H&E staining separation during the training process, so that the model can more sensitively capture the morphological changes of the cell nucleus, thereby improving the model's ability to recognize pathological features. In addition, when generating full-slice pathological images, deviations are inevitably introduced due to differences in tissue preparation, staining schemes, and digital scanners. In order to solve the problem of confounding factors caused by data set deviations, the present invention introduces methods of causal inference and backdoor adjustment. Through this method, the deviation caused by contextual prior information in the data set can be suppressed, thereby improving the prediction accuracy of the model.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a pathological image classification method based on multi-instance learning, comprising the following steps:

[0006] S1, collect full-slice pathological image data, mark pathological labels respectively, and establish a pathological image dataset;

[0007] S2, cropping the pathological image into image blocks of fixed size and without overlap, and deleting the background area to obtain a pathological block instance;

[0008] S3. Establish a pathological image classification network, take the pathological block instances in the data set as input, take the case labels as output, and perform iterative training to obtain an ideal pathological image classification network model;

[0009] S4. Collect the whole-slice pathological images and input them into the ideal model to automatically output the pathological labels and locate the diseased areas.

[0010] A single pathological image is regarded as a "package" in multi-instance learning. The package-level label classification is manually annotated as a binary classification of whether the disease is contained, and an anchor box is drawn to mark the disease area; the instance-level label is classified by the algorithm.

[0011] The pathological image classification network includes a feature extractor network and a classifier network;

[0012] The feature extractor network introduces a data enhancement method for H&E staining separation to distinguish key features, performs contrastive learning through a self-supervised contrastive learning framework SimCLR, and pre-trains pathological images to obtain a domain-specific feature encoder suitable for pathological image analysis; uses the trained feature encoder to extract features from each tile instance of a pathological image of a specific magnification to generate instance-level features;

[0013] The classifier network uses an aggregator to aggregate instance features to generate package-level features; clusters all package features in a training set to approximate confounding factors and suppresses confounding factors through backdoor adjustment, thereby identifying features.

[0014] The feature extractor network processes the input image patch as follows:

[0015] Stage 1: All input image blocks are processed in batches. For each image block x in the same batch, random data enhancement is introduced to generate a pair of positive samples (x1, x2). Other samples in the same batch are regarded as negative samples of image block x.

[0016] Stage 2: Use the basic encoding network ResNet18 to extract features from the enhanced image and obtain features (h1, h2);

[0017] Stage 3: The features (h1, h2) are input into a multi-layer perceptron with a ReLU activation function, which projects the features into a new space to obtain a potential feature vector (z1, z2);

[0018] Stage 4: Use contrastive loss to train the feature extractor network. For each positive sample pair, the loss function maximizes the similarity between positive samples while minimizing the similarity between negative samples.

[0019] The contrast loss is calculated using the dot product of two latent feature vectors (z i,z j ), and adjust the hyperparameter τ to control the smoothness of the similarity score;

[0020] Directly opposite (z i ,z j )’s contrast loss i,j Defined as:

[0021]

[0022] in, Yes i and z j The cosine similarity of k Yes i Negative samples.

[0023] The random data enhancement uses any one or a combination of H&E staining separation strategy, random cropping, and Gaussian blur to generate positive sample pairs in contrastive learning.

[0024] The training classifier network processes the input image patch as follows:

[0025] Stage 1: Use k-means clustering on all package features in the training set to divide the packages into clusters;

[0026] Stage 2: Average the packet features of each cluster to represent a mixed layer C i , the package features of all clusters are averaged to form the confounding factor set C;

[0027] Stage 3: After eliminating confounding factors through backdoor adjustment, package prediction is performed to obtain the pathological image features that truly affect the package label.

[0028] The calculation process to eliminate confounding factors is as follows:

[0029]

[0030] h(X,c i )=α i c i

[0031]

[0032] Among them, the do(X) operation represents the mandatory assignment of a value to the variable X, B is the package feature, W1 and W2 are two learnable projection matrices that project the package feature B and the confounding factor C into the joint space. Used for feature regularization.

[0033] The present invention has the following beneficial effects and advantages:

[0034] First, the present invention improves the instance-level feature extraction process in multi-instance learning, applies contrastive learning to the feature extractor instead of simply using the feature extractor trained on natural images, and can learn the features of pathological images more accurately;

[0035] Secondly, the present invention proposes to apply the H&E staining separation method to the data enhancement link of contrastive learning. Through H&E staining separation, the cell nucleus part in the pathological image can be decomposed, the morphological changes of the cell nucleus can be sensitively captured, more critical pathological features can be extracted, and the contrastive learning method can be better adapted.

[0036] Finally, the present invention introduces a backdoor adjustment method to achieve causal intervention. When generating whole-slice pathology images, confounding factors are introduced due to differences in tissue preparation, staining schemes, and digital scanners. These confounding factors can be found and controlled through backdoor adjustment, and the characteristics that truly indicate the disease can be learned more accurately for more accurate classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a framework diagram of contrastive learning in the feature extraction process proposed by the present invention;

[0038] Figure 2 Schematic diagram of eliminating confounding factors through backdoor adjustment;

[0039] Figure 3 It is a structural diagram of the entire multi-instance learning process proposed by the present invention;

[0040] Figure 4 Visualization of prediction results; DETAILED DESCRIPTION

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0043] In view of the problems that traditional pathological image classification methods have high requirements on the annotation level, are difficult to capture the key features of pathological images, and are affected by confounding factors, the present invention proposes an improved pathological image classification method based on multi-instance learning, and uses a feature extractor based on contrastive learning in the network constructed using this method. In the process of contrastive learning, a data enhancement technology based on hematoxylin-eosin (H&E) staining separation is introduced, and backdoor adjustment is used to suppress the influence of confounding factors. Firstly, an improved feature extractor based on contrastive learning is used. Contrastive learning learns more robust feature representation by comparing the differences between different pathological images, thereby enhancing the network's ability to capture key features in pathological images. Secondly, a data augmentation technique based on H&E staining separation is introduced. The image blocks are randomly converted into images of the H (hematoxylin) channel or the E (eosin) channel with probability p, and the features of the cell nucleus and cytoplasm are separated, so as to sensitively capture the morphological changes of the cell nucleus, which is a key indicator for diagnosing and grading tumors. Finally, the backdoor adjustment technique is used to suppress the influence of confounding factors. By performing backdoor adjustment during the training process, the model can still learn the features that truly indicate the disease and maintain stable classification performance when faced with differences caused by different tissue preparations, staining schemes, and digital scanners.

[0044] Figure 1 The framework structure diagram of contrastive learning in the feature extraction process of the present invention is as follows:

[0045] Contrastive learning is a self-supervised learning method that learns the feature representation of an image by minimizing the distance between pairs of positive samples and maximizing the distance between pairs of negative samples. In the present invention, the contrastive learning framework is used to pre-train the network model so that the model can capture the intrinsic features of the image and effectively distinguish images of the same category in the feature space.

[0046] First, a sample space Ω containing all image patches is defined. These image patches are randomly sampled from a large-scale image dataset. For each image patch x∈Ω in the sample space, random data augmentation such as scaling, cropping, H&E staining separation, etc. is applied to generate two enhanced versions of the image (x1, x2). If (x1, x2) come from the same image patch, they constitute a pair of positive samples; if they come from different image patches, they are negative samples. Next, these image samples x1 and x2 are input into a feature extractor f θ (·). This feature extractor is usually a deep convolutional neural network that is used to extract high-level feature representations of the image.

[0047] The extracted features are then input into a projected multilayer perceptron. This projected multilayer perceptron is used to map the features to a low-dimensional space to facilitate subsequent similarity measurement and contrastive learning processes. Through the projected multilayer perceptron, the present invention obtains latent vectors z1 and z2 for each pair of samples. These vectors capture the latent features of the image and are the basis for similarity comparisons. The latent vectors z1 and z2 are further input into a predictive multilayer perceptron, the goal of which is to maximize the consistency of the positive sample pairs, that is, to ensure that they are close to each other in the feature space by minimizing the distance between them. Through the back-propagation algorithm, the present invention calculates the gradient of the loss relative to the network parameters, and uses an optimizer to update the network weights to minimize the loss function.

[0048] Directly opposite (z i ,z j )’s contrast loss i,j Defined as:

[0049]

[0050] in, Yes i and z j The cosine similarity of k Yes i Negative samples.

[0051] The final loss is calculated over all positive pairs in the batch:

[0052]

[0053] Among them, l 2k-1,2k and l 2k,2k-1 These two terms represent the positive sample pair loss of the kth pair of samples respectively.

[0054] Through this process, the feature representation learned by the network is able to capture the discriminative characteristics of the image, so that images of the same category are close to each other in the feature space, while images of different categories are far away from each other.

[0055] Figure 2 This is a schematic diagram of eliminating confounding factors through backdoor adjustment in the present invention, as shown in the figure:

[0056] X represents the package in multi-instance learning, Y represents the package label, and C represents the package confounding factor. X→Y means that the multi-instance learning model predicts the package label through the package content such as key instances. C→X means that when generating whole-slice pathology images, bias is introduced due to differences in tissue preparation, staining protocols, and digital scanners. C→Y means that the package prediction is affected by the confounding factors in the training dataset.

[0057] In backdoor adjustment, we first need to identify the confounding factors. In this invention, we use clustering to approximate the confounding factors. We use k-means clustering to cluster all the packet features in the training set. We average the packet features in each cluster to represent the confounding layer c. i , resulting in a set of confounding factors of shape d × K, where d is the dimension of the bag features. Because these global clusters are susceptible to visual biases, which are confounding factors, clustering can be used to approximate confounding factors.

[0058] After identifying the confounding factors, these confounding factors are controlled by conditioning. For example, the present invention can calculate the impact of the characteristics of true disease indicators under different tissue preparations and staining schemes on the prediction of the package label separately, and then combine these results to obtain the overall impact. It can be expressed by the following formula:

[0059]

[0060] h(X,c i )=α i c i

[0061]

[0062] Among them, the do(X) operation represents the intervention on the variable X, B is the package feature, W1 and W2 are two learnable projection matrices that project the package feature B and the confounding factor C into the joint space. Used for feature regularization, X represents the bag and Y represents the bag label.

[0063] The present invention considers all possible confounding factors Z and performs backdoor adjustments thereto to block the backdoor path and reduce the interference of confounding factors on model prediction.

[0064] Figure 3 This is a structural diagram of the overall multi-instance learning of the present invention, as shown in the figure:

[0065] In the method of the present invention, the automatic classification process of pathological images first involves preprocessing of the whole slice image. Specifically, the present invention cuts the whole slice image into a series of non-overlapping image blocks {x1, ..., x n}, each block has a uniform size of 256×256 pixels, and background areas that do not contain pathological features are removed. This process not only simplifies the complexity of the image, but also improves the pertinence and efficiency of subsequent processing steps.

[0066] Next, in the image block {x1,…,x n}, and encode each instance into a discriminative feature vector {b1,…,b n These feature vectors can not only reflect the local characteristics of the image blocks, but also reveal the relationship between image blocks, providing rich information for the classification of pathological images.

[0067] Then, the present invention aggregates the feature vectors {b1,…,b n} are assembled into package features B. The choice of aggregator is crucial to model performance. It can be a simple mean or maximum pooling, or a more complex weighted aggregation or attention mechanism. The present invention is not limited to a specific aggregator, which means that the most appropriate aggregation strategy can be flexibly selected according to different application scenarios and data characteristics.

[0068] In order to further improve the prediction accuracy of the model, this paper introduces backdoor adjustment and causal intervention technology. In the training stage, causal intervention is achieved through backdoor adjustment to suppress the deviation caused by confounding factors in the data set. Here, represents a vector concatenation operation, which allows the feature information from different sources to be integrated together. Specifically, backdoor adjustment is performed by introducing a set of learnable confounding factors C, which is obtained from the training data through the k-means clustering method. Each confounding factor c i Represents a potential bias in the data, and by learning the representation of these biases, the model can identify and reduce their impact on the final prediction results. In this way, the present invention can adjust the model training process to focus more on those image features that truly indicate the disease, rather than those irrelevant features that frequently co-occur with the label but may mislead the model.

[0069] The proposed method is compared with a series of existing methods on the TCGA lung cancer dataset. The proposed method is significantly better than other comparison models in terms of evaluation indicators such as precision, recall rate, accuracy and AUC score. The results are shown in Table 1:

[0070] Table 1

[0071] Model Precision Recall Accuracy AUC Mean Pooling 81.21 79.89 81.40 83.54 Max Pooling 83.78 83.85 83.45 86.34 AB-MIL 88.72 88.51 88.57 91.71 DS-MIL 90.26 91.37 90.02 92.53 Trans-MIL 92.05 93.82 91.90 94.55 DTFD-MIL 90.34 91.92 91.95 94.95 Ours 92.73 94.23 92.86 95.67

[0072] Figure 4 The prediction results of the present invention are visualized as shown in the figure:

[0073] The left image shows the original full-slice pathology image, where the disease area is outlined in red; the right image shows the predicted area, marked with red squares to indicate positive disease instances. The present invention accurately depicts the true disease area in the full-slice pathology image, which highlights the excellent attention visualization ability and interpretability of the present invention.

[0074] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should be regarded as within the scope of protection of the present invention.

Claims

1. A pathological image classification method based on multi-instance learning, It is characterized in that The following steps are involved: S1, collect full-slice pathological image data, mark pathological labels respectively, and establish a pathological image dataset; S2, cropping the pathological image into image blocks of fixed size and without overlap, and deleting the background area to obtain a pathological block instance; S3. Establish a pathological image classification network, take the pathological block instances in the data set as input, take the case labels as output, and perform iterative training to obtain an ideal pathological image classification network model; S4. Collect the whole-slice pathological images and input them into the ideal model to automatically output the pathological labels and locate the diseased areas.

2. A case image classification method based on multi-instance learning according to claim 1, It is characterized in that A single pathological image is regarded as a "package" in multi-instance learning. The package-level label classification is a manually annotated binary classification of whether the disease is contained, and an anchor box is drawn to mark the disease area; the instance-level label is classified by the algorithm.

3. A pathological image classification method based on multi-instance learning according to claim 1, It is characterized in that The pathological image classification network includes a feature extractor network and a classifier network; The feature extractor network introduces a data enhancement method for H&E staining separation to distinguish key features, performs contrastive learning through a self-supervised contrastive learning framework SimCLR, and pre-trains pathological images to obtain a domain-specific feature encoder suitable for pathological image analysis; uses the trained feature encoder to extract features from each tile instance of a pathological image of a specific magnification to generate instance-level features; The classifier network uses an aggregator to aggregate instance features to generate package-level features; clusters all package features in a training set to approximate confounding factors and suppresses confounding factors through backdoor adjustment, thereby identifying features.

4. A case image classification method based on multi-instance learning according to claim 3, It is characterized in that The feature extractor network processes the input image patch as follows: Stage 1: All input image blocks are processed in batches. For each image block x in the same batch, random data enhancement is introduced to generate a pair of positive sample pairs (x 1 ,x 2 ), the other samples in the same batch are regarded as negative samples of image block x; Stage 2: Use the basic encoding network ResNet18 to extract features from the enhanced image and obtain features (h 1 ,h 2 ); Stage 3: Features (h 1 ,h 2 ) is input into a multilayer perceptron with a ReLU activation function to project the features into a new space and obtain the potential feature vector (z 1 ,z 2 ); Stage 4: Use contrastive loss to train the feature extractor network. For each positive sample pair, the loss function maximizes the similarity between positive samples while minimizing the similarity between negative samples.

5. A case image classification method based on multi-instance learning according to claim 4, It is characterized in that The contrast loss is calculated using the dot product of two latent feature vectors (z i ,z j ), and adjust the hyperparameter τ to control the smoothness of the similarity score; Directly opposite (z i ,z j )’s contrast loss i,j Defined as: in, Yes i and z j The cosine similarity of k Yes i Negative samples.

6. A pathological image classification method based on multi-instance learning according to claim 4, It is characterized in that The random data enhancement uses any one or a combination of H&E staining separation strategy, random cropping, and Gaussian blur to generate positive sample pairs in contrastive learning.

7. A pathological image classification method based on multi-instance learning according to claim 3, It is characterized in that The training classifier network processes the input image patch as follows: Stage 1: Use k-means clustering on all package features in the training set to divide the packages into clusters; Stage 2: Average the packet features of each cluster to represent a mixed layer C i , the package features of all clusters are averaged to form the confounding factor set C; Stage 3: After eliminating confounding factors through backdoor adjustment, package prediction is performed to obtain the pathological image features that truly affect the package label.

8. A pathological image classification method based on multi-instance learning according to claim 7, It is characterized in that The calculation process to eliminate confounding factors is as follows: h(X,c i )=α i c i The do(X) operation represents the mandatory assignment of a value to the variable X, B is the package feature, and W 1 ,W 2 are two learnable projection matrices that project bag features B and confounding factors C into the joint space, Used for feature regularization.

Images