Deep multi-instance learning method for full-slice image analysis based on multi-scale representation attention mechanism

Abstract

Recently, research on tumor diagnosis and analysis by directly using full-slice images (WSIs) based on a convolutional neural network (CNN) attracts wide attention because the method only uses slice-level labels to carry out model training, and extra labeling is not needed. However, direct processing of a full-slice image with ultra-large pixels is still a challenging task because each WSI contains tens of billion pixels and has internal diversity. In order to solve the problem, the invention provides a novel end-to-end interpretable deep multi-instance learning (MIL) framework which is used for WSI analysis. The framework directly extracts features from all patches of each WSI by using a double-branch deep neural network and a multi-scale attention representation mechanism. Specifically, firstly, each WSI is divided into a packet-level image, a patch-level image and a cell-level image, and slice-level labels are allocated to the corresponding packet-level images, so that a WSI classification problem is converted into a multi-instance learning problem. Besides, a novel multi-scale attention representation mechanism is designed and is embedded into the double-branch deep network, so that packets with correct labels, remarkable patches and cell-level information of the packets are mined at the same time. A large number of experiments show that the proposed framework is superior to the latest advanced method in the aspects of classification accuracy and model interpretability.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning, and more specifically, relates to a deep multiple instance learning (MIL) method applied to medical image processing, specifically for tumor diagnosis and analysis in whole slice images (WSI). Background Art

[0002] Whole slide images (WSIs) are an important tool for pathological analysis in modern medicine. Their resolution is usually very high, often containing billions of pixels. Manually annotating and analyzing these images is not only time-consuming and laborious, but also prone to errors. Computer-aided diagnosis systems (CAD) help pathologists automatically process and analyze these whole slide images through machine learning and computer vision technology.

[0003] Deep learning, especially convolutional neural networks (CNNs), has made remarkable progress in image classification and analysis tasks. However, for whole-slice images, traditional deep learning methods often rely on a large number of image patch annotations and are limited by memory and computing power when processing whole-slice images. To overcome these problems, the multiple instance learning (MIL) framework was proposed, which does not require individual annotations for each image patch, but uses slice-level labels for training.

[0004] Although the MIL method has achieved certain results, existing research lacks full utilization of multi-scale information in whole-slice images. In particular, how to effectively extract important features and explain the prediction results of the model at different spatial levels (such as image bags, image blocks, and cell levels) remains a challenge. Therefore, the present invention proposes a deep multi-instance learning framework combined with a multi-scale representation attention mechanism to simultaneously improve the classification performance and explanation ability of the model. Summary of the invention

[0005] The purpose of this invention is to provide a deep multi-instance learning method based on a multi-scale representation attention mechanism for automated analysis of whole-slide images. This method can not only effectively process large-scale image data, but also explain the prediction results through a multi-level attention mechanism, helping pathologists better understand the model output.

[0006] The present invention has the following beneficial effects:

[0007] 1) Through the multiple instance learning (MIL) framework and multi-scale representation attention mechanism, the present invention can effectively process full-slice images containing billions of pixels without the need to manually annotate each image block. Compared with traditional methods that rely on large-scale fine annotation, the present invention greatly reduces the cost and time of data annotation and significantly improves the analysis efficiency of full-slice images;

[0008] 2) By introducing a multi-scale representation (cell level, image block level, image bag level) attention mechanism, the present invention can automatically filter out the most important image areas at different levels, reduce the interference of noise and irrelevant information on the model, and thus improve the classification accuracy. Experiments show that the classification performance of the present invention on multiple data sets is better than the existing state-of-the-art methods;

[0009] 3) The present invention uses the attention mechanism to screen important areas at different levels of the image, reducing unnecessary calculations, thereby reducing the computing resource requirements of the model, and can process the performance of the ultra-high resolution full-slice image domain model under limited computing conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is the overall framework flow chart of the method of the present invention, including the image processing flow charts at each level (packet level, patch level, cell level) of the multi-scale attention mechanism;

[0011] Figure 2 It is a system framework diagram of the present invention;

[0012] Figure 3 An example of a heat map of important image areas filtered out by the attention mechanism of the present invention is shown. DETAILED DESCRIPTION

[0013] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0014] Example

[0015] Figure 1 It is the overall framework of the method of the present invention, including the image processing flow at each level (packet level, patch level, cell level) of the multi-scale attention mechanism. Figure 1 As shown, the specific implementation flow chart of the present invention for deep multi-instance learning of full-slice image analysis based on a multi-scale representation attention mechanism includes the following steps:

[0016] S101: Segmentation of whole-slice images:

[0017] Full slice images usually contain billions of pixels, so it is not feasible to directly process these large images in terms of computation and storage. In order to reduce the computational complexity, the present invention divides the full slice images into different levels of images step by step:

[0018] Bag-level image: First, the full slice image X nDivide into several non-overlapping packet-level images X ni , where n represents the nth full-slice image and i represents the i-th packet-level image. Each packet-level image usually covers a relatively large area, which is convenient for capturing the macroscopic structure in the image;

[0019] Patch-level image: For each package-level image X ni , further subdivided into multiple patch-level images X nij , where j represents the jth patch-level image. Each patch-level image captures more detailed local information and is usually used to identify tiny lesions in an image. ;

[0020] Cell-level image: patch-level image X nij Further subdivided into cell-level images X nijk Where k represents the kth cell-level image. These cell-level images contain the finest granular cell information in pathological analysis and are an important basis for judging pathological conditions.

[0021] S102: Feature extraction:

[0022] For each layer of image, a convolutional neural network (CNN) is used to extract its feature representation. Assume that the input is a cell-level image X nijk , feature representation is obtained through CNN extraction Similarly, for the patch-level image X nij and package level image X ni , we can obtain the feature representations and

[0023] The extraction process of convolutional neural network can use classic deep network architecture, such as ResNet18. In this network, patch-level images and cell-level images are used as input, and the corresponding feature vectors are generated through convolutional layers, pooling layers and fully connected layers. For each input image, its feature representation is defined by the following formula:

[0024] Z=f θ (X)

[0025] where f θ represents a convolutional neural network, θ is the network parameter, and X is the input image.

[0026] S103: Multi-scale representation attention mechanism:

[0027] In order to effectively extract important information from multi-level images, this paper introduces a multi-scale representation attention mechanism, which is divided into three levels: cell-level attention, patch-level attention, and bag-level attention. Each level of attention mechanism calculates attention weights based on the feature representation of each image to filter out significant image features and discard irrelevant or noise features.

[0028] Cell-level attention mechanism: In the cell-level attention mechanism, the cell-level images in each patch-level image are first filtered. For each cell-level image X nijk , whose characteristics are expressed as By calculating its class vector And normalized, we can get the attention score η of each cell-level image nijk The specific formula is as follows:

[0029]

[0030] in, is the class vector of the kth cell-level image, M c is the number of cell-level images in each patch, η nijk is the attention weight of the cell-level image. By calculating the attention weight of each cell image, the cell images that make important contributions to patch-level classification can be automatically screened out.

[0031] Patch-level attention mechanism: For each patch-level image X in the bag-level image nij , by computing the patch-level feature representation and its class vector Get the attention weight δ nij The calculation formula is:

[0032]

[0033] in, is the class vector of the patch-level image, M p is the number of patches in each package, δ nij is the attention score of the patch-level image, and this mechanism can effectively filter out patches that have an important impact on bag-level classification.

[0034] Bag-level attention mechanism: In the bag-level attention mechanism, for each bag-level image X of the full slice image ni Filtering. Through the feature representation of the package level image and its class vector Calculate the attention weight γ ni To filter, the calculation formula is:

[0035]

[0036] in, is the class vector of the bag-level image, M b is the number of packets in each WSI, γ ni is the attention score of bag-level image.

[0037] S104: Classification based on multiple instance learning

[0038] The present invention performs the final classification of the whole slice image based on the multiple instance learning (MIL) framework. The characteristic of multiple instance learning is that each whole slice image (instance) contains several sub-instances (bags, patches, cell-level images), only the overall label is known, and the label of each sub-instance is unknown.

[0039] In the framework of the present invention, each bag-level, patch-level and cell-level image is regarded as a sub-instance, and after the significant sub-instances are screened out through the multi-scale representation attention mechanism, they are used for overall classification. The classification process is optimized by the cross entropy loss function:

[0040]

[0041] Among them, L MIL is the cross entropy loss function for multi-instance learning, y nr is the true label of the nth full slice image, P nrw is the class probability of the bag-level image, and σ(·) is the Softmax function.

[0042] In order to further improve the performance of the model, the present invention combines the attention weight and L 2,1 -Norm optimization, using the following loss function to remove unimportant image features:

[0043]

[0044] This loss function can effectively remove noise instances that contribute less to the classification results, thereby improving the classification accuracy.

[0045] Figure 3 This is an example of a heat map of important image regions selected by the present invention through the attention mechanism. Figure 3 As shown, an example of a heat map of important image areas filtered out by the attention mechanism of the present invention.

[0046] In order to better illustrate the technical effect of the present invention, the present invention is experimentally verified using specific examples. In this experimental verification, the experiment of the present invention uses multiple public whole slice image (WSI) datasets, including lung squamous cell carcinoma (LUSC), breast cancer (BRCA), gastric adenocarcinoma (STAD) and breast cancer metastatic lymph node (CAMELYON16) datasets. The LUSC, BRCA and STAD datasets are derived from the TCGA project, and each contains hundreds of pathological whole slice images, each of which has a pixel size of up to billions (such as 50,000×50,000 pixels), covering the complex tissue structures of different cancer types; and the CAMELYON16 dataset contains more than 400 high-resolution breast cancer metastatic lymph node slice images, and the image size usually ranges from hundreds of millions of pixels to billions of pixels. The diversity and complexity of these datasets place extremely high demands on the multi-scale feature extraction, attention mechanism and classification accuracy of the model, and are ideal test platforms for evaluating the performance of the framework of the present invention.

[0047] On these datasets, MRAN showed excellent classification accuracy and AUC values ​​on the LUSC, BRCA, and STAD datasets, demonstrating its effectiveness in tumor recognition. In the CAMELYON16 dataset, although the classification accuracy of MRAN was relatively low, it was still better than most other methods. The sensitivity and specificity results showed that MRAN can not only effectively identify positive instances, but also exclude negative instances well, ensuring the overall performance of the model. The improvement in the F1 score further verifies the advantage of MRAN in balancing precision and recall.

[0048] Table 1 shows the classification results of the four data sets in this embodiment (mean ± standard deviation).

[0049]

[0050]

[0051] Table 1

[0052] As shown in Table 1, it can be seen from the results in Table 1 that on the LUSC dataset, the accuracy of the MRAN method reached 97.19% and the AUC reached 99.54%, which is significantly better than the performance of the traditional multiple instance learning method (for example, the accuracy of the MSKCC-MIL method is 85.79%). This verifies the effectiveness of the present invention.

[0053] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A deep multi-instance learning method for whole-slice image analysis, characterized in that: The following steps are involved: S1: a noise detection module, which processes the training samples using a convolutional neural network (CNN), predicts the class probability of the samples, and divides the training samples into clean samples, uncertain samples, and corrupted samples based on the calculated loss value; S2: Provide an anti-noise loss function for optimizing the noise detection module to prevent the model from overfitting to the noise label. The loss function combines the cross entropy loss with a time-related smoothing term and is defined by the following formula: Where b is the number of samples in the batch, p i [y i ] represents the predicted probability of the i-th sample, is the label of the sample, C is the number of categories, and λ(t) is a time-related hyperparameter. S3: The noise detection module divides the training samples into the following categories based on the loss value: clean samples: samples with smaller loss values, whose labels are considered to be correct; uncertain samples: samples with medium loss values, whose labels the model cannot accurately determine whether are correct; corrupted samples: samples with larger loss values, whose labels are speculated to be corrupted or contain noise. S4: further includes a noise cleaning module, which uses a category-based graph convolutional network (GCN) model to correct corrupted and uncertain sample labels, specifically including the following steps: S4.1: Build a graph convolutional network (GCN) model for each category to ensure that similar samples have similar labels in the same category by preserving the local topological structure between samples; S4.2: The noise cleaning module corrects the labels of uncertain and corrupted samples using a semi-supervised learning method based on the embedded features generated from the noise detection module. S4.2.1: The noise cleaning module estimates the noise ratio in the training sample through the Gaussian mixture model (GMM). Specifically, the noise rate is calculated by the following formula: where v i is the judgment result of whether sample i is a noise sample, and n is the total number of samples in the training set. S4.2.2: The noise cleaning module is optimized by the following semi-supervised learning loss function: L ssl =L bce +L mse Where L bce is the binary cross entropy loss for labeled samples: where z i is the true label of sample i, q i is the predicted probability of the sample in the current round of training. L mse is the mean squared error loss for unlabeled samples, representing the difference between the current prediction and the previous prediction: where q i is the prediction result of sample i in the current round, is the predicted value of the sample in the previous round.

2. The method according to claim 1, wherein the noise cleaning module uses a graph convolutional network (GCN) model to correct the labels of uncertain samples and corrupted samples, and determines the corrected labels according to the following formula: where q ic is the probability that sample i is predicted to be category c, and the category corresponding to the maximum probability is selected as the corrected label.

3. The method according to claims 1-2 further comprises a two-layer optimization algorithm, wherein the optimization algorithm alternately optimizes the noise detection module and the noise cleaning module to improve the robustness of the model and the accuracy of label correction, specifically comprising: Upper layer optimization: Optimize the noise detection module by minimizing the anti-noise loss function L r Improve the model's ability to recognize noise labels; Lower-level optimization: optimize the noise cleaning module by minimizing the semi-supervised learning loss function L ssl Enables the model to achieve efficient label correction on both labeled and unlabeled samples.

4. The method according to claim 3, wherein the objective function of the two-level optimization algorithm is defined as follows: Where θ is the parameter of the noise detection module, ω c are the parameters of the graph convolutional network (GCN) in the noise cleaning module; f t (x;θ) represents the output of the noise detection module in round t; A represents the prediction result of the noise cleaning module for category c in round t; c and E c are the adjacency matrix and feature matrix of category c respectively.

Images