Weakly supervised classification method based on dual attention pathological image mixed multi-instance learning

Abstract

The application discloses a kind of weak supervision classification methods for pathological image mixed multi-instance learning based on double attention, comprising the following steps: S1: constructing a feature pyramid of pathological image with image-level label marking;S2: constructing a multi-instance learning network based on two attention branches of Transformer and graph attention network (GAT), inputting the constructed feature pyramid into the transformer and GAT two branches of learning network for learning, combining the output of two attention branches to construct an embedding aggregator for pathological image classification prediction, and completing the construction of the classification model;S3: inputting the unmarked pathological image to be classified into the model for classification and drawing attention heat map.The application solves the problem that ignoring the context information around a single region and the correlation of different regions can miss important information of the pathological image, resulting in a large deviation between the image analysis result and the actual result.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a weakly supervised classification method for pathological images based on dual attention and hybrid multi-instance learning. Background Technology

[0002] Digital pathology images often exceed one billion pixels, making them difficult to analyze effectively using traditional image analysis methods. Currently, mainstream deep learning methods for pathology image analysis primarily include fully supervised learning methods based on small image patches and weakly supervised learning methods based on whole-slide (WSI) pathology images. Fully supervised learning models are constructed using two main approaches: one involves experienced personnel annotating the images in detail, delineating feature regions, and then cropping these regions into several 256×256 image patches to build the deep learning model; the other approach eliminates the need for manual annotation, directly assigning easily obtainable image-level labels to all cropped image patches before building the deep learning model for prediction. Existing weakly supervised learning methods do not require manual annotation and only use easily obtainable image-level labels. However, unlike fully supervised learning methods, weakly supervised methods do not assign the same label to every image patch. Instead, the entire image is treated as a package, and each cropped small image patch is treated as an instance. We only know the label of the package, not the label of each instance. When building a deep learning model for prediction, if any instance is positive, the package is predicted as positive, and vice versa.

[0003] When analyzing pathological images with billions of pixels, existing fully supervised learning methods are not suitable because fully supervised learning methods based on small image patches require detailed manual annotation, while weakly supervised learning methods based on whole slide pathological images (WSI) directly assign easily obtainable image-level labels to all cropped image patches. This will introduce a lot of noise with incorrect labels to the model when dealing with image patches with small feature regions.

[0004] Therefore, in image analysis, weakly supervised methods based on multi-instance learning are widely used. Most existing multi-instance learning methods are based on the assumption that all instances in the bag are independent and identically distributed. However, this assumption does not hold true for pathological images. In actual diagnosis, pathologists need to repeatedly change the magnification of the microscope to simultaneously focus on global and local details. In other words, the contextual information surrounding a single region and the correlation between different regions in a pathological image are crucial for analysis. Existing image analysis methods that ignore the contextual information surrounding a single region and the correlation between different regions miss important information in the pathological image, leading to significant deviations between the image analysis results and reality. Furthermore, while existing classification methods show better classification performance when using images with a maximum magnification of 40×, they consume a large amount of memory. Summary of the Invention

[0005] The purpose of this invention is to provide a weakly supervised classification method for pathological images based on dual attention and hybrid multi-instance learning, in order to solve the problem that existing image analysis methods ignore the contextual information around a single region and the correlation between different regions, which leads to the omission of important information in the pathological image and causes a significant deviation between the image analysis results and the actual situation.

[0006] To address the shortcomings of existing technologies, this invention designs a hybrid multi-instance learning classification model based on transformer and graph attention network (GAT). When classifying pathological images, it considers regional correlation, solving the problem of lost contextual information and regional correlation in the pathological images to be classified. The classification results are more consistent with reality. Furthermore, this invention constructs a feature pyramid for pathological images, allowing the model to flexibly utilize global and local detail information. The feature pyramid in this invention only uses pathological image information at 20×, 10×, and 5× magnification, enabling the classification performance of this invention to reach the classification performance achievable with existing classification methods using images at a maximum magnification of 40×, effectively reducing memory consumption. The technical solution of this invention includes the following steps:

[0007] S1: Construct a feature pyramid for pathological images with labeled image-level tags;

[0008] S2: Construct a multi-instance learning network based on two attention branches: Transformer and Graph Attention Network. Input the constructed feature pyramid into the Transformer and Graph Attention Network branches of the learning network respectively for learning, and obtain the output H′ of these two branches respectively. SA and H′ GAT Combining the outputs of the two attention branches yields H′, i.e., H′ = H′ SA +H′ GATThen, an embedding aggregator is constructed for pathological image classification prediction to complete the construction of the classification model;

[0009] S3: Draw an attention heatmap.

[0010] Furthermore, S1 includes:

[0011] S11: Use the Otsu algorithm to remove blank backgrounds from pathological images at magnifications of 20×, 10×, and 5×.

[0012] S12: Crop the entire pathological image into several 512×512 image blocks to obtain several patch images of the pathological image at various magnifications;

[0013] S13: Use the ResNet50 model pre-trained on the Imagnet dataset to extract features from the patch image at the corresponding magnification factor;

[0014] S14: Compress the 512×512 image block into a 1024-dimensional feature vector, and use the structural characteristics of the digital pathological image pyramid storage to fuse the feature vectors of pathological images at magnifications of 20×, 10× and 5× to form a feature pyramid.

[0015] Furthermore, S3 includes:

[0016] S31: Construct a feature pyramid based on unlabeled pathological images to be classified;

[0017] S32: Input the feature pyramid into the classification model, perform classification, and draw an attention heatmap.

[0018] Furthermore, when constructing the embedding aggregator for pathological image classification prediction in S2, the entire pathological image is regarded as a package, and each small image patch, i.e. each cropped patch, is regarded as an instance. The specific embedding method is to embed the package directly supervised by the package label into the aggregator to generate a package score.

[0019] Furthermore, when the classification model in S32 classifies the input pathological image based on its feature pyramid, it needs to predict the label c(B) of the corresponding bag B. The specific prediction method is as follows:

[0020] Let package B = {x1, x2, ..., x} K}, instance x k Corresponding label y k (0,1), k∈K, where K represents the number of instances in a package, and the label c(B) of package B is:

[0021]

[0022] The multi-instance learning method uses a suitable transformation f and g to predict the label of B, as follows:

[0023] c(B)=g(f(X1),...,f(X K ))

[0024] f represents the ResNet50 feature extractor, and g is the aggregation operator, which specifically represents the following transformation:

[0025] Let H be the input feature pyramid, and H′ be the combination of the outputs of the two attention branches. In the embedding aggregator, N parallel and independent classifiers W are built for H′. c,1 W c,i ,...,W c,N For each class i∈N, the package score is obtained as follows:

[0026]

[0027]

[0028] Where H K Let H′ be the feature vector of the k-th instance in the feature pyramid. (i) The network model outputs the feature value of the i-th class, H′. (i)k Output the feature vector value of the kth instance of the i-th class to the network model; A i,k It is the attention score of the k-th patch of the i-th class after passing through the attention network, h bag,i It is a slide-level representation aggregated based on the attention score distribution of the i-th class, h bag,i ∈R 512×1 R represents the feature vector, 512×1 represents a single feature vector, and the packet score can be obtained through the classifier W. c,i ∈R 1×512 get;

[0029] Finally, the predicted probability S for each category is calculated by applying the softmax function. bag,i The category with the highest predicted probability is selected as the prediction result.

[0030] S bag,i =W c,i h bag,i .

[0031] Furthermore, S12 specifically involves cropping the entire pathological image using a sliding window method.

[0032] Furthermore, when drawing the attention heatmap in S32, the attention score of each instance is obtained by normalizing H′ obtained through the learning network using the softmax function, and the attention heatmap is drawn using the applyColorMap() function in the OpenCV library.

[0033] Furthermore, the attention heatmap in S32 uses a blue-red color scheme, with blue representing low attention areas and red representing high attention areas, and the transparency is set to 0.5, overlapping with the original image.

[0034] The beneficial effects of this invention are:

[0035] 1. This invention designs a hybrid multi-instance learning classification method based on transformer and graph attention network (GAT), which takes into account regional correlation and solves the problem of loss of contextual information and regional correlation of the image to be classified.

[0036] 2. This invention constructs a feature pyramid by combining pathological image information at multiple low magnifications (20×, 10×, 5×), which allows the model to flexibly utilize global and local detail information. Moreover, the feature pyramid in this invention only uses pathological image information at 20×, 10×, and 5×, which enables the classification performance of this invention to reach the classification performance that can be achieved by existing classification methods using images with a maximum magnification of 40×, effectively reducing memory consumption.

[0037] 3. The weakly supervised method for all-slide image classification based on multi-instance learning in this invention only uses image-level labels, eliminating the need for extensive manual annotation, thus reducing human workload. It also avoids assigning the same label to each image patch, preventing the introduction of a large amount of noise with incorrect labels into the model. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of the execution flow of the method of the present invention;

[0040] Figure 2 This is a feature pyramid construction diagram of the method of the present invention;

[0041] Figure 3 This is a network model diagram of the method of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Example 1: The specific operational steps of this invention are illustrated by classifying three subtypes of renal cell carcinoma pathological images (papillary renal cell carcinoma, chromophobe renal cell carcinoma, and clear cell renal cell carcinoma). Figure 1 As shown.

[0044] S1: Construct the feature pyramid, see... Figure 2 .

[0045] Specifically, S1 includes:

[0046] S11: Use the Otsu algorithm to remove blank backgrounds from renal cell carcinoma pathological images with image-level labels at magnifications of 20×, 10×, and 5×.

[0047] Otsu's algorithm, also known as the maximum inter-class variance thresholding algorithm or Otsu's algorithm, is derived from the least squares method and is used for simple threshold determination. For a grayscale image, we can find the maximum and minimum grayscale values, iterate through the minimum to the maximum value, and take each value as the segmentation threshold. The larger the inter-class variance between the background and the foreground, the greater the difference between the two parts of the image. When a foreground is misclassified as background or a background is misclassified as foreground, the difference between the two parts will become smaller. Therefore, maximizing the inter-class variance means minimizing the probability of misclassification. Thus, we can find the corresponding segmentation threshold by maximizing the inter-class variance. This algorithm is simple and can effectively segment images when the areas of the target and the background are not significantly different.

[0048] S12: Using a window sliding method, the entire renal cell carcinoma pathological image with up to one billion pixels is cropped into several image blocks of 512×512 size, resulting in several patch images of the renal cell carcinoma pathological image at various magnifications.

[0049] S13: Use the ResNet50 model pre-trained on the Imagnet dataset to extract features from the patch image at the corresponding magnification factor;

[0050] The ImageNet dataset is a huge image library that can be used for training visual object recognition software.

[0051] ResNet50 is a deep learning model. The "Res" in ResNet50 stands for residual, because this model uses residual learning. The main characteristic of ResNet50 is its depth; it has 50 layers of convolutional neural networks. This depth allows it to learn more complex features, thus improving its accuracy. However, one problem with deep learning models is vanishing gradients, which can make the model untrainable. To avoid this problem, ResNet50 uses residual blocks to implement residual learning. Each residual block contains two convolutional layers and a skip connection, which directly passes the input to the output. Another feature of ResNet50 is its use of global average pooling layers. This layer takes the average value of all pixels in each feature map as the output of that feature map. The purpose of this layer is to reduce the number of parameters in the model, thereby reducing the risk of overfitting.

[0052] S14: Compress the 512×512 image block into a 1024-dimensional feature vector, and use the structural characteristics of the digital renal cell carcinoma pathological image pyramid storage to fuse the feature vectors of renal cell carcinoma pathological images at magnifications of 20×, 10× and 5× to form a feature pyramid.

[0053] Feature pyramids are currently an important component used in object detection, semantic segmentation, and behavior recognition. They are highly effective in improving model performance because visual tasks involve objects of different sizes. Feature pyramids have different resolutions at different scales, allowing targets of different sizes to have appropriate feature representations at their respective scales. By fusing multi-scale information, predictions can be made for targets of different sizes at different scales, thereby obtaining richer image information.

[0054] The use of the feature pyramid effectively reduced memory consumption and model training time. Table 1 reflects the comparison of memory consumption and model training time between using the feature pyramid and using images with 40× magnification (the data in this table is obtained from training tests on 856 renal cell carcinoma pathology images using an NVIDIA Tesla V100 graphics processor).

[0055] Table 1 compares the memory consumption and model training time using the pyramid model and using the maximum magnification of 40× for renal cell carcinoma pathological images in this invention.

[0056] Memory consumption (G) 88.8G 10.1G Training duration (V100) 120h 48h

[0057] As can be seen from Table 1, the feature pyramid reduces memory consumption by about 8 times and training time by about 3 times.

[0058] S2: Construct an image classification model, see Figure 3 .

[0059] Specifically, S2 includes:

[0060] S21: Construct a multi-instance learning network based on two attention branches: Transformer and Graph Attention Network (GAT);

[0061] Multiple Instance (MIL) is a popular weakly supervised learning method. The data unit in the MIL dataset is a bag. Taking binary classification as an example, a bag contains multiple instances. The entire pathological image is considered a bag, and each cropped image patch is considered an instance. If all instances are labeled negatively, then the bag is negative; otherwise, the bag is positive. The specific representation is as follows:

[0062] Let package B = {x1, x2, ..., x} K}, instance x k Corresponding label y k (0,1), k∈K. Here, K represents the number of instances in a bag, and the number of instances in K may vary depending on the bag. When classifying unlabeled pathological images, the label c(B) of the bag B corresponding to the pathological image is determined by:

[0063]

[0064] c(B)=g(f(x1),...,f(x K (2)

[0065] MIL predicts the label c(B) of packet B by using a suitable transformation f and a permutation-invariant g. The selection of f and g is based on the embedding method, where f is a transformation that maps each instance to an instance-level feature extractor of the embedding. The embedding method of this invention is based on packet embedding directly supervised by the packet label to generate packet scores, which generally achieves better accuracy. In this invention, f represents the ResNet50 feature extractor, and g is an aggregation operator based on the packet embedding outputs packet scores. The transformation specifically represented by operator g in this invention is as follows:

[0066] Let H be the input feature pyramid, and H′ be the combination of the outputs of the two attention branches. In the embedding aggregator, N parallel and independent classifiers W are built for H′. c,1 W c,i ,...,W c,N For each class i∈N, the package score is obtained as follows:

[0067]

[0068]

[0069] Where H K Let H′ be the feature vector of the k-th instance in the feature pyramid. (i) The network model outputs the feature value of the i-th class, H′. (i)k Output the feature vector value of the kth instance of the i-th class to the network model.

[0070] A i,k It is the attention score of the k-th patch of the i-th class after passing through the attention network, h bag,i It is a slide-level representation aggregated based on the attention score distribution of the i-th class, h bag,i ∈R 512×1 R represents the feature vector, and 512×1 represents a single feature vector. The package score can be obtained through the classifier W. c,i ∈R 1×512 get.

[0071] Finally, the predicted probability S for each category is calculated by applying the softmax function. bag,i The category with the highest predicted probability is selected as the prediction result.

[0072] S bag,i =W c,i h bag,i (5)

[0073] S22: Input the constructed feature pyramid into the transformer and GAT branches of the learning network respectively for learning, and obtain the output H′ of these two branches respectively. SA and H′ GAT ;

[0074] (1) The specific function representation of the Transformer branch is as follows:

[0075] H l =MAS(LN(H l-1 ))+H l-1 , l=1...L (6)

[0076] H l =MLP(LN(H l ))+H l , l=1...L (7)

[0077] H′ SA =f SA H L (8)

[0078] Where H = {h1, h2, ..., h K},h i ∈Rd These are pathological image features of renal cell carcinoma. MSA represents multi-head self-attention, L is the number of blocks in the MSA, MLP represents multilayer perceptron, LN represents LayerNorm (layer range), and H... L This is the output of the Transformer, finally H. L After a fully connected layer F SA ∈R N×d H′ SA ∈R N×K The output features of the Transformer branch are denoted by N, where N is the number of categories. In this method, d is 512 and L is set to 2.

[0079] The Transformer is a model that utilizes attention mechanisms to improve training speed. Essentially, it's an Encoder-Decoder architecture, where the encoding component consists of multiple encoder layers, and the decoding component consists of decoders with the same number of layers. Each encoder comprises two sub-layers: a Self-Attention layer and a Position-wise Feed Forward Network (FFN). While the structure of each encoder is identical, they use different weight parameters. Self-Attention is the core of the Transformer. The encoder's input first flows into the Self-Attention layer, which allows the encoder to use relevant information from other parts of the input when encoding a particular part (in translation, this can be understood as focusing not only on the current word but also on information from other words). The output of the Self-Attention layer flows into the Feed Forward Network. The decoder also has these two layers from the encoder, but there's an additional attention layer (Encoder-Decoder Attention) between them, which helps the decoder focus on relevant parts of the input information.

[0080] (2) The specific functional representation of the graph attention network branch is as follows:

[0081] e ij =LeakyReLU(a(Wh) i ,Wh j (9) + WSI_adj)

[0082]

[0083]

[0084] Where W∈R N×d It is a learnable shared linear transformation, a:Rd′ ×R d′ →R is a shared attention mechanism. WSI_adj∈R K×K It is a graphical representation of the pathological image of renal cell carcinoma, e ij It is the attention coefficient of each node, a ij It is e ij The score after standardization using the softmax function, h′ i This is the output feature after executing the M-head attention mechanism. We use dropout (P = 0.25) for regularization before the attention backbone. Finally, we use equation (11) to calculate the single-head attention again to obtain the output feature H′ of the GAT network. GAT ∈R N×K

[0085] The LeakyReLU function is a novel activation function for neural network units that can improve the performance of deep neural networks. It can be used to address issues such as vanishing gradients and jitter, and can better fit the model.

[0086] WSI_adj is the graph structure representation of the WSI in a renal cell carcinoma pathological image. This invention utilizes a traditional multi-instance learning aggregation method (max pooling and average pooling) to construct the adjacency matrix of the renal cell carcinoma pathological image. The purpose is to enable the GAT branch to learn the contextual and spatial information of the renal cell carcinoma pathological image. The feature pyramid is passed through a max pooling layer and an average pooling layer, and then the outputs of the two pooling layers are multiplied by a matrix. Finally, this is input into the GAT branch for learning using a graph attention mechanism.

[0087] S23: Combine the outputs of the two attention branches to obtain H′, that is: H′=H′ SA +H′ GAT Then, an embedding aggregator (bag prediction) capable of classifying and predicting pathological images of renal cell carcinoma is constructed, thus completing the construction of the classification model.

[0088] Embedded aggregators are a modeling approach for multi-instance learning.

[0089] S3: Draw an attention heatmap.

[0090] Specifically, S3 includes:

[0091] S31: Construct a feature pyramid based on unlabeled pathological images of unclassified renal cell carcinoma;

[0092] S32: Input the feature pyramid into the classification model, perform classification, and draw an attention heatmap. Specifically:

[0093] The feature pyramid is input into the model, and the output H′ of the two attention branches of the model is normalized using the softmax function to obtain the attention score for each instance. The attention heatmap is then drawn using the applyColorMap() function from the OpenCV library. This invention uses a blue-red color scheme when drawing the attention heatmap, with blue representing low-attention areas and red representing high-attention areas. Smaller attention scores are closer to blue, and larger scores are closer to red. The transparency is set to 0.5, and the heatmap is displayed overlapping with the original image.

[0094] Attention heatmaps are simply aggregated large amounts of data and presented elegantly using a gradient of colors to visually represent the density or frequency of spatial data.

[0095] OpenCV is a cross-platform computer vision and machine learning software library that is lightweight and efficient—consisting of a series of C functions and a small number of C++ classes, while also providing interfaces for languages ​​such as Python, Ruby, and MATLAB, and implementing many common algorithms in image processing and computer vision.

[0096] S4: Evaluate model performance.

[0097] Specifically, S4 includes:

[0098] This invention evaluates the model's performance by calculating metrics such as predicted AUC, accuracy (ACC), precision (Precision), recall, and F1 score (F1), wherein:

[0099]

[0100] AUC (Area Under Curve) is the area under the receiver operating characteristic curve, which is between 0.1 and 1. It can intuitively evaluate the quality of the classifier. The larger the value, the better the classifier. M is the number of positive samples and N is the number of negative samples. (In this embodiment of the invention, papillary renal cell carcinoma, chromophobe renal cell carcinoma and clear renal cell carcinoma are classified into three types. When papillary renal cell carcinoma is set as a positive sample, the other two types of renal cell carcinoma are negative samples; when chromophobe renal cell carcinoma is set as a positive sample, the other two types of renal cell carcinoma are negative samples, and so on.)

[0101]

[0102] Accuracy (ACC) is used to predict the proportion of samples that are correctly predicted (including correctly predicted positive samples and correctly predicted negative samples).

[0103]

[0104] Precision measures how accurately a model predicts a particular class.

[0105]

[0106] Recall represents the proportion of correctly predicted samples in a particular class compared to all true values.

[0107]

[0108] The F1 score, which takes into account both precision and recall, is a harmonic value of precision and recall.

[0109] Wherein, TP: the number of positive classes predicted as positive, FN: the number of positive classes predicted as negative, FP: the number of negative classes predicted as positive, and TN: the number of negative classes predicted as negative.

[0110] Table 2 reflects the comparison results of the classification performance of the present invention and other methods on the TCGA dataset of renal cell carcinoma pathological images, including AUC, accuracy (ACC), precision, recall, and F1 score (F1).

[0111] Table 2 Comparison of the image classification performance of this invention with other methods in renal cell carcinoma.

[0112] mil 0.9697 0.9050 0.9067 0.9050 0.9053 clam_sb 0.9723 0.9118 0.9162 0.9118 0.9124 clam_mb 0.9799 0.9151 0.9188 0.9151 0.9156 dsmil 0.9811 0.9211 0.9232 0.9211 0.9213 transMIL 0.9710 0.9274 0.9336 0.9274 0.9283 This invention 0.9848 0.9309 0.9341 0.9309 0.9315

[0113] As can be seen from Table 2, the method of the present invention is optimal in all five evaluation indicators, namely AUC, accuracy (ACC), precision (Precision), recall, and F1 score, for multi-classification of renal cell carcinoma pathological images.

[0114] This invention solves the following problems:

[0115] 1. This invention designs a hybrid multi-instance learning classification method based on transformer and graph attention network (GAT), which takes into account regional correlation and solves the problem of loss of contextual information and regional correlation of the image to be classified.

[0116] 2. This invention constructs a feature pyramid by combining pathological image information at multiple low magnifications (20×, 10×, 5×), which allows the model to flexibly utilize global and local detail information. Moreover, the feature pyramid in this invention only uses pathological image information at 20×, 10×, and 5×, which enables the classification performance of this invention to reach the classification performance that can be achieved by existing classification methods using images with a maximum magnification of 40×, effectively reducing memory consumption.

[0117] 3. The weakly supervised method for all-slide image classification based on multi-instance learning in this invention only uses image-level labels, eliminating the need for extensive manual annotation, thus reducing human workload. It also avoids assigning the same label to each image patch, preventing the introduction of a large amount of noise with incorrect labels into the model.

[0118] Example 2: The specific operation steps of the present invention are illustrated by realizing the binary classification of whether metastasis has occurred in lymph nodes in breast cancer pathological images.

[0119] S1: Construct the feature pyramid, see... Figure 2 .

[0120] Specifically, S1 includes:

[0121] S11: Use the Otsu algorithm to remove blank backgrounds from breast cancer pathology images with labeled image levels at magnifications of 20×, 10×, and 5×.

[0122] S12: Using a window sliding method, the entire pathological image with up to one billion pixels is cropped into several 512×512 image blocks to obtain several patch images of the breast cancer pathological image to be classified at various magnifications.

[0123] S13: Use the ResNet50 model pre-trained on the Imagnet dataset to extract features from the patch image at the corresponding magnification factor;

[0124] S14: Compress the 512×512 image block into a 1024-dimensional feature vector, and use the structural characteristics of the digital breast cancer pathology image pyramid storage to fuse the feature vectors of breast cancer pathology images at magnifications of 20×, 10× and 5× to form a feature pyramid.

[0125] S2: Construct an image classification model, see Figure 3 .

[0126] Specifically, S2 includes:

[0127] S21: Construct a multi-instance learning network based on two attention branches: Transformer and Graph Attention Network (GAT);

[0128] S22: Input the constructed feature pyramid into the transformer and GAT branches respectively for learning, and obtain the outputs of these two branches;

[0129] S23: Combine the outputs of the two attention branches to construct an embedding aggregator (bag Prediction), thus completing the construction of the classification model.

[0130] S3: Draw an attention heatmap.

[0131] Specifically, S3 includes:

[0132] S31: Construct a feature pyramid based on unlabeled pathological images of breast cancer to be classified;

[0133] S32: Input the feature pyramid into the classification model, perform classification, and draw an attention heatmap.

[0134] S4: Evaluate model performance.

[0135] Specifically, S4 includes:

[0136] This invention evaluates the model's performance by calculating metrics such as predicted AUC, accuracy (ACC), precision (Precision), recall, and F1 score (F1), wherein:

[0137] Table 3 shows the comparison results of the classification performance of the present invention and other methods on the CAMELYON16 dataset of breast cancer pathology images, including AUC, accuracy (ACC), precision, recall and F1 score (F1).

[0138] Table 3 Comparison of the effects of the present invention and other methods in breast cancer pathological image classification

[0139]

[0140]

[0141] As can be seen from Table 3, the method of the present invention is optimal in all five evaluation indicators for binary classification of breast cancer pathological images: AUC, accuracy (ACC), precision (Precision), recall, and F1 score (F1).

[0142] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0143] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A weakly supervised classification method for pathological images based on dual attention and hybrid multi-instance learning, characterized in that, Includes the following steps: S1: Construct a feature pyramid for pathological images with labeled image-level tags; S2: Construct a multi-instance learning network based on two attention branches: Transformer and Graph Attention Network. Input the constructed feature pyramid into the Transformer and Graph Attention Network branches of the learning network respectively for learning, and obtain the outputs of these two branches. and By combining the outputs of the two attention branches, we obtain ,Right now: Then, an embedding aggregator is constructed for pathological image classification prediction to complete the construction of the classification model; S3: Draw an attention heatmap; S1 includes: S11: Use the Otsu algorithm to remove blank backgrounds from pathological images at magnifications of 20×, 10×, and 5×. S12: Crop the entire pathological image into several 512×512 image blocks to obtain several patch images of the pathological image at various magnifications; S13: Use the ResNet50 model pre-trained on the ImageNet dataset to extract features from the patch image at the corresponding magnification factor; S14: Compress the 512×512 image block into a 1024-dimensional feature vector, and use the structural characteristics of the digital pathological image pyramid storage to fuse the feature vectors of pathological images at magnifications of 20×, 10× and 5× to form a feature pyramid. S2 includes: S21: Construct a multi-instance learning network based on two attention branches: Transformer and graph attention network; The specific representation of multiple instances is shown below: Set package , example Corresponding tags Where K represents the number of instances in a package, and the label of package B is... for: The multi-instance learning method uses a suitable transformation f and g to predict the label of B, as follows: , f represents the ResNet50 feature extractor, g is the aggregation operator, and the specific transformation represented by operator g is as follows: set up The feature pyramid is the input. The combination of the outputs of the two attention branches, in the embedding aggregator, is... Establish N parallel independent classifiers The package score for each class is obtained, as shown below: in Let k be the feature vector of the k-th instance in the feature pyramid. Output the feature values ​​of the i-th class to the network model. Output the feature vector value of the kth instance of the i-th class to the network model; It is the attention score of the k-th patch of the i-th class after passing through the attention network. It is a slide-level representation aggregated based on the attention score distribution of the i-th class. R represents the feature vector, 512×1 represents a single feature vector, and the packet score can be obtained through a classifier. get; Finally, the predicted probability for each category is calculated by applying the softmax function. And select the category corresponding to the highest predicted probability as the prediction result; where, S22: Input the constructed feature pyramid into the transformer and GAT branches of the learning network respectively for learning, and obtain the outputs of these two branches respectively. and In the Transformer branch The output features of the Transformer branch are represented by N, where N is the number of classes. In the Graph Attention (GAT) branch, the feature pyramid is passed through a max-pooling layer and an average-pooling layer, respectively. The outputs of these two pooling layers are then multiplied by a matrix, and finally input into the GAT branch for learning using the graph attention mechanism. The output features of the GAT network are... ; S23: Combining the outputs of the two attention branches to obtain ,Right now: Then, an embedding aggregator that can perform pathological image classification prediction is constructed to complete the construction of the classification model; S3 includes: S31: Construct a feature pyramid based on unlabeled pathological images to be classified; S32: Input the feature pyramid into the classification model, perform classification, and draw an attention heatmap.

2. The weakly supervised classification method for pathological images based on dual attention and hybrid multi-instance learning according to claim 1, characterized in that, When constructing the embedding aggregator for pathological image classification prediction in S2, the entire pathological image is regarded as a bag, and each small image patch is regarded as an instance. The specific embedding method is to generate bag scores based on bag embedding directly supervised by bag labels.

3. The weakly supervised classification method for pathological images based on dual attention and hybrid multi-instance learning according to claim 1, characterized in that, S12 specifically refers to cropping the entire pathological image using a sliding window method.

4. The weakly supervised classification method for pathological images based on dual attention and hybrid multi-instance learning according to claim 1, characterized in that, When drawing attention heatmaps in S32, the softmax function is used to optimize the heatmaps obtained through the learning network. After normalization, the attention score for each instance can be obtained, and the attention heatmap can be drawn using the applyColorMap() function in the OpenCV library.

5. The weakly supervised classification method for pathological images based on dual attention and hybrid multi-instance learning according to claim 1, characterized in that, The attention heatmap in S32 uses a blue-red color scheme, with blue representing low-attention areas and red representing high-attention areas. The transparency is set to 0.5, and it is displayed overlaid with the original image.

Images