Gastric cancer subtype classification method based on multi-instance learning and multi-scale feature extraction

Abstract

The invention discloses a gastric cancer subtype classification method based on multi-instance learning and multi-scale feature extraction. The method at least solves three difficult problems in a traditional multi-instance learning tissue pathology picture classification method: the existing multi-instance learning method has single scale limitation, is too high in computing resource demand, and cannot effectively and synchronously pay attention to pathology image context information. In order to solve the problems, the method comprises the following steps: firstly, extracting features of pathological images under different magnification factors by using a ResNet backbone network, obtaining a sub-image Tile with rich information under a lower magnification factor through a parallel attention module and a differentiable module, and combining the sub-image Tile with a tissue pathological image under a higher magnification factor to obtain context associated information; and finally, realizing a gastric cancer subtype classification task by aggregating the features under different magnification times. Compared with an existing method, the method has the advantages that better recognition precision can be achieved by only needing fewer computing resources, and important reference can be provided for diagnosis of gastric cancer patients.

Description

Technical Field

[0001] The present invention relates to the field of pathological image analysis and mining, and in particular to a gastric cancer subtype classification method based on multi-instance learning and multi-scale feature extraction. Background Art

[0002] Gastric cancer is one of the leading causes of death in the world to date, with the second highest mortality rate in the world. Although the incidence rate has decreased in some parts of the world, gastric cancer remains a major clinical challenge due to the late diagnosis of most cases, poor prognosis, and limited treatment options. The World Health Organization mainly divides gastric cancer into common adenocarcinoma (GAC), mucinous adenocarcinoma (MUC), and signet ring cell carcinoma (SRC) based on the tissue structure, cell shape, and degree of differentiation of the cancer. Among them, GAC is the most common, and compared with it, the clinical pathological characteristics of MUC are significantly different: larger size; deeper invasion; higher lymph node metastasis rate. In addition, as a variant of MUC, SRC also produces a large amount of mucin, but clinically manifests as a poorly differentiated adenocarcinoma with a higher degree of malignancy, which is easy to grow diffusely and infiltratively in the gastric wall, has strong invasiveness, high metastasis rate, and the nucleus is squeezed to one side of the cytoplasm to appear like a "signet ring". It is often not discovered until the late stage. It is reported that early and accurate detection of gastric cancer can overwhelmingly increase the 5-year survival rate by about 90%. Therefore, accurate morphological classification of MUC, GAC, and SRC is very necessary in clinical diagnosis, and pathological detection is the gold standard for diagnosing various subtypes of gastric cancer, and the detection results have a great impact on the treatment plan.

[0003] In clinical settings, pathologists usually make diagnoses based on the morphological features of tissue pathology images observed at different scales and referring to medical reports. However, the manual diagnosis process is time-consuming, subjective, and non-repeatable. Currently, researchers have proposed a deep learning method specifically for the field of digital pathology - Multiple Instance Learning (MIL).

[0004] However, existing multi-instance learning is limited by a single scale, high computational cost, and failure to effectively integrate contextual information of pathological images. This often leads to long subtype classification tasks and misjudgment of detection results, and cannot meet the needs of clinical auxiliary diagnosis. Summary of the invention

[0005] The present invention provides a gastric cancer subtype classification method based on multi-instance learning and multi-scale feature extraction for gastric cancer subtype pathological image detection. The present invention uses the ResNet backbone network to extract the features of the pathological image at different magnifications; then, the information-rich sub-image (Tile) at a lower magnification is obtained through a parallel attention module and a differentiable module, and it is combined with the tissue pathology image at a higher magnification to obtain contextual information; finally, the gastric cancer subtype classification task is achieved by aggregating the features at different magnifications. While reducing the computational requirements, it is used for the classification of common adenocarcinoma, mucinous adenocarcinoma, and signet ring cell carcinoma images, improves the accuracy of gastric cancer subtype detection, and can meet the needs of clinical auxiliary diagnosis.

[0006] The technical solution of the present invention is: a gastric cancer subtype classification method based on multi-instance learning and multi-scale feature extraction, comprising:

[0007] S1, collect pathological images of gastric cancer patients and corresponding clinical label data;

[0008] S2, preprocessing each original tissue pathology image WSI to obtain sub-images Tile of the tissue pathology image at different magnifications;

[0009] S3, for the sub-image Tile obtained in S2, the Tile features at different magnifications are extracted through the ResNet backbone network pre-trained on the ImageNet dataset. Finally, the extracted features are packaged into an .h5 file and divided into training set, validation set and test set in a ratio of 8:1:1;

[0010] S4, sending the pathological image features at low magnification obtained in S3 to a multi-instance learning parallel attention module to obtain a channel attention score and a spatial attention score at the magnification;

[0011] The construction process of the multi-instance learning parallel attention module in S4 is as follows:

[0012] S4.1 Multi-instance learning is the basis of subsequent tasks. The input X is a package consisting of multiple instances, denoted by X = {x1,...,x N}; Given a classification task with C class labels, each instance has an unknown label y i ∈C, X has a known label y∈C; the original tissue pathology image WSI is taken as input, and the instance represents the extracted Tile set; among them, the slice-level Tile-level feature extractor h extracts each Tile x i Mapping to feature vector Indicates that each element in the tensor is a D-dimensional real number. Then, the pooling operator g(·) transforms the feature vector h i＝1:N, N is the number of feature vectors, which are aggregated to generate a feature representation of a single WSI; finally, the classifier f(·) uses the aggregated feature representation to predict the WSI-level label The process can be summarized as follows:

[0013]

[0014] Among them, Avgpool is the average pooling operation; Maxpool is the maximum pooling operation; f 7×7 is a 7×7 convolution operation; σ is a Sigmoid activation function; MLP is a fully connected layer with shared weights; finally, the obtained Tile-level attention score s i 、c i The eigenvector h corresponding to the eigenmatrix i Multiply and accumulate them to obtain the WSI-level spatial attention feature map or spatial attention score sa and channel attention feature map or channel attention score ca;

[0015] S5, the channel attention score obtained in S4 is used through a differentiable module to obtain the information-rich tile at a magnification of m times, and a multi-scale concat operation is performed with the features at a magnification of m+1 times obtained in S3;

[0016] S6, for the combined result of S5, perform the same operation as S4 and S5 to obtain the spatial attention score and information-rich tile at this magnification;

[0017] S7, the above process S3-S6 is repeated until the maximum magnification is m+n times, and finally all features under each magnification are aggregated for output prediction;

[0018] S8. Input the training set data of S3 into the multi-instance learning parallel attention module, and combine it with the multi-scale concat splicing in S5 for training. Then use the validation set of S3 to verify the performance of the model obtained from each batch of training, and save the model parameters that best achieve the best indicators on the validation set as the weights of the final classification model; finally, verify the performance of the saved model in classifying gastric cancer subtype pathological images on the test set of S3.

[0019] As a further description of the above scheme: The processing of the S2 original full-view digital slice image WSI includes the following steps:

[0020] Eliminate unmatched data, use a sliding window to crop the tissue pathology images at different magnifications into sub-blocks of preset sizes, and filter out sub-blocks with blank backgrounds exceeding a first preset threshold; color-standardize all sub-blocks. After the above operations are completed, save the cropped sub-blocks separately according to the corresponding tissue pathology images.

[0021] As a further description of the above scheme: S3 extracts Tile features at different magnifications through a ResNet backbone network pre-trained on the ImageNet dataset, specifically including the following steps:

[0022] Multi-scale tile generation: Extract image tiles at different magnifications from the original gastric cancer pathology slice images to obtain multi-scale information;

[0023] ResNet pre-training: Use ResNet pre-trained on the ImageNet dataset as the backbone network for feature extraction. The pre-trained ResNet has learned a large number of common image features, which helps to improve the effectiveness of feature extraction;

[0024] Feature extraction: Tiles at different magnifications are input into the pre-trained ResNet network to extract their corresponding feature vectors; these feature vectors can capture the cell, texture, and structural information of the tile at different magnifications; finally, the features extracted at all magnifications of an entire pathological image are saved as an .h5 file for subsequent use.

[0025] As a further description of the above scheme: the channel attention scores obtained by S4 are used through a differentiable module to obtain information-rich tiles with a magnification of m times. The specific process is described as follows:

[0026] When a WSI is accessible at multiple magnifications, these magnifications are represented by m∈{1,2,...,M}, where M represents the highest magnification. Assuming that the magnification of the WSI at m+1 is twice that at m, in order to identify the information tile at magnification m, the WSI-level channel attention feature map must be calculated first. It contains the channel attention score of the model for each tile. Then, the top K tiles with the highest attention scores are selected, namely the Top-K operation, and further processed at a higher magnification. The feature matrix of the selected tile can be expressed as:

[0027]

[0028] Among them, T m ∈{0,1} N×K is the indicator matrix, H m∈R N×D is the characteristic matrix of Tile when the magnification is equal to m;

[0029] The tile selection at m is directly performed through the predicted output of the classifier f(·), which is a classification head composed of a fully connected layer. This can be achieved by backpropagating from the output of the classifier f(·) to the attention module at m without introducing any additional losses or related hyperparameters. However, formula (4) cannot be directly differentiated because it involves Top-K operations, which are not mathematically differentiable everywhere. Therefore, the perturbation maximum method is used to make the Top-K selection differentiable and apply it to the attention score ca at the magnification m. m Specifically, we first add uniform Gaussian noise Then a linear program is solved for each of these perturbed attention weights and the results are averaged. Therefore, the forward pass of the differentiable Top-K module can be written as:

[0030]

[0031] where <·> is the scalar product before matrix vectorization; therefore Represents the perturbed attention score (ca m +σZ) is repeated K times in the second dimension to form a matrix of size N×K for subsequent operations; the corresponding Jacobian matrix is ​​defined as:

[0032]

[0033] in, represents the optimization goal, represents the decision variable, σ represents the noise standard deviation, is the transposed matrix of Gaussian noise Z;

[0034] The differentiable Top-K operation can learn the weighted attention score of the tile at a specific magnification; this means that the model can automatically learn how to select meaningful tiles without relying on predefined rules; in addition, the present invention also keeps the size of the tile selected at all magnifications unchanged, that is, it will not change due to the magnification; this can keep the number of tiles proportional to the magnification, so that the model can capture various contextual information, which is very important for analyzing tissue pathology sections because they usually contain many cell nuclei, cytoplasm, tissue stroma and diagnosis-related components; in order to combine tiles at other scales, the indicator matrix T m Expand to get from Tile feature (where m'>m) to select meaningful information; by calculating T m and the identity matrix The Kronecker product between them gives the expanded indicator matrix; similar to formula (3), the attention weights from the lower magnification m are used to select tiles at the higher magnification m':

[0035]

[0036] Among them, H m' is the characteristic matrix under magnification m', is the selected feature matrix;

[0037] Through the above calculations, an indicator matrix containing more information can be generated to enable more detailed tile selection at higher magnifications. It can also correspond to the attention weights at lower magnifications, which helps improve the model's ability to process input data.

[0038] As a further description of the above scheme: the output prediction of aggregating all features at various magnifications in S7 is specifically described as follows:

[0039] When the magnification m = 1, the feature matrix H1 passes through the parallel attention module PAM; PAM consists of the spatial attention module PA1 and the channel attention module PA1'. PA1 is used to obtain the optimal pooled WSI-level representation sa1 at low magnification; PA1' aims to provide a meaningful channel attention score ca1 for tile selection at higher magnification; then, the differentiable indicator matrix T1 is used to select the most informative tile, and the tile feature matrix at higher magnification is obtained through formula (7): The above process is repeated until the highest magnification M is reached; then, the Tile feature selected at M is Through the last parallel attention module PAM, the feature representation sa at the highest magnification is generated M In addition, since the features at different scales in pathological images are closely related and the summation exploits the complementarity between features, the attention pooling features of all magnifications are represented as sa1, sa2, …, sa M Aggregation is performed through sum pooling to obtain a multi-scale, context-aware WSI-level representation; the final classifier f(·) maps the WSI-level representation to the corresponding label y∈C and outputs the model prediction This process can be seen as an extension of formula (1), where summation and pooling are performed at multiple magnifications:

[0040]

[0041] The present invention also provides a gastric cancer subtype classification system based on multi-instance learning and multi-scale feature extraction: comprising the modules in the above method.

[0042] Compared with the prior art, the beneficial effects of the present invention are: the present invention solves at least three difficult problems in the multi-instance learning tissue pathology image classification method: the existing MIL method is limited to a single scale, too high a computational cost, and fails to effectively integrate the contextual information of the pathology image. In response to these problems, the present invention first uses the ResNet backbone network to extract the features of the pathology image at different magnifications; then, through the Parallel Attention Module (PAM) and the differentiable module, the information-rich sub-image (Tile) at a lower magnification is obtained, and it is combined with the tissue pathology image at a higher magnification to obtain contextual association information; finally, the gastric cancer subtype classification task is achieved by aggregating the features at different magnifications. Compared with the current advanced methods, the present invention only requires fewer computing resources to achieve a better recognition accuracy to meet the needs of clinical auxiliary diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of the present invention;

[0044] Figure 2 This is a sample of gastric cancer patient data from Yunnan Cancer Hospital;

[0045] Figure 3 This is the confusion matrix diagram of the classification results of the existing advanced methods on the gastric cancer patient data of Yunnan Cancer Hospital;

[0046] Figure 4 This is a confusion matrix diagram of the classification results of the present invention on the gastric cancer patient data of Yunnan Cancer Hospital;

[0047] Figure 5 It is a heat map of the present invention on the pathological image of gastric cancer subtype tissue. DETAILED DESCRIPTION

[0048] The invention is further described below in conjunction with the accompanying drawings and embodiments, but the content of the invention is not limited to the scope of the embodiments.

[0049] Example 1

[0050] A gastric cancer subtype classification method based on multi-instance learning and multi-scale feature extraction, the flowchart is as follows Figure 1 As shown, the specific steps include:

[0051] S1, collect pathological images of gastric cancer patients and corresponding clinical label data;

[0052] S2. Preprocess each original tissue pathology image (WSI) to obtain sub-images (Tile) of the tissue pathology image at different magnifications.

[0053] S3. For the pathological image tiles obtained in S2, the Tile features at different magnifications are extracted through the ResNet backbone network pre-trained on the ImageNet dataset. Finally, the extracted features are packaged into an .h5 file and divided into training set, validation set and test set in a ratio of 8:1:1.

[0054] S4. Send the pathological image features at low magnification obtained in S3 to a parallel attention module to obtain the channel attention score and spatial attention score at the magnification.

[0055] S5, the channel attention score obtained in S4 is used through a differentiable module to obtain information-rich tiles at low magnification, and a multi-scale concat operation is performed with the features at m+1 times obtained in S3;

[0056] S6, for the combined result of S5, perform the same operation as S4 and S5 to obtain the spatial attention score and information-rich tile at this magnification;

[0057] S7. The above processes S3-S6 are repeated until the highest magnification is reached, and finally all features at each magnification are aggregated for output prediction.

[0058] S8. Input the training set data of S3 into the multi-instance learning and multi-scale feature extraction model for training. Then use the validation set of S3 to verify the model performance obtained from each batch of training, and save the model parameters that best achieve the best indicators on the validation set as the weights of the final classification model; finally, verify the saved model performance on the test set of S3.

[0059] Each full-view digital slice image WSI in S2 is preprocessed, including: using a sliding window to crop the images of the tissue pathology image at different magnifications into sub-blocks of a preset size, and filtering out the sub-blocks whose blank background exceeds a first preset threshold (generally set to 20-30%); color standardizing all the sub-blocks. After the above operations are completed, the cropped sub-blocks are saved separately according to the corresponding tissue pathology images.

[0060] In S3, feature extraction is performed on the digital image slices Tiles obtained after segmentation in S2, including: multi-scale Tile generation, extracting image Tiles at different magnifications (e.g., 5x, 10x, 20x) from the original gastric cancer pathology slice images to obtain multi-scale information. ResNet pre-training, using ResNet (such as ResNet-50 or ResNet-101) pre-trained on the ImageNet dataset as the backbone network for feature extraction. The pre-trained ResNet has learned a large number of common image features, which helps to improve the effectiveness of feature extraction. Feature extraction, inputting Tiles at different magnifications into the pre-trained ResNet network to extract their corresponding feature vectors.

[0061] The construction process of the multi-instance learning parallel attention module in S4 is as follows:

[0062] S4.1. Input X is a bag consisting of multiple instances, denoted by X = {x1,...,x N Given a classification task with C class labels, each instance has an unknown label y i ∈C, the package has a known label y∈C. In this application, the input is WSI, and the instance represents the extracted tile set. Among them, the tile-level feature extractor h extracts each tile x i Mapping to feature vector Then, the pooling operator g(·) transforms the feature vector h i＝1:N Aggregation is performed to generate a feature representation of a single WSI. Finally, the classifier f(·) uses the aggregated feature representation to predict the WSI-level label The process can be summarized as follows:

[0063]

[0064] S4.2, use the attention pooling operation to aggregate Tile features and construct an attention multi-instance learning method. is the tile-level feature matrix, and the WSI-level feature representations sa and ca are:

[0065]

[0066] Among them, Avgpool is the average pooling operation; Maxpool is the maximum pooling operation; f 7×7 is a 7×7 convolution operation; σ is a Sigmoid activation function; MLP is a fully connected layer with shared weights. Finally, the obtained Tile-level attention score s i The corresponding element h in the feature matrix iMultiply and accumulate to obtain the WSI-level spatial attention feature map sa and channel attention feature map ca.

[0067] Furthermore, when a WSI is accessible at multiple magnifications, these magnifications are represented by m∈{1,2,...,M}, where M represents the highest magnification. Assuming that the magnification at m+1 is twice that at m, in order to identify the information tile at magnification m, the WSI-level channel attention feature map must be calculated first It contains the model's attention score for each tile. Then, the top K tiles with the highest attention scores are selected, namely the Top-K operation, and further processed at a higher magnification. The feature matrix of the selected tile can be expressed as:

[0068]

[0069] Among them, T m ∈{0,1} N×K is the indicator matrix, H m ∈R N×D It is the characteristic matrix of Tile when the magnification is equal to m.

[0070] Unlike the manual selection method, the present invention directly selects the tile at m through the predicted output of f(·). This can be achieved by backpropagating from the output of f(·) to the attention module at m without introducing any additional losses or related hyperparameters. However, formula (4) cannot be directly differentiated because it involves Top-K operations. Because the Top-K operation is a non-continuous operation, it selects the first K largest (or smallest) elements in the tensor, and its output depends not only on the value of the input tensor, but also on the value of K. Since the Top-K operation is not mathematically differentiable everywhere, it cannot be directly differentiated. In order to solve this problem, the present invention draws on the perturbation maximum method to make the Top-K selection differentiable, and applies it to the attention score ca at the magnification m. m Specifically, we first add uniform Gaussian noise Then a linear program is solved for each of these perturbed attention weights and the results are averaged. Therefore, the forward pass of the differentiable Top-K module can be written as:

[0071]

[0072] where <·> is the scalar product before matrix vectorization; therefore Represents the perturbed attention score (ca m+σZ) is repeated K times in the second dimension to form a matrix of size N×K for subsequent operations. The corresponding Jacobian matrix is ​​defined as:

[0073]

[0074] The differentiable Top-K operation can learn the weighted attention scores for tiles at specific magnifications. This means that the model can automatically learn how to select meaningful tiles without relying on predefined rules. In addition, the present invention also keeps the size of the selected tiles unchanged at all magnifications, that is, it will not change due to the magnification. This can keep the number of tiles proportional to the magnification, allowing the model to capture a variety of contextual information, which is very important for analyzing tissue pathology sections because they usually contain many diagnosis-related components such as cell nuclei, cytoplasm, and tissue stroma. In order to combine tiles at other scales, the present invention converts the indicator matrix T m Extensions to enable access from Tile features (where m'>m) to select meaningful information. Further, the present invention calculates T m and the identity matrix The Kronecker product between them gives the expanded indicator matrix. Similar to formula (3), the attention weights from the lower magnification m are used to select tiles at the higher magnification m':

[0075]

[0076] Among them, H m' is the characteristic matrix under magnification m', is the selected feature matrix.

[0077] Through the above calculations, an indicator matrix containing more information can be generated to enable more detailed tile selection at higher magnifications. It can also correspond to the attention weights at lower magnifications, which helps improve the model's ability to process input data.

[0078] Further: In S7, all features at various magnifications are aggregated to perform output prediction as follows:

[0079] When the magnification m = 1, the feature matrix H1 passes through PAM. PAM consists of a spatial attention module PA1 and a channel attention module PA1'. PA1 is used to obtain the optimal pooled WSI-level representation sa1 at low magnification. PA1' aims to provide a meaningful attention score ca1 for tile selection at higher magnification. Then, the differentiable indicator matrix T1 is used to select the most informative tile, and the tile feature matrix at higher magnification is obtained by formula (7): The above process is repeated until the highest magnification M is reached. Then, the Tile feature selected at M Through the last parallel attention module PAM, the feature representation sa at the highest magnification is generated M In addition, since the features at different scales in pathological images are closely related and the summation exploits the complementarity between features, the attention pooling features of all magnifications are represented as sa1, sa2, …, sa M Aggregation is performed through sum pooling to obtain a multi-scale, context-aware WSI-level representation. The final classifier f(·) maps the WSI-level representation to the corresponding label y∈C and outputs the model prediction This process can be seen as an extension of formula (1), where summation and pooling are performed at multiple magnifications:

[0080]

[0081] Furthermore, during the overall training of the model, the back-propagation algorithm is used for optimization after the loss is calculated, and the Adam optimizer is selected for optimization. It is simple to implement, efficient in calculation, and requires little memory. The hyperparameters are well interpretable and usually require no adjustment or only very little fine-tuning. The update step size can be limited to a rough range. It is very suitable for scenarios with large-scale data and parameters.

[0082] (1) Accuracy: Accuracy is the ratio of the number of correct predictions (including positive and negative) to the total number.

[0083]

[0084] Among them, TP, FP, TN, FN, and TN are true positive, false positive, false negative, and true negative (the same below), respectively.

[0085] (2) Precision (Pre): This indicator measures the ratio of the number of true positive predictions to the total number of positive predictions.

[0086]

[0087] (3) Recall (Re): This metric measures the ratio of the number of correctly predicted positives to the total number of actual positives.

[0088]

[0089] (4) F1-Score (F1): F1-Score takes into account both the precision and recall of the classification model. It can be regarded as a weighted average of the precision and recall of the model. The maximum value is 1 and the minimum value is 0.

[0090]

[0091] Furthermore, in order to prove the effectiveness of the method proposed in the present invention, a large number of experiments were conducted on the gastric cancer subtype data of Yunnan Cancer Hospital and public data sets. The ablation experiment results are shown in Tables 1-3; the comparison results with the existing advanced methods are shown in Table 4; the heat map of the present invention on the gastric cancer patient data of Yunnan Cancer Hospital is shown in Table 6;

[0092] Specifically, the present invention will conduct ablation analysis on the number of parallel attention modules, differentiable modules and Top-K selections to explore their impact on model performance.

[0093] Parallel attention module: Using 5×+10×+20× data, ablation experiments were conducted on variants of the parallel attention module: (1) Only using channel attention to select tiles at higher magnifications and generate features at this magnification, its Acc, Pre, Re, and F1 are 87.32%, 86.67%, 87.76%, and 87.21%; (2) Only using spatial attention, its Acc, Pre, Re, and F1 are 84.68%, 87.04%, 83.93%, and 85.45%. (3) Using channel attention to generate features at this magnification and using spatial attention to select tiles at higher magnifications, its Acc, Pre, Re, and F1 are 85.43%, 84.21%, 86.42%, and 85.30%, respectively. The results are shown in Table 2.

[0094] Comparing the experimental results of different variants, when selecting tiles at higher magnifications, the results of using only channel attention are significantly better than those of using only spatial attention. This may be because the details of WSI are richer at higher magnifications, and each tile may contain more consistent semantic information, but the information differences between different channels are large, and channel attention can effectively capture the differences between these channels, thereby improving the expressiveness of features. In contrast, spatial attention may be affected by excessive image details and increased complexity at high magnifications, resulting in an inability to effectively focus on key areas. In addition, when spatial attention is used to generate features at this magnification, the results are better than channel attention. This may be because spatial attention pays more attention to the spatial details and regional features of the image at a single magnification, which can help the model better focus on important image areas, while channel attention is mainly used to adjust and optimize the weights of different feature channels and is not suitable for such tasks.

[0095] Table 1 Experimental results using different attention (%)

[0096]

[0097]

[0098] Differentiable module: This paper designs a non-differentiable Tile selection module to conduct comparative experiments with the differentiable module to verify the effectiveness of the differentiable method. Specifically, the model only relies on the attention score to select tiles, and does not perform back-propagation updates, which makes the selection process more random and fixed. The experimental results are shown in Table 3. The results show that it is particularly important for the model to be able to perform gradient updates in the process of selecting tiles, because unlike natural images, WSI has diverse features, and relying solely on attention scores for Tile selection may not be able to fully capture the important information in the image. The use of a differentiable method can provide the model with stronger learning and generalization capabilities. By making full use of gradient updates, the model can adjust the attention score according to the feedback signal, making the selected tile more accurate and targeted, thereby improving the accuracy of disease diagnosis and prediction. It is worth noting that although the selection of tiles is fixed in the method of this paper, by combining multiple magnifications, the model can obtain richer information from different scales, thereby achieving better performance in pathological image classification tasks. Compared with using only 10× and 20× data, Acc, Pre, Re and F1 using 5×, 10× and 20× data are improved by 2.28%, 1.17%, 2.38% and 1.64% respectively.

[0099] Table 2 Experimental results using differentiable and non-differentiable modules (%)

[0100]

[0101] Selection of Top-K: The number of selected tiles is crucial in the entire reasoning process of the model. The present invention conducted an ablation experiment analysis on the number of selected tiles under 5×, 10× and 20× data, and selected several most representative values. The experimental results are shown in Table 4. The experimental results show that as the number of selected tiles increases, the FLOPs of the model also increases accordingly, while the accuracy shows a slightly downward trend. This is because the model needs to process more input information, which increases the complexity of the model and increases the risk of overfitting. In this article, K=30 is a more ideal choice. While maintaining a high accuracy (Acc is 89.12%), the computational cost is still relatively low (FLOPs is 14.32).

[0102] Table 3 Experimental results of selecting different numbers of Top-K

[0103]

[0104] The present invention is compared with several classic multiple instance learning algorithms under 5×, 10× and 20× data, and the results are shown in Table 8. The calculation floating point number (FLOPs) and throughput of each model are shown in Table 5. Specifically, the four methods using single-scale WSI, Mean Pooling, Max Pooling, ABMIL, and CLAM, differ by 1.33%-4.73% when facing data with a magnification of 5× compared with data with other magnifications. This may be because in the task of gastric cancer subtype classification, the WSI under a magnification of 5× has fewer details and the tissue structure is not clear enough, while the features of the nuclear morphology, intercellular matrix, and tissue structure of the WSI under a magnification of 10× or 20× are clearer and more detailed, which helps the model to capture key features more accurately, thereby improving the classification effect. It is worth noting that ABMIL uses attention-based MIL pooling for WSI-level learning, which is similar to the present invention, but since the label of each instance is unknown, such training is often insufficient and will introduce additional errors. Its Acc, Pre, Re and F1 at a magnification of 10× are 82.60%, 83.33%, 81.55%, and 81.48%, respectively. In addition, CLAM also combines attention-based learning and additional patch clustering during training, and performs best among the four methods. Its Acc, Pre, Re and F1 at a magnification of 10× are 85.92%, 85.93%, 86.02%, and 85.96%, respectively.

[0105] Table 4 Comparison of prediction performance with existing methods (%)

[0106]

[0107]

[0108]

[0109] Table 5 Comparison of calculation floating point number and throughput with existing methods (%)

[0110]

[0111] Although the above methods can achieve relatively ideal classification results, they usually need to screen the scale that is most suitable for downstream tasks in various magnifications, which is undoubtedly a time-consuming and tedious task. On the contrary, cleverly using the pyramid structure of WSI and combining WSI information at multiple magnifications can not only significantly improve the efficiency and accuracy of classification, but also effectively avoid the tediousness of screening single-scale information. DSMIL combines MIL and self-supervised learning for tumor detection. It first obtains the features of WSI at two magnifications through self-supervised learning, and then sends them to the MIL network for feature splicing of different scales. Finally, the classification results of the combination of two scale information are obtained. The Acc, Pre, Re, and F1 at magnifications of 10× and 20× are 87.50%, 87.37%, 87.50%, and 87.21%, respectively, which is about 2% higher than the optimal result of CLAM. However, the DSMIL algorithm is limited to using two different scales of WSI for reasoning, and fails to fully simulate the comprehensive judgment process of pathologists when observing tissue pathology images. In addition, self-supervised learning often requires powerful hardware environment support, with a FLOPs of 117.61 and a throughput of 9.21 seconds, which is difficult to achieve clinical deployment in most hospitals. In contrast, the method proposed in this paper cleverly integrates WSI features under various magnifications in an orderly manner through parallel attention modules and differentiable Top-K selection modules, thereby greatly reducing the demand for computing resources while ensuring performance. Its Acc, Pre, Re, and F1 under magnifications of 5×, 10×, and 20× reached 89.12%, 89.91%, 88.32%, and 88.69%, respectively, while FLOPs was only 14.32 and the throughput was 4.27 seconds, which is in line with the actual clinical application needs.

[0112] The confusion matrices of different models under 5×, 10× and 20× data are as follows Figure 3 As shown, the confusion matrix of the present invention under different scale data is as follows Figure 4As shown in the figure. Among them, the horizontal axis of the confusion matrix is ​​the predicted label of the model, and the vertical axis is the true label of the sample, which is adenocarcinoma (GAC), mucinous adenocarcinoma (MUC) and signet ring cell carcinoma (SRC). The value of the main diagonal represents the number of correct classifications of the model in each category. The closer the value is to 1, the better the prediction ability of the model. It can be observed from the figure that most models can accurately classify GAC because it has a more obvious cancerous glandular structure; however, for SRC, a cancer type with a small cell morphology and a "signet ring" shape, the models ABMIL and CLAM with attention pooling mechanism show better recognition ability. However, when faced with MUC with a large amount of mucin pool outside the cell nucleus, their recognition ability is relatively insufficient. In contrast, the multi-scale model DSMIL performs well in the recognition of SRC and MUC, which is mainly due to the fact that the multi-scale model can make up for the possible information loss at a single scale and capture the macroscopic tissue structure and microscopic cell characteristics at the same time. The method proposed in this article combines the advantages of attention pooling and multi-scale. It can not only focus on key features, but also comprehensively utilize rich information under different magnifications, thereby significantly improving the accuracy of gastric cancer subtype classification, which is helpful in assisting doctors to make more accurate diagnosis and treatment decisions in clinical practice.

[0113] The present invention further illustrates that the proposed method can effectively capture the characteristic information of medical images through the attention map. Specifically, the present invention first maps the trained Tile attention weight scores to the original pathological image, and visualizes the attention scores using color coding technology. Red represents areas with higher attention scores, that is, areas with greater impact on model prediction. Then, a thumbnail is generated from the superimposed heat map to intuitively display the attention distribution of the model. In addition, in order to better evaluate the attention heat map, the present invention marks the areas with denser and darker red with black frames on the obtained pathological image heat map, and captures images of this area at magnifications of 5×, 10×, and 20× on the original image, as shown in FIG. Figure 5As shown in the figure. For gastric adenocarcinoma (GAC), the red highlights in the attention heat map are obviously concentrated in the areas with dense cancerous glandular structures. In pathological diagnosis, the abnormal proliferation and arrangement of glandular structures are important bases for judging GAC. Through the high-resolution heat map, it can be observed that the model pays high attention to the densely distributed areas of these cancerous glandular structures, which helps pathologists to more accurately identify cancerous areas during the diagnosis process, thereby significantly improving the accuracy and reliability of diagnosis. For gastric signet ring cell carcinoma (SRC), the red areas in the heat map are concentrated in the location of signet ring cells, indicating that the model pays great attention to the typical characteristics of SRC. This information is particularly important in pathological diagnosis because SRC is usually highly invasive and has a poor prognosis. Through early and accurate identification and diagnosis, more accurate treatment plans can be provided to patients, thereby affecting the prognosis of patients. Finally, for gastric mucinous adenocarcinoma (MUC), the high-scoring areas in the attention heat map are mainly concentrated in the areas with a large number of mucin pools outside the cell nucleus. These red areas mark the model's high attention to the characteristics of mucinous adenocarcinoma, which coincides with the key feature points in pathological diagnosis. This refined allocation of attention helps doctors diagnose MUC more accurately and provide patients with timely and effective treatment.

[0114] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A gastric cancer subtype classification method based on multi-instance learning and multi-scale feature extraction, characterized in that: include: S1, collect pathological images of gastric cancer patients and corresponding clinical label data; S2, preprocessing each original tissue pathology image WSI to obtain sub-images Tile of the tissue pathology image at different magnifications; S3, for the sub-image Tile obtained in S2, the Tile features at different magnifications are extracted through the ResNet backbone network pre-trained on the ImageNet dataset. Finally, the extracted features are packaged into an .h5 file and divided into training set, validation set and test set in a ratio of 8:1:1; S4, sending the pathological image features at low magnification obtained in S3 to a multi-instance learning parallel attention module to obtain a channel attention score and a spatial attention score at the magnification; The construction process of the multi-instance learning parallel attention module in S4 is as follows: S4.1 Multi-instance learning is the basis of subsequent tasks. The input X is a package consisting of multiple instances, denoted by X = {x1,...,x N }; Given a classification task with C class labels, each instance has an unknown label y i ∈C, X has a known label y∈C; the original tissue pathology image WSI is taken as input, and the instance represents the extracted Tile set; among them, the slice-level Tile-level feature extractor h extracts each Tile x i Mapping to feature vector Indicates that each element in the tensor is a D-dimensional real number. Then, the pooling operator g(·) transforms the feature vector h i＝1:N , N is the number of feature vectors, which are aggregated to generate a feature representation of a single WSI; finally, the classifier f(·) uses the aggregated feature representation to predict the WSI-level label The process can be summarized as follows: Among them, Avgpool is the average pooling operation; Maxpool is the maximum pooling operation; f 7×7 is a 7×7 convolution operation; σ is a Sigmoid activation function; MLP is a fully connected layer with shared weights; finally, the obtained Tile-level attention score s i 、c i The eigenvector h corresponding to the eigenmatrix i Multiply and accumulate them to obtain the WSI-level spatial attention feature map or spatial attention score sa and channel attention feature map or channel attention score ca; S5, the channel attention score obtained in S4 is used through a differentiable module to obtain the information-rich tile at a magnification of m times, and a multi-scale concat operation is performed with the features at a magnification of m+1 times obtained in S3; S6, for the combined result of S5, perform the same operation as S4 and S5 to obtain the spatial attention score and information-rich tile at this magnification; S7, the above process S3-S6 is repeated until the maximum magnification is m+n times, and finally all features under each magnification are aggregated for output prediction; S8. Input the training set data of S3 into the multi-instance learning parallel attention module, and combine it with the multi-scale concat splicing in S5 for training. Then use the validation set of S3 to verify the performance of the model obtained from each batch of training, and save the model parameters that best achieve the best indicators on the validation set as the weights of the final classification model; finally, verify the performance of the saved model in classifying gastric cancer subtype pathological images on the test set of S3.

2. The gastric cancer subtype classification method based on multi-instance learning and multi-scale feature extraction according to claim 1 is characterized in that: The processing of the S2 original full-field digital slice image WSI includes the following steps: Eliminate unmatched data, use a sliding window to crop the tissue pathology images at different magnifications into sub-blocks of preset sizes, and filter out sub-blocks with blank backgrounds exceeding a first preset threshold; color-standardize all sub-blocks. After the above operations are completed, save the cropped sub-blocks separately according to the corresponding tissue pathology images.

3. The gastric cancer subtype classification method based on multi-instance learning and multi-scale feature extraction according to claim 1, characterized in that: In S3, the Tile features at different magnifications are extracted by using the ResNet backbone network pre-trained on the ImageNet dataset, which specifically includes the following steps: Multi-scale tile generation: Extract image tiles at different magnifications from the original gastric cancer pathology slice images to obtain multi-scale information; ResNet pre-training: Use ResNet pre-trained on the ImageNet dataset as the backbone network for feature extraction. The pre-trained ResNet has learned a large number of common image features, which helps to improve the effectiveness of feature extraction; Feature extraction: Tiles at different magnifications are input into the pre-trained ResNet network to extract their corresponding feature vectors; these feature vectors can capture the cell, texture, and structural information of the tile at different magnifications; finally, the features extracted at all magnifications of an entire pathological image are saved as an .h5 file for subsequent use.

4. The gastric cancer subtype classification method based on multi-instance learning and multi-scale feature extraction according to claim 1, characterized in that: The channel attention scores obtained by S4 are used through a differentiable module to obtain information-rich tiles with a magnification factor of m. The specific process is described as follows: When a WSI is accessible at multiple magnifications, these magnifications are represented by m∈{1,2,...,M}, where M represents the highest magnification. Assuming that the magnification of the WSI at m+1 is twice that at m, in order to identify the information tile at magnification m, the WSI-level channel attention feature map must be calculated first. It contains the channel attention score of the model for each tile. Then, the top K tiles with the highest attention scores are selected, namely the Top-K operation, and further processed at a higher magnification. The feature matrix of the selected tile can be expressed as: Among them, T m ∈{0,1} N×K is the indicator matrix, H m ∈R N×D is the characteristic matrix of Tile when the magnification is equal to m; The tile selection at m is directly performed through the predicted output of the classifier f(·), which is a classification head composed of a fully connected layer. This can be achieved by backpropagating from the output of the classifier f(·) to the attention module at m without introducing any additional losses or related hyperparameters. However, formula (4) cannot be directly differentiated because it involves Top-K operations, which are not mathematically differentiable everywhere. Therefore, the perturbation maximum method is used to make the Top-K selection differentiable and apply it to the attention score ca at the magnification m. m Specifically, we first add uniform Gaussian noise Then a linear program is solved for each of these perturbed attention weights and the results are averaged. Therefore, the forward pass of the differentiable Top-K module can be written as: where <·> is the scalar product before matrix vectorization; therefore Represents the perturbed attention score (ca m +σZ) is repeated K times in the second dimension to form a matrix of size N×K for subsequent operations; the corresponding Jacobian matrix is ​​defined as: in, represents the optimization goal, represents the decision variable, σ represents the noise standard deviation, Z T is the transposed matrix of Gaussian noise Z; The differentiable Top-K operation can learn the weighted attention score of the tile at a specific magnification; this means that the model can automatically learn how to select meaningful tiles without relying on predefined rules; in addition, the present invention also keeps the size of the tile selected at all magnifications unchanged, that is, it will not change due to the magnification; this can keep the number of tiles proportional to the magnification, so that the model can capture various contextual information, which is very important for analyzing tissue pathology sections because they usually contain many cell nuclei, cytoplasm, tissue stroma and diagnosis-related components; in order to combine tiles at other scales, the indicator matrix T m Expand to get from Tile feature (where m'>m) to select meaningful information; by calculating T m and the identity matrix The Kronecker product between them gives the expanded indicator matrix; similar to formula (3), the attention weights from the lower magnification m are used to select tiles at the higher magnification m': Among them, H m' is the characteristic matrix under magnification m', is the selected feature matrix; Through the above calculations, an indicator matrix containing more information can be generated to enable more detailed tile selection at higher magnifications. It can also correspond to the attention weights at lower magnifications, which helps improve the model's ability to process input data.

5. The gastric cancer subtype classification method based on multi-instance learning and multi-scale feature extraction according to claim 1, characterized in that: The specific description of aggregating all features at various magnifications for output prediction in S7 is as follows: When the magnification m = 1, the feature matrix H1 passes through the parallel attention module PAM; PAM consists of the spatial attention module PA1 and the channel attention module PA1'. PA1 is used to obtain the optimal pooled WSI-level representation sa1 at low magnification; PA1' aims to provide a meaningful channel attention score ca1 for tile selection at higher magnification; then, the differentiable indicator matrix T1 is used to select the most informative tile, and the tile feature matrix at higher magnification is obtained through formula (7): The above process is repeated until the highest magnification M is reached; then, the Tile feature selected at M is Through the last parallel attention module PAM, the feature representation sa at the highest magnification is generated M In addition, since the features at different scales in pathological images are closely related and the summation exploits the complementarity between features, the attention pooling features of all magnifications are represented as sa1, sa2, …, sa M Aggregation is performed through sum pooling to obtain a multi-scale, context-aware WSI-level representation; the final classifier f(·) maps the WSI-level representation to the corresponding label y∈C and outputs the model prediction This process can be seen as an extension of formula (1), where summation and pooling are performed at multiple magnifications:

6. A gastric cancer subtype classification system based on multi-instance learning and multi-scale feature extraction, characterized by: The method comprises the module according to any one of claims 1 to 5.

Images