A tumor subtype diagnosis method for whole slide pathological images

Abstract

This invention belongs to the field of tumor subtype diagnosis technology, specifically relating to a method for diagnosing tumor subtypes in whole-section pathological images. First, the unclassified whole-slice indices (WSIs) are preprocessed. Then, a trained tumor subtype diagnosis model is used to diagnose the subtype of the preprocessed WSIs. The diagnostic process includes: extracting the embedded feature representations of the WSIs, measuring the similarity between the WSIs and their respective clusters, and obtaining the subtype diagnosis result based on the similarity. Furthermore, during training, the similarity between the embedded feature representations of WSIs within the same category is increased, while the similarity between the embedded feature representations of WSIs from different categories is decreased. This makes the features of WSIs within the same category more similar and the features of WSIs from different categories more distinct, facilitating more accurate predictions. Moreover, the model can be trained using only image-level label information, significantly reducing the enormous workload of pathologists annotating images.

Description

Technical Field

[0001] This invention belongs to the field of tumor subtype diagnosis technology, specifically relating to a tumor subtype diagnosis method based on whole-section pathological images. Background Technology

[0002] Cancer is one of the most common diseases worldwide, seriously endangering people's health. Pathological analysis of suspicious tissues is the gold standard for clinical diagnosis of many serious diseases, especially malignant tumors (cancer), and is also an important basis for early detection, diagnosis, treatment, and prognosis. Traditional histopathological analysis involves pathologists observing hematoxylin and eosin-stained tissue sections under high-magnification microscopes, analyzing the structure and morphology of tissue cells at different magnifications to determine tumor areas and the degree of cell proliferation, thereby making a pathological diagnosis. However, there is currently a severe shortage of pathologists in China. In some specialized cancer hospitals, pathologists have to observe and analyze thousands of pathological slides every day, and this heavy and time-consuming work may lead to diagnostic errors. In recent years, with the rapid development of deep learning technology, high-magnification scanners can save tissue on slides as high-definition digital pathological images (WSI), making it possible to use computer artificial intelligence to assist in clinical pathological diagnosis.

[0003] With the rapid development of computer hardware and the improvement of computational theory, especially in deep learning algorithms, machine learning methods have been applied to assist medical image diagnosis. However, research and development in the field of computational pathology remains relatively lagging. Early methods of preprocessing pathological images through manual annotation required a significant investment of manpower and time from pathologists, and analyzing tissue sections was time-consuming and labor-intensive for doctors. The main reasons for this are: doctors' judgments of the unique cytological and histological characteristics of pathological sections are influenced by subjectivity and personal experience; different doctors may have different interpretations of pathological sections, with an average diagnostic consistency of only about 75%; doctors spend considerable time and effort studying and annotating tissue pathological images using instruments such as microscopes; and pathologists' accumulated clinical experience is difficult to transfer to others, while doctors' diagnosis and treatment of patients are extremely important to their health. In the initial stages of computer-aided diagnosis, medical experts typically needed to segment images based on cell nuclei, chromosomes, or cells to extract targeted pathological features and train machine learning models based on these features. While computer-aided algorithms at the manual feature stage have addressed some of the challenges in medical image analysis, their pathological diagnostic methods typically rely on pixel-level annotation by professional physicians. This annotation process is extremely costly and time-consuming for doctors. Furthermore, the quality of the annotation depends on the physician's subjectivity and experience, especially for complex cases where different physicians may annotate the same lesion area differently. Moreover, with the development of deep learning, manual features are gradually being replaced by features learned automatically using deep learning. Although some algorithms based on traditional convolutional neural networks (CNNs) are widely used in medical image diagnosis, their application in pathological image diagnosis is still relatively limited. This is due to the limitations of computer hardware memory (especially graphics card memory such as GPUs), which prevent CNN-like models from directly processing large WSI images (e.g., 10^6 pixels). 5 ×10 5 Due to limitations in the size of the WSI dataset (pixels), traditional CNN-based models cannot be effectively trained. Therefore, AI-assisted pathological diagnosis places higher demands on the acquisition and annotation of pathological image data, the effective representation of image information, and the design and implementation of accurate diagnostic methods.

[0004] Pathological images differ from ordinary images. Their extreme image size makes them unsuitable for direct processing by traditional deep learning models. Furthermore, publicly available pathological datasets contain relatively few images, and image annotation requires significant time and effort from pathologists, making pixel-level annotations difficult to obtain. Faced with the problem of only having whole-slice image-level labels and lacking pixel-level annotations, existing mature deep learning networks struggle to achieve effective diagnostic results. Therefore, automated diagnosis of whole-slice pathological images poses a significant challenge to computer vision tasks. In summary, current deep learning techniques for histopathological image analysis still lack a high-precision and feasible solution. Summary of the Invention

[0005] The purpose of this invention is to provide a tumor subtype diagnosis method for whole-section pathological images, in order to solve the problem that there is a lack of high-precision and feasible application of deep learning technology to histopathological image analysis in the prior art.

[0006] To address the aforementioned technical problems, this invention provides a method for diagnosing tumor subtypes based on whole-section pathological images, comprising the following steps:

[0007] 1) Construct a tumor subtype diagnostic model, which includes overall feature extraction, and the overall feature extraction is used to extract the embedded feature representation of the input WSI;

[0008] 2) The tumor subtype diagnostic model is trained using the tumor WSI dataset that has been classified into subtypes as training samples. During the training process, the similarity between the embedding feature representations of WSIs of the same category is increased, while the similarity between the embedding feature representations of WSIs of different categories is decreased. After training, multiple clusters are obtained, and the number of clusters is the number of subtype classifications. Each cluster contains the embedding feature representations of all training samples belonging to that cluster.

[0009] 3) Preprocess the WSIs to be classified, and use the trained tumor subtype diagnostic model to perform subtype diagnosis on the preprocessed WSIs to be classified. The diagnostic process includes: using the feature extraction module to obtain the embedded feature representation of the preprocessed WSIs to be classified; measuring the similarity between the WSIs to be classified and each cluster based on the embedded feature representation of the WSIs to be classified and the embedded feature representation of each training sample in each cluster; and obtaining the subtype diagnosis result of the WSIs to be classified based on the similarity.

[0010] The beneficial effects of the above technical solution are as follows: The tumor subtype diagnostic model designed in this invention can continuously improve the similarity between WSI features of the same category and reduce the similarity between WSI features of different categories during the training process. This makes WSI features of the same category more similar and WSI features of different categories more different, which is more conducive to subsequent partial measurement of the similarity between the WSI to be classified and each cluster, so as to achieve more accurate prediction of the WSI to be classified. Moreover, the model can be trained using only WSI image-level label information and achieves better performance than models that require a large number of pixel-level labeled images for training. Without relying on detailed pixel-level horizontal annotation, it significantly reduces the huge workload of pathology experts in annotating images.

[0011] Furthermore, the similarity between the embedded feature representation of the WSI to be classified and each cluster is measured using any of the following methods:

[0012] Method 1: Measure the similarity between the embedded feature representation of the WSI to be classified and the embedded feature representation of all training samples in each cluster, and take the maximum similarity of each cluster as the similarity between the WSI to be classified and that cluster;

[0013] Method 2: Calculate the average embedding feature representation of all training samples in each cluster and use it as the cluster center of that cluster. Measure the similarity between the embedding feature representation of the WSI to be classified and the cluster center of each cluster, and use this as the similarity between the WSI to be classified and each cluster.

[0014] Method 3: Calculate the average of the results obtained from Method 1 and Method 2 as the similarity between the WSI to be classified and each cluster.

[0015] The beneficial effects of the above technical solution are: different measurement strategies are designed for flexible selection.

[0016] Furthermore, the loss function used in training the tumor subtype diagnostic model is:

[0017]

[0018] In the formula, Loss represents the loss value; B represents the batch size for one training iteration; K i and L i Indicates package X i The similarity scores between the number of similar samples and the number of dissimilar samples are respectively... and γ represents the scaling factor; and Indicates the weighting factor; Δ pos and Δ negThese represent intra-class spacing and inter-class spacing, respectively.

[0019] The beneficial effects of the above technical solution are as follows: The loss function optimizes the distribution of package features in the feature space by forcing the vector similarity between package features within a class to be less than the vector similarity between package features between classes, so that the package feature vectors of each class gather around the center of their respective class, thereby evolving into different class clusters, and the class clusters maintain a certain distance from each other.

[0020] Furthermore, the preprocessing includes segmenting the tissue region in the WSI into multiple image patches.

[0021] The beneficial effects of the above technical solution are as follows: the tissue region is divided into multiple image blocks, and each image block is processed in subsequent processing. This enables the model to be trained using only WSI image-level label information without relying on detailed pixel-level horizontal annotations. The model achieves better performance than models that require a large number of pixel-level annotated images for training. This avoids the dependence of deep learning methods on pixel-level annotations of whole-slice pathological images, thereby significantly reducing the huge workload of pathology experts in annotating images.

[0022] Furthermore, the overall feature extraction includes a feature extraction module, a projection module, and a gated attention module; the feature extraction module is used to extract features from each image patch; the projection module is used to project the features extracted by the feature extraction module into a low-dimensional unit feature space to obtain multiple unit vectors, so that different feature vectors only differ in direction in the unit feature space; the gated attention module is used to fuse the output of the projection module to generate an embedded feature representation.

[0023] Furthermore, the feature extraction module uses ResNet101 as the backbone network, which is used to pass the features output from Stage 3 and Stage 4 of ResNet101 through an adaptive average pooling layer, and then concatenate the outputs of the adaptive average pooling layer to extract the features corresponding to each image block.

[0024] The beneficial effects of the above technical solution are: the extracted features are multi-scale features, and the complete expression of example features is achieved by fusing multi-scale features.

[0025] Furthermore, the projection module includes a trainable BatchNorm1d layer, a fine-tuning layer Proj-Fc parameterized by weights, and an L2 Norm layer; the BatchNorm1d layer is used to normalize the input feature matrix along the feature dimension, and the calculation process of the fine-tuning layer Proj-Fc is as follows: Represents the weight matrix; represents the output of the fine-tuning layer Proj-Fc; ReLU represents the activation function; the L2 Norm layer is used to normalize each feature vector in the output of the fine-tuning layer Proj-Fc to a unit vector.

[0026] Furthermore, the gated attention module includes an Attn-Fc1 layer, an Attn-Fc2 layer, an Attn-Fc3 layer, and a Dropout layer. The Attn-Fc1 layer outputs a compressed feature for the input features and maps the feature values ​​in the compressed feature to the range of positive infinity to negative infinity using the Tanh activation function. The Attn-Fc2 layer acts as a gate and maps the network's output value to the range of 0 to 1 using the Sigmoid activation function to control the output of the Attn-Fc1 layer. The outputs of the Attn-Fc1 and Attn-Fc2 layers are multiplied bitwise and then fed into the Attn-Fc3 layer. The Attn-Fc3 layer generates an attention score for each feature and processes it through the Dropout layer. Then, the attention scores processed by the Dropout layer are used to perform a weighted summation of the unit vectors to obtain the embedded feature representation.

[0027] The beneficial effects of the above technical solution are as follows: The purpose of applying the gated attention mechanism (GAM) is to assign a learnable weight information to each example vector and to generate the embedded feature representation of the bag by fusing all the example feature vectors in a bag.

[0028] Furthermore, the preprocessing includes: first converting WSI from RGB color space to HSV color space; then, using Otsu's method to calculate the segmentation thresholds for the background and tissue regions for the saturation channels in the HSV color space; and based on the segmentation thresholds, binarizing the saturation channels to extract the tissue mask to obtain the tissue regions; and then using a sliding window approach to segment the tissue regions into a series of image blocks of the same size.

[0029] The beneficial effects of the above technical solution are: the Otsu method can effectively remove the background area in WSI.

[0030] Furthermore, after extracting the tissue mask, mean filtering and morphological closing operations are required to extract the tissue region.

[0031] The beneficial effects of the above technical solution are as follows: by using mean filtering and morphological closing operation, noise reduction can be further refined, and a large number of tiny holes and tiny tissue regions existing in the tissue mask can be removed. Attached Figure Description

[0032] Figure 1 This is an overall workflow diagram of the present invention;

[0033] Figure 2(a) is the original WSI diagram;

[0034] Figure 2(b) is a WSI tissue outline diagram;

[0035] Figure 2(c) is a segmented image patch diagram;

[0036] Figure 3 This is a flowchart of the overall feature extraction process for WSI in this invention;

[0037] Figure 4(a) is a schematic diagram of the MaxS measurement strategy adopted in this invention;

[0038] Figure 4(b) is a schematic diagram of the AvgS metric strategy used in this invention;

[0039] Figure 5(a) is a schematic diagram of the impact of the L2 normalization layer on TSMIL on the test set;

[0040] Figure 5(b) is a schematic diagram of the impact of the L2 normalization layer on TSMIL on the training set;

[0041] Figure 6(a) is the first Annotated WSI diagram;

[0042] Figure 6(b) is the attention heatmap of Figure 6(a);

[0043] Figure 6(c) is the second Annotated WSI diagram;

[0044] Figure 6(d) is the attention heatmap of Figure 6(c);

[0045] Figure 6(e) is a thermal index diagram of Figures 6(b) and 6(e). Detailed Implementation

[0046] This invention designs and trains a framework that integrates weakly supervised multi-instance learning and metric-based learning to realize a method for diagnosing tumor subtypes based on whole-slice pathological images. To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0047] An example of a tumor subtype diagnosis method based on whole-section pathological images:

[0048] The specific implementation process of the tumor subtype diagnosis method based on whole-section pathological images of the present invention (hereinafter referred to as TSMIL) is as follows: Figure 1 As shown, the details are as follows:

[0049] Step 1: Obtain the WSI to be classified and preprocess it.

[0050] Since there are a large number of background regions in WSI, such as the white area shown in Figure 2(a), these background regions are meaningless for WSI classification. In order to remove these meaningless background regions and ensure the effectiveness of subsequent classification and diagnosis, WSI is first preprocessed. The preprocessing mainly includes two steps: (1) effectively removing the background regions in WSI; (2) dividing the extracted tissue regions into image blocks of appropriate size for subsequent model analysis. The process of extracting tissue regions in WSI and dividing them into multiple image blocks is as follows:

[0051] First, when extracting tissue regions, each WSI image is converted from the RGB color space to the HSV color space. Then, for the saturation channel in the HSV color space, the Otsu method is used to calculate the segmentation threshold of the background and tissue regions. Based on the segmentation threshold, the saturation channel is binarized to extract the tissue mask. Since there are many tiny holes and tiny tissue regions in the tissue mask, mean filtering is then used for smoothing to handle this noise. Finally, morphological closing operations are used for further refined noise reduction. The extracted tissue regions are shown in Figure 2(b), where the area enclosed by the green curve is the final extracted tissue region.

[0052] After extracting the tissue region, this invention uses a sliding window approach to segment the tissue region into a series of image patches of the same size, as shown in Figure 2(c). For image patch processing, this invention only retains image patches whose tissue area is greater than 80% of the total image patch area. Due to differences in WSI resolution and tissue region size, the number of image patches segmented per WSI varies from hundreds to thousands. Subsequent model training in this invention is performed on these image patches.

[0053] Step two involves constructing a tumor subtype diagnostic model. This model includes global feature extraction, which extracts the embedded feature representation of the input WSI. The constructed tumor subtype diagnostic model is then trained using a pre-classified tumor WSI dataset as training samples. It should be noted that different tumors have different subtype classifications; for example, renal cell carcinoma subtypes include chromophobe renal cell carcinoma, clear cell renal cell carcinoma, and papillary renal cell carcinoma.

[0054] 1. Overall feature extraction.

[0055] The classification and diagnosis of pathological images is based on the overall features of WSI, and the acquisition process of these features includes three modules, such as... Figure 3As shown, these are the Multi-scale Feature Extraction Module (MFEM), Projection Module (PM), and Gated-Attention Module (GAM). After the tissue region in WSI is segmented into multiple image patches, the MFEM is first used to extract the multi-scale features of each image patch in WSI. Then, the PM module projects the features of each image patch into a unit vector in a low-dimensional feature space. Finally, the GAM module fuses the feature vectors of all image patches in a WSI into the global overall features of that WSI.

[0056] 1) Multi-scale feature extraction module (MFEM).

[0057] In WSI weakly supervised classification, a dataset containing N WSI images is represented as D={(X1,Y1),…,(X N ,Y N )}, where each WSI X i There is a unique category label Y i After preprocessing the pathological images, each WSI X i Contains n i Image blocks, i.e. In the corresponding multiple instance learning algorithm, each WSI is regarded as a bag, and the image patch in the WSI is regarded as an instance in the bag.

[0058] When pathologists perform pathological diagnoses, they typically conduct multi-scale observations on glass slides to achieve a comprehensive examination and accurate judgment of suspicious lesions, avoiding missed or misdiagnosed cancerous lesions. Based on this idea, this invention designs a multi-scale feature extraction module (MFEM) to extract features at multiple scales for each example and achieve a complete representation of the example features by fusing multi-scale features. This invention uses ResNet101 as the backbone network and transfers pre-trained parameters from ImageNet. Subsequently, multi-scale features for each example are extracted based on this network, such as... Figure 3 As shown.

[0059] Specifically, the features of each image patch are fused by the features output from Stage 3 of ResNet101. and the characteristics of Stage4 output To obtain multi-scale features, as shown in formula (1). In addition, this invention designs to apply an adaptive average pooling layer after Stage 3 and Stage 4 to ensure that the outputs of the backbone network at different stages are feature values.

[0060]

[0061] Among them, f S3 (·) and f S4 (·) represent different scale features extracted from STAGE3 and STAGE4 in ResNet101, respectively; P(·) represents the average pooling operation; and The dimensions are d s3 peacekeeping s4 dimension;

[0062] A multi-scale joint strategy was then designed to fuse each example x. i,j The specific process of the multi-scale joint strategy for features at different depths is shown in Equation (2).

[0063]

[0064] Here, `concat` represents the concatenation of features at different scales. Each example x... i,j From multi-scale features It means that d r Indicates the feature dimension after fusion; package X i From the characteristic matrix It means that among them

[0065] 2) Projection module.

[0066] Unlike existing classic Multiple Instance Learning (MIL) methods, this invention proposes a projection module (PM) that aims to project instance features into a low-dimensional unit feature space, ensuring that different instance feature vectors differ only in direction within this unit feature space. This invention applies two different regularization operations in the PM to stabilize the training of the encoding layer. Specifically, as follows... Figure 3 As shown, PM consists of three main network layers: a trainable BatchNorm1d layer, a weighted layer, and a depth layer. The parameterized fine-tuning layers Proj-Fc and L2Norm layers are used. The BatchNorm1d layer performs batch normalization, the main function of which is to adjust the feature matrix R. i The feature matrix is ​​placed along the feature dimension. Standardized to feature matrix The specific process is shown in formula (3).

[0067] R′ i =BachNorm1d(R i (3)

[0068] Where BachNorm1d represents batch normalization, and the output of the BachNorm1d layer is: in

[0069] Since the parameters of the backbone network are fixed and transferred during training, the multi-scale features extracted based on the backbone network cannot be optimized during training. To optimize the example features during training, this invention applies a fine-tuning layer, Proj-Fc, in the projection module. Proj-Fc consists of two components: a weight matrix and a weight matrix. and the ReLU activation function, where the weight matrix W p Its main function is to fine-tune example features during training. The ReLU activation function performs a non-linear transformation of the features in the feature matrix. After processing by the fine-tuning layer Proj-Fc, the output feature matrix is Where d p The feature dimension is represented by the fine-tuned layer Proj-Fc. The specific calculation process of the fine-tuning layer Proj-Fc is shown in formula (4).

[0070]

[0071] in, Represents the weight matrix; ReLU represents the activation function; the output of the fine-tuning layer Proj-Fc is

[0072] Furthermore, the normalization shown in formula (5) plays a crucial role in TSMIL, determining whether TSMIL's metric-based feature learning can be effectively trained. Specifically, the operation performed by the L2 standardization layer is L2 normalization, whose main function is to normalize the example feature vectors and eliminate the influence of feature magnitude on the model. In metric learning, the features of each example are regarded as feature vectors in the feature space, and each vector contains two basic attributes: direction and magnitude. Since direction is the only vector attribute of concern in TSMIL of this invention in metric-based feature learning, while the magnitude of the example vector may be a confounding factor. To solve this problem, this invention addresses the issue in the feature matrix H i In the process, L2 normalization is applied to each example feature vector h. i,j Standardized to a unit vector h′ i,j This effectively eliminates the magnitude information of the example feature vectors, avoiding the influence of magnitude information on the bag feature representation. After L2 normalization, each example x i,j Given a unit vector h′ i,j This means that the only difference between different examples is the direction of the vectors, and the subsequent Gated Attention (GAM) module aggregates example vectors from different directions to obtain the bag's feature vector. It's worth noting that GAM assigns a weight to each example's feature vector, and the bag's feature vector will be closer to the feature vector of the example with the larger weight.

[0073] H′ i =L2Norm(H i (5)

[0074] Where L2 Norm represents L2 normalization, and the output of the L2 Norm layer is:

[0075] 3) Gated attention module.

[0076] The purpose of this invention, which applies the Gated Attention (GAM) mechanism, is to assign a learnable weight to each example vector and generate an embedded feature representation of the bag by fusing all example feature vectors in a bag. For example... Figure 3 As shown, GAM is a multi-layer attention structure consisting of four parts: the Attn-Fc1 layer, the Attn-Fc2 layer, the Attn-Fc3 layer, and the Dropout layer. The Attn-Fc1, Attn-Fc2, and Attn-Fc3 layers are each composed of a weight matrix. and Parameterize the parameters and continuously adjust them during training until they are fixed after training is completed.

[0077] In GAM, the Attn-Fc1 layer consists of a weight matrix. It consists of the Tanh activation function, where d p and d a These represent the input and output dimensions of the Attn-Fc1 layer, respectively. Weight matrix V a The algorithm learns information from each example feature and outputs a compressed feature for each example feature. Finally, the Tanh activation function maps the feature values ​​in the compressed feature to the interval from positive infinity to negative infinity. In the multi-layer attention structure, the network layer that plays a gating role is the Attn-Fc2 layer, which consists of a weight matrix. It consists of the Attn-Fc1 layer and the Sigmoid activation function. The Sigmoid activation function maps the network's output values ​​to between 0 and 1, releasing values ​​close to 1 and suppressing values ​​close to 0, thereby regulating the output of the Attn-Fc1 layer. The outputs of the Attn-Fc1 and Attn-Fc2 layers are then passed to the Attn-Fc3 layer after a bitwise multiplication operation.

[0078] Finally, the Attn-Fc3 layer is used to generate an attention score for each example feature. Example features that play a crucial role in classification are assigned larger attention scores, while those that are less important are assigned smaller attention scores. The Attn-Fc3 layer consists of a weight matrix... It consists of the Softmax function, where the weight matrix W aThis is used to output an attention score for each example. The Softmax function normalizes the attention scores of all example features in a bag, making the sum of the attention scores of all example features in that bag equal to 1. For example x i,j The specific calculation process of the attention score is shown in formula (6).

[0079]

[0080] Where ⊙ represents the dot product of vectors; sigm represents the Sigmoid activation function; tanh represents the Tanh activation function; a i,j For example x i,j The corresponding attention score.

[0081] In addition, this invention also designs an application of dropout technology to all attention scores in the bag to enhance the model's generalization ability. This technique involves randomly selecting a bag from which d... p % attention score, set the attention value of the selected example to 0 to achieve random occlusion in WSI d p The percentage of images is used to enhance the robustness of the model during training. The dropout operation on the attention score is shown in Equation (7).

[0082]

[0083] The final feature vector of a packet Obtained by the fusion operator g, which computes the package X i All example vectors h′ i,j And attention score a ′ i,j The weighted sums are then fused to obtain the feature representation of the package. The package fusion process is shown in formula (8).

[0084]

[0085] 2. Loss function.

[0086] Optimizing the features of the bags is a key consideration in TSMIL's choice of loss function. Metric learning is a powerful method to obtain the compactness of features of samples of the same class and the difference in features of samples of different classes during training. This invention designs and uses the loss function shown in formula (9) (instead of the cross-entropy loss function used by most deep neural networks). By forcing the vector similarity between bag features within a class to be less than the vector similarity between bag features between classes, the distribution of bag features in the feature space is optimized, so that the bag feature vectors of each class are clustered around the center of their respective class, thus evolving into different class clusters, and the class clusters maintain a certain distance from each other. During training, the batch size for one iteration of training is denoted as B, and the bags participating in training are denoted as {X1,…,X}. B Assume that within a training batch, for package X... i There exists K i One sample of the same type (package) and L i Let the similarity scores between *n* out-of-class samples in the feature space be denoted as [equation missing]. k∈{1,…,K i}and l∈{1,…,L i The loss function for a training batch is shown in Equation (9).

[0087]

[0088] Two from package X i and X b eigenvectors G i and G b , (i∈{1,…B},b∈{1,…B},i≠b), let the similarity score of these two feature vectors be s, and its similarity is defined as shown in formula (10).

[0089]

[0090] Where, <·,·> is the inner product of two vectors; ‖·‖ denotes the Euclidean regularization of the vector; and It is a weighting factor, Δ pos and Δ neg These represent intra-class spacing and inter-class spacing, respectively, and their specific parameter settings are shown in formula (11).

[0091]

[0092] in,[·] + This indicates a "zero-point cutoff" operation; the scaling factor γ and the interval factor m are two hyperparameters.

[0093] After extracting the overall features of WSI, the classification training process for pathological images can continuously improve the similarity between WSI features of the same category, while reducing the similarity between WSI features of different categories, so that WSI features of the same category are more similar and WSI features of different categories are more different. Figure 3 This demonstrates the overall feature optimization and acquisition process for WSI.

[0094] Step 3: Use the trained tumor subtype diagnostic model to perform subtype diagnosis on the preprocessed WSI to be classified. The diagnostic process includes: using the feature extraction module to obtain the embedded feature representation of the preprocessed WSI to be classified; measuring the similarity between the WSI to be classified and each cluster based on the embedded feature representation of the WSI to be classified and the embedded feature representation of each training sample in each cluster; and obtaining the subtype diagnosis result of the WSI to be classified based on the similarity.

[0095] Based on the feature distribution of the training samples introduced in step two, TSMIL proposes three metric-based bag classification strategies: MaxS, AveS, and HybS. These three different classification strategies all apply the similarity metric shown in formula (10), which can provide larger values ​​for more similar vectors. During testing, the distance between the test sample and each cluster is measured using different metric methods, and the closest cluster category is used as the prediction for the test sample. Assuming that for a specific dataset, the distribution of WSI features in the feature space in the training set is shown in Figures 4(a) and 4(b), where each color represents a category. Clearly, training samples of the same category are more compact in the feature space, while training samples of different categories have a larger distance interval (white blank areas). The three classification strategies designed in TSMIL are as follows:

[0096] 1) MaxS (Maximum Similarity): As shown in Figure 4(a), after all training samples and test samples X are embedded into the feature space, the similarity between the test sample and all training samples in each cluster is measured, and the maximum similarity of each cluster is considered as the prediction score for that class. Three different clusters generate three prediction scores. Finally, all prediction scores are normalized based on the Softmax function.

[0097] 2) AveS (Average Similarity): As shown in Figure 4(b), after the encoding layer embeds all training samples into the feature space, it extracts the average feature of all training samples in each cluster as the "cluster center" of that cluster. During testing, it is only necessary to calculate the similarity between the feature vector of the test sample X and all "cluster centers", and finally perform a Softmax operation on all similarities to obtain the final prediction score.

[0098] 3) HybS (Hybrid Similarity): HybS combines the two classification strategies mentioned above. Specifically, the average prediction score of the two strategies is used as the final prediction result.

[0099] This concludes the overall implementation process of the tumor subtype diagnosis method based on whole-section pathological images of the present invention.

[0100] Experiments were conducted on two publicly available pathology datasets, TCGA-NSCLC and TCGA-RCC, to verify the feasibility of the method of this invention as a general architecture that can be applied to a variety of cancers.

[0101] 1) Experimental dataset.

[0102] ①TCGA-RCC. The Caner Genome Atlas Renal Cell Carcinoma (TCGA-RCC) is a publicly available pathology dataset containing three categories: chromophobe renal cell carcinoma (KICH), clear cell renal cell carcinoma (KIRC), and papillary renal cell carcinoma (KIRP). The TCGA-RCC dataset contains a total of 937 image samples (WSIs), consisting of 121 WSIs belonging to the KICH category, 519 WSIs belonging to the KIRC category, and 297 WSIs belonging to the KIRP category. Clearly, this dataset is multi-class and imbalanced. After preprocessing, at a 20x magnification, the dataset generates approximately 3.9 million image patches, averaging about 4100 image patches per WSI. The only learnable information in this dataset is the class label of the WSI.

[0103] ②TCGA-NSCLC. The TCGA non-small cell lung cancer (TCGA-NSCLC) dataset contains two categories: lung squamous cell carcinoma (LUSC) and lung adenocarcinoma (LUAD). The TCGA-NSCLC dataset contains 1053 pathological images (WSIs), consisting of 541 WSIs belonging to the LUAD category and 512 WSIs belonging to the LUSC category. After preprocessing, at a magnification of 20x, the dataset generates approximately 3.9 million image patches in total, averaging about 3700 image patches per WSI.

[0104] 2) Experimental evaluation indicators.

[0105] Model evaluation refers to measuring the specific performance of a model using certain technical methods. Evaluation metrics are quantitative standards for evaluating model performance and play an important role in comparing different methods. Different metrics reflect different model performances; therefore, evaluating multiple metrics is necessary to obtain a more comprehensive and objective evaluation result. The classification evaluation metrics used in this invention include accuracy and AUC (area under the ROC curve). The derivation of these metrics is based on the confusion matrix, as shown in Table 1.

[0106] Table 1 Confusion Matrix

[0107]

[0108] Accuracy refers to the percentage of correctly predicted samples out of the total number of samples in all prediction results. The specific calculation process is shown in formula (12). AUC (Area Under Curve) refers to the area under the ROC curve, where the horizontal axis of the ROC curve is the False Positive Rate (FPR) and the vertical axis is the True Positive Rate (TPR). The False Positive Rate is the percentage of false positive samples out of all true negative samples. The specific calculation process is shown in formula (13). The True Positive Rate is the percentage of true positive samples out of all true positive samples. The specific calculation process is shown in formula (14). The AUC value ranges from 0 to 1. The closer the AUC is to 1, the better the model performance.

[0109]

[0110]

[0111]

[0112] 3) Experiment setup.

[0113] During training, TSMIL employs a Ranger optimizer with a fixed weight decay factor of 1e-5 to optimize the network. The learning rate is initially set to 0.1 and undergoes cosine decay during training. The training epochs are 300, and the batch size B is 16. Furthermore, TSMIL also employs an early stopping strategy during training with a patience value of 50. In the proposed TSMIL, d in MFEM... s3 d s4 and d rThe values ​​are 1024, 2048, and 3072, respectively. In the Proj-Fc layer, d... p Set to 768, while in GAM p d The value was set to 0.35, d a The value is 128. In the loss function, the scaling factor γ and the spacing factor m are set to 256 and 0.25, respectively. The performance metrics for all experiments are the average of five experiments, and the 95% confidence intervals (CIs) are estimated based on 1000 iterations of the bootstrap algorithm.

[0114] 4) Experimental results.

[0115] ① Experimental results and analysis on TCGA-NSCLC and TCGA-RCC.

[0116] To verify the feasibility and superiority of the proposed method TSMIL, this section selects excellent weakly supervised methods from recent years as comparative methods and compares their experimental results. These methods include ABMIL, DSMIL, CLAM-SB, CLAM-MB, TransMIL, DTFD-MIL, SRCL, and NAGCN. The comparative experimental results are shown in Table 2.

[0117] Table 2 shows the experimental results on TCGA-NSCLC and TCGA-RCC.

[0118]

[0119] As shown in Table 2, the three classification strategies in the proposed TSMIL method exhibit similar performance, with an AUC difference of less than 0.0003 between the two datasets. Clearly, any one of the TSMIL classification methods outperforms other existing methods in both AUC and accuracy. For example, on the TCGA-NSCLC dataset, TSMIL(HybS) and the other two methods achieve similar classification performance, and all three outperform SRCL and TransMIL, specifically by 0.0264 and 0.0391 points in AUC, respectively. On the TCGA-RCC dataset, TSMIL(HybS) and TSMIL(AvgS) show the best AUC performance, while TSMIL(MaxS) achieves the highest accuracy of 98.76% (95% CI, 0.9836–0.9908). Compared with other methods, TSMIL(HybS) achieved an accuracy and AUC that were 3.34% and 0.0072 higher than NAGCN, respectively. TSMIL(MaxS) also outperformed SRCL and TransMIL by 0.0079 and 0.0107 in AUC, respectively. These results demonstrate that the proposed method achieves superior performance using only weakly supervised labels, validating its feasibility for WSI classification.

[0120] ②The necessity of L2 standardization.

[0121] This section demonstrates the impact of the L2 Norm layer on the performance of TSMIL. In WSI feature optimization, the feature representation of a bag is optimized by reducing the angle between bag feature vectors within a class and increasing the angle between bag feature vectors between classes. The bag feature vector is obtained by aggregating the feature vectors of all examples using GAM. Therefore, the feature vectors of examples influence the expression of bag features to some extent. Since only the angle between vectors is optimized, only the direction of the vector is the information that TSMIL focuses on, while the magnitude of the vector may be redundant information and may even affect the performance of the model. Therefore, to verify this idea, this invention designed two sets of comparative experiments on three datasets (TCGA-NSCLC and TCGA-RCC). In one set of experiments, the TSMIL method omits the L2 Norm layer, while the other set still uses the L2 Norm layer. Both sets of experiments use TSMIL(MaxS) as the classification method. The experimental comparison results on the three datasets are shown in Figure 5(a), where w / o L2 Norm indicates that the L2 Norm layer was not used, and with L2 Norm indicates that the L2 Norm layer was used. The accuracy was used as the evaluation metric for the model comparison.

[0122] Experimental results show that the L2 Norm layer significantly improves the classification performance of TSMIL, and may even be a key layer for TSMIL to effectively perform its classification function. Figure 3 As can be seen, TSMIL with the L2 Norm layer achieves 5.49% and 12.04% higher accuracy on the TCGA-RCC and TCGA-NSCLC datasets, respectively, compared to TSMIL without the L2 Norm layer.

[0123] To explore the reasons for the significant improvement in TSMIL classification performance by the L2 Norm layer, this invention demonstrates the change in loss during training on the TCGA-NSCLC dataset, which shows a substantial performance improvement, as shown in Figure 5(b). Solid lines represent models using the L2 Norm layer, while dashed lines represent models without it. Figure 5(b) shows that compared to the model without the L2 Norm layer, performing L2 normalization on the example features effectively reduces the model's loss. The difference in loss between the two models is even higher than 200, indicating that removing the magnitude information of the example features makes the model's classification effective, and also demonstrating the necessity and effectiveness of adding the L2 Norm layer in this invention.

[0124] ③ Explainability.

[0125] This section demonstrates the interpretability of TSMIL. In practical applications, TSMIL aims to provide professional pathologists with a more intuitive visual representation by displaying highly suspicious areas in the WSI, thereby assisting pathologists in interpreting slides and reducing their workload. In TSMIL, each WSI is segmented into a series of image patches. Then, after MFEM extracts the features of each image patch, GAM assigns a weight score to each feature. Here, this invention visualizes the weights of the image patch features into the original WSI, displaying the regions of interest to TSMIL in the form of an attention heatmap. High attention values ​​are indicated in red, representing a high contribution of the image patch to the classification prediction of the WSI, while low attention values ​​are indicated in blue, representing no contribution to the classification prediction of the WSI. Figures 6(a) to 6(d) As shown, two WSIs were selected from TCGA-RCC, with the two WSIs from TCGA-RCC annotated by a pathology expert with more than five years of experience, as shown in Figure 6(e), but not the index diagrams of Figures 6(b) and 6(d). Figures 6(a) to 6(d) In Figures 6(a) and 6(c), the area within the blue curve is an approximate pixel-level annotation by a professional pathologist, representing a large number of positive lesions in that area. Figures 6(b) and 6(d) are probability heatmaps output by TSMIL.

[0126] from Figures 6(a) to 6(d)The probability heatmaps show that TSMIL has the ability to identify cancer-related suspicious areas in positive WSIs. A comparison between the two sets of images clearly shows that TSMIL shows significant positive activation in the expert-annotated areas, indicating that TSMIL identifies a large number of cancer-related suspicious areas in these regions.

[0127] In summary, the present invention has the following characteristics:

[0128] 1) This invention designs a framework that integrates weakly supervised multi-instance learning and metric-based learning for training, realizing a tumor-assisted intelligent diagnostic method for whole-slice pathological images. This method allows for model training using only WSI image-level label information without relying on detailed pixel-level annotations, achieving performance superior to models requiring a large number of pixel-level annotated images for training. This avoids the dependence of deep learning methods on pixel-level annotations of whole-slice pathological images, thus significantly reducing the enormous workload of pathology experts in annotating images.

[0129] 2) The pathological image classification and diagnosis method designed in this invention is based on the global overall features of WSI. The acquisition process of these features includes three modules: a multi-scale feature extraction module (MFEM), a projection module (PM), and a gated attention module (GAM). First, when pathologists perform pathological diagnosis, they usually observe multiple scales on a slide to achieve a complete examination and accurate judgment of suspicious lesions, avoiding missed or misdiagnosed cancer lesions. Based on this idea, this invention designs a multi-scale feature extraction module (MFEM) that can extract features at multiple scales for each example and achieve a complete representation of the example features by fusing multi-scale features. Second, unlike the existing classic multiple instance learning (MIL) method that does not perform any operations on example features, this invention proposes a projection module (PM) that aims to project example features into a low-dimensional unit hypersphere feature space, so that different example feature vectors only differ in direction in the unit feature space; at the same time, this invention applies two different regularization operations in PM to stabilize the training of the encoding layer. Furthermore, the purpose of this invention in applying the gated attention mechanism (GAM) is to assign a learnable weight information to each example vector and to generate an embedded feature representation of the bag by fusing all the example feature vectors in a bag.

[0130] 3) After extracting the overall features of WSI, the classification training process for pathological images can continuously improve the similarity between WSI features of the same category, while reducing the similarity between WSI features of different categories, so that WSI features of the same category are more similar and WSI features of different categories are more different. These properties are more conducive to subsequent metric-based classification and diagnosis of WSI.

[0131] 4) Based on the above training sample distribution, TSMIL proposes three metric-based WSI package classification strategies: MaxS, AveS, and HybS. When classifying and diagnosing test samples, these three different classification strategies measure the distance between the test sample and each cluster using different metrics, and use the closest cluster category as the prediction for the test sample.

[0132] 5) This invention utilizes a weakly supervised multi-instance learning framework, which is more easily transferred and applied to the analysis of various types of histopathological images. It has strong generalization ability, low learning cost, and is easy to implement. Moreover, this invention is a novel deep learning framework that, during testing, achieves assisted diagnosis of whole-slice pathological images in a short time after only one forward propagation of the network. It has high computational efficiency and low hardware requirements.

Claims

1. A tumor subtype diagnosis method for whole-slide pathological images, characterized by, include: Construct a tumor subtype diagnostic model, which includes overall feature extraction; Overall feature extraction is used to extract the embedded feature representation of the input WSI. It includes a feature extraction module, a projection module, and a gated attention module; The feature extraction module is used to extract features; The projection module projects the extracted features into a low-dimensional unit feature space to obtain multiple unit vectors, so that different feature vectors only differ in direction in the unit feature space; the gated attention module fuses the output of the projection module to generate embedded feature representations. The projection module includes a trainable BatchNorm1d layer, a fine-tuning layer Proj-Fc parameterized by weights, and an L2Norm layer; the BatchNorm1d layer is used to normalize the input feature matrix along the feature dimension. ; Output of Proj-Fc for , This is the weight matrix. The activation function is used; the L2 Norm layer is used to... Each feature vector in the vector is standardized to a unit vector; The tumor subtype diagnostic model was trained using the tumor WSI dataset that had been classified into subtypes as training samples. During the training process, the similarity between the embedding feature representations of WSIs of the same category was increased, while the similarity between the embedding feature representations of WSIs of different categories was decreased. After training, multiple clusters were obtained, and the number of clusters was the number of subtype classifications. Each cluster contained the embedding feature representations of all training samples belonging to that cluster. The WSIs to be classified are preprocessed, and the preprocessed tumor subtype diagnostic model is used to perform subtype diagnosis on the preprocessed WSIs. The diagnostic process includes: using the feature extraction module to obtain the embedded feature representation of the preprocessed WSIs to be classified; measuring the similarity between the WSIs to be classified and each cluster based on the embedded feature representation of the WSIs to be classified and the embedded feature representation of each training sample in each cluster; and obtaining the subtype diagnosis result of the WSIs to be classified based on the similarity. 2.The tumor subtype diagnosis method for whole slide pathology images according to claim 1, characterized in that, The similarity between the embedded feature representation of the WSI to be classified and each cluster can be measured using any of the following methods: Method 1: Measure the similarity between the embedded feature representation of the WSI to be classified and the embedded feature representation of all training samples in each cluster, and take the maximum similarity of each cluster as the similarity between the WSI to be classified and that cluster; Method 2: Calculate the average embedding feature representation of all training samples in each cluster and use it as the cluster center of that cluster. Measure the similarity between the embedding feature representation of the WSI to be classified and the cluster center of each cluster, and use this as the similarity between the WSI to be classified and each cluster. Method 3: Calculate the average of the results obtained from Method 1 and Method 2 as the similarity between the WSI to be classified and each cluster. 3.The tumor subtype diagnosis method for whole slide pathology images according to claim 1, characterized in that, The loss function used in training the tumor subtype diagnostic model is: , In the formula, Indicates the loss value; Indicates the batch size for one training iteration; and Indicates package The similarity scores between the number of similar samples and the number of dissimilar samples are respectively... and ; Indicates the scaling factor; and Indicates the weighting factor; and These represent intra-class spacing and inter-class spacing, respectively. Two eigenvectors and similarity score for , It is the inner product of two vectors; Represents the Euclidean regularization of a vector. 4.The tumor subtype diagnosis method for whole slide pathology images according to claim 1, characterized in that, The preprocessing includes segmenting the tissue region in WSI into multiple image blocks.

5. The method for diagnosing tumor subtypes based on whole-section pathological images according to claim 3, characterized in that, weight factor and are respectively: , In the formula, and These represent intra-class spacing and inter-class spacing, respectively. Indicates a zero-point cutoff operation. and These are two hyperparameters, referred to as the interval factor and the scaling factor, respectively. 6.The tumor subtype diagnosis method for whole slide pathology images according to claim 1, wherein, The feature extraction module uses ResNet101 as the backbone network. It processes the features output from Stage 3 and Stage 4 of ResNet101 through an adaptive average pooling layer, and then concatenates the outputs of the adaptive average pooling layer to extract the features corresponding to each image patch. 7.The tumor subtype diagnosis method for whole slide pathology images according to claim 1, wherein, The gated attention module includes an Attn-Fc1 layer, an Attn-Fc2 layer, an Attn-Fc3 layer, and a Dropout layer. The Attn-Fc1 layer outputs a compressed feature for the input features and maps the feature values ​​in the compressed feature to the range of positive infinity to negative infinity using the Tanh activation function. The Attn-Fc2 layer acts as a gate and maps the network's output value to the range of 0 to 1 using the Sigmoid activation function to control the output of the Attn-Fc1 layer. The outputs of the Attn-Fc1 and Attn-Fc2 layers are multiplied bitwise and then fed into the Attn-Fc3 layer. The Attn-Fc3 layer generates an attention score for each feature and processes it through the Dropout layer. Then, the attention scores processed by the Dropout layer are used to perform a weighted summation of the unit vectors to obtain the embedded feature representation. 8.The tumor subtype diagnosis method for whole slide pathology images according to claim 4, characterized in that, The preprocessing includes: first, converting WSI from RGB color space to HSV color space; then, using Otsu's method to calculate the segmentation thresholds for the background and tissue regions in the saturation channels of the HSV color space; and based on the segmentation thresholds, binarizing the saturation channels to extract the tissue mask to obtain the tissue regions; and then using a sliding window method to segment the tissue regions into a series of image blocks of the same size. 9.The tumor subtype diagnosis method for whole slide pathology images according to claim 8, characterized in that, After extracting the tissue mask, mean filtering and morphological closing operations are required to extract the tissue region.

10. The method for diagnosing tumor subtypes based on whole-section pathological images according to claim 1, characterized in that, The features extracted by the feature extraction module are multi-scale features.

Images