WSI classification method based on dynamic graph modeling and knowledge perception attention mechanism

Abstract

The invention discloses a WSI classification method based on dynamic graph modeling and a knowledge perception attention mechanism, and belongs to the field of medical image processing. The method comprises the following steps: performing feature extraction on a plurality of image blocks obtained by WSI segmentation by using a pre-trained visual encoder; constructing a dynamic graph and preliminarily updating a central node; inputting the dynamic directed graph into an attention network, performing information aggregation and updating on nodes in the graph by utilizing a knowledge perception attention mechanism, and generating graph-level embedding according to updated node representation; and performing classification through a classifier according to graph-level embedding to obtain a classification result of the full-view slice image. According to the method, through instance embedding and dynamic graph construction, effective learning can be carried out by utilizing the relationship between nodes under the condition of no accurate annotation; and by aggregating image block features and image structure information and combining classifier output, the accuracy of slice-level classification is further improved.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, and in particular to a WSI classification method based on dynamic graph modeling and knowledge-aware attention mechanisms. Background Technology

[0002] The digitization of whole-slide images (WSI) has driven the rapid development of computational pathology. WSI offers ultra-high resolution and cross-scale histological detail, making it suitable for weakly supervised multi-instance learning (MIL) paradigms that use image patches as instances and slices as packages. It has been widely used in pathological slice classification, subtype identification, and prognostic studies. Compared to fully supervised methods based on pixel-level annotation, MIL relies solely on slice-level labels, balancing annotation costs and clinical usability. However, the tumor microenvironment in WSI is complex, with long-distance dependencies and dynamic interactions between instances. Static aggregation of salient instances alone cannot form a stable and comprehensive global representation, affecting classification accuracy and generalization ability.

[0003] Traditional pathological diagnosis relies on long-term observation and comprehensive judgment by experts under a microscope, which faces problems such as large workload, limited efficiency, and insufficient consistency across observers and within the same observer. WSI (Weighted Infection Samples) is massive in volume, with lesions exhibiting small targets, sparse distribution, and diverse morphologies, resulting in strong tissue heterogeneity. Multi-center acquisition leads to differences in scanner and staining, resulting in distribution bias and noise labels. Existing MIL (Methods for Improving Pathology) often focuses on selecting the "most significant" instance, ignoring the structural relationships between instances; static graph methods based on spatial adjacency construction typically use fixed, undirected topologies, making it difficult to transmit long-distance information and prone to overparameterization and overfitting; while models based on global attention have long-range modeling capabilities, they are sensitive to data scale and lack sufficient utilization of directional dependence and edge semantics, resulting in limited robustness in complex scenarios. Summary of the Invention

[0004] Purpose of the invention: To address the above problems, the purpose of this invention is to provide a WSI classification method based on dynamic graph modeling and knowledge-aware attention mechanisms.

[0005] Technical solution: The WSI classification method based on dynamic graph modeling and knowledge-aware attention mechanism of the present invention includes the following steps:

[0006] Step 1, Feature Extraction:

[0007] Using a pre-trained visual encoder, feature extraction is performed on multiple image patches obtained from the segmentation of the full-view slice image to obtain the instance embedding of each image patch;

[0008] Step 2, Dynamic Graph Construction and Initial Update:

[0009] Based on the similarity between instance embeddings of multiple image patches, the Top-K most similar nodes are adaptively selected as the neighbor nodes for each central node, and a dynamic directed graph consisting of nodes and directed edges is constructed, where the directed edges point from the central node to each of its neighbor nodes.

[0010] The central node is updated based on the original instance embedding, neighbor embedding, and nonlinear transformation;

[0011] Step 3: Knowledge-aware graph embedding:

[0012] A dynamic directed graph is input into an attention network, and a knowledge-aware attention mechanism is used to aggregate and update the information of the nodes in the graph. Graph-level embeddings are then generated based on the updated node representations.

[0013] Step 4, Classification:

[0014] Based on graph-level embedding, a classifier is used to classify the images to obtain the classification results of the full-view slice images.

[0015] Further, step 2 includes:

[0016] Each image patch's instance embedding is used as the initial node. A linear transformation is then performed on the instance embedding to generate the node's head embedding and tail embedding, respectively, using the following formula:

[0017] , ,

[0018] in, For the first The original instance embedding of each image patch, and its corresponding head embedding denoted as . Tail embedding is denoted as ; and It is a learnable linear transformation matrix;

[0019] Calculate the first Head embedding of each node With the Tail embedding of each node The dot product similarity between them is calculated using the following formula:

[0020] ,

[0021] In this case, the similarity between a node and itself is set to a minimum value, i.e. ;

[0022] Based on dot product similarity Before selection The most similar nodes are selected as neighbors. , is represented as:

[0023] ,

[0024] in, Indicates the total number of image patches. For hyperparameters;

[0025] In determining each node neighboring nodes Then, construct a set of directed edges. For each directed edge Utilizing instance embedding Head embedding obtained through linear transformation Tail Embedding The weights are obtained by normalizing the dot product similarity between the two. Perform weighted fusion to generate directional edge embeddings. The formula is:

[0026] ,

[0027] in, It is the normalized similarity weight. It is a standard normalization function.

[0028] Furthermore, step 2 also includes:

[0029] Embed the original instance of the node itself The first updated feature of each node is obtained by fusing the embeddings of its neighboring nodes with the weighted aggregation. , is represented as:

[0030] ,

[0031] in, For the first The original instance embedding of each node, and the instance embedding The same feature representation is used as input for subsequent calculations throughout the graph update process;

[0032] The first update feature The second updated feature is then obtained by performing a nonlinear transformation through two fully connected layers. The second update feature The feature representation of the central node in a dynamic directed graph is expressed as follows:

[0033] ,

[0034] ,

[0035] Where ReLU represents the activation function; and It is a learnable weight matrix. Indicates the dimension of the input features. This indicates the dimension of the hidden layer, which is the dimension of the updated features.

[0036] Furthermore, step 3 includes:

[0037] In an attention network, the attention score for each node is first calculated using the following formula:

[0038] ,

[0039] in, The intermediate attention score, calculated in the knowledge-aware attention mechanism, is used to characterize nodes. Its neighboring nodes The relationship between them; Activation functions are used to capture non-linear relationships. It is a learnable weight matrix;

[0040] Next, through normalization operations, the nodes are obtained. The attention weight between a node and its neighboring nodes is calculated using the following formula:

[0041] ,

[0042] Finally, nodes are generated by weighted summation. Neighbor aggregation features ,use The updated node feature representation is shown as:

[0043] ;

[0044] Graph-level embeddings are generated by weighted aggregation of features from neighboring nodes.

[0045] Furthermore, the steps for generating graph-level embeddings include:

[0046] Based on After completing neighbor-level attention aggregation and obtaining updated node feature representations, a graph-level attention pooling module is introduced to perform second-level attention modeling on all nodes. This attention module first performs attention modeling on each node... Calculate the original graph-level attention score The weights of nodes in the graph-level aggregation are obtained through normalization operations, and are expressed as follows:

[0047] ,

[0048] in, Indicates including all On a set of nodes, nodes Normalized attention weights when generating graph-level embeddings This corresponds to the original attention score;

[0049] The final graph-level embedding is generated by weighted summation. , is represented as:

[0050] ,

[0051] in, A node that incorporates neighborhood aggregation information after combining knowledge-aware attention. Final feature representation.

[0052] Furthermore, for steps 3 and 4, a total loss function is constructed, which consists of two parts: cross-entropy loss and graph structure consistency constraint loss based on the dynamic graph structure, expressed as:

[0053] ,

[0054] in, As a regulating factor, This represents the loss due to graph structure consistency constraints. The cross-entropy loss is expressed as:

[0055] ,

[0056] in, Indicates the number of training samples. Indicates the number of categories. For the first Each sample in category The real labels on The category to which this sample belongs, as output by the slice-level classifier. The predicted probability.

[0057] Beneficial effects: Compared with the prior art, the significant advantages of this invention are:

[0058] 1. Dynamic graph modeling and directional edge embedding enhance the ability to express the correlation between nodes: This invention improves the ability to express the correlation between nodes by introducing dynamic graph modeling and directional edge embedding. Through the dynamic graph modeling technology of this invention, neighboring nodes can be adaptively selected according to the similarity between instances, and directional edge embeddings can be constructed. This dynamic topology structure allows the adjacency relationship to be adjusted in real time according to the semantic relevance between nodes, thereby capturing the dependency relationship between distant nodes more flexibly and accurately; avoiding the limitations of fixed graph structure, and improving the model's ability to express complex relationships;

[0059] 2. Enhanced Efficiency and Accuracy of Node Feature Aggregation Based on Knowledge-Aware Attention Mechanism: This invention introduces a knowledge-aware attention mechanism based on head / tail embedding and edge information. This mechanism dynamically calculates attention weights according to the semantic information of each edge, accurately aggregates information from neighboring nodes, and updates node representations, significantly enhancing the efficiency and accuracy of node feature aggregation. Activation function and Normalized attention weights between nodes are calculated, and the features of neighboring nodes are aggregated by weighted aggregation, thereby improving the accuracy and robustness of information aggregation. At the same time, the updated node features generate graph-level embeddings, which can comprehensively represent the entire WSI image and enhance the global expressive power of features.

[0060] 3. Dynamic graph modeling and feature aggregation strategies improve the model's generalization ability: By dynamically adjusting adjacency relationships, this invention can better model long-distance dependencies between nodes, avoiding over-reliance on local salient instances and effectively improving generalization ability. The dynamic graph mechanism of this invention can dynamically construct topological structures based on different slice images, handling cross-scale and diverse instance features. This dynamic modeling method enables this invention to adapt to different data distributions, improving robustness and generalization ability, especially showing better stability when dealing with problems such as noisy labels and distribution shifts.

[0061] 4. Weakly supervised learning and multi-instance learning framework improve annotation efficiency and accuracy: This invention adopts a weakly supervised multi-instance learning framework, which greatly reduces annotation costs and improves classification accuracy. It trains using slice-level labels without strict pixel-level annotation. Through instance embedding and dynamic graph construction, this invention can effectively learn by utilizing the relationships between nodes without precise annotation. By aggregating image patch features and graph structure information and combining them with classifier output, the accuracy of slice-level classification is further improved. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the entire process of the WSI classification method based on dynamic graph modeling and knowledge-aware attention mechanism.

[0063] Figure 2 Build a process for dynamic graphs;

[0064] Figure 3 For knowledge perception attention mechanisms and interpretability analysis processes;

[0065] Figure 4 This is a schematic diagram of the ROC curve of the method of the present invention on the dataset;

[0066] Figure 5 This is a schematic diagram of the PR curve of the method of the present invention on the dataset. Detailed Implementation

[0067] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the embodiments of the present invention, and not all structures.

[0068] In the following description, specific details such as target system architecture and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0069] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0070] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0071] Furthermore, in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0072] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include the target features, structures, or characteristics described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0073] The WSI classification method based on dynamic graph modeling and knowledge-aware attention mechanism described in this embodiment is illustrated in the framework diagram below. Figure 1 As shown, the method includes the following steps:

[0074] Step 1, Feature Extraction:

[0075] Using a pre-trained visual encoder, feature extraction is performed on multiple image patches obtained from the segmentation of the full-view slice image to obtain the instance embedding of each image patch.

[0076] In one example, the datasets used included three publicly available datasets: CAMELYON16, TCGA-BRCA, and TCGA-NSCLC, totaling over 2430 whole-field section images (WSIs). These images were digitized using different scanners. The CAMELYON16 dataset contained 400 WSIs, annotated by pathologists to distinguish specific histological states in lymph node sections; the TCGA-BRCA dataset contained 977 WSIs, annotated for two breast histological subtypes; and the TCGA-NSCLC dataset contained 1053 WSIs, annotated for two lung histological subtypes. All datasets used only section-level labels. According to the experimental setup, the CAMELYON16 dataset was divided into a training set of 270 images and a test set of 130 images. The TCGA-BRCA and TCGA-NSCLC datasets used a five-fold Monte Carlo cross-validation split, with the training set comprising approximately 80%, and the validation and test sets each comprising approximately 10%. After data collection, foreground and background segmentation was performed on low-resolution HE-stained images. Foreground and background segmentation is achieved using traditional image processing algorithms, including thresholding, hole filling, dilation and erosion, and small object removal. First, a binarization threshold is calculated to obtain a binary image. Then, the image is inverted to brighten the foreground region, and closed small holes are removed. Next, dilation and erosion operations are used to eliminate unclosed holes, and newly generated small holes are removed again. Finally, useless areas outside the tissue, such as contaminant debris, are removed, thus achieving accurate foreground region extraction. Subsequently, image patches of a specified size are extracted from the foreground region for use in subsequent steps. Finally, a data queue index is established to index and queue the panoramic slice image, foreground and background segmentation mask, image patches, and corresponding labels to support efficient weakly supervised training and evaluation.

[0077] An offline feature extractor was used to extract features from the collected data, enhancing the feature extraction capabilities of pathological images. The extraction process was performed on a workstation equipped with an NVIDIA RTX 4060 GPU. After regularization, the input image patches were passed through the final layer of the network, outputting fixed-length feature vectors with 1024 or 512 dimensions depending on the extractor type. The results were then normalized and stored, including image features, starting pixel coordinates of the image patch, size, relative coordinates, panoramic slice image source, and magnification, for easy loading and processing later. This offline extraction mechanism reduces computational overhead during training and preserves spatial information to support masking and cross-resolution consistency.

[0078] Given a WSI First, it is divided into non-overlapping image patches. ,in The number of image patches, each image patch It is treated as a graph node. For a node... The feature extraction process is as follows: ,in, For depth encoders (such as ResNet or Vision Transformer). It is a node The feature vectors of all nodes form a set. The offline feature mechanism ensures efficient training under weak supervision, while preserving instance-level spatial relationships so that invalid instances can be masked in the subsequent graph modeling process, ensuring cross-scale consistency.

[0079] Step 2, Dynamic Graph Construction and Initial Update:

[0080] Based on the similarity between instance embeddings of multiple image patches, the Top-K most similar nodes are adaptively selected as the neighbor nodes for each central node, and a dynamic directed graph consisting of nodes and directed edges is constructed, where the directed edges point from the central node to each of its neighbor nodes.

[0081] The central node is updated based on the original instance embedding, neighbor embedding, and nonlinear transformation.

[0082] Further, step 2 includes:

[0083] Each image patch's instance embedding is used as the initial node. A linear transformation is then performed on the instance embedding to generate the node's head embedding and tail embedding, respectively, using the following formula:

[0084] , ,

[0085] in, For the first The original instance embedding of each image patch, and its corresponding head embedding denoted as . Tail embedding is denoted as Head and tail embeddings better model dynamic interactions by distinguishing the roles of nodes as senders and receivers; and It is a learnable linear transformation matrix;

[0086] Calculate the first Head embedding of each node With the Tail embedding of each node The dot product similarity between them is calculated using the following formula:

[0087] ,

[0088] In this case, the similarity between a node and itself is set to a minimum value, i.e. ;

[0089] Based on dot product similarity Before selection The most similar nodes are selected as neighbors. , is represented as:

[0090] ,

[0091] in, Indicates the total number of image patches. For hyperparameters;

[0092] In determining each node neighboring nodes Then, construct a set of directed edges. For each directed edge Utilizing instance embedding Head embedding obtained through linear transformation Tail Embedding The weights are obtained by normalizing the dot product similarity between the two. Perform weighted fusion to generate directional edge embeddings. The formula is:

[0093] ,

[0094] in, It is the normalized similarity weight. It is a standard normalization function.

[0095] After feature extraction in step 1, dynamic graph construction and preliminary aggregation are performed in step 2. Figure 2 As shown, in step 2, the input to this process is the instance feature tensor (with shape...). ,in For batch size, For the number of nodes, (where the feature dimension is 1), the output is the edge embedding tensor of the dynamic graph (shape 1). ,in (Number of neighbors). First, a linear projection is performed on the instance feature tensor to obtain the head embedding and tail embedding. Then, the similarity matrix between the head and tail embeddings is calculated. Cosine similarity is calculated by normalizing the input features, and diagonal elements are set to minimum values ​​to suppress self-loop interference. For each node, the highest similarity neighbors are selected to form sparse directed adjacency relationships. The number of neighbors is an adjustable parameter, usually set to 5. Next, a directional embedding is generated for each edge. The sender and receiver information are fused by normalized weights, with the weights based on similarity. The computation process begins with the initial aggregation phase, where weights are applied to the neighbor representations and the self-representation to fuse them, obtaining initial node values ​​with enhanced context. The specific algorithm flow is as follows: input is an instance feature tensor, with batch size equal to the number of samples in the batch, number of nodes equal to the number of image patches per slice, and feature dimension equal to the extractor's output dimension; output is a directional edge embedding tensor. The algorithm first performs linear projection on the features and splits them into head and tail embeddings; calculates the similarity matrix; selects the values ​​and indices of the first few neighbors; and calculates... The algorithm employs weighting, collecting neighbor features, and generating head embeddings that are weights multiplied by neighbor features plus complementary weights. This ensures the adaptability and directionality of the dynamic graph, avoids fixed spatial constraints, and reduces computational complexity through sparse adjacency. Subsequently, two fully connected layers process the fused features, using an activation function in between to further refine the node representations, making them more suitable for the input of subsequent attention mechanisms.

[0096] Ultimately, the constructed dynamic graph possesses dynamic topology and directionality, used to capture the diverse relationships between nodes; WSI is modeled as a dynamic graph. ,in It is a set of nodes. It is a set of edges. It is the set of head and tail embeddings of nodes. It is an edge embedding set.

[0097] After the dynamic graph topology and directional edge embedding are constructed, the initial aggregation stage is entered, which aims to fuse the instance features of each node with the weighted features of selected neighbors in order to capture local contextual information.

[0098] Furthermore, step 2 also includes:

[0099] Embed the original instance of the node itself The first updated feature of each node is obtained by fusing the embeddings of its neighboring nodes with the weighted aggregation. This effectively enhances the node's ability to perceive local context, as shown below:

[0100] ,

[0101] in, For the first The original instance embedding of each node, and the aforementioned instance embedding The same feature representation is used as input for subsequent calculations throughout the graph update process;

[0102] The first update feature The second updated feature is then obtained by performing a nonlinear transformation through two fully connected layers. The fused features are mapped to a richer representation space, and the second updated features are... The feature representation of the central node in a dynamic directed graph is expressed as follows:

[0103] ,

[0104] ,

[0105] Where ReLU represents the activation function; and It is a learnable weight matrix. Indicates the dimension of the input features. This indicates the dimension of the hidden layer, which is the dimension of the updated features.

[0106] Step 3: Knowledge-aware graph embedding:

[0107] A dynamic directed graph is input into an attention network, and a knowledge-aware attention mechanism is used to aggregate and update information on the nodes in the graph. Graph-level embeddings are then generated based on the updated node representations.

[0108] After completing the dynamic graph construction and preliminary aggregation in step 2, knowledge-aware attention and graph-level classification are performed in step 3. To more effectively utilize the inter-node relationships generated by the dynamic graph construction, a knowledge-aware attention mechanism based on head-tail embedding and directional embedding is introduced for updating and aggregating node features.

[0109] Combination Figure 3 As shown, step 3 further includes:

[0110] In an attention network, the attention score for each node is first calculated using the following formula:

[0111] ,

[0112] in, The intermediate attention score, calculated in the knowledge-aware attention mechanism, is used to characterize nodes. Its neighboring nodes The relationship between the attention midpoint score and the aforementioned similarity normalization coefficient used to construct the dynamic graph and edge weights; For different parameters: Originating from head embedding With tail embedding The dot product similarity is used for edge construction and initial aggregation, while Neighborhood information aggregation used for further knowledge perception in attention networks; Activation functions are used to capture non-linear relationships. It is a learnable weight matrix;

[0113] Next, through normalization operations, the nodes are obtained. The attention weight between a node and its neighboring nodes is calculated using the following formula:

[0114] ,

[0115] Finally, nodes are generated by weighted summation. Neighbor aggregation features ,use The updated node feature representation is shown as:

[0116] ;

[0117] Graph-level embeddings are generated by weighted aggregation of features from neighboring nodes.

[0118] Furthermore, the steps for generating graph-level embeddings include:

[0119] Based on After completing neighbor-level attention aggregation and obtaining updated node feature representations, a graph-level attention pooling module is introduced to perform second-level attention modeling on all nodes. This attention module first performs attention modeling on each node... Calculate the original graph-level attention score , Intermediate score of attention with neighbor level The attention scoring networks are calculated using different learnable parameters; the former is used for graph-level aggregation, and the latter for neighborhood-level aggregation. The weights of nodes in graph-level aggregation are obtained through normalization, and are expressed as follows:

[0120] ,

[0121] in, Indicates including all On a set of nodes, nodes Normalized attention weights when generating graph-level embeddings The corresponding raw attention score should be noted. It describes nodes Its neighboring nodes Local attention relationships between them are used for neighborhood-level feature aggregation; while The global attention weights are used to aggregate node features into graph-level embeddings; these two belong to different levels of attention computation.

[0122] The final graph-level embedding is generated by weighted summation. This graph-level embedding By fusing features from all nodes and highlighting the instances most important to the classification task based on attention weights, a powerful feature representation is provided for subsequent classifiers, expressed as:

[0123] ,

[0124] in, A node that incorporates neighborhood aggregation information after combining knowledge-aware attention. Final feature representation.

[0125] Furthermore, for steps 3 and 4, a loss function is constructed. This total loss function comprises two parts: cross-entropy loss and graph structure consistency constraint loss based on the dynamic graph structure, expressed as:

[0126] ,

[0127] in, As a regulating factor, This represents the loss due to graph structure consistency constraints. The cross-entropy loss is expressed as:

[0128] ,

[0129] in, Indicates the number of training samples. Indicates the number of categories. For the first Each sample in category The real labels on The category to which this sample belongs, as output by the slice-level classifier. The predicted probability.

[0130] Step 4, Classification:

[0131] Based on graph-level embedding, a classifier is used to classify the images to obtain the classification results of the full-view slice images.

[0132] For the dynamic graph generated in step 3 The process, including the knowledge-aware attention mechanism in step 4, is trained with the goal of generating graph-level embeddings with global representation capabilities for optimizing classification tasks. For the input WSI, the generated graph-level embeddings predict the class probability distribution using a classifier. .

[0133] The training process lasted for 200 epochs, employing early stopping to prevent overfitting. The Adam optimizer was used with a learning rate of 2e-4 and weight decay of 1e-5, dynamically adjusted using a cosine rule to adapt to the training dynamics. Batch size was dynamically adjusted based on the dataset size, typically set to 1 to handle variable-length sequences. Cross-entropy loss was used to optimize the bag-level classification task, generating graph-level embeddings with global representation capabilities, supplemented by graph structure consistency constraints to improve stability. During training, both bag-level and instance-level classification were optimized simultaneously to improve performance. Instance-level optimization optionally used support vector machine loss to further enhance attention to key instances. Validation set performance was monitored during training, and the model state was saved at the optimal fold.

[0134] After training, performance was evaluated. In a comparative experiment between the classification model based on the method described in this invention and the native base model, image features extracted by the PLIP model pre-trained on pathological data and the ResNet-50 model pre-trained on general data were used for subsequent slice-level classification and subtype identification tasks. In terms of accuracy and AUC, the PLIP model trained on pathological images improved performance by approximately 2.78% and 0.76% respectively compared to the native ResNet-50 model (using the CAMELYON16 dataset as an example), indicating that training on specific pathological images improved the model's performance on related downstream tasks. In the comparison between multi-task multi-instance learning and traditional single-task models, several classic single-task models were compared under different tasks. For slice-level binary classification, Mean-pooling, Max-pooling, AB-MIL, CLAM, DSMIL, TransMIL, DTFD-MIL, and R2T-MIL were used for comparison under the same features. For cancer subtype classification, the same baseline model was used for comparison under the same features. In ablation experiments, the dynamic graph strategy model's prediction results were significantly higher than those of the non-graph strategy and the fixed graph strategy. This indicates that dynamic graph modeling plays a mutually reinforcing role across different branches, resulting in higher performance. Finally, the consistency of results between multi-task branches was evaluated. For the instance classifier branch, the instance with the highest attention score was used as the slice-level prediction for that branch. For the slice classifier branch, the prediction result of that classifier was directly used, and the accuracy of the prediction results between the two branches was calculated. Experiments showed that the classification performance of the two branches was similar, indicating that different branches of the model have consistent classification capabilities, improving result consistency and interpretability.

[0135] like Figure 4As shown, the ROC curves and average AUC results of two feature extraction models based on PLIP and ResNet-50 are presented on the Camelyon16, TCGA-BRCA, and TCGA-NSCLC datasets. It can be seen that all models achieved high AUC values ​​on all three datasets, indicating that the method described in this invention possesses excellent classification and discriminative capabilities. Among them, the model based on PLIP features performs particularly well, achieving AUCs of 0.991±0.013, 0.978±0.024, and 0.993±0.010 on the Camelyon16, TCGA-BRCA, and TCGA-NSCLC datasets, respectively, demonstrating stable and consistent high performance. The model based on ResNet-50 features also achieved high AUC performance, with AUCs of 0.980±0.017, 0.965±0.021, and 0.988±0.010 on the three datasets, respectively, indicating that this feature extraction method can also effectively support the model to make accurate judgments.

[0136] like Figure 5 As shown, the corresponding PR curves and average precision (AP) results are presented, further validating the model's performance advantage in positive sample prediction. The model based on PLIP features achieved near-ideal AP values ​​on all three datasets, with APs of 0.991±0.012, 0.953±0.046, and 0.993±0.010 on the Camelyon16, TCGA-BRCA, and TCGA-NSCLC datasets, respectively, indicating that the model maintains high prediction accuracy across different recall ranges. Meanwhile, the model based on ResNet-50 features also demonstrated good positive sample recognition capabilities, achieving APs of 0.979±0.018, 0.886±0.065, and 0.987±0.008 on the same three datasets. Overall, the PR curve results indicate that both feature extraction models effectively support high-precision prediction, with PLIP features exhibiting a more consistent and stable advantage across different datasets.

Claims

1. A WSI classification method based on dynamic graph modeling and knowledge-aware attention mechanism, characterized in that, Includes the following steps: Step 1, Feature Extraction: Using a pre-trained visual encoder, feature extraction is performed on multiple image patches obtained from the segmentation of the full-view slice image to obtain the instance embedding of each image patch; Step 2, Dynamic Graph Construction and Initial Update: Based on the similarity between instance embeddings of multiple image patches, the Top-K most similar nodes are adaptively selected as the neighbor nodes for each central node, and a dynamic directed graph consisting of nodes and directed edges is constructed, where the directed edges point from the central node to each of its neighbor nodes. The central node is updated based on the original instance embedding, neighbor embedding, and nonlinear transformation; Step 3: Knowledge-aware graph embedding: A dynamic directed graph is input into an attention network, and a knowledge-aware attention mechanism is used to aggregate and update the information of the nodes in the graph. Graph-level embeddings are then generated based on the updated node representations. Step 4, Classification: Based on graph-level embedding, a classifier is used to classify the images to obtain the classification results of the full-view slice images.

2. The WSI classification method based on dynamic graph modeling and knowledge-aware attention mechanism according to claim 1, characterized in that, Step 2 includes: Each image patch's instance embedding is used as the initial node. A linear transformation is then performed on the instance embedding to generate the node's head embedding and tail embedding, respectively, using the following formula: ， ， in, For the first The original instance embedding of each image patch, and its corresponding head embedding denoted as . Tail embedding is denoted as ; and It is a learnable linear transformation matrix; Calculate the first Head embedding of each node With the Tail embedding of each node The dot product similarity between them is calculated using the following formula: ， In this case, the similarity between a node and itself is set to a minimum value, i.e. ; Based on dot product similarity Before selection The most similar nodes are selected as neighbors. , represented as: ， in, This indicates the total number of image patches. For hyperparameters; In determining each node neighboring nodes Then, construct a set of directed edges. For each directed edge Utilizing instance embedding Head embedding obtained through linear transformation Tail Embedding The weights are obtained by normalizing the dot product similarity between the two. Perform weighted fusion to generate directional edge embeddings. The formula is: ， in, It is the normalized similarity weight. It is a standard normalization function.

3. The WSI classification method based on dynamic graph modeling and knowledge-aware attention mechanism according to claim 2, characterized in that, Step 2 also includes: Embed the original instance of the node itself The first updated feature of each node is obtained by fusing the embeddings of its neighboring nodes with the weighted aggregation. , represented as: ， in, For the first The original instance embedding of each node, and the instance embedding The same feature representation is used as input for subsequent calculations throughout the graph update process; The first update feature The second updated feature is then obtained by performing a nonlinear transformation through two fully connected layers. The second update feature The feature representation of the central node in a dynamic directed graph is expressed as follows: ， ， Where ReLU represents the activation function; and It is a learnable weight matrix. Indicates the dimension of the input features. This indicates the dimension of the hidden layer, which is the dimension of the updated features.

4. The WSI classification method based on dynamic graph modeling and knowledge-aware attention mechanism according to claim 3, characterized in that, Step 3 includes: In an attention network, the attention score for each node is first calculated using the following formula: ， in, The intermediate attention score, calculated in the knowledge-aware attention mechanism, is used to characterize nodes. Its neighboring nodes The relationship between them; Activation functions are used to capture non-linear relationships. It is a learnable weight matrix; Next, through normalization operations, the nodes are obtained. The attention weight between a node and its neighboring nodes is calculated using the following formula: ， Finally, nodes are generated by weighted summation. Neighbor aggregation features ,use The updated node feature representation is shown as: ； Graph-level embeddings are generated by weighted aggregation of features from neighboring nodes.

5. The WSI classification method based on dynamic graph modeling and knowledge-aware attention mechanism according to claim 4, characterized in that, The steps to generate graph-level embeddings include: Based on After completing neighbor-level attention aggregation and obtaining updated node feature representations, a graph-level attention pooling module is introduced to perform second-level attention modeling on all nodes. This attention module first performs attention modeling on each node... Calculate the original graph-level attention score The weights of nodes in the graph-level aggregation are obtained through normalization operations, and are expressed as follows: ， in, Indicates including all On a set of nodes, nodes Normalized attention weights when generating graph-level embeddings This corresponds to the original attention score; The final graph-level embedding is generated by weighted summation. , represented as: ， in, A node that incorporates neighborhood aggregation information after combining knowledge-aware attention. Final feature representation.

6. The WSI classification method based on dynamic graph modeling and knowledge-aware attention mechanism according to any one of claims 1 to 5, characterized in that, For steps 3 and 4, a total loss function is constructed, which consists of two parts: cross-entropy loss and graph structure consistency constraint loss based on the dynamic graph structure, expressed as follows: ， in, As a regulating factor, This represents the loss due to graph structure consistency constraints. The cross-entropy loss is expressed as: ， in, Indicates the number of training samples. Indicates the number of categories. For the first Each sample in category The real labels on The category to which this sample belongs, as output by the slice-level classifier. The predicted probability.

Images