Full-slice image analysis method based on spatial constraint attention and context awareness

Abstract

The invention discloses a full-slice image analysis method based on spatial constraint attention and context perception, and belongs to the technical field of medical image processing and computer vision. The method comprises the following steps: extracting image blocks from a full-slice image, and mapping each image block into a feature vector; constructing a directional dynamic graph; introducing constraints based on spatial position information, and measuring neighborhood influence by a triple relationship; fusing the node features and the edge features to obtain an attention weighting coefficient; updating and optimizing the node representation; and combining the graph convolutional network with instance-level clustering. According to the method, a context sensing graph representation mechanism is constructed, key histological representation is extracted from a full-slice image, and a spatial constraint attention mechanism is introduced to optimize node interaction; therefore, the stability, accuracy and efficiency of tasks such as category discrimination, subtype distinguishing and prognosis related evaluation of large-scale full-slice images are improved in a multi-data-set scene.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and computer vision, and in particular to a method for analyzing whole-slice images based on spatially constrained attention and context awareness. Background Technology

[0002] Histopathological examination is a crucial tool for diagnosing various diseases and guiding treatment strategies. In clinical practice, pathologists use scanning systems to identify key regions in whole-slice images and extract tumor microenvironment information to support clinical decision-making and survival analysis. However, manually assessing key regions of interest is time-consuming and labor-intensive. Due to the complexity of tissue morphology and the continuity of histological features, the assessment process is highly specialized and subjective. With the increasing size of whole-slice images (e.g., 50,000 × 50,000 pixels) and the high cost of manual annotation, improving the efficiency and consistency of assessment has become an urgent problem to be solved.

[0003] Multiple instance learning (MIL) has become an effective framework for whole-slice image analysis, where the whole slice image is treated as a bag, extracted image patches are treated as instances, and features of these instances are aggregated through a deep neural network to generate slice-level predictions. Although the MIL framework has made significant progress in whole-slice image analysis, current methods often neglect the complex interactions between instances, leading to the loss of crucial tumor microenvironment information and thus degrading model performance. Traditional graph-based methods attempt to model spatial relationships through static topological structures, but face several challenges: fixed topology restricts information flow between distant instances and fails to capture dynamic interactions; most methods use undirected graphs, ignoring directional associations and information flow between instances; static graphs are prone to overparameterization, increasing the risk of overfitting and weakening the model's generalization ability. These limitations hinder the accurate modeling of complex interactions.

[0004] Therefore, it is of great significance to develop a more flexible and efficient framework to overcome these challenges, which can assist the pathology slide reading process, reduce workload, and improve the consistency, interpretability and efficiency of large-scale whole slide image processing. Summary of the Invention

[0005] Purpose of the invention: To address the above problems, the purpose of this invention is to provide a full-slice image analysis method based on spatially constrained attention and context awareness.

[0006] Technical solution: The whole-slice image analysis method based on spatially constrained attention and context awareness of the present invention includes the following steps:

[0007] Data preprocessing steps: Extract multiple image patches from the full-slice image, and perform color normalization on the image patches to construct an image patch set;

[0008] Feature extraction steps: Map each image patch in the image patch set to a corresponding feature vector, and use the feature vector as an initial graph node to form a node set;

[0009] Context-aware graph construction steps: Based on the node set, by calculating the association strength between nodes, neighboring nodes are dynamically selected for each node, and edge features are generated to construct a directional dynamic graph; wherein, the association strength is calculated by bilinear projection and dot product similarity;

[0010] The spatially constrained attention mechanism involves the following steps: On the oriented dynamic graph, constraints based on spatial location information are introduced, and the neighborhood influence is measured by triplet relationships, which include the source node, the edge between the source node and its neighboring nodes, and the target neighboring node; node features and edge features are fused to obtain attention weighting coefficients; the neighborhood information is weighted and aggregated using the attention weighting coefficients, and bidirectional interaction is performed with the original node representation to update and optimize the node representation;

[0011] Model training and output steps: Based on the optimized node representation, a graph convolutional network and instance-level clustering are combined; under slice-level supervision, the cross-entropy loss function is adopted and introduced... Figure 1 The model is trained using a consistency regularization term; during the training process, the adjacency weights of the directional dynamic graph are updated based on the pseudo-labels generated by the model prediction, and finally a classification model for full-slice image analysis is obtained.

[0012] Furthermore, the context-aware graph construction steps include:

[0013] The first Image blocks The eigenvectors are denoted as For eigenvectors Through linear mapping Perform pre-encoding to obtain node features , represented as:

[0014] ,

[0015] For the Node characteristics Semantic projection is performed using two independent linear transformations to obtain the query vectors. and key vector , represented as:

[0016] ,

[0017] ,

[0018] in, and The weight matrix is ​​a learnable matrix;

[0019] Calculate the dot product similarity between any two nodes, and obtain the similarity from the nodes through normalization. Pointing to node Directional weights , represented as:

[0020] ,

[0021] Where N is the total number of nodes;

[0022] Based on directional weights Select distance node Recent A neighborhood is formed by a number of neighbors. The neighborhood is defined as the set of all directional weights. Sort the results by value from largest to smallest and select the highest score. The set consisting of the corresponding node indices, i.e.:

[0023] ,

[0024] in, Indicates a node Sort the directional weights pointing to all other nodes in descending order and select the top one. Operations on each element Indicates the first The index of the selected neighbor node;

[0025] Within the neighborhood, it is a node. arrive Directed edge construction explicit edge embedding , represented as:

[0026] ,

[0027] The full slice image is then represented as a directional dynamic graph in the instance-level feature space. ,in, A set of nodes; Let be a set of directed edges, defined as Each directed edge is associated with an explicit edge embedding; It is a set of biprojective mappings, i.e. ; Represents a set of mappings Embedded with each edge The high-dimensional relationship between them exists .

[0028] Furthermore, the steps of the spatially constrained attention mechanism include:

[0029] For each node and its neighborhood in the oriented dynamic graph The attention score for the fused edge embedding is calculated from the neighbors in the algorithm, using the following formula:

[0030] ,

[0031] in, express Activation function;

[0032] For the neighboring area The attention scores within the range are normalized to obtain the final attention weights. , represented as:

[0033] ,

[0034] in, Indicates the neighboring region All attention scores The operation of exponential normalization is as follows:

[0035] ,

[0036] Using attention weights Neighborhood The information of all image patches in the dataset is weighted and aggregated to obtain the first... Neighborhood information of each node , represented as:

[0037] ,

[0038] Combine neighborhood information with the original node vector The nodes are merged to update them, and the updated nodes are represented as follows:

[0039] ,

[0040] in, It is a non-linear activation function. This represents element-wise multiplication. It is a learnable linear mapping.

[0041] Furthermore, the model training and output steps include:

[0042] For each updated node, calculate the packet-level attention weight using the following formula:

[0043] ,

[0044] in, , For learnable parameters, The node represents the updated node representation, and... One-to-one correspondence;

[0045] Then, a slice-level representation based on graph-enhanced features is obtained through weighted summation. The formula is:

[0046] ,

[0047] Slice-level representation The mapping to slice-level class probabilities is expressed by the following formula:

[0048] ,

[0049] in, , These are the classification header parameters.

[0050] Furthermore, the total loss function for model training is expressed as:

[0051] ,

[0052] in, As a regulating factor, This indicates a regularization term for structural consistency or smoothing of a directional dynamic graph. The cross-entropy loss for slice-level classification tasks is expressed as:

[0053] ,

[0054] in, This represents the number of full-slice images in the training set. Indicates the number of categories. Indicates the first The package in the first Real labels on the class, Indicates by The obtained number The package belongs to the first The predicted probability of a class.

[0055] Beneficial effects: Compared with the prior art, the significant advantages of this invention are:

[0056] 1. This invention constructs a context-aware directional dynamic graph in the instance-level feature space of a full-slice image. By adaptively selecting semantically related neighborhood nodes for each image patch through bilinear projection and dot product similarity, it can explicitly model the contextual dependencies between distant organizational regions. Compared with schemes that only rely on fixed adjacency relationships or local convolution, it has a stronger expressive ability in recognizing cross-regional abnormal morphological patterns.

[0057] 2. This invention introduces a spatially constrained attention mechanism based on coordinate information on a directional dynamic graph. It modulates the influence of the neighborhood by utilizing the spatial distance, directional relationship and geometric consistency of triples between nodes. While preserving key long-range associations, it effectively suppresses false associations with unreasonable spatial positions or inconsistent geometric directions, thereby improving the robustness of the graph structure and attention distribution and reducing interference caused by factors such as coloring differences and scanning noise.

[0058] 3. This invention constructs explicit edge embedding on the edge, integrates directional weights with node projection features, and jointly models node features and edge features through a gating mechanism. When aggregating neighborhood information, it considers both the node's own features and the relationship features between nodes and their neighbors. Compared with aggregation methods based solely on node features, it can more precisely characterize complex tissue structure relationships such as high-risk areas – matrix areas, and high-risk areas – inflammatory response areas.

[0059] 4. This invention adopts a dual-path or dual-interactive node update structure, performs additive fusion and element-by-element interactive modeling on neighborhood information, and then combines them through nonlinear mapping. This not only preserves the overall intensity information, but also explicitly describes the interaction between different feature dimensions. Compared with a single linear aggregation method, it can obtain more discriminative and stable node representations.

[0060] 5. This invention combines graph-level representation with a multi-instance learning framework. Under weak supervision with only slice-level labels, it automatically mines and aggregates the most discriminative image block regions through a bag-level attention mechanism, avoiding dependence on pixel-level or block-level fine annotation, significantly reducing annotation costs, and improving the ability to focus on high-risk areas and key structural regions.

[0061] 6. This invention introduces graph structure consistency or smoothing regularization terms during training and iteratively updates adjacency weights using pseudo-labels predicted by the model, enabling coordinated optimization of graph structure and classification decision-making process. Compared with static graphs or fixed adjacency relationships, this invention can adaptively strengthen the connections between similar tissue regions and weaken the connections in noisy regions, thereby improving the accuracy and generalization performance of whole-slice image classification results.

[0062] 7. The attention weights and directional dynamic graph structures output by this invention can be used to construct visual heatmaps and key region sub-maps, providing interpretable visual evidence for the interpretation of whole-slice images, helping to discover potential morphological patterns and structural features, and have high application value in scenarios such as assisting in whole-slice image evaluation and scientific research analysis. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the entire process of a context-aware graph representation-based full-slice image analysis method with spatially constrained attention.

[0064] Figure 2 Flowchart for building context-aware graphs;

[0065] Figure 3 The process of spatially constrained attention mechanism. Detailed Implementation

[0066] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the embodiments of the present invention, and not all structures.

[0067] In the following description, specific details such as target system architecture and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0068] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0069] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0070] Furthermore, in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0071] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include the target features, structures, or characteristics described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0072] The flowchart of the whole-slice image analysis method based on spatial constraint attention and context awareness described in this embodiment is as follows: Figure 1 As shown, the method includes the following steps:

[0073] Step 1: Data preprocessing steps: Extract multiple image patches from the full-slice image, perform color normalization on the image patches, and construct an image patch set.

[0074] In one example, a sliding window and non-overlapping method are used to extract 256×256 pixel image patches from the full slice image, the full slice image is represented as a set of image patches, and color normalization is applied to the image patches.

[0075] The original whole-slice image is denoted as Weakly supervised multi-instance learning modeling is employed. The entire slice is treated as a bag, and image patches extracted from the slice through a sliding window are treated as instances. That is, the first The nth image patch is represented as the nth One example, .make If the number of instances is the same as the number of image patches, then the set of instances or the set of image patches is denoted as:

[0076] .

[0077] In one example, the datasets used included three publicly available datasets: CAMELYON16, TCGA-BRCA, and TCGA-NSCLC, totaling over 2430 whole-field-of-view (WSI) images. These images were digitized using different scanners. The CAMELYON16 dataset contained 400 WSIs annotated by pathologists to indicate breast cancer metastasis; the TCGA-BRCA dataset contained 977 WSIs annotated with breast cancer subtypes invasive ductal carcinoma and invasive lobular carcinoma; and the TCGA-NSCLC dataset contained 1053 WSIs annotated with non-small cell lung cancer subtypes adenocarcinoma and squamous cell carcinoma. All datasets used only slice-level labels. According to the experimental setup, the CAMELYON16 dataset was divided into a training set of 270 images and a test set of 130 images. The TCGA-BRCA and TCGA-NSCLC datasets used a five-fold Monte Carlo cross-validation split, with the training set comprising approximately 80%, and the validation and test sets each comprising approximately 10%. After data collection, foreground and background segmentation was performed on low-resolution HE-stained images. Foreground and background segmentation is achieved using traditional image processing algorithms, including thresholding, hole filling, dilation and erosion, and small object removal. First, a binarization threshold is calculated to obtain a binary image. Then, the image is inverted to brighten the foreground region, and closed small holes are removed. Next, dilation and erosion operations are used to eliminate unclosed holes, and newly generated small holes are removed again. Finally, useless areas outside the tissue, such as contaminant debris, are removed, thus achieving accurate foreground region extraction. Subsequently, image patches of a specified size are extracted from the foreground region for use in subsequent steps. Finally, a data queue index is established to index and queue the panoramic slice image, foreground and background segmentation mask, image patches, and corresponding labels to support efficient weakly supervised training and evaluation.

[0078] Step 2: Feature extraction step: Map each image patch in the image patch set to a corresponding feature vector, and use the feature vector as the initial graph node to form a node set.

[0079] In one example, a pre-trained visual model is used. For each image patch instance Independent encoding yields dimensions of eigenvectors These instance-level features are aggregated to form a feature representation set. As the initial graph node and its attributes, that is, one graph node and one feature vector correspond to one image, forming a node feature matrix X (N×D). Subsequent graph construction and attention aggregation are based on this set of nodes.

[0080] In one example, a ResNet50 model pre-trained on ImageNet-1K and a PLIP base model pre-trained on OpenPath are used as offline feature extractors to extract features from collected data, enhancing the feature extraction capabilities of pathological images. The extraction process is performed on a workstation equipped with an NVIDIA RTX 4060 GPU. After regularization, the input image patches are passed through the last layer of the network, outputting fixed-length feature vectors with feature dimensions of 1024 or 512 dimensions depending on the extractor type. The results are then normalized and stored, including image features, the starting pixel coordinates of the image patch, its size, relative coordinates, the source of the panoramic slice image, and the magnification, for easy loading and processing later. This offline extraction mechanism reduces computational overhead during training and preserves spatial information to support masking applications and cross-resolution consistency.

[0081] Step 3: Context-aware graph construction steps: Based on the node set, by calculating the association strength between nodes, neighboring nodes are dynamically selected for each node, and edge features are generated to construct a directional dynamic graph; wherein, the association strength is calculated by bilinear projection and dot product similarity.

[0082] Building upon instance-level features, we quantify the latent relationships between instances and construct a dynamic directed graph, thereby generating explicit embeddings for each edge in the graph, laying the foundation for subsequent context-aware learning. By explicitly formalizing the overall structure of the directed dynamic graph and providing a specific construction method for explicit edge embeddings, we represent the full-slice image as a graph data structure that can be used for subsequent spatially constrained attention and graph refinement.

[0083] Combination Figure 2 As shown, the context-aware graph construction steps further include:

[0084] The first Image blocks The eigenvectors are denoted as For eigenvectors Pre-encoding is performed using linear mapping to obtain node features. , represented as:

[0085] ,

[0086] in, It is a learnable weight matrix. It is a bias term. It is the input feature vector.

[0087] For the Node characteristics Semantic projection is performed using two independent linear transformations to obtain the query vectors. and key vector , represented as:

[0088] ,

[0089] ,

[0090] in, and The weight matrix is ​​a learnable matrix;

[0091] Calculate the dot product similarity between any two nodes, and then... Normalization yields the result from the node Pointing to node Directional weights , represented as:

[0092] ,

[0093] Where N is the total number of nodes;

[0094] To reduce the computational complexity of the graph and focus on key local relationships, directional weights are used. Select distance node Recent A neighborhood is formed by a number of neighbors. The neighborhood is defined as the set of all directional weights. Sort the results by value from largest to smallest and select the highest score. The set consisting of the corresponding node indices, i.e.:

[0095] ,

[0096] in, Indicates a node Sort the directional weights pointing to all other nodes in descending order and select the top one. Operations on each element Indicates the first The index of the selected neighbor node;

[0097] After determining the local neighborhood, nodes are defined within that neighborhood. arrive Directed edge construction explicit edge embedding This explicit edge embedding achieves its effect by fusing directional weights. Together with the biprojection features, a continuous and learnable edge representation is formed, which is expressed as:

[0098] ,

[0099] The full slice image is then represented as a directional dynamic graph in the instance-level feature space. ,in, For the set of nodes, and the set of instance-level feature representations Each feature vector in One-to-one correspondence; Let be a set of directed edges, defined as Each directed edge is associated with an explicit edge embedding; It is a set of biprojective mappings, i.e. ; Represents a set of mappings Embedded with each edge The high-dimensional relationship between them exists That is, through the mapping set and directional weights Jointly determine edge embedding The generation method.

[0100] After acquiring instance-level features, a context-aware directional dynamic graph representation is constructed to characterize the nonlocal interactions and directional information propagation between different regions in WSI. Specifically, each instance feature is pre-encoded to obtain node features. Then, through two linear mappings, a query vector (used for actively retrieving semantically similar neighborhoods) and a context vector (used to characterize the instance's influence on others and its passive reception capability) with functional decoupling are obtained. Subsequently, the dot product of the query-context pair is used as a similarity measure, and within each instance, all candidate neighbors are normalized to obtain directional weights, representing the association strength from the instance to its neighbors. To control complexity and suppress noisy edges, only... Similar neighbors form sparse neighborhoods, typically The weights are set between 5 and 20, and can be adjusted according to task complexity and memory budget. Based on the above weights and dual-projection embedding, an explicit edge embedding is generated for each directional edge, fusing self-information (query side) and other information (context side) to encode the strength and semantic attributes of directional adjacency in a continuous vector manner. To improve numerical stability, strategies such as temperature scaling, threshold truncation, and diagonal masking (removing self-loops) can be introduced during the graph construction stage, and extreme high / low similarity scores are pruned.

[0101] For each image patch / instance, the following key features can be generated simultaneously during the graph construction and weighting stages: the query / context vector after linear projection (used for subsequent attention scoring and edge fusion), the similarity score with its neighborhood (facilitating interpretability and parameter tuning diagnosis), and the oriented edge embedding vector (serving as input for spatially constrained attention aggregation and graph convolution). These features, together with the adjacency index, constitute the intermediate representation layer of the oriented dynamic graph, serving both message passing and representation optimization during training, and supporting visualization and retrieval during the inference stage. In practice, to control GPU memory usage, it is recommended to adopt a batch graph construction and sparse tensor caching strategy; in very large packets (with a large number of instances)... In scenarios with extremely high similarity, candidates can be screened first using low-cost metrics, and then rescored using precise similarity to construct the final neighborhood.

[0102] Step 4: Spatial Constraint Attention Mechanism Step 1: On the oriented dynamic graph, constraints based on spatial location information are introduced. The influence of the neighborhood is measured by triple relationships, which include the source node, the edge between the source node and its neighboring nodes, and the target neighboring node. Node features and edge features are fused to obtain attention weighting coefficients. The attention weighting coefficients are used to weight and aggregate neighborhood information and interact bidirectionally with the original node representation to update and optimize the node representation, thereby enhancing the long-range dependency modeling capability while suppressing the influence of noisy nodes.

[0103] Combination Figure 3 As shown, the spatially constrained attention mechanism further includes the following steps:

[0104] For each node and its neighborhood in the oriented dynamic graph The attention score for the fused edge embedding is calculated from the neighbors in the algorithm, using the following formula:

[0105] ,

[0106] in, express Activation function;

[0107] For the neighboring area The attention scores within the range are normalized to obtain the final attention weights. , represented as:

[0108] ,

[0109] in, Indicates the neighboring region All attention scores The operation of exponential normalization is as follows:

[0110] ,

[0111] Using attention weights Neighborhood The information of all image patches in the dataset is weighted and aggregated to obtain the first... Neighborhood information of each node , represented as:

[0112] ,

[0113] Combine neighborhood information with the original node vector To update nodes, a fusion process is employed, utilizing a dual-interaction mechanism to facilitate richer message exchange between nodes. This mechanism aims to simultaneously model both additive and interactive information fusion channels, thereby enhancing the information exchange capabilities between nodes. The updated node is represented as follows:

[0114] ,

[0115] in, It is a non-linear activation function (e.g., LeakyReLU). This represents element-wise multiplication. It is a learnable linear mapping.

[0116] Under weak supervision, using image patches as instances and WSI as packets, the directional dynamic graph is first fed into a graph convolutional network layer to smooth and nonlinearly transform the representations of nodes and edges, further uncovering potential geometric relationships. Then, instance-level clustering mapping is combined to perform prototyping constraints and structured aggregation on high-response instances, resulting in intermediate representations with less noise and stronger discriminative power. Subsequently, a spatially constrained attention mechanism (SCA) is introduced, using triples to score the influence on the neighborhood. The attention coefficient is obtained, the neighborhood context is weighted and aggregated, and the node representation is updated by dual-path fusion through residual addition and element-wise interaction.

[0117] Step 5: Model Training and Output Steps: Based on the optimized node representation, a graph convolutional network and instance-level clustering are combined to refine the graph representation feature space; under slice-level supervision, a cross-entropy loss function is adopted and introduced... Figure 1 The model is trained using a consistency regularization term. During training, the adjacency weights of the directional dynamic graph are updated based on the pseudo-labels generated by the model prediction to enhance graph structure learning, ultimately obtaining a classification model for full-slice image analysis.

[0118] Furthermore, the model training and output steps include:

[0119] For each updated node, calculate the packet-level attention weight using the following formula:

[0120] ,

[0121] in, , For learnable parameters, The node represents the updated node representation, and... One-to-one correspondence;

[0122] Then, a slice-level representation based on graph-enhanced features is obtained through weighted summation. The formula is:

[0123] ,

[0124] Slice-level representation The mapping to slice-level class probabilities is expressed by the following formula:

[0125] ,

[0126] in, , These are the classification header parameters.

[0127] Furthermore, the total loss function for model training is expressed as:

[0128] ,

[0129] in, As a regulating factor, This indicates a regularization term for structural consistency or smoothing of a directional dynamic graph. The cross-entropy loss for slice-level classification tasks is expressed as:

[0130] ,

[0131] in, This represents the number of full-slice images in the training set. Indicates the number of categories. Indicates the first The package in the first Real labels on the class, Indicates by The obtained number The package belongs to the first The predicted probability of a class.

[0132] The updated node input attention aggregation module generates a bag-level (slice-level) representation, and outputs the probability of each class through a multilayer perceptron (MLP) classification head with two hidden layers. In the main setting with only slice-level labels, slice-level cross-entropy is used as the main training loss; when there are a few instance-level or pseudo-labels, instance-level supervision is applied only to these labeled instances, and the remaining instances are implicitly optimized through attention and bag-level supervision.

[0133] To avoid overfitting and attention collapse, the Adam optimizer is used in the optimization process, with a learning rate set to [value missing]. and weight decay Model training employs batch processing and early stopping strategies to ensure convergence performance. Packet-level metrics are monitored on the validation set using early stopping; class imbalance can be addressed with class-weighted or sample-balanced approaches; temperature scaling and entropy constraints are introduced at the attention layer to improve distribution and stability. During inference, the same pre-training process is followed to complete feature extraction, dynamic graph construction, and attention aggregation, outputting slice-level prediction scores.

[0134] During the testing phase, the attention score for each image patch is output by the model to generate a heatmap for visualizing the discriminative regions. The final slice-level prediction is then obtained by combining the output of the bag-level classifier. The trained model is validated using a test set, and the evaluation includes performance comparisons with existing methods, ablation experiments, and interpretability analysis. First, the confusion matrix is ​​obtained by counting the predictions and true results on the test set, including true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Then, various metrics are calculated based on the confusion matrix. For accuracy... For F1 scores, there are For AUC, first, by changing the classification threshold, plot the ROC curve with 1-specificity as the horizontal axis and sensitivity as the vertical axis, and then calculate the area under the curve to get the AUC.

[0135] After model training is completed, performance evaluation is performed. To better verify the advantages of the method of this invention, we conduct comparative experiments with some existing methods. The specific comparison methods include: AB-MIL (Attention-based Deep Multiple Instance Learning), CLAM (Clustering-constrained Attention Multiple Instance Learning), DSMIL (Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning), TransMIL (Transformer-based Correlated Multiple Instance Learning for Whole Slide Image Classification), DTFD-MIL (Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification), and R²T-MIL (Re-embedded Regional Transformer (R²T) for Re-embedding Instance Features Online in Multiple Instances). This is compared with methods such as Learning (online re-embedding of instance features in regional Transformer models) and others.The experimental results are shown in Tables 1 and 2. Table 1 presents the diagnostic and subtype classification performance on the CAMELYON16 and TCGA-BRCA datasets. The performance of each model is given in the form of mean ± standard deviation (%), with the best results highlighted in bold and the second-best results underlined for intuitive comparison of the effects of different methods. Table 2 reports the subtype classification results of the model on the TCGA-NSCLC dataset, as well as the survival prediction performance on the BLCA, LUAD, and LUSC datasets. Again, the results are presented in the form of mean ± standard deviation (%), with the best and second-best results highlighted in bold and underlined respectively, to comprehensively evaluate the performance of the method in classification and prognosis prediction tasks. In terms of accuracy, AUC, and F1 score, the method of this invention improves upon CLAM by 4.13%, 4.10%, and 3.12% on the CAMELYON16 dataset, respectively, and by 3.75%, 2.29%, and 3.38% on the TCGA-BRCA dataset, respectively. In the survival prediction task, the C-index improved by 2.52% to 8.91%, indicating a significant improvement in predictive accuracy, which may better support decision-making and patient prognosis assessment.

[0136] Table 1. Diagnostic and subtype classification performance on the CAMELYON16 and TCGA-BRCA datasets.

[0137] Table 2 shows the model's cancer subtype classification results on the TCGA-NSCLC dataset and its survival prediction performance on the dataset.

[0138] Ablation experiments revealed that the model with context-aware graph representation significantly outperformed the no-graph strategy and the fixed k-node graph strategy, indicating that dynamic graph construction and spatially constrained attention played a key role in achieving higher performance.

[0139] Finally, this example also evaluates the interpretability of the method described in this invention, such as... Figure 3 The method described in this invention uses attention heatmaps to visualize and identify regions. Experiments have shown that the method can accurately locate relevant tumor morphological patterns, thus enhancing diagnostic reliability.

[0140] The context-aware full-slice image analysis method based on spatially constrained attention described in this invention mainly includes data preprocessing, feature extraction, context-aware graph construction, and a spatially constrained attention mechanism. The data preprocessing step extracts image patches from the full-slice image and performs color normalization to construct an image patch set. The feature extraction step maps the image patches to vector representations, serving as initial graph nodes. The context-aware graph construction step introduces a context-aware graph representation mechanism, dynamically characterizing the semantic and spatial relationships between regions through bilinear projection and dot product similarity, adaptively selecting neighborhoods based on similarity to form a sparse oriented graph and edge feature representations. The spatially constrained attention mechanism measures neighborhood influence using triplet relationships, fuses node and edge information to obtain weighting coefficients, aggregates neighborhoods, and interacts bidirectionally with the original node representation, optimizing node representations, promoting the cross-distance propagation of key information, and suppressing noise. Subsequently, graph convolutional networks and instance-level clustering are combined to further constrain and refine the graph representation feature space. Under slice-level supervision, cross-entropy is employed and introduced... Figure 1 The model is trained using consistency regularization, and the adjacency weights are updated based on pseudo-labels to enhance structure learning, resulting in a final model that can be used for full-slice image analysis using the method described in this invention.

Claims

1. A method for analyzing whole-slice images based on spatially constrained attention and context-awareness, characterized in that, Includes the following steps: Data preprocessing steps: Extract multiple image patches from the full-slice image, and perform color normalization on the image patches to construct an image patch set; Feature extraction steps: Map each image patch in the image patch set to a corresponding feature vector, and use the feature vector as an initial graph node to form a node set; Context-aware graph construction steps: Based on the node set, by calculating the association strength between nodes, neighboring nodes are dynamically selected for each node, and edge features are generated to construct a directional dynamic graph; wherein, the association strength is calculated by bilinear projection and dot product similarity; The spatially constrained attention mechanism involves the following steps: On the oriented dynamic graph, constraints based on spatial location information are introduced, and the neighborhood influence is measured by triplet relationships, which include the source node, the edge between the source node and its neighboring nodes, and the target neighboring node; node features and edge features are fused to obtain attention weighting coefficients; the neighborhood information is weighted and aggregated using the attention weighting coefficients, and bidirectional interaction is performed with the original node representation to update and optimize the node representation; Model training and output steps: Based on the optimized node representation, a graph convolutional network and instance-level clustering are combined; under slice-level supervision, the cross-entropy loss function is used and a graph consistency regularization term is introduced for model training; during training, the adjacency weights of the directional dynamic graph are updated based on the pseudo-labels generated by the model prediction, and finally a classification model for full-slice image analysis is obtained.

2. The whole-slice image analysis method based on spatially constrained attention and context awareness according to claim 1, characterized in that, The steps for constructing a context-aware graph include: The first Image blocks The eigenvectors are denoted as For eigenvectors Through linear mapping Perform pre-encoding to obtain node features , represented as: ， For the Node characteristics Semantic projection is performed using two independent linear transformations to obtain the query vectors. and key vector , represented as: ， ， in, and The weight matrix is ​​a learnable matrix; Calculate the dot product similarity between any two nodes, and obtain the similarity from the nodes through normalization. Pointing to node Directional weights , represented as: ， Where N is the total number of nodes; Based on directional weights Select distance node Recent A neighborhood is formed by a number of neighbors. The neighborhood is defined as the set of all directional weights. Sort the results by value from largest to smallest and select the highest score. The set consisting of the corresponding node indices, i.e.: ， in, Indicates a node Sort the directional weights pointing to all other nodes in descending order and select the top one. Operations on each element Indicates the first The index of the selected neighbor node; Within the neighborhood, it is a node. arrive Directed edge construction explicit edge embedding , represented as: ， The full slice image is then represented as a directional dynamic graph in the instance-level feature space. ,in, A set of nodes; Let be a set of directed edges, defined as Each directed edge is associated with an explicit edge embedding; It is a set of biprojective mappings, i.e. ; Represents a set of mappings Embedded with each edge The high-dimensional relationship between them exists .

3. The whole-slice image analysis method based on spatially constrained attention and context awareness according to claim 2, characterized in that, The steps of the spatially constrained attention mechanism include: For each node and its neighborhood in the oriented dynamic graph The attention score for the fused edge embedding is calculated from the neighbors in the algorithm, using the following formula: ， in, express Activation function; For the neighboring area The attention scores within the range are normalized to obtain the final attention weights. , represented as: ， in, Indicates the neighboring region All attention scores The operation of exponential normalization is as follows: ， Using attention weights Neighborhood The information of all image patches in the dataset is weighted and aggregated to obtain the first... Neighborhood information of each node , represented as: ， Combine neighborhood information with the original node vector The nodes are merged to update them, and the updated nodes are represented as follows: ， in, It is a non-linear activation function. This represents element-wise multiplication. It is a learnable linear mapping.

4. The whole-slice image analysis method based on spatially constrained attention and context awareness according to claim 3, characterized in that, The model training and output steps include: For each updated node, calculate the packet-level attention weight using the following formula: ， in, , For learnable parameters, The node represents the updated node representation, and... One-to-one correspondence; Then, a slice-level representation based on graph-enhanced features is obtained through weighted summation. The formula is: ， Slice-level representation The mapping to slice-level class probabilities is expressed by the following formula: ， in, , These are the classification header parameters.

5. The whole-slice image analysis method based on spatially constrained attention and context awareness according to claim 4, characterized in that, The total loss function for model training is expressed as: ， in, As a regulating factor, This indicates a regularization term for structural consistency or smoothing of a directional dynamic graph. The cross-entropy loss for slice-level classification tasks is expressed as: ， in, This represents the number of full-slice images in the training set. Indicates the number of categories. Indicates the first The package in the first Real labels on the class, Indicates by The obtained number The package belongs to the first The predicted probability of a class.

Images