Tumor full-view digital slice classification method based on priority scanning multi-instance learning

Abstract

The invention provides a tumor full-view digital slice classification method based on priority scanning multi-instance learning, and belongs to the field of computational pathology. In order to solve the problem that an existing method neglects a full-slice image space structure and cross-regional similarity, a double-branch SSM architecture is designed: a space branch retains original two-dimensional position information through bit interleaving coding, and a cross-space branch captures similar instance features based on significance scores; and the lightweight feature fusion module is used for integrating double-path features, so that the classification, diagnosis and survival prediction precision of cancer subtypes is improved. Finally, the effectiveness of the proposed system is verified through multiple groups of contrast experiments and ablation experiments.

Description

TECHNICAL FIELD

[0001] The application provides a tumor full-field digital slice classification method based on priority scanning multiple instance learning, and belongs to the technical field of image data processing (G06T) . BACKGROUND

[0002] In the field of computational pathology, the analysis of whole slide images (WSIs) constitutes the core foundation, aiming to assist pathologists in completing key tasks such as cancer diagnosis, subtype classification, and prognosis prediction through automated methods. However, the inherent terapixel-level resolution of WSIs (each image can reach tens of billions of pixels) poses significant challenges to efficient labeling and model training. Traditional diagnostic procedures rely on pathologists manually analyzing WSIs, which is time-consuming, labor-intensive, and costly, and may lead to diagnostic bias due to subjective factors. To address these limitations, the multiple instance learning (MIL) paradigm is widely introduced, treating WSIs as "bags" containing a large number of image blocks (i.e., "instances"), and only requiring weakly supervised labeling at the slide level to train the model. Under this framework, the model predicts the label of the entire WSI by aggregating instance-level features, avoiding the heavy demand for pixel-level labeling. Current mainstream methods can be divided into two categories: instance-level methods generate pseudo-labels through sliding windows and aggregate instances with high malignancy probability, but their clinical applicability is limited by the number of WSIs and their insufficient subtype differentiation ability; embedded-level methods use pre-trained visual models (such as ResNet) to extract block features, aggregate features through learnable attention mechanisms, and then classify them through multi-layer perceptron, becoming a more robust solution.

[0003] Although MIL methods have made significant progress in computational pathology tasks, their performance still faces a bottleneck. The root cause lies in the fact that existing methods mostly assume instance independent and identically distributed, ignoring the complex spatial structure information of pathological tissues in WSIs. The tissues in WSIs usually present irregular distribution (such as scattered morphology, local holes), while traditional methods often use convolution technology based on approximate spatial arrangement to derive the positional relationship between blocks, which is only effective when the tissues present highly regular shape (such as square). In fact, pathologists need to consider the local area environment and the correlation between different areas in diagnosis, and the lack of spatial information makes it difficult for the model to capture key morphological associations. Although some studies try to introduce position embedding technology (such as position encoding in Transformer), its quadratic computational complexity makes it difficult to handle the ultra-long sequence of WSIs (usually containing thousands to tens of thousands of instances). The recently proposed structured state space model (SSM) provides a new way for MIL by modeling long sequence dependencies with linear complexity. However, existing SSM-based methods have obvious limitations: first, their sequence reordering relies on heuristic strategies controlled by hyperparameters or fixed relative position matrices, which cannot fully utilize the original two-dimensional spatial information; second, they ignore the feature association of similar instances across space (such as the relationship between blocks scattered in space but belonging to the same tumor); third, they are not adaptable to irregular tissue morphology, resulting in low utilization of spatial features.

[0004] The root cause lies in the fact that when highly correlated instances (such as two tumor blocks) are adjacent in the sequence, the state transfer of SSM can maintain the continuity of features; if irrelevant instances (such as normal tissues) are inserted between them, the contribution of key features will undergo additional attenuation and introduce noise disturbance, reducing the model's sensitivity to discriminative features. Therefore, how to effectively integrate the original spatial layout of WSIs and the cross-spatial similarity relationship without increasing computational burden has become a key challenge to improve the performance of computational pathology models.

[0005] In summary, the current MIL framework has three shortcomings in the utilization of spatial information:

[0006] (1) Weak spatial structure modeling: existing reordering strategies destroy the original two-dimensional topological relationship, which cannot adapt to the irregularity of pathological tissues;

[0007] (2) Lack of cross-spatial association: ignoring the semantic association between instances with similar features but scattered in space;

[0008] (3) Imbalance between computational efficiency and accuracy: Transformer-based models have high computational overhead, while lightweight SSM methods fail to effectively integrate multi-dimensional information.

[0009] These issues make it difficult for existing methods to meet clinical needs in terms of accuracy and robustness in tasks such as cancer subtyping and prognosis prediction. There is an urgent need for an efficient framework that can simultaneously encode spatial location and cross-spatial feature similarity. Summary of the Invention

[0010] The purpose of this invention is to address the problem of insufficient utilization of spatial information from whole-slice images (WSI) in multi-instance learning frameworks in computational pathology, and to propose an efficient prediction model that integrates spatial location and cross-spatial similarity. The aim is to design a bi-branch structured state-space model that simultaneously captures the original spatial topology of WSI and cross-regional similarity instance features through a priority scanning mechanism, significantly improving the accuracy of cancer subtyping, diagnosis, and prognosis prediction while maintaining linear computational complexity.

[0011] This invention proposes a Priority Scanning Multi-Instance Learning Framework (PSMIL), which includes spatial branching, cross-spatial branching, and a lightweight fusion module. The spatial branch uses bit-interleaved encoding to convert the original two-dimensional coordinates into a spatial sequence, and organizes spatial distribution features through a Structured State Space Model (SSM). The cross-spatial branch rearranges the instance sequence using a saliency scoring module (one-dimensional convolution + ReLU), enabling the SSM to focus on feature associations between similar instances across spaces. Finally, the feature fusion module (layer normalization + adaptive weight allocation) integrates the dual-path features to generate a spatially aware, slideshow-like representation.

[0012] This invention employs the aforementioned model. First, the data samples are preprocessed. In terms of spatial feature extraction, the coordinates are bit-interleaved and encoded to generate a spatial sequence. The spatial features are obtained by processing the sequence through SSM blocks. For cross-spatial feature extraction, the instance weights are calculated using a saliency scoring module and then rearranged in descending order. The similar instance associations are captured by processing the rearranged sequence through SSM blocks. Finally, the dual-branch outputs are layer-normalized and weighted summation is performed to obtain comprehensive features and prediction results.

[0013] This invention includes the following steps:

[0014] Step 1: Construct training data:

[0015] 1. Obtain multiple whole-slice pathological images (WSI), including labeled disease classification tags (such as cancer subtype, diagnosis result) as the true labels of the samples;

[0016] 2. Preprocess WSI: Identify and remove background areas while retaining tissue areas to reduce computational burden; divide tissue areas into 512×512 pixel non-overlapping image patches, using a preset step size to ensure no overlap between patches;

[0017] 3. Use a pre-trained feature extraction model to convert each image patch into a feature embedding vector, generating an instance feature sequence U = [u1, u2, ..., u...]. L ] and its corresponding two-dimensional position matrix, where L is the total number of image patches after a whole slice image (WSI) is segmented.

[0018] Step 2, Spatial Branch Modeling:

[0019] 1. Encode the coordinates (x, y) of each segmented image block: using a bit-interleaved coding method. Generate a spatial location index M(x,y); where (x,y) are the coordinates of the image patch, and k = max{log2(x+1),log2(y+1)} is the maximum number of bits required.

[0020] 2. Reorder the instance feature sequences according to the spatial location index M(x,y) to form an ordered sequence containing potential spatial information;

[0021] 3. Input the reordered sequence into a structured state-space model (SSM), model long sequence dependencies using a linear time-invariant system, and extract spatial features:

[0022]

[0023] y t =Ch t +Du t

[0024] Where h t In the hidden state, u t The input features are A, B, C, and D, which are weight parameters, and Δ is the weight parameter. t Let y be the time step, and t be the time step index, i.e., the position of the t-th instance in the sequence, t∈{1,2,...,L}, where L is the total number of image patches after a full slice image is segmented. This is achieved by outputting y... t Aggregation yields spatial feature vectors:

[0025] f spatial =mean_pooling([y1,y2,...,y t ]);

[0026] Step 3, cross-spatial branch modeling:

[0027] 1. Input the original feature sequence U into a one-dimensional convolutional layer, and generate a saliency score s for each instance using the ReLU activation function. i ,s i This represents the pathological significance score of the i-th image patch.

[0028] 2. Press s i The instance sequences are rearranged in descending order to cluster similar pathological features (such as tumor regions).

[0029] 3. The reordered sequence is input into an independent SSM module, which captures the association features of similar instances across space and outputs a feature vector f. cross-spatial

[0030] Step 4: Dual-branch feature fusion and prediction:

[0031] 1. Regarding f spatial and f cross-spatial Perform layer normalization separately;

[0032] 2. Calculate the fusion weights α1 and α2 using a linear layer and the Tanh activation function. i =Tanh(W i f i +b i ), i∈{0,1}, f i Input features for the branch: i = 0 represents the feature f of the spatial branch. spatial i=1 represents the feature f across spatial branches. cross-spatial W i For the learnable weight matrix, b i This is a bias term.

[0033] 3. Then, the weights are normalized using Softmax. and :

[0034] 4. Weighted fusion of features yields the fused feature vector f. fused :

[0035] 5. Input the fused features into a multilayer perceptron (MLP) and output the predicted probability: For fusion feature f fused The output of the multilayer perceptron (MLP) represents the probability distribution of the predicted category. In cancer subclass classification, It indicates the specific subtype and can be used to guide the selection of treatment plans.

[0036] This invention first preprocesses the data samples. In terms of spatial feature extraction, the coordinates are bit-interleaved and encoded to generate a spatial sequence. The spatial features are obtained by processing the sequence through SSM blocks. For cross-spatial feature extraction, the saliency scoring module is used to calculate the instance weights and rearrange them in descending order. The SSM block processing rearranges the sequence to capture the association of similar instances. Finally, the dual-branch output is layer-normalized and weighted to obtain the comprehensive features and the prediction result. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0038] Figure 2 This is a structural diagram of a tumor full-field digital slice classification method based on priority scan multi-instance learning.

[0039] Figure 3 The comparison results of the model in cancer subtype classification, cancer diagnosis and survival prediction; Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings.

[0041] See Figure 1 The specific processing steps of this invention are as follows:

[0042] 1. Data preprocessing:

[0043] All datasets used in this project are publicly available datasets. The CAMELYON-16 dataset (… https: / / camelyon17.grand-challenge.org / Data / This dataset is a widely used benchmark dataset for detecting lymph node metastases in breast cancer in the field of computational pathology. It contains 400 H&E-stained whole-slice digital pathology images, with 270 training images featuring precise pixel-level tumor region annotations and 130 test images without publicly available annotations to maintain objectivity in the evaluation. These images are derived from sentinel lymph node biopsy samples, encompassing both normal and metastatic breast cancer tissue, and possess extremely high resolution (approximately 300,000 × 150,000 pixels, 40x magnification). In addition to providing high-quality WSI images, the dataset also includes lesion region contour annotations in XML format, offering crucial support for the development of computer-aided diagnostic algorithms.

[0044] BRACS ( https: / / bracs.grand-challenge.org / The BRACS dataset is specifically designed for fine-grained classification of breast cancer subtypes. It contains a large collection of H&E-stained breast cancer tissue slice images, covering multiple breast cancer subtypes, including ductal carcinoma and lobular carcinoma. Each image has been meticulously annotated by professional pathologists, recording key pathological features of the lesion area. The BRACS dataset is designed to support the performance optimization of deep learning models in distinguishing between benign and malignant lesions and in more refined subtype classification tasks. Research shows that models trained on this dataset achieve excellent performance in both 3-class and 7-class subtype classification tasks.

[0045] TCGA-NSCLC ( https: / / xenabrowser.net / datapages / ?cohort=TCGA%20Lung% 20CancerThe TCGA-NSCLC dataset is an important resource in the field of lung cancer research. This dataset integrates whole-slice digital pathology images and comprehensive genomic data, with a particular focus on the two major non-small cell lung cancer subtypes: lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC). The WSI images in the dataset are of high resolution and accompanied by detailed clinical survival data and molecular feature annotations, supporting researchers in conducting multimodal analyses such as histological morphology and gene mutation associations. Due to its large scale and rich case diversity, TCGA-NSCLC has become an important benchmark for evaluating the generalization performance of algorithms.

[0046] TCGA-LUAD( https: / / portal.gdc.cancer.gov / projects / TCGA-LUAD This dataset focuses on prognostic assessment and treatment response prediction for patients with lung adenocarcinoma. It not only contains high-quality whole-slice digital pathology images but also provides comprehensive clinical follow-up data and genomic information, such as the mutation status of key genes like TP53. Researchers frequently utilize this dataset to build prognostic prediction models, explore the relationship between immune cell infiltration characteristics and prognosis, or develop algorithms for predicting chemosensitivity.

[0047] TCGA-LUSC ( https: / / portal.gdc.cancer.gov / projects / TCGA-LUSC This dataset is primarily used for survival prediction and biomarker discovery research in squamous cell lung cancer. It provides comprehensive WSI images, methylation data, and clinical information, enabling researchers to delve into specific biomarkers for squamous cell lung cancer. For example, one study successfully identified six diagnostically valuable methylation sites using this dataset; the combined diagnostic sensitivity of these sites reached an excellent level of 93%-99%. The dataset also supports cutting-edge research directions such as immunotherapy response prediction.

[0048] Next, disease classification labels (such as cancer subtypes and diagnostic results) are labeled. Taking advantage of WSI's gigapixel-level resolution, key preprocessing operations are performed: a threshold segmentation algorithm automatically identifies and removes background regions lacking tissue information, significantly reducing computational burden. The retained pathological tissue regions are precisely segmented into 512×512 pixel non-overlapping image patches, with a preset stride to ensure no overlap between patches. Subsequently, a pre-trained deep model (ResNet50 or CONCH) is used to extract the depth feature vector of each image patch, generating an instance feature sequence, while simultaneously recording the original two-dimensional spatial coordinates of each patch.

[0049] 2. Build a tumor full-view digital slide classification model based on priority scan multi-instance learning:

[0050] like Figure 2 The diagram illustrates the construction of a dual-branch model. The spatial branch preserves the two-dimensional topology of WSI through positional encoding and reordering techniques: utilizing a bit-interleaving algorithm. A one-dimensional spatial encoding value M(x,y) is generated, where (x,y) are the two-dimensional coordinates of the image patch, M(x,y) is the one-dimensional spatial encoding value generated based on the two-dimensional coordinates, and k = max{log2(x+1), log2(y+1) represents the maximum number of bits required. This encoding maps the two-dimensional spatial proximity to the one-dimensional sequence space, rearranging the instance feature sequence in ascending order according to M(x,y) to ensure that adjacent pathological tissue patches in the original image are continuously distributed in the sequence. The rearranged sequence is input into a structured state-space model (SSM), which models long-range dependencies through a linear time-invariant system. Its discretized state transition equation is expressed as:

[0051] y t =Ch t +Du t , where h t In the hidden state, u t The input features are A, B, C, and D, which are weight parameters, and Δ is the weight parameter. t Let y be the time step, and t be the time step index, i.e., the position of the t-th instance in the sequence, t∈{1,2,...,L}, where L is the total number of image patches after a full slice image is segmented. This is achieved by outputting y... t Aggregation yields spatial feature vectors: f spatial =mean_pooling([y1,y2,...,y t ]);

[0052] The cross-spatial branch focuses on extracting cross-regional similarity features to obtain spatial feature vectors f. cross-spatial The original feature sequence U is input into a one-dimensional convolutional layer, which generates a saliency score s. i s i The pathological significance score of the i-th image patch is represented by the sequence rearranged in descending order of score to cluster similar pathological features (such as high-probability malignant regions), and then cross-spatial features f are extracted by an independent SSM module. cross-spatial .

[0053] 3. Feature fusion and classification prediction:

[0054] A dynamic weighting mechanism is used to integrate the dual-branch outputs. First, the spatial feature f... spatial With cross-space features f cross-spatial After performing layer normalization, the initial fusion weights α1 and α2 are calculated using a linear transformation layer and the hyperbolic tangent (Tanh) activation function. The final weight assignment is then obtained after normalization using the Softmax function. and Through spatial features f spatial With cross-space features f cross-spatial The fusion yields the fused feature vector f fused The weighted fusion formula is expressed as follows: The fused feature vectors are input into a multilayer perceptron (MLP) to generate sliding-level disease prediction probabilities. In cancer detection and subclassification tasks, It indicates the specific cancer type or subtype, guiding the selection of treatment options.

[0055] 4. Model training and optimization:

[0056] Implemented on an NVIDIA RTX-3090 GPU hardware platform. WSI-level batch processing (batch size = 1) is used to adapt to long sequence characteristics, the Adam algorithm is selected as the optimizer, and the base learning rate is set to 2×10⁻⁶. -4 It is dynamically adjusted in conjunction with a cosine annealing strategy. The loss function is selected based on the task type, using cross-entropy loss, and the regularization is set with a weight decay coefficient of 10. -4 To prevent overfitting, the training process strictly follows the five-fold cross-validation protocol: the classification task dataset is divided into training set (80%), validation set (10%), and test set (10%). During each iteration of training, forward propagation is performed to calculate the loss, backpropagation is performed to update the SSM parameters, and an early stopping strategy (tolerance for 20 rounds) is adopted to save the model with the best performance on the validation set.

[0057] 5. Performance Verification and Visualization Analysis

[0058] The effectiveness of the model is verified through both quantitative metrics and visual presentation. For example... Figure 3 As shown, in the cancer subtype classification task, PSMIL achieved an AUC of 83.28% (standard deviation ± 4.23%) on the BRACS dataset, improving upon the second-best method, MambaMIL, by 5.13 percentage points; on the TCGA-NSCLC dataset, it reached an AUC of 94.70% (± 1.71%), outperforming the baseline model by 2.63 percentage points. In the survival prediction task, the C-index on the TCGA-LUAD dataset reached 64.91% (± 6.35%), and on the TCGA-LUSC dataset, it reached 61.50% (± 3.54%), both significantly outperforming traditional methods. Ablation experiments further confirmed the synergistic value of the two-branch approach: the AUC on the TCGA-NSCLC dataset was 93.28% when using the original sequence alone; the AUC slightly increased to 93.31% when only spatial branching was enabled; the AUC improved to 93.84% when only cross-spatial branching was enabled; and the AUC jumped to 94.70% after fusion of the two branches, demonstrating the crucial impact of feature complementarity on performance.

[0059] 6. Technological advantages and clinical application value

[0060] The system's technological advantages and clinical application value are reflected in three innovative aspects: First, spatial branching preserves the original two-dimensional spatial relationships through bit-interleaved encoding, overcoming the spatial information loss problem caused by traditional x-axis sorting; second, cross-spatial branching rearranges sequences based on saliency scores, enabling similar pathological features to be processed continuously in the model and reducing the interference of irrelevant instances on state transitions; third, a lightweight dynamic fusion module adaptively integrates heterogeneous features, improving the model's robustness in discrimination. Experiments show that PSMIL comprehensively outperforms nine mainstream MIL methods in five major datasets (CAMELYON-16, BRACS, TCGA-NSCLC, etc.) and diagnostic tasks, with the highest relative performance improvement reaching 14.22% (CAMELYON-16 dataset). In terms of computational efficiency, SSM's linear complexity can efficiently handle long sequences with thousands of instances, demonstrating a significant engineering advantage over the quadratic complexity of Transformer.

[0061] This invention is an information processing method implemented entirely by a computer.

[0062] The above description is merely a specific embodiment of the present invention. Any feature disclosed in this specification may be replaced by other equivalent or similar features unless otherwise specified. All disclosed features, or steps in all methods or processes, may be combined in any way except for mutually exclusive features and / or steps.

Claims

1. A tumor full-field digital slide classification method based on priority scan multi-instance learning, characterized in that, Includes the following steps: Step S1: Construct training data: S1.1 Crops the image patch corresponding to the pathological tissue region from the whole slice image (WSI) and discards the background region to reduce computational complexity; S1.2 records the two-dimensional spatial coordinates of each image patch, forming a position matrix P = {(x1,y1),(x2,y2),...,(x...} L ,y L )}, where (x i ,y i ) represents integer index coordinates, and L is the total number of image patches after a whole slice image (WSI) is segmented; S2.3 uses a pre-trained feature extraction model to convert each image patch into a feature embedding vector, generating an instance feature sequence U = [u1, u2, ..., u...]. L ] and its corresponding two-dimensional position matrix, where L is the total number of image patches after a whole slice image (WSI) is segmented; Step S2, Dual-branch feature extraction: S2.1 Spatial branching process: Based on the position matrix, a spatial position index is generated using a bit interleaving coding method; The instance feature sequence is reordered according to the spatial location index to form an ordered sequence containing potential spatial information; the ordered sequence is processed using stacked structured state space model (SSM) blocks to extract spatial features; S2.2 Cross-spatial branching: The instance feature sequence is input into the saliency scoring module, which contains a one-dimensional convolutional layer and a ReLU activation function, and outputs a saliency score for each instance; the instance feature sequence is reordered in descending order according to the saliency score; the reordered sequence is processed using stacked SSM blocks to extract cross-spatial similarity instance features; Step S3, Feature Fusion and Prediction: S3.1 The dual-branch features are integrated through a lightweight feature fusion module to obtain fused features. The lightweight feature fusion module sequentially performs layer normalization, linear layer + Tanh activation to generate fusion factors, and Softmax to calculate weights and perform weighted summation. S3.2 inputs the fused features into the multilayer perceptron (MLP) and outputs the classification and prediction results of the whole slice image.

2. The method as described in claim 1, characterized in that, The pre-trained feature extraction model is a ResNet50 or CONCH model, the image patch size is 512*512 pixels, and the magnification is 20.

3. The method according to claim 1, characterized in that, The bit interleaving coding method is implemented using the following formula: Where (x,y) are the two-dimensional coordinates of the image patch, M(x,y) is the one-dimensional spatial encoding value generated based on the two-dimensional coordinates, and k = max{log2(x+1),log2(y+1) represents the maximum number of bits required.

4. The method as described in claim 1, characterized in that, The saliency scoring module includes a one-dimensional convolutional layer and a ReLU activation function, which realizes priority modeling of similar instances across space by calculating the importance scores of instance features.

5. The method as described in claim 1, characterized in that, The lightweight feature fusion module performs the following operations: performs layer normalization on the output features of spatial branches and cross-spatial branches respectively; and generates a fusion factor through a linear layer and a hyperbolic tangent Tanh activation function. The fusion weights are calculated using the Softmax function; the normalized features are multiplied by their corresponding weights and then summed to obtain the fusion features.

6. The method according to claim 1, characterized in that, The discretization calculation of the SSM block is based on the following single-step recursive relationship: y t =Ch t +You t Where h t In the hidden state, u t The input features are A, B, C, and D, which are weight parameters, and Δ is the weight parameter. t Let t be the time step, and t be the time step index, which is the position of the t-th instance in the sequence, t∈{1,2,...,L}, where L is the total number of image patches after a full slice image is segmented.

7. The method according to claim 1, characterized in that, The method is applicable to the following computational pathology tasks: Cancer subtype classification; cancer diagnosis; survival prediction.

8. An information processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1-7.

Images