Hierarchical MIL full-slice image classification method based on pseudo-bag hybrid enhancement
Through the multi-level multi-instance learning architecture and pseudo-bag hybrid enhancement strategy, the problems of computational complexity and insufficient model generalization ability of large-scale full-slice image data are solved, and efficient WSI classification effect is achieved.
Patent Information
- Application Number
- CN202510633024.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-16
- Publication Date
- 2025-09-26
AI Technical Summary
Existing multi-instance learning methods are difficult to effectively process large-scale whole-slice image data, have high computational complexity and insufficient model generalization ability, and directly applying the Mixup data enhancement strategy is difficult in WSI classification.
A multi-level multi-instance learning architecture and pseudo-bag hybrid enhancement strategy are adopted to generate pseudo-bags through phenotype clustering. Combined with random mixing mechanism and multi-head self-attention calculation, mixed pseudo-bags are generated and down-sampled, which reduces computational complexity and improves model generalization ability.
The accuracy and AUC index of full-slice image classification were significantly improved, the computational complexity problem in large-scale WSI data processing was solved, and the generalization ability of the model was enhanced.
Smart Images
Figure CN120707920A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical image processing, and in particular to a full-slice image classification method, and more particularly to a multi-level multi-instance learning full-slice image classification method based on pseudo-bag hybrid enhancement. Background Art
[0002] Whole slide images (WSIs), a crucial data type for computational pathology, possess ultra-high resolution, often containing billions of pixels per image. They are the gold standard for digital pathology diagnosis. In recent years, with the advancement of deep learning techniques, automated analysis of whole slide images has become a research hotspot in medical image processing. However, due to the large size of WSIs (e.g., 40,000 × 40,000 pixels) and the lack of region-level annotations, traditional deep learning methods are difficult to directly apply to WSI analysis and classification.
[0003] Multiple Instance Learning (MIL), as a weakly supervised learning framework, can effectively address the problem of missing region-level annotations and has become a mainstream method for WSI classification. In the MIL framework, each WSI is regarded as a "bag" containing multiple "instances" (i.e., organizational blocks or patches). By aggregating the feature representations of multiple instances, the MIL model is able to make bag-level predictions. However, existing MIL methods face two major challenges: (1) single-layer MIL architectures cannot effectively process WSI data containing a large number of instances; (2) the inherent data memorization tendency of neural networks limits the generalization ability of the model.
[0004] In recent years, Transformer-based models have achieved remarkable success in various visual tasks with their powerful self-attention mechanism. However, they are limited by the computational complexity of the self-attention mechanism on the sequence length (O(n 2 Directly applying the Transformer to WSI analysis involving a large number of instances faces significant challenges due to its high complexity. Furthermore, improving the model's generalization capabilities is also a key research issue.
[0005] Mixup is an effective data augmentation method that has been widely used in tasks such as natural image classification. Its basic idea is to generate new training samples through linear interpolation of two input samples, thereby expanding the learning boundary of the model. However, due to the particularity of WSI data (irregular bag representation, inconsistent number of instances, etc.), directly applying Mixup to WSI classification faces many difficulties.
[0006] Therefore, how to combine a multi-level architecture to reduce computational complexity and improve the model generalization ability through data enhancement strategies suitable for WSI is a key issue that needs to be urgently addressed in current WSI classification research. Summary of the Invention
[0007] To address the challenges of existing techniques, this paper proposes a multi-level multi-instance learning method for full-slide image classification based on pseudo-bag hybrid augmentation. By innovatively combining a multi-level multi-instance learning architecture with a pseudo-bag hybrid data augmentation strategy, this method effectively addresses the computational complexity of large-scale WSI processing while significantly improving the model's generalization capabilities.
[0008] Specifically, the method of the present invention comprises the following steps:
[0009] S1: Segment the whole-slice pathology image to generate multiple instance-level image blocks and perform feature extraction;
[0010] S2: using a phenotypic clustering algorithm to perform feature clustering on the instance, and dividing the instance into multiple semantically preserved pseudo bags through a stratified sampling strategy;
[0011] S3: Randomly select instances from two pseudo bags, perform linear interpolation and masking operations according to a preset mixing ratio, and generate a mixed camouflage pseudo bag and a masked pseudo bag;
[0012] S4: taking the mixed disguised pseudo-bag as the initial sub-bag, performing downsampling in the sub-bag to obtain a secondary sub-bag, and appending a learnable MSG token under the secondary sub-bag;
[0013] S5: Perform multi-head self-attention calculation inside the secondary sub-bag to capture the spatial dependencies between instances;
[0014] S6: Collect the MSG token features of each secondary sub-bag and construct a high-level pathology representation by feature concatenation;
[0015] S7: Embed CLS tokens in the high-level pathology representation and give classification prediction results.
[0016] Furthermore, in step 1, the feature extractor adopts a pre-trained ResNet-50 model with a feature dimension d=1024.
[0017] Furthermore, in step 2, the phenotype clustering algorithm includes:
[0018] a. Compute the prototype representation of the WSI bag;
[0019] where m i represents the number of samples in the i-th phenotypic class, ∑ j x i,jRepresents all samples x belonging to the i-th phenotypic class i,j Sum, subscript j traverses every sample in the category, x i,j represents the jth sample in the i-th category, p i It represents the center value (mean) of the i-th category, which is obtained by calculating the average value of all samples in the category and is used to represent the "center position" of the category.
[0020] b. Calculate the cosine similarity between each instance and the bag prototype: s i,j ←cos(p i ,x i,j )
[0021] Where cos(·) represents the cosine similarity function, which is used to calculate the cosine value of the angle between two vectors. The formula is: p i represents the central value (mean) of the i-th category, x i,j represents the jth sample in the i-th category.
[0022] c. Assign instances to l initial phenotypic clusters based on similarity;
[0023] d. Perform k iterations of refinement to optimize the phenotypic clusters.
[0024] Furthermore, the stratified sampling strategy in step 2 specifically includes:
[0025] a. Evenly divide the l feature clusters generated by phenotypic clustering;
[0026] b. Extract instances proportionally from each feature cluster to ensure that each pseudo-bag contains instances of all phenotypic categories;
[0027] c. Maintain the consistency of pathological semantic distribution among pseudo bags by dynamically adjusting the sampling ratio.
[0028] Furthermore, in step 3, the random mixing mechanism includes:
[0029] a. Merge the retained pseudo bags with probability p to generate a mixed pseudo bag:
[0030] in represents the generated mixed bag, M λ ∈{0,1} n is a binary mask with a length of n (the number of pseudo bags), and each element has a value of 0 or 1. The number of 1s in the mask is controlled by λ~Beta(α,α) and satisfies ⊙ represents the element-by-element multiplication operation, which actually means selecting the pseudo bag according to the mask, and represents the pseudo bag corresponding to the position of 0 in the retained mask (from bag A), and Represents the pseudo-bag corresponding to the position with 1 in the retained mask (from bag B), and the ∪ symbol represents the joint operation, merging the pseudo-bags selected by the mask to finally generate a mixed bag containing n pseudo-bags;
[0031] b. Output the masked pseudo bag with probability (1-p)
[0032] Furthermore, in step 3, the mixing ratio λ obeys a Beta distribution with a parameter α, that is, λ~Beta(α,α), where α is a non-negative number.
[0033] Furthermore, the specific steps of downsampling to generate secondary sub-bags in step 4 include:
[0034] a. Dynamic downsampling condition determination: If the total number of instances N in the first-level pseudo-bag of the current level exceeds the preset maximum group size S max , then perform random downsampling operation, otherwise go directly to the blocking step;
[0035] b. Perform random downsampling: Randomly select S from the first-level pseudo bag max instances, forming a downsampled sub-packet Where D is the feature dimension;
[0036] c. Generate secondary sub-bags by block: downsample the sub-bags Divide into M secondary sub-bags in sequence, where: S sup ≤S max , each secondary sub-bag Contains a block of consecutive instances, the last sub-bag is allowed to be less than S sup instances;
[0037] d. Secondary sub-bag feature enhancement: Add a learnable MSG token T to the head of each secondary sub-bag MSG ∈R 1×1×D , forming the enhanced input sequence:
[0038] Furthermore, the specific implementation of the multi-head self-attention calculation in step 5 includes the following steps:
[0039] a. Linear projection and separate processing: embedding features X∈R of all tokens (including MSG tokens) in the secondary sub-bag B ×N×D Through the learnable matrix Projected into query (Query), key (Key), value (Value) vectors, and divided into heads according to the number of heads H, that is:
[0040] where d k =D / H, split_heads is the dimension reordering operation;
[0041] b. Attention weight calculation: Calculate the query-key dot product for each attention head, normalize it by the scaling factor dk and apply Softmax, i.e. where P∈R N×N is a position encoding matrix based on instance space coordinates, used to model spatial dependencies;
[0042] c. Multi-head output splicing and projection: The output of each attention head is spliced and passed through the linear layer W O ∈R D×D Project back to the original dimension, that is, MSA(X)=Concat(head1,head2,...,head H )W O
[0043] d. Residual connection and layer normalization: Connect the multi-head self-attention output to the input feature residual and perform layer normalization, i.e. X out =LayerNorm(X+MSA(X))
[0044] The beneficial effects of the present invention are as follows:
[0045] (1) An innovative pseudo-bag mixed data augmentation strategy is proposed to solve the size inconsistency and semantic alignment problems faced by WSI data in Mixup applications.
[0046] (2) A random mixing mechanism is introduced to generate mixed pseudo bags and masked pseudo bags at the same time, which enhances data diversity and promotes the model's efficient learning of neighborhood samples.
[0047] (3) Through the multi-level architecture and MSG interaction mechanism, the computational complexity of processing large-scale WSI data is effectively reduced from O(n 2 ) is reduced to O(n 2 / g), where g is the number of secondary sub-bags.
[0048] (4) The accuracy and AUC index of WSI classification were significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 1 is a schematic flow chart of the construction method of the present invention;
[0050] Figure 2 It is a schematic diagram of the overall framework of the present invention.
[0051] Figure 3Schematic diagram of the comparison results between the present invention and other methods DETAILED DESCRIPTION
[0052] In order to clarify the design purpose, technical path and innovative value of the present invention, the following will be accompanied by drawings and specific implementation examples to elaborate on the scheme of the present invention. It should be noted that the implementation schemes listed herein are only used as technical demonstrations for understanding the present invention and do not constitute a limitation on the scope of protection. Those skilled in the art should be aware that, without departing from the core idea of the present invention, the technical solution of this patent can be implemented by equivalent replacement or equivalent deformation.
[0053] Implementation example:
[0054] This paper uses the TCGA-LUNG pathology image dataset as an example dataset. The dataset contains 1044 whole-slice images and provides pixel-level annotations. The images are divided into two categories: LUAD (lung adenocarcinoma) and LUSC (lung squamous cell carcinoma). Here we use data with a magnification of 40:
[0055] First, the whole-slice pathology image is segmented to generate multiple instance-level image blocks, and a pre-trained ResNet-50 model is used for feature extraction to obtain a 1024-dimensional feature vector.
[0056] Then, a phenotypic clustering algorithm is used to perform feature clustering based on instance features, and a stratified sampling strategy is used to divide the instances into multiple semantically preserved pseudo bags.
[0057] Then, for the divided pseudo bags, instances from two pseudo bags are randomly selected, and linear interpolation and masking operations are performed according to the preset mixing ratio λ to generate mixed camouflage pseudo bags and masked pseudo bags. The mixed camouflage pseudo bag or masked pseudo bag is selected based on the preset probability P. This step is repeated continuously to finally generate a mixed bag containing n pseudo bags.
[0058] The hybrid enhanced pseudo bag is then downsampled and divided into two sub-bags: Divide into M secondary sub-bags in sequence, where: S sup ≤S max , each secondary sub-bag Contains a block of consecutive instances, the last sub-bag is allowed to be less than S sup instances;
[0059] Append a learnable MSG token T to each secondary sub-bag MSG ∈R 1×1×D And perform multi-head self-attention calculation Thus, the updated MSG token T MSGCarrying local features within the sub-pouch;
[0060] Extract the updated MSG tokens from all secondary sub-bags to form a high-level bag and append a learnable CLS token T in front of the high-level bag CLS ∈R 1×1×D , used for global information aggregation Finally, the classification result is obtained through the fully connected layer.
Claims
1. A hierarchical multi-instance learning full-slice image classification method based on pseudo-bag hybrid enhancement, characterized by: The steps include: S1: Segment the whole-slice pathology image to generate multiple instance-level image blocks and perform feature extraction; S2: using a phenotypic clustering algorithm to perform feature clustering on the instance, and dividing the instance into multiple semantically preserved pseudo bags through a stratified sampling strategy; S3: Randomly select instances from two pseudo bags, perform linear interpolation and masking operations according to a preset mixing ratio, and generate a mixed camouflage pseudo bag and a masked pseudo bag; S4: taking the mixed disguised pseudo-bag as the initial sub-bag, performing downsampling in the sub-bag to obtain a secondary sub-bag, and appending a learnable MSG (Message) token under the secondary sub-bag; S5: Perform multi-head self-attention calculation inside the secondary sub-bag to capture the spatial dependencies between instances; S6: Collect the MSG token features of each secondary sub-bag and construct a high-level pathology representation by feature concatenation; S7: Embed CLS (Classification) tokens in the high-level pathology representation to give classification prediction results.
2. The method for full-slice image classification based on hierarchical multi-instance learning with pseudo-bag hybrid enhancement according to claim 1, characterized in that: The value ranges of the number of pseudo bags n, the number of phenotypic clusters l and the number of iterations k are integers between 8 and 20.
3. The method for full-slice image classification based on hierarchical multi-instance learning with pseudo-bag hybrid enhancement according to claim 1, characterized in that: The phenotypic clustering algorithm includes: a. Compute the prototype representation of the WSI bag; where m i represents the number of samples in the i-th phenotypic class, ∑ j x i,j Represents all samples x belonging to the i-th phenotypic class i,j Sum, subscript j traverses every sample in the category, x i,j represents the jth sample in the i-th category, p i It represents the center value (mean) of the i-th category, which is obtained by calculating the average value of all samples in the category and is used to represent the "center position" of the category. b. Calculate the cosine similarity between each instance and the bag prototype: s i,j ←cos(p i ,x i,j ) Where cos(·) represents the cosine similarity function, which is used to calculate the cosine value of the angle between two vectors. The formula is: p i represents the central value (mean) of the i-th category, x i,j represents the jth sample in the i-th category. c. Assign instances to l initial phenotypic clusters based on similarity; d. Perform k iterations of refinement to optimize the phenotypic clusters. The multi-level multi-instance learning full-slice image classification method based on pseudo-bag mixing enhancement according to claim 1 is characterized in that the stratified sampling strategy specifically includes: a. Evenly divide the l feature clusters generated by phenotypic clustering; b. Extract instances proportionally from each feature cluster to ensure that each pseudo-bag contains instances of all phenotypic categories; c. Maintain the consistency of pathological semantic distribution among pseudo bags by dynamically adjusting the sampling ratio.
4. The method for full-slice image classification based on hierarchical multi-instance learning with pseudo-bag hybrid enhancement according to claim 1, characterized in that: The random mixing mechanism includes: a. Merge the retained pseudo bags with probability p to generate a mixed pseudo bag: in represents the generated mixed bag, M λ ∈{0,1} n is a binary mask with a length of n (number of pseudo packets), and each element has a value of 0 or 1. The number of 1s in the mask is controlled by λ~Beta(α,α) and satisfies ⊙ represents the element-by-element multiplication operation, which actually means selecting the pseudo bag according to the mask, and represents the pseudo bag corresponding to the position of 0 in the retained mask (from bag A), and Represents the pseudo bags corresponding to the positions with 1 in the retained mask (from bag B), and the ∪ symbol represents the joint operation, merging the pseudo bags selected by the mask to finally generate a mixed bag containing n pseudo bags; b. Output the masked pseudo bag with probability (1-p) 5. The method for full-slice image classification based on hierarchical multi-instance learning with pseudo-bag hybrid enhancement according to claim 1, characterized in that: The mixing ratio λ obeys the Beta distribution with parameter α, that is, λ~Beta(α,α), where α is a non-negative number.
6. The method for full-slice image classification based on hierarchical multi-instance learning with pseudo-bag hybrid enhancement according to claim 1, characterized in that: The specific steps of downsampling to generate secondary sub-bags include: a. Dynamic downsampling condition determination: If the total number of instances N in the first-level pseudo-bag of the current level exceeds the preset maximum group size S max , then perform random downsampling operation, otherwise go directly to the blocking step; b. Perform random downsampling: Randomly select S from the first-level pseudo bag max instances, forming a sub-bag after downsampling Where D is the feature dimension; c. Generate secondary sub-bags by block: downsample the sub-bags Divide into M secondary sub-bags in sequence, where: S sup ≤S max , each secondary sub-bag Contains a block of consecutive instances, the last sub-bag is allowed to be less than S sup instances; d. Secondary sub-bag feature enhancement: Add a learnable MSG token T to the head of each secondary sub-bag MSG ∈R 1×1×D , forming the enhanced input sequence:
7. The method for full-slice image classification based on hierarchical multi-instance learning with pseudo-bag hybrid enhancement according to claim 1, characterized in that: The specific implementation of the multi-head self-attention calculation includes the following steps: a. Linear projection and separate processing: embedding features X∈R of all tokens (including MSG tokens) in the secondary sub-bag B×N×D Through the learnable matrix Projected into query (Query), key (Key), value (Value) vectors, and divided into heads according to the number of heads H, that is: where d k =D / H, split_heads is the dimension reordering operation; b. Attention weight calculation: Calculate the query-key dot product for each attention head, normalize it by the scaling factor dk and apply Softmax, i.e. where P∈R N×N is a position encoding matrix based on instance space coordinates, used to model spatial dependencies; c. Multi-head output splicing and projection: The output of each attention head is spliced and passed through the linear layer W O ∈R D×D Project back to the original dimension, that is, MSA(X)=Concat(head1,head2,...,head H )W O d. Residual connection and layer normalization: Connect the multi-head self-attention output to the input feature residual and perform layer normalization, i.e. X out =LayerNorm(X+MSA(X)) 8. The method for full-slice image classification based on hierarchical multi-instance learning with pseudo-bag mixing enhancement according to claim 1, characterized in that: The feature extractor uses a pre-trained ResNet-50 model with a feature dimension of d=1024.
9. The method for full-slice image classification based on hierarchical multi-instance learning with pseudo-bag hybrid enhancement according to claim 1, characterized in that: The model was trained using the Adam optimizer with a learning rate of 1e-4 and a weight decay of 1e-5.