High-resolution medical pathology WSI image analysis method based on scan-invariant Mamba

Abstract

This invention relates to a high-resolution medical pathology WSI image analysis method based on scan-invariant Mamba, comprising the following steps: selecting a public database, wherein the images in the database must contain tissue information; constructing an SMC-MIL network model W, which contains three modules; selecting an image q from the database and inputting it into W, where q passes through the three modules sequentially, finally obtaining the predicted packet label of q. In actual detection, the accuracy of image analysis results can be improved based on the predicted packet label.

Description

Technical Field

[0001] This invention relates to the field of computer image analysis, and in particular to a high-resolution medical pathology WSI image analysis method based on scan-invariant Mamba. Background Technology

[0002] Histopathological image analysis is a key area of ​​modern medicine, typically using whole-slice images (WSI). Due to the high resolution (up to one billion pixels) and lack of pixel-level annotation in WSI, traditional WSI analysis methods face unique challenges. In the most widely used MIL paradigm, instances are converted into offline features, used with a pre-trained model, and then aggregated into bag-level representations for subsequent analysis. In this paradigm, WSI classification can be viewed as a long-sequence modeling problem, treating a large number of instances within a bag as labels, aiming to model the correlation between instances and the overall spatial context within the entire bag to capture discriminative information. Some state-space model (SSM)-based models are designed to efficiently model long sequences. Mamba is a recently proposed representative SSM-based model that offers better long-sequence modeling performance compared to ViT, but requires less computational resources. However, there is a fundamental difference between the Mamba model and the MIL task: Mamba's output is sensitive to different scan modes (scan-sensitive), while in MIL, the model produces invariant results regardless of the scan mode (scan-invariant).

[0003] To reconcile the differences between scan-sensitive Mamba and scan-invariant MIL, previous work primarily involved inputting image patch sequences with different scan patterns into Mamba and then aggregating multiple output sequences. For example, MamMIL combines bi-SSM and 2D-CAB modules to integrate scan pattern information from the original sequence, reverse sequence, and local sequences; MambaMIL reshapes the sequence into rectangles and applies a sequence reordering operation to scan the new sequence vertically. However, the limited number of scan patterns used to generate instance sequences restricts the model's ability to learn scan-pattern-invariant features. Therefore, this does not fundamentally mitigate the impact of scan patterns on Mamba's performance. Summary of the Invention

[0004] In view of the above-mentioned problems in the existing technology, the technical problem to be solved by the present invention is: how to more fundamentally solve the impact of scanning mode on the performance of Mamba, so as to reconcile the difference between scan-sensitive Mamba and scan-invariant MIL, thereby improving the accuracy of high-resolution medical image and pathological WSI image analysis.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: a high-resolution medical pathology WSI image analysis method based on scan-invariant Mamba, comprising the following steps:

[0006] S100: Select a public image database Q, where all images in Q contain organizational information and each image has a known label;

[0007] S200: Construct the SMC-MIL network model W, which includes three parts: an offline data preprocessing module, a sequence comparison enhancement module, and a Mamba aggregation module.

[0008] S300: Select any image q from Q, treat q as a packet, input q into the offline data preprocessing module to obtain the feature original scan sequence Z;

[0009] S400: Input Z into the sequence contrast enhancement module, and perform sequence enhancement branch operations on Z to obtain new sequences Z. easy And perform a sequence mask branching operation to obtain a new sequence Z. hard :

[0010] Sequence augmentation branching operation: Z is processed through an attention mechanism to identify instances with high attention values, resulting in a set of instance attention scores A. A is then processed using the Mixup method and added to Z to obtain a new sequence Z. easy ;

[0011] Sequence masking branch operation: Z is processed through an attention masking mechanism to mask instances with low attention values, resulting in a new masked sequence Z. hard ;

[0012] S500: Z easy and Z hard After passing through the Mamba aggregation module, the corresponding packet features b1 and b2 are obtained, as shown in the following expressions:

[0013] b1=A1·Ψ(Z easy )=φ(Ψ(S A (Z)))·Ψ(S A (Z))

[0014] b2=A2·Ψ(Z hard )=φ(Ψ(S M (Z)))·Ψ(S M (Z))

[0015] Where A1 and A2 are respectively Ψ(Z easy ) and Ψ(Z hard Attention score; S A Indicates sequence enhancement, SM Ψ(·) represents sequence masking; φ(·) represents Mamba, and φ(·) represents the attention mechanism.

[0016] S600: Calculate Z respectively easy and Z hard Corresponding prediction bag tags and The specific expression is as follows:

[0017]

[0018]

[0019] in, Represents a classifier;

[0020] S700: The formula for optimizing the network parameters of the SMC-MIL model is expressed as follows:

[0021]

[0022] Where θ represents the network parameters of the SMC-MIL network model, and γ represents the parameter that balances the classification loss and the consistency loss. Denotes the overall loss function of W. This represents the consistency loss of package features. This represents the consistency loss of package logits / package classification;

[0023] The classification loss for the sequence augmentation branch is expressed as follows:

[0024]

[0025]

[0026] Here, Y represents the true value of the packet; the classification loss is the conventional cross-entropy loss, which is an existing technique aimed at constraining the model's classification results to make them closer to the true value; while the consistency loss ensures that the packet features and classification results of packets from different sequences input to the same Mamba output remain consistent, thus mitigating the differences between sequence-dependent Mamba models and sequence-independent multi-instance learning tasks.

[0027] S800: Set the maximum number of training iterations, perform steps S300-S700 on all images in Q, and according to... Update the network parameters θ, and stop training when the maximum number of iterations is reached to obtain the trained network model W'.

[0028] S900: Select an unknown image containing organizational information and input it into W' to obtain the predicted packet label for that image.

[0029] Preferably, the specific steps for obtaining the original feature scan sequence Z in S300 are as follows:

[0030] S310: Divide q into several tiles, discarding tiles that only contain background areas. Generally, when the information saturation of a tile or patch is less than 15, the information contained in that tile or patch is considered to be background area. Form a set X of the remaining tiles, with each tile representing an instance x. i Specifically, it is expressed as follows:

[0031]

[0032] Where i represents the i-th tile, and I represents the total number of tiles;

[0033] S320: Use a feature extractor to extract features from all patches in X, and then perform linear projection on the extracted offline features to obtain the original feature scan sequence. Among them, z i This represents the offline features of the i-th instance; the feature extractor can be other feature extractors such as ResNet-50 or UNI, which are existing technologies.

[0034] Preferably, the sequence enhancement branch in S400 operates as follows:

[0035] S410: Calculate the attention score for each instance in Z using an attention mechanism, defined as follows:

[0036] A = [a1, ..., a i ,…,a N ]=φ(Z)

[0037] Where φ(·) represents the attention mechanism;

[0038] S411: Sort A in S410, as shown in the following expression:

[0039] I = [j1,j2,…,j N =Sort(A)

[0040] Where j1 represents the index of the instance with the highest attention score, j N The index representing the instance with the lowest score;

[0041] S412: Select the instances with the highest attention scores in the top α% of I as the amplification terms for the Z sequence, ultimately obtaining the new sequence Z. easy The expression is as follows:

[0042]

[0043]

[0044] in, It is a Mixup on the selected instance, S A (·) indicates a sequence enhancement module. Indicates the index of the augmentation item; the attention mechanism here shares parameters with the subsequent aggregation module, therefore, in Figure 1 The Mamba sequence length input in the "easy" branch shown is...

[0045] Preferably, the sequence mask branching operation in S400 is as follows:

[0046] S420: Using an attention masking mechanism, an n-dimensional mask vector M is generated from Z as shown below:

[0047] M = [m1, ..., m r ,…,m N ],

[0048] Where, m r ∈{0,1}, if m r =1, then the r-instance is unmasked; otherwise, it is masked.

[0049] S421: Set the mask rates for high and low attention values ​​to β respectively. h % and β l %, therefore, the sequence index corresponding to the high attention value is The sequence index corresponding to low attention value is therefore,

[0050]

[0051] Finally, the new sequence Z after masking hard Represented as:

[0052] Z hard ←S M (Z) :=M·Z,

[0053] Where, S M (·) represents a sequence mask; during the experiment, let β h =β l .

[0054] Compared with the prior art, the present invention has at least the following advantages:

[0055] 1. This invention improves the existing Mamba model by adding high-interest instances in the comparison sequence and masking low-interest instances, changing the "ordered" restriction to a "disordered" requirement. This improves the adaptability of the improved model and can more fundamentally solve the impact of scanning mode on Mamba performance, thereby reconciling the differences between scan-sensitive Mamba and scan-invariant MIL, and thus improving the accuracy of pathological WSI image analysis in high-resolution medical images.

[0056] 2. This invention proposes a new Mamba-based MIL framework, named SMC-MIL, for computational pathology. SMC-MIL teaches Mamba to learn scan-invariant packet representations through instance sequence contrast learning, which is more suitable for the MIL-based WSI classification paradigm.

[0057] 3. The model of this invention designs a dual-branch instance sequence contrastive learning framework, which aims to achieve consistency between two extremely different input instance sequences and eliminate the influence of sequence arrangement on the final package decision; this framework provides a simple and effective solution for using SSM to process MIL tasks.

[0058] 4. The present invention proposes a sequence contrast enhancement method, which intentionally shuffles, enhances and masks instances based on the difficulty assessment of instance classification. This method not only increases the difference in order and length between the two input contrast sequences, but also expands the difficulty difference in identifying the discriminative regions. Obviously, this method increases the difficulty of contrast learning, thereby forcing Mamba to learn the discriminative features of WSI without considering the arrangement of its tile sequence. Attached Figure Description

[0059] Figure 1 This is a schematic diagram outlining the SMC-MIL model of the present invention.

[0060] Figure 2 A comparison between various variants of Mamba.

[0061] Figure 3 A tile visualization created for the original Mamba (baseline) and SMC-MIL. Detailed Implementation

[0062] The present invention will now be described in further detail.

[0063] This invention proposes a high-resolution medical pathology WSI image analysis method based on scan-invariant Mamba. This method is a solution based on Mamba and scan-invariant MIL. Firstly, based on the Mamba and MIL models, it addresses the issue by increasing high-interest sequence instances and masking low-interest dummy column instances, respectively. This fundamentally solves the performance impact of scan mode on Mamba, reconciling the differences between scan-sensitive Mamba and scan-invariant MIL, thereby improving the accuracy of pathology WSI image analysis in high-resolution medical images.

[0064] See Figures 1-3 A high-resolution medical pathology WSI image analysis method based on scan-invariant Mamba includes the following steps:

[0065] S100: Select a public image database Q, where all images in Q contain organizational information and each image has a known label;

[0066] S200: Construct the SMC-MIL network model W, which includes three parts: an offline data preprocessing module, a sequence comparison enhancement module, and a Mamba aggregation module.

[0067] S300: Select any image q from Q, treat q as a packet, input q into the offline data preprocessing module to obtain the feature original scan sequence Z;

[0068] The specific steps for obtaining the feature original scan sequence Z in S300 are as follows:

[0069] S310: Divide q into several tiles, discarding tiles that only contain background areas. Generally, when the information saturation of a tile or patch is less than 15, the information contained in that tile or patch is considered to be background area. Form a set X of the remaining tiles, with each tile representing an instance x. i Specifically, it is expressed as follows:

[0070]

[0071] Where i represents the i-th tile, and I represents the total number of tiles;

[0072] S320: Use a feature extractor to extract features from all patches in X, and then perform linear projection on the extracted offline features to obtain the original feature scan sequence. Among them, z i This represents the offline features of the i-th instance; the feature extractor can be other feature extractors such as ResNet-50 or UNI, which are existing technologies.

[0073] S400: Input Z into the sequence contrast enhancement module, and perform sequence enhancement branch operations on Z to obtain new sequences Z. easy And perform a sequence mask branching operation to obtain a new sequence Z. hard :

[0074] Sequence augmentation branching operation: Z is processed through an attention mechanism to identify instances with high attention values, resulting in a set of instance attention scores A. A is then processed using the Mixup method and added to Z to obtain a new sequence Z. easy ;

[0075] The operation of the sequence enhancement branch in S400 is as follows:

[0076] S410: Calculate the attention score for each instance in Z using an attention mechanism, defined as follows:

[0077] A = [a1, ..., a i ,…,a N ]=φ(Z)

[0078] Where φ(·) represents the attention mechanism;

[0079] S411: Sort A in S410, as shown in the following expression:

[0080] I = [j1,j2,…,j N =Sort(A)

[0081] Where j1 represents the index of the instance with the highest attention score, j N The index representing the instance with the lowest score;

[0082] S412: Select the instances with the highest attention scores in the top α% of I as the amplification terms for the Z sequence, ultimately obtaining the new sequence Z. easy The expression is as follows:

[0083]

[0084]

[0085] in, It is a Mixup on the selected instance, S A (·) indicates a sequence enhancement module. Indicates the index of the augmentation item; the attention mechanism here shares parameters with the subsequent aggregation module, therefore, in Figure 1 The sequence length for the Mamba aggregation module input in the "easy" branch shown is...

[0086] Sequence masking branch operation: Z is processed through an attention masking mechanism to mask instances with low attention values, resulting in a new masked sequence Z. hard ;

[0087] Preferably, the sequence mask branching operation in S400 is as follows:

[0088] S420: Using an attention masking mechanism, an n-dimensional mask vector M is generated from Z as shown below:

[0089] M = [m1, ..., m r ,…,m N ],

[0090] Where, m r ∈{0,1}, if m r =1, then the r-instance is unmasked; otherwise, it is masked.

[0091] S421: Set the mask rates for high and low attention values ​​to β respectively. h % and β l %, therefore, the sequence index corresponding to the high attention value is The sequence index corresponding to low attention value is therefore,

[0092]

[0093] Finally, the new sequence Z after masking hard Represented as:

[0094] Z hard ←S M (Z) :=M·Z,

[0095] Among them, S M (·) indicates sequence masking; during the experiment, let β h =β l .

[0096] S500: Z easy and Z hard After passing through the Mamba aggregation module, the corresponding packet features b1 and b2 are obtained, as shown in the following expressions:

[0097] b1=A1·Ψ(Z easy )=φ(Ψ(S A (Z)))·Ψ(S A (Z))

[0098] b2=A2·Ψ(Z hard )=φ(Ψ(S M (Z)))·Ψ(S M (Z))

[0099] Where A1 and A2 are respectively Ψ(Z easy ) and Ψ(Z hard Attention score; S A Indicates sequence enhancement, S M Ψ(·) represents sequence masking; φ(·) represents Mamba, and φ(·) represents the attention mechanism.

[0100] S600: Calculate Z respectively easy and Z hard Corresponding prediction bag tags and The specific expression is as follows:

[0101]

[0102]

[0103] in, Represents a classifier;

[0104] S700: The formula for optimizing the network parameters of the SMC-MIL model is expressed as follows:

[0105]

[0106] Where θ represents the network parameters of the SMC-MIL network model, and γ represents the parameter that balances the classification loss and the consistency loss. Denotes the overall loss function of W. This represents the consistency loss of package features. This represents the consistency loss of package logits / package classification;

[0107] The classification loss for the sequence augmentation branch is expressed as follows:

[0108]

[0109] Here, Y represents the true value of the packet; the classification loss is the conventional cross-entropy loss, which is an existing technique aimed at constraining the model's classification results to make them closer to the true value; while the consistency loss ensures that the packet features and classification results of packets from different sequences input to the same Mamba output remain consistent, thus mitigating the differences between sequence-dependent Mamba models and sequence-independent multi-instance learning tasks.

[0110] S800: Set the maximum number of training iterations, perform steps S300-S700 on all images in Q, and according to... Update the network parameters θ, and stop training when the maximum number of iterations is reached to obtain the trained network model W'.

[0111] S900: Select an unknown image containing organizational information and input it into W' to obtain the predicted packet label for that image.

[0112] Experimental Design and Content

[0113] Experimental Dataset

[0114] This invention evaluates the SMC-MIL model through three WSI analysis tasks: cancer diagnosis, subtype classification, and survival prediction. For the cancer diagnosis task, this invention uses the CAMELYON dataset, which is a combination of the CAMELYON-16 and CAMELYON-17 datasets. For the subtype classification task, this invention uses the TCGA-NSCLC, TCGA-BRCA, and BRACS datasets. To evaluate the accuracy of survival prediction, this invention uses TCGA-LUAD, TCGA-LUSC, and TCGA-BLCA to evaluate the performance of the survival prediction task.

[0115] Experimental setup

[0116] For diagnosis and classification, this invention utilizes Accuracy, AUC, and F1-score to evaluate model performance. For survival prediction, this invention reports the C-index for all datasets. By default, this invention employs 5-fold cross-validation, consistent with the baseline approach for performance comparison. Specifically, for the BRACS dataset, this invention also follows the partitioning scheme from [the official dataset name], dividing the dataset into training, validation, and test sets in a 395:65:87 ratio within the official partitioning. This invention performs three independent replicate experiments to minimize random variations in the official partitioning. Furthermore, to maintain consistency with other datasets in the experiments, this invention also performs five cross-validation experiments on the 7-class classification task of BRACS, marked with ★ in Table \ref{tab:main_comp2}.

[0117] Implementation details

[0118] Following previous work, this invention uses ImageNet-1k pre-trained ResNet50 and the state-of-the-art base model UNI as offline feature extractors. UNI is based on a 1×10 6 WSIs' large internal histology dataset 1×10 8 Pre-trained on the patch.

[0119] Experimental results

[0120] Cancer diagnosis and classification

[0121] Table 1. Comparison of cancer diagnostic performance on the CAMELYON dataset and subtype classification performance on the TCGA-NSCLC dataset.

[0122]

[0123] Table 1 presents the performance of various MIL methods on the CAMELYON and NSCLC datasets for cancer diagnosis and subtyping. These results demonstrate that the method of this invention achieves best performance across all metrics on all benchmarks. Specifically, on the CAMELYON dataset, the method of this invention improves accuracy, AUC, and F1-score by 0.55%, 1.26%, and 0.28%, respectively, compared to the second-best method. These figures are 0.57%, 0.4%, and 0.5% on the NSCLC dataset. Also noteworthy is the significant performance improvement of the proposed SMC-MIL compared to SRMamba, which is also based on the Mamba architecture. SMC-MIL achieves AUC performance 1.99% and 1.58% higher than SRMamba on the CAMELYON and NSCLC datasets, respectively. This validates the hypothesis of this invention that eliminating Mamba's dependence on scan mode is better suited for MIL tasks.

[0124] Table 2 BRACS-7 ★ The typing results of TCGA-BRCA.

[0125]

[0126] Table 2 presents a comparison of subtype classification performance on the BRACS dataset (7-class classification task selected in this experiment) and the BRCA dataset. The observations are consistent with those shown in Table 1. The model of this invention achieves state-of-the-art performance on all metrics. Compared to suboptimal methods, this method achieves better performance on BRACS-7. ★ On the dataset, accuracy, AUC, and F1-score improved by 1.81%, 1.51%, and 3.68%, respectively. On the BRCA dataset, improvements of 1.63%, 0.67%, and 2.50% were achieved, respectively. Compared to the srmanba model, BRACS-7... ★ The AUC of the dataset increased by 1.22%.

[0127] Table 3. Official classification of BRACS data subtypes

[0128]

[0129]

[0130] Table 3 reports the subtype results for the 3-class and 7-class classifications using the officially defined subclasses on the BRACS dataset. The results show that the method of this invention still has advantages compared to the baseline. For example, the method of this invention improves the srmanba AUC by 4.62% and 4.65% respectively on these two tasks. This clearly demonstrates that the significant performance improvement of the method of this invention is not solely due to the powerful modeling capabilities of Mamba itself, and further confirms that the scan invariance of the method of this invention can overcome the bottleneck of Mamba in solving the MIL problem.

[0131] Table 4. Cancer diagnosis and subtyping performance using UNI as the offline feature extractor.

[0132]

[0133] Since baseline methods using UNI feature extractors for diagnostic and typing tasks have demonstrated sufficiently strong performance that further comparisons are meaningless (see Table 4), this invention focuses only on the impact of UNI on survival prediction in the main text.

[0134] Survival prediction

[0135] Table 5 shows the survival prediction results on the three main datasets.

[0136]

[0137]

[0138] Table 5 presents the experimental results on three survival prediction datasets. The proposed SMC-MIL model exhibits strong performance, achieving a c-index score of 61.96% on the BLCA dataset, 65.05% on LUAD, and 61.64% on LUSC. It significantly outperforms the compared methods, improving performance by 0.98%, 1.00%, and 0.73% over the second-best method on each dataset, respectively. Furthermore, SMC-MIL yields considerable gains even when utilizing high-quality features extracted by the base model. In particular, compared to SRMamba, it improves performance by 0.56%, 1.87%, and 4.17% on the BLCA, LUAD, and LUSC datasets, respectively. These results highlight the consistency and reliability of the proposed strategy and underscore its effectiveness in predicting survival outcomes.

[0139] ablation experiment

[0140] Effectiveness analysis of different components

[0141] Table 6 shows the performance impact of different components on the three main datasets.

[0142]

[0143] Table 6 reports the impact of different modules in SMC-MIL on the three datasets. OriginMamba simply means integrating Mamba into ABMIL. Aug. indicates the application of sequence contrastive learning. Samp. indicates expanding the "hard" branch using a sampling strategy to match the length of the "easy" branch. Con. indicates the application of consistency constraints. The baseline of this invention involves incorporating the vanilla Mamba architecture into ABMIL. First, the contrastive learning and consistency loss framework are integrated into the baseline model. This strategy improves the AUC on the CAMELYON, NSCLC, and BRCA datasets by 1.50%, 1.18%, and 0.88%, respectively, indicating that the proposed consistency constraints effectively guide Mamba to learn more detailed, sequence-independent features from different scan sequences. Experimental results show that, compared to OriginMamba, the introduction of sequence augmentation and masking modules enhances Mamba's ability to distinguish salient regions within sequences. After introducing these two modules into OriginMamba, the full SMC-MIL achieved the best performance (93.81% AUC on CAMELYON, 96.65% AUC on TCGA, and 93.98% AUC on BRCA).

[0144] Scan Invariance Verification Analysis

[0145] Table 7. Scan invariance evaluation for different Mamba models

[0146]

[0147] To verify that the method of this invention enables Mamba to construct more scan-invariant features, this invention inputs three randomly selected scan sequences into a trained Mamba variant for inference and calculates the mean and standard deviation of their respective AUCs. Experimental results are shown in Table 7. Compared to the baseline, the method of this invention reduces the standard deviation by a factor of 2 on the CAMELYON and NSCLC datasets and by a factor of 3 on the BRACS-7 dataset. Notably, the method of this invention not only achieves the highest mean AUC across all datasets but also exhibits the smallest standard deviation. This underscores the effectiveness of the method of this invention in guiding Mamba models to learn more invariant features of scan sequences, outperforming other Mamba variant methods.

[0148] Comparison of different variants of Mamba

[0149] To further verify that the superior performance of the method of this invention stems from the architecture of this invention rather than Mamba itself, this invention conducted comparative experiments on three variants: Figure 2 The results show that SMC-MIL achieves state-of-the-art performance across all six tasks on all four datasets. The method of this invention not only outperforms the original single-branch Mamba in performance but also exhibits many advantages over multi-sequence (or scan) aggregation methods (e.g., BiMamba and SRMamba). Experimental results demonstrate that the scan-invariant Mamba algorithm has stronger instance modeling capabilities and can extract more discriminative features than the multi-scan ensemble Mamba algorithm.

[0150] Different branch comparison strategies

[0151] Table 8. Different Branch Comparison Strategies

[0152]

[0153] Table 8 lists the performance of SMC-MIL under five different branch contrast strategies. The results show that the "easy vs. hard" strategy produces the best performance. The main reason for this superior performance is that the "easy" branch helps Mamba generate reliable packet representations and classifications, while the "hard" branch forces the model not only to learn irrelevant features but also to explore more discriminative units from the masked sequences.

[0154] For example, consistency between the two branches can be achieved. Furthermore, differences between the two branches can also improve performance. The "hard vs hard" strategy performs the worst. This invention attributes its failure to the lack of reliable guidance from the "easy" branch in package representation and classification.

[0155] Visualization Analysis

[0156] To visually verify the interpretability of the method of this invention, this invention visualizes instances of high attention scores generated by OriginMamba and the method of this invention on the Camelyon-16 dataset, such as... Figure 3 As shown, blue lines delineate the tumor area. Brighter patches indicate higher attention scores. The visualization clearly demonstrates that SMC-MIL is able to select significant patches more accurately compared to the original Mamba.

[0157] This invention proposes a Mamba-based and contrastive learning-based MIL framework to bridge the gap between scan-sensitive Mamba and scan-invariant MIL. By applying consistency constraints to the results of different scan branches, it guides Mamba to learn scan-invariant features of packets from different sequences. Sequence contrast enhancement is used to enhance and mask sequences, forcing Mamba to recognize more discriminative patches. Extensive experiments demonstrate that this framework outperforms other recent methods and effectively mitigates the limitations of Mamba in MIL and other vision tasks, making it more suitable for these tasks and paving the way for further applications of Mamba in vision tasks.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A high-resolution medical pathology WSI image analysis method based on scan-invariant Mamba, characterized in that: Includes the following steps: S100: Select a public image database Q, where all images in Q contain organizational information and each image has a known label; S200: Construct the SMC-MIL network model W, which includes three parts: an offline data preprocessing module, a sequence comparison enhancement module, and a Mamba aggregation module. S300: Select any image q from Q, treat q as a packet, input q into the offline data preprocessing module to obtain the feature original scan sequence Z; S400: Input Z into the sequence contrast enhancement module, and perform sequence enhancement branch operations on Z to obtain new sequences Z. easy And perform a sequence mask branching operation to obtain a new sequence Z. hard : Sequence augmentation branching operation: Z is processed through an attention mechanism to identify instances with high attention values, resulting in a set of instance attention scores A. A is then processed using the Mixup method and added to Z to obtain a new sequence Z. easy ; Sequence masking branch operation: Z is processed through an attention masking mechanism to mask instances with low attention values, resulting in a new masked sequence Z. hard ; S500: Z easy and Z hard After passing through the Mamba aggregation module, the corresponding packet features b1 and b2 are obtained, as shown in the following expressions: b1=A1·Ψ(Z easy )=φ(Ψ(S A (Z)))·Ψ(S A (WITH)) b2=A2·Ψ(Z hard )=φ(Ψ(S M (X)))·Ψ(S M (Z)) Where A1 and A2 are respectively Ψ(Z easy ) and Ψ(Z hard Attention score; S A Indicates sequence enhancement, S M Ψ(·) represents sequence masking; φ(·) represents Mamba, and φ(·) represents the attention mechanism. S600: Calculate Z respectively easy and Z hard Corresponding prediction bag tags and The specific expression is as follows: in, Represents a classifier; S700: The formula for optimizing the network parameters of the SMC-MIL model is expressed as follows: Where θ represents the network parameters of the SMC-MIL network model, and γ represents the parameter that balances the classification loss and the consistency loss. Denotes the overall loss function of W. This represents the consistency loss of package features. This represents the consistency loss of package logits / package classification; The classification loss for the sequence augmentation branch is expressed as follows: Where Y represents the actual value of the packet; S800: Set the maximum number of training iterations, perform steps S300-S700 on all images in Q, and according to... Update the network parameters θ, and stop training when the maximum number of iterations is reached to obtain the trained network model W'. S900: Select an unknown image containing organizational information and input it into W' to obtain the predicted packet label for that image.

2. The high-resolution medical pathology WSI image analysis method based on scan-invariant Mamba as described in claim 1, characterized in that: The specific steps for obtaining the feature original scan sequence Z in S300 are as follows: S310: Divide q into several tiles, discard tiles containing only the background area, and form a set X of the remaining tiles, where each tile represents an instance x. i Specifically, it is expressed as follows: Where i represents the i-th tile, and I represents the total number of tiles; S320: Use a feature extractor to extract features from all patches in X, and then perform linear projection on the extracted offline features to obtain the original feature scan sequence. Among them, z i This represents the offline features of the i-th instance.

3. The high-resolution medical pathology WSI image analysis method based on scan-invariant Mamba as described in claim 2, characterized in that: The operation of the sequence enhancement branch in S400 is as follows: S410: Calculate the attention score for each instance in Z using an attention mechanism, defined as follows: A=[a1,…,a i ,…,a N ]=φ(Z) Where φ(·) represents the attention mechanism; S411: Sort A in S410, as shown in the following expression: I=[j1,j2,…,j N ]=Sort(A) Where j1 represents the index of the instance with the highest attention score, j N The index representing the instance with the lowest score; S412: Select the instances with the highest attention scores in the top α% of I as the amplification terms for the Z sequence, ultimately obtaining the new sequence Z. easy The expression is as follows: Where φ(·) is the Mixup on the selected instance, S A (·) indicates a sequence enhancement module. Indicates the index of the augmentation item.

4. The high-resolution medical pathology WSI image analysis method based on scan-invariant Mamba as described in claim 3, characterized in that: The sequence mask branching operation in S400 is as follows: S420: Using an attention masking mechanism, an n-dimensional mask vector M is generated from Z as shown below: M=[m1,…,m r ,…,m N ], Where, m r ∈{0,1}, if m r =1, then the r-instance is unmasked; otherwise, it is masked. S421: Set the mask rates for high and low attention values ​​to β respectively. h % and β l %, therefore, the sequence index corresponding to the high attention value is The sequence index corresponding to low attention value is therefore, Finally, the new sequence Z after masking hard Represented as: From hard ←S M (Z):=M·Z, Among them, S M (·) represents a sequence mask.

Images