A pathological image classification system and method based on Mamba and hybrid expert models

Abstract

This invention discloses a pathological image classification system and method based on Mamba and a hybrid expert model. The system combines the advantages of Mamba and a hybrid expert model for the challenging task of WSI classification and develops a novel hierarchical network based on Mamba, called Mamba-WSI. To improve feature representation while maintaining efficiency, Mamba-WSI introduces bidirectional Mamba blocks within and across pseudo-packets to explore the correlation between instances within pseudo-packets and the global pseudo-packet representation. Simultaneously, this invention proposes a dynamic memory to store historical information related to label prediction and designs a cross-pseudo-packet fusion module to fuse effective information from a dynamically updated dictionary module, enhancing the current pseudo-packet representation and global modeling. Furthermore, the expert hybrid module adaptively coordinates global and local packet representations to achieve robust packet label prediction.

Description

Technical Field

[0001] This invention belongs to the field of pathological image technology and relates to a pathological image classification method, specifically a pathological image classification system and method based on Mamba and a hybrid expert model (Whole Slide Image ClassificationMeets Mamba and Mixture of Experts, abbreviated as Mamba-WSI). Background Technology

[0002] Histopathological slides contain rich information about biological structures, which is crucial for the accuracy of cancer diagnosis. However, diagnosing tissue sections using manual optical microscopy is a time-consuming and tedious task for pathologists. With the advancement of digital technology, tissue sections can be scanned into Whole Slide Images (WSIs), a technology that opens up new possibilities for traditional pathological diagnosis. In particular, deep learning methods have demonstrated advantages over manual feature extraction methods when parsing tissue and cellular structural information in WSIs. Nevertheless, the gigapixel-level high resolution and complex microstructure of WSIs make collecting pixel-level annotations extremely time-consuming and challenging.

[0003] To address these challenges, Multi-Instance Learning (MIL) methods have garnered significant attention due to their reliance on only bag-level labels as weak constraints. This approach treats each Context-Initiated Sequence (WSI) as a "bag," plucking the WSI into a series of instances and mining the correlations between local and global instances to form bag-level feature representations for classifier training. While the MIL paradigm incorporating Convolutional Neural Networks (CNNs) has achieved significant performance advancements, it still faces limitations in learning global perspectives and capturing long-range dependencies. Unlike CNN-based methods, some studies have successfully captured the spatial and morphological information of large-scale instances by introducing self-attention mechanisms. In particular, Transformers have demonstrated advantages in enhancing global context communication, but their computational cost is often high when applied to high-resolution WSIs. For example, a WSI with a resolution of 106,496×51,968 pixels is tiled into 55,852 instances of size 256×256 at a magnification of 20x. Each instance needs to be encoded as a 1024-dimensional feature vector, which makes the computational cost of the popular Transformer-based TransMIL model as high as 152.03 GFLOPs.

[0004] To address the quadratic complexity issue of Transformer when handling WSI, researchers have proposed various methods to reduce computational cost. A common strategy is to apply a random mask to the input instance features, i.e., randomly discarding some input features, thereby shortening the length of long sequences. However, this method inevitably leads to some information loss, which may affect the capture of spatial patterns in tumor regions. Another approach is to group N instances into g pseudo-packets, reducing the original complexity from O(N^2) to O(N^2). 2 Reduced to Where N is the number of instances. Despite these improvements, the quadratic complexity problem of Transformers remains unresolved, still limiting their application on high-resolution WSI.

[0005] To address the computational bottleneck of Transformers in modeling long sequences, Structured State Space Sequences (S4) have garnered significant attention due to their superior performance in linear complexity. However, since the Mamba model was originally designed for unidirectional one-dimensional continuous data, it exhibits limitations in handling spatial hierarchies and local context analysis. To overcome this weakness, Mamba-based variants have achieved excellent results by designing various scanning patterns to balance local and global representations. For example, some researchers have combined Graph Neural Networks (GNNs) with Mamba to capture local and global spatial relationships between instances. However, attempting to package all instance sequences into GNNs or Mamba models often incurs learning and computational burdens, making it difficult for the model to efficiently handle large-scale WSI data. Summary of the Invention

[0006] To address the unsatisfactory results and prohibitive computational costs of existing methods due to the local connectivity in CNNs and the quadratic complexity of Transformers, this invention provides a pathological image classification system and method based on Mamba and a Mixed Expert Model (MoE). This invention proposes combining the advantages of Mamba and mixed expert models for the challenging task of WSI classification and develops a novel hierarchical network based on Mamba, called Mamba-WSI. To improve feature representation while maintaining efficiency, Mamba-WSI introduces bidirectional Mamba blocks within and across pseudo-packets to explore the correlations between instances within pseudo-packets and the global pseudo-packet representation. Simultaneously, this invention proposes a dynamic memory to store historical information related to label prediction and designs a cross-pseudo-packet fusion module to fuse effective information from a dynamically updated dictionary module, enhancing the current pseudo-packet representation and global modeling. Furthermore, an expert hybrid module adaptively coordinates global and local packet representations to achieve robust packet label prediction.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A pathological image classification system based on Mamba and a hybrid expert model includes a whole-section tissue pathological image instance extraction module, an instance feature extraction module, a cross-pseudo-bag fusion module (CPFM), a global and local bag representation module (GLBRM), and an expert mixture module (EMM), wherein:

[0009] The whole-section tissue pathology image extraction instance module is responsible for extracting tissue pathology image instances. The specific steps for extracting tissue pathology image instances are as follows:

[0010] (1) Given a whole slice dataset of pathological images It contains N WSIX i and its corresponding label Y i ;

[0011] (2) The full slice image WSI is read into the computer at a downsampling resolution and converted from RGB to HSV color space;

[0012] (3) The non-background tissue region of WSI is calculated based on the image saturation and channel threshold after median blurring, and the morphological closure operation is combined to fill small gaps and holes as the final tissue region.

[0013] (4) For each WSI X, based on the calculated tissue region i Crop B at a specified magnification. i Image patch instances Get the collection of instances And store, where H and W represent the height and width of an instance, b∈[1,B] i ] represents the b-th clipping instance;

[0014] The instance feature extraction module is responsible for encoding instance features in tissue pathology images. The specific steps for instance feature encoding are as follows:

[0015] (1) Use a deep neural network pre-trained on ImageNet or a large pathological image dataset as an encoder to encode each instance x. i,b Embedded into a D-dimensional feature vector In this process, a set of feature vectors is obtained.

[0016] (2) To facilitate the exploration of spatial and structural relationships between instances, a feature vector set is provided. Introducing learnable positional encoding

[0017] (3) In order to promote the intrinsic correlation of Mamba-WSI learning instance features and improve feature representation, larger packets are randomly sampled into T smaller pseudo-packets. Where M is the number of instances in the pseudo-package. This represents the features of the pseudo-packet instance generated at time t;

[0018] The cross-pseudo-packet fusion module CPFM includes a bidirectional Mamba block (BiMambaB), a cross-attention module (CAM), and a dynamic memory bank (DMB). At time t, a set of instance features is sampled. Capture using BiMambaB The semantic and spatial features between instances are aggregated into the pseudo-packet representation at time t. middle; Historical knowledge is integrated from the dynamic memory bank DMB using the Cross-Attention Module (CPFM). Entering an enhanced pseudo-packet representation In, among them, Representing the DMB at time t-1, enhanced pseudo-packet representation It is passed to a multi-layer perceptron (MLP) to infer the enhanced pseudo-packet representation. Corresponding pseudo-label probability Simultaneously use the package label Y i Act as a pseudo-label to supervise The relevant process is represented as follows:

[0019]

[0020]

[0021]

[0022] The dynamic memory (DMB) stores K discriminative pseudo-packet feature representations. To facilitate the pseudo-packet representation at time t Entering an enhanced pseudo-packet representation in, This represents the K retained pseudo-packet features. It is the probability of its corresponding category. Based on category confidence Ranked from highest to lowest; for DMB at time t-1, using and As input, through comparison Compared with the data stored before time t-1 Rearrange Introduction As a category-related cue and a softmax function, the stored feature representation Transformed into refined feature representation Then, among them Quantization to optimal feature representation Generate weights Subsequently weight do The coefficient and with The addition operation is performed to aggregate the scattered discriminative information into a set of feature representations with the highest information content. The above operation is represented as:

[0023]

[0024]

[0025] Subsequently, the splicing represents The pseudo-packet representation at time t is obtained through cross-attention computation. Provide more tag-related features;

[0026] The local and global packet representation module GLBRM comprises two parallel sub-branches: a bidirectional Mamba block (BiMambaB) and a convolution max-pooling block (ConvBlock). BiMambaB is used to refine pseudo-inter-packet global packet representations. ConvBlock is used to extract salient features from pseudo-packets, known as local packet representation. The above process is encapsulated as:

[0027]

[0028]

[0029] Predicting global bag representations using linear layers and the Softmax function, respectively. Corresponding category probability Local packet representation Corresponding category probability and pass and Refined into more granular packet representations The process is represented as follows:

[0030]

[0031] The expert hybrid module (EMM) comprises two expert modules and a routing network. Expert module 1 is constructed using 1×1 convolutional layers, ReLU, and max pooling, while expert module 2 is constructed using 3×3 convolutional layers, ReLU, and max pooling. The weights for expert module 1 and expert module 2 are generated through a routing network consisting of linear layers and a softmax activation function. and Used to aggregate the outputs of expert module 1 and expert module 2 and The weighted features are then input into a multilayer perceptron for final bag label prediction. The process is described as follows:

[0032]

[0033] A pathological image classification method based on Mamba and a hybrid expert model includes the following steps:

[0034] Step 1: Given a whole slice dataset of pathological images It contains N WSI Xi and its corresponding label Y i X i Consider it as the i-th "packet"; X i B was cut out i One instance;

[0035] Step 2: Set each instance x i,b Embedded into a D-dimensional feature vector In this process, a set of feature vectors is obtained.

[0036] Step 3: To facilitate the exploration of spatial and structural relationships between instances, a feature vector set is created. Introducing learnable positional encoding

[0037] Step 4: Mamba-WSI randomly samples the larger packet into T smaller pseudo-packets.

[0038] Step 5: Capture using BiMambaB The semantic and spatial structural features between instances are aggregated into the current pseudo-packet feature representation.

[0039] Step Six: In order to enhance The discrimination information in Using CPFM to fuse historical features related to predicted labels in the DMB In the middle, MLP is used to predict labels.

[0040] Step 7: Import the package tag Y i Act as a pseudo-label to supervise In order to Extract more label-related feature representations;

[0041] Step 8: Extract pseudo-cell feature representation using the BiMambaB module. Global package representation in ConvBlock preserves pseudo-room feature representation Local package representation

[0042] Step Nine and Based on the corresponding category score and Packet representation refined to a finer granularity Furthermore, EMM is used to achieve more robust packet label prediction.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] 1. This invention proposes a hierarchical fusion and representation learning framework combining group sampling and Mamba. This framework integrates Mamba and MoE for efficient processing of WSI tasks, aiming to address the challenge of modeling long-range instance dependencies in WSI tasks and improve classification performance. Unlike traditional self-attention methods that explore global relationships among all pixels or packets, this Mamba-based hierarchical fusion scheme decomposes the learning task into two sub-tasks: pseudo-packet intra-fusion and cross-pseudo-packet external representation learning.

[0045] 2. In the pseudo-packet fusion task, this invention designs a bidirectional Mamba block to capture instance correlations within the pseudo-packet, thereby generating a richer representation of the current pseudo-packet. To address the sparsity and redundancy issues of the pseudo-packet representation and enhance its discriminative power, this invention introduces a DMB to store historical information about category dependencies. Furthermore, CPFM is used to integrate informative features into the representation of the current pseudo-packet, thereby enhancing the pseudo-packet representation and providing more refined label dependency features.

[0046] 3. In the pseudo-packet representation learning task, this invention combines the representations of all pseudo-packets to extract local and global packet features. Specifically, this invention designs a BiMambaB module to integrate all pseudo-packet representations and extract global packet representations between pseudo-packets. Simultaneously, it designs a ConvBlock with convolution and max-pooling to retain salient information in the pseudo-packets as local packet representations, avoiding mutual interference and homogenization between pseudo-packet representations. By coordinating the advantages of global and local packet representations of BiMambaB and ConvBlock, this invention achieves finer-grained packet-level representations, significantly improving the classification performance of the WSI task.

[0047] 4. To further enhance the applicability and robustness of the model, this invention introduces EMM to mitigate biases in packet representation, thereby improving the accuracy of packet label prediction.

[0048] 5. Experimental results show that Mamba-WSI outperforms the state-of-the-art ACMIL method on the ESCA dataset, achieving an accuracy of 3.8% and an AUC of 4.1%. These results demonstrate that Mamba-WSI has significant advantages in performance and efficiency, and is better able to address the challenges of WSI tasks. Attached Figure Description

[0049] Figure 1 This is a flowchart of pathological image classification based on Mamba and a hybrid expert model;

[0050] Figure 2 It is a bidirectional Mamba module;

[0051] Figure 3 It is a convolutional module;

[0052] Figure 4 It is an expert hybrid module. Detailed Implementation

[0053] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0054] This invention provides a pathological image classification system based on Mamba and a hybrid expert model, such as... Figure 1 As shown, it includes a whole-section histopathological image extraction instance module, an instance feature extraction module, a CPFM module, a GLBRM module, and an EMM module, among which:

[0055] I. Example module for extracting whole-section tissue pathology images:

[0056] The whole-section tissue pathology image extraction instance module is responsible for extracting tissue pathology image instances. The specific steps for extracting tissue pathology image instances are as follows:

[0057] (1) Given a whole slice dataset of pathological images It contains N WSIX i and its corresponding label Y i X i Consider it as the i-th "packet";

[0058] (2) The full slice image WSI is read into the computer at a downsampling resolution and converted from RGB to HSV color space;

[0059] (3) The non-background tissue region of WSI is calculated based on the image saturation and channel threshold after median blurring, and the morphological closure operation is combined to fill small gaps and holes as the final tissue region.

[0060] (4) For each WSI X, based on the calculated tissue region i Crop B at a specified magnification. i Image patch instances Get the collection of instances And store, where H and W represent the height and width of an instance, b∈[1,B] i ] represents the b-th clipping instance;

[0061] II. Instance Feature Extraction Module

[0062] The instance feature extraction module is responsible for encoding instance features in tissue pathology images. The specific steps for instance feature encoding are as follows:

[0063] (1) Use a deep neural network pre-trained on ImageNet or a large pathological image dataset as an encoder to encode each instance x. i,b Embedded into a D-dimensional feature vector In this process, a set of feature vectors is obtained.

[0064] (2) To facilitate the exploration of spatial and structural relationships between instances, a feature vector set is provided. Introducing learnable positional encoding

[0065] (3) In order to promote the intrinsic correlation of Mamba-WSI learning instance features and improve feature representation, larger packets are randomly sampled into T smaller pseudo-packets. Where M is the number of instances in the pseudo-package. This represents the features of the pseudo-packet instance generated at time t.

[0066] III. CPFM Module

[0067] The purpose of CPFM is to effectively and efficiently model instance relationships to facilitate pseudo-packet instance fusion and packet feature representation, including BiMambaB, CAM, and DMB. The specific operation steps are as follows:

[0068] A set of instance features sampled at time t Taking operation as an example, first design a Figure 2 The BiMambaB module shown consists of a two-layer bidirectional Mamba structure and a linear attention module (LAM) to fully explore the complex relationships between instances within the pseudo-packet and aggregate them into the pseudo-packet representation at time t. middle.

[0069] In order to deal with the current fake packet representation The sparsity of disease characteristics in China Equipped with a cross-attention module (CAM) to integrate historical knowledge from the dynamic memory bank (DMB). Entering an enhanced pseudo-packet representation In the middle. Among them, This represents the DMB at time t-1.

[0070] Finally, the enhanced pseudo-packet representation It is passed to a multi-layer perceptron (MLP) constructed from linear layers and a softmax activation function to infer the enhanced pseudo-bag representation. Corresponding pseudo-label probability The relevant process is represented as follows:

[0071]

[0072]

[0073] The Cross-Attention operation is decomposed into:

[0074]

[0075] Q, K, and V represent the query, key, and value in the cross-attention mechanism, respectively. This represents the DMB at time t-1. This represents a pseudo-packet feature representation at time t. Let t represent the set of instance features formed by sampling M instance features at time t.

[0076] DMB aims to preserve K discriminative pseudo-packet feature representations. To facilitate the pseudo-packet representation at time t Entering an enhanced pseudo-packet representation in, This represents the K retained pseudo-packet features. It is the probability of its corresponding category. Based on category confidence Rank them from highest to lowest. For DMB at time t-1, use... and As input, through comparison Compared with the data stored before time t-1 To rearrange While increasing K may be beneficial, redundancy and computational burden can degrade model performance. Therefore, this invention introduces... As a category-related cue and a softmax function, the stored feature representation Transformed into refined feature representation Then, among them Quantization to optimal feature representation Generate weights Subsequently weight do The coefficient and with The addition operation is performed to aggregate the scattered discriminative information into a set of feature representations with the highest information content. The above operation is represented as:

[0077]

[0078]

[0079] Subsequently, the splicing represents Cross-Attention operation is used to... Provide more tag-related features.

[0080] IV. GLBRM Module

[0081] The CCRM module aims to mitigate pseudo-packet representation bias and promote more comprehensive packet feature representation. It comprises two parallel sub-branches: a bidirectional Mamba block (BiMambaB) and a convolutional and max-pooling block (ConvBlock). The structure of ConvBlock is as follows: Figure 3 As shown. BiMambaB is used to extract the global packet representation between pseudo-packets. To avoid homogenization of pseudo-packet representations, ConvBlock is used to preserve and integrate the best features within the pseudo-packet, which is called local packet representation.

[0082] The above process is encapsulated as:

[0083]

[0084]

[0085] To achieve more accurate packet representation and improve packet label prediction accuracy, this invention predicts global packet representations using both a linear layer and a Softmax function. Corresponding category probability Local packet representation Corresponding category probability and pass and Refined into more granular packet representations The process is represented as follows:

[0086]

[0087] V. EMM Module

[0088] like Figure 4As shown, the expert hybrid module (EMM) includes two expert modules and a routing network. Expert module 1 is constructed using 1×1 convolutional layers, ReLU, and max pooling, while expert module 2 is constructed using 3×3 convolutional layers, ReLU, and max pooling. The weights of expert module 1 and expert module 2 are generated through a routing network consisting of linear layers and a softmax activation function. and Used to aggregate the outputs of expert module 1 and expert module 2 and The weighted features are then input into a multilayer perceptron for final bag label prediction. The process is described as follows:

[0089]

[0090]

[0091] VI. Loss Function

[0092] To achieve efficient global instance modeling and high-quality feature representation for better packet label inference, this invention explores composite cross-entropy loss functions, which involve: optimizing the pseudo-packet label probability loss corresponding to the pseudo-packet fusion module. The commonality and feature representation modules represent the corresponding label probability loss in their commonality package. The feature package represents the corresponding label probability loss. Label probability loss corresponding to expert hybrid module The formulas for these loss functions are as follows:

[0093]

[0094]

[0095] The final total loss is defined as:

[0096]

[0097] Where λ is used to adjust the pseudo-label probability loss. In total losses The proportion of Y was adjusted from 0 to 0.2 by integrating the duration and cosine curve. i is the real label corresponding to WSI, T is the number of generated pseudo-packets, and N is the number of WSIs contained in the dataset.

[0098] This invention also provides a pathological image classification method based on Mamba and a hybrid expert model, the method comprising the following steps:

[0099] Step 1: Given a whole slice dataset of pathological images It contains N WSI X i and its corresponding label Y i X i Consider it as the i-th "packet"; X i B was cut out i One example.

[0100] Step 2: Set each instance x i,b Embedded into a D-dimensional feature vector In this process, a set of feature vectors is obtained.

[0101] Step 3: To facilitate the exploration of spatial and structural relationships between instances, features are assigned to all instances. Introducing learnable positional encoding

[0102] Step 4: Mamba-WSI randomly samples the larger packet into T smaller pseudo-packets. in This represents the pseudo-packet instance features sampled at time t. M is the number of instances in the pseudo-package.

[0103] Step 5: Capture using BiMambaB The semantic and spatial structural features between instances are aggregated into the current pseudo-packet feature representation.

[0104] Step Six: In order to enhance The discrimination information in Using CPFM to fuse historical features related to predicted labels in the DMB In the middle, MLP is used to predict labels.

[0105] Step 7: Import the package tag Y i Act as a pseudo-label to supervise In order to Extract more label-related feature representations.

[0106] Step 8: To improve global packet representation, use the BiMambaB module to extract pseudo-packets. Inter-feature representations are summarized into a global package representation. ConvBlock retains fake packages Inter-feature representation, summarized into local bag representation

[0107] Step Nine and Based on the corresponding category probability and Packet representation refined to a finer granularity Furthermore, EMM is used to achieve more robust packet label prediction.

[0108] To verify the performance of the Mamba-WSI model, extensive experimental comparisons were conducted on the ESCA dataset with several typical WSI analysis MIL methods. These methods included MeanPooling, ABMIL, SetTransformer, DeepAttnMIL, TransMIL, MuRCL, MHIM, SSMMIL, MaxPooling, RNNMIL, DSMIL, CLAM, DTFD, DGMIL, IAT, BCL, CIMIL, SSMMIL, and MambaMIL. Evaluation metrics included Area Under the Curve (AUC), Accuracy, Precision, and Recall, with a threshold set to 0.5. The Mamba-WSI model was trained for 200 iterations using the AdaMax optimizer with an initial learning rate of 0.0004. Quantitative results are shown in Table 1. The Mamba-WSI model achieved competitive performance across all evaluation metrics. In particular, it outperformed the recent state-of-the-art ACMIL method by 3.8% in accuracy and 4.1% in AUC.

[0109] Table 1: Quantitative comparison of the Mamba-WSI results proposed in this invention with representative multi-instance learning methods on the ESCA dataset.

[0110]

[0111]

Claims

1. A pathological image classification system based on Mamba and a hybrid expert model, characterized in that... The pathological image classification system includes a whole-section tissue pathological image extraction instance module, an instance feature extraction module, a cross-pseudo-packet fusion module CPFM, a local and global packet representation module GLBRM, and an expert hybrid module EMM, wherein: The whole-section tissue pathology image extraction instance module is responsible for extracting tissue pathology image instances; The instance feature extraction module is responsible for encoding instance features of tissue pathology images; The cross-pseudo-packet fusion module CPFM includes a bidirectional Mamba module BiMambaB, a cross-attention module CAM, and a dynamic memory bank DMB. At time t, it samples a set of instance features. Capture using BiMambaB The semantic and spatial features between instances are aggregated into the pseudo-packet representation at time t. middle; Integrating historical knowledge from DMB using CPFM Entering an enhanced pseudo-packet representation In, among them, Representing the DMB at time t-1, enhanced pseudo-packet representation It is passed to a multilayer perceptron (MLP) to infer the enhanced pseudo-packet representation. Corresponding pseudo-label probability Simultaneously use the package label Y i Act as a pseudo-label to supervise The relevant process is represented as follows: The dynamic memory (DMB) stores K discriminative pseudo-packet feature representations. To facilitate the pseudo-packet representation at time t Entering an enhanced pseudo-packet representation in, This represents the K retained pseudo-packet features. It is the probability of its corresponding category. Based on category confidence Ranked from highest to lowest; for DMB at time t-1, using and As input, through comparison Compared with the data stored before time t-1 Rearrange Introduction As a category-related cue and a softmax function, the stored feature representation Transformed into refined feature representation Then, among them Quantization to optimal feature representation Generate weights Subsequently weight do The coefficient and with The addition operation is performed to aggregate the scattered discriminative information into a set of feature representations with the highest information content. The above operation is represented as: Subsequently, the splicing represents The pseudo-packet representation at time t is obtained through cross-attention computation. Provide more tag-related features; The local and global packet representation module GLBRM contains two parallel sub-branches: the bidirectional Mamba module BiMambaB and the convolution and max pooling module ConvBlock. BiMambaB is used to refine the pseudo-inter-packet global packet representation. ConvBlock is used to extract salient features from pseudo-packets, known as local packet representation. The above process is encapsulated as: Predicting global bag representations using linear layers and the Softmax function, respectively. Corresponding category probability Local packet representation Corresponding category probability and pass and Refined into more granular packet representations The process is represented as follows: The expert hybrid module (EMM) comprises two expert modules and a routing network. Expert module 1 is constructed using 1×1 convolutional layers, ReLU, and max pooling, while expert module 2 is constructed using 3×3 convolutional layers, ReLU, and max pooling. The weights for expert module 1 and expert module 2 are generated by the routing network, which consists of linear layers and a softmax activation function. and Used to aggregate the outputs of expert module 1 and expert module 2 and The weighted features are then input into a multilayer perceptron for final bag label prediction. The process is described as follows:

2. The pathological image classification system based on Mamba and hybrid expert models according to claim 1, characterized in that... The specific steps for extracting the histopathological image instances are as follows: (1) Given a whole slice dataset of pathological images It contains N WSIX i and its corresponding label Y i ; (2) The full slice image WSI is read into the computer at a downsampling resolution and converted from RGB to HSV color space; (3) The non-background tissue region of WSI is calculated based on the image saturation and channel threshold after median blurring, and the morphological closure operation is combined to fill small gaps and holes as the final tissue region. (4) For each WSI X, based on the calculated tissue region i Crop B at a specified magnification. i Image patch instances Get the collection of instances And store, where H and W represent the height and width of an instance, b∈[1,B] i ] represents the b-th clipping instance.

3. The pathological image classification system based on Mamba and hybrid expert models according to claim 1, characterized in that... The specific steps for encoding the instance features are as follows: (1) Use a deep neural network pre-trained on ImageNet or a large pathological image dataset as an encoder to encode each instance x. i,b Embedded into a D-dimensional feature vector In this process, a set of feature vectors is obtained. (2) To facilitate the exploration of spatial and structural relationships between instances, a feature vector set is provided. Introducing learnable positional encoding (3) In order to promote the intrinsic correlation of Mamba-WSI learning instance features and improve feature representation, larger packets are randomly sampled into T smaller pseudo-packets. Where M is the number of instances in the pseudo-package. This represents the features of the pseudo-packet instance generated at time t.

4. A method for classifying pathological images based on Mamba and a hybrid expert model using the pathological image classification system according to any one of claims 1-3, characterized in that... The method includes the following steps: Step 1: Given a whole slice dataset of pathological images It contains N WSI X i and its corresponding label Y i X i Consider it as the i-th "packet"; X i B was cut out i One instance; Step 2: Set each instance x i,b Embedded into a D-dimensional feature vector In this process, a set of feature vectors is obtained. Step 3: To facilitate the exploration of spatial and structural relationships between instances, a feature vector set is created. Introducing learnable positional encoding Step 4: Mamba-WSI randomly samples the larger packet into T smaller pseudo-packets. Step 5: Capture using BiMambaB The semantic and spatial structural features between instances are aggregated into the current pseudo-packet feature representation. Step Six: In order to enhance The discrimination information in Using CPFM to fuse historical features related to predicted labels in the DMB In the middle, MLP is used to predict labels. Step 7: Import the package tag Y i Act as a pseudo-label to supervise In order to Extract more label-related feature representations; Step 8: Extract pseudo-cell feature representation using the BiMambaB module. Global package representation in ConvBlock preserves pseudo-room feature representation Local package representation Step Nine and Score by corresponding category and Packet representation refined to a finer granularity Furthermore, EMM is used to achieve more robust packet label prediction.

Images