Pathological image classification method based on state space duality

By introducing a multi-instance learning method Mamba2MIL, a state space duality and sequence enhancement, in pathological image analysis, the problem of inefficiency of traditional pathological image analysis is solved, efficient pathological image classification is achieved, and diagnostic accuracy and analysis are significantly improved. efficiency.

CN119048825AActive Publication Date: 2024-11-29BEIHANG UNIV

Patent Information

Application Number
CN202411170605.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-29
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

Traditional pathological image analysis has problems of inefficiency and error-prone, especially in automated analysis of full-slice images, which makes it difficult to effectively process high-resolution and large-scale pathological image data.

Method used

A multi-instance learning method based on state space duality is proposed. Through sequence enhancement and feature weighting selection mechanism, combined with sequence correlation and sequence-independent features, a sequence conversion method adapted to different WSI sizes is designed to extract more Distinguished representation and pathological image classification by computer-executing training models.

Benefits of technology

It significantly improves the classification accuracy and analysis efficiency of pathological images, reduces the computational complexity, and can effectively process long sequence data, achieving better performance on NSCLC and BRACS datasets than existing methods, especially in tumor classification tasks. High AUC and ACC values ​​were achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048825A_ABST
    Figure CN119048825A_ABST
Patent Text Reader

Abstract

The invention provides a computational pathology multi-instance learning (MIL) method based on state space duality (SSD), and provides a novel Mamba2MIL model, and the model can comprehensively utilize sequence dependence and sequence independence characteristics in WSIs to improve the classification accuracy of pathological images. The invention relates to the field of computational pathology, in particular to a multi-instance learning method based on state space duality (SSD), and the multi-instance learning method is used for automatic analysis and diagnosis of full slice images (WSIs). The invention is in the field of image data processing (G06T) and medical diagnosis (A61B). According to the model, through preprocessing, feature extraction, serialization processing and application of a state space dual model, classification decision is finally carried out through a multi-layer perceptron, long sequence data is effectively processed, and the diagnosis accuracy is improved. Experiments prove that the Mamba2MIL method has important application value in the aspect of improving pathological image analysis efficiency and diagnosis accuracy, a valuable auxiliary tool is provided for pathologists, and the Mamba2MIL method has wide clinical application prospects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computational pathology, in particular to a multi-instance learning method based on state space duality (SSD) for automatic analysis and diagnosis of whole slide images (WSIs). The present invention belongs to the field of image data processing (G06T) and medical diagnosis (A61B). Background Art

[0002] As an important branch of modern medicine, pathology undertakes the key tasks of disease diagnosis, classification, grading and prognosis assessment. By observing tissue samples under a microscope, pathology reveals the nature and development of the disease, and provides clinicians with a scientific basis for formulating treatment plans and evaluating patient treatment outcomes. However, traditional pathology diagnosis relies on the professional knowledge and experience of pathologists, and is subjective and labor-intensive. With the increase in medical needs, the workload of pathologists continues to rise, which not only increases their work pressure, but also puts higher demands on the accuracy and efficiency of diagnosis.

[0003] The rise of computational pathology has revolutionized pathology diagnosis. Through high-resolution scanning technology, traditional slide images from optical microscopes can be converted into digital images, namely whole slide images (WSIs). WSIs not only retain all the information of traditional slides, but also provide higher resolution and a wider field of view, allowing pathologists to diagnose and analyze at a more macroscopic level. The application of digital pathology has promoted the digitization and informatization of pathology diagnosis and facilitated remote diagnosis, education and research.

[0004] The development of computational pathology, especially the introduction of deep learning technology, has provided new possibilities for the automatic analysis of pathological images. As the core of deep learning, convolutional neural networks (CNNs) have shown excellent performance in image classification, detection, and segmentation tasks by automatically extracting local features of images and constructing more complex and abstract feature representations layer by layer. However, the high resolution and large-scale data characteristics of pathological images pose challenges to computer processing. Traditional computer vision algorithms may encounter limitations in computing resources and storage space when processing these large-scale images.

[0005] To solve this problem, researchers began to explore the multiple instance learning (MIL) framework. The MIL method regards each WSI as a set of multiple instances (patches), each of which can be a region or a cell in the image. The core idea of ​​MIL is that if at least one instance in the set is positive (that is, contains disease features), the entire WSI is classified as positive; if all instances are negative, the entire WSI is classified as negative. This method allows the model to learn from partially annotated data because it only needs to identify at least one instance containing the disease to classify the entire image.

[0006] In addition, the introduction of attention mechanisms and Transformer models has brought new perspectives to pathological image analysis. These models are able to handle long-distance dependency problems and capture the relationship between different regions in the image. The methods of deep separable convolution and lightweight network structure improve the processing speed by reducing model parameters and computational complexity while maintaining relatively high performance.

[0007] Recurrent neural networks (RNN) and long short-term memory networks (LSTM) play a role in processing pathological image data with time series characteristics. They can capture dynamic features in time series and are suitable for time series analysis in pathological images. Generative adversarial networks (GAN) show great potential in image synthesis and enhancement and can be used for image denoising, super-resolution and data enhancement.

[0008] Graph neural networks (GNNs) are suitable for processing graph-structured data, and they show unique advantages in processing pathological images with complex relationship networks. GNNs can capture the spatial relationships and interactions between cells, providing a new perspective for pathological image analysis. Ensemble learning methods improve overall performance by combining the predictions of multiple models, and can combine the advantages of different network structures or different feature extraction methods to improve the accuracy of classification or segmentation tasks.

[0009] End-to-end deep learning models can learn the final diagnosis results directly from raw image data without manually designing features or performing complex preprocessing steps. These models usually include multiple steps such as data preprocessing, feature extraction, classification or segmentation, realizing an automated process from input to output.

[0010] With the continuous advancement of technology, the field of pathology image analysis is experiencing rapid development and innovation. The introduction of deep learning technology not only improves the accuracy and efficiency of diagnosis, but also provides pathologists with powerful auxiliary tools, which helps to promote the development of pathology in a more automated and intelligent direction. In the future, we can foresee that more innovative models and methods will be developed to meet the various challenges in pathology image analysis. Summary of the invention

[0011] The purpose of the present invention is to propose a new multi-instance learning method, called Mamba2MIL, for improving the analysis efficiency and diagnostic accuracy of WSIs. The Mamba2MIL method combines sequence-dependent and sequence-independent features through sequence enhancement to extract more discriminative representations, and introduces a sequence conversion method that adapts to different WSI sizes to enhance the retention of local sequence information. The method provided by the present invention is all executed by a computer to train the designed Mamba2MIL model. The trained Mamba2MIL model is executed by a computer to process pathological images to obtain classification results.

[0012] Technical Solution

[0013] The purpose of the present invention is to solve the problems of low efficiency and easy errors in traditional pathological image analysis, especially the problem of automated analysis of whole slide images (WSIs). Based on the multiple instance learning (MIL) method of state space duality (SSD), the present invention proposes a novel Mamba2MIL model, which can comprehensively utilize the sequential dependence and sequence independence features in WSIs to improve the classification accuracy of pathological images.

[0014] The Mamba2MIL model designed in the present invention is compared with the structured state space sequence (S4) (L. Fillioux, J. Boyd, M. Vakalopoulou, P.-H. Courn`ede, and S. Christodoulidis, "Structured statespace models for multiple instance learning in digital pathology," in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2023, pp. 594–

[0015] 604.) and the structure of the selective scanning space state sequence model (SSM, i.e. Mamba) (S. Yang, Y. Wang, and H. Chen, "Mambamil: Enhancing long sequence modeling with sequence reordering incomputational pathology," arXiv preprint arXiv:2403.06800, 2024.), further optimizes the state space operation, reduces the computational complexity, and effectively processes long sequence data. The Mamba2MIL model extracts more discriminative feature representations through sequence enhancement and feature weighted selection mechanisms, thereby enhancing the classification ability of WSIs.

[0016] The present invention adopts the above-mentioned Mamba2MIL model to pre-process WSIs, eliminate background interference, segment them into multiple instances, and use a pre-trained deep learning model (such as ResNet50) to embed features in each instance. The model serializes these features, generates sequences of different orders through square and reorder operations, and then effectively models them through the SSD module, and finally obtains the classification results through feature weighted selection and multi-layer perceptron (MLP) classifier.

[0017] The present invention comprises the following steps:

[0018] Step 1: Create training data:

[0019] 1. Obtain sample data from the pathology image dataset. The NSCLC dataset contains H&E stained pathology images of lung squamous cell carcinoma and lung adenocarcinoma. The dataset contains two subtypes: lung squamous cell carcinoma (LUSC) and lung adenocarcinoma (LUAD). A total of 1052 samples are included, with image sizes ranging from 10,000×4617 pixels to 191,352×97078 pixels, and the average sample size is about 3,004 image blocks. BRACS is a dataset containing breast tissue pathology images, especially H&E (hematoxylin-eosin) stained images. The dataset contains seven subtypes: normal (glandular tissue samples without lesions), pathological benign (PB), common ductal epithelial hyperplasia (UDH), flat epithelial cell atypia (FEA), atypical ductal epithelial hyperplasia (ADH), ductal carcinoma in situ (DCIS), and invasive carcinoma (IC). The dataset contains a total of 537 samples, with image sizes ranging from 23,799×15,455 pixels to 181,272×88,334 pixels, and an average sample size of approximately 2,587 image patches. The two datasets were used to verify the effectiveness of the Mamba2MIL model. In the experimental part, the researchers used 5-fold cross validation to evaluate the model performance. In terms of data segmentation, the ratio of 8:1:1 was used to divide the training set, validation set, and test set.

[0020] 2. Preprocess the whole slice images (WSIs) by dividing the images into non-overlapping patches of consistent size and filtering out the background areas to ensure the validity and consistency of the model input data.

[0021] Step 2: Train the Convolutional Neural Network Model

[0022] 1. Build the Mamba2MIL model:

[0023] The Mamba2MIL model is a multi-instance learning framework designed specifically for whole-slice image analysis in computational pathology. The design concept of this model is to make full use of local features in WSIs and improve the accuracy of pathological diagnosis by deep learning these features. The model process starts with image preprocessing, feature extraction, serialization, application of state-space dual model, and finally classification decision-making through multi-layer perceptron.

[0024] In the preprocessing stage, the WSIs image is first segmented into multiple patches, which are local areas of the image, usually 512×512 pixels in size. These patches are then extracted through a pre-trained deep learning model (such as ResNet50) to obtain a high-dimensional feature vector sequence. Since the dimensionality of the original feature vector is very high (for example, 1024 dimensions), directly processing these high-dimensional data will result in large computations and inefficiency. Therefore, the model uses a linear layer to reduce the dimensionality of the feature vector and compress it to a lower dimension, such as from 1024 dimensions to 512 dimensions. This step not only reduces the amount of computation, but also helps to remove redundant information in the features.

[0025] Next, the model uses square sequences and reorder sequences to normalize the sizes of different WSIs. Specifically, after the WSIs are divided into patches, they are embedded into feature vectors (also called instance-level features) using a pre-trained ResNet. This process produces a sequence instance feature S = {x1, x2, …, x N}, where x i ∈R D represents the instance feature, N represents the sequence length, and D represents the feature dimension. The square sequence is converted into the original sequence S by the Concat operation. l Concatenate with the padding feature vector to form a new sequence S', whose length L is the square root of the original sequence length N rounded up. This step is done by the formula S' = Concat(S l ,(x1,x2,…,x M ))) implementation, where And M = Kn to ensure that WSIs of different sizes can be processed uniformly. i represents the intensity feature, Concat represents the feature concatenation operation, N represents the original sequence length, L represents the sequence length after the sequence is squared, and M represents the length of padding in the sequence square operation.

[0026] The reordered sequence generates three different sequences by reordering the patches in the pathological image: the original sequence (S o ), reverse sequence (S f ) and the transposed sequence (S t ). These sequences are modeled by stacking SSD modules to generate features F corresponding to each sequence. O 、F f and F t , to capture the feature information in different orders. The formula is as follows:

[0027] F=Linear(SSD(Conv1D(Linear(S))))

[0028] F∈{F O ,F f ,F t},S∈{S o ,S f ,S t}

[0029] The state-space dual model (SSD) is the core part of Mamba2MIL, which is used to effectively model long sequences of WSIs patches. The SSD algorithm is based on the simplified SSM and achieves efficient processing of sequences by specially designing the parameters of three matrices (A, B, C). In SSD, matrix A is designed as a diagonal matrix with all diagonal elements being the same and other elements being zero, so that it can be simplified to a scalar parameter. This design significantly reduces the computational complexity of the model while maintaining the model's ability to capture sequence information. The core formula of the SSD model is:

[0030] x t =Ax t-1 +Bu t

[0031] y t =Cx t

[0032] Among them, x t is the hidden state at time t, y t is the output, u t is the input signal, A, B, and C are the state transition matrix, input matrix, and output matrix respectively.

[0033] After using stacked SSD blocks to capture the features of different sequences, we use the hyperbolic tangent activation function to implement weighted feature selection. Feature selection is implemented through the hyperbolic tangent activation function (Tanh) and the linear layer (Linear). Specifically, as shown in the formula:

[0034] F c =Concat(F O ,F f ,F t )F c ′=Linear(tanh(Linear(F c )))

[0035] R=F c ×Softmax(F c ′)

[0036] where F c represents the concatenated features, Fc ′ represents the weight vector, and R represents the result after weighted feature selection. The input features are mapped to a lower dimensional space through a linear layer. Subsequently, the Tanh activated features are mapped into weight vectors through another linear layer. These weights are normalized by the Softmax function and multiplied with the concatenated features to obtain weighted feature representations.

[0037] Finally, the model performs further nonlinear transformation on the weighted feature representation through a multi-layer perceptron (MLP) block to generate the final classification result.

[0038] 2. Set the training parameters, use the Adam optimizer, set the learning rate to 0.0002 and the maximum number of iterations to 200, and use the early stopping technique, that is, when the number of training epochs is greater than or equal to 50, and the value of the validation set AUC for more than 20 training cycles is not higher than the value of the best validation set AUC. The training process includes forward propagation to calculate the loss function and back propagation to update the weights of the neural network. Training is continued until the preset convergence conditions are reached.

[0039] 3. After the training is completed, the final trained model parameters are saved for subsequent automatic diagnosis tasks of actual pathological images.

[0040] The present invention improves the MIL model based on a thorough analysis of the deficiencies of the existing model structure. Extensive experimental verification was carried out on the NSCLC and BRACS datasets. The experimental results show that the Mamba2MIL method achieves better performance than existing advanced methods in tumor classification tasks. Specifically, on the NSCLC dataset, Mamba2MIL achieves an AUC of 0.9533 and an ACC of 0.8794; on the BRACS dataset, Mamba2MIL achieves an AUC of 0.7986 and an ACC of 0.4981. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The overall process diagram of the Mamba2MIL method is shown, including steps such as data preprocessing, feature extraction, sequence transformation, SSD modeling, feature weighted selection, and classification prediction.

[0042] Figure 2 The structural diagram of the Mamaba2MIL model is shown.

[0043] Figure 3 Two key steps, “Square Sequence” and “Transpose Sequence”, in processing the local feature sequences of whole slice images (WSIs) in the Mamba2MIL model are demonstrated.

[0044] Figure 4 The comparison between Mamba2MIL and the existing model results on the dataset is shown. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the implementation modes and the accompanying drawings.

[0046] 1. Build a training data set:

[0047] The present invention uses the NSCLC dataset from The Cancer Genome Atlas (TCGA) (https: / / www.cancer.gov / ccg / research / genome-sequencing / tcga) and the BRACS dataset (https: / / www.bracs.icar.cnr.it / download / ), which contain WSIs of tumor tissues annotated by pathologists. The NSCLC dataset focuses on two subtypes of lung cancer: lung squamous cell carcinoma (LUSC) and lung adenocarcinoma (LUAD). A total of 1052 samples are included, with image sizes ranging from 10,000×4617 pixels to 191,352×97078 pixels, and an average sample size of approximately 3,004 image blocks. The BRACS dataset is an H&E (hematoxylin-eosin) stained image. The dataset contains seven subtypes: 40 normal (glandular tissue samples without lesions), 145 pathological benign (PB), 70 common ductal epithelial hyperplasia (UDH), 41 flat epithelial cell atypia (FEA), 48 atypical ductal epithelial hyperplasia (ADH), 61 ductal carcinoma in situ (DCIS), and 132 invasive carcinoma (IC). The dataset contains a total of 537 samples, with image sizes ranging from 23,799×15,455 pixels to 181,272×88,334 pixels, and an average sample size of approximately 2,587 image patches. The two datasets were used to verify the effectiveness of the Mamba2MIL model. In the experimental part, the researchers used 5-fold cross validation to evaluate the model performance. In terms of data segmentation, a ratio of 8:1:1 was used to divide the training set, validation set, and test set. Each WSI was preprocessed, including denoising, contrast adjustment, and background subtraction. These steps are to improve image quality and ensure the accuracy of feature extraction. The preprocessed WSIs are split into patches of uniform size, for example, the size of each patch can be 512×512 pixels. This step is to convert large-scale images into a format suitable for deep learning models. Using the pre-trained deep learning model, ResNet50, feature extraction is performed on each patch to generate a fixed-dimensional feature vector.

[0048] 2. Model setup

[0049] like Figure 2 As shown in the figure, since the dimensionality of the original feature vector is very high (for example, 1024 dimensions), directly processing these high-dimensional data will result in large computational complexity and inefficiency. Therefore, the model uses a linear layer to reduce the dimensionality of the feature vector and compress it to a lower dimension, such as from 1024 dimensions to 512 dimensions. This step not only reduces the amount of calculation, but also helps to remove redundant information in the features. Next, the model uses the method of square sequence and reorder sequence to standardize the size of different WSIs. Specifically, as Figure 3 As shown, the square sequence is converted into the original sequence S by the Concat operation. l Concatenate with the padding feature vector to form a new sequence S', whose length L is the square root of the original sequence length N rounded up. This step is done by the formula S' = Concat(S l ,(x1,x2,…,x M ) is implemented, where And M = LN to ensure that WSIs of different sizes can be processed uniformly. i represents the intensity feature, Concat represents the feature concatenation operation, N represents the original sequence length, L represents the sequence length after the sequence is squared, and M represents the length of padding in the sequence square operation.

[0050] The reordered sequence generates three different sequences by reordering the patches in the pathological image: the original sequence (S o ), reverse sequence (S f ) and the transposed sequence (S t ). These sequences are modeled by stacking SSD modules to generate features F corresponding to each sequence. O 、F f and F t , to capture feature information in different orders. The final feature representation is completed by combining the linear layer (Linear) and the convolutional layer (Casual Conv1D), and the formula is as follows:

[0051] F=Linear(SSD(Conv1D(Linear(S))))

[0052] F∈{F O ,F f ,F t},S∈{S o ,S f ,S t}

[0053] Among them, the state space dual model (SSD) is the core part of Mamba2MIL, which is used to effectively model WSIs patches of long sequences. The SSD algorithm is based on the simplified SSM, and achieves efficient processing of sequences by specially designing the parameters of three matrices (A, B, C). In SSD, matrix A is designed as a diagonal matrix with all diagonal elements being the same and other elements being zero, so that it can be simplified to a scalar parameter. This design significantly reduces the computational complexity of the model while maintaining the model's ability to capture sequence information. The core formula of the SSD model is:

[0054] x t =Ax t-1 +Bu t

[0055] y t =Cx t

[0056] Among them, x t is the hidden state at time t, y t is the output, u t is the input signal, A, B, and C are the state transition matrix, input matrix, and output matrix respectively.

[0057] Feature selection is another key part of the model. Figure 2 As shown in Figure 1, it is a combination of layer normalization, feature compression, softmax function and multiplication operation, which is implemented by hyperbolic tangent activation function (Tanh) and linear layer (Linear). Figure 2 ×L) in the SSD block to capture the features of different sequences and then perform concatenation (i.e. Figure 2 c) in , stacking and concatenation are to merge these feature vectors or matrices in a certain dimension for the next step of feature selection or classification. Then we use the hyperbolic tangent activation function to implement weighted feature selection. Feature selection is implemented through the hyperbolic tangent activation function (Tanh) and the linear layer (Linear). Specifically, as shown in the formula:

[0058] F c =Concat(F O ,F f ,F t )F c ′=Linear(tanh(Linear(F c )))

[0059] R=F c ×Softmax(F c ′)

[0060] Among them, F crepresents the concatenated features, F c ′ represents the weight vector, and R represents the result after weighted feature selection. The result of the linear transformation is nonlinearly activated by the hyperbolic tangent activation function (Tanh). First, a linear layer is used to perform layer normalization and feature compression operations to map the input features to a lower dimensional space. Subsequently, the Tanh-activated features are mapped into weight vectors through another linear layer. These weights are normalized by the softmax function and multiplied with the concatenated features to obtain weighted feature representations.

[0061] Finally, the model performs further nonlinear transformation on the weighted feature representation through a multi-layer perceptron (MLP) block to generate the final classification result. In the MLP layer, the input feature vector first enters the first fully connected layer, and the weights and bias parameters of this layer are learned and updated through the back-propagation algorithm during the training process. The calculation of each neuron can be expressed as z = Wx + b, where W is the weight matrix, b is the bias vector, and x is the input feature. Then, this linear output z passes through the ReLU activation function, which sets all negative values ​​to zero and retains positive values, thereby achieving nonlinear transformation. After the ReLU activation function, the feature vector enters the next layer, and this process is repeated in all hidden layers of the MLP. The output of each layer becomes the input of the next layer. After the last hidden layer of the MLP, the feature vector enters an output layer, which is usually also a fully connected layer, but its output does not pass through the nonlinear activation function. The purpose of this output layer is to map the features to the final category space. After the output layer, a Softmax function is usually used to generate the predicted probability of each category. The Softmax function converts the raw output of the output layer (also called logits) into a probability distribution so that the sum of the probabilities of all categories is 1. In this way, for a given input sample, the model can predict the probability of each category and select the category with the highest probability as the final classification result. When training the MLP layer, the cross entropy loss function is used to measure the difference between the predicted probability distribution and the true label. Optimizers such as Adam automatically update the weights and bias parameters in the MLP layer according to the gradient of the loss function to minimize the prediction error. In this way, the MLP layer achieves efficient classification of full-slice images in the Mamba2MIL model.

[0062] 3. Set parameter training

[0063] The Adam optimizer is used with a learning rate of 0.0002 and a maximum number of iterations of 200, which means that a ratio of 0.0002 is used to adjust the step size when performing the gradient descent algorithm to update the model weights. The Adam optimizer is an optimization algorithm with an adaptive learning rate. It combines the concepts of momentum and RMSProp and usually performs well when training deep learning models. The early stopping technique is used, that is, when the number of training epochs is greater than or equal to 50, and the value of the validation set AUC for more than 20 training cycles is not higher than the value of the best validation set AUC. The training process includes forward propagation to calculate the loss function, and backpropagation to update the weights of the neural network. Training is performed until the preset convergence condition is reached.

[0064] 4. If Figure 4 As shown, we further visualize the performance difference between our model and other models on the BRACS validation set to evaluate the model. When evaluating, we use accuracy (ACC) and area under the curve (AUC). ACC represents the proportion of samples correctly predicted by the model to the total number of samples. AUC measures the performance of the model at all possible classification thresholds. The higher the AUC value, the better the classification ability of the model. Figure 4 -(ac) shows that our proposed method outperforms TransMIL in validation set AUC and ACC during 5-fold cross validation, with a more stable loss behavior, although the loss decreases not as fast as TransMIL. In addition, Figure 4 -(de) As shown, although the performance of our proposed method is very close to that of MambaMIL in terms of validation set AUC and ACC, it performs better on the test set, indicating stronger generalization ability. Figure 4 -f shows that the loss decreases faster and more steadily in our proposed model compared to MambaMIL, highlighting the efficiency of our approach.

[0065] 5. After the training is completed, save the final trained model parameters for subsequent automatic diagnosis tasks of actual pathological images. Using the feature vector corresponding labels, lung cancer and breast cancer can be diagnosed. The breast cancer classification includes seven subtypes: normal (glandular tissue specimens without lesions), pathological benign (PB), usual ductal hyperplasia (UDH), flat epithelial atypia (FEA), atypical ductal hyperplasia (ADH), ductal carcinoma in situ (DCIS) and invasive carcinoma (IC). The labels used in the classification results are 0 to 6 respectively; lung cancer includes two subtypes: lung squamous cell carcinoma (LUSC) and lung adenocarcinoma (LUAD), which are represented by labels of 0 and 1 respectively in the data;

[0066] in conclusion

[0067] The Mamba2MIL method of the present invention has important application value in the field of computational pathology, can significantly improve the analysis efficiency and diagnostic accuracy of pathological images, and has broad clinical application prospects. Through sequence enhancement and feature selection, the Mamba2MIL method can effectively capture the sequential information in WSIs, improve classification performance, and provide a valuable auxiliary tool for pathologists.

Claims

1. A pathological image classification method based on state space duality, characterized in that: The process comprises the following steps executed by a computer: Step 1: Create training data: 1.1 Obtain sample data from the pathological image dataset and use a ratio of 8:1:1 to divide the training set, validation set, and test set; 1.2 Preprocess the whole slice images (WSIs) by dividing the images into non-overlapping patches of the same size and filtering out the background areas to ensure the validity and consistency of the model input data; Step 2: Train the Convolutional Neural Network Model 2.1 Build the Mamba2MIL model, preprocess the data, perform feature extraction, serialization of square sequences and reordered sequences, application of the state-space dual model, feature stacking and concatenation, feature selection, and make classification decisions through a multi-layer perceptron; 2.2 Set training parameters, including model learning optimization method, learning rate and maximum number of iterations, and use early stopping technology; The training process includes forward propagation to calculate the loss function and back propagation to update the weights of the neural network; Train until the preset convergence condition is reached; the convergence condition is when the number of training epochs is greater than or equal to 50, and the AUC value of the validation set for more than 20 training cycles is not higher than the AUC value of the best validation set; 2.3 After the training is completed, the final trained model parameters are saved to obtain the trained Mamba2MIL model, which is used to receive actual pathological images and output classification results.

2. The method according to claim 1, characterized in that The model uses square sequences and reordering sequences to standardize the sizes of different WSIs. After the WSIs are divided into patches, they are embedded into feature vectors using a pre-trained ResNet. This process generates a sequence instance feature S = {x1, x2, …, x N }, where x i ∈R D represents instance features, N represents sequence length, and D represents feature dimension. The square sequence method includes: l concatenated with the padding feature vector to form a new sequence S′, whose length L is the square root of the original sequence length N rounded up; this step is done by the formula S′=Concat(S l ,(x1,x2,…,x M )) is implemented, where And M = LN to ensure that WSIs of different sizes can be processed uniformly; The reordering sequence method includes reordering the patch sequence in the pathological image to generate three different sequences: the original sequence (S o ), reverse sequence (S f ) and the transposed sequence (S t ), these sequences are modeled by stacked SSD modules, thereby generating features F corresponding to each sequence respectively. O 、F f and F t , to capture feature information in different orders, the formula is: F = Linear (SSD (Conv1D (Linear (S)))), F ∈ {F O ,F f ,F t },S∈{S o ,S f ,S t }Where Linear is the linear layer and Conv1D is the convolutional layer.

3. The method according to claim 1, characterized in that: The state-space dual model (SSD) is used to effectively model small patches of WSIs in long sequences. The SSD model is based on the simplified SSM and achieves efficient processing of sequences by specially designing the parameters of the three matrices (A, B, C); In the SSD model, the matrix A is designed as a diagonal matrix with all diagonal elements being the same and other elements being zero, so that it can be simplified to a scalar parameter. This design significantly reduces the computational complexity of the SSD model while maintaining the SSD model's ability to capture sequence information. The core formula of the SSD model is: t =Ax t-1 +Bu t ,y t =Cx t Among them, x t is the hidden state at time t, y t is the output at time t, u t is the input at time t, A, B, and C are the state transition matrix, input matrix, and output matrix respectively.

4. The method according to claim 1, characterized in that After using stacked SSD blocks to capture the features of different sequences, weighted feature selection is implemented using the hyperbolic tangent activation function; feature selection is implemented through the hyperbolic tangent activation function (Tanh) and the linear layer (Linear), as shown in the formula: F c =Concat(F O ,F f ,F t ),F c ′=Linear(tanh(Linear(F c ))),R=F c ×Softmax(F c ′), where F c represents the concatenated features, F c ′ represents the weight vector, and R represents the result after weighted feature selection; first, the input features are mapped to a lower dimensional space through a linear layer; then, the Tanh activated features are mapped into weight vectors through another linear layer, and these weights are normalized by the Softmax function and multiplied with the concatenated features to obtain weighted feature representations; finally, the model performs further nonlinear transformation on the weighted feature representation through a multi-layer perceptron (MLP) block to generate the final classification result.

5. An information processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

Citation Information

Patent Citations

  • Breast cancer full-size pathological image classification method based on attention multi-instance learning

    CN114998647A

  • Encrypted network traffic classification method based on structured state space dual model

    CN120067803A

  • State space duality multi-instance pathological image classification method and system

    CN120125905A

  • Lightweight zero-sample image classification method and system based on state space model

    CN120339693A

  • End-to-End Attention Pooling-Based Classification Method for Histopathology Images

    US20220188573A1

Cited By

  • Sodium-ion battery hard carbon roller furnace real-time monitoring method based on machine vision

    CN119477894A

  • Real-time monitoring method of hard carbon roller furnace for sodium ion batteries based on machine vision

    CN119477894B

  • Pathological image classification system and method based on Mama and hybrid expert model

    CN119851017A

  • High-resolution medical pathology WSI image analysis method based on scanning invariance Mama

    CN120070283A

  • High-resolution medical pathology WSI image analysis method based on scan-invariant Mamba

    CN120070283B