State space duality multi-instance pathological image classification method and system

Abstract

The invention relates to the field of medical auxiliary diagnosis, in particular to a state space duality multi-instance pathological image classification method and system, and the method comprises the following steps: obtaining a lung cancer pathological full-slice image to be detected based on a pathological database; obtaining a tissue area of the lung cancer pathological full-slice image, and cutting the tissue area to obtain an image block set; obtaining a feature vector set of the image block set by using a feature extractor, constructing a fusion feature set based on the feature vector set, and obtaining a weighted feature set according to the fusion feature set; performing dichotomy mapping on the weighted feature set according to a linear classifier to obtain a dichotomy prediction probability vector; and obtaining a cancer prediction category of the lung cancer pathological full-slice image according to the dichotomy prediction probability vector so as to classify the cancer type of the lung cancer pathological full-slice image. The method is used for automatic diagnosis of cancer pathological images and provides efficient and reliable technical support for medical auxiliary diagnosis.

Description

Technical Field

[0001] The present invention relates to the field of medical auxiliary diagnosis, and in particular to a method and system for classifying multi-instance pathological images based on state-space duality. Background Art

[0002] With the continuous development of digital pathology, the analysis of pathological images has gradually shifted towards automated diagnosis based on computational methods. Multiple Instance Learning (MIL), as a weakly supervised learning method, has been widely applied to the classification of pathological images. In pathological image analysis, MIL divides a pathological image into multiple smaller image patches (patches), and each patch participates in training as an "instance". These instances are combined into a "Bag", and finally the entire pathological image is classified. However, existing MIL methods usually assume that instances are independent and identically distributed, which will encounter performance bottlenecks when dealing with pathological images because this assumption ignores the contextual relevance and spatial structure information between different patches.

[0003] To solve the above problems, some researchers have proposed strategies that combine order-dependent and order-independent features, aiming to break the performance bottleneck of existing methods when dealing with pathological images and enhance the model's ability to capture complex spatial and structural information in pathological images. Among them, TransMIL[1] is the first MIL method based on Transformer. It uses a linear attention mechanism based on matrix factorization to capture order-independent features between instances. In addition, TransMIL also enhances order-dependent features by using a Pyramid Position Encoding Generator (PPEG). However, PPEG cannot fully meet the effective utilization of order-dependent features, resulting in limited performance in some cases.

[0004] In addition, methods such as MambaMIL[2] and Mamba2MIL[3] have overcome the limitations of order-independent features to a certain extent. Moreover, the SSD model in Mamba2MIL has higher flexibility and adaptability than the SSM model in MambaMIL and has solved the problem of relatively high algorithm complexity of TransMIL to a certain extent. However, although Mamba2 performs feature fusion through a deep neural network, its fusion process is insufficient in combining features of multi-resolution images and lacks effective modeling of the complex relationship between fine-grained features (such as texture features) and coarse-grained features (such as morphological features) of images with different resolutions, resulting in possible omission when capturing context information between different scales. In view of this, it is crucial to develop a method that can effectively enhance the capture ability of order-dependent and order-independent features and efficiently fuse multi-scale feature information. Summary of the Invention

[0005] Aiming at the defects in the prior art, the present invention provides a state space duality multi-instance pathological image classification method and system.

[0006] To achieve the above object, in the first aspect, the present invention provides a state space duality multi-instance pathological image classification method, and the method includes the following steps: obtaining a whole slide image of lung cancer pathology to be detected based on a pathology database; obtaining a tissue region of the whole slide image of lung cancer pathology, and cropping the tissue region to obtain a set of image patches; using a feature extractor to obtain a set of feature vectors of the set of image patches, constructing a fused feature set based on the set of feature vectors, and obtaining a weighted feature set according to the fused feature set; performing binary classification mapping on the weighted feature set by a linear classifier to obtain a binary classification prediction probability vector; obtaining a cancer prediction category of the whole slide image of lung cancer pathology according to the binary classification prediction probability vector, so as to classify the cancer type of the whole slide image of lung cancer pathology. Through multi-instance learning and state space duality design, the present invention significantly improves the accuracy and efficiency of lung cancer pathology image classification, solves the problems of high resolution and small target dispersion of whole slide images while reducing the computational complexity, and provides an automated analysis solution with high precision and low manual intervention for clinical pathological diagnosis.

[0007] Optionally, obtaining the whole slide image of lung cancer pathology to be detected based on the pathology database includes: constructing a screening criterion for lung cancer pathology images, and obtaining the original data of lung cancer pathology images based on the pathology database in combination with the screening criterion; using tissue section staining technology to stain the original data of the lung cancer pathology images to obtain lung cancer pathology stained images; and constructing the whole slide image of lung cancer pathology based on the lung cancer pathology stained images in combination with digital devices. The present invention significantly improves the data quality and analysis efficiency of the whole slide image of lung cancer pathology through a standardized process and digital technology, converts the stained image into a high-resolution digital section, supports multi-scale analysis, realizes permanent data storage, fast retrieval and remote sharing, combines traditional pathology with digital technology, solves the problems of low efficiency of manual section preparation and poor staining consistency, and provides a reliable basis for the classification of lung cancer pathology images.

[0008] Optionally, obtaining the tissue region of the whole slide image of lung cancer pathology and cropping the tissue region to obtain a set of image patches includes: performing foreground segmentation on the whole slide image of lung cancer pathology using an image segmentation algorithm to obtain the tissue region; cropping the tissue region at a first magnification and a second magnification to obtain a first image patch and a second image patch respectively; obtaining the coordinates of the first image patch and the second image patch in the whole slide image of lung cancer pathology, the first image patches forming a first set of image patches, and the second image patches forming a second set of image patches. The present invention accurately extracts the tissue region through foreground segmentation using an image segmentation algorithm, effectively avoiding the interference of non-tissue parts, cropping the tissue region at different magnifications to obtain a set of image patches, not only retaining tissue details (high magnification) but also retaining overall layout information (low magnification), and at the same time, recording the coordinates of each image patch in the original image for subsequent analysis and positioning, providing a high-quality data basis for in-depth analysis of lung cancer pathology images.

[0009] Optionally, the step of obtaining a feature vector set of the set of image patches by using a feature extractor, constructing a fused feature set based on the feature vector set, and obtaining a weighted feature set according to the fused feature set includes: inputting the first set of image patches and the second set of image patches into the feature extractor to respectively obtain a first feature vector set and a second feature vector set; constructing a pre-fused feature set according to the first feature vector set and the second feature vector set; reordering the pre-fused feature set and the first feature vector set to respectively obtain a pre-fused feature sequence set and a first feature sequence set; constructing a first feature set and a second feature set respectively according to the pre-fused feature sequence set and the first feature sequence set based on a state space duality sequence model; performing feature fusion on the first feature set and the second feature set to obtain a post-fused feature set; and obtaining the weighted feature set weighted by an attention mechanism according to the post-fused feature set based on a feature aggregation model. In the present invention, a feature extractor is used to extract feature vectors from sets of image patches at different magnification ratios, enhancing the diversity of information. A pre-fused feature set and a feature sequence set are constructed, and in combination with a state space duality sequence model, a first feature set and a second feature set are respectively constructed, not only integrating multi-level information but also enhancing the robustness of the features. The feature sets are fused and weighted by using a feature aggregation model and an attention mechanism, not only highlighting key features but also suppressing noise interference, greatly improving the effectiveness of feature representation and providing strong support for the accurate analysis of lung cancer pathological images.

[0010] Optionally, the step of constructing a pre-fused feature set according to the first feature vector set and the second feature vector set includes: obtaining a second feature vector subset of all sub-images in the first set of image patches at the second magnification ratio; and performing feature fusion on the first feature vector set and the second feature vector subset to obtain the pre-fused feature set. In the present invention, by extracting the second feature vector subset corresponding to each sub-image in the first set of image patches at the second magnification ratio and fusing it with the first feature vector set, not only the image information at different resolutions is combined, but also the feature dimension is enriched, enhancing the comprehensiveness and complementarity of the features. The pre-fused feature set provides a more detailed and accurate data basis for subsequent analysis.

[0011] Optionally, reordering the pre-fusion feature set and the first feature vector set respectively to obtain a pre-fusion feature sequence set and a first feature sequence set includes: constructing a sorting model, the functions of the sorting model including horizontal flipping sorting and random reordering; reordering the pre-fusion feature set according to the sorting model to obtain the pre-fusion feature sequence set, the pre-fusion feature sequence set including a pre-fusion feature original sequence set, a pre-fusion feature flipped sequence set, and a pre-fusion feature random sequence set; reordering the first feature vector set according to the sorting model to obtain the first feature sequence set, the first feature sequence set including a first feature original sequence set, a first feature flipped sequence set, and a first feature random sequence set. By reordering the pre-fusion feature set and the first feature vector set, the present invention not only increases the diversity of data, but also helps the model learn more robust feature representations. The pre-fusion feature sequence set and the first feature sequence set provide rich perspectives for subsequent analysis, which helps to improve the adaptability to image feature changes.

[0012] Optionally, the state space duality sequence model includes: Among them, is the output feature set, represents a linear layer, represents a structured state space diagonal model, represents a causal convolutional layer, is the input feature set, is the output original sequence set, is the output flipped sequence set, is the output random sequence set, is the input original sequence set, is the input flipped sequence set, is the input random sequence set. The present invention utilizes the state space duality sequence model, processes the input feature set through the structured state space diagonal model and the causal convolutional layer, and combines the linear layer to output different feature sets, which not only enhances the feature expression ability, but also improves the sensitivity of the model to sequence changes, helps to capture richer information, and provides a solid foundation for subsequent feature fusion and weighting.

[0013] Optionally, obtaining the weighted feature set weighted by the attention mechanism based on the post-fusion feature set includes: normalizing the post-fusion feature set to obtain a normalized feature set; calculating attention weights according to the attention mechanism, the attention mechanism including two-layer linear transformation and an activation function, satisfying the following relationship: Among them, is the attention weight, is the weight matrix of the second-layer linear transformation, represents the activation function, is the weight matrix of the first-layer linear transformation, is the normalized feature set; the normalized attention weight is obtained by transposing the attention weight, and the weighted feature set is obtained by weighted summation of the normalized feature set according to the normalized attention weight, satisfying the following relationship: Among them, is the normalized attention weight, is the non-linear function, is the attention weight, represents transpose, is the weighted feature set, is the normalized feature set. By normalizing the post-fusion feature set, calculating the attention weight in combination with the attention mechanism, and enhancing the feature expression by using two-layer linear transformation and activation function, the present invention effectively highlights the importance of key features. The application of the normalized attention weight further ensures the rationality of weighting. The weighted feature set obtained by weighted summation of the normalized feature set not only improves the quality of the features, but also enhances the model's ability to capture key information, providing a more accurate and reliable feature representation for subsequent pathological image analysis.

[0014] Optionally, the binary classification prediction probability vector is obtained by performing binary classification mapping on the weighted feature set according to the linear classifier, including: compressing the weighted feature set to obtain a compressed feature set, and obtaining a classification score in combination with the linear classifier; normalizing the classification score to obtain a class probability distribution, and obtaining the highest probability class index according to the class probability distribution, and the highest probability class index is used as the binary classification prediction probability vector. By compressing the weighted feature set and applying the linear classifier, the present invention not only simplifies the feature space, but also improves the classification efficiency. After the obtained classification score is normalized, it can intuitively reflect the class probability distribution, facilitating the determination of the highest probability class index, providing a reliable binary classification prediction probability vector, and helping to improve the accuracy and stability of lung cancer pathological image classification.

[0015] Second aspect, the present invention provides a state space duality multi-instance pathological image classification system. The system executes the state space duality multi-instance pathological image classification method provided by the present invention. The system includes an input device, a processor, an output device, and a memory; the input device includes an image acquisition and preprocessing module, the processor includes a feature extractor, a pre-fusion module, a state space duality sequence module, a post-fusion module, a feature aggregation module, and a binary classification module, and the output device includes a result output module; the input device, the output device, the processor, and the memory are interconnected to construct an efficient information processing system. The present invention realizes an efficient information processing flow from pathological image acquisition to binary classification prediction by integrating input, processing, and output modules. The modules work together, not only improving the processing speed, but also ensuring the accuracy of classification.

[0016] In summary, the beneficial effects of the present invention include: making full use of multi-scale feature information, through the pre-fusion and post-fusion modules, it can effectively fuse the features of low-magnification and high-magnification images, thereby capturing the fine-grained and coarse-grained features in cancer pathological images, enhancing the judgment ability of the state space duality sequence model for cancer types; obtaining multiple inputs in different orders through the sorting model, providing a multi-perspective learning method for the state space duality sequence model, thereby enhancing the local feature capture ability of the state space duality sequence model, avoiding information loss that may be caused by a fixed order, and at the same time, through the introduction of a structured state space diagonal model, the state space duality sequence model can effectively process long sequences and global context information in pathological images, improving the comprehensive understanding of the state space duality sequence model for lung cancer pathological images and effectively improving the classification accuracy; by combining multi-resolution pathological image features, using the state space dual sequence model to enhance the ability to capture cancer features, and optimizing feature fusion through the attention mechanism, the accuracy and robustness of cancer classification are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of a state space duality multi-instance pathological image classification method according to an embodiment of the present invention; Figure 2 It is a schematic diagram of the execution of a state space duality multi-instance pathological image classification method according to an embodiment of the present invention; Figure 3 It is a framework diagram of a state space duality multi-instance pathological image classification system according to an embodiment of the present invention; Figure 4 It is a schematic diagram of the modules of a state space duality multi-instance pathological image classification system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0018] Specific embodiments of the present invention will be described in detail below. It should be noted that the embodiments described here are only for illustrative purposes and are not intended to limit the present invention. In the following description, in order to provide a thorough understanding of the present invention, a large number of specific details are set forth. However, it will be apparent to those of ordinary skill in the art that the present invention does not have to employ these specific details. In other instances, well-known circuits, software, or methods have not been specifically described in order to avoid obscuring the present invention.

[0019] Throughout the specification, the mention of "one embodiment", "an embodiment", "one example" or "an example" means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment of the present invention. Thus, the phrases "in one embodiment", "in an embodiment", "one example" or "an example" that appear throughout the specification do not necessarily all refer to the same embodiment or example. In addition, the specific features, structures, or characteristics may be combined in any suitable combination and / or sub-combination in one or more embodiments or examples. In addition, those of ordinary skill in the art should understand that the diagrams provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0020] Please refer to Figure 1 , an embodiment of the present invention provides a state-space duality multi-instance pathological image classification method, and the method includes the following steps: S1. Obtain a whole-slide image of lung cancer pathology to be detected based on a pathology database.

[0021] In this embodiment, an image acquisition module is established to obtain a whole-slide image of lung cancer pathology; first, a strict screening standard for lung cancer pathology images is constructed, and staff collect the original data of lung cancer pathology images from the pathology database of the hospital according to the strict screening standard; subsequently, the hematoxylin-eosin staining method (H&E staining) is used as the tissue section staining technique to stain the original data of lung cancer pathology images to obtain a stained image of lung cancer pathology; finally, the stained image of lung cancer pathology is converted into a whole-slide image of lung cancer pathology (WSI) through a digital device; ensure that the obtained whole-slide image of lung cancer pathology has good quality and no obvious stains, scratches and other factors affecting subsequent analysis.

[0022] S2. Obtain the tissue region of the whole-slide image of lung cancer pathology, and crop the tissue region to obtain a set of image patches.

[0023] Specifically, an image preprocessing module is established to preprocess the whole-slide image of lung cancer pathology.

[0024] Wherein, S2 specifically includes the following steps: S21. Use an image segmentation algorithm to perform foreground segmentation on the whole-slide image of lung cancer pathology to obtain the tissue region.

[0025] In this embodiment, a U-Net model based on deep learning or a traditional threshold segmentation algorithm is selected as the image segmentation algorithm. The image segmentation algorithm is used to perform foreground segmentation on the whole slide image of lung cancer pathology, accurately detect the tissue region of the whole slide image of lung cancer pathology, effectively remove background information, and reduce subsequent computational volume and interference.

[0026] S22. Crop the tissue region at the first magnification and the second magnification to obtain a first image patch and a second image patch respectively.

[0027] The magnification is selected according to actual needs. In this embodiment, based on different magnifications, the tissue region is cropped into image patches (patches, such as 224×224 pixels) of the same size as multi-instances. 5x is selected as the first magnification, and 20x is selected as the second magnification. The tissue region is cropped according to the first magnification to obtain the first image patch, and the tissue region is cropped according to the second magnification to obtain the second image patch.

[0028] S23. Obtain the coordinates of the first image patch and the second image patch in the whole slide image of lung cancer pathology. The first image patches form a first image patch set, and the second image patches form a second image patch set.

[0029] During the cropping process, carefully record the coordinates of each image patch in the whole slide image of lung cancer pathology. All image patches form an image patch set (Bag). The number of image patches in each image patch set is different. At the same time, discard the background image patches with entropy values less than a certain value.

[0030] S3. Use a feature extractor to obtain a feature vector set of the image patch set, construct a fused feature set based on the feature vector set, and obtain a weighted feature set according to the fused feature set.

[0031] Among them, S3 specifically includes the following steps: S31. Input the first image patch set and the second image patch set into the feature extractor to obtain a first feature vector set and a second feature vector set respectively.

[0032] In this embodiment, the SimCLR method is used to train the feature extractor. During the training process, the Adam optimizer is adopted, the initial learning rate is set to 0.0001, the learning rate scheduling uses the cosine annealing (without warm restart) strategy, and the mini-batch size is set to 512.

[0033] Further, all the image patches (multiple instances) in the image patch set are input into the trained feature extractor to extract the feature map sets of each image patch, thereby obtaining a feature vector set. It should be noted that the size of the last-dimensional feature dimension of the feature map set after feature extraction is 512.

[0034] S32. Construct a pre-fusion feature set based on the first feature vector set and the second feature vector set.

[0035] In this embodiment, a pre-fusion module is established to obtain a pre-fusion feature set; the feature map set of the first image patch is obtained at a magnification of 5x, thereby obtaining a first feature vector set; multiple copies of the first feature vector set at a magnification of 5x are replicated and concatenated with the second feature vector subsets at a magnification of 20x corresponding to each sub-image in the first image patch set, thereby constructing a pre-fusion feature set.

[0036] S33. Reorder the pre-fusion feature set and the first feature vector set respectively to obtain a pre-fusion feature sequence set and a first feature sequence set.

[0037] In this embodiment, a sorting module is constructed to obtain a sorting model, whose functions include horizontal flip sorting and random reordering; all the image patches in the cropped image patch set are sorted according to their coordinates to obtain a feature original sequence set; the feature original sequence set is horizontally flipped to obtain a feature flipped sequence set with reversed positions, and the feature original sequence set is randomly reordered to obtain a feature random sequence set with random positions; the feature original sequence set, the feature flipped sequence set, and the feature random sequence set form a sequence set; the following relationship is satisfied: Among them, is the feature original sequence set, is the feature flipped sequence set, represents sequence flip sorting, represents sequence random sorting, is the feature random sequence set, is the sequence set.

[0038] According to the sorting model, the pre-fusion feature set is reordered to obtain the pre-fusion feature sequence set, and the pre-fusion feature sequence set includes a pre-fusion feature original sequence set, a pre-fusion feature flipped sequence set, and a pre-fusion feature random sequence set; the following relationship is satisfied: Among them, is the set of pre-fusion feature sequences, is the set of original pre-fusion feature sequences, is the set of flipped pre-fusion feature sequences, is the set of random pre-fusion feature sequences.

[0039] The first feature vector set is re-ordered according to the sorting model to obtain the first feature sequence set, which includes the first feature original sequence set, the first feature flipped sequence set, and the first feature random sequence set; the following relationship is satisfied: where, is the first feature sequence set, is the first feature original sequence set, is the first feature flipped sequence set, is the first feature random sequence set.

[0040] S34. Based on the state space duality sequence model, a first feature set and a second feature set are respectively constructed according to the pre-fusion feature sequence set and the first feature sequence set.

[0041] In this embodiment, a state space duality sequence model is constructed based on the state space duality sequence module (Mamba2 module). The feature set is input into a linear layer, then into a causal convolution layer, and then into a structured state space diagonal model, and finally through another linear layer to obtain the output feature set, satisfying the following relationship: where, is the output feature set, represents the linear layer, represents the structured state space diagonal model, represents the causal convolution layer, is the input feature set, is the output original sequence set, is the output flipped sequence set, is the output random sequence set, is the input original sequence set, is the input flipped sequence set, is the input random sequence set.

[0042] The Mamba2 module adopts some optimized algorithms and structures, which can achieve fast calculation while maintaining high precision. It significantly reduces the time required for feature extraction in processing large-scale multi-instance data of WSIs, and improves the training and inference efficiency of the entire model.

[0043] Furthermore, set the parameters of the state space duality sequence model as the first state space duality sequence model, and input the pre-fusion feature sequence set into the first state space duality sequence model to obtain the first feature set; set different parameters of the state space duality sequence model as the second state space duality sequence model, and input the first feature sequence set into the second state space duality sequence model to obtain the second feature set.

[0044] By inputting lung cancer pathological whole slide images into the model in different orders (original order, reverse order, random order), the sorting model provides multiple perspectives for the state space duality sequence model to observe and process pathological images; different sequence orders can help the state space duality sequence model focus on local feature correlations at different levels, thus improving the learning ability of the state space duality sequence model and its understanding of pathological images.

[0045] S35. Feature-fuse the first feature set and the second feature set to obtain a post-fusion feature set.

[0046] Specifically, establish a post-fusion module to obtain a post-fusion feature set; perform re-feature splicing on the first feature set and the second feature set in the first dimension to achieve the effect of post-fusion, and obtain a post-fusion feature set, which satisfies the following relationship: Among them, is the post-fusion feature set, represents the set of real numbers, is the batch size. In this example, , is the dimension of the spliced sequence, and 512 is the feature dimension.

[0047] S36. Based on the post-fusion feature set, obtain the weighted feature set weighted by the attention mechanism based on the feature aggregation model.

[0048] In this embodiment, establish a feature aggregation module to obtain a feature aggregation model, normalize the post-fusion feature set to obtain a normalized feature set; calculate the attention weights according to the attention mechanism, and the attention mechanism includes two layers of linear transformation and an activation function, which satisfies the following relationship: Among them, is the attention weight, is the weight matrix of the second layer of linear transformation, represents the activation function, is the weight matrix of the first-layer linear transformation, is the normalized feature set.

[0049] Further, transpose the attention weights and apply the Softmax function on the last dimension to obtain the normalized attention weights, and perform weighted summation on the normalized feature set according to the normalized attention weights to obtain the weighted feature set, satisfying the following relationship: where, is the normalized attention weight, is the non-linear function, is the attention weight, represents transpose, is the weighted feature set, is the normalized feature set.

[0050] S4. Perform binary classification mapping on the weighted feature set according to the linear classifier to obtain a binary classification prediction probability vector.

[0051] Specifically, establish a binary classification module (MLP Block module) and obtain a binary classification prediction probability vector based on the weighted feature set.

[0052] Among them, S4 specifically includes the following steps: S41. Compress the weighted feature set to obtain a compressed feature set, and combine it with the linear classifier to obtain a classification score.

[0053] In this embodiment, the weighted feature set is compressed by a compression operation to remove the dimension with dimension 1, and the compressed feature set after compression is obtained, satisfying the following relationship: where, is the compressed feature set, represents the compression operation, is the weighted feature set.

[0054] Further, input the compressed feature set into the linear classifier to obtain a classification score, satisfying the following relationship: where, is the classification score, is the number of classes, is the weight matrix of the linear classifier, is the compressed feature set, is the bias term.

[0055] S42. Normalize the classification scores to obtain a class probability distribution, and obtain the highest probability class index based on the class probability distribution. The highest probability class index is used as the binary classification prediction probability vector.

[0056] In this embodiment, Softmax normalization is performed on the classification scores to obtain a class probability distribution, which satisfies the following relationship: where, is the class probability distribution, is a non-linear function, is the classification score.

[0057] Furthermore, obtain the highest probability class index from the class probability distribution as the binary classification prediction probability vector, which satisfies the following relationship: where, is the highest probability class index, represents finding the class index where the highest probability is located, is the class probability distribution, is the dimension.

[0058] S5. Obtain the cancer prediction class of the lung cancer pathological whole slide image based on the binary classification prediction probability vector, and classify the cancer type of the lung cancer pathological whole slide image.

[0059] Specifically, establish a result output module to output the classification result; obtain the cancer prediction class of the lung cancer pathological whole slide image based on the binary classification prediction probability vector, and determine which cancer type the current lung cancer pathological section belongs to, so as to classify the cancer type of the lung cancer pathological whole slide image.

[0060] Please refer to Figure 2 , which is a schematic diagram showing the execution process of a state space duality multi-instance pathological image classification method; it details the execution step process of the method of the present invention, and focuses on showing the processes of feature splicing, sorting module, and state space duality sequence model.

[0061] In an alternative embodiment, the TCGA-NSCLC dataset is used for experiments. The dataset contains 1054 whole slide images (WSIs), covering two subtypes of lung cancer, namely lung squamous cell carcinoma (LUSC) and lung adenocarcinoma (LUAD).

[0062] During the training and optimization of the MIL model, the Adam optimizer is used, and the learning rate is fixed at 0.0001 to update the model weights. The training mini-batch size is 1 (i.e., each package is used as a batch), and the epoch size is set to 50.

[0063] AUC (Area Under Curve) and ACC (Accuracy) are used as indicators to evaluate the classification performance of the model, and 5-fold cross-validation is adopted for parameter tuning to ensure the reliability of the model performance.

[0064] Control experiments are carried out using five baseline methods and the method of the present invention. The model of the present invention has obtained the optimal results in various indicators. The results of the control experiments are shown in Table 1: Table 1 In order to deeply understand the role of the fusion module in the model, ablation experiments are carried out. The cases of only the front fusion module (FC), only the back fusion module (LC), and both of them are tested respectively. The results of the ablation experiments are shown in Table 2: Table 2 According to the results of the ablation experiments, the AUC and ACC of only using the front fusion module and only using the back fusion module are both higher than the effect of only using the 20x magnification feature set. And when both are used, the AUC reaches the highest, which fully verifies the effectiveness of the front fusion module and the back fusion module in improving the model performance.

[0065] Through the ways of front fusion and back fusion, the present invention can better combine the features from low-magnification images and high-magnification images; the low-magnification images provide the overall spatial structure information, while the high-magnification images provide the cell-level information with rich details; by reasonably fusing the features of different magnification images, the present invention can comprehensively capture the key features of cancer types, especially the relationship between morphological features and texture features.

[0066] Please refer to Figure 3 , in an optional embodiment, the present invention provides a state space duality multi-instance pathological image classification system. The system includes an input device, a processor, an output device, and a memory; the input device includes an image acquisition and preprocessing module, the processor includes a feature extractor, a front fusion module, a state space duality sequence module, a back fusion module, a feature aggregation module, and a binary classification module, the output device includes a result output module; the input device, the processor, the output device, and the memory are interconnected. Among them, the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to efficiently execute the state space duality multi-instance pathological image classification method provided by the present invention.

[0067] Specifically, please refer to Figure 4 , which is shown as a schematic diagram of a state-space duality multi-instance pathological image classification system module. The system includes: Image acquisition and preprocessing module: It is used to obtain the H&E-stained lung cancer pathological whole-slide image (WSI) to be detected, perform foreground segmentation, crop image patches of the same size at different magnifications, record the coordinates, and form a Bag; Feature extractor: It is used to send the instances in the Bag into it to obtain the feature map sets of all instances in each Bag at different magnifications; Pre-fusion module: Copy and splice the feature sets at different magnifications, and then input them into the Mamba2 module after passing through the sorting module; Mamba2 module: Process the input pre-fused feature set, and is also used to process the separate 5x magnification feature vector set in the post-fusion module; Post-fusion module: Splice and fuse the feature set after the pre-fusion processed by the Mamba2 module with the feature set after the separate 5x magnification feature vector set processed by the Mamba2 module; Feature aggregation module: Normalize the fused feature set, and then perform weighted summation through the attention mechanism to obtain the final output feature; Binary classification module: Input the final output feature into it to obtain the classification prediction probability; Result output module: Obtain the cancer prediction category of the WSI according to the final binary classification probability vector.

[0068] In summary, a state-space duality multi-instance pathological image classification method and system provided by the method of the present invention, by combining multi-resolution pathological image features, using the state-space duality sequence model to enhance the ability to capture cancer features, and optimizing feature fusion through the attention mechanism, significantly improves the accuracy and robustness of cancer classification. The present invention can be widely applied to the automated cancer diagnosis of pathological images, providing efficient and reliable technical support for medical auxiliary diagnosis; the method of the present invention is easy to understand, simple in calculation, and has a small workload, providing a theoretical basis and technical support for the further development of medical auxiliary diagnosis.

[0069] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered by the scope of the claims and the description of the present invention.

Claims

1. A state-space duality multi-instance pathological image classification method, characterized in that: The steps include: Acquire the full-slice image of the lung cancer pathology to be detected based on the pathology database; Acquiring a tissue region of the lung cancer pathology full-slice image, and cropping the tissue region to obtain an image block set; Using a feature extractor to obtain a feature vector set of the image block set, constructing a fused feature set based on the feature vector set, and obtaining a weighted feature set according to the fused feature set; Performing binary classification mapping on the weighted feature set according to a linear classifier to obtain a binary classification prediction probability vector; The cancer prediction category of the lung cancer pathology full-slice image is obtained according to the binary classification prediction probability vector, so as to classify the cancer type of the lung cancer pathology full-slice image.

2. The state-space duality multi-instance pathological image classification method according to claim 1, characterized in that: The method of obtaining the full-slice image of the lung cancer pathology to be detected based on the pathology database includes: Constructing a screening standard for lung cancer pathology images, and obtaining original data of lung cancer pathology images based on the pathology database and the screening standard; Using tissue section staining technology to stain the original data of the lung cancer pathology image to obtain a lung cancer pathology staining image; Based on the lung cancer pathology staining image, the lung cancer pathology full-slice image is constructed in combination with a digitizing device.

3. The state-space duality multi-instance pathological image classification method according to claim 1, characterized in that: The step of acquiring the tissue region of the lung cancer pathology full-slice image and cropping the tissue region to obtain an image block set includes: Using an image segmentation algorithm to perform foreground segmentation on the lung cancer pathology full-slice image to obtain the tissue region; Cropping the tissue region at a first magnification and a second magnification to obtain a first image block and a second image block respectively; The coordinates of the first image block and the second image block in the lung cancer pathology full-slice image are acquired, the first image blocks constitute a first image block set, and the second image blocks constitute a second image block set.

4. The state-space duality multi-instance pathological image classification method according to claim 3, characterized in that: The method of obtaining a feature vector set of the image block set by using a feature extractor, constructing a fused feature set based on the feature vector set, and obtaining a weighted feature set according to the fused feature set includes: Inputting the first image block set and the second image block set into the feature extractor to obtain a first feature vector set and a second feature vector set respectively; Constructing a pre-fusion feature set according to the first feature vector set and the second feature vector set; Reordering the pre-fusion feature set and the first feature vector set to obtain a pre-fusion feature sequence set and a first feature sequence set respectively; Based on the state space duality sequence model, a first feature set and a second feature set are respectively constructed according to the pre-fusion feature sequence set and the first feature sequence set; Performing feature fusion on the first feature set and the second feature set to obtain a post-fusion feature set; According to the post-fusion feature set, the weighted feature set after weighting by the attention mechanism is obtained based on the feature aggregation model.

5. The state-space duality multi-instance pathological image classification method according to claim 4, characterized in that: The constructing a pre-fusion feature set according to the first feature vector set and the second feature vector set includes: Under the second magnification, obtaining a second feature vector subset of all sub-images in the first image block set; The first feature vector set and the second feature vector subset are subjected to feature fusion to obtain the pre-fusion feature set.

6. The state-space duality multi-instance pathological image classification method according to claim 4, characterized in that: The reordering of the pre-fusion feature set and the first feature vector set to obtain a pre-fusion feature sequence set and a first feature sequence set respectively includes: Constructing a sorting model, wherein the functions of the sorting model include horizontal flip sorting and random reordering; Reordering the pre-fusion feature set according to the sorting model to obtain the pre-fusion feature sequence set, wherein the pre-fusion feature sequence set includes a pre-fusion feature original sequence set, a pre-fusion feature flipped sequence set, and a pre-fusion feature random sequence set; The first feature vector set is reordered according to the sorting model to obtain the first feature sequence set, where the first feature sequence set includes a first feature original sequence set, a first feature flipped sequence set, and a first feature random sequence set.

7. The state-space duality multi-instance pathological image classification method according to claim 4, characterized in that: The state space duality sequence model comprises: in, is the output feature set, represents a linear layer, represents the structured state space diagonal model, represents the causal convolutional layer, is the input feature set, To output the original sequence set, is the output flip sequence set, To output a random sequence set, As the input original sequence set, is the input flip sequence set, is a set of random input sequences.

8. The state-space duality multi-instance pathological image classification method according to claim 4, characterized in that: The step of obtaining the weighted feature set weighted by the attention mechanism based on the post-fusion feature set and the feature aggregation model includes: Normalizing the post-fusion feature set to obtain a normalized feature set; The attention weight is calculated according to the attention mechanism, which includes two layers of linear transformation and activation function, satisfying the following relationship: in, is the attention weight, is the weight matrix of the second layer linear transformation, represents the activation function, is the weight matrix of the first layer linear transformation, is the normalized feature set; The attention weight is transposed to obtain a normalized attention weight, and the normalized feature set is weighted summed according to the normalized attention weight to obtain the weighted feature set, satisfying the following relationship: in, is the normalized attention weight, is a nonlinear function, is the attention weight, represents transpose, is the weighted feature set, is the normalized feature set.

9. The state-space duality multi-instance pathological image classification method according to claim 1, characterized in that: The step of performing binary mapping on the weighted feature set according to the linear classifier to obtain a binary classification prediction probability vector includes: Compressing the weighted feature set to obtain a compressed feature set, and combining it with the linear classifier to obtain a classification score; The classification scores are normalized to obtain a category probability distribution, and a highest probability category index is obtained based on the category probability distribution. The highest probability category index is used as the binary classification prediction probability vector.

10. A state-space duality multi-instance pathological image classification system, characterized in that: The system includes an input device, a processor, an output device, and a memory; The input device includes an image acquisition and preprocessing module, the processor includes a feature extractor, a pre-fusion module, a state space duality sequence module, a post-fusion module, a feature aggregation module and a binary classification module, and the output device includes a result output module; The input device, the processor, the output device and the memory are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the state-space duality multi-instance pathology image classification method described in any one of claims 1-9.

Images